Multi-task learning based feature extraction method in signal processing of high resolution remote sensing video images

The channel attention mechanism and the core idea is that not all channels are equally important in a given feature response (i.e., the feature map or feature representation generated by the convolutional layer).

A pooling operation, including average pooling and global pooling, is first performed on each channel to obtain the global statistical features for that channel. This operation reduces the 2D feature map to a single scalar value, which provides the basis for subsequent learning of each channel weight. Next, a small multilayer perceptron (typically containing one or two hidden layers) is used to learn the weights of each channel from the globally pooled features; the number of hidden layers, as well as their dimensions, need to be decided based on the specific application. Finally, a Sigmoid activation function is used to normalize these weights such that each of them has a weight between 0 and 1. These weights are subsequently used to weight the original feature channels, thus improving the network’s sensitivity to important features. Based on the above description, the channel attention is computed using the formula below: (1)Mc(F) = σ(MLP(Avgpool(F))+MLP(Maxpool(F))) = σ(W1(W0(FAvR))+W1(W0(FMax)))$$\begin{array}{rcl} {M_c}(F) &=& \sigma (MLP(Avgpool(F)) + MLP(Maxpool(F))) \\ &=& \sigma ({W_1}({W_0}({F_{A{v_R}}})) + {W_1}({W_0}({F_{Max}}))) \\ \end{array}$$

where σ is the Sigmoid function, F is of size C × H × W, where r is the decreasing rate, and MLP weights W₀ and W₁ are shared parameters for both inputs.

The spatial attention mechanism is an algorithm used to enhance the attention of a neural network to features at specific spatial locations in the input data. In visual tasks, this mechanism recognizes that pixels at different locations in an image contain varying degrees of informativeness, and that certain regions may be more critical to the completion of a particular task. The introduction of spatial attention allows the model to dynamically adjust the focus of its processing, similar to how the human visual system prioritizes those parts of a scene that are meaningful when observing it. In standard CNNs, the convolutional operations at each layer give equal importance to each spatial location of the input feature map. However, in numerous visual tasks, different spatial locations contribute differently to the final output decision. Spatial attention mechanisms are proposed to enable the model to automatically identify and focus on the most useful regions, thereby improving performance. A typical spatial attention module usually takes the following steps:

First global average pooling and maximum pooling are performed, and then the feature maps after global average pooling and maximum pooling are spliced to obtain a feature map dimension of H × W × 2. This captures the global context and helps the network understand which locations are more important. Next a convolution operation is performed on the aforementioned pooled features using a 1×1 convolution kernel to produce a spatial attention graph. The size of this map is identical to that of the input feature map, and each value is associated with the weight of the corresponding position in the input feature map. The weights are then normalized to between 0 and 1 using a Sigmoid activation function. Finally, the normalized spatial attention map is subjected to element-level multiplication operations with the original feature map to enhance or suppress features at specific locations. The computational formula is shown below: (2)Ms(F) = σ(f1×1([Avgpool(F);Maxpool(F)])) = σ(f1×1([FAvg;FMax]))$$\begin{array}{rcl} {M_s}(F) &=& \sigma ({f^{1 \times 1}}([Avgpool(F);Maxpool(F)])) \\ &=& \sigma ({f^{1 \times 1}}([{F_{Avg}};{F_{Max}}])) \\ \end{array}$$

Again, σ is a Sigmoid function, F is of size C × H × W, and f is a 1 × 1-convolution.

In order to enhance the linkages within the features, two self-attention modules are proposed by introducing DANet.

In the positional attention module, a connection is established for the input local features in the positional dimension by first transforming the features into the form of Q, K, and V using convolution, and in order to model the positional features, it is necessary to match the shapes of Q, K, and V to obtain an (H × W) × (H × W) attention weight map, and then dot-multiply the weight map by V to restore it to the original dimensions. In addition, the positional attention module introduces a short-circuit connection, which dot-multiplies the input features with the attention weight map, and the result obtained is summed with the original features, which is beneficial for the transfer of gradients during back propagation and avoids the problems of gradient vanishing and gradient explosion. It is expressed as Eq. (3) and Eq. (4) using Eq: (3)Att=softmax(Q(HW×C)⋅K(C×HW))$$Att = soft\max ({Q_{(HW \times C)}} \cdot {K_{(C \times HW)}})$$ (4)Fout=(V(C×HW)⋅Att).reshape(C×H×W)+Input(C×H×W)$${F_{out}} = ({V_{(C \times HW)}} \cdot Att).reshape(C \times H \times W) + Inpu{t_{(C \times H \times W)}}$$

Where Att means the attention weight map and F_ωu represents the output.

The channel attention module switches the position of the dot product of the position attention module, and the shape of the generated attention weight map is (C × C), so as to establish the influence relationship of features among different channels. The overall structure of the channel attention module is shown in equations (5) and (6): (5)Att=softmax(Q(C×HW)⋅K(HW×C))$$Att = softmax({Q_{(C \times HW)}} \cdot {K_{(HW \times C)}})$$ (6)Fout=(Att⋅V(C×HW)).reshape(C×H×W)+Input(C×H×W)$${F_{out}} = (Att \cdot {V_{(C \times HW)}}).reshape(C \times H \times W) + Inpu{t_{(C \times H \times W)}}$$

Combined with the attention mechanism, the structure of the labeled attention module is shown in Fig. 2. Due to the increase in the number of channels of the attention probability map, the attention probability map is no longer obtained by multiplying Q and K dots, but by direct convolution of the input features. Similarly, the attention output is convolved with the original input and then summed to obtain the final output. Equations (7) and (8) describe the overall structure of the labeled attention module: (7)Att=softmax(Conv(Input(C×H×W))(N×H×W))$$Att = softmax(Conv{(Inpu{t_{(C \times H \times W)}})_{(N \times H \times W)}})$$ (8)Fout=Conv(Conv(Input(C×H×W))(N×H×W)+Att)(N×H×W)$${F_{out}} = Conv{(Conv{(Inpu{t_{(C \times H \times W)}})_{(N \times H \times W)}} + Att)_{(N \times H \times W)}}$$

Figure 2.

The structure of the label attention module

Due to the introduction of additional loss functions in the labeled attention module, the way the parameters are updated during the backpropagation of the training model changes, in the labeled attention module, the parameters will be constrained by two loss functions at the same time, the segmentation loss function and the attention loss function, both of which work together to optimize the parameters as shown in Eqs. (9) and (10): (9)W=W−ηbatch−size∑∂Lseg∂W$$W = W - \frac{\eta }{{batc{h_ - }size}}\sum {\frac{{\partial {L_{seg}}}}{{\partial W}}}$$ (10)W=W−ηbatch−size∑(∂Lseg∂W+∂Latt∂W)$$W = W - \frac{\eta }{{batc{h_ - }size}}\sum {(\frac{{\partial {L_{seg}}}}{{\partial W}} + \frac{{\partial {L_{att}}}}{{\partial W}})}$$

The model employs an encoder-decoder architecture, the encoder is basically the same as the VGG architecture adopted by UNet, and each decoder contains three attention modules containing two self-attention modules and one labeled attention module. As shown in equations (11) and (12): (11)Fi=Upsampling(Conv(PAM(Fe)+CAM(Fe))+LAM(Fe))$${F_i} = Upsampling(Conv(PAM({F_e}) + CAM({F_e})) + LAM({F_e}))$$ (12)F=argmax([F1,...FN])$$F = argmax([{F_1},...{F_N}])$$

The dataset contains a total of N category, F_e represents the feature map output by the encoder, F_i represents the output of each category decoder, and F represents the final multi-class fusion output.

In the label attention module, since each decoder corresponds to only one category, the number of attention channels of label attention is changed accordingly, from the original multi-channel to a single-channel probability map. After each decoder gets the predicted probability of the corresponding category, it directly combines the probability maps of each category, and the category with the highest probability is selected at each point as the final prediction result. Since the labeled attention model has already hinted at the feature extraction region in the attention module, which is equivalent to a layer of weak constraints, there is no need to do further fusion of the overall results to avoid the loss of feature information. In terms of the loss function, due to the splitting of the task, the loss function of each category is calculated separately, as shown in Eqs. (13), (14) and (15): (13)Lossseg=Focal(predict,label)$$Los{s_{seg}} = Focal(predict,label)$$ (14)Lossax=CE(att,labeldown_sampling)$$Los{s_{ax}} = CE(att,labe{l_{down\_sampling}})$$ (15)Losstotal=∑iNLoss−atti+∑iNLoss−segi$$Los{s_{total}} = \sum\limits_i^N L os{s_ - }at{t_i} + \sum\limits_i^N L os{s_ - }se{g_i}$$

The final overall loss function is the sum of all the attention loss functions and the semantic segmentation loss function.

In order to verify the performance of the multitasking network model, the experiments are completed on the servers of the high-performance computing platform, and the configuration of the experimental platform is shown in Table 1.

When evaluating the performance of deep learning network-based models, they are generally categorized into three aspects: accuracy, memory footprint, and frame rate. 1)

AP and mAP

The average accuracy AP is associated with two metrics: precision and recall. True Positive (TP) indicates how many samples with positive labels are predicted to be positive cases, and True Negative (TN) indicates how many negative samples are actually also predicted to be negative cases, and the larger the percentage of TP and TN, the better. False Positive (FP) represents the number of wrong detections, i.e., the number of predicted positive cases that are actually negative samples, and False Negative (FN) represents the number of missed detections, i.e., the number of predicted negative cases that are actually positive samples.

Precision represents the proportion of true positive samples predicted by the network in the overall true samples of the detection results, as shown in Equation (20), which reflects the detection rate of the model: (20)Precision=TPTP+FP$$\Pr ecision = \frac{{TP}}{{TP + FP}}$$

Recall represents the proportion of true-positive samples predicted by the network to the true true samples, as shown in Equation (21), which reflects the model’s check rate: (21)Recall=TPTP+FN$$\operatorname{Re} call = \frac{{TP}}{{TP + FN}}$$

For the performance of the model, the higher the values of Precision and Recall, the better the performance of the model, but these two are often contradictory to each other, in order to better evaluate the performance of the algorithm, different Recall values as the horizontal coordinates, Precision values as the vertical coordinates, to draw the PR curve, which integrates the results of Precision and Recall, the PR curve The accuracy on the curve takes the mean value which is the average accuracy (AP), which is commonly used to evaluate the detection effect of a single category.The area under the PR curve is given by equation (22): (22)AP=∫01P(R)dR$$AP = \int_0^1 P (R)dR$$

mAP is then averaged over all categories of AP for evaluating the effectiveness of multi-category detection, as given in Eq. (23) for calculation: (23)mAP=∑i=1NAPiN$$mAP = \frac{{\sum\limits_{i = 1}^N A {P_i}}}{N}$$

Where N indicates the total number of categories. 2)

Parameters and Parameters Size

Parameters refers to the number of parameters included in the model, i.e., the number of parameters that the model needs to learn in the training process. Take the convolution used in the model as an example, suppose the number of channels of the input feature map is C, the size of the convolution kernel is K × K, and every point of weight on the convolution kernel has to be learned, so the number of parameters that need to be learned for a single channel of the convolution kernel. If the number of channels of the output feature layer is O, the number of input parameters is calculated first, and then multiplied based on the number of output channels, the formula for calculating the number of parameters after the convolution operation is shown in equation (24): (24)Parameters=(K2×C)×O$$Parameters = ({K^2} \times C) \times O$$

Generally use the number of parameters Parameters to measure the model size Parameters Size. the number of parameters directly determines the size of the model, which is measured in units of one, but due to the large number of parameters of the model, it is generally taken as a unit measure of megabytes (M), and corresponds to the unit of the model size of the model is MB. 3)

FLOPS and FPS

Floating point operations FLOPS represents the number of floating point operations per second, i.e., “throughput”, which measures the computational time complexity of the model, and is often used as an indirect measure of the speed of the model, and the larger its value, the better. Frame rate FPS indicates the number of frames processed per second, i.e., the number of images that can be processed per second or the time required to process an image, evaluating the speed of model execution, the shorter the time, the faster the speed.

In the feature extraction algorithm, IoU denotes the ratio between the number of pixels in the common part of the target region obtained from the network prediction and the target region in the corresponding label and the number of pixels covering the concatenation of the two.

The average intersection and merge ratio MIoU, i.e., IoU is calculated on each category and averaged, assuming that the total number of categories is k, which becomes k + 1 with the addition of the background category, the MIoU calculation formula is shown in Equation (26): (26)MIoU=1k+1∑i=0kTPTP+FN+FP$$MIoU = \frac{1}{{k + 1}}\sum\limits_{i = 0}^k {\frac{{TP}}{{TP + FN + FP}}}$$

In order to investigate the impact of multi-task learning, this paper conducts comparative experiments on the selected Vaihingen dataset, comparing different feature extraction methods such as U-Net, EANet, PSPNet, BiseNetv2, CCNet, RefineNet, EMANet, and Deeplabv3+, respectively.

The above model and the MD TANet model of this paper are trained under the experimental environment configuration in Section 4.2.1. During the training process, a multi-threaded approach is used to read data to speed up the data reading speed. The training batch size batch size is set to 64, the initial learning rate are set to 0.01, both use the cos strategy to adjust the learning rate, both use the SGD optimizer for parameter updating, and the epoch number is set to 250.

The difference between the predicted value of the model and the true value of the label is evaluated by the loss value. The smaller the loss value is, the better the model fits the true value. After training the model for 250 epochs, the Loss curve’s change during training can be seen in Fig. 3, and the loss value of the model converges to 0-8, and the MD TANet model converges with the lowest loss value of 0.17, which is the best fitting effect.

Figure 3.

The loss value of different model training

In order to better evaluate the network performance, five metrics, mAP, mIoU, FPS, Parameters and FLOPS, are chosen to evaluate the accuracy of the detection segmentation results, and 0.5 is used as the threshold for the confidence level, so its corresponding evaluation criteria can be expressed as mAP@50 and mIoU@50.

The evaluation of the results of different models on the validation set is shown in Table 2. Experiments show that the MD TANet model has the best recognition accuracy on the Vaihingen validation set, with the mAP@50 and mIoU@50 reaching 78.10% and 75.98%, respectively, the former is 1.16%~5.96% higher than the comparison method, and the latter is improved by 2.43%~6.38%. In terms of speed, the multitasking model MD TANet achieves a high real-time performance of 90.72 FPS while increasing the number of small parameters.The comparative analysis of the above experimental results proves that the improved multitasking model MD TANet achieves a better balance between accuracy and speed, and shows a better performance of remote sensing video image feature extraction.

In order to further validate the effectiveness of the proposed multitasking model MD TANet, feature extraction experiments for specific features are conducted on Vaihingen, WHDLD, and DLRSD datasets, where the comparison experiments conducted on DLRSD dataset are selected to be conducted by several algorithms that have better performance on other datasets. 1)

Comparison experiment results on Vaihingen dataset

Using the Vaihingen dataset, the performance of this paper’s method and the comparison algorithms are compared. The test results on the Vaihingen dataset are shown in Fig. 4, and Table 3 shows the test results of different models on Vaihingen dataset. The mIoU of this paper’s algorithm on the Vaihingen dataset reaches 75.98%, and except for the feature extraction accuracy of the automobile category (68.89%), which is slightly lower than that of RefineNet (69.02%) and CCNet (70.47%), the rest of the categories and the overall accuracy have achieved a certain improvement. Among the comparison methods, PSPNet has the lowest accuracy of 69.60%, mainly due to the difference in the accuracy of the other categories.Deeplabv3+ has a relatively high accuracy of 73.54%. Since the methods in this chapter are optimized for the edges of category objects, consistent optimization for all categories should be able to achieve some improvement in most categories, which is supported by the experimental results. For the MD TANet method in this paper, the accuracy in roads and buildings is better, reaching more than 80%. The lowest accuracy among the main 5 categories is for cars, with only 68.89%. Cars have relatively low segmentation accuracy due to the fact that the objects are significantly smaller and the features are not easily captured.

Figure 4.

Test results on the Vaihingen data set

Figure 5.

Test results on the WHDLD data set

Figure 6.

Test results on the DLRSD data set