Research on infrared image target detection technology based on deep learning

Sigmoid function, also known as Logistic function, is a typical activation function, the mathematical definition of the function can be written: (3) σ(x)=1exp(−x)+1

Tanh function is also called hyperbolic tangent activation function, the main contribution of this function is to solve the mean value problem in Sigmoid function to some extent. The output range of the input data after processing by Tanh function is [–1,1], and the image of this function is centrosymmetric with the origin as the center in the right-angle coordinate system, so Tanh function can be regarded as an improved version of the Sigmoid function after translation and elongation. Although the performance of the Tanh function in the experiment is improved compared to the Sigmoid function, the problem of vanishing gradient still exists: (4) σ(x)=tanh(x)=exp(x)−exp(−x)exp(x)+exp(−x)=2sigmoid(2x)−1

From the derivative function of the ReLU function, it can be seen that the ReLU function has the property that the derivative value is 1 at x ≥ 0, so the ReLU function has an effective treatment of the phenomenon of gradient disappearance, which makes the ReLU function become one of the most commonly used activation functions in various types of convolutional neural networks: (5) σ(x)=max(0,x)={ xx≥00x<0

A typical way of combining modules of a convolutional network is convolution, activation and pooling. The specific process is that firstly, the input feature map is convolved by each convolution kernel on the convolution layer to get the output feature map, after getting the feature map produced by convolution the result is input into the activation function to activate the output features to stimulate the nonlinearity of the network computation, and the last step is the pooling operation, which mainly serves to merge and sparsify features extracted from convolutional and preamble modules, so as to enhance the network’s ability to perceive the overall features and at the same time reduce the network’s computation.

Fully connected layers are more common in classical fully connected networks, and in convolutional neural networks, a small number of fully connected layers are usually used to integrate and adjudicate the image target features and other information extracted by the preceding convolutional layer and other modules, so as to output the entire algorithm.

Integration and judgment are necessary to produce the final result of the entire algorithm. However, the size of the input tensor received by the fully connected layer is generally fixed, so the actual algorithm is often used to replace the fully connected layer with a convolutional layer to form a fully convolutional network, and the output of the network with a convolutional layer can also play a role in integrating the results and drawing conclusions.

Target detection is a core task and an important research topic in the field of computer vision, whose task is to determine whether there is a target object belonging to a predefined object category in an arbitrary image, and if there is such a target object, then the corresponding category and location information of the target object can be returned efficiently and accurately. However, due to the diversity of target morphology and application scenarios, target detection is challenging. In order to overcome the difficulties in the task of target detection, a large number of scholars have invested in algorithmic research in this field, so that the relevant algorithms have been improved in various aspects. At present, deep learning-based target detection algorithms can be divided into two main categories according to the different detection ideas, the first is the Two-stage target detection algorithm based on the generation of candidate regions, and the second is the One-stage target detection algorithm based on regression.

Two-stage target detection algorithms, first extract candidate regions from the input image using a candidate region generation algorithm, and feed the generated candidate regions into the CNN for feature extraction, and then use the extracted features to classify and regress the selected candidate region targets. Such as R-CNN, SPP-Net, Fast R-CNN, Faster R-CNN, R-FCN and so on.

One-stage target detection algorithms eliminate the step of generating candidate regions using correlation algorithms. Instead, they directly implement the steps of feature extraction, target classification, and regression in the same convolutional neural network.Such as YOLO series algorithms, SSD, and RetinaNet.

The quality of the image is directly related to whether the algorithm can play the best level and whether the detection accuracy can reach the optimal performance, so before image recognition and detection, we need to preprocess the acquired dataset images. Batching the basic image preprocessing of the dataset in the early stage can reduce the impact of irrelevant information on image processing to a certain extent, and better focus on the valuable target information.

In this paper, the dataset preprocessing mainly consists of two parts, one is the denoising process according to the characteristics of infrared images, and the other is the data enhancement operation for the insufficient richness of the dataset.

Image denoising is mainly for the image acquisition process may produce a variety of noise processing, especially infrared image acquisition equipment is more complex, acquisition environment is more diverse, so the image preprocessing for infrared image dataset is particularly important.

Classification of denoising algorithms such as shown in Figure 1, the traditional infrared image denoising methods include all kinds of filtering algorithms based on the image spatial domain denoising methods, as well as Fourier transform, wavelet transform and other major image transformation domain denoising methods. In recent years, with the maturity of deep learning technology, many image denoising algorithms based on deep learning have emerged, which can be categorized into four major classes.

Figure 1.

De-denoising algorithm classification

In this paper, we use CBDNet network for image denoising, and its network structure can be divided into two parts, the first part is a five-layer fully convolutional network as a noise estimation sub-network, which is used to convert the noisy observation image x into the estimated noise level Fig. σ^(x) . The second part is a UNet with a residual structure as a denoising sub-network, which is used to get final denoising result y by taking x and σ^(x) as inputs, with residual mapping as: (6) ℜ(x,σ^(x);WD)

The W_D in Eq. (6) denotes the network parameters in the denoising sub-network, and finally the clean image after the denoising process is completed can be expressed as: (7) y=x+ℜ(x,σ^(x);WD)

Considering the excellent detection accuracy and inference rate of the YOLO series target detection network, this paper designs an improved YOLOv5 network based on Transformer network. The improved YOLOv5 network structure is shown in Fig. 2. First, this paper designs Backbone according to the YOLOv5 structure, which is based on the CSPDarknet-53 structure, which stacks multiple CSP layers and convolutional modules. And in Backbone, Focus module and Spatial Pyramid Pooling Fast (SPPF) are introduced, which are used to reduce the image size and merge the contextual information of targets with different scales without loss of information, respectively.In addition, considering the problem of low accuracy of YOLO series in small target detection, a Transformer structure to enhance the network’s ability of feature extraction for small targets. In this paper, the Bot-CSP structure is improved and designed, while the Vision Transformer structure is introduced to improve the performance of the network.

Figure 2.

Improve the YOLOv5 network structure

In the Neck network, the top-down structure of FPN and bottom-up structure of PAN are still used.FPN utilizes up-sampling to generate high-resolution feature maps, while PAN computes a feature hierarchy consisting of feature maps at multiple scales, which is advantageous for multiscale detectionAfter the Backbone processing, FPN utilizes up-sampling to progressively recover the feature maps, which contain the abstract semantic information. Then jump connections merge the feature maps at different scales in Backbone into FPN to preserve the target feature regions after up-sampling. Thus, the FPN structure obtains higher resolution features by up-sampling the spatially coarser but semantically stronger feature maps from Backbone, while the PAN structure enhances these feature maps. Specifically, the merge connections in PAN feature maps of the same spatial scale from both FPN and PAN.Although the feature map of PAN has a lower level of semantic information, its activation location information is more accurate due to its smaller number of subsamples.

This paper investigates deep learning based infrared and visible light image fusion and target detection technology, for the infrared and visible light image fusion part, this paper firstly proposes a two-channel self-coding and decoding image fusion algorithm, designed two-way feature extraction network for infrared and visible light source image respectively, then connects the channel dimensions and finally reconstructs the fused image using the decoding network. Then an image fusion algorithm based on improved generative adversarial network is proposed according to the adversarial network architecture combined with the previous method, which utilizes the game between the generator and the discriminator of the generative adversarial network to achieve better fusion effect and obtain better quality fused images. For the target detection part of the fused image, an improved YOLOv5 target detection algorithm based on Transformer network is designed to improve the network’s ability to obtain information from the fused image, thus increasing the accuracy of detection and reducing the time complexity. In this paper, the realization of infrared light image fusion and target detection technology relies on the embedded platform, image fusion and target detection hardware platform shown in Figure 3. The algorithm and training simulation designed in this paper are mainly developed on the PC platform. After the algorithm verification is completed, the algorithm will be transferred to the embedded platform Hi3559AV100 for actual testing. The main process is as follows: 1)

Use infrared and visible light detectors to obtain data in the same scene, clean and label the data to obtain the final dataset.

Figure 3.

Image fusion and target detection hardware platform

This paper uses the attention mechanism to improve the backbone network, the more popular attention mechanisms include SENet, CA and CBAM, in order to verify the effectiveness of this paper in the backbone network part of the addition of the attention mechanism to improve the effectiveness of the strategy, here with the three attention mechanisms for comparison experiments.

Two types of datasets are used in this paper, one is a near infrared dataset made by ourselves and named Near_Infrare. The other is taken from the thermal infrared dataset released by FLIR. The former is divided into training set, validation set and test set according to the ratio of 8:1:1, and the latter is divided into training set and test set according to 8:2. The labels are all containing three types of targets, where the three types of targets for the Near_Infrare dataset are pedestrians, cars, and motorcycles, and the three types of targets for the FLIR dataset are pedestrians, cars, and bicycles.

The comparison results of the three attention mechanisms are shown in Fig. 4. After experimental verification, SENet and CA are slightly lower in mAP compared to CBAM when dealing with thermal infrared targets, which indicates that the CBAM attention mechanism can better capture the key features and improve the target detection rate when dealing with some scenarios with thermal infrared images, and illustrates the validity of adding the CBAM mechanism in the backbone network in this paper. The introduction of CBAM in the backbone network is 0.53% and 0.44% higher on mAP respectively, compared to SENet and CA. On Precision, it is higher by 3.21% and 2.25%, respectively, although on Recall, it is low compared to the other two attentional mechanisms, which illustrates that there are still some deficiencies in using the CBAM attentional mechanism only in the backbone network, and there is still room for further improvement in the overall effectiveness of the model.

Figure 4.

The comparison of the three attention mechanisms

The up-sampling algorithm in the original YOLOv5 model has the problem of discontinuous gray values, which will have some impact on the detection rate, in order to solve the problem, this subsection hereby performs a simple replacement of the up-sampling algorithm, and conducts a comparative experiment with the other two classical interpolation methods, the bilinear interpolation and the bicubic interpolation. Among them, Bilinear interpolation calculates the changed pixel values by finding the weighted values of the four points around the mapping point, and Bicubic interpolation utilizes the grayscale values of the 16 points around the sampling point to perform three times interpolation, thus obtaining a smoother resampling effect, although the computational complexity is higher. The experiment uses Precision, Recall, mAP, and Inferrence as evaluation criteria for selecting the algorithm. The comparison results of the three types of interpolation are shown in Fig. 5, and from the data, it can be seen that the mAP value of bicubic interpolation is the highest, 0.809, and the nearest neighbor interpolation and bilinear interpolation are relatively low, with the mAP values of 0.792 and 0.807, respectively. Bilinear interpolation’s mAP values are in the middle of the range, although in terms of the inference time, it’s just the opposite, with the bicubic interpolation taking the most time, nearest neighbor interpolation taking the least time the inference time for bilinear interpolation is still in the middle. In order to ensure that the model maintains a certain degree of accuracy under the condition of minimizing the inference time, the double-cubic interpolation interpolation algorithm is chosen as the subsequent up-sampling method in this paper.

Figure 5.

Comparison results of three kinds of interpolation method

To verify the usability of the improved YOLOv5 image detection model algorithm in this paper, the proposed algorithm is experimentally compared with nine current mainstream algorithms.The lightweight algorithm has absolute advantages in terms of parameters, computation, and model size, making it easier to develop and deploy on hardware devices to meet the requirements of real-time detection. The lightweight algorithms were YOLOv3-tiny, YOLOv5s, YOLOv5-ghost, YOLOv5-repvgg, and YOLOv7-tiny, respectively, and the proposed algorithm were compared and tested. Table 1 shows that the weight of the YOLOv5 lightweight network model is 14.38MB, and the size of the optimized model in this chapter is 2.11MB more than that of the original YOLOv5, and the number of parameters and calculations increases slightly, but the accuracy and recall rates reach 85.7% and 81.1%, which are greatly improved compared with YOLOv5, effectively reducing the model false detection rate, and the mAP is increased by 4.8% to 86.2%, which improves the model detection accuracy. Compared with the YOLOv3-tiny, YOLOv5-ghost, YOLOv5-repvgg, and YOLOv7-tiny lightweight models, the mAP of the optimized algorithm in this paper is also far ahead.

At the same time, in order to further verify the universality and generalization of the optimization model in this paper, the second set of comparison experiments are conducted, and the algorithm selects YOLOv5m, YOLOv6s, YOLOv7, and YOLOv8s algorithms with large number of parameters. Experimental comparisons are made with models with higher detection accuracy so as to verify the effectiveness of the algorithms in this chapter, and the results of the comparison of models with large number of parameters are shown in Table 2. The optimized model in this chapter reaches the minimum in the second group of experiments in terms of the number of parameters, the computational amount and the model size.There are five different models with different numbers of parameters in the YOLOv6 and YOLOv8 models, and the YOLOv6s and YOLOv8s models in the second group are the lighter s models in the YOLOv6 and YOLOv8 versions, respectively, with a relatively weak feature extraction capability. The mAP of the optimized model in this chapter is improved by 2.7% compared to YOLOv5m, which is 2.2 and 2.1 percentage points higher than YOLOv6s and YOLOv8s, respectively, and only 0.01% lower than YOLOv7, but the number of parameters is reduced by 27.06M compared to YOLOv7. It can be concluded from the experiments that this paper’s optimized model based on the improvement of YOLOv5 performs better overall than the above models.The model that detects targets with the best overall performance among the above models.