Vehicle Target Detection in Rainy and Foggy Scenes Based on Generative Adversarial Networks and Dynamic Fuzzy Compensation Techniques

Filter-based deblurring is the estimation of a desirable clear image s from an observed blurred image u₀ by some filter estimator, w for the corresponding filter, as represented in equation (1): (1)u^=u^w=w×u0

For a pure denoising process of a noisy image unaffected by blurring, linear filtering can be considered as a natural tool for noise suppression by convolution, and for deblurring, it can be considered as an attempt to remove the effect of a particular convolution operation by another convolution operation [29]. For example, without considering noise, it can be expressed in the form of a Fourier frequency domain, where the Fourier transform is known and the frequency domain quantity is denoted as ω = (ω₁, ω₂): (2)W(ω)=1K(ω)

From the above equation, u^=w×u0≡u is easy to realize for any clear image u. But usually a typical blurring kernel K is a low-pass filter, whose Fourier transform K(ω) tends to cause rapid decay in the high frequency part, leading to instability in its process and even making the recovered image seriously distorted.

In order to solve this instability in the recovery process, the above equation is improved as: (3)W(ω)=K*(ω)K(ω)K*(ω)=K*|K|2

where * denotes the conjugate of a complex number and an attempt is made to regularize the instability of the denominator when it is in the high-frequency part by adding a positive factor r = r(ω): (4)W→Wr=K*|K|2+r

Assuming that the estimate is recorded as u^r , the (5)u^r=Wr×k×u

or in the Fourier frequency domain: (6)WrK(ω)=|K(ω)|2|K(ω)2|+r(ω)

In the low-frequency part of r ≪ |K|², the image recovery is approximately equal to that expected. At r ≫ |K|², the high-frequency part is severely distorted due to the near disappearance of k. Thus the regularization factor r acts as an equivalent of a threshold.

Since image noise affects deblurring, it is particularly important to select an optimal regularization factor, for which we use Wiener’s minimum mean square error for the original implementation.

Generative Adversarial Network (GAN) is a network that contains a class generator and a class discriminator. The class generator generates samples that “look similar to real samples” based on the input noise signal, and the class discriminator is used to distinguish between the samples generated by the class generator and real samples [30]. As an example, a photo is generated (G is the class generator and D is the class discriminator): 1)

G receives a random noise x through which a photo is generated, denoted as G(x).

As the latest form of machine learning, Generative Adversarial Networks (GANs) have the advantages of high-definition output images, high sharpness, and universality to both generators and discriminators compared to general neural networks. Compared to other generative models, GANs no longer require a predefined data distribution and have maximum freedom of fit.

Wiener filtering hypothesis de-blurring estimation results Image u^ is the result of the filtering process of the blurred image u₀ with the optimal filter w(X): (7)u^w=w×u0

Wiener filter w(x) is such that the mean square estimation error: (8)lw(X)=u^w(X)−u(X)

Reach the smallest optimal filter: (9)w=arghminE[ek2]=argkminE(h*u0(X)−u(X))2

which generates orthogonal conditions: (10)E[[(w×u0(X)−u(X)u0(Y))]]=0,∀X,Y∈Ω

can be transformed as per the correlation function: (11)w×Ru0u0(Z)=Ruu0(Z),Z∈R2

This leads to the explicit form of the optimal Wiener filter: (12)W(ω)=Ruu0(Z)Ru0u0(Z)

For fuzzy graph u₀ = k × u + n, there is: (13)Suu0=K*(ω)Suu|K|2Suu+Snn=K*|K|2+r

where the regularization factor r = sm/suu is the square of the noise-to-signal ratio.

The UNIT structure is schematically shown in Fig. 1. The UNIT network can be formally viewed as a combination of two VAE/GAN models. The network consists of three main parts: the autoencoder E₁, E₂, the generative model G₁, G₂, and the discriminative model D₁, D₂, x₁, x₂ representing the pictures in the source and target domains, respectively. For the unsupervised image style conversion task, samples can only be obtained from the respective edge distributions. However, the joint distribution can be obtained from the known edge distributions with countless possibilities, so it is not possible to obtain the joint distribution from the edge distributions without assumptions. Therefore, based on the theory of “shared potential space”, it is considered that any inputs x₁ and x₂ have a common potential code z in a shared potential space, and the image of the corresponding domain can be generated or restored to the original image through the common potential code z.

Figure 1.

Structure of the UNIT

As shown in (a), there are two different styles of picture domains in UNIT, the source domain X₁ and the target domain X₂. In the UNIT network model, encoders E₁ and E₂ are used to realize the mapping of the input pictures x₁ and x₂ from different domains to a certain shared potential space and encode them into a uniform potential code z, with z = E₁(x₁) = E₂(x₂). The generative models G₁ and G₂ are used to realize the conversion of the potential codes z into the corresponding domains’ pictures through different generative models. pictures, and for generative model G₁, whose input is the latent code z, it can be realized to map the latent code z obtained from X₁ the domain into the source domain X₁, and also the latent code z obtained from X₂ the domain into the source domain X₁. Like the discriminative models in the original GAN, the discriminative models D₁ and D₂ are used to determine the authenticity of the images, i.e., to determine whether the input images are real samples or samples generated by the generative model.

As shown in (b), X₁ picture x₁ in the picture domain is obtained by encoder E₁ with potential encoding z, which potential encoding z can be mapped back to the X₁ domain by generative model G₁ to obtain self-reconstructed image x11−1 , and can also be obtained by generative model G₂ to obtain domain-variant image x11−2 . Similarly, picture x₂x₁ in the domain is obtained by encoder E₂ with potential encoding z, which potential encoding can be obtained by generative model G₁ to obtain domain-variant after image x22−1 and also image x22−2 can be obtained by generative model G₂.

1)

Network Structure

In this section, the proposed VAE-CoGAN de-fogging and de-raining model can better handle images in foggy and rainy scenes without preclassifying the images. VAE-CoGAN consists of three parts: encoder, generative model, and discriminative model.

The encoder is responsible for converting the input picture codes into vector form to be used as input for generative models in the GAN. For a blurred picture x₁ with fog or with rain patterns in the source domain and a clear picture x₂ in the target domain, the same potential encoding z is obtained by mapping the picture (x₁, x₂) to the shared potential space using encoders E₁ and E₂, with z = E₁(x₁) = E₂(x₂).

The generative model is responsible for realizing the conversion of the potential encoding z obtained by the encoder conversion into a picture. For the generative model G₁, the generative model yields generative pictures x11−1 as well as x22−1 , where the generative picture x22−1 is a blurred picture obtained by converting a clear picture in the target domain X₂, while the generative picture x11−1 is a self-reconstructed picture that has not been converted to a style, and does not help in the task of converting the image styles investigated in this chapter. Similarly, for generative model G₂, whose input is latent encoding z, generative images x11−2 and x22−2 can be obtained through generative model G₂, where the generated image xi1−2 is a clear picture transformed from a blurred image in the source domain X_i, that is, the clear picture obtained by the foggy or rainy picture that is intended to be obtained in this chapter, and the generated picture x22−2 is a self-reconstructed picture without style conversion, which is not helpful for the task of image style conversion studied in this chapter.

The VAE loss function aims to minimize the objective function, and its loss function consists of two components, regularization and reconstruction error: LVAE=Lpnor+Lllikepixel .

where Lprior=KL(q(z|x)||pη(z)),Lllikepixel=−Eq(z|x)[logp(x|z)] .

Regularization provides a simple way to sample from the latent space, and minimizing the negative log-likelihood term in the reconstruction error is equivalent to minimizing the absolute distance between the image and the reconstructed image. This chapter uses two auto-variable encoders with: (15)LVΛE1(E1,G1)=λ4KL(q1(z1|x1)||pη(z))−λ5Ez1~q1(z1|x1)][logpG1(x1|z1|)] (16)LVAE2(E2,G2)=λ4KL(q2(z2|x2|)||pη(z))−λ5Ez2−q2(z2|x2)][logpG2(x2|z2|)]

where λ₄ and λ₅ are hyperparameters that control the weights of the two items. the KL divergence term indicates the distance between q(z ∣ x) and p_η(z), and the smaller the KL value, the smaller the distance.

In this model, the GAN loss function is used to ensure that the generated images are as similar as possible to the images in the target domain: (17)LGAN1(E2,G1,D1)=λ0Ex1~pq[logD1(x1)]+λ0Ez2~q2(z2|x2)[log(1−D1(G1(z2)))] (18)LGAN2(E1,G2,D2)=λ0Ex2~px2[logD2(x2)]+λ0Ez1~q1(z1|x1)[log(1−D2(G2(z1)))]

Pictures x22−1 and x11−2 with style transformations performed in generative model G_i and generative model G₂ are examined, there are x22−1=F2−1(x2)=G1(E2(x2)) and x11−2=F1−2(x1)=G2(E1(x1)) . The experimental results are made more realistic by incorporating the idea of cyclic consistency, there are: x₁ = F₂₋₁(F₁₋₂(x₁)), x₂ = F₁₋₂(F₂₋₁(x₂)). For cyclic consistency loss there are: (19)dLcc1(E1,G1,E2,G2)=λ6KL(q1(z1|x1)||pη(z))+λ6KL(q2(z2|x11→2)||pη(z))−λ7Ez2−q2(z2|x11→2|)[logpG1(x1|z2)] (20)Lee2(E2,G2,E1,G1)=λ6KL(q2(z2|x2|)||pη(z))+λ6KL(q1(z1|x22→1)||pη(z))−λ7Ez1~q1(z1|x22→1)][logpG2(x2|z1))]−λ7Ez1~q1(z1|x22→1)][logpG2(x2|z1|)]

The formula for VGG loss is: (21)Lcontent(E1,E2,G1,G2)=Ex1~px1[||φi(G1(E2(G2(E1(x1)))))−φi(x1)||0]+Ex2~px2[||φi(G2(E1(G1(E2(x2))))))−φi(x2)||1]

where φ_i is the activation of layer i of the CNN.

The final goal in this network is: (22)G*,F*=argminG,FmaxDX1,DX2,DY1,DY2L(G,F,DX1,DX2,DY1,DY2)

Here λ₀ = 10, λ₄ = 0.1, λ₅ = 100, λ₆ = 0.1, λ₇ = 100.

This section focuses on the network structure of the vehicle detection part and proposes a detection framework with local perceptual enhancement compared to the most commonly used CNN-based detection algorithms. Unlike the detection backbone network used for foggy images, the model feeds the features extracted from the de-raining part to the local perceptual enhancement Transformer backbone network, and the vehicle detection network is shown in Fig. 2. Similar to Swin Transformer, each stage has 2, 2, 6 and 2 blocks respectively.

Figure 2.

Vehicle detection network

First, given a rainy day image of size H × W × 3, the input image H × W × 3 is partitioned into a set of non-overlapping image patches by image block partitioning, where each image patch has size 4 × 4, feature dimension 4 × 4 × 3, and number H4×W4 , and then by a linear embedding, where each patch is regarded as a vector “token”, and its features are viewed as a Each patch is then treated as a vector “token” through a linear embedding, and its features are viewed as a concatenation of the original pixel RGB values. After changing the feature dimension of the segmented patch token to 4 × 4 × C, it is fed into multiple locally aware Swin Transformer blocks for encoding, which contain blocks at different stages. After that the input is merged by image block merging the neighboring image blocks according to 2 × 2. This changes the number of image blocks to H8×W8 and the feature dimension to 4C. Repeat this step for N times for each stage until the number of image block chunks becomes H32×W32 and the feature dimension becomes 8C. Finally it is fed to the regression header for target classification and localization regression. Each stage consists of an image block merging block or a linear embedding block and a stacked local perceptual enhancement Transformer block.

The positional coding in Transformer easily fails to detect local correlation and structural information in the image, Swin-Transformer m used a window-based hierarchical structure to solve the scaling problem and high computational complexity of high-resolution images. Each Swin-Transformer block consists of a normalization layer, a multi-head self-attention module, residual connectivity, and a multilayer perceptron (MLP), and has two fully connected layers with GELU nonlinearity. The window-based multi-head self-attention (W-MSA) module and the shift-window-based multi-head self-attention (SW-MSA) module are applied in two consecutive Transformer blocks, respectively.

Although Swin Transformer constructs a hierarchical Transformer and implements attentional operations in each non-overlapping window, Swin transformer is limited in its ability to encode contextual information, in order to enhance the network’s learning of local relevance and structural information, this paper proposes a locally-aware augmented Transformer. Each block is composed of two consecutive improved Transformer blocks.

The standard convolution kernel dilation process can be viewed as the spacing of the values of the convolution kernel when doing data processing. Expansion convolution introduces hyperparameter i.e. expansion rate r to the convolutional layer, and the expansion rate r of ordinary convolution is 1, i.e. r = 1 when the expansion convolution is standard convolution. The expansion process of dilation convolution can be seen as the convolution kernel horizontal and vertical neighboring weights between the addition of the void parameter 0, the number of intervals is (k − 1), each interval to join the number of 0’s is r − 1. The addition of the void parameter 0 does not need to participate in the convolution operation. So the relationship between the size of the dilated convolution kernel is shown in equations (23) and (24): (23)kd=k+(k−1)×(r−1) (24)kd=k+(r−1)×(k−1)

where k_d represents the size of the dilation convolution kernel, k represents the size of the standard convolution kernel, and r represents the dilation rate. In particular, when r is set to 1, there is no null parameter, i.e., it is a standard convolution. Usually, the dilation convolution which has the same parameters as the standard convolution has more weight parameters and larger size to capture the global information in the image through the enlarged sensory field and discretized weight parameters [31].

Further after the introduction of dilated convolution, the size of the corresponding output feature map is calculated as in equation (25): (25)0=[i+2p−k−(k−1)*(r−1)s]+1 $$0 = \left[ {{{i + 2p - k - (k - 1)*(\>r\> - 1)} \over s}} \right] + 1$$

Where, o represents the output feature map size, i represents the image size size of the input null convolution, p represents the input feature map boundary padding, and s represents the step size. The above formula shows that if the convolution kernel are after dilation convolution operation and maintain its step size, boundary pixel padding and other parameters unchanged, its input the same feature map, after the convolution operation, the size of the output feature map will be correspondingly smaller.

1)

Comparison methods

The algorithm in this paper will be compared with three different classes of rain removal methods, including (1) one raindrop removal method, AttentGAN.(2) six rain pattern removal methods, including DetailNet, RESCAN, PReNet, PReNet, JORDER-E, RCDNet and RLNet.(3) two robust rain removal methods, including Pix2pix and CCN. In addition to the algorithms in this paper, the RadNet-algorithm is also tested in this paper.

1)

Experimental results and analysis on single-type data

Firstly, the performance of all methods is tested under single type strategy, and the quantitative evaluation results on single type dataset (rain pattern) are shown in Table 1, and on single type dataset (raindrop) are shown in Table 2. It can be seen that (1) the performance of this paper’s algorithm is much better than other methods on two real datasets, RS_real and RD_real. (2) Compared with CCN, which is also a robust method, this paper’s algorithm obtains better performance on all but the RainDrop dataset. Specifically, the PSNR of this paper’s algorithm in RS_syn is 5 dB higher than that of CCN.(3) The performance of this paper’s algorithm on the RainDrop dataset is poorer than that of CCN, which may be attributed to the fact that CCN has two independent modules to process rainbars and raindrops separately, so that it can deal with the RainDrop data that contains large blurring regions. (4) In terms of the average score, the algorithm in this paper has a very competitive performance.

2)

Experimental results and analysis of superimposed type data

The task of de-raining under this data strategy is more difficult than the single-type strategy, mainly because the network needs to synchronize the processing of rain patterns and raindrops. The quantitative evaluation results on the superimposed-type data set are shown in Table 3. From the PSNR/SSIM results in the table, it can be seen that (1) the algorithm in this paper obtains the best performance on both synthetic dataset (RDS_syn) and real dataset (RDS_real). Compared with CCN, the algorithms in this paper have 2dB PSNR improvement on RDS_syn and 4dB PSNR improvement on RDS_real. (2) All the methods perform poorly on RDS_real data, which is mainly due to the fact that the image pairs in it do not correspond at the pixel level, which makes it difficult for all the supervised methods. And based on the effectiveness of the proposed FWM, the algorithm in this paper can still achieve excellent results.

3)

Experimental results and analysis of hybrid data

The de-raining task under this data strategy is more difficult than the above two strategies, not only needing to synchronize the two degenerate phenomena of rain patterns and raindrops, but also suffering from the fitting problems caused by the different distributions of the datasets. The results of the quantitative assessment on the hybrid dataset are shown in Table 4. From the PSNR/SSIM results in the table, it can be seen that: this paper’s algorithm achieves the best performance, with 1dB PSNR over RCDNet on the Blended-1 dataset, while on the Blended-2 dataset, this paper’s algorithm outperforms RCDNet by close to 3dB PSNR and 0.011dB SSIM. From the results, it can be seen that the algorithm in this paper is 1dB PSNR higher than CCN*.

The performance of the different methods under the three data strategies is shown in Fig. 3 (Fig. a shows the results of PSNR values and Fig. b shows the results of SSIM values). It can be found that (1) the algorithm in this paper is only slightly weaker than RCDNet on single-class data strategy, while the best performance is obtained in all other cases. (2) This paper’s algorithm improves very significantly on the stacked data strategy and hybrid data strategy, which mainly test the robustness of each method, and thus the robustness of this paper’s algorithm can be fully verified. (3) The algorithm in this paper is far better than another robust rain removal method CCN.

Figure 3.

Different methods are performed under the strategy of three

1)

Dataset library construction

When performing the target detection task under rain and fog conditions, it is crucial to obtain a sufficient number of rain and fog dataset samples. However, there are relatively few rain and fog datasets under public traffic conditions, which poses a challenge to the performance improvement of deep learning network models under rain and fog conditions. To address this problem, this paper uniformly generates fog-containing data samples with different concentrations based on the atmospheric scattering model using the BDD100K dataset as a basis, a process that allows the dataset to contain images of foggy conditions with different concentrations, ranging from light fog to dense fog, covering a variety of foggy conditions. However, the samples generated by relying only on the atmospheric scattering model have certain singularities and limitations. To further enrich the dataset, an adversarial generative network is employed to generate more realistic and diverse fog-containing images, effectively expanding the size and diversity of the dataset. At the same time, real rainy day condition images are manually collected and labeled to enrich the training data, and a rainy fog condition dataset integrated with real, physical models and adversarial generative networks is constructed. The rain and fog images generated by this method have different concentration differences, which improves the breadth and abundance of the dataset and provides more abundant data resources for the subsequent target detection model based on convolutional neural network.

1)

Experimental results and analysis of the effectiveness of data derivation for rain and fog scenes

In order to verify the effectiveness of the data augmentation method, by inputting the Haze Sim dataset, the GANHaze dataset and the AtmoGAN Haze dataset into the model network model of this paper respectively, and using mAP0.5 as the evaluation index, the data augmentation results are shown in Table 5. As can be seen from the table, the dataset achieves 75.6%, 54.8%, and 67.1% in precision (P), recall (R), and mean category accuracy (mAP), respectively, and the AtmoGAN Haze dataset after data expansion improves the detection effect by 4.3% relative to the un-expanded HazeSim dataset, and improves the detection effect by 1.1% relative to the GANHaze dataset Compared to the pre-expansion dataset, the AtmoGAN Haze dataset has better performance.

The detection accuracy in each category is shown in Table 6. Secondly, the detection accuracy in each category shows that the AtmoGAN Haze dataset achieves relatively high mAP for pedestrians, riders, cars, buses, trucks, bicycles, and motorcycles. mAP for pedestrian detection reaches 48.1%, which achieves the highest detection result compared to the unexpanded dataset. It proves that the expanded dataset can detect pedestrians better. For the detection accuracy of riders, cars, buses and trucks, AtmoGAN Haze also achieves 71.1%, 80.9%, 66.8% and 60.4%, respectively, which maintains the leading position compared to the unexpanded dataset. This result demonstrates that the fog-containing dataset augmented with different methods enhances the generalization of the model, and also highlights the superiority and usefulness of the AtmoGAN Haze dataset in image processing tasks, which provides an important reference for further research and applications.

In order to achieve the same good detection results of the model under rainy conditions, the 750 dataset obtained from Rain dataset, 500 images were added to the training set of AtmoGAN Haze dataset, and the other 250 images were added to the test set to form the rainy and foggy condition dataset HydroFogRain for this study. This dataset was constructed to ensure that the model has robust target detection performance in the rainy and foggy conditions with robust target detection performance. Based on the HydroFogRain dataset, the network model of this paper’s model with sensory field amplification is compared with the current mainstream model YOLOV5, and the YOLOV5 comparison results are shown in Table 7. It can be seen that the precision rate of this paper’s model reaches 77.2%, which is higher than that of YOLOV5, indicating that this paper’s model is more accurate in predicting positive samples. Secondly, the recall rate reaches 53.8%, which is also higher than YOLOV5, indicating that this paper’s model can better capture the real positive examples in the positive samples, and the mAP of this paper’s model reaches 66.2%, which is significantly higher than YOLOV5, indicating that this paper’s model performs better in terms of comprehensive performance.

The detection accuracy for each category is shown in Table 8. As can be seen from the table, for various different traffic scenarios, this paper’s model achieves relatively high detection accuracies for pedestrians, riders, cars, buses, trucks, bicycles and motorcycles. First, the detection accuracy of this paper’s model is significantly better than that of YOLOV5 for pedestrians and riders, reaching 53.1% and 73.5%, respectively, compared with only 44.5% and 59.1% for YOLOV5, indicating that this paper’s model is more capable of accurately detecting pedestrians and riders in complex situations. Second, this paper’s model performs equally well on other categories. For example, in the detection accuracy of targets such as cars, buses and bicycles, this paper’s model achieves 80.9%, 62.9% and 66.5%, respectively, which is higher than YOLOV5’s 80.3%, 56% and 46.6%. It shows that this paper’s model has better detection performance for different traffic targets. Finally, the detection accuracy of this paper’s model is also higher than that of YOLOV5 for trucks and motorcycles, which are 60.3% and 75.2%, respectively. In summary, the model in this paper performs well in the detection accuracy of multiple categories, which is especially suitable for the target detection task in complex scenes.

2)

Comparison of mainstream target detection models

In order to verify the effectiveness of the detection model, the proposed target detection model with sensory field amplification This paper’s model is compared with some mainstream deep learning networks based on the HydroFogRain dataset, including Faster R-CNN, SSD, YOLOV3, YOLOV3-SPP, YOLOV4, YOLOV7, YOLOV8, and DETR, and the comparative experiments are shown in Table 9, which clearly shows that the proposed model algorithm in this paper has a much higher mAP than other mainstream detection models and almost twice as much as SSD when compared to other models. Although FasterR-CNN represents a two-stage detection algorithm with better detection accuracy, the detection accuracy is far less than that of this paper’s model and is not applicable to real-time applications. In contrast, YOLOV3, YOLOV3-SPP and YOLOV4 have faster detection speeds but lower detection accuracy and overall poor results. Compared with YOLOV5X, the model in this paper performs well in terms of detection precision and recall, while the number of parameters is much smaller than that of YOLOV5X. Compared with YOLOV7, YOLOV8, and DETR, the model in this paper shows the best performance in terms of precision, recall, and detection speed, although the number of parameters does not dominate. Taken together, the target detection algorithm proposed in this study for this paper’s model achieves the best level in terms of detection precision, and although the detection speed is slightly reduced compared to some YOLO series models, it is still the optimal choice in terms of comprehensive performance.