Innovative research on multispectral image fusion technology for defect detection of composite insulators in ultra-high voltage transmission lines

To solve the problem of low alignment accuracy in scenes with inconsistent image target scales, the Laplace pyramid (LP) scale space is constructed in the ORB algorithm, and feature points of different scales are extracted at each level of the pyramid [21]. The LP pyramid is realized by calculating the differences between different layers of the Gaussian pyramid as: 1Gg(x,y)=down(12πσ2e−x2+y22σ2⋅Gg−1(x,y)) Where: when g = 1, G₀(x, y) is the original image, σ¹ is the variance of the Gaussian kernel. The size of σ² is positively correlated with the degree of smoothing of the image, and a larger value of σ indicates a higher degree of smoothing, i.e., the blurrier the image, indicating the Gaussian scale of the image. down() is the downsampling operation of the image.

Each layer of the LP pyramid L_i(x, y) is composed of the difference between two neighboring layers of the Gaussian pyramid. The specific formula is: 2Li(x,y)=Gg(x,y)−Expand(Gg+1(x,y)), Where: Expand(G_g+1,(x, y)) denotes the interpolated expansion of layer g+1 of the Gaussian pyramid so that it has the same dimensions as layer g.

The definition of feature point in Fast algorithm is that a pixel may become a feature point if its gray value is much larger or smaller than a certain range of pixel gray values and a certain number of pixel gray values [22]. If there is K consecutive pixel I_k on a circle with o as the center and the radius of the circle is determined by the image resolution, k = 1,2,…,K, the method to determine whether it is a feature point is as follows: 3CRF={ 1,| Io−Ik |<t0,| Io−Ik |≥t Where: I_k is the pixel value of a point on the circle and I_o is the image cord value of point o. When the number of CRF = 1 is greater than a particular fixed value, a suitable threshold t, o is selected as a candidate point based on the gray scale distribution of the image.

Feature description is the basis of feature matching and binary descriptors are tens or even hundreds of times faster than SIFT and SURF. The BRIEF algorithm selects some pairs of points around the key points by a specific method and the binary code string resulting from the comparison of the pairs is the descriptor [23]. The process of obtaining the binary bit string is to compare the grayscale values of random point pairs: 4τ(p;x,y)={ 1,p(x)<p(y)0,p(x)≥p(y) where p(x) denotes the pixel value p of the point x. For the feature point o, its characterization is represented as a n binary feature vector based on n pairs of pixel points in the domain: 5fn(p)=∑i=1n2i−1τ(p;xi,yi)

Where: n is the length of the feature vector and n = 256 is taken considering the speed, distribution and accuracy of the descriptor generation.

To solve the problem that BRIEF algorithm does not have rotational invariance, the grayscale prime method is used to make the descriptors rotationally invariant, which is obtained in a small image block: 6mab=∑x,yxaybI(x,y)

According to Eq. (6), the 0th order moments m_ω and 1st order moments m₀₁, m₁₀ can be obtained, and the coordinates of the center of mass of the image block can be found through the moments: 7C=(m10m00,m01m00)

Connect the geometric center O and the center of mass C of the image block to obtain a direction vector OC, with the direction of the feature point being the angle between the direction vector OC and the X -axis: 8θ=arctan(m10m00,m01m00)=arctan(m01m10)

In order to obtain a n-bit description, it is necessary to select n the set of test by introducing a 2×n matrix Q which can be defined as follows: Q=[ x1x2⋯xny1y2⋯yn ] . After obtaining the rotational directions θ of the feature points by the gray scale center of mass method, the corresponding rotation matrix R_θ is obtained, which leads to the construction of the rotational matching pair matrix Q_o: 9Qθ=RθQ=[ cosθsinθ−sinθcosθ ][ x1x2⋯xny1y2⋯yn ]

The rotational invariant descriptor is: 10fn(P,θ)=fn(P)|(xi,yi)∈Qo

To solve the contrast inversion problem in visible and infrared image alignment, a descriptor reorganization strategy is introduced so that θ = θ + θ_r,θ_r is the inversion angle. The inversion angle θ_r=π is found by the experimental law, then the reorganized rotational invariant descriptor is: 11fn(P,θ+θt)=fn(P)|(xi,yi)∈Q(θ+θi)

The descriptor distance indicates the degree of similarity between two feature points.

The power field has high requirements for the accuracy and real-time performance of testing equipment, but the power field environment is often more complex. For composite insulator defect detection, due to its production of various materials, production process is complex, in the process of production and transportation is prone to produce a variety of defects. The existence of these defects reduces the service life and performance of composite insulators. Common defects such as breakage, cracks, etc., are interspersed with the complex background, which greatly increases the difficulty of its detection. And the spectral reflectance of each defect is different, and the form of characterization is also different, which provides a research direction for this paper. For composite insulator images with changing defect spectral ranges, applying multispectral image fusion to suppress the complex background and highlight the weak defect features is an effective idea to solve the problem, and the algorithm in this section is proposed based on this idea.

Based on multispectral images, a non-uniform texture complex surface defect saliency fusion algorithm is proposed in this section. The pixel values of multispectral images of composite insulators can describe the spectral reflectance of the object with illumination independence. By utilizing this property, the influence of complex background on defect detection can be reduced to a certain extent, and the spectral reflectance shows a gradual decreasing trend as the band grows from 380 nm to 780 nm, so 450 nm, 500 nm, 550 nm, 600 nm, and 650 nm are selected as the fusion source images in this section. The algorithm firstly suppresses the complex background by using the structure-texture decomposition model as a preprocessing means, but at the same time, a large amount of information is lost, so it needs to be reconstructed by the method of image fusion. Then it is decomposed into approximation layer and detail layer by 2D tensor empirical wavelet transform multiresolution decomposition. The detail layer contains a large amount of defective feature information, which is analyzed for significance, and the contrast of the weighted part is enhanced. Finally the fused image is obtained by 2D tensor empirical wavelet inverse transform. The multispectral image saliency fusion algorithm proposed in this section well solves the interference problem of complex background for weak defect features, and provides help for the defect detection of composite insulators in UHV transmission lines.

In this paper, the least squares method of straight line fitting is used to detect the insulator self-explosion. From the insulator image after image segmentation, the insulator is basically on a straight line. The equation of this straight line is set as: 21y=ax+b

Where: a and b denote the slope and intercept of the line, respectively, the coordinates of the points at the edges of the segmentation line (x_i, y_i) and hence the values of a and b were calculated by fitting the line. The minimum error between the observed and actual values is calculated: 22Δ=∑i=1N[ yi−(bxi+a) ]2

Calculate the partial derivatives of a and b: 23δδa∑i=1N[ yi−(bxi+a) ]2=−2∑i=1Nyi−(bxi+a)=0 24δδb∑i=1N[ yi−(bxi+a) ]2=−2∑i=1N(yi−(bxi+a))xi=0

Organized and available: 25aN+b∑ xi=∑ yia∑ xi+b∑ xi2=∑ xiyi

The values of a and b are calculated as: 26a^=(∑ xi2)(∑ yi)−(∑ xi)(∑ xiyi)N(∑ xi2)−(∑ xi)2 27b^=N(∑ xiyi)−(∑ xi)(∑ yi)N(∑ xi2)−(∑ xi)2

The values of a and b are calculated to obtain the fitting equation and establish the mathematical model of insulator string. Then Hough transform is used to detect the insulator string to get the ellipse center parameter on the fitted straight line, and compare the identified number according to the number of standard insulator pieces of composite insulators, and when the identified number is lower than the actual number, it means that the insulators are self-destructing/dropping the string.

In this paper, the length and width of insulator sheet cracks are calculated and analyzed using the pixel integral projection method, and when the crack width or length changes, the projected length ratio in the horizontal and vertical directions changes. After image segmentation, the image has become a binarized image, so the projection of each pixel in the horizontal direction is summed up as the horizontal integral projection, and similarly the pixel in the vertical direction is summed up as the vertical integral projection. The equations for the integral projection in the horizontal and vertical directions are as follows: 28L=1x2−x1∫x1x2f(x,y)dx 29V=1y2−y1∫y1y2f(x,y)dy

Where: f (x, y) denotes the pixel value of the image. L denotes the horizontal integral projection. V denotes the vertical integral projection.

When the insulator surface is contaminated, its shape characteristics are basically the same as those of a clean insulator, but its surface color makes a difference. Therefore, it is possible to discriminate the insulator filthiness by identifying the exterior color. The images of dirty insulators and clean insulators are captured and the color characteristics of their surfaces are extracted. The insulator slices are decomposed into six channels, and then the feature values of each channel are calculated. The calculation method is:

Mean value: 30mean=∑igipi

Max: 31max=maxgi

Variance: 32var=∑i(gi−mean)2pi

Gray scale entropy: 33entro=∑i−pilg(pi)

Where: g_i and p_i denote the gray level of the image and its probability of occurrence, respectively.

To filter the above feature vectors using Fisher's criterion, let the total number of samples be N, which can be divided into n classes, and the number of samples in each class is N_i. Calculate the interclass and intraclass variance of these samples as:

Inter-class variance: 34Sxj=∑i−1nNiN(mij−mj)2

Intraclass variance: 35Sxj=1N∑i−1n∑α∈Li(aj−mij)2

Where: mij and mⁱ denote the mean of the i -category sample and the mean of all samples, respectively; a^j denotes the j th dimensional feature of sample a; and L_i denotes the feature category. The Fisher criterion of a feature is expressed as: 36F=SxjSrj

After calculating the eigenvalues of each channel, the mean, maximum, and variance of s of F are the highest, so these three features are used to discriminate the degree of insulator fouling.

The dataset used in this experiment is the photos of composite insulators of UHV transmission lines taken by UAVs, and the initial dataset contains 658 photos of insulators taken by UAV inspections, and using the synthesis technique, the insulators placed on the green screen are cut and pasted on a complex background through the image enhancement technique, and the defective insulators are each placed into different simulated UAVs to take photos of complex backgrounds to get 274 synthesized images, a total of 932 images of the original dataset were obtained for the next step of data enhancement.

In order to prove the effectiveness of the improvements in this paper, in this section, the improved ORB algorithm (IORB) is firstly compared with the traditional ORB algorithm in terms of its alignment, and then a side-by-side comparison between IORB and other algorithms, such as SIFT, PIIFD, etc., is conducted to prove the superiority of the IORB algorithm for multispectral composite insulator image alignment.

In this section, the alignment error err of the alignment point pair is used as a measure of the alignment algorithm, and err is defined as follows: 37err=1n∑i=1N(x1i−T(x2i))2+(y1i−T(y2i))2

Where (x_li, y_li.) is the coordinates of the feature points in the first image, (x_2i, y_2i) is the coordinates of the feature points of the second image paired with (x_li, y_li), T is the transformation matrix between the two images computed using the feature point pairs inputted by the algorithm, N is the transformation matrix between the two images computed using the feature point pairs inputted by the algorithm, and N is the total number of matched feature point pairs. When err < 3, the two images are considered to be matched successfully.

For the training set, in this paper, the blue spectral band image is used as the reference image, and the near-infrared spectral band image is used as the floating image. For the experiments in the test set, the image with a center wavelength of 400 nm is used as the reference image and the image with a center wavelength of 958 nm is used as the floating image. Since the long-wave infrared images in the test set have a low gray level and a small number of feature points, which makes it difficult to achieve the alignment, the recall recall is also used as the evaluation index of the algorithm, i.e., the ratio of the number of image pairs that have been successfully aligned to the number of all image pairs.

In the experiments comparing the IORB and ORB algorithms, in this paper, the floating images are transformed with a mixture of rotational transformation, scale transformation and rotation plus scale, respectively, as a way of comparing the robustness of the IORB and ORB algorithms to rotational transformation, scale transformation and rotation plus scale.

In the experiments containing only rotational transformations, the reference image is fixed, the rotation angle ∆θ of the floating image with respect to the reference image is used as a variable, and the scale of the floating image is kept constant. Statistics ∆θ show the performance metrics of the ORB and IORB algorithms at 0°, 90°, 180° and 270°, respectively.

Tables 1 and 2 show the number of aligned point pairs and alignment errors of IORB and ORB on the training and test sets, respectively, for different rotation angles err. When the rotation angle is 0°, both the ORB algorithm and IORB algorithm can match more correct feature point pairs. As the rotation angle increases, the ORB algorithm matches fewer and fewer feature point pairs, while the IORB algorithm in this paper is able to get many matched feature points even at 270°. Overall, whether in the training set or in the test set, the IORB algorithm proposed in this paper has a smaller alignment error and more matched point pairs, indicating that IORB is more robust to rotation than ORB.

In the experiments containing only the scale transformation, in this paper, the reference image is fixed, the scaling multiplier σ of the floating image with respect to the reference image is used as a variable, and the scale of the floating image is unchanged. The performance metrics of the ORB and IORB algorithms when statistic σ is 1, 0.8, 0.6, and 0.4, respectively. Tables 3 and 4 show the metrics of the two algorithms on the training and test sets, respectively. From the table, it can be seen that the number of matching point pairs of IORB and ORB is about the same when the scaling multiplier σ is 1, i.e., there is no scaling, but the number of matching point pairs of ORB decreases drastically as σ decreases, and ORB is no longer able to get the aligned feature points in the test set when σ is 0.4, whereas the matching point pairs of IORB proposed in this paper are relatively small with the change of σ, and it can output more pairs of points even if σ is 0.4. Output more pairs of points, so the robustness of IORB to scale transformations is better than the ORB algorithm.

Since the performance of the ORB and IORB algorithms when rotational transformation plus scale transformation has been analyzed in detail in the previous section, in the following, when comparing the robustness of the two algorithms to the hybrid rotational plus scale transformation, only the alignment results and algorithmic metrics of the training set of experiments are shown, in this set of experiments, the floating image is rotated by 90° with respect to the reference image and the scaling multiplier is 0.5, and the experimental results are shown in Table 5. From the table, it can be seen that the ORB algorithm cannot output correctly aligned feature point pairs when the rotation angle is 90° and the scaling is 0.5, while the IORB algorithm in this paper can still output more feature point pairs.

In order to demonstrate the superiority of the IORB algorithm proposed in this paper in the task of multispectral image alignment, this section also compares the alignment performance of the IORB algorithm with the SIFT, PIIFD, MatchNet and RF-Net algorithms on multispectral composite insulator images. Among these four compared algorithms, SIFT and PIIFD are traditional algorithms, and MatchNet and RF-Net are deep neural network based algorithms for homospectral image alignment. In this paper, the performance of these algorithms are compared on the training set and test set respectively, without geometric transformation of the images to be aligned in the experiment, the performance metrics of the different algorithms on various datasets are shown in Table 6. In the table, for the training set, SIFT, PIIFD, RF-Net and IORB have good alignment results, and the average number of alignment pairs for the IORB algorithm is 546 pairs. While for the test set, SIFT successfully matches 107 pairs of images, PIIFD only successfully matches 13 pairs of images, and the IORB algorithm of this paper matches 387 pairs successfully with an error of only 0.38. It can be seen that the improved image alignment method of ORB in this paper is more suitable for multispectral composite insulator image alignment compared to other alignment methods.