Research on Deep Learning-based Image Processing and Classification Techniques for Complex Networks

This paper establishes the degree matrix of an image under different thresholds according to the static statistics of complex networks, completes the texture description of an image by counting the degree distribution of network nodes in each state, and proposes a method to establish a complex network model of an image, which regards each pixel of an image as a node of a complex network and considers that each node is connected to each other with edges, and the weights of the edges are determined by the distances between the two pixels and the gray level difference The weighted sum of the two pixels is determined. The initial complex network complete graph model is thresholded dynamically by setting a series of thresholds for edge weights, and edges with weights higher than the threshold are deleted, resulting in edges between pixels with smaller distances and similar pixel values. In order to simplify the complex network model, this paper selects the 28 nodes around the node with its distance less than 3 as the neighborhood, only the nodes in the neighborhood can have an edge connected between the two nodes i(x,y) and j(x′,y′) in the weight of the edge between the two nodes w(v_(x,y),v_(x′,y′)) for the distance between the nodes and the node represents the difference between the gray value of the pixels of the weighted sum of the nodes, in order to make the node’s degree distribution is more uniform, this paper adopts the weights of the value of the w for the direct summation of the two items above, as shown in the formula (1) shown: (1) w(v(x,y),v(x′,y′))=(x−x′)2+(y−y′)2+| I(x,y)−I(x′,y′) | \[\begin{align} & w\left( v(x,y),v\left( x\prime ,y\prime \right) \right)=\sqrt{{{\left( x-x\prime \right)}^{2}}+{{\left( y-y\prime \right)}^{2}}} \\ & +\left| I(x,y)-I\left( x\prime ,y\prime \right) \right| \end{align}\]

After normalizing the obtained edge weights, a series of thresholds t are set, and edges between nodes with edge weights higher than the thresholds are deleted to obtain the neighborhood θ(v_t) and degree deg(v_t) of each node corresponding to the thresholds. As shown in Eqs. (2)(3): (2) θ(vr)={ v′∈V|(v,v′)∈E&w(v,v′)≤t } \[\theta \left( {{v}_{r}} \right)=\left\{ v\prime \in V\left| \left( v,v\prime \right) \right.\in E\And w\left( v,v\prime \right)\le t \right\}\] (3) deg(v1)=| θ(vt) | \[\deg \left( {{v}_{1}} \right)=\left| \theta \left( {{v}_{t}} \right) \right|\]

The method proposed in this paper is based on the degree matrix, where the degree of a node is used as the point weight of the node. By counting the number of elements with the same degree in the degree matrix, feature vectors are constructed to realize the texture description of the image. By controlling the number of thresholds, the result of controlling the number of degree matrices and controlling the length of the feature vector data can be achieved. The flow of the algorithm is shown in Figure 1.

Figure 1.

The flow chart of image texture feature extraction algorithm

In this paper, the Harris corner points of the image are regarded as the nodes of the complex network, and the initial complete graph network model is established as a result. In the complete graph model each node is connected to each other by edges, which is a kind of regular network and cannot be used as distinguishing image topology features, using the dynamic evolution process of the complex network, a series of sub-networks can be generated, and the shape feature extraction of the image is accomplished through the statistics of each sub-network’s degree, jointness, shortest path length, average path length clustering coefficients, and other static statistical feature descriptions.

Dynamic evolution, as an important feature of complex network models, is a process that can be accomplished through the distribution of attributes of edges or through the distribution of attributes of nodes. General complex network evolution models are based on the properties of edges, such as the inter-value evolution method, the minimum spanning tree evolution method, and the K -nearest neighbor evolution method. The minimum spanning tree evolution method chosen in this paper ensures that the sub-networks generated by evolution are all connected, and does not produce the problem of difficult to compute the statistical eigenvalues of the network caused by the disconnection of sub-networks in the threshold evolution method. The process is as follows:In Eq. (4), G₀ is the complete graph of the initial network, MST is the minimum spanning tree algorithm, T_n(n = 1,2,3…) is the minimum spanning tree generated by the n th evolution, and G_n is the subnetwork generated by the n th evolution. That is, the minimum spanning tree T_n of the n th generation is the difference between the initial network G₀ and the n–1 th evolution generating subnetwork G_n–1, and the n th generated subnetwork G_n is the sum of the n th generated subnetwork G_n–1 and the n th generated minimum spanning tree T_n.

Shape feature extraction of an image is realized by calculating the average path length, network diameter, clustering coefficient, maximum degree and maximum number of kernels for each sub-network G_i. And it can be achieved by controlling the number of evolutions to control the data length of the feature vectors and the time required for the procedure.

In this paper, we propose a complex network image description where the feature vector consists of texture feature vector V_T and shape feature vector V_s. Namely: (5) V=[ VT,VS ] \[V=\left[ {{V}_{T}},{{V}_{S}} \right]\]

The texture eigenvectors consist of distributions of degree matrices at different thresholds: (6) VT=[ dst1,dst2,dst3⋯dstN ] \[{{V}_{T}}=\left[ ds{{t}_{1}},ds{{t}_{2}},ds{{t}_{3}}\cdots ds{{t}_{N}} \right]\] (7) dsti=[n1,n2,n3⋯nL],i∈[ 1,N ] \[ds{{t}_{i}}=[{{n}_{1}},{{n}_{2}},{{n}_{3}}\cdots {{n}_{L}}],i\in \left[ 1,N \right]\]

dst_i is the distribution of the degree matrix under different thresholds, N is the number of selected thresholds, n_k is the number of elements with degree k in the current degree matrix, and L is the maximum value of the number of other elements in the selected neighborhood.

The shape feature vector is constructed according to the Harris corner points and then dynamically evolved with the minimum spanning tree algorithm to get the sub-networks at different moments G_i and calculate the average path length, network diameter, clustering coefficient, maximum degree and maximum kernel of each sub-network corresponding to achieve the shape feature extraction of the image. The mathematical expression is shown below: (8) VS=[ F1,F2,F3…FM ] \[{{V}_{S}}=\left[ {{F}_{1}},{{F}_{2}},{{F}_{3}}\ldots {{F}_{M}} \right]\] (9) Fi=[ Li,Di,Ci,degmaxi,Kmaxi ],i∈[ 1,M ] \[{{F}_{i}}=\left[ Li,{{D}_{i}},{{C}_{i}},\deg _{\max }^{i},K_{\max }^{i} \right],i\in \left[ 1,M \right]\]

F_i is the static feature statistic of subnetwork G_i at different moments, L_i, D_i, C_i, degmaxi\[\deg _{\max }^{i}\], Kmaxi\[K_{\max }^{i}\] are the average path length, network diameter, clustering coefficient, maximum degree and maximum number of kernels of the subnetwork at the corresponding i moments, and M is the number of dynamic evolutions selected.

Image semantic segmentation methods play a very critical role in the process of parsing image content, the effect of the extracted feature information will have a direct impact on the performance of subsequent methods. According to the actual situation, in the collection of image data, because some images will exist in the light imbalance situation, the light situation more or less will cause the lack of object texture and color and other feature information in the image [24].

In this paper, we propose an image semantic segmentation model DECANet, which aims to solve the problem of invalid information and loss of local detail information of the image in the process of semantic segmentation, and to improve the semantic segmentation accuracy. The overall model architecture of DECANet is shown in Fig. 2.

Figure 2.

Structure of decanet network model

The coordinates of the unknown point (x,y) are inferred from a straight line to the point. Linear interpolation, as shown in Figure 4, assumes that the known coordinates (x₀,y₀) and (x₁,y₁), (x₀,y₀) and (x₁,y₁) are connected to a straight line, and finds the value y of a point x on the line in the interval [x₀,x₁]. This solution process is the process of linear interpolation.

Figure 4.

Linear interpolation

As can be seen from the figure: (15) y−y0x−x0=y1−y0x1−x0 \[\frac{y-{{y}_{0}}}{x-{{x}_{0}}}=\frac{{{y}_{1}}-{{y}_{0}}}{{{x}_{1}}-{{x}_{0}}}\]

Since the value of x is known, the value of y can be obtained from Eq: (16) y=y0+(x−x0)y1−y0x1−x0=y0+(x−x0)y1−(x−x0)y0x1−x0 \[y={{y}_{0}}+\left( x-{{x}_{0}} \right)\frac{{{y}_{1}}-{{y}_{0}}}{{{x}_{1}}-{{x}_{0}}}={{y}_{0}}+\frac{\left( x-{{x}_{0}} \right){{y}_{1}}-\left( x-{{x}_{0}} \right){{y}_{0}}}{{{x}_{1}}-{{x}_{0}}}\]

Known y The procedure for finding x is the same as above, except that x and y are swapped.

First do linear interpolation in the x direction to figure out f(R₁) and f(R₂): (17) f(R1)≈x2−xx2−x1f(Q11)+x−x1x2−x1f(Q21)WhereR1=(x,y1) \[f\left( {{R}_{1}} \right)\approx \frac{{{x}_{2}}-x}{{{x}_{2}}-{{x}_{1}}}f\left( {{Q}_{11}} \right)+\frac{x-{{x}_{1}}}{{{x}_{2}}-{{x}_{1}}}f\left( {{Q}_{21}} \right)Where{{R}_{1}}=\left( x,{{y}_{1}} \right)\] (18) f(R2)≈x2−xx2−x1f(Q12)+x−x1x2−x1f(Q22)WhereR2=(x,y2) \[f\left( {{R}_{2}} \right)\approx \frac{{{x}_{2}}-x}{{{x}_{2}}-{{x}_{1}}}f\left( {{Q}_{12}} \right)+\frac{x-{{x}_{1}}}{{{x}_{2}}-{{x}_{1}}}f\left( {{Q}_{22}} \right)Where{{R}_{2}}=\left( x,{{y}_{2}} \right)\]

Then do a linear interpolation in the y-direction to figure out f(P): (19) f(P)≈y2−yy2−y1f(R1)+y−y1y2−y1f(R2) \[f\left( P \right)\approx \frac{{{y}_{2}}-y}{{{y}_{2}}-{{y}_{1}}}f\left( {{R}_{1}} \right)+\frac{y-{{y}_{1}}}{{{y}_{2}}-{{y}_{1}}}f\left( {{R}_{2}} \right)\]

This allows you to calculate the desired result f(x,y): (20) f(x,y)≈(x2−x)(y2−y)(x2−x1)(y2−y1)f(Q11)+(x−x1)(y2−y)(x2−x1)(y2−y1)f(Q21)+(x2−x)(y2−y)(x2−x1)(y2−y1)f(Q12)+(x−x1)(y−y1)(x2−x1)(y2−y1)f(Q22) $\begin{align} & f\left( x,y \right)\approx \frac{\left( {{x}_{2}}-x \right)\left( {{y}_{2}}-y \right)}{\left( {{x}_{2}}-{{x}_{1}} \right)\left( {{y}_{2}}-{{y}_{1}} \right)}f\left( {{Q}_{11}} \right) \\ & +\frac{\left( x-{{x}_{1}} \right)\left( {{y}_{2}}-y \right)}{\left( {{x}_{2}}-{{x}_{1}} \right)\left( {{y}_{2}}-{{y}_{1}} \right)}f\left( {{Q}_{21}} \right) \\ & +\frac{\left( {{x}_{2}}-x \right)\left( {{y}_{2}}-y \right)}{\left( {{x}_{2}}-{{x}_{1}} \right)\left( {{y}_{2}}-{{y}_{1}} \right)}f\left( {{Q}_{12}} \right) \\ & +\frac{\left( x-{{x}_{1}} \right)\left( y-{{y}_{1}} \right)}{\left( {{x}_{2}}-{{x}_{1}} \right)\left( {{y}_{2}}-{{y}_{1}} \right)}f\left( {{Q}_{22}} \right) \end{align}$

In this paper, after extracting the features in the encoder stage, we use bilinear interpolation to do a 4-fold upsampling operation on the output feature map, and then the features are fused, and the fused feature map is subjected to a 4-fold upsampling operation, so that the size of the feature map is restored to the same size as the input image size.

Group convolution can improve model performance and reduce the number of parameters. Mostly, the input feature maps are processed in different groups, and then the different outputs are merged again.

Group convolution reduces the number of parameters compared to standard convolution using feature grouping. Here is an example of calculating the number of parameters for each of these two convolutions. Suppose, using C*H*W for the input and N for the number of convolution kernels, the output is also N. If the size of each convolution kernel is C*K*K, the total number of parameters for the standard convolution is N*C*K*K.

The group convolution improves the previous standard convolution, the basic input and output parameters are the same as above, the group convolution introduces the parameter G, with G indicating the number of groups. If the input feature map is to be divided into G Groups, then the parameter of each group will be changed accordingly. By analyzing and calculating, the number of parameters of group convolution is expressed by P, then the number of parameters of group convolution can be expressed as follows: (21) P=N*CG*K*K \[P=N*\frac{C}{G}*K*K\]

where CG$\frac{C}{G}$ denotes the number of input channels per group, the output is NG$\frac{N}{G}$, CG*K*K$\frac{C}{G}*K*K$ is the convolution kernel, the number of convolution kernels is N, and the number of convolution kernels per group is NG$\frac{N}{G}$. As can be seen from the above example, the total number of parameters of the convolution operation is reduced by some amount to become the original 1G$\frac{1}{G}$, which effectively proves that group convolution can reduce the number of parameters.

Suppose, the input feature map is C*h*w and the output feature map is N*h*w. Different image features are extracted by convolving the input feature map, usually this process can be represented as: (22) Y=WX+b \[Y=WX+b\]

where input X ∈ R^C×h×w, output Y ∈ R^N×h×w, C is the number of input channels, N the number of output channels, W denotes the weights, and W ∈ R^C×h×w×N, b denote the bias terms.

Typically, the bias terms are generally ignored to simplify the notation, and the convolution between complete feature maps can be expressed as: (23) yi=∑jwijxij \[{{y}_{i}}=\sum\limits_{j}{{{w}_{ij}}}{{x}_{ij}}\]

Expanding the convolution equation (23) yields the following matrix expression: (24) [ y1y2⋮yN ]=[ W11W12⋯W1CW21W22⋯W2C⋮⋮⋱⋮WN1WN2⋯WNC ][ x1x2⋮xC ] \[\left[ \begin{matrix} {{y}_{1}} \\ {{y}_{2}} \\ \vdots \\ {{y}_{N}} \\ \end{matrix} \right]=\left[ \begin{matrix} {{W}_{11}} & {{W}_{12}} & \cdots & {{W}_{1C}} \\ {{W}_{21}} & {{W}_{22}} & \cdots & {{W}_{2C}} \\ \vdots & \vdots & \ddots & \vdots \\ {{W}_{N1}} & {{W}_{N2}} & \cdots & {{W}_{NC}} \\ \end{matrix} \right]\left[ \begin{matrix} {{x}_{1}} \\ {{x}_{2}} \\ \vdots \\ {{x}_{C}} \\ \end{matrix} \right]\]

where y_i is the i nd output value, i = 1,2,⋯,N, w_ij are each element of the weight matrix representing each parameter, and the convolution kernel size is K*K, x_ij is the input value, and j = 1,2,⋯,C.

In order to reduce feature redundancy and feature loss and extract more effective features, the lightweight segmentation convolution divides all input feature map channels into two main parts according to the ratio α, where one part uses the more complex 3*3 convolution to extract typical feature information, and the other part uses the simpler 1*1 convolution to extract the feature information for self-calibration in order to supplement the information of some tiny hidden details. The arithmetic of this whole process is as follows: (25) [ y1y2⋮yN]=[ W1,1⋯W1,aC⋮⋱⋮WN,1⋯WN,αC ][ x1⋮xαC ]+[ W1,ac+1⋯W1,C⋮⋱⋮WN,αC+1⋯WN,C ][ xαC+1⋮xC] \[\left[ \begin{matrix} {{y}_{1}} \\ {{y}_{2}} \\ \vdots \\ {{y}_{N}} \\ \end{matrix} \right]=\left[ \begin{matrix} {{W}_{1,1}} & \cdots & {{W}_{1,aC}} \\ \vdots & \ddots & \vdots \\ {{W}_{N,1}} & \cdots & {{W}_{N,\alpha C}} \\ \end{matrix} \right]\left[ \begin{matrix} {{x}_{1}} \\ \vdots \\ {{x}_{\alpha C}} \\ \end{matrix} \right]+\left[ \begin{matrix} {{W}_{1,ac+1}} & \cdots & {{W}_{1,C}} \\ \vdots & \ddots & \vdots \\ {{W}_{N,\alpha C+1}} & \cdots & {{W}_{N,C}} \\ \end{matrix} \right]\left[ \begin{matrix} {{x}_{\alpha C+1}} \\ \vdots \\ {{x}_{C}} \\ \end{matrix} \right]\]

where α is the proportion of channel divisions and takes the value α = 0.5. The left half of the matrix w_ij is convolved over αC channels using a 3*3 convolution kernel, j = 1,…,αC and the right half of the matrix w_ij is convolved over (1–α)C channels using a 1*1 convolution kernel, j = αC,…,C.

Cityscape: The Cityscape dataset is a large-scale dataset for urban segmentation from the perspective of the automobile. The dataset contains multiple stereoscopic video sequences from 50 different city street scenes containing 4000 high quality finely annotated images, which are further divided into 2555, 400 and 1045 images according to the criteria for training, validation and testing respectively. The resolution of the images is 2048 × 1024, which poses a great challenge for real-time semantic segmentation methods. The labeled images have 30 classes, out of which 19 classes are used for the semantic segmentation task. To ensure a fair comparison with other methods, only 19 classes are used in the experiments of this paper.

Cam Vid: The Cam Vid dataset is a small-scale dataset for road scene segmentation, which is also from the perspective of a car. There are a total of 621 images with high quality pixel-level annotations extracted from video sequences, of which 314 are used for training, 89 for validation and 218 for testing. The images have the same resolution of 960 × 720. The annotated images are provided with 32 categories of truth labels, out of which a subset of 11 categories is used for the experiments in this paper.

The experiments were first conducted on the Cityscapes dataset to compare the models in this paper, focusing on comparing the network PreactResNet in this chapter with the mainstream real-time lightweight semantic segmentation models for urban landscapes, ENet, ICNet, BiSeNetV1, BiSeNetV1-L, BiSeNetV2, BiSeNetV2-L, BiSeNetV3-1 and BiSeNetV3-2 were compared and the comparison results are shown in Table 1. As mentioned in the previous subsection, the image resolution of the Cityscapes dataset is too high, so the images in the dataset are cropped at different scales according to different model settings. γ denotes the downsampling ratio, and a downsampling ratio of 0.5 corresponds to a size of 512 × 1024 after cropping, 0.75 denotes a size of 768 × 1536 after cropping, and 1.0 means that the original image is trained according to the size of the original image without downsampling. Backbone denotes the backbone network where each model is trained. Secondly, for fair comparison, separate experiments were also conducted using STDC1 and STDC2 as the backbone networks.

The evaluation of the experiments shows that the method proposed in this paper achieves a better balance between accuracy and speed compared with other methods, and the segmentation accuracy can reach 69.6% and the speed can reach 255.8 FPS when using STDC1 as the backbone network, which means that the model in this paper achieves a high inference speed. When using STDC2 as the backbone network, the segmentation accuracy of the model can reach 73.6% and the inference speed can reach 184.3FPS, which achieves the highest accuracy.

To further validate the performance of the deep neural network images in this paper, comparison experiments were also conducted on the CamVid dataset. To ensure a fair comparison with other methods, the experiments used an input image resolution of 940 × 710 for training and prediction. The experimental results are shown in Table 2.

From the experimental results, it can be seen that the method in this paper achieves an inference speed of 221.5 FPS and a segmentation accuracy of 69.6% when using STDC1 as the backbone network. Meanwhile, when STDC2 is used as the backbone network, the segmentation accuracy of the image can reach up to 75.4%, and the inference speed is 143.2 FPS.Taken together, the method in this paper achieves a better balance between speed and accuracy on the CamVid dataset.