Development of real-time video target recognition system based on convolutional neural network

If r^′ = T[r] is a linear single-valued function, the grayscale transformation determined by it is called a grayscale linear transformation, or linear transformation for short [30]. The specific transformation, is a kind of function that maps the gray value of each pixel in the original image to another value domain interval according to this linear transformation relationship.

If the gray value of the image is r = f(x,y), as shown below undergoes one of the following linear transformations, whose expression is.

It is possible to extend the range of values of the transformed grayscale r^′ to [ra′,rb′]$[r_{a}^{\prime },r_{b}^{\prime }]$. In the extreme case ra′=rmin′,rb′=rmax′$r_{a}^{\prime }={{r}_{\text{mi}{{\text{n}}^{\prime }}}},r_{b}^{\prime }={{r}_{\text{ma}{{\text{x}}^{\prime }}}}$ it follows that the human eye perceives a difference in luminance between two neighboring pixels only when their grayscale values (corresponding to their luminance) differ by a certain degree. When the grayscale value r is only in the smaller interval [r_a,r_b], the total number of levels of luminance difference that the human eye can perceive for the figure is very small. If the difference between the gray value of a target and the gray value of its background is small, the human eye will not be able to detect it, and after such a linear transformation, the total number of levels of luminance difference that can be perceived by the human eye for the transformed image will be increased, and the transformed image will look much clearer.

There is also a linear transformation is a segmented linear transformation, segmented linear transformation when the value domain of the image is divided into a number of value domains and different linear transformations of an algorithm. Using segmented linear transformation, according to the requirements of the transformation, compression of a portion of the gray-scale interval, expanding into another portion of the gray-scale interval. The transformation expression is shown in equation (5): (5) r′=rc′−ra′rc−ra(r−ra)+ra′,r∈[ra,rc] ${{r}^{\prime }}=\frac{r_{c}^{\prime }-r_{a}^{\prime }}{{{r}_{c}}-{{r}_{a}}}(r-{{r}_{a}})+r_{a}^{\prime },r\in [{{r}_{a}},{{r}_{c}}]$ (6) r′=rb′−rc′rb−rc(r−rc)+r′,r∈[rc,rb] ${{r}^{\prime }}=\frac{r_{b}^{\prime }-r_{c}^{\prime }}{{{r}_{b}}-{{r}_{c}}}(r-{{r}_{c}})+{{r}^{\prime }},r\in [{{r}_{c}},{{r}_{b}}]$

The segmented linear transform can both form a unit function with multiple folds and approximate a curve. At the same time, this transformation extends the gray scale range of the image details of interest and enhances their contrast. At the same time, the gray scale range of the image details that are not of interest can be compressed, reducing its contrast. It is worth noting that with this segmented linear transformation, the total gray scale range of the entire image before and after the transformation is unchanged.

The histogram of an image is an important statistical feature of an image and can be thought of as an approximation of the density function of an image’s gray scale distribution. Usually the gray level distribution density function of an image is related to the location of the pixel. Let the density distribution density function of the image at point (x,y) be p(z;x,y), then the gray level density function of the image is: (7) p(z)=1S∬Dp(z;x,y)dxdy \[p(z)=\frac{1}{S}\iint\limits_{D}{p}(z;x,y)dxdy\] Where D is the defined domain of the image and S is the area of region D. In general, it is difficult to obtain the density function of the gray level distribution of an image precisely, so in practice the histogram of the image is used instead. The gray level histogram is a discrete function that represents the correspondence between each gray level of a digital image and the probability of occurrence of that gray level. Let the total number of pixels in a digital image be N, there are L gray levels, and there are n_K pixels of gray level r_K with the kth gray level. Then the probability of occurrence of the kth gray level is: (8) hK=nKN,k=0,1,…,L−1 \[{{h}_{K}}=\frac{{{n}_{K}}}{N},k=0,1,\ldots ,L-1\]

The histogram grayscale adjustment described above is based on probability theory. In fact, a common method for changing the shape of the histogram is histogram equalization. The basic idea of this method is to transform the unbalanced histogram of the original image into a uniformly distributed form, which increases the dynamic range of the gray values, thus achieving the effect of enhancing the overall contrast of the image.

The image enhancement transform function needs to fulfill two conditions.

The first condition above ensures that the inverse transformation exists and that each gray level of the original image remains in the order from black to white after the transformation, preventing the appearance of some inverted gray levels in the transformed image. The second condition ensures the consistency of the dynamic range of gray values before and after the transformation, which can also be said that the original image and the transformed image have the same range of gray levels. For Equation t = T(s), the inverse transform can be expressed as: (9) s=T−1(t),0≤t≤L−1 \[s={{T}^{-1}}(t),0\le t\le L-1\]

And it can be proved that Eq. (8) should also satisfy the above conditions (1) and (2).

The gray levels of an image can be regarded as random variables on the interval [0,L – 1], and it can be proved that the cumulative distribution function (CDF) satisfies the above two conditions and can transform the distribution of s into a uniform distribution of t . In fact, the CDF of s is the cumulative histogram of the original image, in this case there: (10) tk=T(sk)=∑i=0knin=∑i=0kps(si),k=0,1,2,…,L−1 \[{{t}_{k}}=T({{s}_{k}})=\sum\limits_{i=0}^{k}{\frac{{{n}_{i}}}{n}}=\sum\limits_{i=0}^{k}{{{p}_{s({{s}_{i}})}}},k=0,1,2,\ldots ,L-1\]

From Eq. (10), the gray value of each pixel after histogram equalization is calculated directly from the image histogram. Of course, in the actual digital image processing but also to t_k rounding to meet the requirements. The inverse transformation can be written as: (11) sk=T−1(tk),0≤tk≤L−1 \[{{s}_{k}}={{T}^{-1}}({{t}_{k}}),0\le {{t}_{k}}\le L-1\]

In its usual form, convolution refers to a mathematical operation on two functions of a real variable. In the field of general function analysis, convolution denotes the process of generating a new function by means of two functions f(x) and g(x), and can also be characterized as the integral of the product of the function values of the overlapping parts of the functions f(x) and g(x) after flipping and translating over the overlap length.

Let f(x),g(x) be two productable functions on R. The integral: (12) ∫−∞∞f(t)g(x−t)dt \[\int_{-\infty }^{\infty }{f}(t)g(x-t)dt\] is called the convolution of f(x) with g(x) and can be denoted as: (13) s(x)=(f*g)(x) \[s(x)=(f*g)(x)\]

In a convolutional neural network processing image data, the first parameter f(x) of the convolution operation represents the input image, the second parameter g(x) represents the convolution kernel function, and s(x) represents the convolution result image. All subsequent references to convolution in this paper refer to the convolution operation expressed in Equation (14). The input in image processing can be viewed as a two-dimensional discrete point set, therefore, the convolutional neural network is generally a two-dimensional convolution operation, which can be written: (14) S(i,j)=(I*K)(i,j)=∑m∑nI(m,n)K(i−m,j−n) \[S(i,j)=(I*K)(i,j)=\sum\limits_{m}{\sum\limits_{n}{I}}(m,n)K(i-m,j-n)\] Where, s(i,j) represents the convolution result image, I represents the original image, K represents the convolution kernel, (m,n) represents the size of the convolution kernel, and (i,j) represents the position of the pixel on which this convolution operation acts. Mathematically, the convolution operation has an exchange property, i.e., Equation (13) can also be written as s(x) = (g*f)(x) with no change in the result. This property is manifested in two-dimensional convolution operations in images as the convolution kernel function K(m,n) can be flipped, but since convolution kernel flipping has no special significance for the application of convolutional neural networks, usually convolution operations in neural networks do not flip the convolution kernel function.

The pooling layer refers to the statistical results of the neighborhood of a certain position of the original image as the value of the output image, and its statistical methods are maximum pooling, average pooling, and weighted average pooling. Typically, if one is more concerned with the overall features of the image rather than the fine local features, then the pooling operation will play an approximate invariant to the representation of the input. In layman’s terms, if the input image is transformed by a small number of local transformations, and then the same pooling operation is used on both the before and after transformations, the representations of the two cases will turn out to be approximately the same. If you only need to determine whether a certain feature appears in the input image, and do not care about the specific location of the feature, the pooling layer can help to complete this judgment.4 An example of the pooling operation is shown, in which the pooling kernel has a size of 2 × 2, and the pooling operation has a step size of 2. The step size is determined by the offset where the pooling kernel is moved in the input image.

Typically most convolutional neural network models have a fully connected layer, which serves to make an integrated connection between the outputs of the convolutional or pooling layers, so that the previously localized features become global features after passing through the fully connected layer. The fully connected layer looks at the output of the previous layer and determines which category these features better match. Given different weight values, the probability of being correct for the different categories will be higher.The fully connected layer typically produces a vector that is multidimensional and has dimensions that are linked to the number of target categories.

In a convolutional neural network, the convolutional layer, the pooling layer, and the fully connected layer can usually be referred to as hidden layers, so the neural network model can be simplified to an input layer, a hidden layer, and an output layer.

In the YOLO family of algorithms, the model’s prediction frames are output and backpropagated based on the initial anchor frames, so the initial size of the anchor frames has a direct impact on the final accuracy of the model. Prior to YOLO-V5, the determination of the anchor frame size usually relied on the K-Means algorithm, which had to be performed separately before model training. YOLO-V5, on the other hand, simplifies the process by integrating the anchor frame size calculation process into the model training. It is no longer necessary to run the program independently. At the same time, if the adaptively generated anchor frame size is not satisfactory, the user can still customize the anchor frame.

The output part of YOLO-V5 is responsible for determining the category of the target and predicting its bounding box.The loss function of YOLO-V5 consists of three main components, the bounding box prediction error, the confidence score error, and the category prediction error. The total loss function expression is shown in equation (15). Where box_gain,cls_gain,obj_gain denotes the weight of the corresponding loss, respectively, with general default values of 0.05, 0.5, and 1.0: (15) Loss=box_gain×bbox_loss+cls_gain×cls_loss+obj_gain×obj_loss \[Loss=box\_gain\times bbox\_loss+cls\_gain\times cls\_loss+obj\_gain\times obj\_loss\]

The bounding box loss function discards the earlier version of the GIoU-Loss computation in favor of the CIoU-Loss computation.The GIoULoss computation is shown in Equation (16): (16) GIoULOSS=IoU−|C/(A∪B)||C| \[GIo{{U}_{LOSS}}=IoU-\frac{|C/(A\cup B)|}{|C|}\] where A is the predicted box, B is the true box, and C denotes the minimum outer rectangle of the two. Although this approach solves the problem of bounding box mismatch, more accurately represents the degree of overlap between the two bounding boxes, and speeds up the convergence of the model, the problem of degradation of the intersection and merger ratio (IoU) still occurs if one bounding box completely contains the other box. CIoULoss, on the other hand, considers the aspect ratio of the predicted box and the real box, the overlap area between the two boxes, and the center distance at the same time, which is better by considering the difference between the predicted box and the real box through more dimensions.The computation of CIoULoss is shown in Eq. (17): (17) CIoULOSS=1−IoU+ρ2(b,bgt)c2+αν \[CIo{{U}_{LOSS}}=1-IoU+\frac{{{\rho }^{2}}(b,{{b}^{gt}})}{{{c}^{2}}}+\alpha \nu \] (18) ν=4π2(arctanwgthgt−arctanwh)2 \[\nu =\frac{4}{{{\pi }^{2}}}{{(\text{arctan}\frac{{{w}^{gt}}}{{{h}^{gt}}}-\text{arctan}\frac{w}{h})}^{2}}\] (19) α=ν(1−IoU)+ν \[\alpha =\frac{\nu }{(1-IoU)+\nu }\]

In Eq. ρ represents the Euclidean distance between the center points of the bounding box, c represents the diagonal length of the minimum closed region of the predicted and real bounding boxes. And b and b^gr represent the coordinates of the center points of the predicted and real boxes, respectively. v is used to measure the consistency of the aspect ratio and the expression is shown in equation (18). α is the weight parameter and the expression is shown in equation (19).

YOLO-V5 is usually calculated using the binary cross entropy function when dealing with classification loss, the definition of which is shown in Equation (20): (20) L=−ylogp−(1−y)log(1−p)={ −logp,y=1−log(1−p),y=0 \[L=-y\text{log}p-(1-y)\text{log}(1-p)=\left\{ \begin{array}{*{35}{l}} \quad -\text{log}p,y=1 \\ -\text{log}(1-p),y=0 \\ \end{array} \right.\]

In the formula, y denotes the actual label of the input sample (positive class samples are labeled 1 and negative class samples are labeled 0), while P denotes the probability that the model predicts the sample to be a positive class.

The confidence level of each prediction frame reflects the accuracy of that prediction frame, with a higher confidence level indicating a better match between the prediction frame and the actual frame.

As with calculating the classification loss, YOLO-V5 uses the quadratic cross-entropy function by default to calculate the confidence loss.

In order to enhance the feature extraction capability of the network structure, a lightweight convolutional attention module CBAM is introduced into the YOLO-V5 detection model, which simultaneously introduces the attention mechanism in both channel and spatial dimensions, improving the feature extraction capability of the network model without significantly increasing the computational effort.

The channel attention mechanism structure first passes the input feature map through two parallel average pooling and maximum pooling layers to change the height and width of the feature map to 1. Then it passes through the ShareMLP module, in which the number of channels is compressed and then expanded to the original number of channels, and goes through the ReLU activation function to get the two activated results. The two output results are summed element by element and then weighted by a Sigomid activation function feature to get the output result, which is finally multiplied by the original map to change back to the original size.

The structure of spatial attention mechanism firstly goes through two parallel average pooling layers and maximum pooling layer to highlight the feature region and get two 1×H×W feature maps, next it goes through a convolutional layer convolved into 1-channel feature maps, and then normalized by a Sigmoid function to get the new feature maps and generate the spatial attention matrix. Finally after weighting operation to add spatial attention to the feature maps to get new feature maps.

The V3 version of the YOLO family of algorithms includes multi-scale feature fusion to increase the network’s predictive capacity by combining shallow detail information with deep semantic information.

YOLO-V5 employs the Feature Pyramid Network FPN and the Path Aggregation Network PANet as its architecture for multiscale feature fusion [33]. FPN consists of bottom-up feature extraction and top-down horizontal feature fusion. Although FPN facilitates the transfer of deep semantic information to shallow layers, detailed information may be lost in the process.

Compared to PANet, BiFPN has the following design changes [34].

1) Removal of nodes that only serve as inputs. If a node only has input connections and does not perform feature fusion, its contribution to a network designed to integrate different features is minimal. Therefore, removing it not only has little impact on the performance of the network, but also helps simplify the architecture of the network.

2) Fusing more features without adding much cost. If the original input and output nodes are in the same layer, BiFPN adds an extra edge between the original input and output nodes.

3) A feature network layer containing bidirectional paths is constructed. Compared to PANet, which contains only one top-down and one bottom-up path, BiFPN achieves deeper feature fusion by integrating a set of bi-directional paths into a single network layer and by performing multiple iterations on the layer. A set of bidirectional paths as a feature network layer. Unlike PANet, which has only one top-down and one bottom-up path, BiFPN achieves a higher level of feature fusion by using a set of bi-directional paths as one network layer and repeating the same layer several times.

4) A weighted summation is used to assign differentiated weights to different features. The specific weighted fusion process is demonstrated in Eq. (21). This method is similar to the softmax-based fusion strategy in terms of fusion efficiency and accuracy: (21) O=∑iωi×Iiε+∑jωj \[O=\sum\limits_{i}{\frac{{{\omega }_{i}}\times {{I}_{i}}}{\varepsilon +\sum\limits_{j}{{{\omega }_{j}}}}}\] Where ω_i denotes the weight parameter of the model, and ω_i ≥ 0, I_i denote the inputs of the different layers, and ε is a small value to ensure the stability of the values.

The use of dual-spectrum camera to capture video images, the camera is placed on the turntable and rotate with the turntable along the horizontal direction, you can collect 360 ° of video data.

The collected video image data is recognized by the target detection algorithm based on deep learning, and then the recognized pseudo-targets are rejected, based on the time-domain characteristics of the image sequence to determine whether the target is a real moving target, if it is a real target, then send an early warning signal to the alarm, if it is a pseudo-target, then it will be rejected directly.

The recognized video image is displayed on the program interface in real time, and if a moving target is detected, it is selected with a bracket box and its category is indicated, meanwhile, the image of the detected moving target and its result are automatically stored in the local disk for later viewing.