Design of Convolutional Neural Network Optimization Algorithm Based on Embedded System and Its Application in Real-Time Image Processing

Convolution operation is one of the most important operators in CNN and is used in the convolutional layer of CNN. The convolution operation here is the same as convolution in image processing, where a convolution kernel performs a weighted summation of the image from left to right and from top to bottom. The difference is that in traditional image processing, the parameters of the convolution kernel are obtained in advance, such as the soble operator for edge detection, which can extract the horizontal and vertical edge features of the image. But here, the convolutional kernel parameters need to be obtained by continuous learning during the training process.

Convolution operation is inspired by this, each of its convolution kernel represents a feature, multiple channels of the convolution kernel on multiple channels of the input image convolution sum to get an output feature map, another set of multi-channel convolution kernel convolution to get another channel of the output feature map, the convolution process is shown in Figure 1. For convenience, the input and output feature maps are uniformly referred to as feature maps, or FMAP for short. The input channel of fmap is the same as the input channel of the convolution kernel filter, and the output channel of fmap is the same as the group number of the filter. Thus, the number of convolutional kernels per layer is the product of the number of input channels and the number of output channels.

Figure 1.

Convolutional diagram

Where C denotes the number of input channels, S denotes the width of the filter, R denotes the height of the filter, W denotes the width of the input feature map, H denotes the height of the input feature map, M denotes the number of channels of the output feature map, F denotes the width of the output feature map, and E denotes the height of the output feature map. Also when there are multiple sets of input feature maps, N will be used to indicate the number of batches, and a batch size of N is not indicated in Fig. 1. The expression for the convolutional layer is shown in equation (1): 1O[m][x][y]=Activation(B[m]+∑i=0R−1∑j=0S−1∑k=0C−1I[k][Ux+i][Uy+j]×W[m][k][i][j])$$\begin{array}{l} O[m][x][y] \\ = Activation\left( {B[m] + \sum\limits_{i = 0}^{R - 1} {\sum\limits_{j = 0}^{S - 1} {\sum\limits_{k = 0}^{C - 1} I } } [k][Ux + i][Uy + j] \times W[m][k][i][j]} \right) \\ \end{array}$$

where, 0 ≤ m < M, 0 ≤ x < F, 0 ≤ y < E, E = (H − R + U)/U, F = (W − S + U)/U, U are the step values. The time complexity CconvT$$C_{conv}^T$$ of the convolutional layer computation and the space complexity CconvS$$C_{conv}^S$$ of the weights are 2CconvT=O(M⋅C⋅R⋅S⋅E⋅F)$$C_{conv}^T = O\left( {M \cdot C \cdot R \cdot S \cdot E \cdot F} \right)$$ 3Cconvs=O(M⋅C⋅R⋅S)$$C_{conv}^s = O\left( {M \cdot C \cdot R \cdot S} \right)$$

The output of the convolution is usually immediately followed by the activation function, which serves to bring about nonlinearity. The introduction of nonlinear activation functions makes deep neural networks less of a linear combination and theoretically approximate arbitrary functions. It is worth noting that the activation function is often represented in a separate layer of the network when implemented in software; here, since the input and output of the activation function are single variables that do not need to be cached into memory, it is subsumed within the convolutional layer for hardware implementation.

Nowadays, the common CNN models use ReLU as the activation function more. First of all, the form of the ReLU function is very simple, the part greater than 0 is equal to itself, and the parts less than or equal to 0 are all 0. Its derivative is also very simple, only 0 or 1, which is very easy to realize. Second, ReLU’s derivative of 0 or 1 avoids the gradient vanishing problem in CNNs, making training easier to converge. Finally, ReLU will make the output of a part of neurons is 0, which brings the sparsity of the network, and can solve the overfitting problem.ReLU also has a disadvantage, because its derivative has a part of 0, it is easy to have a large number of neurons with the value of 0 when the network is training, which leads to the stop of the training process. Later on, many modified models were proposed, such as Leaky ReLU and Exponential LU, etc. The idea is to set the interval of x < 0 not to be set to a value of 0, but set it to the form of multiplying with the functions of α and x, and to correct α during the training process.

Pooling abstracts and compresses the results of convolution while preventing overfitting, making the network more robust.The conv layer of a CNN is generally followed by a pooling layer, making it resilient to image translation, scaling, and slight rotation. The pooling operation is very similar to conv, and the commonly used average-pooling and max-pooling is to average or take the maximum operation on the feature map within a 2*2 or 3*3 kernel.

The computational time complexity of pooling is 4CpoolT=O(M⋅E⋅F)$$C_{pool}^T = O\left( {M \cdot E \cdot F} \right)$$

where M denotes the number of channels of the output feature map, F denotes the width of the output feature map, and E denotes the height of the output feature map.

The Fully Connected Layer, also referred to as the FC layer, acts as a “classifier” in a convolutional neural network. It can represent a nonlinear combination of features learned from previous layers by learning weights. It processes the input and outputs a N-dimensional vector, where N corresponds to the number of classifications. Since it is fully connected, every neuron in the previous layer is connected to every neuron in this layer. Assuming that the inputs and outputs are in the form of vectors, the fully connected layer operation can be represented by equation (5). Where all the inputs of the previous layer are represented by vector x, y_o is an output element in the output vector y, w_o is a corresponding vector in the weight matrix woi$$w_o^i$$, and bias b_o is a corresponding element in the bias vector b. Activation denotes the activation function. 5yo=Activation(x⋅wo+bo)$${y_o} = Activation\left( {x \cdot {w_o} + {b_o}} \right)$$

The fully-connected layer is usually immediately followed by the pooling layer, and in the case of multiple input channels, the fully-connected layer can be viewed as into a convolution operation, which is equivalent to a convolution with a kernel size equal to the size of the input fmap. At this point, the multidimensional input data can also be viewed as a one-dimensional vector and computed according to equation (4). The fully connected layer is computationally intensive and parameter intensive, and the parameters will be much larger than the convolutional layer with the same number of input and output neurons as the convolutional layer. This is because the parameters of the convolutional layer are shared within a single input channel, unlike the fully connected layer where each input neuron has different parameters. The computational time complexity CfcT$$C_{fc}^T$$ and the space complexity CfcS$$C_{fc}^S$$ of the weights for the fully connected layer are respectively 6CfcT=O(M⋅C)$$C_{fc}^T = O(M \cdot C)$$ 7Cfcs=O(M⋅C)$$C_{fc}^s = O(M \cdot C)$$

where M is the length of the output vector and C is the length of the input vector.

The softmax layer is used in the final classification process to map the outputs of multiple neurons to the interval (0,1)$$\left( {0,1} \right)$$, mainly through the normalized exponential operation, and the result after normalization is used as the final classification probability output. Assuming a vector v, v_i represents the ith element of v, then the softmax value of this element is 8Si=evi∑jevj$${S_i} = \frac{{{e^{{v_i}}}}}{{\sum\limits_j {{e^{{v_j}}}} }}$$

The softmax layer allows for a greater probability of output results for large input data and a smaller probability of output results for small input data and normalization.

In convolutional neural network forward propagation, the convolutional module has a high similarity to the pooling module in that both have outstanding sliding parallelism of windows, which can be generalized to other layers using the convolutional layer as an example. The huge multiply-accumulate operation of the convolutional layer consumes a major portion of the computational resources of the entire system, and the development of the parallelism of the convolutional layer is thus a major part of the overall forward propagation parallelism. The essential operation of convolution is multiply-accumulate, and the principle of convolution operation is shown in Fig. 2, which has the following four main parallel characteristics.

Figure 2.

Multiplication-cum-operation diagram

There is a correlation between CNN layers, and the extracted output features of the previous layer are often the input features of the next layer. This correlation increases the difficulty of parallelism, so inter-layer parallelism is different from intra-layer parallelism, inter-layer parallelism focuses on the overall performance of the system, allowing functions or tasks to be executed in an overlapping manner, improving the throughput of the overall design, reducing the inter-task interval, and increasing the speed to improve the computing performance from the whole process.

FPGA is a programmable logic gate array that can be programmed and configured by the user to perform various logic functions. Due to its advantages of high flexibility, reconfigurability, and the ability to realize high-speed computing, it is widely used in a variety of fields such as digital signal processing, communications, image processing, and control systems. Here are the basics of FPGAs.

Structure of FPGA: FPGA consists of programmable logic units, input/output blocks, clock management units, digital signal processor multipliers, etc. These units are connected through programmable interconnect resources.

Programming of FPGA: FPGAs are programmed in two ways, one using hardware description languages such as Verilog and VHDL, and the other using high-level synthesis tools HLS, such as VIVADO HLS.

Timing analysis of FPGA: Due to the programmability of logic units and interconnect resources in FPGA, timing analysis is a very important part of FPGA design, which requires the setting of timing constraints and the verification of timing analysis.

FPGA design flow: The design flow of FPGA usually includes design, synthesis, realization and verification, etc., in which synthesis and realization are the core links in FPGA design.

In summary, FPGA is a programmable logic device, which is one of the important tools in the design of digital circuits, and is now widely used in many fields.

Zynq UltraScale+ is a highly integrated programmable SoC (system-on-chip) from Xilinx that combines the latest generation of Xilinx’s UltraScale+ FPGAs (field-programmable gate arrays) and ARMCortex-A53/Cortex-R5 multicore processors, which utilize the TSMC 16nm FinFET +process with excellent performance and low power consumption. It supports many different types of IO standards, including PCI Express Gen3, USB3.0, Gigabit Ethernet, DisplayPort, etc. It also integrates high-speed memory controllers, video codecs, digital signal processors, and a variety of peripheral interfaces, providing users with a wealth of hardware resources and interfaces.Zynq UltraScale+ MPSOC is also equipped with programmable logic and a hardware debugger that allows users to design, simulate and debug using the VIVADO development kit. In addition, Xilinx provides a wealth of development tools and software libraries to help users quickly develop efficient and reliable systems. Currently Xilinx launched products equipped with this type of chip include Ultra 96 v2, ZCU102, ZCU104 and ZCU106, which are mainly used in machine vision, surveillance, automobile automatic driving and other scenarios. Figure 3 shows the architecture of the ZCU104. The architecture diagram shows a quad-core ARM Cortex-A53 and a dual-core ARM Cortex-R5, as well as a DDR4 memory controller, on the ZCU104 development board. In addition, the development board includes a programmable logic area FPGA and multiple high-speed serial connectivity ports, including PCI Express and DisplayPort.MIO and HP are multiple input/output ports in the processor subsystem that can be used to communicate with external devices.

Figure 3.

Architecture diagram of ZCU104

Overall, the Zynq UltraScale+ fabric integrates high-performance FPGAs and multi-core processors, providing powerful computing and communication capabilities for a variety of application scenarios, making it an excellent programmable SoC chip.

In order to simplify the difficulty of implementing CNN algorithms, deep learning frameworks are needed to accomplish the deployment of their models. Deep learning frameworks encapsulate the regular operations of the algorithm into operators that can be called directly, and encapsulate the hardware architecture to reduce the complexity of the algorithm and the difficulty of development.Darknet framework is a lightweight, open source, language-based neural network framework that supports both CPU and GPU computation, and can be freely coded to configure the configuration file to customize the structure of the network and so on.The Darknet framework has the advantages of easy installation, good portability, no dependencies, etc. It is ideally suited for the deployment of convolutional neural networks on embedded platforms. The Darknet framework has the advantages of easy installation, good portability, no dependencies, etc., and is very suitable for the deployment of convolutional neural networks in embedded platforms.The block diagram of the Darknet architecture is shown in Figure 4.

Figure 4.

Darknet framework block diagram

In the task of image processing, Im2col is a kind of transformation process to convert pictures into matrix columns, which is one of the earliest important ways to perform convolution for convolutional neural networks in the Caffe framework. In Darknet framework, Im2col is also used to convert the 3D tensor to a 2D matrix, and then make full use of the generalized matrix multiplication algorithm to perform the convolution operation for different convolutional neural network models, and finally convert the obtained 2D matrix results to a 3D matrix output through Col2im. Compared with the traditional direct convolutional computation, the Im2col+Gemm approach is widely used in current convolutional neural networks.

Usually, when processing an image, the number of channels of the image also needs to be considered, and the multi-channel convolution calculation process is a convolution calculation of a multi-channel input feature map with multiple convolution kernels in a three-dimensional space. The three-dimensional convolution can be simply expressed as a simple superposition of the two-dimensional convolution, which is calculated as shown in Equation (9). Where Z is the convolution output, I is the convolution input, K is the convolution kernel, c is the input channel, h is the image height, w is the image width. 9Z(i,j)=I(i,j)*K(i,j)=∑c∑h∑wIc(i−h,j−w)Kc(h,w)$$Z(i,j) = {I_{(i,j)}}*{K_{(i,j)}} = \sum\limits_c {\sum\limits_h {\sum\limits_w {{I_c}} } } \left( {i - h,j - w} \right){K_c}\left( {h,w} \right)$$

Im2col is to transform the input feature map into a matrix for convolutional computation by sacrificing space, i.e., converting a three-dimensional matrix into a two-dimensional matrix by spatial dimensionality reduction. The idea of the transformation is that the algorithm implements the process by arranging all the data needed for each loop into column vectors, and then splicing them into a matrix in the column direction according to the order of the channels.

It can be seen from the algorithm that im2col reduces the number of loop layers by reducing the data dimensions, thus reducing the complexity of time. Meanwhile, after the three-dimensional matrix is transformed into a two-dimensional matrix, the data required for the operation is stored in the computer in a continuous memory, which greatly improves access speed. Moreover, the number of data elements after dimensionality reduction is greater than the number of data before dimensionality reduction, resulting in increased memory usage. So Im2col is an algorithm that sacrifices space for time.

After converting the 3D tensor into a 2D matrix by Im2col algorithm, it is necessary to call the Gemm computational library to compute the convolution of the input matrix with the convolution kernel.Gemm is an important way to compute matrices in deep learning. More than 99% of the computations in convolutional neural networks are in the convolutional and fully connected layers, and the matrix computations in these two layers are basically realized by Gemm.

From the matrix multiplication algorithm, it can be seen that the time complexity of its algorithm is O(n3)$$O\left( {{n^3}} \right)$$. Its complex multiply-accumulate operation will occupy a large amount of computational resources in the embedded platform, and the resources in the embedded platform are limited, so it needs to be optimized for Gemm. Therefore, it can be considered from the following two aspects, the first is based on the algorithm analysis approach, according to the computational characteristics of matrix multiplication, optimized from the mathematical point of view, such as Winograd algorithm. Secondly, it can also be optimized based on the software method, which selectively adjusts the computation order according to the hierarchical structure characteristics of the computer storage system system, mainly including loop splitting, vectorization, memory rearrangement and so on.

A total of 3 groups of parameters are used for experimental comparison. The first group uses only the Cortex-A9 single core at the PS side of the Zynq chip to test the performance of the CNN gas pedal, and the PL side of the Zynq is not involved in the operation; the second group uses the Intel i7-9750H model CPU to implement the CNN network; and the third group adopts the Zynq-7035 chip in combination with optimized CNN, which is co-designed by the software and hardware at the PS side and the PL side to accelerate the the computing process of CNN network. The CNN networks built under different hardware configurations are compared, and the average time spent on each layer and the total time spent on CNN network operation are recorded, and the relevant test results are shown in Table 1.

The PS side and PL side of the Zynq-7035 chip combine the optimized CNN to accelerate the CNN network by adopting cyclic chunking, cyclic water flow, cyclic unrolling, and cache optimization, and embedding the computation of convolutional, pooling, and fully-connected layers in the PL side of the Zynq, which has a high degree of parallelization and an improved computation rate of the CNN network, even though its operating frequency is only 100 MHz. On the other hand, the degree of parallelization of CNN network under Cortex-A9 single core and Intel CPU is low, and the operating frequency of Cortex-A9 single core is low, both of which are not suitable for operations with high parallelism.

From Table 1, it can be seen that compared to the Cortex-A9 single core and Intel CPU realizing the whole CNN network, the PL-side accelerated CNN network can achieve 658.12 and 23.18 times speedup, respectively. The convolutional layer operation occupies most of the computing time of the CNN network, no less than 95% of the total time, because the computation of the convolutional layer is significantly larger than that of other layers. Although there are more fully connected layer parameters, the computation is still simple and fast.

In different convolutional layers, the first convolutional layer has less computation time, although the size of input and output features is larger, but the number of input and output channels is smaller, so the computation process of the first convolutional layer can be completed in a shorter time; on the contrary, the last convolutional layer, although the number of input and output channels is larger, but the size of the input and output features is smaller, and there is not a great computation, and the computation spent In contrast, the last convolutional layer has more input and output channels, but the size of the input and output features is small, which is not very computationally intensive and the computation takes less time; the 4th and 6th convolutional layers, which are in the middle position, do not have the largest number of input and output channels and sizes of the features, but the computation process is more cumbersome, and taking the computation of the pooling layer into account, the calculation takes more time. Among the different fully-connected layers, the first fully-connected layer takes the most time to compute, and the latter two layers have a rapidly decreasing time to compute because of the sharp decrease in the number of input and output channels.

When the Zynq-7035 chip is used to realize CNN acceleration, the resources of the PS side and PL side of the chip will be used at the same time, and after the PL side of the Zynq generates hardware circuits, the corresponding FPGAs with various resource occupancy are shown in Figure 5. In this paper, when optimizing the algorithm and subsequently designing the CNN gas pedal, the input features, weights and biases stored in the external memory DDR3 are loaded into the PL side of Zynq as much as possible to reduce the amount of data transfer between the PL side and the external memory. While the FPGA commonly stores IP cores as FIFO or RAM, these need to occupy BRAM resources. Figure 5 shows the utilization of different resources for CNN acceleration.

Figure 5.

Diagram of different resources used by CNN acceleration

From Fig. 5, it can be seen that the CNN acceleration takes up the most BRAM resources, accounting for 90% of the total, which is consistent with the design requirements. In addition, the DSP resources in the CNN acceleration also occupy a lot of resources, mainly realizing the multiplication and addition operations in the convolutional layer and the fully connected layer, and the other resources occupy less.

Zynq combined with optimized CNN is applied in person recognition detection, and hardware-level acceleration of target detection is carried out using images from different scenarios, such as multi-person scenarios, small target scenarios, and mask recognition scenarios. The experiments are summarized in Table 2:

In order to verify the Zynq performance of combining optimized CNNs, tests of Zynq under CPU, GPU and FPGA platforms with the same images were carried out, where the CPU and GPU platforms were tested on the COCO dataset under the Pytorch framework, and the arithmetic data types were all Float32. From Table 2, it can be seen that, in the case of changing only the data quantization, the recognition of the mAP is not greatly cut down. However, in terms of real-time performance, the FPS of Zynq combined with optimized CNN can reach 220, and it only takes about 4.5ms to recognize a graph, compared with the 355FPS achieved by GPU, this design can achieve 220FPS with less than 4.5W power consumption, which is much lower than the 185W of GPU, while CPU, due to the characteristics of its serial processing, can only get Due to its serial processing nature, the CPU only achieves 30FPS and consumes 70W. Since the performance gap between the ARM side of Zynq and the CPU is too large, it is not included in the discussion. Combining optimized CNNs, Zynq achieves slightly lower acceleration than GPUs and much higher acceleration than CPUs with 2.5/100 of the power consumption of GPUs and 6.4/100 of CPUs. Overall, the acceleration of CNNs on Zynq development boards with hardware and software collaboration is realized.

Combined with the optimized CNN of Zynq in the specific implementation process will consume a large number of FPGA logic resources, in the development tool Vivado to generate the hardware resource consumption report, the resource consumption statistics are shown in Table 3. Combined with Table 3 can be seen, a total of 336 DSPs are consumed, BRAM resources 93.6, LUT resources 54,891, FF resources 66,130.

In order to evaluate the processing capability of Zynq combined with optimized CNNs, performance tests are conducted in this paper. Since the model is trained based on the GTSDB dataset, the previously segmented test set is used as the test data with a total of 100 sheets and the number of traffic signs it contains is counted. In addition, in order to make the test results more reliable, this paper carries out 5 times of random segmentation on the produced dataset and completes the corresponding model training, and its test set is then used as the algorithm test data.

After a number of performance tests to get Table 4, from Table 4 can be seen in the algorithm in the process of detecting and identifying traffic signs, for the detection of traffic signs, there are a certain number of leakage cases, but the leakage rate is kept at a low level of 8.28%, analyzing the reason is mainly due to the candidate frame extraction algorithm is affected by certain complex environments, such as extreme illumination, before and after the scene similarity, and then the training of the model is not For the recognition of traffic signs, the average recognition rate of 97.8% shows better recognition performance, thus reflecting the great advantage of CNN network in solving the classification problem. For traffic sign image single detection time consuming, although the single time consuming given by the performance test has not reached the industrial level of real-time, it is enough to confirm in order to evaluate the feasibility of the application of Zynq combined with optimized CNN in real-time recognition of traffic sign detection.

Comprehensively, the above test results and analysis show that Zynq combined with optimized CNN has a better detection and recognition ability, which basically meets the demand index of the initial application. Through subsequent research and improvement, its detection and recognition level can be further improved so as to meet the real real-time image processing requirements.