Applying Deep Learning Networks to Identify Optimized Paths in Gymnastic Movement Techniques

Where, S = (S₁,S₂,...,S_J), S_J ∈ R^w×h, j∈{1,…J}, indicates that J predictions need to be made for each joint location with J confidence maps.

L = (L₁,L₂,...,L_C), L_C ∈ R^w×h×2, c ∈ {1,…,C} indicates that C vector fields are obtained for each prediction of each part of the detected target.

OpenPose overall uses a VGG network as a skeleton, and its network structure is shown in Fig. 1.The VGG network is mainly divided into two main parts, each of which consists of a continuous convolutional network [18]. The upper half of the network predicts the confidence map of skeletal points, which belongs to the first branch, and the lower half of the network predicts the site association vector field, which belongs to the second branch. The two branches are able to iteratively predict and regress the confidence map and site-association vector field at the same time, and each regression completes one iteration. The whole detection system is completed after t∈(1...T) iteration.

Figure 1.

OpenPose network structure

F is the input for the beginning stage of the two branches, a set of mappings, obtained from the analysis of the original input data by the CNN. A set of data S¹ = ρ¹(F) and L¹ = Φ¹(F), i.e., confidence mapping and association vector field, are obtained in the first stage of starting training, and S¹, L¹, and F need to be concatenated in order to predict the results more accurately. The prediction result at stage t of training is: (1) St=ρt(F,St−1,Lt−1),∀t≥2 (2) Lt=φt(F,St−1,Lt−1),∀t≥2

Where, ρ^t, Φ^t are the results obtained after t rounds of iterations in the CNN middle stage.

After obtaining the prediction results, the loss function needs to be calculated once in the termination segment of each stage in order to get better recognition accuracy. Meanwhile the OpenPose algorithm adds the L2 loss function in order to determine the error between the prediction result and the real data. For the case that there may be undetected data in some of the datasets that are not completely labeled, it is considered to be caused by the weighting problem of the loss function, so it is spatially weighted and points out those regions that are not detected, where the loss function for stage t is shown below: (3) fSt=∑j=1J∑pW(p)⋅‖ Sjt(p)−Sj*(p) ‖22 (4) fLt=∑c=1c∑pW(p)⋅‖ Lct(p)−Lc*(p) ‖22

where Sj* denotes a confidence map of the target joint locations of the actual dataset, Lc* denotes the site association domains in the labeling of the actual dataset, and W is a binary mask used to avoid error penalties in some cases. The OpenPose algorithm can still make predictions even when p is not labeled by the detection, and the gradient is periodically corrected during the training of the prediction to prevent missing gradients, and the objective function can be expressed as: (5) f=∑t=1T(fst+fLt)

The general flow of skeletal point extraction using the OpenPose algorithm is shown in Figure 2.

Figure 2.

OpenPose's extraction of the skeleton network process

Step 1: Input color image, preprocessing, use VGG-19 to train the processed image, complete the selection of color image external feature points.

Step 2: Divide into two network branches for iterative prediction, one branch is trained to predict the skeletal points of the human body, and the other branch is trained to predict the connecting lines of the skeletal points, and the two branches can be processed in parallel, and then the data information is stored in JSON data format.

Step 3: Select the skeletal point data information, convert the information stored as JSON format to text format, and store the efficient data information.

Step 4: Preprocess the skeletal point data information.

The OpenPose model can obtain the 18 major skeletal points information of the human body as shown in Figure 3, and different skeletal points correspond to different code numbers.

Figure 3.

The skeleton diagram for OpenPose acquisition

OpenPose algorithm is developed with Caffe as the framework, an open source, human posture estimation model for research use.OpenPose model is richer in features, mainly the following:

1) Multi-platform operation, applicable to a variety of operating systems, such as Windows system, Ubuntu system and so on. C++, Python API interface is provided, which can meet the programming needs of different people.

The VGG-19 network used in the OpenPose model is responsible for skeleton feature extraction, and its network structure is schematically shown in Fig. 4, which results in slow processing speed due to its high computational cost and deep network structure. To alleviate this problem, a lighter weight network architecture or improvement of existing algorithms can be considered to enhance the real-time performance of the model while maintaining or improving its accuracy.

Figure 4.

VGG-19 network structure

VGG-19 network design is relatively simple, its main focus is on the network depth, it increases the network depth by using the same type of convolutional layer several times, thus improving the feature extraction ability, but its parametric quantity is large, the computational complexity is high, its calculation requires a large amount of computational resources, and its inference is relatively slow, which is not conducive to carry out real-time detection, in order to solve the above problems of the VGG-19, the present study adopts the lightweight MobileNet network to replace the VGG-19 network in the OpenPose network model.

In 2019, the Google team introduced the latest version of the MobileNet family, MobileNet-V3 [19]. This version inherits the deep separable convolution technique from V1 and the inverted residuals and linear bottleneck design from V2. In addition, the V3 version integrates the SENet module, which enables automatic tuning of model parameters. To achieve network optimization without sacrificing accuracy, MobileNet-V3 is used in this study to replace the original VGG-19 network architecture.

The traditional convolutional kernel used in VGG-19 tends to cause the gradient to disappear as the network depth increases, which affects the training effect. In contrast, MobileNet-V3 uses a depth-separable convolutional kernel technique with a residual structure to mitigate the problem of gradient vanishing. The structure utilizes the strategy of expanding the input dimension and then reducing the input to enhance the effect of gradient propagation, thus effectively reducing the storage requirements during the computation process, which not only reduces the consumption of computational resources, but also improves the speed of data processing.

MobileNet-V3 follows the depth-separable convolution of the V1 version and inherits the features of the V2 version, including the inverted residual structure and linear bottleneck structure. In addition, MobileNet-V3 introduces the SENet module, which can automatically optimize the model parameter settings.

In MobileNet-V3, the improved h-swish activation function is used instead of the ReLU activation function, whose formula is shown in Eq. h-swish can not only mitigate the influence of the ReLU activation function, but also is smoother than ReLU, which improves the performance of the model on low-dimensional data, and reduces the hyper-parameters when the neural network layers are deepened to reduce the computational cost. : (6) h−swish[ x ]=x⋅ReLU6(x+3)6

MobileNet-V3 optimizes MobileNet-V2, and one of the improvements is for the V2 network tail. In MobileNet-V3, the network structure was optimized. To maintain a sufficient number of feature maps, the 1 × 1 convolutional layer is placed after the average pooling layer. At the same time, the 3 × 3 DConv and other 1 × 1 convolutional layers were removed. This adjustment not only maintains the accuracy of the model, but also significantly reduces the number of parameters and computational requirements of the model.

The main innovation of the improved version of OpenPose is that its feature extraction part consists of a Bneck module (bottleneck module). The process starts with a 1 × 1 convolutional layer to extend the channel, followed by a batch normalization (BN) layer with an h-swish activation function layer, and ends with a 1 × 1 convolutional layer for channel compression, or compression after average pooling. In order to avoid the loss of information due to the deepening of the network layers and to improve the flow of feature information, the module also incorporates the concept of residual connectivity.

From the above, it can be seen that MobileNet-V3 employs the h-swish activation function to improve the performance of the model in the low-dimensional feature space and enhances the importance adjustment of the feature mapping through the SENet module, and these improvements enable MobileNet-V3 to achieve a better balance between efficiency and performance.

The OpenPose network structure needs to go through five 7 × 7 convolution operations in both the upper and lower branches from stage2, and the overhead of the 7 × 7 convolution operations is very large. To ensure that the complexity of the OpenPose network model is reduced without any decrease in accuracy, this experiment replaces the large convolution kernel with a smaller convolution kernel to reduce the complexity of the model and to improve the efficiency of the model. The ensuing problem is that the small convolution kernel reduces the receptive field (i.e., the size of the input region that can be observed by the convolution), and hollow convolution is introduced to fill the missing receptive field during the convolution process.

In human behavior recognition, where spatial feature information is particularly important, the use of cavity convolution avoids oversampling in the convolutional network and helps maintain the spatial resolution of the feature map. In cavity convolution, a certain number of spaces are inserted between elements within the convolutional kernel to increase the range over which the convolutional kernel acts.

The 7 × 7 convolution in the original OpenPose network structure is improved as follows: a 1 × 1 convolution kernel with two 3 × 3 convolution kernels is utilized to replace the original 7 × 7 network, and the last layer of 3 × 3 convolution kernel is a null convolution with an expansion factor of 2 to solve the problem of missing receptive fields. A jump connection consisting of 1 × 1 convolutional kernels is added between the three convolutional kernels, which is to cope with the gradient vanishing problem that occurs as the depth of the network increases.

In summary, the improved OpenPose network model is shown in Fig. 5, with the Bneck module of MobileNet-V3 as a component, and the internal structure of the Bneck module is designed to efficiently extract key information to optimize the skeleton detection performance.The Bneck module to perform feature extraction, the network processes the input training set images by first using a normal convolutional layer with a step size of 2 operation to increase the number of channels, followed by feature extraction, and finally performs the downsampling process.

Figure 5.

Improved OpenPose-MobileNet-V3 network structure