Data-driven Optimization of Folk Dance Inheritance and Protection Strategies and Their Realization Paths

In this paper, a binocular stereo vision depth motion vision image camera is used to collect video images of folk dance movements and record the whole process of dancers’ training, and the accuracy of the collected data is improved by constructing a binocular stereo vision model [31].

The Q point in space is described by Q_x(X_x, Y_x, Z_x) in the camera coordinates, and the Q point in the world coordinates is described by Q_w(X_w, Y_w, Z_w), and the expression of the relationship between them is: (1)[ Xx Yx Zx]=[ R T 0¯ 1][ Xw Yw Zw 1]$$\left[ {\begin{array}{*{20}{c}} {{X_x}} \\ {{Y_x}} \\ {{Z_x}} \end{array}} \right] = \left[ {\begin{array}{*{20}{c}} R&T \\ {\bar 0}&1 \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {{X_w}} \\ {{Y_w}} \\ {{Z_w}} \\ 1 \end{array}} \right]$$

where the translation vector is described by T; 3 × 3 the rotation matrix is described by R. Based on the stereo imaging theory, the image coordinate system and camera coordinate relationship are derived as follows: (2)Zχ=[ x y 1][ f 0 0 0 0 f 0 0 0 0 1 0][ Xw Yw Zw 1]$${Z_\chi } = \left[ {\begin{array}{*{20}{c}} x \\ y \\ 1 \end{array}} \right]\left[ {\begin{array}{*{20}{c}} f&0&0&0 \\ 0&f&0&0 \\ 0&0&1&0 \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {{X_w}} \\ {{Y_w}} \\ {{Z_w}} \\ 1 \end{array}} \right]$$

where the camera focal length is described by f. The relationship between the world coordinates and image coordinates of the Q points can be found by deriving Eq. (1) and Eq. (2) into Eq. (3): (3)Zχ=[ cx 0 α0 0 0 cy β0 0 0 0 1 0][ R T 0˜ 1][ Xw Yw Zw 1]$${Z_\chi } = \left[ {\begin{array}{*{20}{c}} {{c_x}}&0&{{\alpha_0}}&0 \\ 0&{{c_y}}&{{\beta_0}}&0 \\ 0&0&1&0 \end{array}} \right]\left[ {\begin{array}{*{20}{c}} R&T \\ {\tilde 0}&1 \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {{X_w}} \\ {{Y_w}} \\ {{Z_w}} \\ 1 \end{array}} \right]$$

where the internal parameters of the camera are described by c_x, c_y, respectively. The above model can accurately capture video images of dancers’ motion movements at spatial point locations, laying the foundation for the following folk dance movement recognition.

The original folk dance visual images collected are color 3-channel images, including a total of 2500 color variations. Therefore, this paper implements image preprocessing to transform the collected original folk dance visual images into grayscale images through the average value method, which reduces the amount of data and can improve the data processing capability. The calculation formula of the average value method is as follows: (4)Gray=(R+G+B)3$$Gray = \frac{{(R + G + B)}}{3}$$

Where the red channel is described by R; the blue channel is described by B; and the green channel is described by G.

Thresholding is applied to the folk dance visual grayscale image to produce a binarized folk dance visual image, and the thresholding formula is as follows:

Where, (i, j)-point gray gradient function is described by ε(i, j); pixel points are described by (i, j); and (i, j)-point gray value is described by f(i, j).

The background subtraction of binarized folk dance visual images is implemented by Gaussian model [32] in the following process:

Step 1: Construct the model. Set the pixel point values in the binarized folk dance visual image at the γ moment described by A, and at the γ moment, the vth Gaussian distribution weight is described by ω_v,γ, and its probability expression for the occurrence of this value is: (5)P(Aγ)=∑v=1κωv,γ×τ(Aγ,μv,γ,σv,γ)$$P({A_\gamma }) = \sum\limits_{v = 1}^\kappa {{\omega_{v,\gamma }}} \times \tau ({A_\gamma },{\mu_{v,\gamma }},{\sigma_{v,\gamma }})$$

where the mean is described by μ_v,γ; the probability density function by τ(A_γ, μ_v,γ, σ_v,γ); the period variance by σ_v,γ; and the number of Gaussian distributions by K.

Step 2: Update and match the model. A new input 1-frame image is used to determine the matching of pixel values and Gaussian model in this image by the following formula: (6)|Aγ−μv,γ−1|≤2.5σv,γ−1$$|{A_\gamma } - {\mu_{v,\gamma - 1}}| \leq 2.5{\sigma_{v,\gamma - 1}}$$

When the conditions of Eq. (6) are not met, foreground points are derived; when the conditions of Eq. (6) are met, background points are derived.

The Gaussian model is used to derive the background-subtracted folk dance visual image, but it is not conducive to the effect of folk dance movement recognition in the later stage because of the large amount of noise in the foreground environment. Therefore, the median filtering method is used to implement noise reduction on the background-subtracted folk dance visual image, and the specific process is as follows: 1)

Find the point where the center of the template and the center pixel point of the folk dance visual image coincide by moving the template, and take this point as the center of the window to complete the window construction.

where the window is described by ψ; the output and output pixel gray values are described by g(i, j), f(i − ψ, j − ζ), respectively. The gray value of the pixel at the center of the window is the value calculated by the above equation.

In this paper, we propose a neighboring frame feature processing module to capture the relationship between neighboring frames for learning, and add the feature relationship between neighboring frames to capture the subtle transformations while preserving the original temporal features. The spatial feature processed feature sequence is set to Xin={Xini|i=1,...,s}∈Rc×s×h×w$${X_{in}} = \{ X_{in}^i|i = 1,...,s\} \in {R^{c \times s \times h \times w}}$$ where c, s, h, w corresponds to the number of feature channels, the total number of time frames, and the height and width of the feature sequence in the input feature sequence, respectively. The neighboring frames are divided into different blocks and used as the mean feature map. The above process can be expressed as: (8)Xstat=Sm(Xdipt)$$X_{sta}^t = {S_m}(X_{dip}^t)$$

where S_m(·) denotes the average value taken along dimension s.

Obtaining the sequence of skeleton features between neighboring frames, the processing flow of the neighboring frame feature processing module is applied to t and t + 1 frames. Namely: (9)Xmotiont=Xint−Xstat$$X_{motion}^t = X_{in}^t - X_{sta}^t$$

where Xint$$X_{in}^t$$ is the feature sequence of frame t and Xint+1$$X_{in}^{t + 1}$$ is the feature sequence of frame t.

After extracting the temporal feature sequences from the neighboring frames, 1 × 1-convolution operation is performed on them and the Sigmoid activation function is used to process the relevant features, and then the residual connection is used to fuse them with the original feature maps, and this processing flow can be expressed as follows: (10)XAF=M(Xin,δ(conv(Xmotion)))$${X_{AF}} = M({X_{in}},\delta (conv({X_{motion}})))$$

Where, conv(·) denotes 1 × 1 convolution function, δ(·) denotes Sigmoid activation function and M(·) denotes element-by-element multiplication.

In order to fuse the features between all the frames after processing, the Hadamard product is taken for aggregation. The feature sequence after the neighbor frame feature processing module can be expressed as: (11)XAF=BN(M(Xmotion1,Xmotion2,⋯,XmotionT))$${X_{AF}} = BN(M(X_{motion}^1,X_{motion}^2, \cdots ,X_{motion}^T))$$

where BN(·) denotes batch normalization.

For temporal feature extraction, multi-scale temporal modeling is adopted [33]. Four different branches are described in detail below.

Each of the four branches inputs the human skeleton structure information after spatial feature extraction XS={XSi|i=1,…,T}∈Rn×C×T×V$${X_S} = \left\{ {X_S^i|i = 1, \ldots ,T} \right\} \in {R^{n \times C \times T \times V}}$$.

The first branch performs a 1×1 convolution operation on the input information, with the main purpose of reducing the channel dimension. After that, an expansion convolution with an expansion rate of 1 is performed to enrich the sensory field;

The second branch introduces the neighboring frame feature processing module to the input information to get the relationship between neighboring frames. After that 1×1 convolution with 5×1 convolution is introduced;

The third branch first performs 1×1 convolution on the input information and introduces a 3×1 maximum pooling layer after the convolution operation;

The fourth branch simply performs 1×1 convolution for channel dimension adjustment.

MTSGCN single-stream folk dance action recognition mainly includes three parts: spatial feature extraction, temporal feature extraction, and folk dance action classification. For the spatial feature extraction part, temporal extraction part, and folk dance action classification part, MCTTE and MTNFF models are used respectively to input the feature sequences after spatio-temporal feature extraction into the classifier to obtain the corresponding action classification. The specific network structure of MTAGCN [34] is shown in Fig. 1.

Figure 1.

MTSGCN single-flow human action recognition framework

Firstly, the input sequence is denoted as Xin={Xini|i=1,...,s}∈RN×C×T×V$${X_i}n = \{ X_{in}^i|i = 1,...,s\} \in {R^{N \times C \times T \times V}}$$, where N is the number of batches, C is the number of channels, T is the time dimension and V is the number of human joints, and the spatial feature extraction is performed on the input sequence using the MCTTE model, which is represented by the following formula: (12)Xdat=Conv(δ(Conv(Tm(Xin))))$${X_{dat}} = Conv(\delta (Conv({T_m}({X_{in}}))))$$

where T_m(·) denotes averaging along dimension s, Conv(·) denotes convolution operation, and δ(·) denotes Sigmoid nonlinear activation function.

The feature sequence X_dat after spatial feature extraction is fed as a new input sequence into different branches for temporal feature extraction.

The first branch performs a 1×1 convolution operation on the input information with the main purpose of reducing the channel dimensions. After that, an expansion convolution with an expansion rate of 1 is performed to enrich the sensory field;

The second branch first performs information fusion between neighboring frames on the input sequence, and the formula is expressed as follows: (13)XAF=BN(M(Xdat1,Xdat2,⋯,XdatT))$${X_{AF}} = BN(M\:(X_{dat}^1,X_{dat}^2, \cdots ,X_{dat}^T))$$

where BN denotes batch normalization, M(·) denotes element-based multiplication operation, and Xdatn,n∈(1,T)$$X_{dat}^n,n \in (1,T)$$ denotes a T-frame feature sequence processed for spatial features.

The 1×1 convolution with 5×1 convolution is introduced to the processed feature sequence.

The third branch first performs 1×1 convolution on the input information and introduces a 3×1 maximum pooling layer after the convolution operation;

The fourth branch simply performs 1×1 convolution for channel dimension adjustment.

Afterwards, the feature sequences processed in the four branches are concatenated and matrix addition is performed with the original input, after which the ReLU activation function is used to improve the nonlinear relationship between the layers of the network. The formula is represented as follows: (14)XMT=ReLU(Xdat+Concat(XMT1,XMT2,XMT3,XMT4))$${X_{MT}} = ReLU({X_{dat}} + Concat({X_{MT1}},{X_{MT2}},{X_{MT3}},{X_{MT4}}))$$

where X_MT1, X_MT2, X_MT3, X_MT4 corresponds to the results of the four temporal feature processing branches, respectively, and ReLU(·) denotes the ReLU activation function.

Therefore, spatial feature extraction is performed on the initial feature sequence first, and the feature sequence after spatial feature extraction is fed into the temporal feature extraction module to obtain the feature sequence after spatio-temporal feature extraction X_MT.

The sequence after feature extraction needs to go through the classifier to get the corresponding classification result. The probability value of the corresponding action category is output. The formula is expressed as follows: (15)Xout=Softmax(FC(AvgPool(XMT)))$${X_{out}} = Soft\max (FC(AvgPool({X_{MT}})))$$

where Softmax(·) denotes the activation function, FC(·) denotes the fully connected layer, and AvgPool(·) denotes the global average pooling operation.

The MTSGCN network proposed in this paper consists of four different input streams to perform spatio-temporal feature extraction on the respective feature sequence information and obtain the corresponding folk dance movement classification probabilities. Finally, a post-fusion approach is taken to sum up the corresponding probabilities to obtain the final folk dance movement classification. The complete network structure is shown in Fig. 2.

Figure 2.

MTSGCN network structure

In order to verify the validity of the added modules, ablation experiments are conducted in this section, and the three models are compared two by two, and the accuracy rates of the three input modes on the two datasets are shown in Table 1. In dataset 1, comparing the three input modes, it is found that the three models “Model 1, Model 2, and Model 3” have the highest accuracy rates in Rgb+Flow input mode, which are 88.37%, 93.81%, and 97.41%, respectively. In the Rgb and Flow input modes, the accuracies of the three models ranged from 82.84%-86.44% and 86.11%-91.75%, respectively.

In dataset 2, it is still the Rgb+Flow input mode that has the highest accuracy of the three models, whose accuracy is 72.09%, 73.05% and 77.39%, respectively. In Rgb and Flow input modes, the accuracy rates of the three models are less than 70%. From the table, it can be seen that the models in this paper have higher accuracy of the network trained on dataset 1. This is because the scenarios in dataset 1 are less complex and larger in size compared to dataset 2.

In dataset 1, this paper investigates 9 folk movements of “fanning, twisting, throwing waist, kicking, lowering the waist, pacing, swinging arms, jumping, and squatting”, which are named 1~9. In dataset 2, this paper investigates 9 kinds of movements, namely “pair fan, double waist twist, double waist turn, double arm swing, double kick, arm extension, double paw hitting the ground, double hip twisting, and double exchange step”, which are named 11~19 respectively. The confusion matrix can provide a good summary of the recognition accuracy of each action category, so this section uses the confusion matrix to visualize the recognition accuracy of actions in dataset 2 and dataset 1.

The confusion matrix for the identified results on dataset 1 is shown in Figure 5. The vertical axis in the coordinate diagram represents the true category of each action, the horizontal axis represents the predicted category of each action, and the diagonal line represents the correct recognition accuracy of each action category. The results show that on dataset 1, the recognition accuracy of the model for “waist twisting, fan turning, kicking-fan turning, swinging arm-throwing waist, jumping-kicking” is less than 70%. The recognition accuracy of “kicking-lower waist, lower waist-fan, lower waist-throwing waist, pacing-throwing waist, jumping-throwing waist, jumping-lower waist, squatting-fanning, squatting-jumping” is more than 95%. The recognition accuracy between the rest of the actions is within 70%-95%. Overall, the recognition accuracy of the model is good.

Figure 5.

Data set 1 identifies the confusion matrix of the result

The confusion matrix of the recognition results on dataset 2 is shown in Fig. 6. As can be seen from the figure, data set 2 also contains nine types of actions, in which the recognition accuracy between the other actions is above 60%, except for “two-person arm swing - two-person waist turn, two-person foot strike - two-person waist turn”. Among them, the recognition accuracy of two-person kicking and two-person twisting reaches 99.07%, followed by “two-person exchange step - arm extension, two-person arm swing - two-person twisting, two-person exchange step - two-person twisting, two-person exchange step - two-person twisting” and “two-person exchange step - two-person twisting”. -The recognition accuracies were 96.02%, 94.38%, 92.88% and 90.15%, respectively. Except for the above-described movements, the accuracies between the remaining movements are between 60% and 90%. The model’s recognition results are better.

Figure 6.

Data set 2 identifies the confusion matrix of the result

An overall analysis of the recognition accuracies of the two datasets reveals that the model has a higher recognition accuracy for dataset 1 than for dataset 2. This may be related to the difficulty of the folk dance movements in both datasets. Because the movements in dataset 2 are produced by two-person interactions, the difficulty of recognition increases compared to the single-person movements in dataset 1, so it leads to a slight decrease in the recognition accuracy. However, overall, the model has better recognition accuracy in both datasets.

In this paper, five existing action recognition methods “ST-ResNet, KVMF, TBN, ActionVLAD and STRA-Net” are selected to compare with the MTSGCN method in this paper. The accuracy comparison results of different methods on dataset 1 and dataset 2 are shown in Table 2. From the table, it can be seen that the accuracy of the proposed method is higher than the five algorithms (97.06% and 89.01%), and it is 5.9%-18.85% and 8.46%-17.28% higher than the other five algorithms on dataset 1 and dataset 2, respectively. This shows that the algorithm in this paper has very high accuracy in recognizing folk dance movements and has a good application prospect.

In order to verify the effectiveness of folk dance movement recognition, this paper conducted experiments on the FD1 dataset. The experimental results of the effectiveness of the overall network structure are shown in Table 3. In the table, the experimental comparison will be made between the baseline model and the model in this paper, and the validation performance of the model is evaluated using the TOP metric. According to the experimental results, the action recognition accuracy top metric of the model proposed in this paper on the validation set is 92.21%, which is an improvement of 8.75% compared to 83.46% of the baseline model. The experimental results show that the model in this paper has obvious advantages in the recognition and extraction of multi-branch time and neighbor frame features, which further improves the recognition performance of folk dance actions.

Figure 7 shows the recognition accuracy of nine single folk dances. It can be seen that the recognition accuracy of this paper’s model for nine kinds of movements is improved compared with the baseline model, in which the two models have the largest difference in the recognition of the “fan turn” movement, followed by the lower back and swinging arm movements, and this paper’s model has a higher recognition accuracy than the baseline model by 28.9%, 24.88%, and 21.57%, respectively. In addition, the recognition accuracies of this paper’s model were 16.04%, 16.88% and 12.61% higher than that of the baseline model for the waist-swinging, lower back and squatting movements, respectively. The recognition errors for both jumping and twisting movements are within 10%. Comprehensive analysis reveals that the average recognition accuracies of the algorithm proposed in this paper and the baseline model for the nine movements of single folk dance are 90.7% and 72.44%, respectively, which shows that the algorithm in this paper performs well.

Figure 7.

The accuracy of the 9 types of individual folk dancing

Figure 8 shows the recognition accuracy results of 9 types of two-person interaction-related actions. By comparing the results in the figure, it can be seen that the recognition accuracy of the model in this paper ranges from 81.24% to 88.51% for the other 6 types of two-person interaction actions, except for “two-person waist turning, two-person arm swinging, and two-person leg kicking”, which are below 80%. In contrast, in the baseline model, the recognition accuracies of the other 8 action categories were below 70%, except for the two-person exchange step. The difference between the two models in recognizing the two-person interaction actions is large, and the recognition accuracy of this paper’s model increases by 13.39%-24.1% compared to the baseline model with respect to the nine actions. Overall, the average recognition accuracy of this paper’s model (82.31%) increased by 16.81% over the baseline model (65.5%). It can be seen that the algorithmic model proposed in this paper still shows excellent performance on such recognition-difficult action samples, which proves that the algorithmic model proposed in this paper is able to effectively extract the characteristics of the subtle interactions between the action nodes, and thus is able to more accurately recognize the categories of difficult action samples.

Figure 8.

The results of the action recognition accuracy of nine pairs of pairs

The rational use of digital equipment has greatly facilitated the preservation and transmission of folk dances. For example, the camera can film the dance performance and play it repeatedly; the scanner can completely copy the picture information and save it to the computer for viewing and printing at any time; the Internet database can store the information of a large number of materials and can be output in reverse without the limitation of time and space.

In today’s society, this paper can objectively and realistically record the performance form, dance vocabulary, dance props and other related things of folk dances with the help of intelligent devices (e.g., cameras, video cameras, etc.), duplicate and scan the precious dance literature, record the voice information of folk dances and store it for a long time. In addition, information related to folk dances can be summarized through the Internet for public inquiry and retrieval. The collected materials can also be organized and stored in different categories to establish a digital archive. The publicity of folk dances can be promoted through official websites, WeChat public numbers and other means, and a website for folk dance cultural resources can be created to promote the publicity of folk dances on the Internet.

Folk dance is a comprehensive and culturally rich art form, each art form and dance movement has a specific connotation, in particular, there are some folk dances that are popular for a long time and have less existing data, it is difficult to fully express its connotation and pass it on with only collected text and video data, which requires the use of more advanced digitization technology. Taking motion capture technology as an example, it was initially applied to the production of animation, in which the recorders projected the movements of the dance performers on the computer screen through the motion capture equipment, and the later staff produced the animation picture according to the projected movements of the actors. In addition, through the digital technology to analyze a variety of representative dance clips of folk dance, and then analyze the laws and characteristics of folk dance movements, you can achieve the capture of the common laws of the folk dance movements, in order to produce a digital model of folk dance, so that the lost art form to model in front of the eyes of the people.