Research on the Integration of Student Behavior Analysis and Curriculum Education Strategies in Colleges and Universities under Deep Learning Framework

Deep learning is a kind of deep constructive learning style, which requires learners to be able to critically learn new knowledge on the basis of understanding, relate it to the existing cognitive structure, effectively internalize, transfer and apply the knowledge, relate the knowledge to new real-life situations and solve practical problems, so as to obtain the development of higher-order abilities. Deep learning focuses on the accumulation of knowledge, as well as the emotional experience, value recognition, and ability development of learners during the process of knowledge exploration.

Teaching methods such as problem-based learning and task-driven learning are considered to be effective ways to promote deep learning, and these teaching methods emphasize the initiative and participation of learners, encouraging them to keep exploring and reflecting in the process of solving practical problems. At the same time, deep learning also puts forward high requirements for the context of knowledge learning, suggesting that teaching should be closely linked with practice and real situations, so that learners can experience the power and value of knowledge in real or simulated situations. At present, deep learning has become an important trend in the development of higher education, and is also the goal pursued by blended teaching in the context of informationization. In course construction and teaching, it is necessary to effectively connect teaching time and space, strengthen teaching and learning interaction, purposefully design and carry out learning tasks based on problems, tasks and real situations, provide students with rich learning resources, stimulate their interest and motivation to learn, and guide them to actively explore and actively practice, so as to achieve deep learning.

In this paper, we design a deep learning-based behavior analysis system for students in colleges and universities. For the subsequent development of the system, the system architecture should follow the principles of modularization, high cohesion, and low coupling when designing, in order to meet the needs of scalability, stability, and easy maintenance of the system. The overall architecture of the system can be divided into four layers from bottom to top: data storage layer, data processing layer, business logic layer and user interface layer. 1)

Data Storage Layer: Select the database (relational, non-relational and distributed storage system) according to the structure and access mode of the data, and also need to ensure the security, integrity and efficient access to the data.

In order to better meet the needs of student behavior recognition, the extracted video images are preprocessed as follows: 1)

Take the center point as the benchmark, the image is uniformly scaled and cropped to 432 × 368 size.

For the detection of student targets in video frame images, this paper uses the Tiny_YOLOv3 target detection algorithm for student target detection. Due to the shallow network layer of Tiny_YOLOv3, the detection speed is relatively fast, but the accuracy is not enough, in order to improve the accuracy of detection, this paper incorporates the attention mechanism SENet to improve the network structure of the original model [30]. SENet is a typical method for enhancing or suppressing channels for different tasks, analyzing the importance of each feature channel to improve accuracy, as a typical channel attention mechanism. In this paper, we incorporate SENet into the Tiny_YOLOv3 network structure as a way to improve detection accuracy.

Human posture estimation based on OpenPose algorithm is mainly divided into two kinds of 2D posture estimation and 3D posture estimation, among which 2D posture estimation is mostly studied, and 2D posture estimation is further divided into single-person posture estimation and multi-person posture estimation. The single-person pose estimation methods mainly contain coordinate regression-based methods, heat map detection-based methods, and hybrid models based on coordinate regression and heat map detection. Multi-person pose estimation is mainly categorized into two types: 1)

Top-down methods, i.e., detect each person first and then perform pose estimation for each person, typical models are RMPE, MaskR-CNN, etc.

Figure 1.

OpenPose model structure

The input to the first stage of the OpenPose network is the feature map F, which is processed by a series of CNNs to obtain the joint 2D confidence map S¹ and partial affinity L¹ [31]. And from the second stage onwards, the input to the network contains a total of three parts, F, S^t − 1 and L^t − 1, as shown in equation (3): (3)St=ρt( F,St−1,Lt−1 ) t≥2 Lt=φt( F,St−1,Lt−1 ) t≥2$$\begin{array}{l} {{S^t} = {\rho ^t}(\:F,{S^{t - 1}},{L^{t - 1}}\:)}&{t \ge 2} \\ {{L^t} = {\varphi ^t}(\:F,{S^{t - 1}},{L^{t - 1}}\:)}&{t \ge 2} \end{array}$$

Through repeated iterations of the multi-stage convolutional neural network until the network converges. Eventually, at the time of prediction, whether this joint belongs to the same person or not is measured by the affinity (PAF) between the joint pairs d_j1 and d_j2, as shown in equation (4): (4)E=∫a=0a=1Lc( p( u ) )×dj2 −dj1|dj2 −dj1|2du$$E = \int_{a = 0}^{a = 1} {{L_c}} (\:p(\:u\:)\:) \times \frac{{{d_{j2}}\: - {d_{j1}}}}{{|{d_{j2}}\: - {d_{j1}}{|_2}}}du$$

Where: p(u) denotes the pixel point between consecutive image cable points d_j1 and d_j2 as shown in equation (5): (5)p( u )=( 1 −u ) dj1 +udj2$$p(\:u\:) = (\:1\: - u\:)\:{d_{j1}}\: + u{d_{j2}}$$ (2)

OpenPose network improvement

The feature extraction network of the original OpenPose model adopts VGG19 convolutional neural network, and the research shows that: when the convolutional neural network reaches a certain depth, not only can’t improve the performance, but also cause the network to converge slower, and the detection performance deteriorates. In this paper, we propose MobileNet, a lightweight network for mobile with depth-separable convolution as the core, which has fewer parameters and lower computing cost than standard convolutional neural networks. Depth separable convolution is composed of depth convolution (DW) and pointwise convolution (PW). The structure of standard convolution and depth separable convolution is shown in Fig. 2, (a), (b), and (c) denote standard convolution, depth convolution, and point convolution, respectively.

Assuming that the input feature map size is D_i × D_i × M, the convolution kernel size is D_K × D_K × M, and the output feature map size is D_θ × D_θ × N, the number of parameters to be convolved using the standard convolution of Fig. 2(a) is: (6)Wstand=( DK×DK×M) ×N$${W_{s\tan d}} = (\begin{array}{c} {{D_K} \times {D_K} \times M} \end{array})\: \times N$$

Figure 2.

Standard convolution and depth-separable convolution structures

Whereas the convolution kernel size for depth convolution in Fig. 2(b) is (D_K, D_K, 1) with M kernels, the convolution kernel size for point convolution in Fig. 2(c) is (1, 1, M) with N kernels, the number of parameters for depth and point convolution are: (7){ Wdepthwise=( DK×DK×1 )×M Wpointwise=( 1×1×M) ×N$$\left\{ {\begin{array}{c} {{W_{depthwise}} = (\:{D_K} \times {D_K} \times 1\:) \times M} \\ {{W_{po\operatorname{int} wise}} = (\:1 \times 1 \times M)\: \times N} \end{array}} \right.$$

Therefore, the number of depth-separable convolutional parameters is: (8)WD = Wdepthwise + Wpointwise =( DK ×DK ×1 ) ×M+( 1 ×1 ×M) ×N$${W_D}\: = \:{W_{depthwise}}\: + \:{W_{po\operatorname{int} wise}}\: = (\:{D_K}\: \times {D_K}\: \times 1\:)\: \times M + (\:1\: \times 1\: \times M)\: \times N$$

Therefore, the parametric ratio of the number of depth-separable convolutional parameters to the number of standard convolutional parameters is: (9)η=WDWstand=1N+1DK2$$\eta = \frac{{{W_D}}}{{{W_{s\tan d}}}} = \frac{1}{N} + \frac{1}{{D_K^2}}$$

The convolutional kernel sizes of VGG19 are all of size 3 × 3, so the number of parameters will be reduced by about 1/9 by replacing VGG19 with a MobileNet network with a deeply separable convolutional core.

Through the above analysis, this paper starts from improving the feature extraction network, and compares the impact of different feature extraction networks on the final results by comparing four feature extraction networks, namely VGG19, MobileNet, MobileNetV3-small, and MobileNetV3-large. The original model uses a large number of 7 × 7 convolutional kernels, which causes an increase in computation, this paper adopts a residual structure of three 3 × 3 convolutions to replace one 7 × 7 convolution, and continues to improve each 3 × 3 convolution by using a form of depth-separable convolution.

Target classification is mainly used to classify the extracted human joint point data to determine the current learning behavior of students. 1)

Skeletal key point direct coordinate method

In this paper, we classify and identify the two learning behaviors of raising hands and stretching by the direct coordinate method of skeletal key points, and the identification situation is shown in Fig. 3, with (a) and (b) indicating the skeletal key points of raising the left hand and stretching posture, respectively. The method used is to determine which learning behavior is present by calculating and comparing the geometric relationship between the detected coordinate positions of the skeletal key points in different parts of the body.

As shown in Fig. 3(a), it can be seen that when the key point of the human left hand hand is higher than the key point of the nose, it can be determined that the action at this time is lifting the left hand, and by the same token, it can be determined that lifting the right hand. The specific calculation method is as follows: take the upper left corner of the picture as the coordinate origin, when there is a left hand hand key point in the picture, the longitudinal coordinate of the hand key point H is less than the longitudinal coordinate of the point N and the longitudinal coordinate of the right hand hand key point is greater than N, it is determined that the action at this time is to raise the left hand.

As shown in Fig. 3(b), when the key point of the right and left hand of the human body is higher than the key point of the nose, it can be determined that the action at this time is stretching. The specific calculation method is as follows: take the upper left corner of the picture as the coordinate origin, when there are left and right hand hand keypoints in the picture, the vertical coordinates of the hand keypoints H1 and H2 are less than N1, it is determined that the action at this time is stretching.

Figure 3.

Skeletal key points of left hand raised and extended posture

The skeletal key point direct coordinate method for determining learning behaviors with obvious geometric relationships not only has a high recognition accuracy, but also has a simple recognition method and a fast calculation speed. 2)

Skeletal key point relationship feature extraction method

The process of extracting students’ actions through the skeletal key point relationship feature extraction method is shown in Figure 4. (a), (b) and (c) represent the skeletal keypoint connection diagrams of the three actions of lying down, playing cell phone and writing respectively. It can be seen that the differences in the key point relationship features of these three actions are mainly reflected in the arms and head, so we will extract the skeletal key point relationship features for the head and arms respectively, so as to categorize and recognize these three actions.

In the classroom application scenario, it is necessary to identify the learning behaviors of each student in the classroom, and each student in the classroom has a different body size and height, and the distance of the students from the camera is also different, so the process of key point feature extraction for the three kinds of actions should not only ensure that the extracted feature vectors can differentiate between the three kinds of learning behaviors, so that the support vector machine can be used to carry out effective classification, but also to satisfy that the feature vectors can be applied to each student to ensure the invariance of learning behavior detection.

Figure 4.

Key bone points of lying, playing mobile phone and writing posture

Through the above analysis, this paper extracts the feature vectors by representing the three key points on the arm as two vectors and transforming the positional relationship between the key points into the relationship between the vectors. The specific calculation method is shown in Eq. (10), in which the three key points S1, E1, and H1 of the left arm in Fig. 4(a) are represented by vectors E1H1→$$\overrightarrow {E1H1}$$ and E1H1→$$\overrightarrow {E1H1}$$, respectively, and the projection length of E1H1→$$\overrightarrow {E1H1}$$ in the E1S1→$$\overrightarrow {E1S1}$$-direction can be obtained by dividing the two vectors by the modulus of E1H1→$$\overrightarrow {E1H1}$$ after dot-multiplying them, and then dividing this projection length by the modulus of E1S1→$$\overrightarrow {E1S1}$$ yields the ratio of the projection length of E1H1→$$\overrightarrow {E1H1}$$ in the E1S1→$$\overrightarrow {E1S1}$$-direction to the length of E1S1, W1. W1 is the feature vector of the left arm of the human body, which achieves the purpose of learning behavior detection invariance by avoiding the influence of the feature vector due to the student’s body size and distance in the real application scenarios by means of the ratio: (10)W1=E1H1→⋅E1S1→|E1S1→|2$$W1 = \frac{{\overrightarrow {E1H1} \cdot \overrightarrow {E1S1} }}{{|\overrightarrow {E1S1} {|^2}}}$$

The calculation of the eigenvectors of the head is shown in Eq. (11), where the key point between the neck and the head is represented by vector C1N1→$$\overrightarrow {C1N1}$$ with the vector direction C1 pointing to N1, and vector S1S2→$$\overrightarrow {S1S2}$$ represents the vector between the key point of the left shoulder and the right shoulder. Dotwise multiplying the two vectors and dividing by the mode of S1S2→$$\overrightarrow {S1S2}$$ yields the length of the projection of C1N1 in the direction of S1S2, and then dividing this length of projection by the mode of S1S2→$$\overrightarrow {S1S2}$$ yields the ratio of the length of the projection of S1S2→$$\overrightarrow {S1S2}$$ in the direction of S1S2→$$\overrightarrow {S1S2}$$ to the length of S1S2, Y1. Since vectors are directional, the value of Y1 is negative when N1 is below C1. When NI is above C1, the value of Y1 is positive. Thus, a distinction can be made between these two action features of head down and head up: (11)Y1=C1N1→⋅S1S2→|S1S2→|2$$Y1 = \frac{{\overrightarrow {C1N1} \cdot \overrightarrow {S1S2} }}{{|\overrightarrow {S1S2} {|^2}}}$$ 3)

Learning behavior classification based on support vector machine

In this paper, support vector machine is used as the classification algorithm of learning behavior, and the feature vectors extracted from students’ learning behavior based on human skeletal key points are taken as inputs, and the learning behavior is classified after the training of support vector machine, and the learning behavior classification model is finally obtained [32]. The specific process is to take the double-arm feature vector W and the head and neck feature vector Y in the feature vectors of skeletal key points extracted in the previous section as the input vectors of the support vector machine, and at the same time, we label the three kinds of learning behaviors that need to be classified, and the learning behavior lying on the floor is labeled as 1, playing with the cell phone is labeled as 2, and writing is labeled as 3.

In order to avoid overfitting for classification, in this paper, the image data of the three actions are acquired according to the criteria of multiple angles, multiple gestures and multiple numbers. The image data of the three actions of the authors of this paper were automatically acquired by a camera, and 10,000 images were acquired for each action, resulting in a total of 30,000 image data. Then the obtained picture data are categorized into three learning behaviors, and the picture data of each category are randomly sorted in the process of classification training, and then 70% of the picture data of each category are divided into the training set, and the remaining 30% are used as the test set. Finally, these picture data are trained with feature vectors using support vector machines, and finally the classification model is generated.

Since there is no open dataset of students’ classroom behavior in China, this study used a SONY FDR-AX30 digital 4K camcorder to collect image data of students’ classroom behavior using a single shot and a single image intercepted from a classroom-recorded video for the construction of a dataset of students’ classroom behavior with 600 students from a university in A city.

In this study, the collected images were cropped and saved according to the upper body region of the human body constructed based on the key points of the human skeleton, and then scaled uniformly to an image with a size of 112 × 112 (with blanks complementing the zeros) using the long side as a base. By tagging and categorizing the 4600 collected images with students’ classroom behaviors, this study summarizes seven typical classroom behavior images that appear more frequently in students, including: raising hands 514, listening 825, looking around 806, reading 503, writing 522, standing up 847, and sleeping 583, which constitutes the Student Classroom Behavior Dataset (SCBID).

1)

The dataset of students’ classroom behaviors is disrupted, and four-fifths of each of these students’ classroom behaviors are randomly selected as training samples, and the remaining one-fifth is used as testing samples.

Under the same experimental conditions and after 10 randomized experiments, this study compares the accuracy of CNN-10 and the student classroom behavior recognition method designed in this paper in recognizing multiple classroom behaviors on the SCBID dataset, and the accuracy comparison results are shown in Fig. 5. By calculation, this study concludes that the average recognition accuracy of CNN-10 is 92.15%, while the average recognition accuracy of the student classroom behavior recognition method designed in this paper is 98.04%, which is up by 5.89% compared to CNN-10.

Figure 5.

Accuracy comparison results

The confusion matrix of CNN-10 on the students’ classroom behavior dataset is shown in Fig. 6.Using CNN-10, there is a large gap in the recognition accuracy of different classroom behaviors, e.g., the recognition accuracy of raising hands and reading is 83.22% and 83.26%, respectively. Among them, 13.96% of hand-raising is misrecognized as listening, while 8.77% of reading is misrecognized as listening, and 5.43% is misrecognized as sleeping. The reasons for this are mainly: (1) The recognition of hand-raising behavior mainly relies on local information such as hands, which is easily interfered by factors such as students’ physique, dress code, and classroom background. Students’ reading behavior has a certain range of movement, and if it is too small, it is easy to misinterpret it as listening. Some of the reading and sleeping behaviors are too similar and the distinction is subtle, which can also lead to misjudgments. (2) Due to the limitations of the student classroom behavior dataset, it is difficult to classify all students’ classroom behaviors by using only convolutional neural networks to obtain behavioral features directly from the original image data.

Figure 6.

CNN-10 is the confusion matrix of the class behavior data set

The confusion matrix of a method for recognizing student classroom behavior based on human skeleton information and deep learning on student classroom behavior dataset is shown in Fig. 7. The gap of recognition accuracy of each classroom behavior under the method of this paper is narrowed. The recognition accuracy of the writing behavior is even higher than 100%, and the reading behavior with the lowest recognition accuracy also reaches 94.28%, in which 3.75% of the reading behaviors are misidentified as standing up.

Figure 7.

The confusion matrix of the class behavior data set is learned in this article

In order to exclude any unnecessary influence on the experimental results of this study due to the differences in the level of deep learning that the students in the experimental group and the control group have previously had, a test questionnaire was distributed to the two groups of students before the start of the experiment, and the resulting data were analyzed through SPSS 20.0. 1)

Descriptive statistical analysis

First of all, this study analyzed the pre-test data with descriptive statistics, and the results of the descriptive statistics of the pre-test of the questionnaire are shown in Table 1. Before the start of the formal experiment, except for the learning input dimension where the data of the two groups are basically the same, in the three dimensions of learning motivation, learning strategy and learning outcome, the mean values of students’ ratings of the experimental group are higher than those of the control group by 0.79, 0.61, 0.36 respectively, and the situation of the two groups are not much different.

Figure 8.

Independent sample t test results

1)

Descriptive statistical analysis

After the end of the formal experiment, by filling out the same learning questionnaire again by the students of the two groups, this study obtained the post-test data, and its descriptive statistical results are shown in Table 2. The experimental group’s scores on the four dimensions of deep learning motivation, deep learning engagement, deep learning strategy, and deep learning outcome were higher than those of the control group by 4.05, 3.68, 3.48, and 3.11, respectively, and the scores increased significantly compared with the pre-test. Therefore, this study concludes that the curriculum education strategy incorporating student behavior analysis has a positive effect on students’ learning level. The next step will be to analyze the difference through independent samples t-test to determine whether this effect reaches the level of significance.

Figure 9.

Independent sample t test results