An Intelligent Classification Method for Online Resource Data of College Language Teaching Based on Deep Reinforcement Learning

Deep reinforcement learning consists of deep learning and reinforcement learning [26], and its main principle framework is shown in Fig. 1. Among them, deep learning is mainly used for system perception, while reinforcement learning makes reasonable decisions based on system perception to achieve specific goals. The overall process of deep reinforcement learning is as follows: the intelligent body first observes the environment to get the high-dimensional features of the environment, and uses the deep learning method to perceive to get the specific state representation. Then, it evaluates the expected value of all the actions based on the expected return and selects the corresponding actions by a specific selection strategy (e.g., greedy strategy). Finally, the environment is changed by the actions and enters a new state. Through a continuous cycle of the above steps, the optimal policy is finally learned.

Figure 1.

DRL model framework

The DQN algorithm is a common deep reinforcement learning algorithm [27] that uses deep neural networks to replace the value function in traditional Q-learning algorithms. Due to the ability of deep neural networks to learn high-dimensional abstract features from data, DQN can handle high-dimensional state spaces and obtain a score corresponding to each action. In DQN, after an intelligent body observes the state from the environment and performs an action, the environment provides the intelligent body with a reward and a new state. The intelligent body uses this information to update the estimate of the value function, making the estimate as close as possible to the true value. Meanwhile, since the interactions of the intelligent body are constantly changing in reinforcement learning, updating the estimate of the value function using only the current interaction history may lead to instability of the estimate, which in turn affects the training effect. Therefore, DQN employs the technique of experience playback to cache and randomly replay the previous interaction history. With the empirical replay region, DQN can update the estimate of the value function smoothly and can overfit the i.e., forgetfulness problem.

High-quality learning resources can improve learners’ learning efficiency, help online learning platforms establish trust with learners and teaching staff, and improve the authority of online learning. It is also conducive to the joint maintenance of the online teaching environment by learners and teachers, and promotes the positive cycle of resource learning.

Online education teaching resources are characterized by knowledge-intensive, rich text content and clear logical structure. Therefore, most resource classification models focus on extracting semantic features from resource text content. However, false and misleading resources are characterized by strong concealment and incitement, which are difficult to be identified directly by virtue of semantic features.

Misleading resources disguise themselves by learning to imitate the line structure and writing style of high-quality teaching resources, and induce readers to spread them quickly through inflammatory remarks. Research has shown that inflammatory text content stimulates readers’ emotions such as fear, disgust, and shock, thus increasing their willingness to spread and interact, and emotional incitement is an important reason for the spread of false misleading information. Misleading information usually uses expressions with strong emotional colors and tendencies, while genuine teaching resources tend to be objective and truthful.

The review of online learning resources must ensure that the content is accurate, authoritative and in line with academic standards to prevent the spread of false and misleading information. Learning resources also need to comply with relevant national education laws and regulations to ensure compliance of online education resources.

Suppose that N isomorphic intelligences are in a static unknown environment, and each of them is randomly initialized with its position in the environment at the beginning. At each time step, the intelligences collect partial observations of the environment from the environment at the current position and process the observations locally. Afterwards the intelligences choose a specific action to change their position based on the observations. The common goal of the intelligences is to recognize the key features of the resource information from a limited number of categories and classify the resources in a limited time step. Due to the architecture of centralized training distributed execution, the observations of all the intelligences are learned by the centralized training network. The specific process is shown in Fig. 2.

Figure 2.

Multi-agent image feature extraction process

In order to satisfy the characteristic of decentralized execution, the movement strategy of each intelligent body can only rely entirely on its own local information. Therefore, the intelligences need to learn to extract relevant features from partial observations and rationally plan their traveling paths in the environment to find the most valuable resource information and reliably solve the classification problem. The intelligent body observation module is shown in Figure 3.

Figure 3.

Agent observation module

It is assumed that a partial observation o_i(t) of the environment by the intelligent body i at moment t can be obtained from the current position p_i(t) of the intelligent body: (1)oi(t)=o(I,pi(t))

where I is the whole resource, this function ensures the repeatability of the observations, and the same local observations are obtained when different intelligences are at the same location. The resource processing module performs feature extraction on the observations, internally a parameterized feature extraction network: (2)fi(t)=fθ1(oi(t))

where θ₁ is a trainable neural network parameter and the nonlinear mapping f consists of two single-layer convolutional neural networks and a fully connected layer.

The current position of the intelligence is also useful information, so it is more efficient to learn it together with the locally observed features of the intelligence. For the current position can be handled by a function mapping: (3)λi(t)=λθ2(pi(t))

where λ consists of a fully connected layer and a CeLU activation function and θ₂ is the trainable tensor.

The trajectory in the environment is a sequence of observations, and in order for it to learn long sequences of knowledge, a Long Short-Term Memory (LSTM) unit is used as the perceptual network of the intelligent body. The hidden state of the LSTM module of the smart body i at moment t ≥ 0 is h_i(t) and the state of the unit is c_i(t). The inputs to the perceptual network u_i(t) include information about the current smart body position and local features: (4)ui(t)=[fi(t)Tλi(t)T]T

The evolution of the perceptual network over time is shown in Equation (5): (5)[ hi(t+1) ci(t+1)]=bθ3([ hi(t) ci(t)],ui(t))

The intelligences also need to use the existing learned knowledge to categorize the information and predict the network to use the final joint hidden state h_i(t) and mapping qθ4 : (6)qi(t)=qθ4(hi(t))

where q consists of a fully connected layer plus a ReLU activation function plus a fully connected layer, and then the joint prediction result vector is obtained after averaging the predictions of all the intelligences: (7)q¯=1N∑i=1Nqi

Finally the most probable category is taken out as the result: (8)qc=argmaxj∈{1,2,⋯,M}Softmax(q¯)

Instead of reaching a consensus on categories, the intelligences combine the local perceptions of all intelligences to reach a single prediction.

The global reward of the environment is computed based on the category prediction and the true label of the information r. The reward function is defined as follows: (9)r=−(q¯i−ec)2

q¯i is the predicted probability, and e_c is the uniquely hot coding of the information’s true category label. The loss function of the perceptual network is defined as follows: (10)L(θ)=1N(∑i=1NQ(o,a)r+r)

Q(o,a) is the joint Q value of all optional actions of the smart body under the current local observation.

Deep Q-network is the core of DRL method, using deep neural network can analyze the input features of nodes and calculate the score of each node [28]. Let the DQN be computed as a state-action function Q(S, a), where S={s1,s2,…,sn|si∈{0,1},1≤i≤n}n is the state of the node and a ∈ [1, n] represents the selection of node v_a. For all i ∈ {1, 2, …, n}, s_i = 1 means that node v_i has been selected and s_i = 0 means the opposite. Performing action a represents the selection of node v_a and sets the state s_a of v_a to 1. In addition to this, node features (usually embedded features H) are also required as inputs to the DQN. In detail, the inputs to the DQN include node feature matrix H, node states S and other related features P. Therefore, let H′ be the input matrix of the DQN, which can be formulated as H′ = [H ⊕ S ⊕ P], where ⊕ is the matrix splicing notation. Taking the 3-layer MLP as an example, Q(S, a) the computation is shown below: (11)Q(S,a)=[DQN(W1,W2,H′)]a=[Relu(H′,W1)⋅W2)]a

where W¹ and W² are the weight parameters of the first and second layers of the DQN, respectively. Relu is an activation function that handles the nonlinear relationship between the eigenvalues and the result by setting features with negative eigenvalues to zero. The output of the DQN is a n-dimensional vector, and [DQN(W1,W2,H′)]a denotes the ath score of its result.

Markov decision-making process is an important part of reinforcement learning [29]. Reinforcement learning aims to learn a function [30] that maps state-action pairs to a score, while Markov decision-making executes actions based on this function to obtain a reward and the next state. This process can be represented as a quaternion (St,at,rt,St+1) , where S_t is the state at moment t, a_t is the action chosen at moment t, r_t is the reward for executing the action at state S_t, and S_t+1 denotes the new state obtained after the action is executed. In the node combination optimization problem, S_t denotes the node selection state of the complex network G, r_t is the reward after selecting a node, and S_t+1 is the new state where the vat th value is set to 1.

In order to obtain a higher cumulative reward value, Markov decision making uses a greedy strategy [31] to select the action based on Q(S,a) the highest score. In the node combination optimization problem, the Markov decision will select the node with the highest score, which has the highest expected return and can help the model to obtain better performance in classifying teaching resources. Specifically, the Markov decision process can be expressed as follows: (12)Q(St,at)=maxaQ(St,a) (13)at=argmaxaQ(St,a)

Where, Equation (12) calculates the maximum Q value in state S_t and selects the corresponding node a as the target node. Equation (13) sets the node with the largest Q value as a_t. (14)St+1=St;[St+1]a=1

where S_t denotes the node selection state at moment t and S_t+1 denotes the node selection state at moment t + 1. Equation (14) updates state S_t and obtains S_t+1 by setting the state of the newly selected node to 1. (15)rt=F(St+1)−F(St)

where F(St) denotes the score of S_t and r_t is the reward for performing action a_t in state S_t. Equation (15) obtains the actual reward for node selection by calculating the difference between the scores of states S_t+1 and S_t.

Through DQN as well as Markov decision process, the node combination optimization problem can be equivalently transformed into an optimization problem with DQN to improve the model’s classification on teaching resources. Given a DQN, the solution state S can be obtained through a Markov greedy decision process. Therefore, in order to optimize the solution S, it is necessary to continuously adjust and find the appropriate DQN parameters W¹ and W². Therefore, the node combination optimization problem can be concretely expressed as: (16)maxF(S)=F(W1,W2) s.t.Q(St,a)=DQN(W1,W2,H,S)

During the training process, RL employs a wandering strategy (ε-greedy strategy) to collect 4 samples and store them in the empirical playback area B. At the same time, the DQN is updated by performing a small batch of stochastic gradient descent on the loss function L. From this point of view, the training of the DQN is a minimization process of the loss function. In order to achieve a good balance between exploration and utilization, the MSE loss function and dual DQN methods are used. Dual DQN uses two DQNs: a prediction network Q and a target network Q^ . The prediction network Q is updated once per round, while the target network Q^ is updated once per c round. The loss function L(W1,W2) is evaluated: (17)L(W1,W2)=E(St,atrtSt+1∈B)[rt+γmaxa′Q^(St+1,a′;W^)−Q(St,at;W)]2

In the online teaching resources of university languages, text and image are the highest and most important types of resources, and this subsection provides an in-depth investigation of the image classification effect of DRML model.

4.2.2

Feature visualization and analysis

In order to study the feature extraction capability of DRML more intuitively, this subsection samples 20 samples per category on the QHGIM dataset, and uses DRML to show their distributions in the feature space, and the feature distributions of the test samples are visualized as shown in Fig. 4, where (a) and (b) are the results of the feature visualization for the benchmark models ProtoNet and DRML, respectively. The results show that DRML is able to generate more accurate decision boundaries so that different categories are distinguished more obviously.

Figure 4.

The visualized characteristics of the test sample

In order to investigate the overall generalizability of different models on the QHGIM dataset, this section uses different methods to visualize the feature distributions of all the samples within the QHGIM dataset. The visualization of the feature distributions of all the samples within the QHGIM dataset is shown in Fig. 5, and (a) to (d) are the original feature distributions, the ProtoNet, the SUN, and the DRML feature distributions, respectively.

Figure 5.

The visualized characteristics of all test samples in QHGIM dataset

As shown in Fig. 5(a), the original QHGIM dataset has high overlap of different categories, which makes classification challenging. As shown in Fig. 5(b), the conventional prototype network has limited classification performance on the QHGIM dataset, which is only able to distinguish the classes of a small number of samples, and most of the samples still have significant inter-class overlap. As shown in Fig. 5(c), SUN achieves effective classification for most of the samples, but the class spacing between different classes is too close, and there is still class confusion for some samples. As shown in Fig. 5(d), in contrast DRML exhibits the best classification performance, effectively reducing the inter-class overlap and improving the overall results. This further demonstrates the effectiveness of DRML on image classification tasks.

The DRML model of this paper was used in the teaching of language at school in School S. Questionnaires were distributed to the subjects in the school to investigate whether the classification of resources for teaching language at school based on the DRML model has gained the satisfaction of the users. Evaluation questionnaires were distributed to 442 teachers and students, and the data from the recovered questionnaires were counted, and the results are shown in Table 5, where -2, -1, 0, 1, and 2 stand for Strongly Disagree, Disagree, Uncertain, Agree, and Strongly Agree, respectively, and Fi stands for the score rate.

As can be seen from Table 5, the percentage of subjects who expressed agreement/strongly agree in the 10 indicators is more than 75%, the percentage of those who expressed disagreement is not more than 2%, and the number of those who expressed strong disagreement is not more than 5%. The mean scores of the 10 indicators are in the range of 0.88-1.16, which can be seen that this paper’s DRML model of categorization of online language teaching resources in colleges and universities achieves a more satisfactory evaluation of users’ Results.