Research on the Optimization of Intelligent English Vocabulary Teaching Paths Based on Reinforcement Learning Models

Personalized learning is the core goal of computer-assisted instruction, and personalized learning systems are constructed through AI technology to tailor learning paths and resources according to learners’ interests, abilities, and progress. This requires collecting and analyzing learners’ learning data, using machine learning algorithms to identify learning patterns and needs, and then designing personalized learning paths and resource recommendation strategies to improve learning efficiency, satisfaction, and stimulate motivation and creativity.

Intelligent teaching aids are key applications of AI technology in computer-aided teaching, covering intelligent Q&A systems, virtual teaching assistants, online labs, etc., which can answer questions, provide real-time feedback and simulate real experimental environments. In order to optimize these tools, it is necessary to continuously improve their algorithms to increase accuracy and response speed, and focus on interface design to enhance the ease of use and friendliness of the user experience. At the same time, a user feedback mechanism should be established to collect and process learners’ opinions in order to continuously improve the tools.

Reinforcement learning technology model [18] in the intelligent body’s task is to continuously explore in the constructed environment, the specific process is to observe the state in the environment, and then take the corresponding action, thereby obtaining the environment to feedback to the intelligent body’s current exploration of the action of the rewards, and finally through the maximization of rewards to learn and update the policy, and constantly interact with the environment, so as to find the optimal solution to reach the end state.

Reinforcement learning is the core element of reinforcement learning, which simulates the environment explored by the intelligent body as the environment in which the current learning task is performed. The environment interacts with the intelligent body and gives the intelligent body some rewards for its exploration in the current state, and changes the current state of the intelligent body, and so on until the intelligent body reaches the final state in the environment.

The state of the smart body in the environment can be understood as the attributes of the smart body itself, which can be location information or other variables that describe the environment. The state can be a discrete variable or a continuous variable and is continuously updated as the intelligent body interacts with the environment.

Actions are behaviors generated by the intelligent body during the interactive exploration of the current environment.

Reward is a certain reward that the environment gives back to the intelligent body according to the action taken by the intelligent body in the current state during the interactive exploration of the current environment. Usually, when the intelligent body chooses the next state, it will consider the reward of the current action in the environment, and the reward of the interaction in each time window is also related to the subsequent training process.

Reinforcement learning problems are often described as Markov Decision Over (MDP). That is to say, in reinforcement learning, the interaction process between the intelligent agent and the environment has to satisfy the Markov decision process, i.e., the future state of the intelligent agent is only related to the current state. The Markov decision process has dependencies between states and a temporal order, which is a sequential decision process. The Markov decision process contains five elements, which can be represented by a quintuple, i.e. M = (S, A, P, R, γ). where S is a finite state space, A is a finite action space, also called the set of actions, P={Psa(⋅)|s∈S,a∈A} is the probability of transferring to the next state, R is the reward function, and γ is the discount factor.

where the transfer probability matrix P is specifically denoted as P(s,a,s′)=P(s,a)r(st+1=s′|st=s,at=a) and the immediate reward r is denoted as r=R(s,a,s′)=E{rt+1|st=s,at=a,st+1=s′} .

The specific Markov Decision Process (MDP) is shown in Figure 1. s₀ is the initial state of the intelligent body, and after taking an action a₀ in this state, it transfers to state s₁ according to the probability p₀ and receives an immediate reward r₁ from the environment, and then repeats the above process continuously.

Figure 1.

Markov decision process

In Markov Decision Process (MDP), which is based on the theory of stochastic processes, it is stipulated that the current state of the intelligent body Agent is only related to the state of the previous moment and the action taken. Meanwhile, the reinforcement learning approach based on this regards the recommendation process as a Markov decision process, specifically, the intelligent body selects an exploratory behavior (action) according to the state transfer probability matrix Pst,st+1π through the policy π and performs the action to interact with the environment, and after the interaction, the current reinforcement learning environment feeds back its immediate reward and the current intelligent body will be transformed to the next established state. Thus, a reinforcement learning-based English vocabulary exercise recommendation assisted teaching system can characterize the sequence of user behaviors and can capture the dynamic preferences of users. The model also dramatically improves the diversity of recommendation results by allowing the intelligences to fully explore the state space, action space according to their exploration mechanism. This type of model generally improves the accuracy of English vocabulary exercise recommendations by maximizing the cumulative benefits of the English vocabulary exercise recommendation aid teaching system as the optimization goal, focusing on the long-term feedback from users, and thus executing the updating of the recommendation strategy, focusing on the improvement of the long-term satisfaction of users, as well as improving the accuracy of the English vocabulary exercise recommendations.

The Q-Learning algorithm [19] is an offline policy algorithm and a value-based reinforcement learning algorithm. Where Q refers to the Q value, the Q function in the algorithm, the

i.e., Q(state,action) , is denoted as the Q-value, also called Q − value, obtained by executing action A in state S. The Q-Learning algorithm aims to maximize Reward by selecting the optimal action A, i.e., the best action, in state S to obtain the maximum gain, with the ultimate goal of maximizing the Q-value.

The Q-Learning algorithm always maintains a Q table to store the respective Q values corresponding to different actions taken in different states. The Q values are modified and the Q table is updated based on the immediate rewards r obtained and the algorithmic rules during the interaction of the intelligence with the environment. As the number of iterations increases, the intelligence improves its understanding of the environment and the Q function is gradually fitted until it converges or reaches the set number of iterations.

The Value update formula for the Q-Learning algorithm is shown below. (1)Q(s,a)←Q(s,a)+α[R(s,a)+γmaxQ′(s′,a′)−Q(s,a)] s←s′

where α is the learning rate, which represents the update magnitude, R(s, a) is the immediate reward, maxQ′(s′,a′) is the Q value obtained after choosing the action that maximizes the next state, and γmaxQ′(s′,a′) represents the future long-term reward. The R(s,a)+γmaxQ′(s′,a′) whole is represented as the true Q value of taking an action a_t in the current state s_t and consists of the immediate reward R and the long-term reward. The Q(s, a) in parentheses is the estimated Q value, and the difference between the actual Q value and the estimated Q value is denoted as ΔQ(s, a), and as this difference converges to 0, Q(s, a) it stops changing and the whole converges to a convergent state.

Item response theory (IRT) has been widely used in educational and psychological settings, particularly in educational assessment, to assess learner competence. Tests are usually constructed and evaluated using various IRT models by analyzing the characteristics of the items (item discrimination and item bias).

The probability function of a two-parameter normal shoulder-type model for a learner correctly answering a 0-1 scoring question i is given below. (2)Pi(θ)=∫−∞ai(θ−bi)e−z222πdz

Where P_i(θ) - the probability that the learner will answer the question correctly.

θ - The corresponding learning ability of the learner.

a_i - The differentiation parameter of question i.

b_i - Difficulty parameter of question i.

Similarly, based on the two-parameter normal shoulder model, the three-parameter normal shoulder model can be derived by adding the guessing coefficients of the questions to the model (3)Pi(θ)=ci+(1−ci)∫−∞ai(θ−bi)e−z222πdz

Where c_i - the guessing parameter of topic i, which indicates the probability of being able to mask the right answer even if the topic is answered randomly.

Although the normal shoulder model theoretically lays down the basic form of the IRT model, the calculation process utilizes the integral form of calculation, which leads to the normal shoulder model is not very suitable in both parameter estimation and practical application. Currently the commonly used IRTlogistic model is the three-parameter logistic model: (4)Pi(θ)=ci+1−ci1+e[−Dai(θ−bi)]

Each set of English vocabulary test questions in the Exercise Resource Library contains not only 0-1 scoring questions such as single-choice questions, fill-in-the-blank questions and judgment questions, but also multi-level scoring questions. Compared with the 0-1 scoring questions, the multi-level scoring questions can evaluate the learners’ learning ability more objectively.

Like the 0-1 scoring questions, the parameters of the multi-level scoring questions also include differentiation parameter, difficulty parameter, and there is no guessing parameter unlike the 0-1 scoring question model, because it is basically impossible for learners to guess to answer a multi-level scoring question correctly. Unlike the 0-1 scoring IRT model, the multilevel scoring IRT model subdivided the difficulty parameter of the question, i.e., for each additional share of the score, the difficulty of the question is a little more difficult, and the reason why it did not subdivided the differentiation parameter as well.

Therefore, a generalization of equation (4) can be obtained as a logistic model formula for the multilevel scoring model: (5)Pk(θ)=11+exp[−1.7ai(θ−bik)]

Where P_k(θ) - the probability that the learner scores k points on this question.

θ - The corresponding learning ability of the learner.

a_i - The differentiation parameter of question i.

b_ik - Difficulty parameter for topic i when it scores k.

In Eq. The value of k is taken from 1 because there is no difficulty for the learner to get 0 points on a multilevel scoring question.

This section discusses how to assess the difficulty of English vocabulary exercises. Further, after evaluating the difficulty of exercises, we can more easily and accurately match students’ learning ability with the difficulty of English vocabulary exercises to achieve better recommendation results.

To calculate the difficulty of English vocabulary exercises, we first need to analyze the questions in the exercise bank. In practice, we find that different questions can differentiate students with different ability levels, and the degree of differentiation of the questions is the amount of information that the questions can provide for the respondents. The amount of information, but for students who have learned the basics very firmly, this question is a bonus question, so the question can not play a role in testing such students, that is, the value of the amount of information provided is very small, so it is not possible to answer this question based on the answer to the question to say that which student’s ability to learn is stronger.

Item response theory based on the idea of Fisher’s information defines the relationship between the amount of information contributed by the question in evaluating the learning ability of students as: (6)Ii(θ)=[pi′(θ)]2pi(θ)(1−pi(θ))

Where p_i(θ) - the probability that a learner with an ability value of θ answers question i correctly.

p_i′(θ) - the first order derivative function of p_i(θ) for θ.

The information content of the questions derived for different proficiency level values is different. Of course, we can similarly define the notion of informativeness for evaluating an individual student at the level of the entire test, which is the test information function, which is, in fact, an accumulation of the information functions of the questions included in the test, i.e: (7)I(θ)=∑i=1mIi(θ)

The information function of the items allows for the classification of the questions into several categories:

Standard item information profiles, where the peak corresponds to a learning ability value of slightly less than 0 and the peak is high, indicate good differentiation and the item corresponds to a low level of difficulty, which can serve as a good test for a significant portion of the student population.

The ideal question information curve, with a peak corresponding to a learning ability value of 0, a higher peak and steeper sides of the peak, indicates that the question is of moderate difficulty, has good differentiation, and can be a good test for students whose learning ability is around 0, and the guessing result of the question has a small error.

The information curve of the biased type of questions, the corresponding learning ability value of the wave peak is low, the wave peak is short and the two sides of the wave peak are gentle, which indicates that the differentiation of this type of questions is small, and the difficulty is low, and because the span of the wave peak is large, it can’t provide effective information for the assessment of learning ability of a small interval, so this type of questions should be given less weight in the assessment of the difficulty of the exercises, or even directly not to let this type of questions participate in the assessment of the difficulty of the exercises. Therefore, this type of questions should be given less weight in the assessment of the difficulty of the exercises or even be excluded from the assessment of the difficulty of the exercises.

Wrong type of question information curve, no matter how the waveform is, the whole curve is located below the horizontal coordinate, indicating that it can not provide valid information for any ability assessment, this type of question design is a failure, and this type of question should be excluded from the assessment of the difficulty of the exercise.

The effective topic information curve weighted fit can be obtained after fitting a multi-peak graph, select the first peak as the lowest difficulty of the exercise, the last peak as the highest difficulty of the exercise, the average difficulty of the exercise is calculated by normalizing the information content of all valid topics as a weight and then multiplied by the difficulty of each topic to do a linear addition to obtain the average difficulty of the exercise. The formula for calculating the average difficulty of the exercises can be obtained: (8)D=α1b1+α2b2+α3b3+⋯⋅+αnbn

Where α_i - the weight of topic i.

b_i - Difficulty of topic i.

This study utilizes reinforcement learning algorithms and further analyzes the environmental features based on the characteristics of learners and questions to select the appropriate actions and achieve reward rewards. This experiment adopts offline experiment to obtain effective feature information from learners’ history of answering records of exercises to complete the research on the model of personalized exercise recommendation assisted teaching system based on reinforcement learning. Firstly, the answer records of learners’ questions are collected, the ability level interval of learners is classified according to the answer results, and the average pass rate and average score data of exercises are provided for the quantification of the difficulty of the questions. Then the reinforcement learning model is trained to recommend the exercises to the learners. The structure of the experimental model is shown in Figure 2.

Figure 2.

The recommended model

In the exercise recommendation model based on reinforcement learning Q-Learning algorithm, the simulation of the learner’s “nearest development zone” is a two-dimensional model constructed by the learner’s ability level and the difficulty of the topic, and the experimental code in this study is written in Python language, and the simulation platform is built using Tkinter. A grid of 10 × 10 was constructed as the environment model for the experiment. The horizontal and vertical axes in this grid environment represent the learner’s ability level and the difficulty of the questions, respectively, and the values of both of them are standardized and normalized by data preprocessing and mapped in an interval with a value range of [0,1] . Each cell in the environment model is a state, and the total number of states is 100. An intelligent body selects an action in the current state s, arrives at the next state s′, computes the value function Q(s,a) and updates the Q table according to the output Q value, selects the next action, and receives the corresponding instantaneous reward payoff according to the reward strategy r. Each step of the state updating represents the change of the intelligent body doing the question of different difficulty at different ability levels.

Recommendation based on reinforcement learning Q-Learning algorithm is an exploratory process, and the reinforcement learning intelligent body selects the optimal next action through the value function Q(s,a) after learning experience through continuous exploration. In this study, each frame moved by the intelligent body in the environment model indicates the occurrence of an action, and according to the changing characteristics of the learner’s ability level and the difficulty of the topic, there are three actions in the action set A of this experiment. In order for the intelligent body to learn enough knowledge and experience, a lot of exploration and trial and error should be carried out to help it make the correct judgment, but too much exploration will lead to a decrease in the efficiency of model training, while insufficient exploration will lead to a lack of experience and the inability to make the optimal choice. So in order to choose the optimal path to accumulate the maximized reward, we choose ε − greedy greedy algorithm [20] as the action selection strategy for the experiment to weigh the proportion of exploitation or exploration.

The purpose of the intelligent body in reinforcement learning is to find the optimal path through learning to obtain the maximum reward, the reward is the feedback of the environment on the selection and execution of the action during the learning process, the execution of a poor action to obtain a small reward or even get punishment, then the probability of the next time that the action is selected to be executed will become smaller. While executing an optimal action to get the maximum reward, then the probability that the next optimal action will be selected for execution will become larger, a reasonable reward strategy [21] can improve the efficiency of model training, so it is very important for the design of the immediate reward strategy.

According to the concept of “learning area” in psychology, the questions we do to learners each time should not be too difficult, so that learners will be afraid of difficulties, nor can they be too simple, causing users to lose interest in learning. According to the definition of “learning area”, ideally, the algorithm should be able to give a moderately difficult exercise from the learning area each time, which is reflected in the fact that the correct answer rate finally converges to a fixed value. Let the number of questions that learners answer each time is N and the number of correct answers each time is n, and the construction bias metric is as follows: (9)err=(nN−δ)2

where δ ∈ [0, 1], x ≤ n. err denotes the deviation of a learner’s single score from our expectation, and obviously, the more the English vocabulary question that the learner answers is in the learner’s learning zone, err the closer the difference is to 0. The actual experiment looked at the mean value of the recommended deviation in each round or specified zone, i.e: (10)ERR=∑i=1merrim

where m is the number of responses in the round or the specified block.

Reinforcement learning is relatively difficult to measure, and is usually evaluated using the score at the end of the current round or the average of the results of several rounds, so the evaluation in this experiment was based on analyzing the results of 200 rounds of user responses. In order to observe the stability of the algorithm, the number of correct answers for each of the 200 rounds is shown in a line graph. Figure 3 shows the distribution of the number of correct answers for individual users with training, and Figure 4 shows the distribution of the number of correct answers for all users with training.

Figure 3.

The distribution of the correct number of individual households

Figure 4.

All households are distributed

δ∗In the last 200 rounds of training, the number of times learners answered correctly each time was δ∗N was significantly increased, and the highest accuracy rate reached 71% and 69% respectively in the single user answer and all user answer experiments, and the number of correct answers was mainly distributed around δ∗N, and the frequency of other correct values was significantly reduced. The model in this experiment can effectively learn the English vocabulary questioning strategy during training, and the level of English vocabulary questioning from the learner’s learning area is increasing and stabilizing with the number of training sessions.

In order to evaluate the strategy more consistently, we look at the learning process of reinforcement learning through another dimension of the metric, namely the value function of the directly used action. Because the state space of this experiment is large and there is no way to select a fixed user state, we choose the value function of the initial state of a single user in each round to see whether the model in this experiment is stable in its decision-making on the sub-states during the training process. The value function of the initial state of a single user in each round is shown in Fig. 5.

Figure 5.

The value function of the initial state of a single door

The q value in the figure represents the expected discounted reward value that the reinforcement learning model can obtain for using the best action for that state. In order to accurately assess the change in the q value, we filtered the q value when the optimal action was taken in the initial state of the round each time, and there were 15,000 values. The value of q value oscillates in the pre-training in the number of 0-7500 iterations, oscillating in the range of 0-16.8, and in the number of 7501-15000 iterations, the trend of the training stabilizes, with an average value of about 16.3. It indicates that the experimental model gradually stabilizes the value of q value in the middle and late stages of training during the process of exploring English vocabulary learning strategies, from which it shows that the reinforcement learning model has been trained and achieved better learning results.

At the same time, based on the given recommendation bias metric, it is possible to see the decision-making of the model selection strategy each time. The value of ERR is calculated once per round. Figures 6 and 7 show the mean and the variance of ERR per 200 rounds for a single user, and Figures 8 and 9 show the mean and the variance of ERR per 200 rounds for all users. At the beginning of the training period, both the mean and the variance were large, with the mean and variance of ERR per 200 rounds for individual users reaching a maximum of 0.0391 and 0.0352, respectively, and that of ERR per 200 rounds for all users reaching a maximum of 0.041 and 0.084, respectively. However, as the training period progressed, both the mean and the variance of ERR per 200 rounds for individual users and the mean and variance of ERR per 200 rounds for all users reached a maximum of 0.041 and 0.084, respectively. However, as the training progresses, both the mean and the variance of ERR per 200 rounds for individual users and the mean and the variance of ERR per 200 rounds for all users decrease and stabilize. The mean and variance of ERR per 200 rounds for individual users eventually converged to 0.0041 and 0.0040, while the mean and variance of ERR per 200 rounds for all users stabilized at 0.01 and 0.03, respectively, which indicates the gradual stabilization of the model’s decision-making on the selection of topics in this paper, and the gradual reduction of bias in the model’s recommendation of English vocabulary exercises and its stabilization.

Figure 6.

The average of every 200 rounds ERR

Figure 7.

The variance of err per 200 rounds

Figure 8.

The average of every 200 rounds ERR

Figure 9.

The variance of every 200 rounds of ERR

The experiments use precision, recall, and NDCG to validate the performance of the model.

Precision is used to measure the proportion of the classification model’s predictions that are correct. The formula for calculating precision is given below: (11)Precision=∑i−1N|A(u)∩B(u)|∑i−1N|A(u)|

where N represents the number of samples, A(u) represents the set of items that are the final output of this model, and B(u) represents the set of items for which there is indeed a preference interaction in the interaction record.

Recall, also known as sensitivity or true instance rate, is a measure of the proportion of positive instance samples that the model is able to correctly find. The recall rate is calculated using the following formula: (12)Recall=∑i−1N|A(u)∩B(u)|∑i−1N|B(u)|

The discounted cumulative gain (NDCG) is a metric used to assess the quality of ranking systems, especially in information retrieval and recommender systems. It is obtained by summing the relevance scores of each item in a recommendation list weighted by position and then normalizing them in order to compare the quality of different lists. The value of NDCG usually ranges from 0 to 1, and the closer it is to 1, the better the performance of the system in terms of ranking and recommendation.

The formula for NDCG is: (13)NDCG@k=DCG@kIDCG@k

where DCG@k and IDCG@k are calculated as shown in Eqs. (14) and (15): (14)DCG@k=∑i=1krelilog2(i+1) (15)IDCG@k=∑i=1krel(i)log2(i+1)

In this paper rel_i denotes the relevance score of the ind English vocabulary exercise in the recommended list, rel_(i) denotes the score of the ith English vocabulary exercise after sorting the true relevance scores in descending order, and k is the number of the first k English vocabulary exercises considered.

In order to verify the superiority of the modeling algorithms, the following algorithms are used to compare with the experiments, respectively, and the comparison datasets use three more authoritative public datasets MovieLens, Yelp, and Last.fm. 1)

BPR algorithm

The BPR algorithm uses a pairwise learning approach to learn user preferences by comparing pairs of items that users have interacted with to achieve a personalized ranking of the recommendation list.

In order to verify the effectiveness and sophistication of this paper’s algorithm, a comparison test is conducted using this paper’s algorithm and the above multiple comparison algorithms on the same dataset as well as in the same operating environment, and the results of the comparisons at different k-values are shown in Tables 1, 2, 3, and 4.

According to the given data tables, it can be observed that the performance of each recommendation algorithm on different datasets is different in terms of Precision, Recall and NDCG metrics.

First of all, it can be noted that the algorithm in this paper is the most prominent among all indicators in all datasets, for example, the Precision@3 in Yelp dataset is 0.1159, which is 20.60% higher than the suboptimal algorithm. Secondly, for other algorithms, the BPR algorithm and Dijkstra algorithm perform poorly than other algorithms in all datasets, and the NDCG@5 is only 0.3452 and 0.1961 in the Last.fm algorithms, and the algorithms in the middle are NeuCF, FPMC and DDPRG.

In contrast, the reason why the BPR and Dijkstra algorithms may perform poorly is because they are not designed to adequately take into account the dynamics of user behavior in the dataset. Taking k=5 as an example, the recall of the BPR algorithm in Yelp, Movilens, and Last.fm algorithms is 0.1582, 0.1154, and 0.3359, respectively. Although the BPR algorithm can be trained with implicit feedback data, it ignores the sequential information of the user and the item, as well as the user’s behavioral evolution process, resulting in a poor performance in terms of the English vocabulary exercise recommendations accuracy and poor performance in coverage. Dijkstra’s algorithm is limited by the large state space and slow convergence speed, which leads to unstable performance in the practical application of English vocabulary exercise recommendation, and the performance is not as good as other algorithms.

The NeuCF, FPMC, and DDPRG algorithms have some limitations compared to the algorithms in this paper. First, for example, with k=7, the accuracy, recall, and NDCG of NeuCF in the Movielens dataset are 0.1270, 0.5632, and 0.3999. The accuracy, recall, and NDCG of FPMC in the Yelp dataset are 0.1284, 0.5235, and 0.4314, and the accuracy, recall, and NDCG of the DDPRG algorithm in the Last.fm with accuracy, recall, and NDCG of 0.1800, 0.5782, and 0.4090, and NeuCF, although it employs a neural network model, it usually can only capture the static feature information between users and items, ignoring the dynamic characteristics and temporal relationships of user behavior. This may lead to the inability to fully consider the user’s historical behavioral sequences in the recommendation process, resulting in less accurate recommendation results. Second, although FPMC considers the interaction relationship between users and items, it focuses more on modeling the co-occurrence relationship between users and items, while ignoring the temporal order of user behavior sequences. Therefore, FPMC may not perform well when facing time-sensitive recommendation scenarios. In addition, although DDPRG combines reinforcement learning algorithms, it may be limited by unstable training and too large state space during the training process, resulting in less stable recommendation performance. In summary, although these three algorithms are able to provide personalized English vocabulary exercises recommendation to a certain extent, they are not as good as this paper’s algorithm in terms of considering the dynamic characteristics of user behavior, temporal relationships, and recommendation effects.

Finally, the algorithm in this paper adopts an intelligent personalized recommendation model for English vocabulary exercises based on a reinforcement learning model, which can effectively capture the temporal relationship and dynamic characteristics of user behavior. For example, when k=9, the accuracy, recall and NDCG of the proposed algorithm in Yelp are 0.1162, 0.5453 and 0.4070, respectively, the accuracy, recall and NDCG in the Movielens dataset are 0.1357, 0.5852 and 0.4441, and the accuracy, recall and NDCG in Last.fm are 0.1574, 0.5454, 0.3990, the proposed algorithm can model the user’s historical behavior sequence, so as to better understand the user’s behavioral preferences and interest evolution. Secondly, the algorithm in this paper utilizes reinforcement learning algorithm for recommendation decision-making, which can dynamically adjust the recommendation strategy according to the user’s feedback and environmental changes, so as to realize personalized recommendation service. After comprehensively comparing the above several algorithms, the model shows superiority and advancement.