A Study on the Effectiveness of Designing and Applying Computer-Assisted Grammar Error Correction System in College English Writing Teaching

Machine translation is a task of generating another target language that expresses the same meaning by automatically reading a sequence of natural language. Machine translation models can provide a variety of candidate translations. Due to the differences between languages, many candidate translations have problems such as poor syntax and semantics, and do not conform to the daily expression habits. However, expanding the training data by forming parallel sentence pairs of candidate and target results provides a solution idea for data augmentation methods.

An end-to-end neural machine translation model that utilizes neural networks to achieve mapping from source language text to target language text. The main idea is “encoding-decoding”: given a source utterance x, x=(x1,x2……,xT) , an encoder is used to map it into a continuous, dense vector c, and then a decoder is used to transform the vector into a sentence in the target language y, y=(y1,y2,……yT) . p(y) represents the output probability of the target sentence. The formula is shown below: (1)p(y)=∏t=1Tp(yt|{y1,y2,,….,yt−1},c)

1)

Transformer model

The function of an encoder is to transform a non-fixed-length semantic sequence into a fixed-length context vector and encode the information of the input semantic sequence. Transformer’s encoder is composed of N identical neuromodules, where the output of the previous neuromodule is the input of the next neuromodule. Each neuromodule has a multi-head attention layer and a feedforward network layer [28]. The Multihead Attention Layer consists of multiple Attention Layers spliced together, and each layer uses the Attention mechanism. The ATTENTION mechanism in the Transformer allows the model to focus on the valuable information in the context. When encoding words, not only the current word is considered, but also the contextual context of the current word. The entire contextual context is incorporated into the current word vector. The encoder consists of a number of layers and each layer includes two sub-layers, the self-attention layer and the feed-forward neural network layer.

The decoder has the same number of layers as the encoder and consists of a self-attention layer, an encoding-decoding attention layer and a feedforward neural network layer. The encoding-decoding attention layer helps the decoder to focus its attention on the relevant parts of the input sentence. Each sub-layer in the encoder and decoder uses residual concatenation and normalization. The hidden layer in the neural network uses normalization to normalize the output of each layer to a standard normal distribution, which ultimately speeds up training and convergence. The formula for calculating the multiple attention layers is as follows: (2)MultiHead(Q,K,V)=Concat(head1,……headn)W0 (3)headi=Attention(QWiQ,KWiK,VWiV) (4)Attention(Q,K,V)=softmax(QKTdi)V

Where, Q denotes the query vector, K denotes the relevance vector, and V denotes the vector of queried information, i.e., the real-valued matrix. d is the fixation factor, and d is the hidden layer dimension. 2)

C-Transformer model

The principle of Copy-Augmented Transformer is to copy the unchanged words from the original utterance to the target utterance. In the text correction process, only a few words need to be corrected in each utterance, and the remaining text needs to be copied directly into the target utterance. The Transformer model based on the copying mechanism is able to determine whether the predicted output of the next word is generated by copying or from the word list space [29]. Ptgen represents the output of the decoder and denotes the probability that t moments are generated after word list decoding using regular attention mechanisms. α1copy is a balancing factor in order to balance the word list generation operation with the original text copying operation. Ptcopy represents the probability that the decoded word at moment t is copied from the original text. The final generation probability distribution of the target word is determined by the generated decoding probability Ptgen and the copying probability Ptcopy . The target language decoding formulas with copying mechanism are shown in Eqs. (5) to (8): (5)Pt(w)=(1−αtcopy)*ptgen(w)+(αtcopy)*ptcopy(w) (6)αtcopy=sigmoid(WT∑(AtT⋅V)),AtT=(qtTK)T (7)Ptcopy(w)=softmax(At) (8)qt,K,V=httrgWqT,HsrcWKT,HsrcWvT

where q_t is the mapping of the decoded hidden state at moment t, K is the different transformations of the input encoded hidden state, and W is the parameter matrix.

A machine translation system consists of multiple components, each of which will provide multiple, different candidate target sentences. Due to the variability among different languages, the quality of translations obtained using machine translation varies, and even many translations are ungrammatical. In addition, in the text generation task, the output of the model in the previous time step affects the results of the next time step, and the final generation of a number of this paper will be selected using conditional probabilities based on historical results, i.e., the model only judges the corrected results by the probability of the value of the value of the good or bad. Therefore the corrected output of the model may not be the optimal corrected result. Nowadays, the candidate translation with the highest comprehensive score is selected as the output by using the reordering method of multiple features, which makes the final correction result optimized.

In this paper, we will regard the English grammar error correction task as a machine translation task. For the input source utterance (x1,…,xN) containing grammatical errors and the corresponding corrected target sentence (y1,…,yT) , the sequence generation process of the corrected sentence can be represented by the following equation: (9)h1…Nsrc=encoder(Lsrcx1…N) (10)ht=decoder(Ltrgyt−1…1,h1…Nsrc) (11)Pt(w)=softmax(Ltrght)

where L is the word embedding matrix, h1…NsrcN is the implicit state of the source utterance after encoding it through an encoder, h_t is to predict the implicit state of the target word, and P_t(w) represents the probability distribution of the target word.

However, the English grammar error correction model based on neural machine translation still faces great challenges. A complex neural network has a large number of parameters, and often a huge amount of data is required to make the model adequately trained. The number of sentence pairs with manual annotation for grammatical error correction is currently low, with only more than one million relevant annotated sentence pairs, and even less annotated data of higher quality. This greatly restricts the research progress of syntactic error correction algorithms based on neural machine translation. The scarcity of training data has become an important bottleneck in the research of grammatical error correction, which is a problem that needs to be solved urgently.

In this paper, we will expand the training data by creating pseudo-parallel sentence pairs, and study the effect of different pseudo-parallel sentence generation methods on the model effect.

The three different ways of generating a pseudo-parallel corpus used in this paper are as follows: 1)

Noise Perturbation

The simplest way of generating syntactic errors is considered first, i.e., taking the approach of adding noise directly to a correct sentence. In order to avoid the situation of too many unlogged words, the sentence is first subjected to BPE segmentation, and then words are deleted, replaced, masked and other words are randomly inserted with random probability. This method can perturb the correct sentences and generate a large number of incorrect sentences, constituting a large number of parallel sentence pairs to augment the training data.

In practice, the subword to be manipulated in a sentence is first randomly selected, the word is replaced with “<UNK> ” with a probability of 0.1, i.e., the current word is masked, the original word is kept with a probability of 0.2, i.e., the sentence is not altered, and the current word is randomly replaced by a word from the whole corpus with a probability of 0.35, and the current word is deleted with a probability of 0.35 probability of deleting the current word. Briefly, the probabilities of masking, keeping, inserting and deleting the words in the sentence are denoted by p_mask, p_keep, p_insert and p_delete respectively, and the probability of the simplest noise addition selected in this paper is [pmask,pkeep,pinsert,pdelete]=[0.1,0.2,0.35,0.35] .

By the above method, a syntax error generation model can be trained, and by inputting some English corpus without syntax errors to the model, we can obtain English text with syntax errors, and generate a large number of parallel sentence pairs of “wrong sentence, right sentence”. In this experiment, this paper will use the BERT-fused model as the reverse grammatical error generation model [30].

In the English grammar error correction context, it is often a small portion of words that have grammatical errors, and most of the rest do not contain grammatical errors. In order to avoid the grammatical error correction algorithm from perturbing the correct sentences, this paper constructs a Transformer model (C-Transformer) based on the copying mechanism, i.e., those words that do not contain grammatical errors are directly copied into the target sentence. The probability distribution of the words in the target sentence is a mixture of the probability distribution Ptgen generated by the error correction model and the probability distribution Ptcopy copied from the source utterance, and the specific mechanism is shown in Equation (14): (14)Pt(w)=(1−αtcopy)*ptgen(w)+αtcopy*ptcopy(w)

where αtcopy∈[0,1] , αtcopy are the balance parameters used to control the generation probability and replication probability at each time step t.

The structure of the Transformer model based on the replication mechanism is shown in Fig. 1. In this structure, the probability distribution of the target word is generated by the underlying Transformer model, and a replication score is computed using the implicit state h1…Nsrc(Hsrc) of the input word of the source utterance and the implicit state h^trg of the target word. The specific replication-based attention score is computed in the same way as the underlying Transformer attention score, as shown in Eqs. (15) to (17): (15)qt,K,V=httrgWqT,HsrcWkT,HsrcWvT (16)At=qtTK (17)Ptcopy(w)=softmax(At)

where q_t, K, and V are the query vector, key vector, and value vector that are used to compute the attention distribution and the replicated hidden layer, respectively. See Eq. (18) for the computation of the normalized attention distribution as the replica score and the replica hidden layer to estimate the equilibrium parameters αtcopy , αtcopy , and Eq. (19) for the loss function of the model: (18)αtcopy=sigmoid(WT∑(AtT⋅V)) (19)lce=−∑t=1Tlog(pt(yt))

Figure 1.

Transformer model structure based on replication mechanism

Adam (Adaptive moment estimation) is a stochastic optimization algorithm developed from the stochastic gradient descent (SGD) algorithm that can optimize neural network models very effectively [31]. It discards the method of updating the neural network parameters with the same learning rate in SGD, and chooses to use the methods of first-order estimation and second-order estimation to generate different adaptive learning rates for updating the network parameters. Adam’s algorithm combines the advantages of Adaptive Gradient Algorithm (AdaGrad) and Root Mean Square Propagation Algorithm (RMSProp), which is not only capable of solving the problem of the presence of sparse gradients, but also there are advantages such as high computational efficiency and better handling of non-stationary problems.

For the objective function f(ω), the parameters to be optimized ω and the initial learning rate α, Adam’s iterative process for parameter ω_t in step t is shown in the following equation: (20)gt=∇f(ωt) (21)mt=ϕ(g1,g2,…,gt) (22)Vt=ψ(g1,g2,…,gt) (23)ηt=α⋅mtVt (24)ωt+1=ωt−ηt

where g_t denotes the gradient of the objective function with respect to the parameters at the current moment, and m_t and n_t refer to the first-order and second-order momentum at the time of gradient computation. η_t and ω_t+1 refer to the gradient and the updated parameters at moment t, respectively. The detailed calculation process of the first-order momentum and second-order momentum refers to the following equation: (25)mt=β1⋅mt−1+(1−β1)⋅gt (26)Vt=β2⋅Vt−1+(1−β2)gt2

The English Grammar Automatic Error Correction System can be divided into five layers according to different functions, and each layer refines the functions of the system into sub-tasks, which reduces the interdependence of the various parts of the system, facilitates the development and debugging of the system, and improves the scalability of the system. The overall framework of the system is shown in Figure 2, which is divided into four layers: data layer, application layer, representation layer and feedback layer. 1)

Data layer: this layer is used to receive the text to be detected sent by the user and preprocess the data, including removing html tags, removing irregular symbols and so on.

Figure 2.

Overall system framework

In this paper, the English grammar automatic error correction system is divided into three parts: grammar error correction module, user feedback module and visualization module. 1)

Grammar Error Correction Module

The flow of the grammar error correction module is shown in Figure 3, which is mainly composed of five parts: text preprocessing, pinyin detection algorithm, grammar error correction algorithm, result optimization module, and grammar error correction model training. The main function of text pre-processing is to receive and pre-process the user request data, because manual writing may not be standardized, so this module needs to process the html tags and non-standard punctuation marks contained in the text. The text of the user’s request is cut according to the sentence level, and the text is broken. Grammatical error correction is performed sentence by sentence on the broken text, firstly, pinyin detection is performed on the text to find out the words in the statement that conform to the pinyin rules, and grammatical error checking and correction is performed on the statement. In order to avoid the types of errors caused by pinyin, the error correction results are optimized to remove those that conform to the pinyin rules in the set of error correction results, and finally, the detection results are returned to the user in the format of splicing processing for the user’s feedback on the error correction results.

Figure 3.

Syntax correction process

Figure 4.

User feedback process

The experimental data were chosen to be obtained from the English corpus, and the dataset was named CoNLL2023 English Writing Corpus, which contains a total of 25,000 sentences with 160 different kinds of grammatical errors in English writing, and the types of grammatical errors mainly include grammatical error types such as gerund tense errors, errors in the use of articles, errors in prepositions, and errors in acronyms.

The specific method of preprocessing the corpus is as follows: firstly, the English utterances are checked for spelling errors. Then the corpus is expanded, i.e., a mixture of news corpus, ESL corpus and corrected English corpus is used. Finally, manual error generation is performed, i.e., the news corpus is compensated for manual errors during model training to narrow the gap between the corpora. After data preprocessing in the above way, 28,000 English utterances containing grammatical errors are obtained. In order to better validate the performance of the proposed grammatical error correction model for English writing, the experiment will divide this dataset into a training set, a testing set and a validation set according to the allocation ratio of 7:2:1.

In order to improve the model experimental effect, the Adam gradient descent optimizer is used for model training in this paper. Among them, the hyperparameter mini-batch of the model is set to 35, the initial learning rate is set to 0.0001, and the neuron disconnection probability p in the Dropout layer is set to 0.6.

The experiment chooses F_0.5 values, accuracy (P) and recall (R) as the evaluation indexes of English grammar error correction. The specific expressions of the three evaluation indexes are: (29)P=∑i=1n|gi∩ei|∑i=1n|ei| (30)R=∑i=1n|gi∩ei|∑i=1n|gi| (31)F0.5=(1+0.52)×R×PR+0.52×P

In the above equation, n denotes the total number of sentences employed in the test. g_i and e_i denote the set of standard and system-given modification results in sentence i, respectively.

In order to verify the performance advantages of the proposed C-Transformer-based English writing grammar error correction model, the experiments will be conducted by using the CoNLL2023 English writing corpus applied to this model and the traditional CAMB English grammar error detection and correction model for comparison experiments, respectively, and the model performance comparison results are obtained as shown in Table 1.

From the comparison results in Table 1, it can be seen that the accuracy, recall and F_0.5 metrics of the present model take the values of 37.52%, 42.75% and 38.46%, respectively, which are 16.68%, 20.38% and 17.33% higher than those of the traditional CAMB grammar error correction model, respectively. Comprehensive analysis shows that this model has higher accuracy and precision for detecting and correcting grammatical errors in English writing, better detection effect, and more superior model performance.

In order to further validate the error correction effect of the proposed English writing grammar error correction model, the experiment will be based on the test set in Table 2 for six different types of English writing grammar errors. The number of different error types and their percentage in the dataset are shown in Table 2. The total number of English sentences in this test set is 5600, among which the number of sentences with English writing grammar errors is 2241, and the data set can be divided into Short and Long parts according to the length of the sentences, through which the validation of English writing grammar correction will be carried out.

Based on the corpus of coronal errors and prepositional errors in the test set of Table 2, the experiment compares the present model with the traditional UIUC method and Corpus GEC method for grammatical error correction, and obtains the comparison results of the three grammatical error correction methods as shown in Table 3.

From the comparison results in Table 3, it can be seen that in the coronal error detection, the error correction accuracy, recall and F0.5 of the present method take the values of 61.48%, 59.82% and 61.14%, respectively, which are higher than the other two grammatical error correction methods. And the F0.5 value of the present method is 12.00% and 7.14% higher than the UIUC method and Corpus GEC method, respectively. In preposition error detection, the error correction accuracy, recall and F0.5 of the present method are also greater than the corresponding values of the UIUC method and Corpus GEC method, reaching 42.14%, 45.63% and 42.80%, respectively. This shows that this method is effective in correcting grammatical errors of articles and prepositions in English utterances, and it improves the efficiency and precision of error correction for articles and prepositions by adding a replication mechanism in the decoding process of the Transformer model.

In order to further verify the error correction effect of the proposed grammatical error correction for English writing, the experiments will be carried out on the four typical error types of articles, prepositions, nouns and verbs respectively, and compared and analyzed with the long and short sentences as Long and Short respectively, and a comparison of the results of error correction of the four different grammatical errors by this method is obtained, as shown in Fig. 5.

Figure 5.

Error correction results of the method for four different syntax errors

As can be seen from the results in Fig. 5, the error correction F_0.5 value of this method for prepositions, nouns and verbs is significantly higher than that of short sentences, mainly due to the fact that the grammatical errors of English sentences in long sentences occur more frequently, and there are dependencies between individual words that are farther away, with less error-interfering information. Comprehensive analysis shows that the F_0.5 values of this method for four different grammatical error correction are kept above 55%, which indicates that the grammatical error correction results of this method satisfy the error correction needs of the English grammar error correction system.

In order to investigate the practical effect of the designed computer-aided grammar error correction system in college English writing teaching, this paper designs a teaching experiment. The experimental subjects were selected from 95 freshman undergraduates who were not majoring in English in College A. The experiment was conducted from November 1, 2023 to May 31, 2024, excluding one month of winter vacation, a total of six months, and the experimental process was divided into three stages: the first round of action experiment, the second round of action experiment and the third round of action experiment.

The first round of action experiment is from November 1, 2023 to December 31, 2023, the second round of action research is from January 1, 2024 to March 31, 2024, and the third round of action research is from April 1, 2024 to May 31, 2024. Composition training was conducted once every two weeks in all three phases, and the automatic English grammar error correction system designed in this paper was used to correct grammatical errors, with a total of four compositions in each training.

By correcting the pre-test paper, this paper obtained the statistical results of the score rate of the pre-test paper as shown in Table 7 according to the eight lexical aspects examined in grammar: prepositional errors, pronoun errors, coronal errors, noun errors, adjective errors, adverb errors, verb errors (including predicate verbs and nonpredicate verbs), and conjunctions errors (including parallel conjunctions and subordinate conjunctions). Among the 20 questions tested, the highest score rate was 88.02% in question 5 which examined adverbs. The lowest scoring question was question 20 examining conjunctions with only 5.47%.

The test results before and after the first round of the action experiment are shown in Table 8. Before and after the first round of the action experiment, the score rate of the questions in 5 categories, such as prepositional, pronoun, noun, predicate verb and parallel conjunction, increased, but the score rate of the questions in 5 categories, such as coronal, adjective, adverbial, nonpredicate verb and subordinate conjunction, on the contrary, decreased. Among them, the biggest increase in the score rate of the noun category was 31.83%, while the biggest decrease was in the question of the adjective category, which amounted to 31.32%. Taken together, before and after the first round of the action experiment, although the fluctuating values of the students’ scores in individual categories varied greatly, the final average scores had few ups and downs and were more influenced by randomness.

The results of the paired samples t-test for the pre- and post-test scores of the first round of the action experiment are shown in Table 9. As can be seen from Table 9, the value of significance is 0.952, indicating that there is no significant correlation between the changes in the pre- and post-test scores and the first round of the action research. Combined with the fluctuations in the score rates of the various types of grammar questions, it can be seen that the first round of the action experiment did not have a great impact on the students’ knowledge of grammar.

The test results before and after the first round of the action experiment are shown in Table 10. Before and after conducting the second round of the action experiment, all types of grammar questions showed an increase in score rate, except for adverbs, which showed a slight decrease in score rate. The category with the highest increase in score rate was the predicate verb category, which reached 32.58%, and only the score rate of adverbs decreased slightly. Overall, before and after the second round of the action experiment, students’ scores on grammar questions increased significantly, and the fluctuation value of the average score increased more significantly compared to the first round of the action experiment.

A paired samples t-test for the pre- and post-test scores of the second action experiment is shown in Table 11. This time, the t-value is 3.138 and the significance value is 0.012<0.05, indicating that the changes in the pre- and post-test scores are significant and statistically significant. Combined with the fluctuation of the score rate of each type of grammar questions, it can be concluded that the second round of the action experiment has significantly improved the students’ mastery of each type of grammar.

After three rounds of action research, this paper administered a post-test to the students, using test questions of the same type and difficulty as the pre-test questions. The statistical results of the pre and post-test score rate comparison are shown in Table 12. Comparative analysis of the data in the table shows that the correct rates of the pre-test of 14 verb questions and 20 conjugation questions are 22.86% and 5.47% respectively, and the correct rates of the post-test are 40.35% and 21.81%, compared with other questions, the correct rates are not high, but the students’ grammatical learning of the types of questions has progressed significantly, and some of them have been able to analyze the errors better. Taken together, most students have improved their understanding of all types of lexemes. Although there was a decrease in scores on one quiz for articles (-8.71%), there was an increase in scores on the other quizzes, and it may be that this is just an isolated phenomenon and that these data show progress in students’ grammatical understanding and application.

In addition, this paper also conducted paired-sample t-tests on students’ pre and post-tests on single-sentence correction (A), short-text correction (B), single-sentence fill-in-the-blank (C), single-sentence translation (D), essay fill-in-the-blank (E), and written expression (F), and the results of the paired-sample t-tests for the different question types are shown in Table 13.

As can be seen from Table 13, compared with the scores on the pre-test, single-sentence correction, short-text correction, single-sentence fill-in-the-blank, single-sentence translation, composition fill-in-the-blank, and written expression increased by an average of 5.164 points, 5.164 points, 3.285 points, 3.447 points, 1.841 points, and 3.423 points, respectively. The increases in the scores for each question type varied, but proved that the students had made some progress. In summary, the post-test performed better than the pre-test on all items mentioned. The t-value proves that such an improvement is statistically significant and the significance value of 0.000 for each item also indicates the significance of the improvement. This means that all the teaching activities, whether it is single sentence correction, short correction, single sentence fill-in-the-blank, single sentence translation, essay fill-in-the-blank, or written expression have achieved effective improvement. This fully demonstrates that the C-Transformer-based English grammar automatic error correction system designed in this paper can be applied to the teaching of English writing and can significantly improve students’ mastery of grammar.