A Study on the Influence Mechanism of English Corpus on Translation Quality in Multilingual Website Translation

To for the current input source-target utterance pair < x,y >, the coding layer encodes the source utterance x by means of a bi-directional encoder that incorporates the knowledge of the words before and after each word into the coding vector as shown in Eqs. (5) and (6): (5) h→ix=RNN(xi,h→i−1x) \[\vec{h}_{i}^{x}=RNN({{x}_{i}},\vec{h}_{i-1}^{x})\] (6) h←ix=RNN(xi,h←i−1x) \[\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\leftarrow}$}}{h}_{i}^{x}=RNN({{x}_{i}},\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\leftarrow}$}}{h}_{i-1}^{x})\]

The splicing is performed to obtain the expression for each word, as shown in Equation (7): (7) hix=[h→ix;h←ix] \[h_{i}^{x}=[\vec{h}_{i}^{x};\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\leftarrow}$}}{h}_{i}^{x}]\]

Meanwhile, the expression before and after splicing toward the final moment is used as the final expression of the sentence, as shown in Equation (8): (8) hx=[h→Nq;h←1q] \[{{h}^{x}}=[\vec{h}_{N}^{q};\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\leftarrow}$}}{h}_{1}^{q}]\]

Similarly, the same operations are performed on the target statement v as on the source statement to obtain the final expression h^y of the target statement.

The clustering layer divides the source-target utterance pairs in the current corpus into K different source-target semantic cluster pairs by constructing a memory module. Specifically, after obtaining all the source-target utterance expressions in the English corpus through the encoding layer, all the source utterance expressions are divided into K different source semantic clusters by using the K-Means method, and all the target utterance expressions corresponding to all the source utterances under each source semantic cluster are put into the target semantic clusters corresponding to the source semantic clusters. Thus, the K group of source-target semantic cluster pairs is formed. The source semantic cluster group is denoted as Cx={C1x,C2x,⋯,Cix,⋯,CKx}${{C}^{x}}=\{C_{1}^{x},C_{2}^{x},\cdots ,C_{i}^{x},\cdots ,C_{K}^{x}\}$, where Cix$C_{i}^{x}$ is the cluster center expression of the ith source semantic cluster, and similarly, the target semantic cluster group is denoted as Cy={C1y,C2y,⋯,Ciy,⋯,CKy},Ciy${{C}^{y}}=\{C_{1}^{y},C_{2}^{y},\cdots ,C_{i}^{y},\cdots ,C_{K}^{y}\},C_{i}^{y}$, which is the cluster center expression of the i th target semantic cluster, and Cix$C_{i}^{x}$ corresponds to Ciy$C_{i}^{y}$ one-to-one.

For the current input source utterance expression h^x, its similarity with the cluster center of each source semantic cluster in the source semantic cluster group is first calculated as the correlation between the utterance and each source semantics αix$\alpha _{i}^{x}$ as shown in Equation (9): (9) αix=hxWaCix \[\alpha _{i}^{x}={{h}^{x}}{{W}_{a}}C_{i}^{x}\]

The weighted sum of all the correlations yields the memory module embedding expression M^x for the current input source utterance, as shown in Equation (10) and Equation (11).

Similarly, for the current input target utterance expression h^y, its similarity to the cluster centers of each target semantic cluster in the target semantic cluster is calculated as the correlation of this target utterance with each target semantic αiy$\alpha _{i}^{y}$, as shown in Equation (12): (12) αiy=hyWbCiy \[\alpha _{i}^{y}={{h}^{y}}{{W}_{b}}C_{i}^{y}\]

The weighted sum of all the correlations is used as a memory module for the current input target semantics embedded in expression M^y. As shown in Eqs. (13) and (14): (13) αiy=hyWbCiy \[\alpha _{i}^{y}={{h}^{y}}{{W}_{b}}C_{i}^{y}\] (14) My=∑i=1kαiyCiy \[{{M}^{y}}=\sum\limits_{i=1}^{k}{\alpha _{i}^{y}}C_{i}^{y}\] where Cix$C_{i}^{x}$ is the center of the ind cluster of the source semantic cluster, Ciy$C_{i}^{y}$ is the center of the ith cluster of the target semantic cluster, and W_a and W_b are parameter matrices.

In particular, the memory module embedding expressions M^x and M^y are obtained based on the overall semantics of the source and target utterances, respectively, and will be used as attentional information at the decoding level to guide the generation of translations at the sentence level.

The self-encoding layer mainly consists of the target utterance self-encoder. This module is mainly used to update all target utterance expressions in the corpus, thus updating the target semantic cluster expressions in the memory module. As shown in Eqs. (15) and (16): (15) Sy=MLP[hx;hy] \[{{S}^{y}}=MLP[{{h}^{x}};{{h}^{y}}]\] (16) y^′=LSTM(Sy) \[{{\hat{y}}^{\prime }}=LSTM({{S}^{y}})\]

The memory self-encoding layer loss is shown in Equation (17): (17) Lme=−E(x,y)∈ClogP(y^′|x,y) \[{{L}_{me}}=-{{E}_{(x,y)\in C}}logP({{\hat{y}}^{\prime }}|x,y)\]

The decoding layer uses a unidirectional decoder, and the initial hidden state S₀ is obtained by using a multilayer fully-connected neural network after first splicing the encoded vector h^x of the source utterance with the retrieved memory information M^x and M^y, as shown in Equation (18): (18) S0=MLP([Mx;hx;My]) \[{{S}_{0}}=MLP([{{M}^{x}};{{h}^{x}};{{M}^{y}}])\]

The decoding process utilizes the attention mechanism to capture the contextual attention information of the source utterance. Specifically, assume that the decoder obtains the hidden state S_t at moment t, which is normalized as a query vector with each word of the source utterance after computing the similarity as shown in Eqs. (19) and (20): (19) βt,i′=StWchix \[\beta _{t,i}^{\prime }={{S}_{t}}{{W}_{c}}h_{i}^{x}\] (20) βt,i=exp(βt,i′)∑i=1Nexp(βt,i′) \[{{\beta }_{t,i}}=\frac{exp(\beta _{t,i}^{\prime })}{\sum\limits_{i=1}^{N}{e}xp(\beta _{t,i}^{\prime })}\] where w_c is the parameter matrix. Weighted summation of all similarities yields the contextual attention H_t at the current moment, as shown in Eq. (21): (21) Ht=∑i=1Nβt,ihix \[{{H}_{t}}=\sum\limits_{i=1}^{N}{{{\beta }_{t,i}}}h_{i}^{x}\]

For the generative translation utterance decoding process, different target cluster attentions are computed for each moment since semantic shifts may occur at each moment and the target cluster semantics of interest are not exactly the same. Specifically, taking S_t as the query vector, its similarity to the cluster center of each target cluster in the target semantic cluster is calculated and normalized as shown in Eqs. (22) and (23): (22) ut,i′=StWdCiy \[u_{t,i}^{\prime }={{S}_{t}}{{W}_{d}}C_{i}^{y}\] (23) ut,i=exp(ut,i′)∑i=1Nexp(ut,i′) \[{{u}_{t,i}}=\frac{\text{exp}(u_{t,i}^{\prime })}{\sum\limits_{i=1}^{N}{\text{exp}}(u_{t,i}^{\prime })}\] where W_d is the parameter matrix. The target cluster attention at the moment of t is obtained by weighted summation of all similarities Z_t. As shown in Eq. (24): (24) Zt=∑i=1Nut,iCiy \[{{Z}_{t}}=\sum\limits_{i=1}^{N}{{{u}_{t,i}}}C_{i}^{y}\]

In particular, the target cluster attention is computed once in the clustering layer and the relevant target semantic cluster information is obtained, while it will be obtained again at each moment in the decoding process, which is a two-level attention mechanism. The statement expression is used as the query vector in the clustering layer, and the attention information based on the overall semantic level is obtained, which guides the whole translation process at the sentence level, while the current hidden layer expression is used as the query vector in the decoding, and the attention information based on the currently decoded word is obtained, which guides the current translation moment at the word level. The overall attentional information ensures semantic accuracy and fluency of the overall translated utterance, while the attentional information of individual decoded moments allows for better alignment of the translation at the word level.

After fusing the three parts of information, the decoder generates the current t-moment word as shown in Equation (25): (25) p(y^t)=softmax(Wp[St,Ht,Zt]+bp) \[p({{\hat{y}}_{t}})=softmax({{W}_{p}}[{{S}_{t}},{{H}_{t}},{{Z}_{t}}]+{{b}_{p}})\] where W_p is the parameter matrix and b_p is the bias vector.

The loss function of the decoding layer is shown in Eq. (26): (26) Ld=−E(x,y)∈ClogP(y^′|x,My) \[{{L}_{d}}=-{{E}_{(x,y)\in C}}\text{log}P({{\hat{y}}^{\prime }}|x,{{M}^{y}})\]

The total model loss function consists of two parts, the decoding layer loss and the self-coding layer loss, as shown in Equation (27): (27) L=Lme+λLd $L={{L}_{me}}+\lambda {{L}_{d}}$ where λ is the hyperparameter.

In the training phase, within the current epoch, the encoding results of all source-target utterance pairs are first obtained and K-Means clustering is used to obtain the memory module, after which training is performed. At the end of the current epoch, all source-target utterance pair expressions are re-acquired, the memory module is updated, and the next round of training is performed.

For testing, only the source utterances are used as inputs, and the similarity of the source utterances in the source semantic clusters is used to replace the similarity of the target utterances in the target semantic clusters.

In order to verify the superiority of the constructed neural machine translation model incorporating translation memory, it is compared with other neural machine translation models in experiments on several divine machine translation (NMT) tasks, and the experimental results of different NMT systems are summarized in Table 1. Table 1 demonstrates the BLEU scores of the six NMT systems, and “Ours” denotes the NMT system that uses the NMT model construction method of fusing translation memories proposed in this paper.

In the six translation directions, compared with the baseline model, the model building method for fusing translation memories proposed in this paper can effectively improve the BLEU scores. The method in this paper has a significant effect on improving the translation quality of RNN models. Compared with the RNN model, the model in this paper not only achieves an improvement of 2.91 BLEU scores in the direction of English-German translation, but also achieves an improvement of 1.65 BLEU scores in the direction of Russian-English translation. For the Transformer model, the NMT system with this paper's method improves 1.75 BLEU scores and 1.54 BLEU scores in the German-English and Vietnamese-English translation directions, respectively. This paper's method improves the BLEU score of the mRASP model from 30.82 to 31.85 in the English-German translation direction, which has the greatest improvement effect on the multilingual pretrained model. In addition, the mBART model improves the BLEU score from 29.81 to 30.92 in the same translation direction, which also has an improvement effect.In summary, it can be seen that the modeling method proposed in this paper that incorporates translation memory achieves better performance in multiple translation directions for the multilingual neural machine translation model.

In order to better understand the proposed NMT model construction method for fusing translation memories, this paper further compares the BLEU scores obtained using the mRASP baseline model and the improved model of this paper to compare the advantages of this paper's method on six resource translation tasks. The BLEU score curves on the validation set under different translation directions are shown in Fig. 4. Where (a) ~ (f) represent the RLEU score curves on the six translation directions of English-German, English-Vietnamese, English, Russian, German-English, Vietnamese-English, and Russian-English, respectively.

Figure 4.

The curves of BLEU scores on validation set for different translation directions

As can be seen from Fig. 4, the NMT model with fused translation memory applied converges faster than the baseline model. This phenomenon indicates the better performance of the NMT model with fused translation memory during the model training process. One explanation for the NMT model with fused translation memory to have better performance momentum than the baseline model at the later stage of the training of the NMT model (i.e., after all the training samples are included) may be that when translating the relevant information through the retrieval memory module, it fuses both global and local attentional information, guiding the translation process from both the sentence-level and the word-level perspectives, which results in higher-quality translated utterances.

Meanwhile, when the source language is English, the performance of the mRASP model with the fused translation memory modeling approach outperforms the baseline at the outset. Several factors can explain this observation. First, the mRASP model was pre-trained using a large-scale multilingual corpus and learned shared information between multiple languages, benefiting from this prior knowledge, which focuses on the comprehension and generation of English sentences at that stage. This prior knowledge gives the mRASP model an initial advantage.Secondly, English examples tend to have relatively regular syntax and structure. This simplicity allows the mRASP model to quickly capture and learn patterns of expression in the language. Finally, the mRASP model is exposed to a large number of English sentences in the early stages of training, which helps the model learn English language features and expressions more quickly.

In order to investigate the mechanism of the influence of English corpus on translation quality in the translation of multilingual websites, this paper selected 80 English majors in the third year of undergraduate colleges as the experimental subjects, and designed an experiment of English parallel corpus-assisted manual translation. The experimental group used parallel corpus for the experiment, and the control group used conventional reference resources such as dictionaries for the experiment. In order to facilitate the comparison between the experimental group and the control group and control the variables as much as possible, the experiment was a 90min time-limited translation. At the same time, except for the difference in the use of reference tools, the conditions of the two groups were identical in all other aspects. Limited to the conditions of the existing English parallel corpus resources, only the German-English form was chosen for this experiment.