Improving Mandarin Emotional Vocabulary in Application Writing Using Deep Learning Models

It is generally believed that in ensuring efficiency at the same time, jieba participle is a better participle effect of the participle tool, so the choice of jieba participle tool for participle. At present, jieba has a total of four kinds of word separation, respectively, the precise mode, full mode, search engine mode and paddle mode. Precise mode refers to the sentence by default method in order to seek the most accurate cut, for the movie review data, more suitable for the use of this mode.

Deactivated words often refer to a type of word that occurs more frequently in a text, but does not have an impact on the information contained in the text. For sentiment analysis, these words are also usually neutral, so they can be removed first. First, the Baidu deactivation word list was chosen for the text of the application because it works best for most corpora on average. Second, a small proportion of the deactivated words in the word list were found to overlap with words in the Mandarin Emotion Vocabulary Ontology. Finally, these overlapping words were excluded from the deactivation word list to avoid affecting the subsequent calculations.

Symbols or emoticons in text carry some sentiment information though, and they are also used in sentiment analysis studies. However, since the existing manually labeled sentiment lexicons generally do not have entries for symbols or emoticons, this information is not considered in this study for the time being and needs to be removed all together.

CBOW model is a model that predicts the occurrence probability of the current word based on the words in the context. The ECBOW model diagram is shown in Fig. 3. The left half represents the CBOW model, whose neural network structure is divided into input layer, projection layer and softmax layer, and the vocabulary w represents the one-hot vector, and V represents the m×|V| word vector table [16]. After the conversion of the word vector table into the projection layer, the vocabulary vectors v_i are superimposed as shown in Equation (6). (6)vt=∑ivi,i∈{t−m,....,t+m}

Figure 3.

ECBOW model diagram

In this paper, we set m = 2. The maximum probability of w is output at the output layer, and then the maximum likelihood is normalized to the function, as shown in Equation (7). (7)fw=1|c|∑i|c|logp(w|wi−m,wi−(m−1),...,wi+m)

The ECBOW model proposed in this paper is based on the principle that based on the original CBOW model that constrains the syntactic structure of the context, the added emotion constraint network structure constrains the emotion information of the text with positive and negative emotions [17]. In the added network structure, the input layer is the word w in the text with positive and negative polarity, and the projection layer s_j represents the one-hot vectors of all the words in the text with positive and negative meanings obtained by transforming them into word vectors and summing them through the word vector table v, as shown in Equation (8). (8)sj=∑invi,i∈{ 1,2,....,n}

The text is categorized into tweets with positive and negative sentiments, so a logical neuron is utilized to output the probability of being positive and negative as shown in Equation (9). (9)P(hj|sj)={ σ(sj⋅H)p(sj)=[1,0] 1−σ(sj⋅H)p(sj)=[0,1]

Where H is the vector parameter and notation p(s_j) is the text praxis, if the projection layer s_j is projected by praxis text then p(s_j) = [1, 0]. If the projection layer s_j is projected by praxis text then p(s_j) = [0, 1]. Thus maximizing the objective function as shown in equation (10). (10)ζ=α1|s|∑j=1|s|logP(hj|sj)+(1−α)fw

Where S denotes the set of sentences in the dataset, θ parameters include U, V, α, η, U for the left half of the projection layer to softmax layer vector parameters. α denotes the weight parameter. η denotes the learning rate. e denotes the magnitude of vector change. Where: V(w) denotes the vector of vocabulary w. (11)xs=∑wesjV(w) (12)hj={ 1 p(sj)=[1,0] 0 p(sj)=[0,1]

Huffman semantic dependency binary tree is described by lexical nodes and lexical dependencies and Huffman encoding, Huffman semantic dependency structure binary tree construction is generated based on the semantic dependency tree structure, the semantic dependency structure binary tree diagram is shown in Figure 4.

Figure 4.

Binary tree of semantic dependency structure

Since the center word can usually represent the main syntactic and semantic features of the phrase, it is considered to have strong predictive ability. In this paper, the root node of Huffman bifurcation semantic dependency structure is specified as the center word, and the path from words to the center word in the text is called bifurcation semantic dependency path in this paper.

In this paper, with the help of language modeling, the bifurcated semantic dependency path of each word in the text is taken as a text sequence, and the product of the probability of occurrence of the bifurcated semantic dependency path of all the words in the whole sentence, i.e., P_L(s), is computed, as shown in Eq. (13). (13)PL(s)=∏i=1sPL(wi)

where w_i is a leaf node in a semantic dependency binary tree. L denotes the sequence of paths of w_i. P_L(w_i) is the probability of occurrence of the binary semantic path of this node w_i. The bifurcation semantic dependency path feature extraction using bidirectional LSTM neural network is shown in Fig. 5. The input layer is the node in the bifurcation semantic dependency path of each word. The bifurcation semantic path feature acquisition is carried out through the bidirectional LSTM neural network, whose feature vector is f_p, and whose dimensionality is n_p. The neural network is finally made to converge by maximizing the objective function as shown in Eqn. (14). The feature acquisition is carried out through the bidirectional LSTM neural network. The vector representation of the feature layer i.e. the bifurcated semantic dependency path feature for each word in the text. (14)f=1N∑i=1Nlog(PL(wi))

Figure 5.

Extraction of binary semantic dependency path feature map

In the utterance, the center word has a strong predictive role, it can skip some auxiliary or adverbial thus achieving long-distance structural constraints, usually the center word can represent the main syntactic and semantic features of the phrase. Relative position indicates the position of words in the utterance relative to the center word. In this paper, the relative position feature is mapped into a n_rp-dimensional vector f_rp.

In this paper, we view the sentiment dictionary classifier construction problem as a sequence labeling problem, for each word in the text contains four features respectively: sentiment word feature f_e, bifurcated semantic dependency path feature f_p, center word information f_c and relative position feature f_p. The final synthesized feature f₀ is denoted as: (15)f0=[fer,fpr,fcr,frpr]r

Remember that the vector dimension of emotion word feature f_ϵ is d_e, the vector dimension of bifurcated semantic dependency path feature f_p is d_p, the vector dimension of center word information f_c is d_c, and the vector dimension of relative position feature f_rp is d_rp. Therefore, the dimension of the final combined vector f_θ is dim(f₀) = d_c + d_p + d_c + d_rp, and in this paper, we use a bidirectional LSTM network to process the sentence, so as to obtain the forward representation of the sentence h→ , and the inverse representation of the sentence h← , and choose tanh as the activation function to produce the final representation of the sentence as shown in Equation (16). The construction of emotion word classifier graph using bidirectional LSTM neural network is shown in Fig. 6. (16)h=tanh(h→+h←)

Figure 6.

Constructing emotion word classifier using bidirectional LSTM neural network

In this paper, the emotional lexical features are extracted through the text labeling framework, and this paper defines the PNO framework as the text labeling framework, noting the frame label T ∈ {P, N, O}, where the frame labels are denoted as follows: if there is a word with positive emotional color in the text, this paper labels the word as P: if there is a word with pejorative emotional color in the text, this paper labels the word as N: if there are words with positive emotional color and pejorative emotional color neither reflected in the text, this paper labels the word as O. color and pejorative emotional color are not reflected, this paper marks the word as O.

The output layer is calculated as shown in Equation (17). (17)O=softmax(Wsh+bs)

The dimension of the output node in the output layer is 3, with [1, 0, 0], [0, 0, 1], [0, 1, 0] to indicate that the output result of the term is positive (P), neutral (O), and negative (N), respectively. Then it is processed by softmax layer to find out the probability of belonging to a certain category, as shown in equation (18). (18)p(xi)=exp(xi)∑jexp(xj)

In this paper, the loss function cross entropy is used as the objective function, as shown in equation (19). (19)L=−∑i∑iyiilog(preii)

Where, y_n refers to the ird value in the actual label at the t moment and pre_ε refers to the ith value in the predicted label at the i moment. Where: e(w) ∈ D, the bidirectional LSTM sentiment classifier is first trained to complete the construction of the dynamic sentiment dictionary, and then the test text set is predicted by this sentiment classifier, if the output is P-labeled, then the current vocabulary is a positive-sense sentiment word and the positive-sense sentiment word is deposited into the positive-sense lexicon D_pos. If the output is N-labeled, then the current vocabulary is a negative-sense sentiment word and the negative-sense sentiment word is deposited into the negative-sense lexicon D_neg. If the output is O-labeled, then the current vocabulary is neutral and finally the primary extended static sentiment lexicon is obtained.

Firstly, the most popular social group in the website was selected, then the IDs of the users in the community were collected, and finally 396,1207 users’ IDs were collected, for which, after de-emphasizing the object of the comment by using the ISBN number, a total of 12,118,836 applications were collected, and an average of 3.3 applications were commented by each user. For each comment, fields are captured including “user ID”, “rating”, “comment content”, “web link”. In this paper, we mainly consider the construction of Mandarin sentiment lexicon, so we exclude comments that do not contain Chinese characters, comments that are a mix of Chinese and English, and comments that contain only emoticons as well as gibberish, leaving a total of 2313436 comments. After removing the three-star ratings representing neutrality, the statistical information table of short comments in the word frequency statistics stage is shown in Table 1.

Due to the experimental needs, the relevant fields of each comment data include “User ID”, “Rating”, “Comment”, “PageUrl” If the content of a field is empty, the comment is excluded. The fields of the final experimental corpus mainly include vocabulary and the distribution of vocabulary on ratings. Sample vocabulary rating statistics are shown in Table 2.

The experiment as a whole consists of two interrelated segments, lexicon filtering and sentiment score calculation. According to the rules and process of generating the candidate word set of the sentiment lexicon, the short comments in the database are extracted to generate a collection of ultrashort comments. First, among all 197,078,085 entries, preliminary filtering is carried out using the length of the comments, retaining the short comments with a length of 10 characters or less, and if the short comments appear repeatedly, they are counted as appearing only once, and the score is the score with the highest frequency, which yields 929,250 candidate emotion words. According to the calculation and filtering ideas in the previous section, the candidate emotion words are processed and the non-emotion words are eliminated. All the rules are formulated with a total of 9 rules. The rule set formulation and specific application in the model are as follows. 1)

Candidate words with a total word frequency of less than 3 were eliminated. According to the observation of the data, the illustrative users of the words with too low frequency are fewer and not representative. After this cleaning work, the score of each candidate emotion word is calculated according to SO-PMI, if the subsequent culling work is carried out first, it will result in the loss of emotional information, according to the characteristics of the data, the threshold H is set to 2. Sample results of the emotion word calculation are shown in Table 3. The positive and negative frequency comparisons of candidate words show that users are very inclusive of resources, and when the threshold H of SO-PMI is set to 1, words such as “embarrassment” and “feel general” are still positive. However, when the threshold H is set to 3, it is too harsh, and the intensity of common positive words such as “healing” and “not bad” does not even reach 1. Therefore, taking the above considerations into account, the threshold is set to 2.

After some processing operations, a sentiment lexicon based on large-scale ultra-short comments oriented to the field of application writing is formed, and the Top30 high-frequency sentiment words are shown in Table 5.

In order to evaluate the sentiment dictionary constructed in this paper, two indicators, recall rate and accuracy rate, are introduced and compared with other general dictionaries. Recall rate and accuracy rate are important evaluation indexes in the field of information retrieval and information recommendation, and they are also widely used for the quality evaluation of emotion dictionary. Define the confusion matrix for evaluating the experimental effect, and the confusion matrix for evaluating the experimental effect is shown in Table 6.

Table 7 shows the comparison results of sentiment dictionary pairs. By analyzing the words included in the dictionary constructed in this paper but not included in the other two general dictionaries in the recall sample set, it is found that the dictionary better contains the characteristic words in the field of applied writing, such as “illustrated and illustrated”, “typo”, etc. Dictionaries are better at discovering new words, such as “unclear”, “powerful”, etc. Dictionaries do a good job of including colloquial words, such as “read in one sitting”, “worth seeing”, etc. A dictionary can contain word ontologies and collocations, such as “pretty”, “too long”, etc.

The differences in the characteristics of writing texts from the five types of institutions for the four new composite statistical indicators (analytical thinking, impact, authenticity, and emotional tone) are shown in Figure 7. It can be found that the differences between the writing texts from the five types of institutions in these four comprehensive indicators are more obvious, especially in the two of authenticity and emotional tone, the writing texts from high schools are significantly higher than the other four types of institutions. In terms of the emotional tone of the text, the mean value of writing texts from middle schools was much lower than that of the other four types of institutions. The writing texts from key universities, secondary institutions, vocational institutions, and high schools all scored well above 75, showing a significant positive affect. In contrast, the overall emotional tone of middle school students’ writing texts had a mean of nearly 50, indicating that middle school students’ writing did not show a clear positive or negative emotional tendency. The standard deviations of each indicator for writing texts from all five types of institutions were large, suggesting that even individual writers from the same type of institution can vary widely in their performance on the same indicators.

Figure 7.

The characteristics of writing in five schools are different

The five types of institutional applications as positive and negative emotions in writing texts and the decomposition of negative emotions are shown in Figure 8 (Figure a represents emotional processes and Figure b represents negative emotions). By examining the two secondary categorical indicators of positive and negative emotions in the emotion lexicon, this study found that the frequency of words expressing positive emotions was much higher than the frequency of words expressing negative emotions in writing texts other than middle school. Taking positive emotion words as the statistical target, the frequency of positive emotion words appeared in the writing texts of high school students was the highest, while middle school was the lowest. The frequency of positive emotion words in the composition texts of key universities, second colleges and vocational colleges resided between junior and senior high school students, and there was no significant difference within the three. Among the negative emotions, the frequency of words in the texts of middle school students was the highest among the five types of schools and much higher than the other four types of schools.

Figure 8.

Positive, negative and negative to emotional decomposition

The top 20 positive affective lexical items that appeared most frequently in the writing texts of the five types of institutions are shown in Figure 9. In the high-frequency word use of positive emotion-related words, the various types of schools reflected certain common emotion lexical items, but each still had some unique high-frequency emotion lexical items. Taken together, the general positive emotion words such as good, well, better, great, best, and love were widely used in their own right and ranked high in frequency in students’ writing texts at all levels. Words commonly used to modify specific people or objects appeared more frequently in the texts of junior high and high school students’ compositions. For example, the word frequency of beautiful ranked 5th among the positive affective words used by junior high school students and 9th among the positive affective words used by high school students, both of which were relatively high, and the word did not appear among the top 20 high-frequency positive affective words in the texts of the students from key universities, second-base colleges and vocational colleges.

Figure 9.

The top 20 positive emotional entries in the writing text of five schools

The top 20 negative affective lexemes that appeared most frequently in the writing texts of the five types of institutions are shown in Figure 10. The top 20 high-frequency words used by middle school students were more clustered in the use of negative affective words in the essays of the five types of institutions. For example, the negative affective word broke, which ranked 20th in frequency in the middle school text, appeared much more frequently than words that ranked much higher in frequency in the essays of students from key universities, second colleges, and vocational schools, such as bad. In contrast, high school students’ use of negative affective vocabulary was relatively dispersed, with even the word DIFFICULT having a relative frequency of only slightly more than 0.022%.

Figure 10.

The top 20 negative emotional entries in the writing text of five schools

In order to further explore the affective differences and their causes among learners writing texts in different types of institutions, the categorical indicator of drive was selected for quantitative analysis in this study. A comparison of application writing texts based on the “drive” categorization indicator is shown in Figure 11. The analysis revealed an increase in the frequency of vocabulary related to a sense of community, accomplishment, and honor in composition texts from middle school to college.

Figure 11.

The application text is based on the comparison