Research on multi-label short text categorization method for online education under deep learning

Text classification refers to extracting features from raw text data and predicting the labels of the text data based on these features. The multi-label text classification process is shown in Fig. 1, which can be roughly divided into the steps of text preprocessing, extracting text features and constructing a classifier [25].

Figure 1.

Multi-label essay the overall process of the class task

Text data is characterized by unstructured, varying lengths, and interspersed data noise. Therefore, preprocessing the raw input text is necessary, and is especially important in the classification of short texts. Preprocessing methods are effective techniques to reduce data sparsity and help improve low-quality text.

In text data, different text records may have different writing styles, there may be the use of emoticons, abbreviations, misspellings, and the use of urls, etc., and the use of appropriate text preprocessing can help computers learn good text representations. Text preprocessing generally involves splitting words, cleaning data, and analyzing statistics. After the text data is preprocessed, the text representation method is used to obtain a form of representation that is easier to understand and has the least loss of information for the computer. Generally speaking, a good text representation has richer semantic features, which can greatly improve the algorithm’s effect. Finally, the represented text is fed into the classifier according to the selected features.

For multi-label short text categorization tasks, the performance of the classifier can be evaluated using either micro-averaging (Micro) or macro-averaging (Macro) computational methods. The micro-averaging method does not consider the specific label information, treats all the categories as a binary classification task, and only judges whether the classifier correctly classifies them, and calculates all the TPs, FPs, TNs, and FNs first, and then calculates the corresponding metrics. The computational method of micro-averaging can evaluate the performance of the classifier better when the label distribution is relatively uniform, but when the label distribution is not uniform, the label categories with fewer samples tend to result in poorer classification of sparse label categories due to their lower weights in the computational process. In contrast, macro averaging is calculated by first calculating the accuracy, recall, and F1 composite score for each label category, and then averaging over all the categories, so it can also achieve better results on the more sparse label categories.

The macro-averaging is calculated as follows: 1Macro−P=1n∑i=1nTPiTPi+FPi 2Macro−R=1n∑i=1nTPiTPi+TNi 3Macro−F1=2*Macro−P*Macro−RMacro−P+Macro−R where i represents the test statistics associated with the i nd category label, and n is the total number of categories.

Word Frequency-Inverse Text Frequency (TF-IDF) constructs word feature representations by considering the importance of words in the text from a frequency perspective. The method asserts that words that can effectively represent the current text category should be those that appear more frequently in the current text and less frequently in other texts. In addition, a word should also gain importance if its total frequency of occurrence in all texts is low. To this end, TF-IDF uses both word frequency (TF) and inverse text frequency (IDF) to obtain a feature representation of a word, and then constructs a text vector space model to be used in the task of multi-label short text classification.

Word frequency, inverse text frequency and TF-IDF are calculated as follows: 4TF=nwk∑i=1Lnwi 5IDF=log(NNwk+1) 6TF−IDF=TF×IDF

Where w_k denotes the k nd word, L denotes the total number of words in the text, N denotes the total number of texts, and N_wk denotes the total number of texts containing the k th word.

The pre-trained language model is trained on a large-scale corpus by setting up pre-training tasks to enable the model to obtain a generalized comprehension of the language, and then fine-tuned in downstream tasks according to specific task scenarios to achieve better results. In order to avoid the problems of unlimited increase of feature dimension with the increase of the number of words and the independence of words from each other in the traditional text feature representation, the pretrained language models represented by Word2Vec, GloVe, and BERT represent the use of fixed-size dimension to represent words, and the use of a large amount of corpus data to obtain the generalized linguistic knowledge, so that semantically similar words will have a similar feature representation.

BERT is a pre-trained language model that learns a rich linguistic representation by training on a large-scale unsupervised corpus [26]. The BERT model is trained in a bi-directional way, which allows us to take into account both the left and right contextual information of a word, resulting in a richer representation. The BERT model’s foundation is the Transformer architecture, which is made up of multiple layers of Encoders. Each Encoder includes a multi-head self-attention mechanism as well as a forward propagation neural network. The multiple layers of encoders stacked in the BERT model allow the model to learn different levels of linguistic representations, which can be used for a wide variety of text categorization tasks. By fine-tuning the pre-trained BERT model, good performance can be obtained on a small amount of labeled data. In the working mechanism of BERT model, pre-training and fine-tuning together constitute the core working principle of BERT model.

Convolutional neural network (CNN) is an important deep learning model in deep learning, the use of CNN for feature extraction of textual information features, so that the model can get a good textual representation, the application of TextCNN to the field of multi-label short text classification makes the classification more accurate. The structure of CNN is shown in Fig. 2. The CNN uses the convolution kernel as a sliding window, the convolution kernel multiply and sum the features at the corresponding position by bit, this operation is called convolution operation. CNN uses the convolutional kernel as a sliding window, and the convolutional kernel is multiplied and summed with the features at the corresponding positions, which is called convolutional operation, and the convolutional operation enables CNN to efficiently perform feature dimensionality reduction of the feature values.The main model of CNN consists of the structure of the input layer, the convolutional layer, the pooling layer, and the fully-connected layer, and so on.

Figure 2.

CNN’s network structure

The input layer of CNN is mainly to input data to the model, and in text categorization it is necessary to vectorize the text information to convert it into a word vector matrix of text information. The main role of the convolutional layer is to capture semantic features from textual features through convolutional operations, which are realized by using a fixed-size convolutional kernel that can be slid. The convolution kernel is moved in the feature matrix in specific steps. Each step multiplies the convolution kernel with the feature values bitwise, composing the result into a new matrix for a new computation. The pooling layer is responsible for the operation of sampling feature values, and it mainly includes the average pooling layer, maximum pooling layer, and so on. The pooling layer can reduce the total number of feature values, discard unimportant feature values, and reduce the complexity of feature calculation. The fully connected layer is generally connected to the feature vector obtained from the pooling layer, through the fully connected layer even if the feature vector is converted into different output categories.

In standard LSTM, the transmission of cell states is unidirectional from front to back, so the LSTM model can only learn text features from past moments but not from future moments. In contrast, Bidirectional Long Short-Term Memory Network (BiLSTM) has two unit state conveyor belts, which transmit information from front-to-back and back-to-front, respectively, enabling the BiLSTM model to utilize the text data information of the past moments while being able to learn the features of the text information of the future moments, and to recursively and feedback on them, with more accurate prediction results than the unidirectional LSTM. Mining the connection between the past and future data of the time series to improve data utilization better use of the temporal features of the time series in order to improve the model prediction accuracy [27].

Suppose ht→ is the hidden layer state of the forward LSTM network at moment t, which can be regarded as a single-layer LSTM network, and the process of computing state ht→ at moment t from state h→t−1 at moment t – 1, and x_t is the input at moment t. Then: 7ht→=LSTM(xt,h→t−1)

Where ht→ is the hidden layer state of the forward LSTM network at moment t, x_t is the input at moment t, and h→t−1 is the hidden layer state of the state forward LSTM network at moment t – 1. Similarly ht→ is the hidden layer state of the reverse LSTM network at moment t, then: 8ht←=LSTM(xt,h←t−1)

Where: ht← is the hidden layer state of the forward LSTM network at moment t, x_t is the input at moment t, and h←t−1 is the hidden layer state of the state forward LSTM network at moment t – 1. The BiLSTM network output is the combination of the two parts of the hidden layer states ht→ and ht← , which constitutes the overall hidden state of the network h_t.

At its core, the attentional mechanism draws on the fact that the human visual system, in the course of a thorough scanning of the area a person sees, focuses its primary attention on the more critical information and ignores the ineffective information in it.

The attention mechanism can be described as a set of sequences of key-value pairs, where Key = (k₁, k₂, k₂,…,k_d), value Value= (v₁, v₂, v₃,…,v_d), and query sequence Query = (q₁, q₂, q₃,…, q_m), and the process of attention computation is a mapping process of key-value pairs in the query sequence [28]. The main computation steps are the following 3 steps:

First, the attention scores of Key and Value are computed score(q_t, k_t), i.e., e_ti. There are five main formulas to compute the attention scores, summation, splicing, dot product, scaled dot product, and bilinear, which are computed as: 9eti=score(qt,ki)=vTtanh(Wqt+Uki) 10eti=score(qt,ki)=vTtanh(W[ qtT;ki ]) 11eti=score(qt,ki)=qtTki 12eti=score(qt,ki)=qtTkidk 13eti=score(qt,ki)=qtTWki

Next, the output from the previous step is normalized using the SoftMax function so that the sum of all the obtained attention scores e_it is 1. i.e: 14ati=Softmax(eti)=exp(eti)∑n=1Nexp(etn)

Finally, the weight coefficients a_ti are assigned to the corresponding values v_i for weighted summation and finally the output of the attention mechanism is obtained. Namely: 15Attention(qt,K,V)=∑iativi

In traditional models, fixed weights are usually used for calculation, while the attention mechanism can adaptively assign weights according to different parts of the data, so that the model can pay more attention to the key information in the data, and screen and filter a large amount of redundant information in the model, thus improving the classification performance and model interpretability.

In order to be able to realize the accurate classification of multi-labeled short texts in online education, this paper is based on a bit of BERT model, the output of BERT is accessed into BiLSTM-CNN hybrid network as an Embedding layer, and the short text-level word vectors output from BERT are fused with the output from BiLSTM-CNN layer to form a BERT-BiLSTM-CNN hybrid model to improve the accuracy of classifying multi-labeled short texts for online education, and the structure of the model is shown in Fig. 3 [29].

Figure 3.

BERT-LSTM-CNN model structure

The BERT-BiLSTM-CNN model proposed in this study utilizes the BERT model’s feature of different output vectors for the same word at different positions in the text to solve the problem of multiple meanings of a word that may occur in a short text. Utilizing the feature that BiLSTM is able to capture context-dependent information in both forward and backward directions, the model is able to capture a more comprehensive global feature of the argumentative text. Utilizing the feature that CNN can abstract the text information in a higher dimension, so that the model can capture the local features of the text, combined with the advantages of deep learning techniques so that the model can better complete the task of multi-labeled short text classification for online education.

The specific steps for building the network structure of BERT-BiLSTM-CNN model are as follows:

Step1 Preprocess the text dataset to be classified to get the input text, denoted as E = (E₁, E₂,…,E_i,…,E_n), where E_i(i = 1, 2,…,n) denotes the i rd word of the text.

The word embedding technique can be understood as a distributed representation of words, which improves the computational efficiency and the semantic information representation of word vectors compared to the feature sparsity problem caused by traditional machine learning methods. For the task of encoding words into model inputs, traditional word embedding approaches use language models such as Bag-of-Words, Skip-gram, and Word2Vec models. Word embedding can be considered as a data pre-training process. In this study, the online education multi-label short text classification task belongs to the small sample task, the sample size is not enough to support the model to learn enough parameters, and it is easy to overfit on the training set, so it is necessary to use the pre-training language model as the word embedding representation.The BERT model uses a self-supervised approach to obtain generalized knowledge from the massive unsupervised data, and then fine-tunes it with a small amount of annotated data on teaching behaviors, to be apply to downstream text multi-label classification tasks.

In this paper, the BERT model is used to obtain the vector representation of the input text, and the sentence pairs composed of the labeling information constructed above and the original sentences are input using sentence pairs, which are represented as: 16input=[ CLS ]+xi+[ SEP ]+li+[ SEP ]

Where x_i = {w₁, w₂,…, w_m} represents the input sentence of a multi-tagged short text, [CLS] represents the start marker of the input sequence, [SEP] represents the split marker of the input sentence sequence, and l_i represents the tag information. Semantic coding based on the BERT pre-training model can transform the input text into word vectors by performing word embedding operations, and the probabilistic prediction of [MASK] tags is performed by the MLM task in the pre-training model.

After the input text data is obtained to BERT word vectors and sentence vectors through the text representation layer, the semantic features are further extracted by the parallel combination strategy of Multihead Self-Attention Mechanism, CNN and RCNN models.

Firstly, global feature enhancement of BERT word vector Token Embedding is performed with the help of the long-distance dependent learning ability of multi-head self-attention mechanism, which strengthens the ability to capture key features between sentences, and obtains the multi-head attention matrix MultiHeadToken.

Secondly, CNN model is used to extract local features. The N-gram features of MultiHeadToken are combined and screened using a convolutional kernel of height h in the CNN model to obtain local semantic information and word order features. Namely: 17ci=f(w·MultiHeadTokeni:i+h−1)+bc=[ c1,c2,⋯,cN−h+1 ]

Where, MultiHeadTokeni_i:t+h−1 denotes the local features in rows i to i + h – 1 of the Multi Head Attention Matrix Multi Head Token, b is the bias parameter and fis the nonlinear function. Then the most important text features are obtained by global maximum pooling, i.e: 18c^=max{ c }

After the local features are extracted by individual convolution kernels, the final convolution result CNNToken is obtained. i.e.: 19CNNToken=[ c^1,c^2,⋯,c^m ]

At the same time, Multi Head Token is input into CNN+ATTENTION model to further extract the global and local features of the text sequence. The context-dependent information of Multi Head Token bidirectional RNN Token is extracted by Bi-LSTM. i.e: 20h→t=f(W→xt+V→h→t−1+b→)h←t=f(W←t+V←h←t−1+b←)yt=g(Wy[ h→t,h←t ]+by)RNNToken=[ y1,y2,⋯,yN ] where h→t and h←t denote the hidden layer vectors at the moments of forward and backward propagation, respectively, W→ and W← denote the weight matrices at the moments of forward and backward propagation, respectively, b→ and b← are bias terms, and f and g are nonlinear functions.

BiLSTM model is prone to lose the semantic information at the beginning of the sentence at the decoding end, so the multi-head self-attention mechanism is used to strengthen its key feature information in the global perspective to obtain the attention matrix RNNMHA. Then, based on the optimization idea of RCNN model, a new CNN structure is added, and the n-gram local feature information of the attention matrix is enhanced to obtain the feature vector RCNN Token.Finally, it is spliced with CLSEmbedding and CNNToken to obtain the final text feature vector. Namely: 21y=concat(CLSEmbedding,CNNToken,RCNNToken)

After obtaining the final multi-labeled short text representation vectors through the feature fusion layer, the classifier is constructed using the fully connected layer and the Sigmoid activation function, where the fully connected layer maps the dimensionality of the input vectors to the dimension of the number of labels, and the Sigmoid activation function calculates the probability that the tech text belongs to each label. Then: 22y^=softmax(WFM(E)+b)

Where y^ is the classification result predicted by the model, indicating the probability value that the current technology text belongs to each label, and a threshold is set, if the probability value predicted by the current label is greater than the threshold then the label is considered to be the classification result of the text. W_F is the parameter matrix that maps the text vector to the output space, and b is the deviation term. Finally, the cross-entropy loss function is used as the loss function and the model is trained by minimizing the value of the loss function. Then: 23Lloss=−∑i=1N∑j=1l(yijln(y^ij)+(1−yij)ln(1−y^ij)) where N is the total number of texts, l is the number of labels, and y_ij is the true value of the i th tech text corresponding to the j th label.

How to embed text vectors and what method to train word vectors are the first issues to be considered in this study, in order to verify the impact of different word embeddings on this experiment, this study selects four commonly used word embedding models in NLP, Word2Vec, ELMo, GloVe, and BERT, which are capable of generating word vectors and also have their own characteristics. Word2Vec is a kind of shallow neural network, GloVe is a word vector generation method based on global matrix decomposition, and the advantage of BERT is that it is able to learn bi-directional relationships between words. This study designs a comparison experiment to verify the effect of different word embeddings on this experiment, and the results of the comparison experiment are shown in Table 1.

As can be seen from the table, Word2Vec’s word vector training approach has a poor classification effect, with the training and testing accuracy ranging from 92.64% to 93.98% on the two datasets, in which the reason may be that CBOW and Skip-gram have a weak ability to process long texts, thus resulting in a poorer classification effect. ELMo and GloVe word vectors have moderate performance, and both of them have a multilabel short text classification accuracy are between 94% and 95%, this is because ELMo will carry out basic pre-training for multi-labeled short text, and its core idea is closer to BERT, while GloVe mainly utilizes co-occurrence matrix to learn the correlation between multi-labeled short text, which helps to improve its classification accuracy for multi-labeled short text. The best performance of the BERT-based model is over 96% on both datasets, which is due to the fact that BERT consists of pre-training and fine-tuning phases. After word vector extraction of multi-labeled short texts in the pre-training phase, labeled multi-labeled short text data are used in the fine-tuning phase to adjust the parameters of the BERT model to achieve the best performance on the multi-labeled short text categorization task. Performance.

The performance of CNN is very much determined by the size of the convolutional kernel, considering the effect of different convolutional kernels on this model, three convolutional layers are designed to obtain better performance according to the experimental ideas of existing research. The performance of [1,2,3], [2,3,4], [3,4,5], [4,5,6], [5,6,7] convolutional kernel sizes are designed on the combination of convolutional kernels on this model. Holding other parameters constant, Table 2 shows the experiments performed on THUCNews and EduData data. In the THUCNews dataset, the best performance is achieved when using a convolutional kernel size of [2,3,4], with classification accuracy of more than 98% in all cases. This shows that the model is able to capture important features of multi-labeled short text data using a small convolutional kernel size. On the other hand, the best performance is achieved using larger convolutional kernel sizes [5,6,7] in the homemade EduData dataset, which has a classification accuracy of greater than 98% for multi-labeled short text. This may indicate that the features in the homemade EduData dataset are more dispersed in the text and require a larger convolutional kernel to capture them efficiently. Although the hyperparameter tuning process is usually time-consuming and computationally expensive, it can significantly improve the model accuracy.

The EduData dataset is used to train the multi-label short text classification model that was created in this paper. During the training phase of the model, the Adam optimizer is used to train the model to update the weights, and 50 rounds of iterative training are set up, with 1000 iterations in each round, and the number of training samples in each batch is 24. After every 100 iterations during the training process, the loss value and Macro-P of the model are recorded locally. In order to reduce the risk of overfitting the model during the training process, the training is terminated early when the loss value exceeds 5000 iterations and does not decrease. At the end of training, the loss values and Macro-P curves of the multi-label short text classification model during training are plotted as shown in Fig. 4 Where the left axis is the Macro-P training curve and the right axis is the loss value training curve.

Analysis of the loss value change curves shown in the figure shows that in the initial stage of model training, the loss value curve oscillates sharply and decreases dramatically. With the increase in the number of training times, the optimal value of the model parameter region, the fluctuation amplitude of the loss value gradually decreases, the loss value gradually tends to stabilize, and the model training is ended at 5*10⁵ steps, at which time the loss value of the model is about 0.085. From the change of the Marco-P curve of the model, its trend is opposite to the loss value, and it oscillates sharply and is in the stage of rapid increase before 3*10³ steps. After this, it gradually converges and the overall Marco-P stabilizes around 96.05% after 5*10⁵ steps. The change curves of the model training loss and Marco-P can be used to understand the specific parameters of the model convergence, and then better ensure the accuracy of the model in the classification of multi-label short texts.

Figure 4.

Change curve of Loss and Macro-P value

In order to verify the effectiveness of the multi-label short text classification model based on BERT-BiLSTM-CNN model proposed in this paper, the model in this paper is compared and experimented with TextCNN, BERT, RoBERTa, MacBERT, ERNIE, ERNIE-CNN, CRC-MHA. The models are trained using uniform hyperparameters in the experiments, and Table 3 shows the results of the comparison experiments of different models on different datasets.

In the classification experiments on the THCNEWS dataset, the BERT-BiLSTM-CNN model proposed in this paper achieves the best results in terms of Marco-P, Marco-R, and Macro-F1 values for multi-labeled short text classification. Compared to BERT and its improved models RoBERTa and MacBERT, the Macro-F1 values have improved by 5.05%, 3.74%, and 4.21%, respectively, and all of them have achieved a substantial lead. Among them, the difference in the performance of BERT compared to RoBERTa and MacBERT on the THCNEWS dataset is not significant, and the RoBERTa model even performs less well than the BERT model, which is presumed to be related to the difference in its pre-training process. Compared to the ERNIE and ERNIE-CNN models, the Macro-F1 values improved by 3.62% and 2.35%, respectively, which is a more significant improvement than the effect of the ERNIE model. The Macro-F1 index of ERNIE-CNN model is improved by 1.1% compared with ERNIE model, which indicates that there is still more hidden information in the output feature vector of the pre-trained model, and the effect of using CNN network for extraction has been significantly improved. Compared with the CRC-MFA model, the Macro-F1 value is elevated by 1.55%, and the classification effect is also improved to a certain extent.The CRC-MFA model adopts a multi-dimensional approach to feature fusion, but there are semantic and information density differences between news headlines and keywords in the dataset, and direct splicing may cause semantic confusion. The BERT-BiLSTM-CNN model proposed in this paper adopts a multi-field attention fusion, where the title and keywords of the multi-labeled short text are separately feature extracted and then feature fused, which can better retain the feature information, and therefore better classification results can be achieved.

In addition, in the classification experiments on the EduData dataset, this paper’s model improves the Macro-F1 value by 3.22%, 2.89%, 2.67%, 2.23%, and 1.48% compared to the BERT, RoBERTa, MacBERT, ERNIE, and ERNIE-CNN models, respectively, which all achieve a substantial lead.

Compared with the CRC-MHA model, the Macro-F1 value only improved by 0.94%, which is not a significant improvement. The reason may lie in the style of the dataset, the EduData dataset contains a large number of fields of online education related event text, and does not reflect the advantages of multi-field feature fusion, but the final classification effect is similar. It shows that the effect of this model in the classification of multi-label short text with multiple fields is similar to the mainstream model, and it has good generalization ability.

In order to verify the time performance of the BERT-BiLSTM-CNN model based on BERT-BiLSTM proposed in this paper, the comparison model in Section 4.2.2 is still chosen to compare the training time of different models on different datasets for experiments and to analyze the time complexity. The experimental results that are related are displayed in Fig. 5.

Figure 5.

Model time consumption contrast

On both THCNEWS and EduData datasets, the training duration of TextCNN is within 2 hours. The time consumption of TextCNN on THCNEWS dataset is 0.41 hours lower than that of BERT, while the time consumption of TextCNN on EduData dataset is 0.51 hours lower than that of BERT. TextCNN is a convolutional neural network based on convolutional kernels of different sizes, which is able to reduce the number of parameters of the model through the parameter-sharing mechanism, and can perform data BERT is based on the Transformer model, which is characterized by multi-head attention mechanism and hierarchical stacking structure, and has more parameters than traditional deep learning models, thus its training time is higher than that of TextCNN model. Overall, the training time of the multi-label short text classification model designed in this paper is significantly higher than the comparison model, which is due to the fact that the model in this paper adopts the BERT model for short text representation, in which the embedded Transformer structure will significantly consume time, and the BiLSTM model utilizes the long and short-term memory network to model the multi-label short text in both forward and backward order, but due to the fact that long and short-term memory network is serial in nature, it can only be processed serially in both directions, which will also increase the training time to some extent.

Although the temporal performance of the multi-label short text classification model based on BERT-BiLSTM-CNN model proposed in this paper is worse than that of the traditional deep learning model, it achieves a better classification effect than the traditional deep learning model within the acceptable training time, so it can also be better applied to the classification of multi-label short texts in online education, and provides reliable We also propose that the model can be better applied to classify multi-label short texts in online education, providing reliable technical support for improving the quality of online education.