Combined Application of Speech Recognition and Natural Language Processing Technologies in the Electric Power Industry

The out-of-set word speech recognition system proposed in this chapter is shown in Figure 1. The out-of-set word speech recognition system is composed of an acoustic model, a language model and an out-of-set word correction model, and the speech synthesis model is only used in the training process. The acoustic model, language model and out-of-set word correction model are all neural network models based on Transformer.

Figure 1.

Speech recognition system for extra-set words

In the training process, the speech synthesis model uses the out-of-set word text to generate synthetic audio and realize audio data augmentation, the augmented synthetic audio and the real audio go through the acoustic model respectively to get their respective recognition results, and then finally the out-of-set word correction model is trained by using the two types of recognition results and the corresponding real values. In the decoding process, the acoustic model combines with the language model to predict the recognition results. After that, the out-of-set word correction model corrects the out-of-set word error part of the recognition results to realize out-of-set word speech recognition.

The structure of the acoustic model contains an encoder, a decoder and a Softmax prediction module. The encoder/encoder is composed of a positional embedding layer, a stack of several encoding/decoding blocks. Each encoding block is composed of two sub-layers, a multi-head self-attention layer and a feed-forward neural network layer. The decoding block is composed of three sublayers: a multi-head self-attention layer, an implicit multi-head self-attention layer, and a feedforward neural network. Residual connections are used between blocks, Layer Normalization ensures the stability of feature distribution, and Dropout technique mitigates the gradient vanishing problem of deep networks. Among them, the empirical value of 12 is selected for encoding block layer M and 6 for decoding block layer N.

Fbank audio feature x is used as the input to the encoder, and since the self-attention mechanism ignores the positional information of the temporal features, it is necessary to embed the positional information of the Fbank features. After that, x is sequentially encoded layer by layer through stacked coding blocks to extract the semantic information, and the output of the coding block at the top layer is used as the coded feature of the audio, and the Encoder function is shown in Equation (1): (1) Henc=Encoder(X)

The Decoder combines the encoded features and the previous prediction sequence to iteratively compute the decoded features for the current time step, Decoder function as in equation (2): (2) Hdec=Decoder(Henc)

The decoder starts predicting from the character ‘<start>’ and iteratively predicts the sequence of characters until the end of the predicted character ‘<end>’. In a time step, decoded feature H^dec is mapped into potential feature after linear mapping layer, the potential feature dimension is the same as the dictionary dimension of the acoustic model, the potential feature goes through the activation function to get the probability vector, the maximum value dimension is used as the state code to complete the character prediction for the current time step: (3) y′t=MaxValueIndex(Softmax(Linear(Hdec))) where y′t denotes the predicted character for the current time step and Eq. (3) denotes the prediction for one time step. Attention-based Seq2Seq loss: (4) lossseq2seq=Seq2Seq(X,Y)

The coded features are mapped, through a linear mapper and Softmax function, into probability vectors with vector dimensions equal to the dictionary length of the acoustic model, and finally a CTC loss calculation is performed: (5) lossctc=CTC(X,Y)

The loss function of the optimized acoustic model is a hybrid CTC and Seq2Seq loss, which is a weighted sum of both CTC and Seq2Seq losses: (6) loss=αlossctc+(1−α)lossseq2seq where α takes the value of [0,1].

In the decoding inference process, the probability vectors obtained from the two decoding methods are weighted to determine the decoding result. In order to obtain as accurate a recognition result as possible, the bundle search method is used. The bundle search method is a temporal prediction N-Best decoding strategy that is used only in the inference process. Bundle search retains the N-Best result in the prediction of each time step as an input or probability calculation for the next time step, thus achieving a wider search surface and improving the recognition results despite sacrificing some time and space.

The synthetic speech model and language model are not the main modules of the method proposed in this paper and will not be repeated, the language model in this paper is an end-to-end Chinese language model based on Transformer.

The Transformer-based out-of-set word correction model, shown in Fig. 2, is composed of an encoder, a decoder, and a Softmax prediction module. The encoder/decoder has a number of layers of encoding/decoding blocks stacked together, with residual connections between blocks. The structure of the encoder/decoder is the same as the structure of the acoustic model. The number of block layers of the encoder is taken as empirical value M = 6 and the number of block layers of the decoder is taken as empirical value N = 6.

Figure 2.

Transformer based OOV-SC model

Unlike the acoustic model, the location information embedding method of the out-of-set word correction model is the OneHot coding method. Sin-function based coding approach can describe the relative position information well. In the process of speech recognition, the information provided by the nearby context is more useful for speech recognition than the distant context. However, the absolute coding approach based on OneHot can describe the absolute position information well and is simple to implement, the recognition of out-of-set word error regions requires more semantic association based on the overall context.

The out-of-set word correction model works in a similar way to machine translation, where the input to the encoder is the recognition result of the acoustic model, assuming that the recognition result of the acoustic model is A = [a₀,a₁,…,a_L], and a_i denotes the character in the sequence of the prediction result of the acoustic model. The recognition result A is used as input to the out-of-set word correction model, which needs to undergo label smoothing and positional information embedding. Label smoothing is a common method to prevent overfitting and is commonly used as a method for natural language processing and speech recognition. The encoder performs feature abstraction on the incoming text to get the encoded features: (7) Fenc=Encoder(A)

The decoder iteratively computes the decoded features for each time step using the encoded feature F^enc and the previously predicted text sequence C^t−1: (8) Ftdec=Decoder(Fenc,Ct−1) where C^t−1 = [c₀,c₁, …, c_t−1], c_i denote the predicted characters.

In each time step, the decoded features are linearly mapped to potential spatial features, which have the same data dimensions as the out-of-set word dictionary size, and the potential spatial features are passed through a Softmax function to obtain a probability vector P^t, from which state codes are obtained, one by one, and mapped to characters in the dictionary: (9) Pt=Softmax(Linear(Ftdec)) (10) ct=MaxValueIndex(Pt)

The optimization of the training model is composed of the Seq2Seq loss of the decoder weighted by the alignment loss: (11) Loss=βlossseq2seq+(1−β)lossalign where loss_seq2seq denotes Seq2Seq loss and loss_atign denotes alignment loss. Weight β ranges from [0,1].

A complete speech communication contains three main factors: the subject, the discourse entity and the context, and the close combination of the three can completely express the meaning of a piece of speech, in which the contextual information determines the current speech situation and background, and at the same time has a certain constraint on the semantic expression. A text error will inevitably lead to the text of the sentence semantics does not make sense, so to find the error in the sentence, that is, to find the sentence does not meet the text semantics of the word, which leads to the text semantics does not make sense of the word can be categorized into the following three cases: does not meet the current syntax of the sentence, semantics or context. In this paper, we temporarily disregard the error checking at the syntactic level, and realize text error checking from both semantic and contextual aspects. In most existing natural language processing algorithms, the contextual knowledge of a word is usually represented by one or two words around the target word, or by some main words in the chapter and paragraph where the target word is located, called context words. In this paper, we take the several words adjacent to the words before and after the current error-checked word as the context words to characterize the textual context knowledge, and characterize the semantic knowledge with the core words and their fixed collocations in the text, which together provide the basis for the error-checking. As shown in Figure 3 is the overall framework of text error checking after speech recognition.

Figure 3.

Text error checking process after speech recognition

From Figure 3, we can get the process of text error checking after speech recognition: firstly, we use the training corpus to construct the n-gram model and the core word collocation thesaurus, then we preprocess the text after speech recognition, and firstly, we use the constructed n-gram model to calculate the contextual harmony of the words in the preprocessed text in order to realize the initial error checking, and then we use the core word collocation thesaurus to calculate the collocational aggregation degree of the words after the initial error checking. Then we use the core word collocation thesaurus to calculate the word collocation aggregation degree of the words after the initial error checking to carry out the secondary error checking, and finally mark the wrong words with the results of secondary error checking.

The main idea of error checking is: take the prediction ability of n-gram model on neighboring words or words as the basis of error checking, for the limitation that n-gram model can only use historical words (context above) to check errors, firstly, adopt a weighted allocation method based on n-gram model to calculate the context harmony of words in the text after speech recognition, and preset a threshold for the value of context harmony, and words and characters below the threshold will be marked as true word errors in the text. errors, realize the initial error checking of true word errors and loose string errors in the text; then for the error checking defects of true word errors in the n-gram model, based on the core word collocation thesaurus, calculate the language collocation aggregation degree of phrases and core words in the text after the initial error checking in turn, and preset a threshold for the language collocation aggregation degree, and the word errors below the threshold value, realize the second error checking of true word errors in the text after the speech recognition, and at the same time realize the correction of the error checking results of the n-gram model. -gram model error checking result correction, and finally take the second error checking labeling result as the final error checking output.

There are two main ideas for error checking based on n-gram language model: 1)

Calculate the probability of the current word appearing under the premise of historical word occurrence, if the n-gram probability value of the current word satisfies the pre-set threshold β₁, then the phrase is correct, if it is smaller than the pre-set threshold, then the phrase is wrong, which can be calculated word-by-word or word-by-word, and the method can specifically locate the position of the wrong word.

Comparing the above two kinds of error checking ideas, this paper takes the first idea as the main idea of error checking, by calculating the contextual harmony p(x_i) of each word in the text, if p(x_i) is greater than the threshold β₁, it means that x_i is in line with the current contextual context x_i is correct, otherwise x_i is wrong, if x_i is a phrase, then it will be labeled R, and if it is a single-word loose string, then it will be labeled N. The specific flow of error checking is shown in Fig. 4.

Figure 4.

Error checking flow of weighted distribution method based on n-gram model

The experiments in this chapter are supplemented by the use of the Tsinghua University’s open Chinese dataset THCHS-30 in addition to the data from Hill Shell’s open source AISHELL-1 Chinese Mandarin speech data to do the experiments, which is used in this study as the training set to complete the model training, and the test set to complete the test experiments. In the experiments, Spectrogram features, Fbank features and MFCC features are extracted as inputs to train the end-to-end speech recognition model of this paper’s model, respectively.

After completing the training, the experimental results of the test set are used to compare the word error rate (CER) of these three different input features, so as to prove which of the three speech features, Fbank features, Spectrogram features and MFCC features extracted from the speech signal is more suitable to be used as the input features of this paper’s model, and the experimental results are shown in Table 1.

For the two different datasets used in the experiments, as can be seen from the data in Table 1, both datasets have low word error rates (CER) of 15.72% and 15.54%, respectively, when the Spectrogram feature is used as an input feature, while the second highest word error rates (CER) are obtained for both datasets when the Fbank is used as an input feature, with results of 17.28% and 16.77%, while when MFCC is used as an input feature, both datasets have the worst results for word error rate (CER). 18.68% and 19.30%, respectively. This is due to the fact that the MFCC feature requires more manual intervention, which can lead to loss of information and greatly reduce the performance of the system. Fbank does preserve the original characteristics of the audio signal as much as possible, but it can also lead to the presence of redundant information, which can affect the model’s ability to not capture real usable audio information when the dataset size is insufficient.

By conducting a comparison between these two sets of experiments, it can thus be concluded that the Spectrogram feature of the speech signal is more suitable as an input feature for the model in this paper, which reduces the word error rate (CER%) of the speech recognition model, which in turn is able to improve the performance of the system.

A comparison experiment is done between the model proposed in this paper and the baseline models CNN-BiLSTM and CNN-BiGRU models in the AISHELL-1 dataset, and we choose the four metrics of Accurary, Precision, Recall, and F1 as the performance metrics of this experiment to determine whether the performance of the model incorporating the attentional mechanism has been improved or not. The results of the experiment are shown in Fig. 5.

Figure 5.

Performance comparison of different models (%)

As shown in Fig. 5, it indicates the best performance of the model in this paper.

At present, most of the open-source speech corpora are based on mass news and other corpora, but this type of corpus basically does not contain the content of power dispatching instructions and some specialized vocabularies such as equipment and operation, which do not meet the application requirements of power dispatching scenarios. Therefore, according to the corpus characteristics of power dispatching instructions, a design scheme for the voice data set in the field of power dispatching is proposed, a 360-article power dispatching industry corpus is constructed, a total of 5,000 voice dispatching instructions are recorded, and after the construction of the data set is completed, it is expanded by using the method of speed perturbation.

The content of dispatching instructions of a power supply company in Guangdong Province is selected as a template to construct a text corpus of power dispatching according to the rules and characteristics of its dialogues. The text covers the common equipment name, dispatching, operation and other specialized terms in the power system, and the text corpus information. The terminology of commonly used equipment names in the power system basically contains the names of major equipment mentioned in daily dispatching. The corpus deals with numbers, units, special symbols, etc. in the electric power industry, all of which are described in Chinese characters, e.g., the Chinese character expression for “10KV” is “ten kilovolts”. The scheduling instruction corpus example contains the basic order format of daily scheduling instructions.

In order to verify the speech recognition effect of the proposed model, comparison experiments are conducted on the power speech dataset established in this paper using the more popular models. The experimental results are shown in Table 2. The experimental results on the dataset show that the number of parameters of the method in the paper is 25, which is at a medium level. Its word error rate (WER) is 8.21%, which indicates that the method has the highest recognition accuracy on this dataset. The real-time rate (RTF) is 0.017, indicating that the method has the best real-time performance on this dataset.

By analyzing Table 2, it can be seen that the proposed method in this paper performs very well on the above dataset, not only achieving a low WER and RTF while keeping a low number of parameters, demonstrating its efficiency and practicality in speech recognition tasks, and proving that the algorithm has good generalization performance.

Power scheduling system in addition to the need for speech recognition accuracy to meet the actual scheduling needs, but also need to meet the timeliness, in order to verify the feasibility of the proposed algorithm in the power scheduling system, the next step will be to verify the proposed algorithm by comparing the timeliness of the different voice information processing. As shown in Table 3, the response time of different types of different voice duration system is analyzed by different types of different voice duration information input system, for different voices, it can be seen that the voice response time of the input voice system of different durations increases with the increase of the response time of the voice duration, but the latency is low, and the response time of the three modes of the system is the longest of 9.7 ms for the voice duration of 60 s, which meets the actual use requirements of the dispatching system. The longest response time of the three modes is 9.7 ms at 60 s, which can meet the actual demand of the scheduling system and further illustrates the feasibility of the proposed algorithm.