The role of musical feature extraction model MFE in multicultural music fusion innovation

The internal structure of the GRU loop body contains two gating functions as well as a hidden state, the reset gate r_i, the update gate z_i and the hidden state h_i.

r_i is used to control the ratio of the hidden state h_t–1 from the previous time step combined with the input x_i from the current time step. It receives h_i–1 and x_i as inputs and computes a set of reset coefficients between 0 and 1 for the features in each dimension: (1) rt=σ(Wr[ht−1,xt])

where W_r is the learnable parameter of the reset gate, [h_i–1,x_i] denotes two vector splices, and σ(·) denotes the Sigmoid function. Then the reset coefficients are multiplied back on h_i–1 to realize a partial reset of the information of the previous time step, and work with x_i to generate a candidate hidden state h˜i for the current time step: (2) h˜t=tanh(Wi[ri*hi−1,xi])

where wh¯ is the learnable parameter for computing h˜t and ^* denotes the element-by-element multiplication z_t determines the ratio of information forgotten and updated in the hidden state. It also computes a set of values between 0 and 1 as update coefficients through x_i and h_t–1: (3) zt=σ(Wz[ ht−1,xt ])

where W_z is the learnable parameter of the update gate. Finally, the hidden state h_t of the current time step is computed by z_t,h_i–1 and h˜i . Partial forgetting of past information is achieved by multiplying (1–z_i) with h_i=1, and partial retention of current information is achieved by multiplying z_i with h˜i . The update of h_i is accomplished by adding the two parts: (4) ht=(1−zt)*ht−1+zt*h˜t

It can be seen that GRU controls the degree of forgetting the past information and updating the current information through r_i and z_i, which ensures the ability to memorize long time-series features. Compared with LSTM, GRU has a simpler structure with one less gating function in the loop body and omits the cell state, which is more advantageous in terms of the number of parameters and computational efficiency.

TDNN is a feed-forward neural network for processing sequential data. Different from the traditional fully connected network, it adopts a neuron structure with time delay, each neuron has a certain width, and the output of the neuron is only related to the input in a certain time range.Each point of TDNN represents a frame feature vector, and the neurons of the TDNN move along the temporal direction in a certain step to realize the extraction of local features. During the moving process, the weights of the neurons are shared, which means that the neurons at each time step use the same weights to process the input data. Such a shared weight mechanism helps to reduce the number of model parameters and improves the generalization ability of the model. As the number of network layers deepens, the temporal range of TDNN layer connections gradually becomes wider, thus abstracting the global feature representation.

The width of the TDNN neurons is the size of the convolution kernel, and the frames within the time span are weighted and summed by the parameters in the convolution kernel during computation. These weights are continuously adjusted to optimize the model performance by training the minimization loss function.

The operations in these network layers are linear, and in order to fit the real data, the model needs to be capable of nonlinear transformations. Therefore, the output of the network layer needs to be nonlinearly transformed using some functions, which are called activation functions. Common activation functions include Sigmoid function, ReLU function, tanh function and so on, as shown in Equation (5), Equation (6) and Equation (7):

These functions have different properties and can be mapped to different ranges of input signals by nonlinear transformations, thus increasing the expressive power and fitting ability of the network.

The structure of the MFE is shown in Fig. 1, for the input FBank features, in addition to the one-dimensional convolutional layer used in the previous sections of this paper, an additional branch of the GRU is added by parallelization, where the dimension of the hidden state h_t is set to 64. Afterwards, the features generated by the two branches are spliced together along the channel direction, and a convolutional layer with a convolution kernel size of 1 fuses the two parts of the features, which then undergoes a BN operation and then using tanh as the activation function.

Figure 1.

The structure of MFE

It should be noted that the time cost required by GRU is high due to the need to iterate time step by time step during the computation process. In the PyTorch framework, in order to improve the parallelization of the computation, GRU computes a batch of data at the same time step simultaneously, and therefore uses a different input data format than the one-dimensional convolutional layer. One-dimensional convolutional layers require F_in ∈ ℝ^B×C×T for the input data, i.e., the first dimension of the feature is the batch size, the second dimension is the number of channels, and the third dimension is the timestep length. Whereas the input data format required by GRU is F_in ∈ ℝ^T×B×C, i.e. the first dimension is the timing length, the second dimension is the batch size and the third dimension is the number of channels. So after getting the FBank features with data format F_FBamk ∈ ℝ^B×C×T, the dimensions of the features need to be exchanged before inputting to GRU. Accordingly, the output data format of the GRU is F_out ∈ ℝ^T×B×(H×D), H represents the dimension of the hidden state, D is related to the direction of the GRU and has a value of 2 when using bi-directional GRUs and 1 when uni-directional GRUs are used.After obtaining the output of the GRU, the dimensions of the features are exchanged back to the original format, i.e., Fout′∈ℝB×(H×D)×T in order to facilitate the transfer in the subsequent network.

The Transformer model consists of four parts, which are the input, output, encoder, and decoder parts.

The input part of the Transformer model consists of a data embedding layer and a positional coding layer, and the output part of the model consists of a linear layer and a Softmax layer.The role of the data embedding layer is that the model needs to transform the input textual data as well as numerical data into vector representation.The purpose of the location coding layer is to include location information in the input data. The role of the linear layer is to deal with the linear changes in the previous step to achieve the effect of transforming the dimensions.The role of the Softmax layer is to scale the numbers of the vectors in the last dimension into the value domain of 0-1, and make their sum to 1, and according to the probability of the proportion at this point in time to get the most suitable output.

The biggest problem with Transformer is that there is no way to model sequences that exceed its maximum length, if the sequence exceeds that length it needs to be intercepted and then the intercepted fragments are re-encoded, the above method is feasible but it leads to the problem of contextual fragmentation, which means that each intercepted fragment is modeled individually without sufficient contextual information between each other. Transformer-XL, which is based on Transformer, suggests implementing a fragmentation recursion mechanism and replacing absolute position encoding with relative position encoding to address the issue of context fragmentation and long-term dependency.

Transformer-XL is also input with fixed-length segments during training as Transformer-similarly, the difference is that the state of the previous segment of Transformer-XL is cached, and then the hidden layer state of the previous time slice is reused when calculating the current segment. Because the features of the previous segment are reused in the current segment, this gives Transformer-XL the ability to model long-term dependencies. The details of the fragment recursion can be seen according to equation (8):

Where h denotes the hidden state of the model, τ denotes a fragment, SG denotes that no gradient needs to be computed, and [A ∘ B] denotes the splice of two vectors. Starting from the second fragment, the hidden state of each fragment is spliced with that of the previous fragment to complete the recursion between fragments and achieve the purpose of obtaining long-term dependency.

Since different segments in Transformer have the same position encoding for the same position, the model has no way to correctly distinguish the position information of different segments, so Transformer-XL proposes to use relative position encoding instead of absolute position encoding. The specific operation is to consider only the relative position relationship between the Query vector and the Key vector when calculating the attention score, and add this relative position relationship, to the calculation of attention at each level. The formula is shown in (9):

where E_x denotes the embedding of the token input, W_k,E corresponds to the weight matrix of the embedding, W_k,R corresponds to the weight matrix of the positional encoding, R_i–j denotes the relative positional encoding, and u^T and v^T are both learnable vectors.

Considering the unique “cloze in the blank” mechanism of the BERT model, this paper applies this training method to Transformer-XL, so that the improved Transformer-XL can clearly see the past information and future information, and the predicted results are fully combined with the “context”.The Transformer-XL model’s Mask mechanism masks all information after the predicted position, resulting in the model only focusing on past information. In view of this, this paper improves the Mask mechanism so that it only masks the next moment, which is what is about to be predicted, during training, and does not mask information from the next position onwards.

Since Transformer-XL combines Transformer and RNN model ideas and proposes a fragment recursion mechanism, which recurses the features of the previous fragment to the next fragment, the model’s past information includes not only all the information before the current token prediction moment, but also the information recursed from the previous fragment, which is also the information of the MEMORY token. Therefore, the model can make better use of past information when predicting, avoiding the problem of generating music with poor wholeness and fragmentation between bars.

In modeling music, the main framework of the model is based on Transformer-XL, after which an improved Mask mechanism is added to it, which gives the model the ability to see bidirectional information. In the training process, in this paper, the complete music is divided into multiple segments, and the layer n – 1 hidden features of the τ st segment in the music can be expressed as: (10) h˜τn−1=[ SG(hτ−1n−1)∘hτn−1 ]

Among them, the features of the τ st fragment utilize the memory features of the previous fragment, and although it does not participate in BP computation, it is also spliced in dimension, which is equivalent to the current fragment reusing the information of the previous fragment, which increases the dependency of the model, expands the length of the sequence processed by the model, and effectively solves the fragmentation problem.

When the model is being trained, the information of each fragment of the input music is represented by the Query vector, Key vector and Value vector: (11) qτn,kτn,ντn=hτn−1WqT,h˜τn−1WkT,h˜τn−1WνT

In the recursive approach to self-attention computation, the query vector Query is represented only by its own features without incorporating the recursive mechanism, and both the Key and Value vectors combine the features of the previous fragment, W being the weights to be learned by the model.

Since the improved Transformer-XL model also uses the relative position encoding approach, the layer n hidden features of the τ nd fragment in the model can be computed from the layer n – 1 hidden features: (12) Aτ,i,jn=qτ,inTkτ,jn+qτ,inTWk,RnRi−j+uTkτ,j+νTWk,RnRi−j $$A_{\tau ,i,j}^n = q_{\tau ,i}^{n\>T}k_{\tau ,j}^n + q_{\tau ,i}^{n\>T}W_{k,R}^n{R_{i - j}} + {u^T}{k_{\tau ,j}} + {\nu ^T}W_{k,R}^n{R_{i - j}}$$ (13) ατ,in=Improved−Mask−Softmax(Aτ,in)ντ,in (14) ατn=[ατ,1n∘ατ,2n∘…∘ατ,mn]TWn

Where ατ,t* is the Query vector, the Key vector and the Value vector are computed with self-attention to obtain the attention features of the ind head. After that it is subjected to a series of computations of normalization, residual join and position feedforward network to get the potential feature representation of each position in each segment: (15) zτn=LayerNorm(Linear(ατn)+hτn−1) (16) hτn=max(0,zτnW1n+b1n)W2n+b2n

Where hrn denotes the layer n hidden feature of the τ nd fragment in the score, and the output hr′ can be calculated by Softmax function to predict the probability of notes under each moment: (17) Pθ=exp(hτ,N*)∑Nexp(hτ,N*)

Figure 2 shows the specific process of music generation, firstly, this paper converts the music into audio in MIDI format, after that, using the MFE method, the features of the music are transformed into specific events, and then these events are sent to the model for training, and finally the music with the characteristics of multicultural fusion can be obtained.

Figure 2.

The process of music generation

Since there are no suitable automated metrics to evaluate the music generation task, based on the knowledge in the music domain, the experiment was manually evaluated in four aspects: rhythm, melody, integrity, and singability. Ten music experts were invited to evaluate the multicultural fusion music generated by different models. All models generated one piece of music combined into one round of scoring samples, and a total of 60 rounds of scoring were conducted. All the experts ranked the generated music according to the proposed manual evaluation indexes, with scores ranging from 6 to 1. After obtaining all the scoring data, the scores obtained by each model were statistically analyzed.

Figure 3 shows the results of the melody generation evaluation for the different models. The Improved Transformer-XL model achieved the best results on all indicators, verifying the effectiveness of the model on the music generation task, with an overall average score of 3.76 for generating music, which is 6.21% to 94.82% higher than the scores of other models. Meanwhile, the improved Transformer-XL model outperforms the original Transformer-XL model in several metrics, indicating that the improved Mask mechanism can indeed enhance the effectiveness of multicultural fusion music generation.The RNN-based music generation model significantly outperforms the GAN-based model, indicating that the RNN model is more suitable for the problem of sequence generation.

Figure 3.

Evaluation results of different model melodic melody

The experiments were further analyzed to improve the performance of chord progression in the Transformer-XL music generation model, defining chord accuracy to assess whether the chords of the generated melody match the input chord sequence:

where P is the number of phrases, just the ind chord detected in the generated melody, and y˜i is the ith corresponding chord in the given chord progression.

Figure 4(a) shows the results of analyzing the chords of the generated music, and the average chord accuracy of the melody generated by the improved Transformer-XL model is 87.24%. Figures 4(b)∽(e) show the effect of the chord accuracy of the generated melodies on the metrics of rhythm, melody, completeness, and singability, respectively, verifying that the quality of the melody generation improves significantly as the chord accuracy increases.

Figure 4.

The analysis of the chords of generation music

Pauses are common in music, i.e. a period of time without the appearance of any notes. On the one hand, pauses can be used to divide the music into sections, dividing the long and short sections of the music, making the music more rhythmic, and on the other hand, they also provide the listener and the singer with rest intervals and time to change breath. A good-sounding piece of music must maintain a good dynamic balance between its musical activities and pauses.Figure 5 shows the distribution of different phrase lengths in both the generated music and the real music. In this case, each phrase is divided by pauses, and the maximum and minimum percentages of different degrees of phrases of generated music and real music are between 0 and 24% and 0 and 14%, respectively. It can be seen that the generated music and the real music have similar distributions in terms of phrase lengths, which indicates that the improved Transformer-XL model can learn the pauses of the phrases in multicultural fusion music very well, and can maintain good structural properties.

Figure 5.

The distribution of music in different lengths

The online audition effect scoring platform was developed using front-end and back-end separation, with the front-end developed in React and the back-end in Java.After the platform was developed, 15 test arrangements were placed on the platform, four of which were from the demonstration audio set, notated as D1∽D4.Four songs are from the midishow website, which is a professional music sharing and communication platform where users can upload their own music creations, labeled as D5∽D8. 7 songs are generated by improving the Transformer-XL model, labeled as D9∽D15. The duration of each arrangement is interrupted for 30 seconds to avoid listening fatigue of the testers.The testers were given the opportunity to listen to 15 arrangements online and rate them based on their subjective listening experience. The testers invited for the online audition evaluation are music lovers, and the scoring system is a five-grade scale.

The scores of each piano piece were counted and the average score was calculated, and the rankings of the online audition evaluation are shown in Fig. 6. The multicultural fusion arrangement D10 generated by the improved Transformer-XL model enters the top three in the ranking, with a score of 4.23. It shows that the testers cannot completely distinguish between the human work piece and the automatically-generated arrangement of this paper, and also indicates that the automatically-generated arrangement of this paper meets the testers’ appreciation requirements. In addition, the scores of D11, D13 and D15 of this paper are below 3.5 and ranked at the back of the list, which indicates that there is a difference in the quality of the automatically generated arrangements, and there is room for further optimization of the network model of automatic composing.

Figure 6.

Rankings of online audition scores

The offline performance evaluation invited professionals who have extensive experience in music performance. The professionals specified five indicators for this evaluation, namely “melody, texture, harmony, tension, and aesthetics”, each of which is worth 100 points, and then performed the selected 15 arrangements live and recorded the scores of the five indicators for each arrangement.

For the weight size of each indicator, this paper uses the entropy weight method to calculate the specific weight. According to the information provided by each indicator, the entropy method utilizes the information entropy to determine the size of the role of the indicator in the comprehensive evaluation, and then obtains a specific weight value.

The final results of the offline performance evaluation are shown in Figure 7. The demonstration audio and the MS-created arrangement works ranked in the top three, with evaluation scores above 92. Three of the automatically generated arrangements in this paper rank in front of the manually created arrangements, and the difference in D7 scores between them and the MS-created arrangements is small, all of which are above 88. It shows that the automatically generated arrangements with the improved Transformer-XL model in this paper can reach the level of general manual creation, but the score gap is larger compared to the works with demonstration audio. The lower-scoring arrangement D15 is also an automatically generated arrangement in this paper, which indicates that there is a difference in the quality of automatically generated multicultural fusion arrangements, and the network model for automatic composition needs to be further optimized.

Figure 7.

The final result of the offline performance evaluation