Research on the Application of Artificial Intelligence Technology in Guzheng Music Cultural Inheritance

China is a multi-ethnic country with a long history and deep cultural heritage. In the field of music, there are different styles of ethnic musical instruments which are carriers of ethnic music culture and need to be protected and inherited. Most of the listeners are not familiar with ethnic musical instruments, the use of computers and artificial intelligence means of detecting ethnic musical instruments, can make people hear ethnic music at the same time to understand the instrument played, in the fun of learning to understand and love the ethnic musical instruments, at present, this is a more valuable and meaningful research direction.

For example, the founder of Yi Guzheng proposed a brand new teaching method of artificial intelligence-assisted guzheng construction, combining artificial intelligence with traditional guzheng teaching. Yi Guzheng Intelligent Music Classroom, hand in hand with domestic experts in many fields such as guzheng education, artificial intelligence, audio recognition, etc., combines traditional Chinese art with modern technology, and combines traditional guzheng culture with the developmental characteristics of modern children.

Guzheng is an ancient Chinese plucked instrument with a wide range of music and beautiful sound. Due to its flexibility and versatility in playing, the guzheng was already popular in Qin during the Warring States period, and in the long history of music, the guzheng has developed a very Chinese cultural heritage. The zheng is rich in techniques, which can be pressed, played, monophonic, and polyphonic, and has a relatively large dynamic range of volume among plucked instruments. With the introduction and popularization of western musical instruments, the performance technique of guzheng has also been developed, and many composers and performers have fused western musical elements on the basis of traditional techniques, constantly reforming the performance technique of guzheng to promote the development of guzheng.

Not only do they use the traditional playing techniques of the guzheng that have been developed over a long period of time, such as the right hand playing techniques: single-tone playing fingerings (chopping, resting, wiping, picking, etc.), double-tone playing fingerings (double resting, double chopping, double picking, etc.), continuous-tone playing fingerings (flower finger), and sustained-tone playing fingerings (finger wheeling, finger rocking). As well as the left hand techniques (press, vibrate, knead, note, chant, point, slide). The traditional techniques of the guzheng still dominate, and are the essence of every player. There are also a number of modern techniques used in the guzheng. With the continuous influx and integration of western musical elements, the guzheng not only integrates the playing techniques of other instruments, but also frees itself from the left and right-handed activities in the traditional way of playing, so that the sound and acoustic performance of the guzheng is more rich and diversified. In the design of the music, the bamboo of the yangqin and the bow of the cello are used to combine the guzheng’s performance, as well as new techniques such as tapping the strings of the zheng, the zither box, sweeping, and the left side of the yard, etc. The guzheng’s sound and the sound of the left hand have become richer and more diversified.

There are two main ideas of data processing, one is to convert pitch information into its corresponding MIDI number as input to estimate fingering, and the other is to use the relative position information between the characterizing keys as input, and the paper refers to the processed data as pitch difference (PD) data, which is a representation that contains information about the physical positional intervals of neighboring pitches along with temporal information, which is more conducive to the model’s exploration of the relationship between pitch and fingerings. In this processing method, the left and right hand data are processed separately and trained separately, and the specific representation is as follows: (1)d(t)={ 100n t=1 x(t)−100x(t−1)n |x(t)−x(t−1)|<12,t>1 80sgn[x(t)−x(t−1)] |x(t)−x(t−1)|≥12,t>1$${d^{(t)}} = \left\{ {\begin{array}{l} {100n}&{t = 1} \\ {{x^{(t)}} - 100{x^{(t - 1)}}n}&{\left| {{x^{(t)}} - {x^{(t - 1)}}} \right| < 12,t > 1} \\ {80\operatorname{sgn} \left[ {{x^{(t)}} - {x^{(t - 1)}}} \right]}&{\left| {{x^{(t)}} - {x^{(t - 1)}}} \right| \geq 12,t > 1} \end{array}} \right.$$

Where: x denotes the Midi number corresponding to the pitch of the note. t denotes the sequential index of the note. n indicates the number of notes at the current moment, i.e., the number of notes contained in the polyphony, and n is specified as 0 when the current note is a monophonic note.

This paper adopts the common fingering representation of sheet music, the following “1 finger” that is, “thumb”, “2 fingers” that is, the index finger, and so on. The left and right finger numbers are shown in Figure 1. The sequence labels of the data used in this paper are also represented by five numbers from 1 to 5.

Figure 1.

The left and right finger number

When the pitch rises or falls continuously, finger-turning fingering is often used instead of hand position translation. Although the basic bidirectional long-term short-term memory (BI-LSTM) network can contain remote pitch difference information and the relationship between pitch difference and fingering, it cannot directly reflect the ergonomic constraints between fingers. These observations inspired us to further improve the BI-LSTM network to incorporate a priori knowledge between finger sequences and note contexts. Ergonomic constraints of the hand are important for fingering annotation. To address this issue, we introduce the finger transfer probability matrix W_T into the model, which counts the fingering transfer patterns of single tones in the statistics set to constrain the neighboring fingerings in the BI-LSTM output. Finger shifts are related to the rise and fall of a monophonic sequence, so the fingering shift probability matrices W_T↑ and W_T↓ are defined according to the rise and fall of the input sequence pitch, respectively. Their representations are shown in Eq. (10): (10)WT=12[sgn(d(t))⋅(WT↑−WT↓)−sgn(d(t))⋅(WT↑+WT↓)]$${W_T} = \frac{1}{2}\left[ {\operatorname{sgn} \left( {{d^{(t)}}} \right) \cdot \left( {{W_{T \uparrow }} - {W_{T \downarrow }}} \right) - \operatorname{sgn} \left( {{d^{(t)}}} \right) \cdot \left( {{W_{T \uparrow }} + {W_{T \downarrow }}} \right)} \right]$$

W_T↑ and W_T↓ constrain only the fingering choices of adjacent single tones. where P_ij is the likelihood of transfer from the ith finger to the jth finger. In the model, the parameters of this matrix are trained together with the parameters of the mapping part, which are updated during backpropagation.

Based on the input pitch difference sequence D, the fingering transfer matrix W_T and the output fingering sequence Λ obtained from the BI-LSTM, the output Y is represented as follows: (11)y(t)=WT*y(t−1)+λ(t)$${y^{(t)}} = {W_T}*{y^{(t - 1)}} + {\lambda ^{(t)}}$$ (12)y^(t)=softmaxy(t)$${\hat y^{(t)}} = {\rm{softmax}}\left( {{y^{(t)}}} \right)$$ (13)Y=(y^(1),y^(2),⋯y^(n))$$Y = \left( {{{\hat y}^{(1)}},{{\hat y}^{(2)}}, \cdots {{\hat y}^{(n)}}} \right)$$

where yi(1)=λi(1)=Λ(1,i)$$y_i^{(1)} = \lambda _i^{(1)} = \Lambda (1,i)$$, y^t can also be expressed in the following form: (14)y(t)=(y1(t)+y2(t)+y3(t)⋯+yk(t))T$${y^{(t)}} = {\left( {y_1^{(t)} + y_2^{(t)} + y_3^{(t)} \cdots + y_k^{(t)}} \right)^T}$$

yi(t)$$y_i^{(t)}$$ denotes the probability of finger i appearing at time t.

The finger with the highest probability at time t, i.e., the finger number φ^(t) corresponding to the current pitch during training, is: (15)φ(t)=argmaxi[yi(t)]$${\varphi ^{(t)}} = \arg {\max _i}\left[ {y_i^{(t)}} \right]$$

The finger transfer matrix learned during training is affected by the model structure, data diversity, and number of samples, and can only characterize the weak transfer relationship between adjacent fingers. If the current fingering is not realizable, the transfer probability between fingers is set to 0, otherwise the current output of the model is maintained. In left-handed fingering, there is a 5-finger to 1-finger hand translation within an octave, so this fingering is not subject to prior knowledge. The decision function T is shown in equation (16): (16)T={ 12{sgn[4.5−f(t−1)⋅f(t)]+1} right: −12<d(t)<12and sgn(d(t))⋅(f(t)−f(t−1))<0 −12<d(t)<12and 12{sgn[5.5−f(t−1)⋅f(t)]+1} left: sgn(d(t))⋅(f(t)−f(t−1))>0 other$$T = \left\{ {\begin{array}{c} {\frac{1}{2}\left\{ {\operatorname{sgn} \left[ {4.5 - {f^{(t - 1)}} \cdot {f^{(t)}}} \right] + 1} \right\}}&{right:\begin{array}{c} { - 12 < {d^{(t)}} < 12and} \\ {\operatorname{sgn} \left( {{d^{(t)}}} \right) \cdot \left( {{f^{(t)}} - {f^{(t - 1)}}} \right) < 0} \\ { - 12 < {d^{(t)}} < 12and} \end{array}} \\ {\frac{1}{2}\left\{ {\operatorname{sgn} \left[ {5.5 - {f^{(t - 1)}} \cdot {f^{(t)}}} \right] + 1} \right\}}&{left:\begin{array}{c} {\operatorname{sgn} \left( {{d^{(t)}}} \right) \cdot \left( {{f^{(t)}} - {f^{(t - 1)}}} \right) > 0} \\ {other} \end{array}} \end{array}} \right.$$

When the decision function is equal to 0, there are fewer fingering paths between the two moments. Some of the transfer paths are pruned when the left hand music pitch rises or when the right hand pitch falls.

After adding the path pruning condition, the expression of y(t)˜$$\widetilde {{y^{(t)}}}$$ in the recognition phase is shown in Equation (17): (17)y(t)˜=(T⋅WT)T*y(t−1)+λ(t)$$\widetilde {{y^{(t)}}} = {\left( {T \cdot {W_T}} \right)^T}*{y^{(t - 1)}} + {\lambda ^{\left( t \right)}}$$

At time t, we choose the fingering with the highest probability as the final fingering estimate, and the modeled fingering sequence number φ^(t) for the current pitch is: (18)φ(t)=argmaxi[y2(t)˜]$${\varphi ^{\left( t \right)}} = {\arg \max} _i\left[ {\widetilde {y_2^{\left( t \right)}}} \right]$$

Although score-fingerprint data enhancement cannot be directly applied to data enhancement methods in the field of natural language processing, some of the ideas contained therein are still inspiring, for example, the data enhancement method of synonymous substitution is very common in tasks such as lexical annotation of text sequences or machine translation. Inspired by the fact that synonymous substitution achieves data enhancement by maintaining the semantic equivalence between two words, data enhancement can be achieved by maintaining the equivalence of the mapping relationship between the score data and the fingering for score-fingering data.

In the fingering annotation task, any fingering is feasible unless the note is preceded or followed by other notes, i.e., fingering is essentially a problem of finding an optimal sequence of smooth state transitions. Therefore, guzheng playing can be interpreted as a process of generating a sequence of playing notes from a sequence of states of finger positions, based on which a data augmentation method based on the Hidden Markov Model (HMM) is proposed, and the HMM has also been used directly for fingering estimation tasks before with good results.

The fingering state is the hidden state, and the pitch difference is the observed state to construct the HMM.To build a first-order HMM, for example, based on the score fingering dataset, the author statistically determines the frequency of the initial fingering F(f(1))$$F\left( {{f^{(1)}}} \right)$$, the size of the initial set of fingering, the frequency of the occurrence of the current fingering F(f(t)|f(t−1))$$F\left( {{f^{(t)}}|{f^{(t - 1)}}} \right)$$ in the case of the previous fingering determination, and the frequency of the occurrence of the subsequent fingering f^(t) corresponding to the pitch difference F(PD(t)|F(t))$$F\left( {P{D^{(t)}}|{F^{(t)}}} \right)$$ in the information of the existing scores.Where the parenthesized superscripts are the pitch difference or the fingering’s time index, and the contents of the parentheses indicate its time step. In addition, since chords occur together, each chord is counted as a whole element.

Based on this the three elements of the first order HMM: initial probability π, state transfer matrix A and firing matrix B are constructed. Where initial state π=(F(f1(1)),F(f2(1)),⋯,F(fk(1)))$$\pi = \left( {F\left( {f_1^{(1)}} \right),F\left( {f_2^{(1)}} \right), \cdots ,F\left( {f_k^{(1)}} \right)} \right)$$, which is the probability of the initial state of the fingering. The state transfer probability matrix A=[ai,j]$$A = \left[ {{a_{i,j}}} \right]$$, where a_{i, j} denotes the probability that at any moment, if the fingering state is s_i, the state at the next moment is s_j, i.e., ∑i=2nF(f(t)=sj|f(t−1)=si)$$\sum\limits_{i = 2}^n F \left( {{f^{(t)}} = {s_j}|{f^{(t - 1)}} = {s_i}} \right)$$ in the statistical results. Where n denotes the sequence length. The firing matrix B=[bi,m]$$B = \left[ {{b_{i,m}}} \right]$$, where b_{i, m} denotes the probability that an observation o_m will be acquired at any moment if the state is s_i, i.e. ∑i=1nF(PD=om|f(t)=si)$$\sum\limits_{i = 1}^n F \left( {{\text{PD}} = {o_m}|{f^{(t)}} = {s_i}} \right)$$ in the results.

With sufficient amount of data, statistics F(f(t)|f(t−1),f(t−2))$$F\left( {{f^{(t)}}|{f^{(t - 1)}}} \right.,\left. {{f^{(t - 2)}}} \right)$$, etc. can be used to generate higher order HMMs, linking more fingering relations to generate higher quality fingering sequences. The data enhancement process based on the first-order HMM is as follows.

Assuming that the newly generated initial fingering fa(1)$$f_a^{(1)}$$ obeys the initial state π, e.g., the probability P{fa(1)=2}=F(2(1))$$P\left\{ {f_a^{(1)} = 2} \right\} = F\left( {{2^{(1)}}} \right)$$ that the newly generated initial fingering is 2-fingered, the initial fingering fa(1)$$f_a^{(1)}$$ is: (19)fa(1)=random_(π)$$f_a^{(1)} = random\_(\pi )$$

where random_(π) denotes the generation of values in its sample space based on the probability matrix π. After the initial fingering is obtained, its corresponding pitch difference is: (20)PDa(1)=random_(B(si=fa(1)))$$PD_a^{(1)} = random\_\left( {B\left( {{s_i} = f_a^{(1)}} \right)} \right)$$

The next fingering is: (21)fa(t)=random_(A(si=fa(t−1)))$$f_a^{(t)} = random\_\left( {A\left( {{s_i} = f_a^{(t - 1)}} \right)} \right)$$

The corresponding pitch difference is: (22)PDa(t)=random_(B(si=fa(t)))$$PD_a^{(t)} = random\_\left( {B\left( {{s_i} = f_a^{(t)}} \right)} \right)$$

This generates an augmented sequence of the specified length.

The experiment involved recruiting eight participants from a music program at a university. All eight participants had no relevant learning experience in guzheng and other musical instruments, were between the ages of 22 and 26, and were all right-handed. Before the beginning of the experiment, the purpose of the experiment, the content of the experiment, the inclusion criteria and the potential risks were explained to the participants in detail. The eight experimental participants were randomly divided into two groups:

The experimental group, including 4 participants, was used after the guzheng lesson to perform fingering exercises in the after-session independent practice session.

The control group, including 4 participants, was used to perform fingering practice after the guzheng lesson in a traditional mode, i.e., practicing on their own without any external tutoring, in the after-session independent practice session.

In addition, the experiment invited a guzheng teacher with 13 years of guzheng learning experience and 6 years of guzheng teaching experience as the teacher of basic guzheng knowledge teaching in the pre-test period to train and teach guzheng knowledge points to the 8 experimental participants.

5.2.1

Performance level and fingering mastery dimensions

The experiment calculated and counted the correct rate of 31 indicators for each participant separately to verify whether the performance level and fingering mastery of the practitioners were improved after practice. The percentage of the indicators with less than 60% correct rate of the test before and after the after-school independent practice for both groups of participants is shown in Figure 3. Before the independent practice, the performance level and mastery of the target fingerings were basically the same in both groups, with some participants in the control group having a lower number of indicators below 60% than those in the experimental group. However, after the after-school independent practice, the number of indicators with less than 60% correctness of the four participants in the experimental group decreased significantly, from an average of 51.3% before the practice to an average of about 7.25% after the practice, a decrease of about 85.87%. In contrast, the four participants in the control group decreased from an average of 55.59% before the exercise to 41.78% after the exercise, a decrease of about 24.84%. It is reasonable to speculate that compared to the four participants in the control group, the four participants in the experimental group effectively practiced by improving their fingering performance level and mastery.

Figure 3.

The pre-test and post-test accuracy rate was lower than that of 60%

Further, the correct rates of the indicators with less than 60% correct in the pre-test for both groups of participants were further analyzed. The comparison results are shown in Figure 4, which shows the change in the average correct rate before and after practicing the indicators with less than 60% correct rate in the pre-test of the participants in the two groups. As can be seen from the figure, the four participants in the experimental group showed a significant improvement in the correct rates of the indicators whose original correct rates were lower than 60% after the system-assisted practice, and the average correct rates of the indicators whose original correct rates were lower than 60% reached more than 70% for all four participants after the practice.

Figure 4.

The average change in the accuracy of the accuracy rate

In contrast, the four participants in the control group showed less improvement in the indicators that were less than 60% correct on the pre-test, and the indicators remained less than 60% correct on average after practice. In the analysis of changes in correctness before and after practice for all indicators with less than 60% correct, the four practitioners in the experimental group were able to achieve about 1-2 times the original improvement in correctness after practice, even though the indicators had very low correctness in the pre-test. On the other hand, the four practitioners in the control group had very low correct rates on the pre-test, and their correct rates did not increase significantly or did not increase after practice. Therefore, it is reasonable to speculate that, with the assistance of the system, the four participants in the experimental group, by obtaining the feedback information provided by the system, could keep abreast of the results of their playing behaviors and correct their erroneous cognitions and muscle memories, which led to improved fingering performance and enhanced fingering mastery for effective practice.