A study on the mechanics, finger movement, and finger function of computer vision technology in guzheng playing posture recognition

Performing music is a way of communicating beauty to the listener. As a musician, a lifelong goal is to explore ways to convey and express beauty to the audience. With the development of the society, people’s material living standard has been improved, and the level of spiritual civilisation has also been improved, and the appreciation of music is not only about the sound, but also about the overall aesthetic enhancement of the performer [1-4].

Some artists point out that the posture is an important part of zheng performance, and it is inseparable from the complete interpretation of zheng works. Regarding the posture of guzheng performance, it refers to the movements of the body (the traditional form of guzheng performance is mostly sitting) and the hands. For example, the back and forth movement of the body and the up and down, back and forth, up and down movement of the hands [5-6]. The guzheng is mainly played by touching the strings with the fingers, so we mainly talk about some auxiliary playing postures other than finger movements. Some guzheng learners in the study of guzheng performance noted that some excellent performers in the performance of music has a lot of body movements, some students will be imitated, most often seen in the play a section of the wrist in the air shaking two or two circles to take over the next section, or the body shaking two, and the result is always like a non-like, hard to do, the cause of which is to imitate the external movement of the hand, and did not grasp the connotation of it [7-9]. Body language expression in performance The expression of body language in the performance must be combined with my breath, the connotation of the repertoire, that is, the shape of God, so look at a good player, playing music, he gives the feeling of static and dynamic, slow when the action is shown to be very slow, do not procrastinate, fast when fast is also fast with a very good method, not impatient, clean, easy to use, in fact, this move has a corresponding breath in the support, so we are learning! When we learn it, we must not move for the sake of movement, but from the connotation of the musical work and the player’s own breath [10-11].

Multidisciplinary cross-fertilisation has become an inevitable trend in the development of science and technology in today’s era, and the development of music artificial intelligence is in the ascendant, and the issue of AI-enabled intelligent identification and evaluation of instrumental performance techniques spans the cross-cutting fields of science and art [12-13]. Interdisciplinary research has high professional knowledge and technical barriers, how to seek breakthroughs and combinations is the key to solving interdisciplinary research. Guzheng is an ancient, unique, highly representative and rhythmic national musical instrument in China. In recent years, the guzheng has become the folk music with the largest number of learners, and the teacher’s power is difficult to meet the actual demand of the rapid growth. The traditional teaching mode of guzheng mostly adopts the one-to-one or one-to-many face-to-face teaching form, and the students can’t get the real-time feedback from practicing the zither after the lesson, which will easily lead to the development of bad playing habits and thus affecting the subsequent learning of the advanced level, and they will meet the bottleneck of the mastery of the technique at an early stage. Therefore, the demand for intelligent assisted identification and evaluation systems of guzheng playing techniques that are both intelligent and professional has increased [14-15].

Intelligent recognition and evaluation of guzheng playing hand shape and fingering can provide timely problem feedback and improvement advice to zheng practitioners, which can quickly improve the efficiency of practice and thus guarantee the quality of teaching the instrument. Ding, H et al. evaluated six automatic guzheng recognition techniques and based on them, proposed a guzheng fingering recognition strategy based on machine learning algorithms, which emphasises on fingering attributes recognition and the fingering recognition accuracy was close to 91% in the simulated test with fingering recognition accuracy close to 91% [16]. Zhou, Z et al. conceived a teaching system to assist guzheng fingering practice, which can provide three types of students with guzheng fingering practice scenarios and learning feedback, and effectively improve students’ proficiency in guzheng fingering [17]. Li, S et al. combined Unity 3D and Kinect camera, as well as the virtual reality gesture recognition technology, to constructed a virtual performance demonstration model of Chinese traditional instrument guzheng, which promoted the intelligent and digital development of guzheng performance [18]. Chen, J et al. conceptualised a fully automated deep learning framework with a new guzheng music input as the core logic, and based on the framework, they designed a semblance animation, which could be effectively adapted to the rhythm and melody of the guzheng music [19].

Although the guzheng playing technique has gone through three innovative revolutions in the changing times, its development is not only the pursuit of technical level, but also has a deep emotional resonance and aesthetic touch. The sound of the guzheng’s strings not only echoes the ebb and flow of the tune, but also contains the ebb and flow of the player’s inner emotions. This beauty is not only embodied in the melody and harmony of the music, but also in the player’s gesture, demeanour and emotional expression. Liu, Y reviewed the origin and development of the guzheng culture, as well as the status quo of guzheng playing in today’s world, and believed that there is a need to carry on the good art and culture of the guzheng music art and propagate it [20]. Wang, X described the performance characteristics and development history of guzheng music art, and discussed the traditional rhyming techniques in guzheng performance, aiming at removing its essence and pushing forward the art of guzheng music [21]. Jingqi, L talked about the traditional guzheng playing techniques and deeply studied the innovation and development of the three guzheng playing techniques, namely, Topi, Shake Straight, and Wheel Straight, and emphasised the significance of the linear art for the training of guzheng performance, pointing out that the linear art promotes to a certain extent the improvement of guzheng learner’s playing skills and level [22]. Wang, S discusses the innovation and adaptation practices on the Youtube platform for the guzheng music art and argues that these guzheng innovations and changes expand more artistic styles for the guzheng art, which is conducive to the convergence of Chinese traditional music art and international music art, as well as having a positive significance for the circulation of the guzheng music art [23]. Based on an interdisciplinary research approach, Du, C et al. examined the path of Chinese guzheng music towards internationalisation from three perspectives: traditional music, cultural space and music culture, and put forward targeted opinions, which were mainly related to the preservation and development of Chinese guzheng music [24].

The study mainly focuses on the recognition and judgement of the three basic playing techniques of “lifting”, “pinching” and “shaking”, which are related to the mechanics principle in guzheng performance, using computer vision technology, and it is divided into It is divided into two parts. The first part is the recognition of finger movement posture. Based on the 3D-DGR network model, a finger motion gesture recognition model is constructed using Mean Shift joint positioning and target point position estimation based on displacement velocity. The second part is the judgement of finger function. The tangent comparison method was used to detect and calculate the finger joint angles, and the changes in the joint angles compared to the standard fingering method were used to judge the player’s finger function. Finally, the effectiveness of the application of computer vision technology is examined by combining experiments and professional scoring methods.

2

Mechanics in Basic Performance Technique

2.1

Projectile lifting

Tibiao is a kind of ancient boxing way to pluck the strings through the small joints of the fingers, and the rest of the joints move towards the palm with the small joints, and the basic playing techniques in Tibiao are “Tuo, Wiping, Hooking, Big Handicap, Small Handicap, and so on”.

The transverse vibration of the string is the main vibration that determines the pitch of the fundamental frequency. Through the decomposition of force in three-dimensional coordinates, it can be seen that the component force of plucking force in the X-axis is along the direction of the string, which can only produce the longitudinal vibration of the string, so in order to make the plucking force play the maximum effect, the component force F_2−X in the X-axis should be taken as the minimum value when θ = 90°, F_2−X = 0, at this time, all the plucking force applied by the player is the effective plucking force. In other words, when the player uses the pick, the direction of the plucking force should be in the plane formed by the Y-axis and the Z-axis, i.e., the direction of the force is perpendicular to the string. Therefore, when playing the pick, the player is required to hold an empty fist, open the tiger’s mouth, protrude the joints, and the first joint of the fingers from the oblique downward to the oblique upward to the direction of the palm, all in order to ensure that the direction of plucking force is perpendicular to the strings in the picking.

The direction of finger movement is towards the centre of the palm, and the fingers must produce plucking force through muscle contraction, and at the same time, they need to overcome part of their own gravity. In short, the plucking force on the strings when lifting the strings is F_Plucked = F_Muscle − G′, in which F_Plucked is the plucking force exerted on the strings, F_Muscle is the force produced by the finger muscles, and G′ is a part of the gravity of the hand out (fingers).

On the other hand, lifting and playing is characterised by crisp articulation and clean sound quality. From the momentum and energy in guzheng performance, it can be seen that the amplitude of the strings is related to the size of the plucking force, the time of the plucking force and the distance of the plucking force (Note: the “distance of the plucking force” here refers to the “distance of the plucking force in its direction”, hereinafter), and the amplitude of the strings determines the intensity of the sound. The amplitude of the string determines the intensity of the sound. Therefore, in order to ensure the articulation characteristics of the pick, the player must control the amount, duration and distance of the plucking force. In short, when using the pick, the player is required to control the strength (i.e., control the size of the plucking force), and quickly release the force and quickly restore the relaxation (i.e., control the time and distance of the plucking force).

2.2

Clipping

The “pinch play” is to use the power of the fingers and palm of the hand to apply force towards the panel, and the nail plane is pressed down at an angle of about 45 degrees to make the strings vibrate with the downward force. The ring finger and the little finger are placed against the strings in a vertical position, the big finger is vertical, the other four fingers are bent and the centre of gravity is placed on the big finger, and the force is generated from the top to the bottom, using a straw grip. The basic playing techniques include the “tor, wall, wipe, hook, big handful and so on”.

From the decomposition of force, the plucking direction of clip and lift is opposite, but the angle of plucking is similar, so the reason for the difference in tone between the two methods is not the direction of plucking, but the size, time and distance of plucking force. The player will use part of the gravity of the hand to act on the big finger, plus the downward force generated by the muscle contraction of the big finger, so the plucking force of the strings during the clip bomb is F_Plucked = F_Muscle + G′, it can be seen that, due to the full use of the human body’s own gravity during the clip bomb, coupled with the force generated by the muscles, the plucking force generated by the clip bomb is much greater than the lifting of the bomb, and at the same time in order to ensure that the plucking force in the clip bomb is given full play to, the clip bomb will be reliably placed on the next string after it is fired, which is a good idea. At the same time, in order to ensure the full play of the plucking force in the clip, after the clip is finished, it is reliable on the next string, which fully guarantees the action time and action distance of the plucking force, and also makes the strings in the clip after the clip is finished, its deformation of the stored elastic potential energy and kinetic energy is greater, naturally, after the clip is finished, the pronunciation is also thicker, and the sound strength is greater.

2.3

Finger shaking

Finger shaking is a technique that the big finger can get a continuous long tone through the periodic shaking of the arm and wrist, respectively, there are pile shaking, hanging wrist, forefinger, multi-finger, etc. When learning it, the basis is the shaking of the finger, so this article mainly takes the shaking of the finger as an example. Learning to stake shake as the basis, so this article mainly to stake shake as an example. Playing the inside of the index finger pinch one-half of the big finger Yijia, little finger standing on the right side of the Yue Shan, the middle finger, ring finger relaxation bending, big finger Tuo, cleave natural articulation, rapid alternation, strength, frequency uniformity, playing the wrist to drive the small arm, relaxation to find the inertia, to avoid the wrist, the small arm, the big arm tension shaking the finger can not be sustained.

The purpose of finger-swinging is to obtain continuous long tones through the periodic input of energy, so in order to ensure the stability of the tone when finger-swinging, it is necessary for the strings to obtain a similar amount of energy during each vibration, i.e., the amplitude of the strings is similar each time. The mechanical energy of the string is converted into acoustic energy after each vibration, so the amplitude of the string will be weakened with the increase of the vibration times. In order to maintain the stable vibration process of the string, it is necessary for the player to keep the size of the plucking force uniform in the process of shaking the finger, and at the same time to keep the speed of the finger shaking uniform (i.e. to ensure the uniformity of the plucking force’s action time and its action distance), which are to ensure the uniformity of the work done to the string in the process of shaking the finger each time. All these are to ensure that the work done on the string during each finger-swing is uniform, so that the mechanical energy of the string is similar during each vibration.

3

Finger movement gesture recognition and finger function judgement methods

3.1

Finger movement gesture recognition model construction

The pre-processed “lifting, pinching, and shaking” standard fingering images and research images are simultaneously fed into the 3D-DGR network model, which is divided into two main branches, the upper branch is responsible for processing the standard fingering images and the lower branch is responsible for processing the research images. Each branch consists of two main parts, the DCNN module is responsible for extracting features in the spatial domain of the image and the LSTM module is responsible for processing the extracted features in the time domain [25]. In the DCNN module, four convolutional blocks and three fully connected layers are used. The convolutional block consists of Conv3×3, MaxPooling2×2, BN layers and ReLU layers and the second convolutional block does not contain MaxPooling2×2. The computation of the convolutional block $C B_{1} (X_{R})$ $$C{B_1}\left( {{X_R}} \right)$$ which contains MaxPooling operation and the convolutional block $C B_{2} (X_{R})$ $$C{B_2}\left( {{X_R}} \right)$$ which does not contain MaxPooling operation are shown in Equation (1) and Equation (2) respectively: (1) $C B_{1} (X_{R}) = δ (B (M (C o n v_{3} (X_{R}))))$ $$C{B_1}\left( {{X_R}} \right) = \delta \left( {B\left( {M\left( {Con{v_3}\left( {{X_R}} \right)} \right)} \right)} \right)$$ (2) $C B_{2} (X_{R}) = δ (B (C o n v_{3} (X_{R})))$ $$C{B_2}\left( {{X_R}} \right) = \delta \left( {B\left( {Con{v_3}\left( {{X_R}} \right)} \right)} \right)$$

where δ denotes the ReLU activation function. B denotes the Batch Normalisation layer. M denotes MaxPooling operation. C²v₃ denotes 3 × 3 Convolution operation. X_R denotes the standard finger image. Then the overall convolution module output can be expressed by $X B (X_{R})$ $$XB\left( {{X_R}} \right)$$ as in Eq. (3): (3) $X B (X_{R}) = C B_{1} (C B_{1} (C B_{2} (C B_{1} (X_{R}))))$ $$XB\left( {{X_R}} \right) = C{B_1}\left( {C{B_1}\left( {C{B_2}\left( {C{B_1}\left( {{X_R}} \right)} \right)} \right)} \right)$$

The final feature extraction of the DCNN module is then completed by a 3-layer fully connected layer, as shown in Equation (4): (4) $L X B (X_{R}) = L i n e a r (L i n e a r (L i n e a r (X B (X_{R}))))$ $$LXB\left( {{X_R}} \right) = Linear\left( {Linear\left( {Linear\left( {XB\left( {{X_R}} \right)} \right)} \right)} \right)$$

where Linear indicates that a fully connected layer is used. The output feature map of DCNN module $L X B (X_{R})$ $$LXB\left( {{X_R}} \right)$$ is fed into an LSTM block consisting of an LSTM layer and a fully connected layer. The operation of LSTM block $L L (X_{R})$ $$LL\left( {{X_R}} \right)$$ can be expressed in equation (5): (5) $L L (X_{R}) = L i n e a r (L S T M (L X B (X_{R})))$ $$LL\left( {{X_R}} \right) = Linear\left( {LSTM\left( {LXB\left( {{X_R}} \right)} \right)} \right)$$

Where Linear denotes the fully connected layer. LSTM denotes the temporal network LSTM function. The connection between the fully connected layers (FC3 and FC4) and the LSTM layer is shown in Fig. 1.

The features learnt in the FC3 layer are sent to the LSTM layer for dynamic feature extraction in the temporal domain, followed by the FC4 layer, which compares the spatio-temporal features obtained from the two branches, and finally the information is finally classified using the fully connected layer (FC5).3D-DGR Network Outputs $O u t p u t (X_{R}, X_{D})$ $$Output\left( {{X_R},{X_D}} \right)$$ The calculation of Equation (6) is as follows: (6) $O u t p u t (X_{R}, X_{D}) = L i n e a r (L L (X_{R}) + L L (X_{D}))$ $$Output\left( {{X_R},{X_D}} \right) = Linear\left( {LL\left( {{X_R}} \right) + LL\left( {{X_D}} \right)} \right)$$

where X_R denotes the standard fingering image. X_D denotes the study image.

3.2

Finger Motion Tracking Recognition Technology

3.2.1

Mean Shift based joint positioning technique

The applied MeanShift joint localisation method, the MeanShift algorithm process is a continuous iterative process. It first calculates the mean position of the interior points of a circle of radius bandwidth and uses it as the centre of the new circle, and then iterates in this way until a certain condition is met and no more iterations are performed, at which point the centre of the circle is the centre of the density and the algorithm requires the solution of a vector that can move the centre of the circle towards the point with the highest density [26].

Where the basic form of the MeanShift vector is defined as: (7) $M_{h} (x) = \frac{1}{k} \sum_{x_{i} \in S_{h}} (x_{i} - x)$ $${M_h}(x) = \frac{1}{k}\sum\limits_{{x_i} \in {S_h}} {\left( {{x_i} - x} \right)}$$

where k denotes the number of points in the S_h-sphere region, S_h denotes a high-dimensional spherical region of radius h, and $(x_{i} - x)$ $$\left( {{x_i} - x} \right)$$ denotes a vector of x-points to x_i, which can be expressed as the set of points y satisfying the following equation: (8) $S_{h} (x) \equiv {y : {(y - x)}^{T} (y - x) \leq h^{2}}$ $${S_h}(x) \equiv \left\{ {y:{{(y - x)}^T}(y - x) \le {h^2}} \right\}$$

In the specified region range S_h, for a given circle centre x, the offset vectors of the circle centre x pointing to all the other points inside the circle are found separately, and the average of all the offset vectors is obtained. If sample point x_i in region range S_h is obtained by sampling some probability density function, the direction of the probability density gradient will point in the direction of the largest probability density, and thus most sample points in region S_h may be distributed along the direction of the probability density gradient.

Suppose that a discrete data set, which is in fact obtained by sampling the density function, has a corresponding probability density function: (9) $P (x) = \frac{1}{n} \sum_{i = 1}^{n} K (x - x_{i})$ $$P(x) = \frac{1}{n}\sum\limits_{i = 1}^n K \left( {x - {x_i}} \right)$$

and the kernel function is selected: (10) $K (x - x_{i}) = c k [{‖ \frac{x - x_{I}}{h} ‖}^{2}]$ $$K\left( {x - {x_i}} \right) = ck\left[ {{{\left\| {\frac{{x - {x_I}}}{h}} \right\|}^2}} \right]$$

For the finger joint point localisation method based on the Mean Shift algorithm, the basic idea is to get several corresponding candidate points in each part region as joint point candidates by the MeanShift method, and then calculate the position of each candidate joint point, and if a certain joint point is within a certain range of the other candidate joint points, this point will be used as the joint point.

3.2.2

Displacement velocity based target point position estimation technique

When occlusion occurs on the finger in the image, the displacement velocity based target point position estimation technique (Kalman) can quantitatively estimate the next moment based on the state of the previous moment, and its main part is divided into two parts: state estimation and prediction function [27]. Its state equation and measurement equation are divided into: (11) $x (k) = A x (k - 1) + B w (k)$ $$x(k) = Ax(k - 1) + Bw(k)$$ (12) $z (k) = C x (k) + g (k)$ $$z(k) = Cx(k) + g(k)$$

where w(k) and g(k) denote the noise of the system process and the measurement process, respectively, which are Gaussian white noise. where the target point state vector is: (13) $x (k) = [\begin{matrix} u (k) \\ u' (k) \\ v (k) \\ v' (k) \end{matrix}]$ $$x(k) = \left[ {\begin{array}{*{20}{c}} {u(k)} \\ {u^\prime (k)} \\ {v(k)} \\ {v\prime (k)} \end{array}} \right]$$

Where, u(k), u′(k), v(k), v′(k) are the coordinates and velocities of the target on the x-axis and y-axis respectively. The observation vectors of the joints are: (14) $z (k) = [\begin{array}{l} u_{c} (k) \\ v_{c} (k) \end{array}]$ $$z(k) = \left[ {\begin{array}{*{20}{l}} {{u_c}(k)} \\ {{v_c}(k)} \end{array}} \right]$$

Where, u_c(k), v_c(k) are the coordinates of the target’s x-axis and y-axis respectively.

The final Kalman filter model is: (15) $[\begin{matrix} u (k) \\ u' (k) \\ v (k) \\ v' (k) \end{matrix}] = [\begin{matrix} 1 & t & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & t \\ 0 & 0 & 0 & 1 \end{matrix}] [\begin{matrix} u (k - 1) \\ u' (k - 1) \\ v (k - 1) \\ v' (k - 1) \end{matrix}] + [\begin{matrix} \frac{1}{2} Δ t^{2} \\ Δ t \\ \frac{1}{2} Δ t^{2} \\ Δ t \\ \frac{1}{2} Δ t^{2} \\ Δ t \end{matrix}] w (k)$ $$\left[ {\begin{array}{*{20}{c}} {u(k)} \\ {u^\prime (k)} \\ {v(k)} \\ {v^\prime (k)} \end{array}} \right] = \left[ {\begin{array}{*{20}{c}} 1&t&0&0 \\ 0&1&0&0 \\ 0&0&1&t \\ 0&0&0&1 \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {u(k - 1)} \\ {u^\prime (k - 1)} \\ {v(k - 1)} \\ {v^\prime (k - 1)} \end{array}} \right] + \left[ {\begin{array}{*{20}{c}} {\frac{1}{2}\Delta {t^2}} \\ {\Delta t} \\ {\frac{1}{2}\Delta {t^2}} \\ {\Delta t} \\ {\frac{1}{2}\Delta {t^2}} \\ {\Delta t} \end{array}} \right]w(k)$$ (16) $[\begin{matrix} u_{c} (k) \\ v_{c} (k) \end{matrix}] = [\begin{array}{l} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \end{array}] [\begin{matrix} u (k) \\ u' (k) \\ v (k) \\ v' (k) \end{matrix}] + [\begin{array}{l} 1 \\ 1 \end{array}] g (k)$ $$\left[ {\begin{array}{*{20}{c}} {{u_c}(k)} \\ {{v_c}(k)} \end{array}} \right] = \left[ {\begin{array}{*{20}{l}} 1&0&0&0 \\ 0&1&0&0 \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {u(k)} \\ {u^\prime (k)} \\ {v(k)} \\ {v^\prime (k)} \end{array}} \right] + \left[ {\begin{array}{*{20}{l}} 1 \\ 1 \end{array}} \right]g(k)$$

3.3

Finger Function Judgement Based on Changes in Knuckle Angle

The captured images, which are converted to digital images by the image capture card, are input to the computer, and the hand image and the image of the grasped hand posture are extracted by the thresholding method: (17) $f (x_{i}, y_{i}) = {\begin{array}{l} \bar{A} & i f & | f (x_{i}, y_{i} + 1) - f (x_{i}, y_{i}) | > T \\ A & i f & | f (x_{i}, y_{i} + 1) - f (x_{i}, y_{i}) | \leq T \end{array}$ $$f\left( {{x_i},{y_i}} \right) = \left\{ {\begin{array}{*{20}{l}} {\bar A}&{ {\text{ }}if{\text{ }}}&{ \left| {f\left( {{x_i},{y_i} + 1} \right) - f\left( {{x_i},{y_i}} \right)} \right| > T} \\ A&{ {\text{ }}if{\text{ }}}&{ \left| {f\left( {{x_i},{y_i} + 1} \right) - f\left( {{x_i},{y_i}} \right)} \right| \le T} \end{array}} \right.$$

Or: (18) $f (x_{i}, y_{i}) = {\begin{array}{l} \bar{A} & i f & | f (x_{i}, y_{i}) - f (x_{i} + 1, y_{i}) | > T \\ A & i f & | f (x_{i}, y_{i}) - f (x_{i} + 1, y_{i}) | \leq T \end{array}$ $$f\left( {{x_i},{y_i}} \right) = \left\{ {\begin{array}{*{20}{l}} {\bar A}&{ {\text{ }}if{\text{ }}}&{ \left| {f\left( {{x_i},{y_i}} \right) - f\left( {{x_i} + 1,{y_i}} \right)} \right| > T} \\ A&{ {\text{ }}if{\text{ }}}&{ \left| {f\left( {{x_i},{y_i}} \right) - f\left( {{x_i} + 1,{y_i}} \right)} \right| \le T} \end{array}} \right.$$

Where, T is the grey scale threshold. $f (x_{i}, y_{i})$ $$f\left( {{x_i},{y_i}} \right)$$ is the grey level of the detected image point. $f (x_{i} + 1, y_{i})$ $$f\left( {{x_i} + 1,{y_i}} \right)$$ and $f (x_{i}, y_{i} + 1)$ $$f\left( {{x_i},{y_i} + 1} \right)$$ are the image greyscale of the next row or column of $f (x_{i}, y_{i})$ $$f\left( {{x_i},{y_i}} \right)$$, respectively. A is the set current position image grey value. In the study, the background image grey level is set to 0 and the image grey level of the other images is set to 255. Using raster scanning, the image is scanned from both directions in rows and the above operations are performed at the same time, so that the missing image information due to some reasons can be avoided.

Since the object of the study is the joint angle of the finger, in the two-valued image extracted with the threshold segmentation method described above, all object images other than the image of the hand are irrelevant object images. In order to facilitate the analysis of finger postures, all irrelevant object images are removed by the raster scanning marking method, leaving only the image of the hand to be analysed.

The 2D image coordinates of the outer edge points of the thumb and index finger of the hand are extracted using a scan-and-detect method at n-row intervals. They are deposited into two edge point matrices M_T and M_I, respectively. The choice of the value of n is a more critical issue, as scanning by rows interferes with the analysis of the overall trend of the line of edges if there will be some irrelevant inflection points (e.g., the natural morphology of the skin on the hand) in the edge point matrices. If the amount of interlaced lines is too large, a joint corner may be missed.

After obtaining the two edge point matrices, they are analysed separately to find the angle of tangency between the first edge point and the line formed by the points in that point matrix: (19) $t g α_{i j} = \frac{y_{i j} - y_{i 0}}{x_{i j} - x_{i 0}}$ $$tg{\alpha_{ij}} = \frac{{{y_{ij}} - {y_{i0}}}}{{{x_{ij}} - {x_{i0}}}}$$

Detection is carried out using the method of tangent angle comparison. The analysis can be started by taking any one of the above two matrices and the analysis process is the same. From the two-dimensional coordinates $(x_{i 0}, y_{i 0})$ $$\left( {{x_{i0}},{y_{i0}}} \right)$$, $(x_{i 1}, y_{i})$ $$\left( {{x_{i1}}} \right.,\left. {{y_i}} \right)$$ of the first and second edge points, the slope of the straight line between the two points is found tgα₀, which is used as the comparison base, the first point is the starting point of the line segment $(x_{i 0}, y_{i 0}) = (x_{i s}, y_{i b})$ $$\left( {{x_{i0}},{y_{i0}}} \right) = \left( {{x_{is}},{y_{ib}}} \right)$$, and the second point is used as the temporary end point of the line segment $(x_{i 1}, y_{i 1}) = (x_{i e}, y_{i e})$ $$\left( {{x_{i1}},{y_{i1}}} \right) = \left( {{x_{ie}},{y_{ie}}} \right)$$. where i is the serial number of the line segment. Then, from the third point onwards, the slope of the line between the current point and the start point is found tgα_ij, and the difference between the two is found D by comparing it with the benchmark tgα₀. The edge points that are smaller than a certain value are regarded as being on the same line as the last neighbouring point, and are taken as the new end point coordinates of the line. When D is greater than a given threshold, the current point is regarded as the starting point of the next straight line, and the above comparison process is repeated from this point, and so on, to find a straight line that describes the composition of the knuckles. From the slopes of two neighbouring straight lines, the angle between the lines, i.e. the angle of each knuckle, can be found.

When solving for this, since the angles of the joints are constantly changing during the movement, sometimes the angle of a joint is 180, and the number of joints in the thumb and index finger are not equal, the number of joint angles in the edge point matrix is not a constant value, and thus cannot be used as a loop termination condition. We use the fingertip coordinates (the last set of coordinates) as the termination condition.

The grasping process of the hand is continuously captured, and the change in the angle of the finger joints between two adjacent images is compared to obtain the amount of change in the angle of each finger joint.

4

Empirical analyses

The study selects a video of guzheng performance by students of a conservatory of music to conduct a comparative experiment of the computer vision technique designed in this paper. And the finger function of the player in the video is analysed according to the change of the angle of the finger joints. The judging results of this paper are compared with the judging results of professionals to verify the executability of the method of this paper.

4.1

Comparative experiments

4.1.1

Data set construction

All the 1000 collected fingering data were integrated into a complete initial dataset, which will be used for further research on fingering pose recognition and analysis of guzheng playing. Before training on the data, the acquired dataset needs to be manually labelled. This requires not only labelling the fingering names for individual initial data files, but also labelling the valid pose segments in each fingering after the valid data segments have been intercepted, in order to facilitate the accuracy assessment of subsequent finger movement pose recognition models.

4.1.2

Experimental results

In order to validate the recognition effect of the models in this paper, the 3D-DGR models of Model-1 without the MeanShift joint localisation method and Kalman technique, Model-2 with only the MeanShift joint localisation method, Model-3 with only the Kalman technique, and Model-4 with both the MeanShift joint localisation method and the Model-4 with Kalman technique for experimental comparison. The recognition results of each model are shown in Figure 2.

Through the comparative analysis, it can be clearly seen that the finger motion gesture recognition model proposed in this paper, which incorporates the MeanShift joint localisation method and Kalman technique, significantly outperforms the other comparative models in terms of accuracy, with a recognition accuracy of 96%. The recognition process takes less time, 1.02ms less compared to the original model.

4.2

Comparison of finger movement change angles with standard fingerings

A student’s guzheng performance video was randomly selected as the data to be predicted, and three playing clips containing three basic playing techniques were intercepted. The finger movement gesture recognition model proposed in this paper is used to identify the movements of “lifting”, “pinching” and “shaking”, and the tangent comparison method is used to detect and calculate the angle changes of the finger joints. The changes of finger joint angles are detected and calculated using the tangent comparison method.

Comparison of knuckle angles during performance with the standard angles is shown in Figure 3. (a)~(c) are the changes of the knuckle angle for the movements of “lifting”, “pinching” and “shaking”, respectively. As can be seen from Fig. 3, the overall trend of the changes in the angle of the knuckle is more consistent with that of the standard angle during the student’s performance. The average errors of “lifting”, “pinching” and “shaking” with the standard angle were 0.402, 0.437 and 1.244 respectively, while the average error of “shaking” was the largest. During the playing of “finger shaking”, the error between the angle of the knuckle and the standard angle was the largest, which may be due to the fast change of movement and large movement amplitude of this fingering, which makes it more difficult to play compared with the other two fingering methods. The student’s changes in the angle of the knuckles in the basic playing technique were cnt’s joint mobility was more flexible.

4.3

Evaluation of Finger Functions in Basic Performance Techniques

The finger function of the performer was analysed from the angle change above, and in this subsection, a questionnaire was administered to the professionals to statistically assess the subjective evaluation of the student’s finger function in six aspects, namely, basic technique (A), hand shape and fingering (B), timbre and strength (C), intonation and rhythm (D), expressiveness of the piece (E), and musical comprehension (F), so as to verify the method of determining finger function based on the change of the joint angle The accuracy of the method of determining finger function based on changes in joint angles was verified.

The study scale was scored on a 7-point Likert scale, which was recorded as 1, 2, 3, 4, 5, 6, and 7 points from “very disagree”, “not compliant”, “somewhat non-compliant”, “somewhat non-compliant”, “somewhat compliant”, and “very much compliant”. The evaluation scores of 30 professionals were counted, and the students’ finger function was judged as a whole according to the average score.

The overall evaluation of the student by 30 professionals was that “lifting” was the best technique, followed by “pinching” and finally “shaking”. The results of the professionals’ evaluation are consistent with the results of the finger function judgement based on the change of the angle of the knuckle, which verifies the accuracy of this paper’s method.

The evaluation analysis of the “lifting” technique is shown in Table 1. The mean value of the overall score was 6.291.

Table 1.

Evaluation of Bomb performance technique

	A	B	C	D	E	F
1	6.34	6.41	5.79	6.45	5.99	6.69
2	6.47	5.61	6.13	6.31	6.18	6.5
3	6.73	5.93	6.29	6.09	6.07	5.65
4	5.72	5.81	6.71	6.83	6.62	6.18
5	6.32	5.81	5.98	6.78	6.22	6.27
6	6.49	5.65	5.56	5.75	5.74	6.67
7	6.71	6.91	6.63	6.48	6.3	6.66
8	5.7	6.65	6.57	5.84	6.57	5.93
9	6.81	6.54	5.62	6.69	6.19	6.17
10	6.15	6.34	6.13	5.86	6.91	6.87
11	6.34	5.9	5.61	6.46	6.96	6.03
12	6.6	6.81	6.24	6.76	5.57	5.68
13	6.84	6.49	5.66	5.52	6.09	6.54
14	6.58	5.51	6.89	5.8	6.95	5.96
15	6.05	6.88	7	5.94	6.67	6.81
16	6.26	6.49	5.57	5.78	6.95	6.33
17	6.45	5.59	5.55	6.89	6.43	5.96
18	6.05	5.57	6.6	6.77	6.18	5.62
19	6.68	6.98	6.91	5.83	6.13	6.49
20	6.92	6.89	5.67	5.83	5.91	6.41
21	6.82	5.57	5.73	6.02	6.38	6.65
22	6.08	5.95	6.78	6.28	6.95	6.97
23	5.96	5.62	5.77	5.6	6.1	6.36
24	6.99	6.24	6.06	5.6	6.67	5.56
25	5.99	5.75	6.87	6.69	5.61	6.54
26	6.93	6.73	6.81	6.44	6.38	5.65
27	6.9	5.81	6.93	6.55	6.15	6.83
28	6.09	6.31	6.32	5.89	6.97	6.64
29	6.92	6.39	6.39	6.05	6.44	6.24
30	6.33	6.4	6.55	5.92	6.89	6.57
Mean	6.091	6.174	6.052	6.133	6.120	6.176

The evaluation of the “clip” technique is analysed in Table 2. The overall mean score was 6.124.

Table 2.

Evaluation of Clip performance technique

	A	B	C	D	E	F
1	6.45	5.55	5.92	5.73	6.36	6.24
2	6.57	5.83	5.63	5.83	6.43	6.47
3	6.52	6.11	6.64	5.98	6.27	6.53
4	5.85	6.2	6.49	6.37	5.78	5.54
5	5.97	6.67	5.52	6.61	5.6	6.18
6	5.64	6.47	5.77	5.99	6.61	6.05
7	6.35	6.58	6.46	6.55	6.28	5.84
8	6.69	6.06	5.51	6.39	5.85	6.3
9	5.52	5.72	6.49	6	5.85	5.8
10	6.64	5.61	5.69	5.51	5.76	6.11
11	5.7	6.68	5.53	5.66	5.66	6.67
12	6.49	6.43	6.3	6.19	6.6	6.58
13	6.4	6.43	6.07	6.55	6.02	6.39
14	6.08	6.55	6.19	5.66	5.57	6.32
15	5.56	5.5	5.54	6.52	5.63	6.46
16	5.6	6.07	5.69	6.02	6.3	6.39
17	5.82	6.16	5.88	5.74	6.4	6.57
18	6.01	5.78	6.39	6.35	6.53	6.7
19	5.6	6.19	6.27	6.66	6.53	6.11
20	6.46	5.57	6.61	6.53	5.7	6.47
21	5.66	5.96	6.56	6.38	5.88	5.73
22	6.28	6	5.79	6.4	6.33	6.38
23	6.06	6.61	6.23	6.02	6.69	5.54
24	6.45	6.46	6.34	6.28	6.43	5.6
25	6	5.61	6.11	6.26	5.84	5.61
26	6.61	6.6	5.99	6.12	6.12	5.58
27	5.61	6.66	6.26	6.59	6.42	6.1
28	5.96	6.67	6.44	5.69	5.83	6.09
29	6.57	6.59	5.58	5.64	5.69	6.27
30	5.61	5.91	5.67	5.77	6.63	6.67
Mean	6.091	6.174	6.052	6.133	6.120	6.176

The evaluation of the “finger-swinging” technique is analysed in Table 3. The overall mean score was 6.031.

Table 3.

Evaluation of Vibrato performance technique

	A	B	C	D	E	F
1	6.16	6.37	6.02	5.55	6.02	5.78
2	6	6.41	6.07	6.03	5.59	6.12
3	5.96	5.69	6.17	6.18	5.86	6.23
4	6.2	6.38	5.65	5.81	5.71	6.36
5	6.05	5.83	6.42	6.15	5.68	6.54
6	6.54	5.76	6.25	6.33	5.95	5.72
7	6.04	6.31	6.49	6.48	5.55	6.41
8	6.16	5.94	5.79	6.15	6.09	6
9	6.06	6.12	5.76	6.5	6.37	5.74
10	6.16	6.01	5.78	6.3	5.56	5.68
11	6.02	6.45	5.71	5.84	5.59	5.77
12	6.27	6.35	6.09	5.89	5.91	6.01
13	5.93	5.93	6.33	6.3	5.86	6.25
14	6.08	6.4	5.87	6.52	6.37	6.38
15	6.38	6.29	6.39	5.79	5.52	5.88
16	6.47	6.54	6.17	5.96	5.88	6.01
17	6.11	5.7	5.5	5.99	6.19	6.08
18	5.7	5.91	6.11	5.85	5.84	5.64
19	5.89	6.39	5.99	6.3	6.45	5.95
20	5.82	5.86	6.03	5.65	6.08	5.56
21	6.18	6.03	6.41	5.56	6.44	5.78
22	5.6	6.18	6.15	5.9	5.74	6.4
23	5.53	5.88	5.87	6.27	6.22	6.13
24	5.66	6.39	5.84	5.5	6.54	5.6
25	6.15	6.17	6.25	5.51	5.84	6.35
26	5.69	5.91	6.34	5.96	5.9	5.78
27	5.97	6.02	5.73	6.09	5.66	5.79
28	5.61	5.68	5.89	6.22	6.53	5.97
29	6.11	5.77	5.94	6.48	6.54	5.81
30	6.31	6.34	6.49	6.07	6.13	5.86
Mean	6.027	6.1	6.05	6.038	5.987	5.986

5

Conclusion

The study constructed a finger motion gesture recognition model to identify the basic playing techniques involving mechanical principles in guzheng performance, and used the tangent comparison method to judge the player’s finger function. 1)

The recognition accuracy and recognition time of the constructed finger motion gesture recognition model are 96% and 1.07ms, respectively, which are the most efficient in comparison with other experimental models.

2)

The tangent comparison method was used to count the angles of the finger joints in “lifting”, “pinching” and “shaking”, and the average errors of the angle changes with the standard angles of the three basic playing techniques were 0.40% and 0.40%, respectively. The average errors of the angle changes with the standard angles of the three basic playing techniques were 0.402, 0.437, and 1.244, respectively, indicating that the player had the best mastery of the “lifting” technique, followed by “pinching”, and lastly, “shaking finger”. This indicates that this player has the best mastery of the “lifting” technique, followed by “pinching”, and finally “shaking”. In addition, the mean values of the three basic techniques evaluated by 30 professionals were 6.291, 6.124 and 6.031 respectively, which were consistent with the results of the tangent comparison method, indicating that the method of this paper can make a correct judgement on the finger function of guzheng players.

Język:: Angielski

Częstotliwość wydawania:: 1 razy w roku
Dziedziny czasopisma:: Nauki biologiczne, Nauki biologiczne, inne, Matematyka, Matematyka stosowana, Matematyka ogólna, Fizyka, Fizyka, inne

Kanał RSS czasopisma

A study on the mechanics, finger movement, and finger function of computer vision technology in guzheng playing posture recognition

Dan Lu

Data publikacji: 29 wrz 2025

Otrzymano: 07 sty 2025

Przyjęty: 20 kwi 2025

DOI: https://doi.org/10.2478/amns-2025-1120

Słowa kluczowe3D-DGR, Mean Shift joint localisation, Target point position estimation, Posture recognition, Computer vision

© 2025 Dan Lu, published by Sciendo

This work is licensed under the Creative Commons Attribution 4.0 International License.

Słowa kluczowe
3D-DGR, Mean Shift joint localisation, Target point position estimation, Posture recognition, Computer vision