Optimization Strategy of Piano Teaching Interaction Experience Based on Virtual Reality Technology

Among the diverse forms of modern piano performance, digital notation and automated accompaniment technologies are of particular interest, both of which have greatly expanded the boundaries of the performer’s repertoire and have also facilitated music creation and teaching like never before. Exploring the art of piano in this digital convergence reveals far-reaching implications and possibilities.

Digital notation technology makes complex scores more readable, allowing performers to quickly convert and adjust scores through software, while customizing displays to individual needs, such as note size, spacing, color, and emphasis can be set according to personal preference, greatly enhancing visual comfort and notational accuracy. Further applications, such as dynamic notation, which automatically scrolls as the player progresses, free the musician from the momentary distraction of flipping pages, allowing them to focus on the performance itself. In addition, by combining this technology with performance recordings, it provides data for subsequent analysis and review, making it easier to identify and correct errors in music education and self-training.

The development of automated accompaniment technology, on the other hand, makes a single piano performance more colorful. Through pre-programmed or real-time generated accompaniment, the performer can interact with the virtual sounds of various instruments as if they were in a full orchestra. The technology is often based on complex algorithms that recognize the rhythm, tempo, and musical style of the performer and generate a matching accompaniment. In addition to adding interest to individual practice, this technology brings an additional audiovisual dimension to performances by allowing players to perform complex multi-part musical works without the presence of other musicians, greatly enriching the style and level of the performance.

Through a careful exploration of these technologies, it becomes clear that the integration of digital technology into piano performance is not just a simple addition to existing performance forms, but a profound artistic revolution. In this process, the performer is no longer limited by traditional ways of playing, and more creativity and personal expression can be shown on the keys. With the advancement of technology, the future of piano art will surely present more unexplored possibilities, bringing more surprises and delights to music lovers all over the world.

Piano analog and timbre synthesis technologies have undoubtedly pushed the boundaries of modern music and piano performance, introducing unparalleled innovation and accessibility, and these technologies, which span the spectrum from commercially popular music production software to cutting-edge audio engineering research, have become powerful tools for musicians, composers, and educators to create and perform music [26]. As the digital music world continues to evolve, piano analog technology can now accurately replicate and even surpass the sonic quality and expressiveness of traditional instruments. Through deep sampling coupled with complex triggering algorithms, software can not only reproduce a straight piano sound, but also simulate ambient acoustics, such as acoustic feedback in echo chambers.

Tone synthesis, on the other hand, offers even more freedom of musical expression. Using synthesis technology, artists are no longer limited by the physical constraints of traditional instruments and are able to create new and unprecedented sounds. This technique allows musicians to design unique timbres and integrate them into their compositions, enriching the emotional breadth and depth of the music. For example, by using a waveform synthesizer, musicians can mix multiple waveforms to create unique sound colors, giving each performance a personalized style. More than just mimicking the sounds of existing instruments, tone synthesis opens the door to a wide range of natural and electronic sounds, empowering artists to transcend traditional boundaries and express unlimited creativity in their compositions.

Virtual Reality (VR) technology, on the other hand, fully immerses the user in a three-dimensional musical performance space by creating a fully digitalized environment, where the user wearing a VR headset can be immersed in a fictional concert hall, a chamber orchestra rehearsal scene, or any other virtual musical scene, and perform with virtual musicians or alone [27]. The applications of this technology are not limited to entertainment and rehearsal. It can also provide a risk-free environment for professional musicians to experiment with new performance styles or arrangements, or even to simulate a real performance experience without an actual audience, which is especially valuable for artists preparing for large-scale recitals. Such technology allows users to explore musical innovations without physical constraints, pushing the development and refinement of their personal artistic style.

In this paper, the HTC VIVE virtual reality kit is used as the hardware platform for the virtual piano. A complete HTCVIVE virtual reality set consists of a headset, two grips and two localizers (the two small square boxes in the figure).

Viv uses Lighting House indoor positioning technology, two rows of LED lights in the positioner emit a scanning beam six times per second, respectively, in the horizontal and vertical direction to take turns to scan the positioning space of 15 X 15 feet. HTC Vive headset and handle has more than 70 light-sensitive sensors. When the beam sweeps through the helmet at the same time, the helmet starts counting, the sensor receives the scanning beam, the sensor position and the time of receiving laser, then the scanning beam will be counted. Using the relationship between the sensor position and the time the laser is received, the exact position relative to the localizer can be calculated. With the help of localization technology, the user can walk around the virtual piano within the detection range, and even play the piano on the other side of the keyboard, further increasing the sense of three-dimensionality and immersion.

HTC VIVE in normal circumstances, you need to operate the menu through the handle, but holding the handle is not convenient to play the virtual piano, put on the helmet and can not observe the surrounding environment, can not pick up the handle, so this paper develops the virtual piano does not use the handle, but through the gesture recognition of the virtual piano control, such as timbre settings and rhythm adjustment and so on.

The virtual piano developed in this thesis includes a hand modeling module, a piano modeling module, a MIDI sound generation module and a gesture recognition module.

Unity3D is a professional game engine developed by Unity Technologies, which enables game developers to easily complete a variety of game ideas and 3D interactive development [28]. Unity3D has rendering resources such as physics simulation, normal mapping, screen space ambient occlusion, dynamic shadows, etc. It has two main advantages over other game development tools: it provides an excellent visualization workflow and multi-dimensional cross-platform support. Visual workflows make it easy to edit scene layouts, bind resources, and code interactive object scripting languages. Unity3D can be deployed across multiple platforms, running on 21 platforms including Windows, Mac, Wii, iPhone, WebGL, Windows Phone 8, and Android.

An ordinary piano consists of 88 black and white keys, which are separated by a semitone from the white keys, with the bass, middle, and high registers moving from left to right. For simplicity, the virtual piano developed in this paper has 7 tones in the bass, middle and treble regions, namely, do, re, mi, fa, so, la, si, with a total of 21 white keys and 15 black chromatic keys.

Take the example of building a bone-driven hand model, the steps are as follows:

The use of 3Ds Max software to build the palm model of the hand, first make a box object, appropriate subdivision and collapse into multiple quadrilateral, adjust the shape of the quadrilateral close to the palm shape. Use the cut command to extrude the finger surface, continue to adjust the length of the five fingers, continue to adjust the model while subdividing, use UV processing for texture mapping, increase bone bone parameters, etc. The completed resource has been added to the prefabricated parts file in Unity3D.

Since the influence of physiological factors, such as hand size and finger function, needs to be eliminated when measuring finger executive ability in response to cerebral cognitive ability, it is necessary to design experiments to measure basic information about the fingers, including span and dexterity, before measuring cerebral cognitive ability. Finger span refers to the distance between fingers, and finger dexterity refers to the ability of each finger to perform fine movements. Then, based on the knowledge of piano chord decomposition and the principle of finger movement, we set up a piano sight-reading experiment to quantitatively measure the brain cognitive ability by using a simple and easy-to-understand piano fingering expression, namely, the acoustic hand score.

Forty physically and mentally healthy subjects were recruited as the experimental group and confirmed by questioning that they had no significant cognitive impairment. The age of these volunteers ranged from 22 to 28 years old, with 20 males and 20 females. All the volunteers were right-handed, but one had a history of left-handed injury. In addition to this, 18 additional mentally and physically healthy subjects were recruited as a validation group. Students in the experimental group were taught interactive piano instruction in VR, while students in the validation group were taught traditional piano instruction.

This experiment was conducted in a quiet, well-lit laboratory. The laboratory contained a large table and a chair for the subjects to perform the experiment. The virtual piano described herein was placed on the experimental table, with a basic level note variable tuning positioning tape placed on the piano, the piano was connected to a control scanning circuit module through an FPC, and the control scanning circuit module was connected to a processor through a serial port, which was connected externally to a monitor and a camera, which was placed directly above the piano.

The distribution statistics of the continuous keystroke time of the fingers are shown in Fig. 1, most of the subjects in the experimental group have the continuous keystroke time between 1187ms and 1681ms, and a very small number of subjects have the continuous keystroke time within 1100ms, which is regarded as a very flexible finger, and a very small number of subjects have the continuous keystroke time greater than 1900ms, which is regarded as a not flexible finger. The limit of continuous keystroke time is about 1000ms due to the gravity rebound speed of the test equipment, in general, the distribution of continuous keystroke time shows a normal distribution trend. After further K-S normal test, the distribution of the continuous keystroke time of the finger can be considered to obey the normal distribution with the mean value μ of 1464ms and the standard deviation σ of 178ms.

Figure 1.

The experimental group’s finger is a continuous keystroke time statistic

Table 1 shows the statistics of the continuous keystroke time and dexterity of each finger in the experimental group. In order to facilitate the understanding of the finger numbering in this chapter, the English symbols L and R are used to denote the right and left hands respectively, and the subscripts 1-5 are used to denote the thumb, the index finger, the middle finger, the ring finger, and the little finger respectively. Since the experimental subjects were all right-handed, the dexterity scores of each finger of the right hand were significantly higher than the scores of each finger of the left hand, and the average dexterity scores of the right hand and the left hand were 71 and 62, respectively. Meanwhile the index finger of each hand was the most flexible and had the best independence (L2 and R2 were 65 and 75, respectively), and the ring finger, which was checked by the little and middle fingers, was the least flexible, with the flexibility of the left and right ring fingers being 57 and 68, respectively. The standard deviation of the little finger was the largest (L5=179, R5=163), and the scoring results were realistic and well reflected the degree of dexterity of each finger.

Fig. 2 shows examples of left and right hand finger spans, Fig. (a) shows the left hand span and Fig. (b) shows the right hand span, which are examples of left and right hand spans of the same subject in the experimental group obtained after processing the data collected from the test platform. The finger spans are represented by piano interval differences to facilitate the calculation of distance fit later. It can be seen that there is a difference in the span of each finger of the subjects, and the finger span of the right hand is larger than that of the left hand fingers, for example, in the spans of R4 and L4, the spans of R4 are 3.5, 6, and 9, and the spans of L4 are 3, 5.5, and 9, respectively, which shows that there is a necessity to measure the finger spans.

Figure 2.

Left and right hand finger span example

Figure 3 is a histogram of time fit for finger dexterity, which shows the time fit during sight-reading of the subjects in the experimental group, i.e., the difference between their playing time and the actual standard time. As can be seen from the graph, most of the subjects’ sight-playing time fit was between -500ms and 250ms, and very few subjects had sight-playing time fit to the left beyond -750ms and to the right beyond 750ms, and their response speed lagged far behind that of the others. By analyzing the distribution of sight-reading time fit, it can be found that the peak was reached around 23.1ms, and the number of people gradually decreased to the left and right sides, showing a bell-shaped distribution with a high middle and low sides, similar to a normal distribution.

Figure 3.

The time of flexibility of the finger is attached to the histogram

Table 2 shows the scores and accuracy of the difficulty tests, and the overall speed and accuracy scores of the experimental groups were calculated according to different playing difficulties. It is easy to see that there is a trade-off relationship between speed and accuracy in the process of score change in the two rounds of test for difficulty 1, and as the difficulty of playing increases, the scores show a decreasing trend, with the right hand’s accuracy dropping from 99% to 95%, and the left hand’s accuracy dropping from 95% to 87%, which is basically in line with common sense.

Figure 4 shows the cognitive ability scores of the five subjects in the validation group in the difficulty test, with rounds 1 and 2 belonging to difficulty 1, round 3 belonging to difficulty 2, and round 4 belonging to difficulty 3. With the exception of round 1, which was due to the unfamiliarity of the subjects with the vocal score and the piano equipment, the cognitive ability scores of the subjects gradually declined with the increasing difficulty of playing the piano, which was basically in line with the overall trend of change. Some subjects showed better cognitive scores in round 3 than in round 2, as well as cognitive scores in round 4 exceeded those in round 3, for example, subjects scored 65 in round 2 and 67 in round 3 of D. Such fluctuations may be affected by a variety of factors, such as psychological quality, difficulty adaptation, and degree of concentration, etc., and the effect of the virtual piano interactive teaching experience is remarkable.

Figure 4.

Five subjects scored a cognitive score on difficulty testing

Table 3 shows the teaching effectiveness, and the survey data shows that all the students in the experimental group believe that the following six items are “able to feel the beauty of piano music and the pleasure it brings”, “be proficient in reading musical scores”, “be able to play music at the correct rhythm”, “be able to play music with both hands” and “learn some piano playing skills” are all the knowledge and skills that students should master through piano lessons, and the percentage of cases is 100%. All the students in the experimental group believed that piano teaching has the following two promoting effects on music learning, namely “stimulating the potential of music learning” and “developing intelligence and training coordination ability”. The total number of cases was 40, accounting for 100% of the cases. Overall, in the investigation of the promoting effect of piano teaching on music learning, the teaching effect of the experimental group was better than that of the verification group.

One-way ANOVA analysis of male and female students in the experimental group, Table 4 shows the one-way way analysis of evaluation methods in different genders, the mean and standard deviation of different genders in each evaluation method, the results of the t-test descriptive language evaluation, listening to the students’ playing the p-value of 0 is less than 0.05, there is a significant difference. The descriptive language evaluation, including listening to playing, in male grades is higher than in female grades. The difference in scores is 1.0201, 0.4016, and 0.6575, respectively.

Table 5 shows the one-way ANOVA of teaching methods in different genders, the mean and standard deviation of different genders in each teaching method, the results of t-test p-value of lecture method, demonstration method, and instructional method are 0.0158, 0, 0, which is less than 0.05, and there is a significant difference. The performance of males under didactic, demonstration and instructional methods were higher than that of females with a difference of 0.3471, 0.491 and 0.5497 respectively.