Innovative Application of Human-Computer Interaction Technology in Higher Education Music Education

The user input layer mainly consists of input devices that are responsible for the interaction between the user and the virtual learning environment.With the help of input devices, users can interact with the virtual characters in the virtual learning environment through voice, posture, and other means to obtain specialized knowledge and other information contained therein.

The application function layer mainly includes the functions realized by the virtual learning environment, through the simulation of the real classroom environment, so as to improve students’ motivation to learn and let students have a better sense of immersion.

In order to realize the functions of each application scene, this paper designs an algorithm model layer in the virtual learning environment for music education in colleges and universities. The realization of each application scenario is a step-by-step process that cannot be separated from the support provided by the model layer, and these models play different supporting roles for different scenarios.

The development tool layer of the system mainly includes the construction platform of the virtual learning environment interaction system, which mainly includes the development platforms of 3DS MAX, Unity3D Engine virtual game engine, and PyCharm development tool, etc. 3DS MAX is responsible for the geometric modeling of the 3D model, while Unity3D is mainly responsible for the construction of the 3D scene and user interaction interface. Among them, 3DS MAX is responsible for the geometric modeling of the 3D model, while Unity3D is mainly responsible for the construction of the 3D scene and the construction of the user interaction interface, and PyCharm is mainly responsible for the running of the intelligent Q&A deep learning training code.

The data layer of the system mainly stores system-related information and expandable knowledge resources. In this paper, the related data resources are categorized into four types: IPS-VLQA Q&A library, open-domain Q&A library, control algorithm library, and virtual intelligent body model library.

The IPS-VLQA Q&A library mainly stores knowledge points related to music education.

The open-domain Q&A library stores Q&A statements from daily life.

The control algorithm library mainly stores the algorithms and parameters of virtual instrument experiments related to music education.

Virtual Intelligent Body Model Library, which mainly stores 3D models and character animations of intelligent bodies.

The virtual learning environment cannot be created without a variety of 3D models, such as walls, doors and windows, blackboards, schoolbags, textbooks and so on, which are needed for the scene [23]. First of all, through the 3DS MAX software to create some standard basic body, and then in the basis of these basic models for extrusion, patchwork and other operations for the expansion of the model.At the same time, the parameters of the model are set to conform to the real-world properties.

For some complex 3D models, each structural component model is still an independent entity after the completion of solid modeling, and it is necessary to adjust the parent-child relationship for them to make them become a completed object. At the same time, the attributes of each structural component are adjusted to make its position, size, and angle more consistent with reality.

After adjusting the parent-child relationship of the completed solid model, it is also necessary to carry out material processing. By adding these materials to the three-dimensional model, it can have a more realistic appearance. This is an important part of the geometric modeling process.Different objects need to be created with different materials to create a unique three-dimensional texture, thus creating a realistic three-dimensional effect.

Figure 2.

Schematic diagram of geometric modeling

Three-dimensional objects obtained through solid modeling usually need to be optimized, as the complexity of the model directly affects the performance of the entire system. Therefore, it is necessary to delete and merge the connecting surfaces and three-dimensional nodes of the three-dimensional object model, so as to streamline the number of surfaces and models, and improve the loading speed of the virtual scene as well as the system’s operation speed. At the same time, some complex 3D static models can be mapped to achieve the purpose of model optimization without affecting the visual sensory premise.

Through the above process, this paper finally builds a virtual learning environment for music education in colleges and universities.

Intelligent character motion control is mainly intended to directly control the body movements of intelligent characters, and the quality of intelligent character motion control will directly affect the immersion and authenticity of the virtual learning system.At present, intelligent character motion control technology mainly includes parametric keyframe technology, kinematics method, dynamics method, and motion capture technology.In this paper, we mainly model the motion control of intelligent characters using the dynamics method.The advantage of this method is that it makes the movements of the intelligent character more in line with physical laws and has good realism.

The primary purpose of the dynamics method is to determine the force and moment applied to the joints by analyzing the joint state of the intelligent character and the trajectory of the end effector. The more commonly used method is the Newton-Euler equation, which derives the forces and moments applied to the joints when the angular velocity, linear velocity and acceleration of the joints are known: (1) { F=mid2ridt2L=Iiωi+ωi×(Iiωi)

Where the force F is obtained from Newton’s second law and the moment L is expressed through Euler’s equation, where I_i is the inertia matrix, ω_i is the angular velocity, r_i is the position vector and m_i is the i th linkage mass. The kinematic approach increases the physical characteristics of the intelligent character’s movement and can be realized to control every detail of the intelligent character’s action.

The view cone is the basic method to simulate human vision, the view cone is shown in Figure 3, only the objects within the range of the view cone can be perceived by the intelligent character. Where L represents the effective distance that the intelligent character can see in the virtual environment, and R represents the range of the angle of the intelligent character’s field of view, in the virtual environment, R ≤ 30° is usually chosen to simulate the optimal visual zone of human vision.

Figure 3.

Visual cone

The point occlusion detection algorithm is a more commonly used line of sight detection method. Firstly, the center point O of the line between the two eyes of the intelligent character is used as the origin to establish the right hand local coordinate system of the intelligent character. The x-axis is the head direction of the smart character, and the ray emitted from the O-point is used to simulate human vision. Assuming P is a point on the object, if ║O–P║≤L, and the angle between this ray and x axis is less than R/2, then point P falls within the visual range of the smart character, and then line-of-sight detection is needed. Let the world coordinates of point O be (x₀,y₀,z₀) and the world coordinates of P be (x_p,y_p,z_p), then the line OP can be represented as: (2) x=xo+t(xp−xo),y=yo+t(yp−yo),z=zo+t(zp−zo) (0≤t≤1)

To facilitate the detection, a hexahedron is used as an enclosing box for the occluder and it is projected on the xy-face to obtain a rectangle, and the coordinates of its vertices are set to be (x₀,y₀,z₀)⋯(x₃,y₃,x₃) in order, then the equations of the sides of the rectangle are: (3) x−xixi+1−xi=y−yiyi+1−yi i={0,1,2,3}

Then, substitute Eq. (2) into Eq. (3) to solve for t. If t has no solution then OP has no intersection with the rectangle, which means that OP is not obscured by the intervening object, and the smart character can see P. If = Tt is found, then OP has an intersection with the rectangle. Then bring t = T into equation (2) and calculate the coordinates of z. If the coordinates of z, are within the height of the occluding object in the direction of the z axis, then P is not visible. Otherwise, P is visible.

In order to realize the visual perception of the intelligent character, this paper creates a visual perceptron for the intelligent character, in which the visual range of the intelligent character is defined, and visual triggers are also created for the objects that the intelligent character wishes to see. When the visual trigger is activated within the visual range of the smart character, it informs the visual perceptron, leading to the smart character’s appropriate behavior.

Once the virtual environment is represented as a raster, it is necessary to perform path search using a certain search strategy to find an optimal path from the start node to the goal node from the free raster. Currently, optimal path search can be divided into two categories: blind search and heuristic search.

Blind search is a type of brute force search algorithm, which generally searches according to a predetermined strategy without considering the characteristics of the problem itself.This method is not efficient and doesn’t always find the optimal path, which is not suitable for solving complex problems. For the problem of low efficiency of blind search, heuristic search has been proposed. Heuristic search adds enlightening information related to the problem, which makes the search proceed in the most ideal direction and reduces the complexity of the search. Heuristic search usually evaluates the importance of nodes using an evaluation function, which has the general form: (4) F(n)=G(n)+H(n)

Where F(n) is the evaluation function, G(n) is the cumulative cost, and H(n) is the estimation function, which reflects the heuristic information. By introducing the heuristic information, the search is carried out according to the size of the evaluation function to approach the target node, instead of conducting a diffuse search in every possible direction.

A* algorithm is the most commonly used heuristic search algorithm, has the advantages of fast speed and high efficiency, and is widely used in path planning, the evaluation function of A* algorithm is: (5) f(n)=g(n)+h(n)

where f(n) denotes the sum of the costs of the shortest paths for node n to reach the target node, g(n) denotes the actual cost for the initial node to reach node n, and (n)h is the estimated cost of the optimal path for node n to reach the target node. The mathematical expressions of g(n) and f(n) can be decided based on the actual situation, and in this paper, g(n) will be denoted as: (6) g(n)=∑i=inD(Ai−1,Ai)

Where A₀ is the starting node and D(A_i–1,A_i) denotes the distance between node A_i–1 and node A. When node A_i–1 and node A_i are connected at right angles, then D(A_i–1,A) = 1. If node A_i–1 and node A_i are connected diagonally, then D(Ai−1,Ai)=2 . Whereas the selection of h(n) is the key to path search and determines whether the optimal path can be found quickly.

For the raster method A* algorithms are better known valuation functions are the following 3 kinds:

The division of the virtual learning scene levels in this study is mainly based on the pedagogical interaction hierarchy tower, as well as the three-dimensional interaction hierarchy relationship, and modifications and additions are made on the basis of the division of these two types of interaction hierarchies, adding the main features of the virtual learning scene. For example, roaming and manipulating in the scene, etc. Finally, the hierarchical relationship of 3D interaction of the virtual learning scene is divided into three levels: geometric roaming layer, selective manipulation layer and collaborative interaction layer, forming the interaction hierarchy of the virtual learning scene as shown in Figure 4.

Figure 4.

Hierarchical relationship between virtual learning scenarios

When carrying out interaction design, we can start from the hierarchical relationship of interactions and analyze the required interaction at each layer according to the specific requirements of the virtual learning scene.In addition, it is also necessary to consider the technical way of achieving the interaction.Generally speaking, the technical way of interaction can be divided into several major interaction methods, such as visual gaze interaction, handle ray interaction, position tracking interaction and voice interaction:

1) Visual gaze interaction

Visual gaze interaction, also known as “timed anchor point”, is the most basic interaction method. It means that after the learner puts on the helmet, he/she can select the interaction through the anchor point directly in front of his/her eyes. When the learner focuses his/her gaze on an interactive interface or virtual object in the scene for a certain period of time, the interaction will be triggered by this object.

When designing interactions in virtual learning scenarios, the type of form of interaction is mainly considered to be the interaction between the person and the scenario. Therefore, when designing the interaction of the scene, the two different interaction categories, explicit and implicit, are mainly considered in the design of the scene.

In this paper, explicit and implicit interactions are further subdivided into four types of interaction forms as shown in Figure 5, based on the two aspects of the initiative of device use and the attention of user experience. In this case, the interaction is divided into four categories, with the initiative of the device being the horizontal axis and the attention being the vertical axis. The white area indicates explicit interaction and the gray area indicates implicit interaction, and the four quadrants correspond to the four types of interaction. From the division of explicit and implicit interactions, it can be seen that each quadrant represents an interaction mode, and each interaction mode plays a different interaction function.

Figure 5.

Explicit interaction and implicit interaction

From a comprehensive point of view, the interactions in virtual learning scenarios can be simply divided into two categories: explicit and implicit interactions:

First, explicit interaction mainly refers to learners’ interactions with the obvious iconic elements in the scene, which can be directly accessed through observation. Learners can trigger corresponding actions through the interaction interface and menus to realize interaction anytime and anywhere.

Second, implicit interaction refers to the fact that the interaction symbols are not always observed by the learner in the scene and are not always presented in the interface. The main role of implicit interaction is to gain insight into the learners’ real needs, provide temporary guidance or hints for the learners, and eliminate the obstacles in the learning process in time to ensure the smooth progress of the learning process.

Based on the data from the above questionnaire research, the average score situation of 49 students on the importance of each design requirement point of the virtual learning system for music education in colleges and universities was calculated to facilitate the subsequent data analysis work, and the details of the scores of each requirement point are shown in Table 3. From the data scores, it can be seen that the students have higher demands on perceived controllability (R8, 4.35), ease of operation (R9, 4.25), scenario aesthetics (R11, 4.35), and design style consistency (R3, 4.25), which meets the students’ pursuit of visual perception and interactive operation of virtual learning environments.

However, evaluating the importance of user requirements solely from the students’ perspective is too one-sided and subjective, so it is necessary to synthesize the views of five experts on the design requirements.The scores of the design requirements were weighed accordingly to arrive at an adjusted level of importance for the user requirements. Although the total number of student evaluations is higher than that of the experts, their depth of research and understanding of the VLE is weaker than that of the experts, so in order to better weigh the two, the student evaluations and the expert scores were each given a 50% weighting.

In addition, in order to emphasize the key user requirements and the subsequent determination of design goals, the weighting of the final adjusted design requirements was controlled so that all the design requirements summed up to 1. The details of the specific scores are shown in Table 4, with the highest weighted score being content fit (R4, 0.0960), followed by ease of operation (R9, 0.0956). The weighted scores are more credible than doing an evaluation of the importance of user needs from the students’ perspective alone.

Similar to the user needs study, each participant will be required to fill out a background survey that collects general demographic information, whether they have had experience with VLEs, and whether they have had experience with VR.Each of the 49 participants will receive approximately five minutes of rapid VLE training due to the fact that the participant may be exposed to virtual reality for the first time. Therefore, subjective evaluations may be influenced by exposure to something new. Next, participants will be required to perform the following two tasks. The order in which these two tasks are presented will be balanced between participants:

For Task 1, the system provided a virtual piano model as the target shape, and participants were asked to use the virtual piano to complete the piano piece “To Alice”.

For Task 2, the system provided a virtual piano model identical to the one in Task 1, and participants were asked to use the virtual piano to compose freely.

Each task was preceded by a training session in which the experimenter explained how to interact with the system and allowed participants to practice. After practicing, participants were asked to get as close as possible to the target shape and complete the virtual model as quickly as possible. Participants were then asked to complete questionnaires about usability and presence.

In order to evaluate the overall HCI effectiveness of the virtual learning system constructed in this paper for music education in higher education, this study collected objective user interaction performance, as well as subjective metrics related to the evaluation of usability and presence. For user interaction performance, this study measured completion time and accuracy. Interaction accuracy was calculated from the correlation coefficient of the data between the user’s performance score of playing the piano piece “To Alice” in the virtual learning system, and the standardized score. For sense of presence, this study used the Slater-Usoh-Steed (SUS) sense of presence questionnaire to collect user feedback after each task. To evaluate the usability of a virtual learning system for music education in higher education using HCI, this study used its own usability questionnaire consisting of a 7-point Likert scale.

The experimental results of 49 users completing Task 1 and Task 2 in the usability study of human-computer interaction are shown in Figure 6. Where (a) is the completion time, (b) is the accuracy (c) is the presence and usability score. An t-test was conducted in this study and the results showed that there was no significant difference in the average time to complete Task 1 and Task 2 for the 49 users in the virtual learning system. For interaction accuracy, this study also conducted a t test, which showed that Task 1 (Mean=0.854) was significantly more accurate (t =6.45, p<0.001) than Task 2 (Mean=0.806). For the Subjective Presence Questionnaire, the results of the t test indicated that Task 1 had a significantly higher presence score than Task 2 (t =11.73, p<0.001). Similarly, on the interaction usability score, students scored significantly higher in Task 1 than Task 2 (t =9.41, p<0.001).

Figure 6.

Usability analysis of human-computer interaction of the system

Overall, familiarity improves users’ interaction performance in the virtual learning environment. From the results, it was found that participants’ completion speed and interaction accuracy for the familiar piano piece “To Alice” were better than the free creation performance in Task 2. This virtual course is mainly for college students and graduate students majoring in music-related fields of art in colleges and universities, and it is also considered to provide online teaching services for students in other colleges and universities across the country. Therefore, when designing a virtual learning system for music education, it is necessary to take into account the characteristics of such students and design the virtual course in a targeted way.As music students are more sensitive to the visual appearance of the virtual learning environment, including color and other factors, they have a strong curiosity and desire to learn new things. And their thinking is relatively strong, so when designing the teaching scene of the virtual learning environment, we need to control the design quality, design style and interface design layout according to these characteristics of the students. Try to use scenes that students are familiar with when developing the design to reduce cognitive barriers in the process of student use.