A Practical Study of Creative Stage Performance in the Integration of Vocal Singing and Artificial Intelligence

With the continuous development of Internet technology, online concerts have become increasingly popular, providing a brand new platform for musical performances. Artists and bands are able to connect directly with audiences around the world through online platforms, breaking the limitations of geography and physical space. Today, online concerts have become an important way for musicians to interact with their audience. Through this format, audiences can enjoy high-quality musical performances from the comfort of their own homes, while providing a way for artists to maintain their prestige and income.

As one of the important research topics in the field of artificial intelligence, virtual reality technology comes with the reproducibility, reorganization, computation, and transmission, the interactivity, immersion, virtuality, and intelligence of the social communication process, the diversity of forms of artistic creation, the non-linearity of artistic narrative, the real-time nature of the information transmission, and the convenience of the creation tools. The application of virtual reality art in the era of artificial intelligence has a rich and colorful performance. The technical relationship is shown in Figure 1, the application of virtual reality and augmented reality technology is bringing music performance into a whole new dimension. Virtual reality technology can simulate realistic three-dimensional music performance scenes, and through VR helmets, listeners can be immersed in an all-encompassing music experience, as if they were actually standing in front of the stage or in a fictional music world, and feel the charm of the music performance.AR technology allows virtual elements to be integrated into real-world scenes, adding extra visual effects to live concerts. With these two technologies, the boundaries of musical performances are expanded, allowing audiences to experience performance forms in the virtual world that are not possible in real life.

Figure 1.

New technical schematic diagram

The rise of live-streaming platforms has provided another form of new platform for music performances. Platforms such as Twitch, Instagram Live, and Facebook Live allow artists to interact with their audience in real time, not just by performing music, but also by answering questions, sharing the creative process, and even improvising. In this interactive experience, musical performance practices take on a distinctly contemporary character.

In the art of contemporary musical theatre, the integration of virtual reality technology has brought revolutionary changes to the construction of stage space. The virtual reality stage space not only expands the boundaries of traditional stage art, but also creates an unprecedented interactive and immersive experience for the audience and actors. In this process of technology-driven innovation, three core elements – “virtual scenes”, “action spaces”, and “viewing spaces” – become key to building a virtual reality stage space. Virtual reality technology makes “virtual scenes” possible, reshaping the audience’s perception of the stage space through a highly realistic three-dimensional environment.This technology can not only accurately reproduce scenes in the real world, but also create a fictional world that completely transcends reality according to the creator’s imagination. This spatial flexibility and expandability greatly enriches the expressive power of the musical and the depth of the audience’s experience. The concept of “action space” has been newly interpreted on the virtual reality stage. In the virtual reality environment, the actor’s performance is no longer limited by the constraints of physical space, the action space is greatly expanded. In this environment, actors are able to move and perform freely in a wider virtual space, and at the same time interact with virtual elements, bringing more possibilities and dynamics to the performance. The “viewing space” occupies a crucial position in the construction of virtual reality stage space. This concept reshapes the traditional viewing experience, enabling the audience to freely choose their viewing position and angle in the virtual environment, and even interact with the performance. This personalized viewing experience not only enhances the immersion of the audience, but also allows each audience member to enjoy a unique performance experience. Through an in-depth discussion of these three core elements, we are able to more fully understand how virtual reality technology has redefined the stage space in the art of musical theater and opened up a new field of artistic exploration. This not only demonstrates the infinite possibilities of the integration of technology and art, but also points out a new direction for the future development of musical theater art.

Unity3D is a virtual reality presentation by Unity Technologies Unity3D can be said to be the overthrow of the traditional way of presentation, growing the expressive power, and most of all to be able to benefit the colleagues of the working group in the practical application, saving time and at the same time to be able to see the realistic effect. Under this demand, we use modern technology, combined with the choreography, of course, at the same time it also brings challenges to the work of the choreographer, to be very detailed consideration of every little detail, at the same time this is their opportunity to perform.

3dsMax is used to build the scene, modeling, character modeling software, as well as photography and lighting, etc., can be more realistic to show the required scene content. (1) Simulation model construction: the modeling lines have been done with the “extrusion” option, to be able to turn the line into a body. (2) props simulation construction: the grass, trees, balconies, chairs, etc. to build a collection of grass, trees, balconies, chairs, and then modify the content. To use 3dsMax system TreeStorm to simulate the simulation. (3) task simulation: at first, we should start from the skeleton structure, and then some points connected to the line using the command to build into the surface. (4) In the creation panel of 3dsMax, then take the light and camera projection, and edit the surface.

Not only need to build the stage three-dimensional three-dimensional model, but also need texture data. That is, by establishing the one-to-one correspondence between the spatial coordinates of the feature and the spatial coordinates of the texture, the two-dimensional feature texture image is mapped onto the surface of the three-dimensional feature model. The essence is to use the 2D plane image to replace the simulatable or non-simulatable details in the feature model. Typically, 2D texture data can be defined as an object with one gray value in texture space, either by analytic representation of a mathematical function or by discrete definition using a digitized image. Mostly, we realize the affine relationship between 2D texture data and 3D object space by affine variation.

For a larger number of detail parts, complex structure and easy to increase the number of required surfaces, a mapping texture can be used instead. This not only reduces the overall number of surfaces, but also reduces the overall landscape model complexity and improves the display speed when rendering the landscape model. In order to reduce the number of surfaces and the possibility of errors, the use of Boolean operations and cutting tools should be minimized when modeling fine models. At the same time, make sure that the normal direction of the two face models of Boolean operation is the same, to prevent the generation of bad faces and waste faces. In order to reduce the size of the output file in the later stage, you should try to use as many straight lines and straight faces as possible, and minimize the number of fixed points and faces on curves and surfaces. In order to prevent operational problems with translations and rotations, the model should be changed to stack collapse as much as possible when exporting the 3D model. Delete redundant, unused faces and invisible backs, minimize redundant polygons, reduce the number of 3D model face points, reduce the overall number of 3D scene polygons, and achieve landscape model optimization. The use of the 3DMAX object’s associated copy (Instances) algorithm makes it possible to increase only the number of similar objects without increasing the 3D scene running overhead and the overall number of rectangular face models. However, if you change any of the attributes of a similar object, such as size, pose, material, and texture mapping, the attributes of the other similar objects will change as well. The use of associative copying can greatly reduce the size of the document.

The human-computer interaction stage design process is shown in Figure 2, where the theater uses visual images to realize different sensory experiences. It can also combine infrared human-computer interaction technology, mobile interaction technology, augmented reality technology, so that when the viewer communicates with the screen, it can drive the image and thus form a randomized digital art effect. The introduction of new media technology into the opera performance stage mainly broadens the presentation mode of opera performance. For example, with the help of new media technology, the surround screen outside the theater plant carries out the real-time interaction of the character image in the play following the viewer’s movements. When the walking viewer stays in front of the screen and makes movement changes to the screen, the new media equipment will capture the viewer’s movements, so that the viewer’s movements can be fed back to the screen in a timely manner, and the viewer’s body movements and gestures can be captured by the 3D somatosensory camera, and the new technological art is presented through the new technology. Through the 3D body sensing camera to sense and capture the viewer’s body movements and gestures, the visual image is shown through the new technology and art presentation and data analysis to achieve the human-computer interaction mode.

Figure 2.

Human-computer interaction presentation technology flowchart

Inside the theater, there is a large LED screen combined with screen projection to show the real and unreal performance effects, through multimedia, three-dimensional modeling technology, real-time video transmission and display control technology, multi-sensor fusion technology, etc., the scene presented in the plot will be virtualized and superimposed on the real theater space. Through the new media AR technology with the help of computer vision presentation method, establish the mapping relationship between the real world and multiple screens, 3D modeling of the drawn scenes in the play attached to the large screen, 3D image projection in the 2D screen and add positioning of the virtual scenery or characters in the three-dimensional stage space.

In recent years, musical stage art creators are constantly exploring new forms of performance, and are committed to exploring new forms of interpretation so that the immersion of the performance can be fully developed. The development of stage performance forms is shown in Table 1, “Thunderstorm” is one of the classics of Chinese drama performance, which was premiered in December 2, 1934 in Zhejiang Province and has been brought to the stage many times since then, and it is the reserved repertoire of Beijing People’s Art Theatre. The Phantom of the Opera premiered at Her Majesty’s Theatre in London on October 9, 1986 and won seven Tony Awards in 1988, making it one of the most successful musicals of all time. The musical has been a box office hit to this day whenever there is a touring production. The realistic feel of the stage design, the vivid performances of the actors, and the beautiful music keep the audience clapping. Since its premiere in London on May 11, 1981, the musical “Cats” has been translated into more than 20 languages and has been performed in every corner of the world.In 2019, it was re-performed in Kunming, and due to the big ticket sales, the number of performances was increased by several days, and the performance venue of the actors was extended from the stage to the audience, so that the audience could watch the actors’ performances up close and personal.The end of the Kunming tour of the musical “Cats” received a lot of positive reviews. After the tour in Kunming, “Cats” received a lot of favorable comments.

The immersive musical Alice’s Adventures in Wonderland was first officially staged in October 2015 at the Watts Theatre, located in Waterloo, London, U.K., and earned an excellent sell-out crowd.The emergence of the immersive show Sleepless Nights on December 14, 2016, explored a new narrative form for the development of stage performances. Immersive musical performances allow the audience to watch actors perform up close and personal, and watch different branches of the plot by following different actors. This approach is more in line with VR movie viewing, where the user can choose which performance they want to see in the open world, and follow different branching routes to see different plot developments. This way of viewing gives the audience more space for choice, transforming the passive mode of viewing into an active collection of information, which greatly mobilizes the audience’s curiosity and desire to explore.The 3D sci-fi stage play “Three Bodies II” premiered in Shanghai on April 21, 2019, and the show extends the stage space by means of digital media technology and light and shadow technology, and some of the scenes in the performance process require audiences to wear special 3D glasses to view the show. The 3D sci-fi stage play is another exploration of stage performance forms, although the form is very innovative, due to the limitations of theater viewing, the placement of stage props in the space affects the viewing experience of the audience in the back rows and the immersive experience. Despite the mixed feelings, the development of the performance form has shown a more inclusive and open state in front of the audience. The development of stage performance presents a more open form with the advancement of technology, and stage creators are constantly integrating new technologies and combining new platforms to find a more marketable interpretation form for stage performances.

In order to verify the effect of this paper’s 3d Max method for designing 3D virtual stage scenes, this paper’s method is used to design a stage landscape. In order to highlight the quality of the stage virtual scene designed by the method of this paper, the stage virtual design method based on Vega and the stage virtual design method based on Web3D are used to compare with the method of this paper. The experiments were conducted using the three methods for virtual simulation of the five sub-models of the stage model, and the comparison of the fidelity of the virtual stage scene designed using each method. The comparison results are shown in Table 2, the average value of the fidelity of the stage virtual design images using this paper’s method is about 24.8% higher than the average value of the fidelity of the stage virtual design based on the Vega method, and about 26.8% higher than that of the stage virtual design method based on Web3D. The fidelity of each sub-model designed by this paper’s method is not less than 90%, which plays a stable role, and relatively speaking, the 3D virtual stage scene designed by this paper’s method is more realistic.