Digital Intelligence Technology-Driven Transformation and Innovation Research on Choreography Design

1)

Large model selection. Stable Diffusion (SD) is a deep learning model based on artificial intelligence, which belongs to a kind of Generative Adversarial Networks (GAN), and is specifically applied to the field of image generation, and is a powerful authoring tool [33]. WebUI is the web interface of SD, and by analyzing a large amount of image data and its related textual descriptions, SD learns to transform textual information into visual images, which opens up digital authoring to a a whole new dimension, making the conversion from text to image possible.

1)

LoRA is a fine-tuning technique for tuning large language models (LLMs) [34]. It significantly reduces the number of trainable parameters, memory requirements, and training time by introducing low-rank matrices at each layer of the Transformer architecture and training only these matrices while keeping the original model weights unchanged. LoRA is a Parameter Efficient Fine-Tuning (PEFT) technique that achieves model tuning by introducing a small number of parameters while keeping the original model unchanged. With the LoRA model, customized tuning can be performed to fit a specific style and maintain high operational efficiency when generating images using SD tools.

LoRA, like Control Net, uses a small amount of data to train a painting style or character without modifying the SD model to achieve the customization requirements, and requires much smaller training resources than training the SD model, which is expressed in the data equation (1) as follows: (1)W=W0+BA

Where W₀ is the parameter of the initial SD model (Weights), BA is the parameter of the LoRA model, and W represents the parameter of the final SD model after being influenced by the LORA model. 2)

WebUI image generation process: select model - input prompt words - adjust parameters - click to generate images, the actual machine operation process: prompt words prompts - lexical elements Tokens - text EncoderText Encod-er-Set parameters-Noise Predictor-Noise Predictor-Variable Auto-Encoder VAE-Output image. The core principle of LoRA model training is to embed the “repeated impressions” in the noise predictor network.

In the field of stagecraft, for lighting designers, the dataset can be fixture specifications and documentation, and for stage designers and costume designers, the dataset can be costume styles, well-known designers and works of different style genres.

Generative adversarial network consists of a generator and a discriminator. Where the generator is responsible for learning the sample distribution and generating the most realistic image possible [35]. The discriminator is responsible for recognizing the real data and the generated data. The model training ends when the data generated by the generator successfully deceives the discriminator. The loss function of the generative adversarial network can be expressed as equation (2): (2)minmaxV(D,G) = Ex~pdata(x)[logD(x)] +Ez~pz(z)[log(1−D(G(z)))]

where z denotes random noise, real data is denoted by x, generator is denoted by G, and discriminator is denoted by D, G(z) denotes samples generated by the generator network, D(G(z)) denotes the output result of the discriminator on the samples generated by the generator, and log(1−D(G(z))) denotes the result of misclassification of the generator-generated samples as real samples by the discriminator. E denotes the expectation. D(x) denotes the result of the discriminator’s output of the real sample. log D(x) denotes the result of the discriminator’s judgment of the generator-generated sample data. E_x~pdata(x) denotes the expectation of the true sample. E_x~pdata(x)[log D(x)] When maximized, the goal of the discriminator is to correctly classify the true sample as a true sample, thereby improving its accuracy in identifying true and generated samples. The generator and the discriminator are trained by the model and continuously optimized to finally reach the Nash equilibrium.

The diffusion model constructs a Markov chain that uses a diffusion process to transform known distributions (e.g., Gaussian) into target data distributions. The diffusion model is shown in Figure 1.

Figure 1.

Diffusion model

where x₀ denotes a real data observation, such as a natural image, and x_t denotes a pure Gaussian noise, and the diffusion process starts at x₀ and follows this chain structure of gradually adding Gaussian noise to obtain pure Gaussian noise x_t. q(xt|xt−1) denotes a conditional probability distribution for the state x_t at the current time step t, conditioned on the state at the previous time step x_t−1. This distribution uses the output of the previous state as its mean. The joint probability distribution of the entire network is denoted by p(x0;T) . If this joint probability is chain-split in left-to-right order it can be expressed as equation (3): (3)p(x0;T)=q(x0)∏t=1Tq(xt|xt−l)

The latent diffusion model (LDM) belongs to the conditional control model significantly reduces the computational requirements compared to pixel-based diffusion models. The latent diffusion model can not only be trained on limited computational resources, but also ensures the quality and flexibility of the generated images.

Stabilized Diffusion (SD) model is proposed on the basis of potential diffusion model, the difference between stabilized diffusion model and potential diffusion model is that stabilized diffusion model uses CLIP as a text encoder and self encoder (VAE) as an image encoder. The stable diffusion model uses LAION-5B as the dataset for model training. The similarity between the stable diffusion model and the potential diffusion model is that both use U-Net network.

1)

Attention mechanism

In order to further enhance the expression ability of object embedding vectors, the attention mechanism is added to the graph convolutional network, i.e., the attention coefficients are assigned to the neighboring nodes when the aggregated neighboring nodes output the feature vectors, and thus the ability of the object to perceive other neighboring nodes can be enhanced.

A shared parameter matrix is first used W∈RF1F2 . Where parameter matrix W represents the weight parameters in the model used to transfer information between different layers and to perform feature transformations, and R represents the real number domain. F₁ denotes the number of matrix rows, and F₂ denotes the number of matrix columns. The object vectors and relationship vectors in the scene graph are transformed into high-level feature vectors using the parameter matrix, so as to make the object and relationship feature vectors more expressive, and the attention coefficients between the objects are calculated using the high-level feature vectors of the edges in the scene graph. The calculation formula is expressed as equation (4): (4)eij=φ(W[hoi,hrk,hoj])

where (hoi,hrk,hoj) is the vector of object embeddings corresponding to edges in the input scene graph, and φ denotes the attention computation network. e_ij is the degree of contribution of any edge (oi,r,Oj) instance object O_j to instance object O_i of the scene graph obtained after computation. 2)

Scene Layout

In this paper, we use a scene layout network based on a regional convolutional network using four parameters t_x, t_y, t_w, t_h as learning objectives. t_x, t_y are used to adjust the offset of the center point position of the bounding box. By adjusting the center point position, the position of the object in the image can be located more accurately. t_w, t_h are used to adjust the scaling of the width and height of the bounding box. t_x, t_y, t_w, t_h are obtained through regional convolutional network learning, and the bounding box can be adjusted according to the extracted features and the features of the target object, so that the final target box more accurately frames the target object and improves the accuracy and stability of target detection. Parameters t_x, t_y, t_w, t_h are calculated as shown in Eqs. (5)~(8): (5)tx=Gx−PxGw (6)ty=Gy−PyGh (7)tw=log(GwPw) (8)th=log(GhPh)

where G_x, G_y denote the coordinates of the center point of the bounding box provided by the real data, G_w, G_h denote the width and length of the border provided by the real data. P_x, P_y denote the coordinates of the center point of the predicted bounding box, P_w, P_h denote the width and length of the border of the predicted data. The learned set of parameters is denoted as W∈RF2×4 , and the object vector h_o is operated by Eq. to obtain the bounding box transformation parameter vectors t_*, *∈(x,y,w,h) , viz: (9)t*=Wh′0

First, a border P⁰ = (x, y, w, h) is initialized for the object, and the coordinates of the center point of the border of the generated image are set to Px0 . Py0 The length and width of half of the generated image are set to Pw0 , Ph0 . The four learning parameters are continuously optimized through training to make the border close to the real border. Through continuous training and learning, the border transformation parameter vector t_* is gradually brought closer to the real data provided by the border regression network t^* . The formula for t^* can be expressed as Eqs. (10) to (13): (10)t^x=Gx−PX0Gx (11)ty^=Gy−Py0Gh (12)tw^=logGwPw0 (13)th^=logGhPh0

1)

CLIP text encoder

The attention mechanism uses the multi-head attention mechanism to introduce the three concepts of Query, Key and Value, Query denotes the query, which is represented by the symbol Q. Key denotes the key, which is represented by the symbol K. Value denotes the value, which is represented by the symbol V. By calculating the correlation between Q and K, the attention weights are obtained, and then the weights are used to weight the combination of V, and finally the output of the attention mechanism is obtained. The attention mechanism expression is represented as equation (14): (14)Attention(Q,K,V)=softmax(QKTdκ)V

where Q, K, V are the matrices obtained by doing a linear mapping of the embedding vectors X_embedding1 of the previous layer. QK^T is the attention matrix. The attention function used is the dot product attention function, and dk denotes the dimensions of Q and K. The matrix is made normally distributed by scaling factor. Normalization is done using Softmax function and finally the output of the attention mechanism is obtained.

The text embedding vectors output from the text encoder are high dimensional, and when the data is limited, the text description cannot fill the hidden space, and the data flow shape is discontinuous, which affects the training effect, so the conditional enhancement technique is introduced to get more hidden variables by using random sampling in Gaussian distribution, and the expression for the Gaussian distribution is shown in Eq. (15) [36]: (15)N(U0(φt),∑0(φt))

where ∑0(φt) denotes the diagonal covariance matrix and U0(φt) denotes the mean function of the text embedding vector (φt) . In this way, the model can generate more text-image training pairs on limited text-image data for deeper understanding of the hidden information in the text, thus enabling the sublayer network to generate clearer images. 2)

Residual network generation

The residual network can be used to deeply understand the multi-peak features of text and images, and this network can alleviate the model gradient vanishing as well as the overfitting problem. The first layer of the network will generate the image through a convolutional kernel window size of 4 × 4, step size of 2 convolutional layer, batch attribution layer and LeakyReLU activation function composed of the network for downsampling, to get the image features. The combination of image features and hidden variables are fed into the residual network for learning the multimodal representation between the image and the hidden variables to improve the image accuracy. The residual network consists of a series of residual blocks, in this paper, four residual blocks are used, and each residual block can be expressed by equation (16): (16)xl+l=h(xl)+F(xl,Wl)

The residual block consists of two parts, the direct mapping and the residual, where h(x1) is the direct mapping and F(xl,Wl) is the residual. Denotes the residual block. F(xl,Wl) is the residual, which consists of a convolutional layer with a convolutional kernel window size of 3 × 3 and a step size of 1, a batch normalization layer and a ReLU activation function. The learned features are upsampled to get the output image. The objective function of the sublayer network generator is expressed as equation (17): (17)L(G)=Es0~pG0,t−pdata[log(1−D(G(s0,c^),φt))]

Where G₀ denotes the first layer network generator, G and D denote the generator and discriminator of the second layer network respectively, s₀ denotes the output of the first layer network of CGADM, c^ is the hidden variable generated by the conditional enhancement technique, φ_t is the text embedding vector obtained from the text through the text encoder, and t denotes the text. 3)

Match-aware discrimination

The discriminator of the sublayer network uses a match-aware discriminator. In the training process, the discriminator divides the real image, the generated image and its corresponding text description into two cases: positive sample and negative sample. Only the combination of the real image and the matching text is a positive sample, and the rest are negative samples The discriminator uses a deep convolutional neural network containing two downsampling layers, the downsampling layer consists of a convolutional layer with a step size of 2 and a convolutional kernel window size of 4 × 4, a batch normalization and a Leaky ReLU.

The objective function of the sublayer discriminator network is expressed as equation (18): (18)L(D) = E(I,t)~Pdata[logD(I,φt)] +Es0~pG0,t~Pdata[log(1−D(G(s0,c^),φt))]

Where G, D, G₀, z, c^ , φ_t represent the same meaning as represented in the objective function of the generator, and I represents the real data. When L is maximum it means that the training is completed and the discriminator network realizes the classification between positive and negative samples. The final model reaches convergence by alternating training of the residual network generator and the match-aware discriminator.

First, the Mosaic data enhancement technique significantly improves the system’s ability to adapt to various scenes and the accuracy of target detection by randomly cropping, scaling, and rearranging the images. In addition, all images are resized to a uniform size at the input side in order to maintain the consistency of the training data and improve the processing speed of the model. This step ensures the standardization of data input and lays a solid foundation for model training.

The Backbone architecture integrates three key architectures, Focus, CSP and SPP. the Focus architecture specializes in image slicing and generates finer-grained feature maps through convolutional operations. The CSP structure introduces a varying number of residual components to further optimize the feature extraction process. SPP (Spatial Pooling Pyramid) enhances the model’s adaptability to changes in image size by performing a maximum pooling operation on feature maps of different sizes.

The role of the Neck module is to connect the Backbone to the Head, which contains both FPN and PAN structures. FPN efficiently conveys feature information through a top-down approach, while PAN employs a bottom-up approach that enhances the detailed representation of features.

Finally, the Head module disposes of the mismatch between the prediction frame and the target frame through loss function and non-maximum suppression (NMS) operations, and effectively removes redundant prediction frames through a weighting method, which increases the accuracy and clarity of detection.

During the matching process, each feature point is assigned the corresponding 3D coordinates one by one. By applying the Nearest Neighbor Rule Classification Matching Algorithm (KNN Algorithm), the distance between each sample, as well as the difference in their directions, can be calculated to minimize the error in the matching process.

The calculation of the distances follows equation (19): (19)d=∑i=1n(Ai−Bi)2

where: n is the number of dimensions, A_i and B_i are the values of the two feature points in the i-dimensional space, respectively, and d is the distance between the two samples.

In order to measure the difference in direction, Eq. (20) can be utilized to calculate the direction cosine value between two feature points: (20)cosθ=x1x2+y1y2x12+x22×y12+y22

Where: x₁, y₁ are the coordinates of the 1st actor, x₂, y₂ are the coordinates of the 2nd actor, and cos θ is the cosine value of the angle formed by the two actors and the coordinate origin.

In the process of calculation, the system will take the center of the stage as the origin, establish a coordinate system with a spacing of 0.5m, and calculate the 3D coordinates of each matching point group through the built-in parameters and position of each camera device, and then after deriving the 3D coordinates, the system will convert the 3D coordinates of each matching point group into the corresponding 2D coordinates of the projection onto the stage in the way of perspective projection.

Cross dataset training, i.e., training the localization algorithm by collecting additional data as a supplementary training set. By collecting data under different environments, different lighting conditions and different actor postures as supplementary training sets, the stability and extensiveness of the system can be effectively improved, thus enhancing the system’s adaptability to situations that may occur during improvisation.

And the motion posture prediction can be divided into two cases. The first case is uniform motion, i.e., the actor’s route in a certain time is uniform, in this case, the system can calculate the next coordinate point based on the 2D projection coordinates of the previous feature points, as shown in Eq. (21) and Eq. (22): (21)xnext=xcurrent+t×(xcurrent−xlast)tp (22)ynext=ycurrent+t×(ycurrent−ylast)tp

Where: (Xcurrent,Ycurrent) is the current coordinates, (Xlast,Ylast) is the last coordinates, (Xnext,Ynext) is the next coordinates, t is the next time to be predicted, and t_p is the time sampling period.

By using the above formula, the next route of the actor can be effectively predicted.

The second case is non-uniform motion, i.e., the actor carries out irregular, and the speed changes at any time on the stage. In this case, the system can ensure real-time by its own real-time tracking technology as well as high-speed computation capability to output images in a very short time.

Color theory provides a series of guiding principles on how to influence the viewing experience through color combinations, such as color harmony, contrast, and color symbolism. The direction of the audience’s emotions can be guided through color combinations in stage design, e.g., warm tones are often used to create a warm, exciting or tense atmosphere, while cool tones are used to express calmness, sadness or contemplative emotions.

Lighting and lighting design is an extremely core component of choreography. Lighting not only provides the necessary illumination, but is also a key tool for constructing the atmosphere of the scene, guiding the audience’s emotions and reinforcing the visual impact, the core of lighting design lies in the mastery of the quality, direction, color, and intensity of the light as well as how to interact with the stage space and the actors through these elements. Lighting design must be closely integrated with the content of the script and the visual intent of the director, supporting the development of the plot and deepening the expression of emotions through changes in lighting.

Spatial layout focuses on how the stage space is utilized including the depth, width and height of the stage as well as the configuration of reality and reality in the scene, while composition involves how the visual elements of the stage are distributed in the space, which includes the position of the actors, the arrangement of props and the design of the backdrop. Effective use of space can guide the audience’s eyes to create a visual focus, so that the audience’s attention is focused on the key elements or actions on stage.

1)

Eye movement

In previous studies, the presentation time of the stimulus materials for choreography was usually controlled at 7 s to 20 s depending on the number and type of the stimulus materials, and in this experiment, the Tobii Pro Glasses 2.0 eye-tracking device was used to collect data to present the samples through a screen with a resolution of 1,920 × 1,080, and the presentation time of the choreography samples was 10 s, with a gap of 2 s between the photographs, and all the photographs were randomly shown after being disrupted in order, and only one subject was allowed to enter the experimental site during the experiment. All photos were randomly played after disrupting the order, and only one subject was allowed to enter the test site during the experiment. The sampling rate of the eye movement data obtained in this experiment reached more than 80%, which was in line with the collection standard, and the data were reliable. After the eye movement experiment was completed, subjective data were collected from 20 samples.

Table 1 shows the analysis of the sample gaze time, this paper counts the gaze time of the subjects in each region in the sample by dividing the eye movement interest region, and quantifies the eye movement indexes to analyze the degree of eye-catchingness, degree of attention, and attractiveness of the spatial elements of each choreography design. After analyzing all the samples, this paper divided the spatial elements in the samples into the following six categories: spatial interface elements, costume elements, prop elements, choreographic composition, lighting elements, and set elements, and computed and analyzed the average gaze time of automatic optical inspection (AOI) in each region. In order to reduce the experimental error, the boundaries of each region were divided in such a way that they followed the original element boundaries as much as possible, and the boundaries of each region did not overlap. Among the six categories of choreographic design space elements, the attention of choreographic composition is significantly weaker than that of the other five categories of choreographic design space elements, indicating that the choreographic composition elements are less eye-catching and lack attractiveness, but they are more recognizable and comprehensible, with a total gaze duration value of 3.95 s. The total duration of attention is 3.95 s, which is the same as that of the other five categories of choreographic design space elements.

Props elements are widely present in the choreography scene and account for a larger proportion in some samples, but the degree of eye-catchingness is weaker than that of other major elements in the choreography scene in subjects’ observation, with a higher degree of recognizability but a slightly weaker attraction to the subjects. The spatial elements that received more attention from the subjects were the spatial interface elements and set elements, which were more visible and attractive than the other four spatial elements, with the total gaze duration of 104.624 s and 63.552 s. However, due to the large amount of information contained in the elements, the subjects’ comprehensibility and recognizability of these two types of elements were lower.

As the primary component of spatial form, spatial interface elements influence the perception of space. As secondary elements, spatial interface elements are more eye-catching in choreography, but in terms of attractiveness, they are again weaker than elements.

The normality test was performed on the overall eye movement data of the scene and the eye movement data of each AOI using spss26.0 software. Table 2 shows the S-W test results of the overall eye movement indexes, according to the S-W test results of each index the absolute value of the kurtosis is less than 10 and the absolute value of the skewness is less than 3, the kurtosis is 0.02796, 1.03498, 0.63486, and the skewness is 0.31486, 0.57964, -0.39486, respectively. Figure 3 shows the histogram of the overall normal distribution of the eye movement indexes, which shows that the overall eye movement index data distribution satisfies the normal distribution, and the mean values of the three indexes are 1.97909, 1.3409, and 3.38583, respectively.

Figure 3.

The overall normal distribution histogram of the eye movement index

2)

Heart rate variation

Due to the complexity of the human physiological signals and the limited precision of the measuring instruments, the values fluctuate greatly, and the demand for local data analysis is very small, and it is only suitable for analyzing the general trend of a large amount of data. In this paper, the Euclidean distance calculation is performed on the physiological signal data of the simulated subjects in different color environments of the choreography scene to analyze the differences and then compare the differences of human physiological state in different color environments. Specifically, Euclidean distances were calculated for 40 groups of physiological signals under the simulated test environment of light blue, white, warm white, orange, red and green, respectively, and 40 groups of physiological signals under the natural conditions of the choreographic environment. As shown in Eq. (33), D represents the sum of physiological signal data acquired by a single subject during 10 min, i.e., the product of sampling frequency and time. a_k and b_k represent the physiological signal data in the natural conditions of the choreographic environment and the physiological signal data in the simulated experimental environment, respectively. D(a, b) represents the Euclidean distance between the two, reflecting the difference between the physiological signal data in different simulated test environments and the choreographic environment: (33)D(a,b)=(∑k=1d(ak−bk)2)1/2

Heart rate is the number of times the heart beats per minute and is the most common measure of cardiac activity. A normal adult’s heart rate is about 70 beats per minute and an athlete’s heart rate is about 50 beats per minute. In a choreographic environment, it is important for the audience to maintain a smooth heart rate activity.

In the simulation of the test environment for stage performances to watch the heart rate signal measurement experiments, T-Sens heart rate sensor sampling frequency of 16Hz, the T-log wireless data logger in the SD card data reading, by CAPTIV software processing can be obtained within 10 minutes, 40 subjects heart rate data, due to the sample size is too large, this paper only placed four typical convergence of the collection of results Due to the large sample size, only 4 typical convergent acquisition results are placed in this paper. Fig. 4 shows the heart rate signal acquisition results of 4 subjects, Fig. (a)~(d) represent the 1st-4th subjects respectively, and the mean values of heart rate of 4 subjects in 10 minutes range from 70 to 86 beats/min.

Figure 4.

The heart rate signal of four subjects was collected

1)

Color

The Euclidean distances of the six simulated experimental environment colors of the choreography design were compared with those of the natural environment heart rate signals, and Figure 5 shows the results of the experiment. Warm white has the smallest Euclidean distance among the six choreography colors, ranging from 800 to 1,500 beats/min, and the heart rate signals in the choreography design color simulated experimental environment are closest to the Euclidean distance in the natural indoor environment.

Figure 5.

Test result

Figure 6.

The tracking algorithm is solved

Figure 7.

Dance beauty spatial feature D/H ratio

The visual entropy of the designed choreography is calculated by writing the program language through matlab, the operation method is to read in the choreography image and grayscale it, obtain the grayscale histogram of the image, calculate the probability of each grayscale in the image appearing in the image, and calculate the entropy by the definition of entropy. Figure 8 shows the visual entropy statistics of the choreography scene, through the program calculation, it is found that the visual entropy of each choreography scene is between 7 and 8.5, mainly concentrated between 7.5 and 8, a few lower than 7.5 and higher than 8, and the numerical difference is reflected in the 2 decimal places, which on the one hand indicates that the choreography image samples collected in this collection are very uniform in terms of the type, and the information is close to the information, which is helpful for concentrating the focus of the research. On the other hand, it shows that there is a difference in the visual information carried by the visual elements of the external space of the choreography, and this difference is reflected in the details of the color composition, material texture and so on. The data of visual entropy are standardized and data segmented, which are divided into 3 segments: low visual entropy (7.30~7.50), medium visual entropy (7.50~7.70) and high visual entropy (7.70~7.90).

Figure 8.

Visual entropy statistics for dancing scenes

Table 3 shows the statistics of the audience’s overall perception, and the analysis of the audience’s overall perception in this paper is divided into two aspects: the audience’s communication method and overall feeling towards the stage performance. The results presented in the survey indicate that the evaluation of the communication mode of this stage performance is very good and good evaluation of the majority, accounting for 62.5% and 17.5% of the total number of people respectively. At the same time, 65% of the audience thought that the overall feeling brought by the stage performance was very good, and no audience thought that the overall feeling of the stage performance was very bad.

The audience’s satisfaction with the stage performance elements of the survey is divided into four aspects of the overall stage modeling, stage lighting effects, stage performance and stage scheduling form of generalization, satisfaction scores in accordance with the level of scoring, very satisfied with 5 points, more satisfied with 4 points, and so on. Table 4 for the audience on the performance of the elements of satisfaction analysis, the audience stage lighting effect satisfaction is the highest (score of 178), the rest of the satisfaction in descending order: the form of stage scheduling (score of 171), the overall stage modeling and stage performance scores are equal, are 170 points.

Table 5 shows the performance music audience satisfaction, performance music audience satisfaction from high to low in order of ranking: music and stage picture between the cooperation, music and lighting between the rhythmic sense of embodiment, music and performance style combination, music, sound production effect, satisfaction with the total score were 184, 179, 176 and 174.

Table 6 shows the other sensory experience, in the stage performance, added a perfume rain link, this link experience satisfaction survey, the audience’s satisfaction in descending order: perfume rain and stage atmosphere catering to (satisfaction score of 181), the olfactory experience of perfume rain (satisfaction score of 176).