Automatic Generation of Cinematic Animated Characters and Their Behavioral Characterization Using Graph Generation Networks

Translation refers to the process of relocating a point along a linear path from one coordinate position to another. The formula for translational transformation is: (1) [ x′y′ ]= [ x+Txy+Ty ]

Where T_s, T_y are called translation vectors and indicate that the point has moved a distance of T_x and T_y in the x and y directions.

Proportional transformation refers to a point relative to the origin of the coordinates along the x-direction deflation S_x times, along the y-direction deflation S_y times. The formula for proportional transformation is: (2) [ x′y′ ]=[ xy ]⋅[ Sx00Sy ]

Rotation is the process of relocating a point p by turning it by an angle θ (counterclockwise is positive, clockwise is negative) around the coordinate origin to obtain a new point p′. Its formula is: (3) [ x′y′ ]=[ xy ]⋅[ cosθsinθ−sinθcosθ ]

In the application of graphics, it is sometimes necessary to produce a unisexual object deformation processing, which will use the wrong cut transformation, also known as shear, dislocation transformation. Its calculation formula is: (4) [ x′y ]=[ xy ]⋅[ 1bc1 ]

In the animation production, the key frame can be played directly, while in the two key frames in the middle of the ordinary frame need to do complementary animation, in order to realize the movement of the picture, such as animation character action effects, scene changes and so on. Complementary animation is calculated by the program and does not require the user to draw or edit it. Complementary animation is one of the very important functions, using it, you can set the size, position, color, transparency, rotation and other attributes of the symbols, and can change a variety of wonderful incredible dazzling animation effects, and graphic movement and deformation of natural and smooth. When the animation is running, every frame has an image in it. Image is a bitmap composed of pixel points, can use digital arbitrary description of pixel points, intensity and color, is the object to a certain resolution resolution after each point of information presented in a digital way, can be directly and quickly displayed on the screen.

Frame, as the smallest unit in animation, contains a single image frame, which represents a grid on the timeline of animation software, representing a point in time. The key frame is the sudden change point of the graphic image, the qualitative change point of the picture, which is equivalent to the original painting in 2D animation, referring to the frame where the key action in the movement or change of the component or graphic image is located. The animation between the key frame and the key frame can be calculated automatically by the program, which is also known as the ordinary frame or intermediate frame, which is known as complementary animation.The concept of keyframes and complementary animation has long existed in traditional cartoon production, where skilled painters design the keyframes of the cartoon, and then the intermediate frames are designed by average painters. In short, fill-in animation is a gradual process, which is a quantitative process from one keyframe to the next keyframe.

The calculation process of motion interpolation animation is actually an interpolation process, which interpolates the symbols in ordinary frames according to the difference between the symbols in the first and the last two key frames, so that the symbols change smoothly from the first key frame to the second key frame. This is also the significance of proposing the transformation matrix in the symbols. The principle of generating motion filler frame animation is to interpolate the transformation matrix of the symbols in the first and last keyframes and the position of the common frame to find the corresponding symbols in the common frame. The interpolation is a uniform process, and the formula for calculating the amount of change in the nst frame is: (5) δ=Δ⋅(n/Num)

During the animation, the graph will gradually become twice the size of the first graph, and the size of the graph is incremental, i.e., it will not become smaller and then larger in the middle. Therefore, the variables in the transformation matrix also change gradually, increasing or decreasing. But the speed of change can be different for each frame, starting with an introduction to the concept of tween_ease, the acceleration between complements.

Acceleration in physics is the ratio of the amount of velocity change and the time taken for this change to occur, it is a physical quantity that describes how fast or slow an object’s velocity changes, and its formula is: (6) s=v0t+1/2at2

However, Gong its not applicable to the complementary acceleration tween_ease, because the amount of change between two keyframes is fixed, and the time of change is also fixed, which is not enough to calculate the complementary acceleration tween_ease, and thus the formula is used to calculate the complementary acceleration, that is: (7) δ+(1+a)δ+(1+2a)δ+⋯+(1+(number−1)a)δ=Δ

Where Δ indicates the first two key frames for the amount of change between the motion patch, number indicates the number of frames before and after the two key in the middle of the frame, a indicates the patch acceleration, δ indicates the amount of change between the first pause and the second pause, (1+a)δ indicates the amount of change between the second frame and the third frame, and so on.

The formula can be introduced: (8) ration=(2n+n(n−1)a)/(2num+num(num−1)a)

where a is the complementary acceleration tween_ease, num is the number of frames between the first and last keyframes, and ratio_n is the ratio of the transform of the nth frame of the complementary interval to Δ.

Deep learning is derived from artificial neural networks and is characterized by deeper network layers and more complex network structures than the original artificial neural networks. In the early days when artificial neural networks were proposed, the number of layers of neural networks was generally shallow due to the limitations of the machine’s arithmetic power and storage capacity, and the corresponding data processing capacity of these networks was weak. Accompanied by the further development of core hardware such as CPU and GPU in scientific computers, neural networks are able to achieve improved data processing benefits through layer stacking and complex structure design.

The original generative adversarial network is based on game theory ideas, and the corresponding overall framework is mainly divided into two parts, one part is the generator, and the other part is the discriminator, which are built by two separate groups of deep neural networks.

In the entire adversarial training process, the goal of the generator is to deceive the discriminator, and the corresponding loss function of the generator during training is described as follows: (9) minGEz∼Pz[ log( 1−D(G(z)) ]

The goal of the discriminator is to distinguish between the generated false image and the real image, its corresponding input is a picture, and the corresponding output is the judgment of the authenticity of the picture, its loss function is as follows: (10) maxDEx∼Pr[ logD(x) ]+Ez∼Pz[ log( 1−D(G(z)) ]

Where D is the discriminator, G is the generator, P_r is the distribution of the real data, x is the sampling of the real picture, P_z is the noise distribution of the input to the generated data, and z is the sampling noise.

The two loss functions can be described jointly: (11) minGmaxDV(D,G)=minGmaxDEx∼Pr[logD(x)]+Ez∼Pz[log(1−D(G(z))]

For the field of generative adversarial networks to realize image generation, a deep convolution based generative adversarial network DCGAN (GAN) has been proposed. In the network model of DCGAN, all fully-connected hidden layers present in the original generative adversarial network are removed.Most of the structural designs suggested by DCGAN are adopted by a large number of subsequent variants of generative adversarial networks.

In order to be able to improve the stability and efficiency of training, a batch normalized neural network is added to all network layers except the output layer of the generator and the input layer of the discriminator. For the activation function, the effect of various types of activation functions was tested and a better design was finally arrived at: for the generator, the Relu activation function was used for all layers except the output layer which used Tanh as the activation function. For the discriminator, LeakyRelu is used as the activation function for all network layers.DCGAN has a certain improvement in the quality of image generation compared to the original GAN network, as well as having fewer training parameters, taking up less memory, and a more efficient training speed.

The Wasserstein distance is also known as EM distance, which is defined accordingly: (12) W(Pr,Pg)=infγ∼π(Pr,Pg)E(x,y)∼γ[ ‖ x−y ‖ ]

where π(P_r,P_g) denotes the set of all joint distributions of combinations P_r, P_g whose marginal distributions are P_r, P_g.

For any joint distribution γ of P_r, P_g, sampling any of the samples (x,y) yields the corresponding sample distance ║x–y║, E_(x,y)~γ[║x–y║] is the expectation of the sample distance under that joint distribution. The Wasserstein distance, on the other hand, describes the lower bound of the expected distance across all joint distributions.

The KL scatter will be meaningless when the support sets of the two distributions do not overlap or do not overlap almost everywhere, and the JS scatter is a constant, neither of which is a valid measure of the difference between the two distributions. The Wasserstein distance, on the other hand, is still able to portray how far apart the two distributions are under this condition.

The Wasserstein distance, although significantly superior in measuring the difference between two distributions, cannot be solved directly in practice because its expression contains infγ∼π(Pr,Pg) . By the KR theorem, where ║f║_L≤K means that the Lipschitz constant of function f does not exceed K. That is, for any x₁, x₂ belonging to the domain of definition of f, |f(x₁)–f(x₂)|≤K|x₁–x₂| is satisfied.

Thus it is only necessary to find the f that satisfies the K-Lipschitz continuum such that E_x~Pr[f(x)]–E_x~Pg[f(x)] is maximized to obtain the corresponding Wasserstein distance.

In this way the original Wasserstein GAN (i.e., WGAN) is designed.

The deep neural network is utilized to construct the fit f (discriminator D) such that the corresponding value can be maximized under the K-Lipschitz constraints, and thus the corresponding loss function obtained is as follows: (13) LossD=−Ex∼Pr[ D(x) ]+Ez∼Pz[ D(G(z)) ]

And the loss function at this point is equivalent to the approximate Wasserstein distance between the true and generated distributions.

The generator part, on the other hand, aims to minimize the Wasserstein distance, and considering that the left half of Eq. is not directly related to the generator, the corresponding loss function of the generator is: (14) LossG=−Ez∼Pz[ D(G(z)) ]

Nowadays, animation production enters the stage of generative design, especially for movie-level animation production, the traditional production method is extremely time-consuming and labor-intensive, so the combination of intelligent algorithms and animation production is a big trend. The generation and creation of animation characters are carried out using the graphic generation adversarial network. The animation character generation and creation process is shown in Figure 2, and the animation character generation and design creation process is divided into six steps.

Figure 2.

The production and creation process of the artist

The first step is to construct the animated character dataset, which is intended to provide training data for the generative adversarial network and provide data reference for extracting the animated character imagery vocabulary.

The second step is to construct a graphic generation network model, compare the advantages and disadvantages of various generative adversarial networks as well as the application areas, and add visdom to the constructed model for real-time feedback.

The third step is to train the model, using the constructed training dataset to train the network model, in the training process, the generator continuously generates new animated characters, while the discriminator will judge the authenticity of these animated characters. Through repeated iterative training, the performance of the generator and discriminator will gradually improve.

The fourth step establishes selection criteria, which can be carried out simultaneously with the third step, to classify the data while training the network, extract the animation character intention vocabulary, match the relationship between morphology and emotion vocabulary, and determine the design intention of the animation character, so as to assist the producer in the creation of animation.

The fifth step automatically generates animated characters, generates animated characters through the trained model, and then adjusts the parameters to control the number of generated animated characters as well as the generating effect of the network to meet the output of high-quality programs.

The sixth step of screening optimization and final animation character determination, through the establishment of a good selection criteria to select the initial program to meet the requirements, and then by the expert group scoring to determine the final optimization of the program, reference to the optimal program for the final animation character generation, design and production.

Four datasets of different sizes and connectivity are considered in this chapter. The experiment divides all the datasets into a test set (20%) and a training set (80%).The experiments are validated using the 20% training set, and the final number of samples generated is equal to the number of samples in the test split of each dataset. To assess the quality of the generated graphs, the experiment compares the distribution of the generated graphs of the animated characters to the distribution of the real graphs by measuring the Maximum Mean Difference (MMD) of the graph statistics, capturing how close their distributions are. The graph statistics used by the model to evaluate the graphs of the generated animated characters are Node Degree Distribution (Degree), Clustering Coefficient Distribution (Clustering), Mean Orbit Count Distribution (Orbit), Mean Number of Nodes (Node), and Mean Number of Edges (Edge). The statistics listed above can be used to evaluate the structural distribution difference between the generated graph data and the real graph dataset in a more comprehensive and quantitative way, which further reflects the advantages and disadvantages of the model.

In order to verify the effectiveness of the graph generation network improvement in this paper, the traditional GAN, CGAN, ACGAN and the generative network model in this paper are compared. The maximum mean difference (MMD) comparison results can be seen in Fig. 3, and Figs. (a) ~ (c) are the MMD score comparison results of node degree distribution, clustering coefficient distribution and average orbit distribution, respectively. The average MMD values of this paper’s model in node degree distribution, clustering coefficient distribution and average orbit distribution are 0.748, 0.503 and 0.523, respectively. Compared with the ACGAN model which has better data, the average value of each MMD is still higher than that of the ACGAN model by 0.116, 0.035 and 0.076, which shows that the improved optimization of this paper’s model is very effective.

Figure 3.

Comparison result(MMD)

In order to further validate the effectiveness of the models in this paper in the overall algorithm, the four models are tested on a new synthetic COKK dataset. Node Degree Distribution (Degree), Clustering Coefficient Distribution (Clustering), and Mean Orbit Count Distribution (Orbit), and the results of the model testing are shown in Table 1. It can be seen that the node degree distribution, clustering coefficient distribution, and average orbit count distribution of this paper’s model on the COKK dataset are 0.178, 0.185, and 0.076, respectively, which, in a comprehensive view, have achieved the best results, further illustrating the superiority of this paper’s model.

Since real-life graphs typically have thousands or even tens of thousands of nodes, the scalability or generalizability of any generation method is crucial. Generalizability is one of the key aspects of graph generation algorithms and refers to the ability of an algorithm to handle larger and more complex datasets while maintaining the high performance and quality of the generated graph. Generalizable algorithms can effectively handle increasing amounts of data, making them suitable for applications that require large-scale graph generation or applications that generate large graphs. Therefore, ensuring the generalizability of graph generation algorithms is important for their practical use in various industries.

Since there is no standardized metric for generality, in this experiment, the metric will be set to the time required to generate graphs with different numbers of nodes. The untrained model is initialized to generate multiple random graphs, and the number of nodes generated, n, is continuously expanded to perform this experimental measure.In this experiment, the GAN, CGAN, ACGAN methods, and the proposed model in this paper were tested on the same device environment. The generalization test results are shown in Fig. 4. It can be seen that after the number of nodes is greater than 11000, the graph generation time of the three models of GAN, CGAN, and ACGAN is more than 10 s. The time required to generate the image of this paper’s model is less than 10 seconds, which is a significant reduction in time compared to other models.

Figure 4.

Usability test results

The method in this chapter uses the Yolov3 network as a target extractor for video frames.Yolov3 is used as a target detection network, which enables the extraction of features for each frame of video, as well as the labeling of the target location, and the categorization of the detection target. Using the characteristics of the detection network, it is able to extract the entities, their category information, and the location information of the behavior in each frame. Specifically, for each entity O₁, iϵ{0,1,2,⋯,N}, N indicates the number of entities involved in the behavior, when i=0 for the whole video, the entity information is summarized as: (16) Poi=detect(boij,Video,Oi)

where P_oi denotes the set of types of entity O_i.

When the category information of the entities in the video is obtained, the category information needs to be transformed into semantic information, using Word 2Vec to represent the entity category information in the video using word vectors: (17) foi=φ(Poi)

Where φ denotes the Word 2Vec algorithm, which transforms the category information into semantic information, and f_oi is the word vector of each entity, for the whole video, the word vector information of all its entities can be represented as: (18) Fword={ fo1,fQQ,…,fox }

where f_o0 denotes the word vector representing the person. The relative position of the entity where the behavior occurs throughout the video is: (19) boi=∑j=1Fra(xooj,yoij,woej,hoij)Fra

Where Fra represents the total number of frames of a video, b_a represents the relative position of entity O_i in the video, and the position of O_i is denoted as [x_Qi,y_oi,w_oi,h_oi].

The construction of the person-object relationship graph is based on the construction of nodes as well as the construction of edges. For the construction of nodes, the method uses the word vectors of the extracted entities as the nodes of the graph, i.e., F_word. In the whole graph, F_word is denoted as the feature matrix X. For the edges of the graph, the person-object relationship is modeled considering that the relationship between the entities in which the behavior occurs is a person-object interaction, i.e., the edges of the graph are denoted by using the adjacency matrix. For the relationship measure of two nodes, the similarity of the features of the two nodes used by most methods for the measure, or calculating the intersection and concurrency ratios of the positions of the two nodes, is not applicable for the method proposed in this chapter. This is because the nodes of this method have clear categories and are highly distinguishable, and using similarity for the metric would result in too small a value for the edge. Using the intersection and merge ratio of node positions is also inappropriate because many behavioral object positions do not intersect with human positions, which would result in zero values for edges that should be present. Based on this, the defined join operation is as follows: (20) connect(Xi,Xj)=α(xoi−xoj)2+(yoi−yot)2

The connection operation represents the calculation of the distance between two entities. The relationship between two nodes is represented by defining the connection operation. For the adjacency matrix M, its dimension size is N×N and each value inside is: (21) Mij={ connect(Xi,Xj)ifi=00

For the above formula, the values on the edges of the person-object relationship graph are constructed, as the human nodes are connected to other object nodes in the graph, considering that the behavior occurs mainly by the interaction between the person and the object.

The method in this paper feeds the constructed person-object relationship graph into a graph convolutional network for inference, and uses graph convolutional networks to reason about the graph. Unlike ordinary convolutional networks, graph convolution can aggregate information from neighboring nodes to the current node, therefore, graph convolution allows information to pass through the graph and update each node. The output of the graph convolution layer is represented as follows: (22) Z=MXW

W is the weight of the convolutional layer, X is the features of all nodes, M is the adjacency matrix of the graph, and Z represents the output of the graph after convolution. For the graph structure generated by this method, the representation of the graph is done by using global pooling to transform the person-object relationship semantic graph into behavioral semantic features F_word, specifically: (23) Fword=pooling(Z)

where pooling(·) denotes the global pooling operation that changes the dimension of the feature matrix X from N×d to 1×d to generate behavioral semantic features. For this graph convolution, the loss function defined is: (24) L=−1N(∑c=1Nyclog(δ(Fword)))

where C denotes the number of classes of behaviors, δ(·) denotes the penultimate layer of fully connected mapping functions, and N denotes the number of entities.