Application of Chinese Painting Techniques in Modern Hand-painted Renderings in Information Age

The unique artistic language and expression form of painting needs to be expressed and presented through painting techniques [9, 10]. Different techniques can endow works with different visual effects and forms of expression and make paintings present different artistic forms. As a traditional art form, Chinese painting has rich and diverse techniques. Its characteristics are mainly reflected in the brushwork and ink method. In terms of brushwork, Chinese painting mainly uses various techniques such as hooking, chapping, dotting, and dyeing [11]. In terms of ink methods, Chinese painting adopts many different methods, such as baking, dyeing, breaking, and accumulation [12]. In addition, in the traditional expression techniques of Chinese painting, highly representative traditional styles such as meticulous brushwork, freehand brushwork, and white drawing have also been widely used. Different painting techniques, painting themes, and forms of expression all give the works different artistic forms.

According to different painting themes and creative motives, the techniques of Chinese painting can present different characteristics and functions [13, 14]. For example, Song Yingxing’s “Tiangong Kaiwu” is an ancient Chinese scientific work. The overall picture uses the technique of Chinese painting white drawing, and the simple and general artistic features of lines make the picture information accurately convey.

In the paintings of flowers and birds in the Song Dynasty, Huang Sheng’s Sketch of Rare Birds was created by meticulous painting techniques. In terms of painting techniques, Huang Sheng emphasizes true sketching and attaches importance to shape and texture. His paintings are mostly hooked with light ink, rendered with heavy colors, and the pen is extremely fine and then smudged with color, with almost no ink. This “double-tick color filling” method is a common technique in Chinese meticulous painting, which is characterized by outlining the outline of the line with a fine pen first and then filling it with paint, which makes the picture rich in texture and layering [15]. In addition, the color of insects in Sketching Rare Birds is very close to that of real insects, and the sense of realism is stronger from the details.

When drawing macro scenes, the composition of Chinese painting is very ingenious. For example, “Three Friends and Hundred Birds” adopts a panoramic composition method, and the picture has a sense of layering. Even the scenes with lush branches and rich species are not messy and have a sense of rhythm. After clearly explaining the attachment relationship between the subject and the scene, the picture is interactive, such as birds seem to be playing with each other. The picture, like “a hundred birds pay homage to the phoenix,” is very vivid and interesting.

The techniques and expressions of Chinese painting are very rich, not only in brushwork and ink but also in composition. Especially compared with modern illustration art, the rich techniques and expressions of Chinese painting provide reference value and distinctive national characteristics for the field of modern illustration [16].

The expression of the so-called hand-drawn graphics in graphic design is not only the design that designers use all kinds of freehand-drawn graphics as media, materials, elements, skills, materials, and techniques, but also the conception, establishment, construction, and piecing together in a vivid and intuitive way of explanation, so as to convey the important work of design information and information, abandoning the creation of computers and images, and completely based on the creation of hand-drawn. Its classification is also covered by the concept of graphics [17, 18]. A good design scheme and conception must have strong explanatory and persuasive power so as to play an important role in design expression and present the most valuable part of the design scheme truly and objectively so as to facilitate the common discussion and research of the design scheme. Such decision-making and appeal require hand-drawn graphics, so hand-drawn graphics are an indispensable and effective resource in graphic design.

With the application of computers, today’s design works are more and more abundant, the pictures are more and more beautiful, the design cost is getting lower and lower, and the work efficiency has been greatly improved. All kinds of superior conditions make designers feel happy and satisfied, but at the same time, there is also an unknown worry and confusion [19]. Long before the advent of computers, designers used paints and brushes to record their fleeting inspirations and instant thoughts and convey certain ideas. Therefore, paper and pencil tools became their carry-on items, and it is precisely because of their convenience and powerful recording ability to express instant memory that paper and pencil tools are liked and favored by designers, which is the unique artistic charm that computers can never replace. For design, hand-drawn graphics, usually every link from conception to work formation, is the expression of the original intention of the author’s design ideas, which can timely and efficiently capture the instant spark of thought in the author’s heart and can reflect the synchronous process of the author’s creativity. In the process of the designer’s creation exploration and practice, hand-painting can record the author’s thinking image more vividly and vividly, reflect the author’s creative passion, and inject this passion into the works.

As an effective carrier of information dissemination, hand-drawn graphics have played an increasingly important role in modern society [20 21]. At present, in various design grand prix, many designers make full use of the combination of hand-drawn graphics and graphic design to create influential works. It grafts the original materials into modern graphic design through drawing, sketching, copying, and so on, and presents an illustrative pattern image in the works, vividly expounding the connotation and significance of the works.

Inspired by HED [22], two aspects are mainly considered: (1) in order to be able to extract more bottom-level and high-level features, the network model needs sufficient depth; (2) Using images with different resolutions to realize feature representation, that is, through the fusion of multi-resolution images, the intrinsic multi-scale information of Chinese painting techniques can be obtained. Because convolutional neural networks have the nature of hierarchical learning, multi-scale and multi-level learning strategies can meet the above two requirements. The above learning strategy can be considered from two aspects: First, this learning strategy can be embedded in the neural network for implementation, which is embodied in the continuous increase of the receptive field and the continuous reduction of the size of the feature map generated by downsampling, that is, in the spatial dimension, the neural network can complete the above learning process through multiple convolutions and pooling; Secondly, this strategy can be realized by a scale transformation of the input natural image during network training. For the training of neural networks, if you start from scratch, multi-stage deep neural networks will encounter great difficulties. In fact, if fine-tuning is performed on the basis of traditional models, it can well solve many low-level tasks and middle-level tasks in images, such as edge detection [23], image classification [24], and object detection [25].

Through the above analysis, it can be found that VGGNet is a network framework that is very suitable for the generation of Chinese painting techniques. First, VGGNet achieved excellent results in the ImageNet Image Classification Challenge. Secondly, VGGNet has very deep depth (16 convolution layers) and high-density perception (convolution kernel with a step size of 1). Finally, VGGNet divides the convolution into five stages, each followed by a downsampling layer with a step size of 2. With the increase of the receptive field, each convolutional layer can capture more useful information, and this rich multi-level information is very helpful to the recognition and extraction of image contour.

The framework of the multi-scale Chinese painting technique generation model is shown in Figure 1. According to the characteristics of Chinese painting techniques, VGGNet is improved as follows: (1) The last convolution layer of the four stages is convolved respectively to obtain four different side output layers, thus obtaining rich semantic information, that is, convolution operations are performed on convolution layer 1, convolution layer 2, convolution layer 3 and convolution layer 4 respectively to obtain receptive fields with sizes of 5 × 5, 14 × 14, 40 × 40 and 92 × 92 respectively: (2) The fifth stage of VGGNet is removed, because the discontinuity of detection lines appears in the high-level output map, and the results obtained by upsampling increase the error with the labeled data; (3) Remove the fifth pooling layer and all fully connected layers of VGGNet. The main reason is that the side output of different scales will output a very small feature map when the step size is 32, and the map generated by interpolation is too fuzzy for the task of generating Chinese painting techniques.

Figure 1.

Network framework for generating original Chinese painting techniques

In the network training stage, the training data set S consists of the original image X and the corresponding annotated image Y. Composition, that is, S = (X, Y), n takes values from 1 to N. The goal of the n= is to generate Chinese painting techniques that are close to the annotated image from the original image through network training and feature learning. For the convenience of expression, the parameters in the standard network are uniformly represented by W, the number of side output layers is represented by M, and the network parameters of each side output layer are represented by w=(w(1),w⁽²⁾,..,w^(M). For M-side outputs, the objective function definition (1) is as follows: (1) Lside (W,w)=∑m=1Mαmlside (m)(W,w(m)) Where l_side represents the loss of each side output. α represents the output of the activation function. For Chinese painting techniques, the pixel occupancy of Chinese painting techniques in the whole image is less than 10%, so simply using loss to calculate the positive and negative classes in a single image will cause an imbalance [26, 27]. Therefore, the class balance weight is used for each pixel to calculate the class balance cross-entropy loss between the Chinese painting technique area and the blank area of the image, and its form (2) can be expressed as: (2) lside(m)(W,wm)=−β∑j∈Y+logPr(yj=1∣X;W,w(m))−η∑j∈Y−logPr(yj=0∣X;W,w(m)) Wherein,β = |Y₋|/(|Y₊| + |Y₋|),η = |Y₊/(|Y₊| + |Y₊|). |Y₊| and |Y₋| represent a blank area label set and a Chinese painting technique area label set in the reference image, respectively. Pr(y_i=1\X;W,w^(m))=δ(aj(m)∈[0,1] is the j pixel calculated by the activation function. Use weight fusion to calculate the loss value of all side output results and labeled images, as shown in equations (3)-(5): (3) Lfive (W,w,h)=D(Y,Yfiuse ) (4) Yfiuse =δ(∑m=1MhmFside (F)) (5) Fside (m)={ aj(m),j=1,2,…,| Y+ |+| Y− | } Among them, Y and Y_fuse represent the original image and the fused image respectively, and D (,) represents the distance between the fused feature map and the labeled data. δ is the activation function, hm represents the fusion weight, and F^(m)side represents a mapping from input to output. Finally, the loss function of the whole network is equation (6), where L_fuse represents the loss of the fusion layer and L_side represents the edge loss in the training process.

In order to describe Chinese painting techniques, this study proposes a double-branch convolutional neural network combining appearance and modern hand-painted features [28, 29]. As shown in Figure 2, the first branch is the appearance feature extraction network, which is mainly responsible for extracting the appearance features of Chinese painting techniques: the second branch is the modern hand-painted feature extraction network, which is mainly responsible for extracting the modern hand-painted features of Chinese painting techniques.

Figure 2.

Double-branch convolutional neural network framework

Notably, the two neural networks have different input types. For the appearance feature extraction network, the original Chinese painting techniques are input into the network, and the appearance features of Chinese painting techniques are extracted through operations such as convolution, pooling, and fully connected layers [30]. For a modern hand-painted feature extraction network, a series of point sets are obtained by point sampling of Chinese painting techniques, and then multi-layer stacking, including input alignment module, multi-layer perceptron layer, pooling layer, and full connection layer, is used to extract modern hand-painted information of Chinese painting techniques. Finally, the trained appearance network and modern hand-painted network are used to extract the appearance features of Chinese painting techniques and modern hand-painted features, respectively. When different feature representations are obtained, the two feature vectors are normalized and connected to obtain a mixed feature representation. For the image recognition task, the features are input into the trained SVM classifier to obtain the predicted probability value of the category. For the image retrieval task based on Chinese painting techniques, the features of Chinese painting techniques and the features of the edge map generated from the image data set are extracted, respectively, and then the final retrieval results are obtained by calculating the distance and sorting the results.

First, the original Chinese painting techniques are uniformly scaled to 225 × 225 pixels. In order to avoid the deformation of the Chinese painting technique caused by scaling operation, the long side of the Chinese painting technique is adjusted to 225 pixels. First, the scaling ratio α is calculated at the same time, and the short side is scaled based on α, and finally, the white pixel block is used to fill the deficiency. Then, choose to acquire a series of points x to realize a modern hand-drawn representation, x = {x₁, x₂ ... x_n}. In fact, the sampling points can be selected randomly, but if the sampling points are selected evenly at a fixed distance, the final modern hand-painted representation of Chinese painting techniques will work better. Therefore, iterative Farthest Point Sampling (FPS) is employed to achieve the collection of points. Compared with random sampling, this strategy can achieve full coverage of Chinese painting techniques and reduce the error of modern hand-painted representation. In this study, 512 points were collected, and each point was expressed as two-dimensional coordinates (x, y).

When traditional convolutional neural networks designed for images solve the problem of geometric invariance, their main strategies focus on using data enhancement techniques such as flipping, random cropping, translation, scaling, etc. However, the ability of these methods in rotation, translation, and scaling changes is still very limited. The semantic information of Chinese painting techniques based on point sampling should be invariant to some transformations, that is, the invariance of rotation and translation transformation. Realizing the alignment of different point sets to regular space is still a problem worth studying. Aiming at the rotation and translation of Chinese painting techniques, the modern hand-painted extraction network introduces an alignment network to standardize point features. For the first stage of input alignment, this section designs an alignment network to predict the 2 × 2-point transformation matrix and then multiplies the prediction result with the input point set Chinese painting technique matrix so as to achieve the alignment of Chinese painting techniques in the input space. In this study, an alignment network for affine transformation matrix learning is designed, which consists of five multi-layer perceptrons, a pooling layer, and three fully connected layers. For the third stage of feature space normalization alignment, the feature correction is realized by learning the feature transformation matrix. The network structure is the same as that of the first stage, but the input and output dimensions have changed; that is, the output of the multi-layer perceptron is 64, 128, 256, 512, and 1024, respectively, and the output of the fully connected layer is 1024, 512 and 256, respectively. The difference is that the input and output of the third stage network are 64 × 64 matrices. Because the feature dimension of the third-stage transformation matrix is higher than that of the original input space, the difficulty of network optimization will be greatly increased. Using the point cloud processing method for reference, the regularization term is added after the loss function, which makes the network easier to optimize.

In order to learn the modern hand-painted features of Chinese painting techniques, the loss function of network training is defined as follows (7)-(9):

Where J(θ) represents the cross-entropy loss function, α represents the regularization weight of ℒ, ℒ represents a regularization term, m represents the number of images, k represents the number of image categories, l(y⁽ⁱ⁾=j) represents the indicator function, and exp is the exponential function. Take 1 if the predicted label y⁽ⁱ⁾ is consistent with the actual label, otherwise take 0.a_k = ∑j W_j Z_j represents the output of the hidden unit j. I represent the network input, and M represents the feature transformation matrix learned by the alignment network. By orthogonal transformation with the input, more original information of Chinese painting techniques can be retained. Experiments show that by adding regularization term to the loss function, the optimization of the network can be more stable and the model can be optimal.

Appearance feature extraction network extracts appearance features from the input original Chinese painting techniques. Existing deep convolutional networks have many mature network architectures when extracting image appearance features, so this section draws lessons from the design idea of Alexnet to build an appearance feature extraction network. Overall, the appearance feature extraction network includes 8 convolutional layers and 3 fully connected layers. For the 1st, third, and seventh convolutional layers, followed by the maximum pooling layer, the number of neurons in the last fully connected layer is equal to the number of dataset categories. The appearance feature extraction network is trained by a cross-entropy loss function, and the calculation method is as follows (10)(11): (10) J=∑k=1Nyklogy^k (11) y^k=g(W·x(i)+b) Where k represents the number of image categories. N represents the input number of samples, yk represents the label where the input image is true, y^k represents the label of the input image prediction, g represents the softmax activation function, x(i) represents the output of the hidden layer, and W and b represent the weight and bias terms of the network, respectively.

After the training of the two networks is completed, the feature fusion method is used to merge the appearance features of Chinese painting techniques and modern hand-painted features, that is, to connect l₁ normalized Ffc2app ,Ffc2shape , “F+” and denoted as “F⁺”. In the training stage, the SVM classifier is trained by feature vectors composed of mixed features F⁺. In the testing stage, the test image is first preprocessed, then the trained double-branch network is used to extract features, and finally the SVM classifier is used for prediction, as shown in Equation (12).