Research on Traditional Costume Culture Information Extraction and Digital Reconstruction Methods Based on Artificial Intelligence

The U-Net model as a whole can be divided into two parts, the encoder part and the decoder part, the role of the encoder part is used for feature extraction of the input image, as a way to gradually obtain the higher-order semantic information, and the role of the decoder part is to carry out cross-scale feature fusion and output the predicted results. Firstly, the encoder of the U-Net model is improved, except the first convolutional block in the U-Net model, the convolutional block used for downsampling in the encoder is replaced by the Residual module group in the ResNet model, and each Residual module group contains two Residuals, the first one is used for downsampling to obtain higher-order semantic information, and the second one is used to speed up feature transfer. Residual is used to speed up feature transfer and prevent model overfitting. Then, the decoder of the U-Net model is improved by adding the SKNet Attention in the SKNet model after the first three splices in the decoder part, the SKNet Attention serves to suppress irrelevant feature information in the feature maps generated during the up-sampling process, and the ParNet in the ParNet model is added after the fourth splices, the ParNet is able to increase the model’s feature expression ability, so that the model has more nonlinear ability, and at the same time has the ability of multi-scale feature fusion, ParNet has the role of preventing model overfitting.

In the encoder part a Residual module group consisting of two Residuals is used to replace each downsampling of the U-Net model, Residuals are proposed in the ResNet model, and there are two structures of Residuals used in this paper, where two Residuals are used simultaneously at a time, and their formulas can be expressed in equations (1) and (2), respectively: (1)F=σ(conv3×3(convv3×3(X))+conv1×1(X)) (2)F=σ(conv3×3(conv3×3(X))+X)

Where, X is the input features, F is the output features, conv_3×3 is the convolution kernel of 3 × 3, conv_1×1 is the convolution kernel of 1 × 1 and σ is the activation function.

The first Residual operation is to perform different convolution operations on the input feature maps before fusion, in which the down-sampling to capture the features is realized due to the fact that the first 3×3 convolution kernel has a move step of 2, while the second 3×3 convolution kernel has a move step of 1. Due to the complexity and variety of the content of the traditional dress pattern, rich color, fine texture, and more details, Residual can better extract these features and perform feature transfer to reduce the loss of the detailed content of the traditional dress pattern in the down-sampling process.

The second Residual operation is to convolve the input feature map before fusing it with the input. Since both 3×3 convolution kernels have a move step of 1, it does not change the resolution size of the input feature map, and it does not change the number of channels of the input feature map. Since the traditional dress pattern has more features, this Residual serves to prevent the model from overfitting, as a way to improve the generalization ability of the model, so that the model can effectively learn the various features of the traditional dress pattern, and thus accurately perform the segmentation of the traditional dress pattern.

SKNet attention consists of three parts, namely Split, Fuse, and Select. the Split part learns the features of the feature maps from multiple scales using different convolutional kernels, the Fuse part performs channel attention operations on the feature maps obtained from different convolutional kernels so that the model can assign higher channel weights to more important feature maps, and the Select part re-aggregates feature maps with weight coefficients from different convolutional kernels so that the model can obtain feature maps with weight coefficients. The Select part is to reaggregate feature maps from different convolutional kernels so that the model can obtain feature maps with weight coefficients to enhance the feature representation of the model.

The whole SKNet attention can be expressed by the following equation: (3)Uk=fk(X),k=1,…,K (4)U=∑k=1KUk (5)z=φ(BN(W(fGAP(U)))) (6)Skc=eWicz∑k=1KeWicz,c=1,…,C (7)F=∑k=1KSkUk

Where, X is the input features, U_k is the features formed after different convolution kernels, φ is the offset, BN is the batch normalization, f_GAP is the global average pooling, and F is the output features.

In this paper, the reason why SKNet attention is added after up-sampling is because the up-sampling process in the decoder generates irrelevant feature information, and feature fusion by splicing can reduce the impact of irrelevant feature information on the subsequent output segmentation results. However, it is not possible to realize the segmentation of fineness, and the detail content of the traditional dress pattern is large, SKNet attention can make the model recover certain details of the traditional dress pattern after up-sampling.

The structure of ParNet can be divided into three parts, which are SSE and two different convolutional kernels. The SSE part contains the global average pooling operation and the Sigmoid activation function.

ParNet can be represented by the following equation: (8)F=δ(conv3×3(X);conv1×1(X);SSE(X))

where X is the input feature, conv_3×3 is the convolution kernel of 3 × 3, conv_1×1 is the convolution kernel of 1 × 1, SSE is a branching operation, δ is the SiLU function, and F is the output feature.

The formula for the SiLU function can be expressed as: (9)f(x)=x1+e−x

Where x is the input and f(x) is the output.

The most commonly used loss function in the field of image segmentation is the BCE loss function with the following formula: (10)LBCE=−1n∑n=1N(yn×lnxn+(1−yn)×ln(1−xn))

where N is the number of image pixel points, n is the number of times per training, x_n is the independent variable of loss function training, and y_n is the dependent variable of gradient change.

Due to the complexity of traditional dress patterns, there are usually far more segmented target pixels than background pixels in traditional dress patterns, so the traditional dress pattern dataset generally has the problem of sample category imbalance, so the BCE loss function is no longer applicable. To address such a problem, this paper uses the Lovász-hinge loss function for segmentation calculation of traditional dress patterns, which can be optimized for loss calculation by optimizing the IoU. The IoU is also known as the Jaccard index, and its calculation formula is as follows: (11)Jc(y*,y˜)=|{y*=c}∩{y˜=c}||{y*=c}∩{y˜=c}|

The formula for Jaccard loss is: (12)ΔJc(y*,y˜)=1−Jc(y*,y˜)

Where: y^* is the label vector, y¯ is the prediction vector of the model, and c is a certain category.

The main basis of the stabilizing diffusion model is the denoising diffusion implicit model (DDIM), which is mainly divided into a forward diffusion process and an reverse inverse diffusion process.

The forward process asymptotically transforms the input real image x₀ into an image x_T of pure Gaussian noise by adding T noise to the initial image, and at each step of the noise addition process, x_t−1 a Gaussian noise is added to generate a new hidden variable x_t, and the process of noise addition to the image from step t − 1 to step t can be expressed in terms of a Gaussian distribution as: (13)q(xt|xt−1)=N(xt;1−β1xt−1,βtI)

where β_t is a set constant, 1−βtxt−1 denotes the mean of the Gaussian distribution, β_tI denotes the variance of the Gaussian distribution, a hyperparameter that increases gradually with t, and I denotes a unit matrix with the same dimension as the input sample x₀. The forward process of the diffusion model can be represented as a non-Markov chain from t = 1 to t = T moments: (14)q(xLT|x0)=q(xT|x0)∏t=2Tq(xt−1|xt,x0)

The DDIM model inverse process is to generate an image by gradually denoising a noisy image by learning to predict the variance and mean of a Gaussian distribution for the inverse diffusion process through a neural network. Given a noisy x_t, a x₀ is first predicted and then sampled through qθ(xt−1|xt,x0) . For xt~q(x0) , ε_t ~ N(0, I), x_t can be obtained from the DDIM forward process Eqs. xt=αtx0+1−αtε , ε ~ N(0, I), where α_t = 1 − β_t, on the contrary, can be predicted using the model ε_t and x_t to obtain x₀, defined here as denoising: (15)fθ(t)(xt)≔(xt−1−αtεθ(t)(xt))/αt

The forward process is defined through a fixed prior pθ(xT)=N(0,I) and the reverse process is defined as: (16)pθ(t)(xt−1|xt)={ N(fθ(1)(x1),σ12I) t=1 qσ(xt−1|xt,fθ(t)(xt)) t>1

where qσ(xt−1|xt,fθ(t)(xt)) is the backward sampling process defined above and x₀ uses fθ(t)(xt) instead. When t = 1, a Gaussian noise (with variance σ²I) is used here to make the forward process work smoothly.

During training, the final optimization goal is to make the network predicted noise consistent with the true noise with the following loss function: (17)L=Ex0,ε,t[‖ε−εθ(α¯x0+1−α¯ε,t)‖2]

Where t denotes the uniform sampling of [1, N], ε ~ N(0, 1), ε_θ denotes the noise predicted by the network, and α¯t=∏s=1tαs .

The traditional stable diffusion model requires a large number of datasets to be trained, and the generation results have a large randomness, which is not effective to be directly applied to the generation of traditional dress patterns, an improved stable diffusion model structure is proposed, which consists of a stable diffusion model combined with a LoRA fine-tuning network, plus a discriminative network.

The main stabilizing diffusion models are the CLIP text encoder, the VAE model, and the hidden layer space.

The text encoder converts the input textual information into digital information that can be understood by the computer.

Variational Auto-Encoder (VAE) model is composed of two parts, encoder and decoder, which transforms between image and Latent Space.

The hidden space is where the noise iteration is performed. The 4×64×64 feature vector output from the decoder is passed into the hidden layer space, and next the de-noising begins by inverting the time step of the DDIM sampler and looping over it. The noise vector, iteration step, and cue word condition are passed into the hidden layer space together, and the noise at this moment in time is computed by combining iteration step t and the text vector. The original input Gaussian-distributed noise is then subtracted from the noise at this moment in time, and this loops over the number of iteration steps required, resulting in a final output of a 4 × 64 × 64 feature layer.

The LoRA fine-tuning network freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for the downstream task.The LoRA model cannot be used alone, but needs to be paired with the original model, and the two can be merged to obtain complete weights.

There are many dense parameter matrices during the fine-tuning training process of the large model, but the change matrices of the parameters during the fine-tuning process are not full of rank, as shown in Eq. (18): (18)[ a b c … d 2a 2b 2c … 2d 4a 4b 4c … 4d ⋮ ⋮ ⋮ ⋱ ⋮ a 2b 3c ⋯ 4d]d×d=[ a b c … d ⋮ ⋮ ⋮ ⋱ ⋮ a 2b 3c ⋯ 4d]d×r

The d × d-dimensional matrix is transformed by matrix determinant transformation to finally obtain a d × r-dimensional matrix.

Using this feature, for the pre-trained weight matrix W0∈ℝd×k , it can be represented by a low-rank decomposition with updates W₀ + ΔW = W₀ + BA, B∈ℝd×r , A∈ℝr×k and rank r ≪ min(d, k). During training, W₀ is frozen and does not receive gradient updates, while A and B are trainable parameters. Both W₀ and ΔW = BA are multiplied by the same input. For h = W₀x, the forward propagation becomes: (19)h=W0x+ΔWx=W0x+BAx

A random Gaussian initialization is used for matrix A, and matrix B is initialized with 0, so that ΔW = BA starts with 0 at the beginning of training.

In the improved stable diffusion model, the fine-tuning task can be accomplished by adding the LoRA fine-tuning technique to the U-net’s attention mechanism layer, which is the Spatial Transformer layer.

Adding a discriminative network based on the stable diffusion model to determine whether the generated image is a traditional dress pattern can improve the discriminative effectiveness. The discriminative network adopts the adjusted VGG-16 network. The input generated 512×512×3 image is subjected to feature extraction to obtain the feature layer of 224×224×64, followed by multiple feature extraction to obtain the feature layers of 112×112×128, 56×56×256, 28×28×512, 14×14×512h, and 7×7×512, respectively, and then the fully connected layer, which obtains the feature layers of 1×1× 4096, 1×1×1024 and 1×1×2 feature vectors, and finally the output is two dimensions, i.e., yes or no, to discriminate the input image to determine whether it is a traditional dress pattern or not.

Crawler technology is used to obtain traditional dress images from Baidu images, Taobao, Jingdong and other websites, and region detection technology is used to obtain screen shots of film and television dramas, documentaries and other screen shots with traditional dress. In order to guarantee the reliability of the data in the later stage, according to the quality of the image, wearing, characters accounted for the ratio of the image as the screening conditions, screened out the images that match the experiments in this chapter, eliminated the noisy images, processed the images using a unified standard, and utilized the computer technology of rotating, brightness change to supplement and expand the data to get the data set Traditional dress.

The fine-grained featured semantic labels defined for the unique styles, accessories and other attributes of traditional ethnic costumes that meet the characteristics of each ethnic costume are manually labeled using the LabelImg tool, which divides the traditional costumes into six parts: head, chest, arms, legs, feet and waist.

By analyzing the actual training effect and loss curves, it is found that the learning rate higher than 1 × 10⁻⁴ or lower than 1×10^-6 will lead to model training misfit. Take 1 × 10⁻⁴ and 1 × 10⁻⁶ as the upper and lower limits of the learning rate respectively, take a certain interval for training respectively, set epoch as 100, batch size as 8, the loss curves of model training with different learning rates are shown in Fig. 2, and (a)~(d) represent the loss curves with learning rates of 3 × 10⁻⁶, 5 × 10⁻⁶, 3 × 10⁻⁵ and 5 × 10⁻⁵ respectively. When the learning rate is 3 × 10⁻⁶, the loss value fluctuates between 0.5 × 10⁻⁶, which is relatively small, and the loss curve has an obvious downward trend, indicating that the model training has achieved good results.

Figure 2.

Model training loss curve of different learning rate

Due to the limited hardware power, RTX A5000 can set the maximum Batch size to 8. According to the actual training effect and background data analysis results, the relatively effective Batch size is between 1 and 8. Take 1 and 8 as the upper and lower limits of Batch size, take a certain interval for training, set the epoch as 100, and the learning rate as 5×10^-5, and the training loss curves of models with different Batch sizes are shown in Fig. 3, and (a)~(d) represent the loss curves of batch size 1, 3, 5, and 7, respectively. When the Batch size is 7, the fluctuation of the loss value is relatively small, and the loss curve has a more obvious downward trend, indicating that the model training has achieved good results.

Figure 3.

Model training loss curve of different batch size

The paper evaluates the degree of model fitting by comparing the similarity of the improved stable diffusion model generating traditional dress pattern images with the training set images, and the introduced image similarity evaluation indexes include the following four: 1)

Structural similarity SS. When the value of structural similarity is above 0.9, the images possess extremely high similarity, and when the value of structural similarity is around 0.7, there is a certain degree of similarity between the images, but there are differences that can be detected by the naked eye.

Using the improved stabilized diffusion model to generate the corresponding traditional dress patterns under typical cue words, the similarity between the generated patterns and all the patterns under the corresponding classification in the training set is evaluated using several similarity evaluation indexes, and the results of the similarity of the patterns of different elements are shown in Table 3. The mean values of structural similarity, mean square error, root mean square error and peak signal-to-noise ratio of different elemental patterns are 0.745, 0.816, 0.887 and 18.712, respectively. The traditional dress pattern patterns generated by the improved stable diffusion model with the cue words have a certain degree of similarity with those in the training set and at the same time have enough differentiation, i.e., the model is well fitted and the training has achieved better results.