Application of Artificial Intelligence-based Content Generation Technology in News Publishing

AIGC technology, i.e., the technology of generating content based on artificial intelligence, is a new type of technical means that combines artificial intelligence and creative content. Through advanced algorithms such as deep learning and neural networks, this technology simulates human thinking patterns and creativity to autonomously generate content with certain practical value.AIGC is not only limited to the generation of multimedia content such as text, images, audio, video, etc., but also includes applications in a variety of fields, such as game design and the construction of virtual reality scenarios. Its core lies in the fact that through a large amount of data training, the machine is able to learn the laws and techniques of content creation, and then independently complete high-quality content creation without direct human intervention.The emergence of AIGC technology greatly improves the efficiency and diversity of content production, and provides creators with a new source of inspiration and means of creation. At the same time, it also provides a more personalized and intelligent content consumption experience for ordinary users. It can be said that AIGC technology is a major innovation in the field of content creation in the digital era, and it is gradually changing the way practitioners access and enjoy cultural products.

The basic principle of AIGC technology mainly relies on deep learning and big data processing technology. First of all, AIGC technology can build a huge database by collecting massive creative materials and data, which not only contains rich and diverse creative elements, but also records the patterns and preferences of human creation. Secondly, AIGC technology can utilize deep learning algorithms to train and learn from data, so that the machine can understand and simulate the human creative process.In specific operations, AIGC technology will extract key features and rules by analyzing the creative elements and patterns in the database. Again, based on these features and rules, combined with specific creation needs, AIGC technology can autonomously generate new content. In this process, it not only simulates human creative skills, but also incorporates its own “creativity” to generate works that are both aesthetically pleasing and novel. Additionally, AIGC technology has the ability to self-optimize and learn. By constantly analyzing and learning from new data, it can gradually improve the level and diversity of its creations, so that the content it generates is closer to human creations, and even surpasses human creations in some aspects. This self-evolving characteristic makes AIGC technology a system of continuous progress and innovation, bringing unlimited possibilities to the future of content creation.

In order to fully extract the feature information in long text content, this paper adopts the Fastformer model to model and analyze the input word vectors and output the feature vectors containing contextual information. Fastformer is a variant of Transformer that uses additive attention [27] to achieve effective contextual modeling with small linear complexity.Compared with the Transformer model, it is much more efficient in the computation process, which can save computational resources, and its modeling performance in long text is also better than the Transformer model.Since most news texts are long, in order to save computational resources and improve performance, this paper constructs a sequence encoder based on the Fastformer model and combined with the Bi-LSTM model [28].

During the computation of the Fastformer model, the query sequence is first transformed into a global query vector using the additive attention mechanism.Then the global query vector will be combined with each key vector through elementwise product operation, and at the same time, the combined sequence will be summarized to generate the global key vector through additive attention mechanism; finally, the global key vector will be multiplied with the value sequence through elementwise product operation, and then the attention sequence containing global information will be obtained through linear transformation, and this attention sequence will be combined with the query sequence and combine the attention sequence with the query sequence to form the final output.

First, the Fastformer model transforms the input matrix of length N into the query, key, and value attention matrices, which are denoted as Q = [q₁, q₂,⋯, q_N], K = [k₁, k₂,⋯, k_N], V = [v₁,v₂,⋯, v_N], respectively, by means of a separate linear Transformation layer.

Secondly, the Transformer model uses dot product operation when modeling the context information of the input sequences through the interaction between Q, K, and V. The dot product operation not only brings large computational complexity, but also makes the model’s performance degrade when modeling the context information of long sequences. Fastformer uses the additive attention mechanism to solve this problem when modeling the interaction between Q, K, and V. It uses the additive attention mechanism to compress the global context information in the Q matrix, which is summarized as a global query vector, and the attention weight a_i of the i st query vector is computed as shown in Eq. (1), where wqT is the learnable parameter vector The global query vector q is computed as shown in equation (2).

Then, the global query vector is modeled interactively with each key vector in the K-matrix by the element product operation, which is an efficient operation for modeling the interaction between two vectors.The Fastformer model combines the global query vectors with each key vector through an elemental product operation in order to learn the global context-aware key matrix, where the ist vector in the matrix is denoted as p_i and summarized as a global key vector containing global context information through an additive attention mechanism.The formula for the attention weight a_i of the ird key vector is shown in Equation (3), where wkT is the learnable parameter vector, and the global key vector k is computed as shown in Equation (4).

Finally, the global key vector is modeled by interacting with each value vector in the V matrix, using the same computational approach of elementwise product operation, the global context-aware value matrix is learned by elementwise product operation between vectors, and then the final hidden representation between key-values is learned by a layer of linear Transformation layer, which will be finally The obtained global attention matrix is denoted as R = [r₁,r₂,⋯,r_N]. The obtained matrix R is summed with matrix Q, which is the final output of the Fastformer model.

After obtaining the final output of the Fastformer model, the input vectors and the final output vectors are residually connected and normalized as the input of the Bi-LSTM model, the residual connection can effectively reduce the overfitting, and finally the hidden state h sequence is encoded and output through the Bi-LSTM model.

The pointer generation network model combines a pointer network with a sequence-to-sequence model based on an attentional mechanism, which not only helps to improve the accuracy of the copied information by copying words from the source text via pointers, but also retains the ability to generate new words by performing word generation via a vocabulary.

After the input vector passes through the sequence encoder to generate the hidden state sequence h, the word vector generated in the previous moment is received by the bi-directional long short-term memory network decoder to get the decoded state sequence s_t. From the encoder hidden state sequence h and decoded state sequence s_i can be derived from the attention distribution a^t, and the attention distribution a^t is used to determine the characters that need to be paid attention to among the characters of the output sequence in the moment of t, which is computed as shown in Eqs. (5), (6) shown, where v,W_h,W_s,b_att are parameters obtained by training.

The computed attention distribution a^t is then used to generate a context vector c_i by weighted averaging of the encoder hidden states h_i. The context vector c_i is concatenated with the decoded state sequence s_i, and then two linear mappings are performed to generate a word list distribution P_vocab, which is computed as shown in Equation (7), where V^′,V,b^′,b is a parameter learned through training.

After calculating the word list distribution P_vocab, the model will determine whether to copy words from the original text or to generate new words through the vocabulary list by means of the generation probability P_gcπ, and the generation probability Pgcπt at the moment of t is calculated as shown in Eqs. (8) and (9), where w_k,w_s,w_x,b_psr is the parameter obtained during the training process, σ refers to the sigmoid activation function, and x_t is the input sequence of the decoder.

In order to solve the unlogged word (OOV) word problem, the pointer generation network model selects whether the current word is copied from the original text or a new word is generated from the vocabulary list by using P_seκn as a switch during the decoding process, and the probability calculation process is shown in Equation (10).

The title generator chooses the pointer generation network, which is different from the traditional text summarization model, and can not only generate words in the summary, but also point to the words in the original text through the “pointer” method. By introducing the pointer mechanism, the pointer generation network can effectively deal with some uncommon words and entities in the original text.

User interest injection is the key to personalize the title. The modeled user interest is added to the computation of the pointer generation network, which in turn affects whether the words are generated from the table or copied from the original text in the decoding stage.

Specifically, in the encoding phase, the Transformer encoder is utilized. A sequence of word vectors v = [w_v1,…,w_vn] of the candidate news text is input to the two-layer positional encoder, which adds positional coding to the embedding vectors: (11) PE( pos,2i)=sin(pos/1 00002i/d*) (12) PE( pos ,2i+1)=cos(pos /100002i/dw) where pos is the word position and i is the dimension. Each word embedding with position information epos′ can be represented as: (13) epos'=(epos+PEpos)⊕Wpos[l] where ⊕ denotes splicing, and multi-head subattention is used to capture interactions between words and sentences. Therefore, the encoder hides state h = h₁ ⊕ h₂,…,h_k.

where WiQ,WiK,WiV is the learnable parameter matrix and E^′ denotes the word sequence embedding in the candidate news.

During the decoding process, the hidden state s_i step t of decoding can be derived after a given input x_i and the distribution of attention in the hidden state of the encoder is computed as h: (15) at=softmax(Van·tanh(Whh+Wsst+bant)) where Van·,Wh,Ws,Wu and b_an are trainable parameters. The decoder generates the next word based on the attention distribution.

The attention distribution is used to generate a weighted sum of the encoder’s hidden states to compute the distribution of the generated vocabulary list: (16) Pvocab(wt)=tanh(Vp[st;ct]+bv) (17) P(wt)=pgentPweab(wt)+(1−pgent)∑j:wj=wtat,j where V_p and b_v are learnable parameters. Context vector c_t is a fixed-size representation read from the news body at time step t, and P_vocap(w_t) represents the probability distribution of predicted words for all words in the glossary at time step t. The embedding of user interests is represented in the computation of pgen′ . pgen′ is computed as: (18) pgen'=softmax(VTtanh(Whh+Wsst+Wuut+bant)) where V,W_h,W_s,W_u and b_ant are trainable parameters. u_t is the filtered user interest vector. In decoding step t, pointer pgen′ is used to select whether to generate words from a vocabulary list with probability P_vocab(w_i) or to copy words from a sample of news body text with attention distribution.

In order to further improve the degree of personalization of the generated headlines as well as the authenticity of the headline content, this paper uses a reinforcement learning algorithm to optimize the model during the training process. Reinforcement learning algorithms are mainly optimized by learning strategies during the interaction between the intelligence and the environment to achieve the purpose of maximizing the return.

Reinforcement learning is a type of unsupervised learning, which differs from supervised learning in that it does not have labeled data and does not require prior knowledge or experience, but learns through interaction with the environment.Intelligent agents in reinforcement learning collect reward signals through interaction with the environment, which indicate whether the action taken by the agent is correct or conducive to reaching a goal. By trying out different actions and collecting reward signals, the intelligent body can gradually learn which actions lead to higher rewards, thus improving its decision-making and action strategies.

The basic idea of reinforcement learning is for the agent to perform various actions in its environment in order to maximize its future rewards.This process can be simulated using computer code or realized through real-world physical experiments.

Intelligentsia in reinforcement learning usually consist of the following three main components:

1) Environment: the external environment in which the intelligent body operates and learns.

The main goal of reinforcement learning algorithms is to learn the best strategy by maximizing the long-term cumulative reward. Algorithms for reinforcement learning can be categorized into two main groups: value function and strategy gradient.

The value function approach learns the optimal strategy by estimating the value of each state. The value function represents the expected value of the long-term reward for taking an action in a given state, and it can be used to assess how good or bad it is for an intelligent to take a given action. Common value function algorithms include Q-Learning, State-Action-Reward-State-Action, and so on.

Q-Learning is a value function based reinforcement learning algorithm [29] for learning an action value function Q(s,a) that represents the long term payoff of taking an action a in state s. Q-Learning uses Bellman’s equation to update the Q value function and selects the next action based on maximizing the Q value function. This algorithm is widely used to solve many control problems and robotics tasks.

SARSA is another reinforcement learning algorithm based on value functions and a model-based learning algorithm. Unlike Q-Learning, SARSA uses the value function of the current action for updating and selects the next action based on the current strategy.This algorithm is suitable for those cases where the current strategy needs to be taken into account, such as multi-intelligence game problems.

The strategy gradient approach directly optimizes the strategy function itself to find the optimal strategy. The strategy function is a mapping that translates the intelligence states to the appropriate actions. Common strategy gradient algorithms include Actor-Critic and others.

A3C is an asynchronous dominant actor-critic algorithm for solving massively parallel reinforcement learning problems. In A3C, multiple intelligent agents run concurrently, each with a separate neural network for learning policies. This algorithm uses an actor-critic structure to learn strategies and uses multiple threads for asynchronous updates.

Reinforcement learning has a wide range of applications in artificial intelligence, such as training agents to play games in gaming, training agents to perform various tasks in robot control, and training cars to learn to drive in autonomous driving.

Rouge is a significant set of metrics that is utilized to evaluate machine translation and evaluate automatic text summarization. In this paper, news headlines generated by a news headline generator are compared with human-written news headlines provided in the original text to evaluate news headlines based on the co-occurrence information of N-means in the news text.Rouge is an evaluation metric oriented towards N-means recall, which evaluates the quality of news headlines by counting the number of basic units (n-means grammars, word sequences, and word pairs) that overlap between the two. This section scores the model using the Rouge-N (including Rouge-1, Rouge-2) and Rouge-L metrics, and uses these scores to compare with other models proposed in the past.

In order to make the experimental results more convincing, this chapter averages the experimental results and takes the average of 10 experimental results as the final experimental result data. After conducting several comparative experiments, the effectiveness of the proposed news headline generation model in this paper has been verified on both LCSTS and CSTS datasets.Tables 1 and 2 show the comparison of different models on the LCSTS dataset and the CSTS dataset, respectively. Tables 1 and 2 demonstrate that the news headline generation model proposed in this paper performs better than the other six models on either the LCSTS dataset or the CSTS dataset. Analyzing the news headlines generated by the models, it is found that the LSTM+Point model pays more attention to learning the textual information in the original news body, which is introduced into the newly generated news headlines through the form of pointer insertion, while the news headline generation model in this paper pays more attention to the readability and authenticity of the news headlines, and provides a generalized summary of the news document. Regarding the LCSTS dataset, this paper’s model improves on the Rouge-1, Rouge-2 and Rouge-L metrics compared with the LSTM+Point model by 16.43%, 6.31% and 0.75% respectively, which illustrates that the news headlines generated by the news headline generation model proposed in this paper are improved in readability and authenticity.

In this paper, the different performance of the model in the case of using different decoding selection strategies, the model on the CSTS dataset Rouge-1, Rouge-2 and Rouge-L three indicators of the different performance comparison graph is shown in Figure 2. A total of four decoding selection strategies are selected in this chapter on LCSTS dataset including: greedy algorithm, beam search, Bi-LSTM, joint use of Fastformer and Bi-LSTM.The comparison reveals that the Rouge metrics of the model are best in the case of the proposed model in this chapter with the joint use of Fastformer and Bi-LSTM.31.28, 12.68, and 28.31 on the Rouge-1, Rouge-2, and Rouge-L metrics on LCSTS, respectively.Comparing the Rouge metrics under different hyperparameters shows a significant improvement, which indicates that the decoding strategy using Fastformer in conjunction with Bi-LSTM designed in this paper is reasonable and efficient.

Figure 2.

Comparison of different decoding strategies in CSTS dataset

Figure 3 shows the change of loss values during the training process using negative sampling in word-level text processing, negative sampling in Chinese character-level text processing, and using Fastformer combined with Bi-LSTM. First of all, it can be seen that the title generation model using Chinese character-level text processing converges to a lower loss value than the title generation model using word-set text processing, which also proves that the title generation model using Chinese character-set text processing can achieve better performance. In addition, it can be seen from the figure that in the headline generation model with Chinese character-level processing, the use of Fastformer combined with Bi-LSTM can make the model get faster convergence, and the model begins to converge when the number of iterations is 2, and the loss value finally converges to 17.5.

Figure 3.

Training loss value

Figure 4 shows the PR curves of title generation of the model on different news types, in order to make the results more complete, on the basis of analyzing the PR curves, the F1-score is added to analyze the results, and Table 3 shows the highest generation results of the model on different news types related to the three indicators.

Figure 4.

Generate the PR curve of different types of titles

From the PR curve of news headline generation in the figure and the results in the table, it can be seen that the model can obtain high headline generation performance, which can be seen that the headline generation performance of the model has a high precision value, and the recall rate is relatively low, this is because in practice, a piece of news can often belong to more than one category, i.e., there is a cross-category labeling, and the model generates the news headlines with the The model generates news headlines about the health genre with high precision, recall, and F1, with the highest precision of 0.9741 and F1-score of 0.8789.