1. Introduction

Internet products have become deeply integrated into people's daily lives in recent years. Accompanied by algorithmic advancements and increased computational power, deep learning technology has made significant progress in various aspects, including computational efficiency, parameter quantity, and expressive capabilities. Many researchers are applying deep learning technology to dialogue research and everyday life, with some products already in mass production, such as Microsoft's Xiaoice, Alibaba's intelligent customer service Xiaomi, and Apple's Siri. Particularly, the rise of large language models (LLMs) has brought ChatGPT into the public eye. People are amazed at the rapid progress of intelligent dialogue systems and can easily foresee their broad application prospects and research value.

Therefore, this paper will explore popular personalized language models through comparative methods, analyze their strengths and weaknesses, and propose feasible improvements to offer new perspectives and positive directions for exploration.

2. Project approach

In the rapidly evolving landscape of technology, dialogue systems have become an integral part of various industrial applications. These systems can be broadly classified into two distinct categories: task-oriented dialogue systems and open-domain dialogue systems, also known as casual conversation systems.

Task-oriented dialogue systems are designed with a specific purpose, usually to assist users in accomplishing particular tasks within a specialized domain. For example, these systems can help users book flights, make reservations, or navigate through customer service inquiries. They operate based on the user's input and aim to provide accurate and efficient services tailored to the task. A prime example of this type of system is Tmall Genie, an intelligent customer service platform that assists users in various specialized tasks.

Open-domain dialogue systems are more versatile and flexible on the other side of the spectrum. Unlike their task-oriented counterparts, these systems do not have a predefined purpose or topic limitations. This allows for a more free-flowing and natural interaction between the machine and the user. Voice assistants like Xiao Ai and advanced language models like ChatGPT fall under this category. These systems offer a more personalized and engaging user experience, as they can adapt to various topics and conversational styles.

Given the increasing demand for more personalized and dynamic interactions, this report strongly emphasizes the research and exploration of open-domain dialogue systems. The aim is to delve deeper into these systems' capabilities, limitations, and future potential to enhance user engagement and satisfaction.

There are mainly two methods for building open-domain dialogue systems: one is based on retrieval, and the other is based on generation. The retrieval-based method first constructs a search corpus and then uses a semantic matching model to calculate the similarity score between the user's input and the responses. Based on this score, an appropriate response is selected as the final result. Since these responses are extracted from the corpus, they are generally fluent. However, these responses are highly dependent on the existing sentences in the corpus and cannot generate new replies. Therefore, this method has significant limitations in open-domain dialogue systems.

On the other hand, the generative approach involves incorporating training data from multiple topics during the training process. The dialogue sequences are then fed into a seq2seq model to predict responses (as shown in Figure 1) [1]. This method can generate sentences that have never appeared in the corpus, making it more suitable for open-domain human-machine dialogue modeling. However, its drawback is that the model often generates meaningless answers due to training samples like "I don't know" and "Okay."

Figure 1. Classic seq2seq framework structure [1].

Many researchers have proposed different methods to address this issue from various angles. For example, some use entropy-based algorithms to filter out generic responses from the dataset. In contrast, others use memory modules to extract and store helpful information from the training corpus, among other methods, all aimed at reducing meaningless replies.

This study focuses on two types of methods: 1) Based on deep learning, it uses end-to-end neural dialogue models to learn character role information from large-scale datasets, thereby giving open-domain robots personality and improving the diversity and personalization level of the dialogue. 2) Based on reinforcement learning, it continuously optimizes the quality of generated replies during user interactions and adjusts reply strategies based on user feedback, thus generating more natural and appropriate replies. By comparing the experimental results, the study aims to determine which direction is more effective for open-domain dialogue systems. The following is a detailed experimental introduction.

3. Experimental analysis

3.1. Deep learning

For the first type of method mentioned above, this study focuses on an improved personalized language model based on the Transformer, called IMDPchat [2]. It mainly consists of four parts:

(1) Personalized History Encoder and Global User Role Information: Utilizes a personalized language model based on the Transformer to encode historical replies and constructs global user role information based on the representation of dialogue history replies.

(2) Personalized Post Encoder: Uses BiGRU to encode the input context into a personalized context representation [3].

(3) Post-Aware Personalized Selection Module: Adopts a custom multi-hop memory mechanism to calculate the relevance between the current and historical dialogue contexts. Then, highly relevant historical dialogue replies are selected from the key-value memory network, and dynamic user role information is constructed based on the corresponding historical dialogue replies.

(4) Personalized Response Decoder: Predicts replies during the decoding phase based on the generated personalized representation, global user role information, and dynamic user role information.

Figure 2 is a schematic diagram of the various components of this model.

Figure 2. IMDP model structure diagram [2].

Regarding the training aspect of the dialogue model, the primary objective is to optimize the model so that it can most accurately generate the desired or target response based on the user's immediate context and historical dialogue data. In technical terms, the model aims to maximize the likelihood of producing the correct output reply given the input variables, including the current conversational context and past interactions with the user.

A specific loss function is employed to quantify the model's performance during the training phase. This loss function denoted as: \( L=-\sum _{t=1}^{{S_{y}}}log{[p({y_{t}}{y_{ \lt t}},X,H)]-}φ{S_{y}} \) , serves as a mathematical representation of the difference between the model's predictions and the actual target replies. The goal is to minimize this loss function, making the model's generated responses increasingly aligned with the expected or target replies. Doing so makes the model more effective and reliable in generating contextually appropriate and accurate responses in real-time conversations. \( Φ \) : A specific hyperparameter is employed to regulate the penalty associated with the length of the generated output. This allows for fine-tuning the model's responses to be either more concise or detailed, depending on the context.

\( p({y_{t}}{y_{ \lt t}},X,H) \) : The likelihood of producing a particular word, denoted as \( yt \) , is determined by considering both the input context and the historical interactions with the user. This probability guides the model in generating the most appropriate word in a given conversational scenario.

3.2. Reinforcement Learning

For the second type of method mentioned above, reinforcement learning is a machine learning approach whose main goal is to enable intelligent agents to learn how to maximize reward signals through interaction with the environment. In dialogue systems, reinforcement learning can be used to learn the optimal dialogue strategy to improve the quality and efficiency of the conversation.

The design of dialogue strategies based on reinforcement learning usually involves three main components: state representation, action selection, and reward design. State representation is the process of converting the dialogue state into a form that the computer can process for handling within the intelligent agent. Action selection is the process by which the intelligent agent chooses the optimal action in a given state. Reward design is the process of defining the reward signal so that the intelligent agent can learn the optimal dialogue strategy. Figure 3 shows a dialogue simulation between two agents.

(1) Action ( \( {A_{t}} \) ): The action, denoted as \( {A_{t}} \) , involves generating a response in the dialogue, which is formulated as \( {A_{t}} = gen (S) \) . Here, \( gen (S) \) refers to the sequence outputted by the Decoder. The scope of possible actions is limitless, as the Agent can produce sentences of varying lengths.

(2) State( \( {S_{t}} \) ): By inputting the information that must generate a response into the current dialogue box (state), the dialogue box is converted into a vector representation.

(3) Policy: The policy is represented as \( {P_{RL}}({P_{i+1}}|{P_{i}}) \) , where \( {P_{i+1}} \) is the reply generated in the context of a specific dialogue. In this setup, the policy is the probabilistic mapping between a given dialogue state and the subsequent actions, which are also dialogues. This means that the policy dictates the likelihood of generating specific responses based on the current state of the conversation.

(4) Reward: The model incorporates three distinct internal rewards to address the challenges of generating natural language in seq2seq frameworks. These are Responsiveness ( \( {r_{EA}} \) ), which gauges how adequately the generated text answers the user's question; Logical Consistency ( \( {r_{SC}} \) ), which assesses the rationality and relevance of the generated dialogue; and Emotional Awareness ( \( {r_{EI}} \) ), which evaluates the emotional tone and appropriateness of the response. These internal incentives guide the model in creating more meaningful, contextually relevant, and emotionally sensitive replies.

Figure 3. Dialogue simulation between the two agents [4].

3.3. Dataset

The research employs data from two online discussion platforms: Weibo and Reddit, some indicators of the dataset are shown in Table 1. The Weibo data is a specialized subset of a larger dataset known as PChatbotW, encompassing a year's worth of Weibo posts starting September 10, 2018 [5].

Weibo is a widely-used social networking site in China, where users can share brief messages, commonly known as "contexts." These messages can receive replies from other users, and each message and reply is tagged with a unique user ID for identification. On the other hand, the Reddit data is extracted from posts made on the Reddit online forums from December 1, 2015, to October 30, 2018. Unlike Weibo, Reddit conversations can form hierarchical, tree-like structures of replies. This allows for exploring these tree structures to pair each parent comment with its subsequent child comments, thereby creating various sets of historical conversational contexts and their corresponding replies.

Table 1. Some indicators of the dataset.

3.4. Comparative models

To evaluate these two models' effectiveness, this study compared several representative baseline models, which can be categorized into four types.

(1) Non-personalized generative models:

Seq2SeqWA: This is a Seq2Seq model based on GRU and equipped with an attention mechanism [6].

MMI: This is also a Seq2SeqWA model. However, it uses Maximum Mutual Information as the new objective function to reduce the proportion of generic responses, thus generating more diverse and interesting replies [7].

(2) Personalized generative models using user role vectors:

Speaker: This model based on Seq2SeqWA inputs user role vectors into the decoder to assist in dialogue generation [8].

PersonaWAE: This personalized Wasserstein autoencoder uses learned user role vectors as conditions to construct a Gaussian mixture distribution [9].

(3). Personalized generative models using predefined role information:

PerWAE: This model employs a memory-enhanced architecture to mine character roles in the context and combines it with a conditional variational autoencoder model to generate more diverse responses [10].

PEE: This model mines and associates existing dialogue corpora using a topic model based on variational autoencoders [11].

(4). Personalized generative models using implicit role information:

VHRED-P: This is a latent variable hierarchical recurrent encoder-decoder model trained by maximizing the variational lower bound on the log-likelihood [12].

ReCoSa-P: This model uses self-attention mechanisms to calculate attention weights between each dialogue context and response representation [13].

3.5. Evaluation metrics

(1) BLEU-1, BLEU-2, ROUGE-L: These are evaluation metrics used to measure the word-level accuracy between the generated responses and the ground truth labels. Higher values indicate higher similarity between the generated and actual responses [14, 15].

(2) Dist-1/2: This is an important metric for evaluating the performance of text generation models. It assesses the similarity between the generated text and the reference text. A higher Dist-1/2 value means the generated responses have greater diversity [16].

(3) Greedy Matching: This uses a metric method based on embeddings to calculate the semantic relevance between the generated and actual responses through cosine similarity. This helps assess the responses' quality and similarity [17].

4. Results and discussion

4.1. Experimental results

The experimental results are shown in Table 2.

Table 2. Experimental results.

4.2. Experimental analysis

From the above experimental data, it can be observed that in terms of evaluation criteria, both the improved personalized language model based on the Transformer architecture (IMDPchat) and the neural network models trained through reinforcement learning outperformed traditional baseline models. Furthermore, when comparing the test results of these two methods, it is clear that they perform similarly in terms of response accuracy, linguistic diversity, and semantic relevance, with little difference.

This indicates that both models have demonstrated excellent performance in generating personalized dialogues. This also means that significant progress has been made in this field, whether using end-to-end neural network methods or models trained through reinforcement learning. These achievements provide powerful tools for generating personalized dialogues and positively impact the field of natural language processing.

5. Conclusion

Open-domain generative dialogue systems represent a widely applicable artificial intelligence technology that can simulate natural language conversations, offering enormous potential across various fields. However, traditional dialogue systems often require a large corpus for input to generate dialogue content, making them less adaptable to different scenarios and contexts. A more challenging issue is that they sometimes provide predictable, safe, and uncreative responses, making it difficult for users to experience refreshing conversations.

Moreover, the shortcomings of traditional dialogue systems also include the potential to overlook user input, lack of personalized responses, and limitations due to rigid generation methods. These issues indicate that existing personalized dialogue generation models still have various shortcomings in modeling dialogue generation tasks.

Therefore, this study aims to analyze their performance in generative dialogue systems by comparing two methods: an improved personalized language model based on the Transformer, and a neural network model trained through reinforcement learning. The evaluation metrics focused on in this study include the accuracy of responses, linguistic diversity, and semantic relevance. The research found that both methods can achieve better results in this field than previous models.

Furthermore, combining these two methods may yield even more outstanding results. A more powerful and innovative personalized dialogue generation system can be built by fully leveraging their strengths, providing users with a more attractive and satisfying conversational experience. This integrated approach is expected to lead the development of the generative dialogue system field, compensating for the shortcomings of traditional methods, and paving the way for further advancements in intelligent dialogue technology.