1. Introduction

Sentiment analysis, also known as emotional disposition analysis or opinion mining [1], focuses on how to extract users' opinions, attitudes, and emotions, etc. from text, audio, or images, etc., among which textual sentiment analysis focuses on emotionally-charged text and mines its embedded emotional tendencies. Text sentiment analysis has grown in importance as a field of study within natural language processing during the past few years, necessitating the development of more sophisticated technical approaches. Previous related research identifies three basic approaches to text sentiment analysis: those that rely on a sentiment lexicon, those that use conventional machine learning, and those that use deep learning [2].

Deep learning is a significant subfield of machine learning theory that tries to model the human brain in order to more precisely and efficiently extract features from sample data for the purposes of machine recognition and analysis. Today, deep learning is widely used for several tasks, including speech recognition, image recognition, and NLP [3]. Deep learning for natural language processing was first explored in 2008 when Collobert et al. applied it to common NLP tasks like POS tagging using embeddings and multilayer one-dimensional convolutional structures [4]. This paper follows the framework established by the literature [2], which categorizes deep learning-based text sentiment analysis methods into four categories: analysis with a single neural network; analysis with a hybrid (combined, fused) neural network; analysis with the introduction of an attention mechanism; and analysis with a pre-trained model. This paper aims to examine deep learning's potential as a technical solution for text sentiment analysis, to categorize the most important models of deep learning-based text sentiment analysis from the research literature available in CNKI, and to make suggestions for further study in this area.

2. Deep learning based approaches

2.1. Neural network

The interconnected neurons of a real neural network are mimicked in an artificial neural network. There are three layers to a neural network model: the input layer, the hidden layer, and the output layer. An input layer, where each neuron represents a feature and receives the input signal, is fully connected to a hidden layer, where the number of neurons varies based on the circumstance, and an output layer, where the number of neurons reflects the number of classifications, according to the literature [5]. These days, CNNs, RNNs, LSTMs, and Bi-LSTMs are the mainstays of neural network learning.

A convolutional neural network, or CNN, has a hidden layer made up of a convolutional layer, a pooling layer, and a fully connected layer. A convolutional group is made up of the convolutional layer and one or more of the pooling layers, which are selected in a random order and placed in an alternating pattern. The learning process of CNN proceeds layer-by-layer, from the local features to the global features, and ultimately achieves the classification through the fully connected layer. Experimental results on the COAE2014 Task 4 corpus showed that CNNs are effective for sentiment analysis of Chinese text [6], and researchers Liu Longfei, Yang Liang, Zhang Shaowu, et al. used word-level vectors of vocabulary as raw features for microblogging sentiment tendency analysis with CNN. Using character-level, vocabulary-level, and concept-level distributed representations as three-channel inputs, ICNNSTCM proposed by Gao Yunlong, Wu Chuan and Zhu Ming extracts their numerical features through convolutional operations, and adds sparse self-coding measure rate in the fully-connected layer to decrease the model's complexity while simultaneously increasing its generalization ability and experimental results [7].

Recurrent neural networks, or RNNs, are a type of neural network that are typically employed to handle sequences of varying lengths. In contrast to a CNN, an RNN may pick up on contextual semantics and use previously memorized information to better understand the context of the current event. Due to its susceptibility to gradient vanishing and explosion when processing lengthy text sequences, Hochreiter et al. proposed LSTM as a more robust structure of RNN [8]. Based on this, Niu Chengming, Zhan Guohua, and Li Zhihua proposed a Word2Vec and LSTM improved RNN model for Chinese microblogging sentiment analysis [9]. This involves replacing each hidden layer of RNN with a cell with memory function, which has a better processing capability for time series and linguistic text sequences. The Bi-LSTM model consists of two LSTM networks, each of which is fed into a separate LSTM neural network in forward and reverse order for feature extraction, and then the resulting feature vector is spliced together. Since a lot of linguistic knowledge and sentiment resources are underutilized in sentiment analysis tasks, Li Weijiang and Qi Fang proposed a multi-channel Bi-LSTM based sentiment analysis model called Multi-Bi-LSTM [10], which models the existing linguistic knowledge and sentiment resources in the sentiment analysis task, generates different feature channels, and allows the model to fully learn the sentence's sentiment information, it achieving better performance than Bi-LSTM, CNN combined with sentiment sequence features, and traditional classifiers in experiment.

Since traditional RNNs are unable to recall information for a long time and a single CNN cannot accurately represent the contextual semantics of text, Yang Yunlong, Sun Jianqiang, and Song Guochao proposed a sentiment analysis model GCaps that integrates GRU and capsule features [11], which captures the contextual global features of the text through GRU to obtain the overall scalar information, and then iteratively processes the captured information via a dynamic routing algorithm at the initial capsule level to obtain the vectorized feature information that represents the overall attributes of the text, and finally the combination between features is carried out in the main capsule part to obtain more accurate text attributes and analyze the sentiment polarity of the text according to the intensity of each feature. Experimental results on the benchmark dataset MR demonstrate that compared to CNN+INI and CLCNN, GCaps achieves a 3.1% and a 0.5% improvement in classification accuracy. Liang Zhijian, Xie Hongyu and An Weigang proposed a text classification method based on Bi-GRU and Bayesian classifier [12]. They use Bi-GRU to extract text features, assign weights by TF-IDF algorithm, and use Bayesian classifier to discriminate classification, which improves the shortcomings of GRU's insufficient dependence on post-text, shortens the training time of the model, and improves the efficiency of text classification. Comparative simulation experiments are carried out on two types of text data, and the experimental results show that the classification algorithm can effectively improve the efficiency and accuracy of text classification compared with the traditional RNN.

Due to the fact that each neural network method has its own set of benefits and drawbacks, many researchers have turned to hybrid methods while doing text sentiment analysis. With the gradient explosion problem of RNN limiting the accuracy of text classification, and the existence of backward and forward dependency in the structure of natural language, Li Yang and Dong Hongbin proposed a CNN and Bi-LSTM feature fusion model that uses CNN and Bi-LSTM to extract the local features of text vectors and the global features related to the text [13]. Experimental results demonstrate that the suggested feature fusion model significantly boosts text categorization accuracy over competing methods. To address the shortcomings of previous approaches to text sentiment analysis—namely, inaccurate results, lengthy processing times, and a lack of relevant features—Zhao Hong, Wang Le, and Wang Weijie developed a Bi-LSTM-CNN serial hybrid model [14]. This model first employs Bi-LSTM to extract the text's context, then CNN to extract local semantic features from the extracted features, and finally Softmax to derive the text's emotional tendency. This model improves the overall evaluation index F1 by 1.86% compared to LSTM-CNN and by 0.76% compared to Bi-LSTM-CNN parallel feature fusion. A text sentiment analysis model that combines CNN and Bi-GRU was proposed by Miao Yalin, Ji Yichun, Zhang Shun et al. to address the issue of heavy workload caused by conventional sentiment analysis methods and the disregard of network training speed by most deep learning approaches [15]. Extracted via a CNN and Bi-GRU, the text's local static and sequential features are coupled to a GRU layer for further dimensionality reduction before being passed to Sigmoid for sentiment classification. Experiments conducted on the self-created Douban movie and TV review dataset demonstrate that, in comparison to the CNN-BLSTM model of the same complexity, this model improves classification accuracy by 2.52% and training rate by 41.43%.

2.2. Attention mechanisms

The first to introduce the attention mechanism into the field of natural language processing was Bahdanau et al. [16], who applied it to the field of machine translation. The nature of the attention mechanism is a set of weight values distribution, which is manifested in the field of natural language processing as words with higher weights are more important throughout the text and play a greater role in the overall classification task [17]. The attention mechanism can extend the ability of neural networks to approximate more complex functions to focus on specific parts of the input.

Feng Xingjie, Zhang Zhiwei, and Shi Jinchuan combined a convolutional neural network (CNN) with an attention model (AM) for text sentiment analysis, and their experiments demonstrated significant improvements in accuracy [18], recall, and F1 measure compared to both traditional machine learning methods and pure AM methods. Using deep learning for sentence-level sentiment analysis tasks as a starting point, Guan Pengfei, Li Baoan, Lyu Xueqiang, and others suggested an attention-enhanced Bi-LSTM model [19]. The model employs Bi-LSTM to learn the semantic information of the text, which improves the classification effect through parallel fusion, and the attention mechanism to learn the weight distribution of each word on the sentiment tendency of a sentence directly from the basis of word vectors. Experimental results on the NLPCC2014 sentiment analysis corpus show that this model performs better than others at classifying the tone of individual sentences. Shi Lei, Zhang Xinqian, and Tao Yongcai et al. built the SAtt-TLSTM-M model by combining the self-attention mechanism with the introduction of Maxout neurons at the output of the Tree-LSTM model in order to address the issues of information memory loss [20] and the negligence of the correlation between context-discontinuous words, and gradient dispersion in RNN models. Using the COAE2014 evaluation dataset, the accuracy of the model was shown to be higher than that of the standard SVM, MNB, and LSTM models for sentiment analysis by a margin of 16.18%, 15.34%, and 12.05%, respectively. Zhang Jin, Duan Liguo, Li Aiping, and others suggested a text sentiment classification model that combines Bi-GRU-Attention and a gating mechanism to perform aspect-level fine-grained sentiment analysis based on user comments [21]. They combine the negation dictionary and lexical information to increase the user evaluation sentiment knowledge, and use the user evaluation sentiment knowledge as the user review sentiment feature information to integrate the existing sentiment resources. The seed sentiment dictionary is the HOWNET evaluation sentiment dictionary, and the SO-PMI algorithm is used to expand the user review sentiment dictionary. Next, they use Bi-GRU to do deep feature extraction on the text by introducing the word feature and sentiment feature information, and then combining these as model inputs. At the output layer, text sentiment analysis is conducted, and the final sentiment polarity is obtained via Softmax. This process begins with the acquisition of information about a text's aspect words and continues with the extraction of contextual sentiment features related to those words using the gating mechanism and the attention mechanism. Improved experimental outcomes are achieved by testing the model on the Chinese dataset of AIchallenger2018's fine-grained sentiment analysis, where it achieves a MacroF1score value of 0.7218.

Some researchers have also implemented the attention mechanism on the basis of hybrid neural networks, expanding on what has previously been done with single neural networks. A text categorization technique based on a hybrid model of LSTM-Attention and CNN was proposed by Teng Jinbao, Kong Weiwei, and Tian Qiaoxin et al. to address the limitation of standard LSTM and CNN in not being able to express the importance of each word in the text while extracting features [17]. After CNN has extracted the relevant local information from the text, the whole text semantics can be integrated. The attention mechanism is then introduced after LSTM to extract the attention score of the output information, and LSTM is utilized to extract the text context features. Finally, CNN's output is fused with LSTM-Attention's, enabling the focus of attention to be directed on key words via efficient feature extraction. The model's accuracy is 90.23%, and the F1 measure is 90.12%, according to experimental results on three public datasets; these numbers are higher than those of LSTM and CNN. Since the information between neurons in the same layer of traditional CNN cannot be transmitted to each other, unable to make full use of the feature information in the same level, and the lack of the representation of sentence system features leads to the limited feature learning ability of the model, Wang Liya, Liu Changhui, and Cai Dunbo et al. proposed a model based on the introduction of the attention mechanism of the joint CNN-Bi-GRU network [22]. The model employs CNN to extract deep phrase features, Bi-GRU for serialized information learning to obtain sentence system features, strengthened linking of CNN pooling layer features, and attention mechanism to complete effective feature screening. Experiments comparing the model to multiple groups on the dataset reveal that it is able to enhance text categorization accuracy for minimal effort and time investment. The deviation of weight allocation is caused by the fact that both the traditional self-attention mechanism and the Bi-GRU network disregard the local dependencies that exist between the texts, leading to inaccurate predictions. SAttBiGRUMCNN was proposed by Chen Kejia and Liu Hui [23]; it is a text categorization model built on Bi-GRU with an enhanced self-attention mechanism and a multi-channel CNN. Bi-GRU is utilized to obtain the text's contextual semantic information at the global level, and then the text's local features are extracted using the optimized multi-channel CNN. Based on this, they incorporate a position weight parameter and redistribute the self-attention weight probability value based on the text vector training location before using Softmax to classify the sample labels. The experimental findings on two standard datasets demonstrate the model's accuracy of 98.95% and 88.1%, respectively, outperforming FastText, CNN, RCNN, and other classification methods.

Furthermore, Gao Jiaxi and Huang Haiyan suggested a text sentiment analysis model based on the TF-IDF and multi-head attention Transformer model to deal with the issue that current computational methods cannot appropriately handle text datasets with high complexity and confusion [24]. The TF-IDF algorithm is used in the text preprocessing stage to initially screen the words that affect the text's sentiment tendency to a greater extent, while ignoring the common stop words and other proper nouns that have less influence. Finally, the Transformer model encoder trained with multi-head attention is employed for feature extraction to further understand and generalize the text's semantics. When applied to the Ec_60k dataset, for instance, the model has an accuracy of 98.17%.

2.3. Pre-training model

In NLP, pre-trained models are deep network architectures that have been trained on a large, unlabeled text corpus. Word2vec, a static pre-training model, has been the most popular text representation method since the advent of NLP technology; however, it has the flaws of learning only a shallow representation of the text and being context-independent, so it has little bearing on the improvement of subsequent tasks. After much investigation, researchers created dynamic pre-training models (ELMo and BERT, primarily) to address the aforementioned issues. Since the introduction of the BERT model ushered in a new age, numerous more pre-training models have been developed, most of which may be categorized as either better BERT-based models or XLNet [25].

The ELMo model is made up of a forward and backward Bi-LSTM language model and an objective function to maximize the likelihood of the model. In contrast to more standard word vector models, this one allows each word to map to exactly one word vector. To address the limitation that word embedding methods like Word2Vec and GloVe only generate a single semantic vector for polysemous words, Zhao Yaou, Zhang Jiachong, Li Yibin, et al. presented an ELMo-MSCNN hybrid model for sentiment analysis [26]. The model learns the pre-trained corpus with ELMo, generates contextually relevant word vectors, and initializes the embedding layer of ELMo with the pre-trained Chinese character vectors, which accelerates the training speed and improves the training accuracy; then the features of word vectors are extracted twice with MSCNN, and feature fusion is performed to generate the overall semantic representation of the sentence; finally, the classification of textual sentiment tendency is realized after Softmax excitation function. Experiments are run on two publicly available datasets (hotel reviews and NLPCC2014 task2), with the results showing an improvement in model accuracy of 1.08% compared to the attention-based Bi-LSTM model on the hotel reviews dataset and an improvement in accuracy of 2.16% compared to the LSTM-CNN model on the NLPCC2014 task2 dataset. To get around the fact that CNN is unable to directly extract bidirectional semantic features of sentences and that conventional word embedding methods are unable to effectively represent the multiple meanings of a word, Zhao Yaou, Zhang Jiachong, Li Yibin et al. proposed a hybrid model based on ELMo and Transformer [27]. This model employs the ELMo model to produce word vectors, incorporates contextual features of the sentence into the word vectors, and produces distinct semantic vectors for the various semantics of polysemous words; the obtained ELMo word vectors are then fed into the Transformer model with modified Encoder and Decoder structures for sentiment classification. Since the model can extract semantic properties of sentences from multiple perspectives, the resulting semantic information is completer and more nuanced. The experimental results show that compared with the current mainstream methods, this one improves the classification accuracy by 3.52% on the NLPCC2014 Task2 dataset, and by 0.7%, 2%, 1.98%, and 1.36%, respectively, on the four subdatasets of hotel reviews. It was proposed by Wu di, Wang Ziyu, and Zhao Weichao that they use a model called ELMo-CNN-Bi-GRU [28]. The model uses ELMo and Glove pre-training models to generate dynamic and static word vectors, respectively, generates input vectors by stacking and embedding the two word vectors, and adopts the self-attention mechanism to process the inputs; then, the internal word dependencies are calculated to construct a dual-channel neural network structure fusing CNN and Bi-GRU, and the text local features and global features are acquired simultaneously; finally, the dual-channel neural network structure is trained to predict the Compared to the H-Bi-GRU model, which excels among similar sentiment classification models, the experimental findings reveal that the model improves accuracy and F1 value on the IMDB, yelp, and sentiment140 datasets.

The BERT model, first presented by Google, takes into account the contextual semantics of words by using WordPiece embedding as word vectors in the input, as well as position vectors and sentence tangent vectors, and the bidirectional transformer mechanism for the language model [29]. Chen Zhiqun and Ju Ting used the BERT model to extract semantic feature representations of microblog comment texts, and then fed the acquired word semantic features into the Bi-LSTM model for propensity classification to address the issue of word polysemy that cannot be solved by traditional language models in word vector representations [30]. Experiments conducted on Sina Weibo comment data demonstrate that this model achieves an F1 value of 91.45%, placing it above other widely used propensity analysis models. In order to address the imbalance in evaluation, Liu Ji and Gu Fengyun proposed the model M2BERT-BiLSTM [31], which combines BERT and Bi-LSTM. The model first converts the sequences in the hidden layer of the BERT model into vectors, then splices them along the dimensions of sentence length according to the pooling of the mean and maximum values, and finally inputs the semantic features of the spliced words into the Bi-LSTM for the textual sentiment analysis in order to alleviate the imbalance in the evaluation. The experimental findings demonstrate that the model provides a more accurate assessment of the indicators. To address the limitations of sentiment characteristics derived by conventional pre-training models, Huang Zemin and Wu Yinggang suggested a BERT-CNN(LRN)-BiSRU model for text sentiment classification [32], which combines BERT with a convolutional bidirectional simple recurrent network. In order to obtain dynamic word vectors that incorporate sentence context, the model is pre-trained with BERT; the word vector features are then extracted twice with a multi-granularity convolutional neural network; the pooled and injected into the local response normalization layer LRN to normalize the feature map; and finally, the two-way simple loop unit is used to further learn the contextual semantic information. The model achieves a high F1 value of 91.27% in experiments, demonstrating its usefulness in real-world settings. The BERT model has been further improved by the addition of the attention mechanism proposed by a number of academics. An integrated BERT-TCN and attention mechanism model for text sentiment analysis was proposed by Zhang Jian [33]. The transformer-based BERT model is used first to obtain the text word vectors containing contextual semantics; the TCN model is then used to further extract the contextual semantic features of the text word vectors; the attention mechanism is then introduced to focus on the crucial sentiment features in the context; and finally, the Softmax classifier is used to perform the sentiment classification. The experimental results suggest that the model outperforms the BERT-TCN model in terms of precision, recall, and F1 score.

The XLNet model is an improvement over the BERT model in that it can be trained in a bidirectional context without having to mask some of the predicted words. Li Dongjin, Shan Rui, and Yin Liangkui et al. proposed a text sentiment analysis model combining a generalized autoregressive pretrained language model, using XLNet to represent text features, extracting local features of text vectors through CNN, then using Bi-GRU to extract deeper contextual information of the text, introducing an attention mechanism to assign different weights to the features according to their importance, and performing a text sentiment polarity analysis [34]. The simulation experiment contrasts the accuracy of this model to that of five commonly used sentiment analysis models, and the results demonstrate that this model is significantly more accurate.

3. Conclusion

Now more than ever before, people use the Internet to share their opinions and emotions with the world. Interaction via text not only replicates occurrences but also conveys feelings. Given this context, text sentiment analysis becomes a powerful tool for understanding the state of affairs and making informed predictions about the future, with potential uses in areas as diverse as public opinion research, user profiling, and more. This paper examines the use of deep learning in text sentiment analysis technology by reviewing relevant CNKI articles and providing an explanation of how it works. Three primary technique models are introduced, and their potential enhancements are discussed.

Deep learning-based approaches for text sentiment analysis start with CNN and to improve the problem of accessing contextual semantics, RNN is used to apply memorized content. In addition to CNN and RNN, the researchers suggested LSTM and GRU for model optimization, and then suggested Bi-LSTM and Bi-GRU by rearranging the input order. Scholars are also interested in studying and applying hybrid neural network models, with the majority of this work focusing on CNNs paired with LSTMs or GRUs. Researchers have built on this foundation by adding attention processes to the model or by employing pre-trained models to enable horizontal model extension. When training on a large scale with the dataset, the pre-training model learns contextual semantics and the semantics of polysemous words, which greatly improves the classification effect. The attention mechanism uses the importance of words in the text to better capture the contextual information, leading to more accurate text classification. Text sentiment analysis is a promising area of study. Researchers in this area have developed numerous approaches and models for use in real-world settings, and it is believed that this field will have richer research results in the future.