1. Introduction

The proliferation of e-commerce platforms has profoundly altered consumer behavior, with a growing number of individuals turning to online shopping. As a result, user-generated comment data has become a valuable resource for both consumers and businesses alike. These comments offer rich insights into product experiences, serving as a crucial reference for users and a valuable source of feedback for companies. [1-2] Within the realm of text mining, sentiment analysis, particularly aspect-based sentiment analysis, has emerged as a pivotal research direction. Unlike traditional sentence-level sentiment analysis, aspect-based sentiment analysis delves deeper into user opinions by considering multiple aspects within a single sentence, posing increased difficulty in prediction. While deep learning methods have been widely employed to tackle sentiment analysis tasks, shallow neural network models often struggle to effectively extract relevant features from comments. To address this limitation, our approach combines original comment texts with pre-trained language models to construct auxiliary training samples, [3] facilitating improved feature extraction. Additionally, we introduce an enhanced attention mechanism that focuses on important features pertinent to the current aspect category, enhancing the accuracy of polarity prediction. Experimental validation across multiple aspect sentiment datasets validates the efficacy of our proposed model, while further analysis sheds light on factors influencing its performance, providing valuable insights for future research endeavors and practical implementation in e-commerce settings.

2. Related Work

Aspect-based sentiment analysis has evolved significantly over time, progressing from rule-based methods relying on sentiment lexicons to machine learning techniques focusing on more effective text representation. Currently, deep learning neural networks are at the forefront of research in this field. For instance, the [4] Seq2Path model challenges the idea of fixed positional relationships between sentiment tuples and utilizes a seq2seq encoder-decoder architecture to generate sentiment paths resembling tree structures. On the other hand, the [5] PASTE model employs [6] Bi-LSTM and BERT to capture sentence features and employs a pointer network to accurately extract aspect words, sentiments, and opinions, enhancing triple extraction accuracy.

Another notable advancement is the BARTABSA [7] model, which addresses multiple subtasks within aspect-based sentiment analysis using a unified architecture based on BART. By transforming inputs into vectors and utilizing a pointer network for aspect category labeling, BARTABSA achieves comprehensive analysis within a single framework. Similarly, the [8] DOER model takes a multi-task approach, jointly modeling aspect sentiment analysis and aspect term extraction through sequence labeling. It utilizes shared framework units and cross-framework units to effectively extract and propagate information within the model, streamlining the complexity of the tasks.

Lastly, TAS-BERT represents a jointly trained model capable of identifying targets and sentiments simultaneously, addressing implicit aspects within sentences. By leveraging pre-trained language models and incorporating CRF or softmax decoders, TAS-BERT [9] achieves robust sentiment analysis while considering the nuanced relationships between targets and sentiments.Overall, these advancements underscore the importance of leveraging deep learning techniques and unified architectures for more accurate and comprehensive aspect-based sentiment analysis.

In this paper, we combine BERT fine-tuning tasks with a cross-attention mechanism for aspect sentiment polarity analysis. We construct auxiliary samples, fine-tune with pre-trained BERT models, extract features using a cross-attention network, conduct sentiment polarity classification, and achieve the best results to date.

3. Methodology

In this model as shown in Figure 1, the features of text and aspect markers are extracted by BERT. Auxiliary training samples are first constructed, transforming aspect sentiment tasks into reading comprehension or natural language inference tasks to extract text features and sentence features, Comments features are represented by \( Vevtor-D \) , while aspect tag features are represented by \( Vevtor-Q \) . Aspect tags can be single words or phrases. Applying document attention to \( Vevtor-D \) yields the result represented by \( Vevtor-S \) . To ensure the model focuses on the most relevant features for a given aspect, we perform cross-attention feature extraction between comment features and aspect tag word features. We denote the result of this cross-attention as \( Vevtor-R \) , which is then concatenated with \( Vevtor-S \) . Finally, we can predict the sentiment polarity of the given aspect category text through a classification layer consisting of fully connected layers and softmax. Specifically, there are various methods for constructing auxiliary samples. In this paper, we use a mixture of methods to construct auxiliary samples. Sample question-answer method: Based on the provided sample, ask how to evaluate a specific aspect, with the correct answer being the sentiment polarity of that aspect. Sample inference method: Provide a template, similar to a fill-in-the-blank question, "The user's evaluation of a certain aspect in the sentence is". Then, concatenate the auxiliary samples after the corresponding samples and input them into the [10-12] BERT model. For each word, its input representation is constructed by adding corresponding word, segment, and position embeddings bitwise. For classification tasks, the first token position of each sentence sequence is a special token ([CLS]). Use the first special token [CLS] of the final hidden state layer as the final vector representation of the input sequence.

Figure 1. The architecture diagram of the model proposed in this paper.

The document attention can be represented as follows:

\( {u_{t}}=tanh⁡({d_{t}}{W_{w}}+{b_{w}}) \) (1)

\( {α_{t}}=\frac{exp⁡({u_{t}}{u_{w}})}{\sum _{t=0}^{n}exp⁡({u_{t}}{u_{w}})} \) (2)

\( s=\sum _{t=0}^{n}{α_{t}}{d_{t}} \) (3)

In this formula, we utilize a module composed of attention weights and parameters \( {W_{w}} \) and \( {b_{w}} \) , which constitute the bias terms, to extract output weights for various features, integrating them into a vector \( s∈{R^{H}} \) . For each different aspect category, we have a corresponding document attention layer, which, through a cross-attention mechanism, guides the network to focus on key parts of the comment text related to the aspect category. We can calculate the matching score between \( D∈{R^{(n+1)×H}} \) and \( Q∈{R^{(m+1)×H}} \) as follows:

\( {M_{(i,j)}}=d_{i}^{T}∙{q_{j}} \) (4)

Where \( M∈{R^{(n+1)×(m+1)}} \) . Then, by applying max pooling to \( M \) to obtain the maximum values along each row, the result is represented by \( E∈{R^{(n+1)}} \) . Since \( M \) represents the matching scores between text features and aspect labels, and each element in \( E \) can be interpreted as the relevance between a text word and a given aspect category [13]. By scaling \( E \) using softmax, we can obtain \( β∈{R^{(n+1)}} \) , and with this, weighted summation according to relevance of text features, we can obtain \( r∈{R^{H}} \) :

\( β=softmax(E) \) (5)

\( r=\sum _{t=0}^{n}{β_{t}}{d_{t}} \) (6)

Connecting the vectors \( s∈{R^{H}} \) and \( r∈{R^{H}} \) together, we can obtain \( z∈{R^{2H}} \) . Classification of sentiment polarity can be achieved through a classification layer, where \( {p_{j}} \) represents the sentiment polarity of the current aspect category.

\( {p_{j}}=softmax(\sum _{i=1}^{n}{W_{i}}z) \) (7)

The loss function of the model is as follows:

\( {loss_{i}}=-\sum _{j}\sum _{k}{weight_{ik}}y_{j}^{k}log\hat{y}_{j}^{k} \) (8)

\( {weight_{ik}}={log_{base}}\frac{all\_sample}{{class\_sample_{ik}}} \) (9)

\( {loss_{total}}=\sum _{i}{loss_{i}} \) (10)

4. Experimental Results and Analysis

4.1. Datasets

Experimental work was conducted on three datasets (Table 1), the details of which are as follows:

• ASAP [14]: This dataset is for sentiment analysis and consists of restaurant review data. It includes real user review data from online food ordering platforms. Samples in this dataset contain users' star ratings for their dining experiences, as well as their evaluations of various aspects of the dining experience, such as food taste, environment, service, and 18 other predefined aspect categories. The dataset was randomly divided into training, validation, and test sets.

• ACOS [15]: This dataset is a quadruple-extraction dataset for aspect sentiment, namely aspect term - aspect category - opinion - sentiment quadruples. It contains text data from restaurant and laptop reviews. The laptop review data is sourced from the Amazon shopping website and is twice the size of past datasets. Samples in the dataset not only include explicitly expressed aspects and opinions but also implicit aspects and opinions, increasing the difficulty of learning from this dataset.

• ASTE-Data-V25 [16]: This dataset is a sentiment multi-tuple dataset involving aspect categories, sentiment polarities, and sentiment causality relationships. It is a highly challenging dataset in the aspect sentiment domain. Each sentence contains multiple aspect categories and corresponding sentiments. It includes explicit aspect categories when aspect terms appear in the sentence and implicit aspect categories when aspect terms do not appear in the sentence. For example, [('battery', 'durable', 'Positive').

Table 1. Datasets description

4.2. Training Protocol

To mitigate the risk of overfitting during model training, a data augmentation method involving random deletion of text content was employed. All methods incorporated a hyperparameter called Maximum Deletion Ratio ( \( MDR \) ), where \( MDR \) defined the maximum random deletion rate of input text during training, and the average deletion rate was set to \( MDR / 2 \) . Pre-trained uncased BERT-base was used for fine-tuning. The model consisted of 12 Transformer blocks, with a hidden layer size of 768 and 12 self-attention modules, totaling 1.1 million parameters. During fine-tuning, a dropout probability of 0.1 was maintained, and the number of training epochs was set to 10. The initial learning rate was set to \( 2e-5 \) , with a batch size of 24. \( Adam \) optimizer was utilized for optimization. Each model was run 5 times, and the best results for all models were reported on the test dataset.

4.3. Evaluation Strategy

This paper utilizes \( Accuracy \) and \( F1 \) scores for evaluation.

\( F1=2×\frac{precision×recall}{precision+recall} \) (11)

\( Accuracy=\frac{TP+TN}{TP+TN+FP+FN} \) (12)

4.4. Results Analysis

Table 2. Experimental Result

Overall, as shown in Table 2, Our Model performs the best in terms of accuracy and F1 scores across all datasets. It is noteworthy that although Our Model's performance varies across different datasets, it remains the most stable model overall, indicating its strong generalization ability. It can be observed that all models exhibit similar performance on the [17] ACOS dataset compared to the ASTE-Data-V25 dataset, but a decrease in performance is noted when compared to the Data1 dataset. However, these models offer shorter training times and higher computational efficiency, making them suitable for lightweight applications. This highlights the advantage of simple model structures and efficient computational efficiency. Therefore, for simple tasks requiring efficient execution, shallow neural network-based models are recommended, while scenarios demanding high performance may benefit from models based on large language frameworks.

5. Conclusion

With the development of the Internet, people's shopping habits have undergone significant changes, from offline to online. E-commerce platforms have increasingly become the carriers of users' shopping needs. While purchasing goods, users express their opinions on products and services through comments. These comment data contain rich user experiences, which are crucial for companies to understand user sentiments. Sentiment analysis of comment text is a hot research direction in text mining. Compared to sentence-level sentiment analysis, aspect-based sentiment analysis can provide more in-depth and fine-grained information. Aspect sentiment analysis initially relied on rule-based methods, followed by machine learning methods that focused on finding more effective sentiment features at the text representation level. Now, the mainstream research method is based on deep learning approaches. This paper proposes an aspect category sentiment analysis model based on BERT, which can provide more accurate product feedback for platforms and merchants. The model extracts features using BERT and applies a cross-attention mechanism to improve model performance. Experimental results demonstrate that the model performs excellently on three datasets: ASAP, ACOS, and ASTE-Data-V25, validating the effectiveness of the model.