1. Introduction

With the rapid growth of social media and e-commerce platforms, user-generated content has become a key resource for understanding consumer opinions and behavior. Xiaohongshu (also known as Little Red Book) is a popular social commerce platform in China where users share shopping experiences, reviews, and lifestyle tips. The sentiments expressed by Xiaohongshu users toward products and services provide valuable data for sentiment analysis. By extracting and analyzing these sentiments, businesses can gain insights into customer preferences, optimize product development, and enhance marketing strategies.

Sentiment analysis, a vital task in Natural Language Processing (NLP), aims to determine the emotional tone of a text. Traditional approaches, such as machine learning models based on bag-of-words and TF-IDF, have shown effectiveness in analyzing structured text but struggle to capture the complex semantics and context inherent in user-generated content [1]. The advent of deep learning and pretrained language models has provided more powerful tools for in-depth text understanding in sentiment analysis.

Among these models, BERT (Bidirectional Encoder Representations from Transformers) and its variants have become state-of-the-art for many NLP tasks [2]. RoBERTa (A Robustly Optimized BERT Pretraining Approach) builds on BERT by optimizing training strategies to achieve better performance across tasks. RoBERTa, combined with bidirectional Long Short-Term Memory (BiLSTM) networks and attention mechanisms, provides a robust solution for analyzing sentiment in complex user reviews [3]. The proposed model combines RoBERTa’s pretraining, BiLSTM, and attention mechanisms, making it well-suited for Xiaohongshu’s informal and multi-dimensional language.

This paper presents a novel approach to sentiment analysis of Xiaohongshu texts using the RoBERTa model combined with BiLSTM and attention mechanisms. By leveraging RoBERTa’s pretraining on large corpora and adding layers to capture both forward and backward semantics, this study aims to enhance the accuracy and depth of sentiment classification for Xiaohongshu reviews. The research will compare the performance of the proposed RoBERTa-based method with other commonly used sentiment analysis models and evaluate its effectiveness in identifying positive and negative sentiments within Xiaohongshu’s user-generated content.

2. Literature Review

2.1. Sentiment Analysis in Natural Language Processing

Sentiment analysis, also known as opinion mining, aims to identify the emotional tone of text and categorize it as positive, negative, or neutral. Early methods used lexicon-based approaches and traditional machine learning algorithms like Naive Bayes and Support Vector Machines (SVM) [4,5], relying on handcrafted features such as bag-of-words (BoW) and term frequency-inverse document frequency (TF-IDF). However, these methods struggled to capture complex semantics and context [6]. The advent of deep learning shifted sentiment analysis towards feature learning directly from data, eliminating the need for manual inputs, especially with recurrent neural networks (RNN) and long short-term memory (LSTM) networks that effectively preserve contextual information in sentences [7,8,9]. Nonetheless, these models faced challenges in handling long-range dependencies and deep semantic information [10].

2.2. Pretrained Language Models: BERT and RoBERTa

A major breakthrough in NLP came with pretrained language models, particularly BERT (Bidirectional Encoder Representations from Transformers). BERT was designed to understand bidirectional word context by pretraining on large corpora with objectives like Masked Language Model (MLM) and Next Sentence Prediction (NSP), significantly improving tasks like sentiment analysis and text classification [11]. RoBERTa (A Robustly Optimized BERT Pretraining Approach) was introduced as an enhanced version of BERT, removing the NSP objective, increasing batch size, and employing dynamic masking, which improved model generalization. These optimizations allowed RoBERTa to achieve state-of-the-art performance in many NLP benchmarks, making it a preferred model for complex tasks like sentiment analysis [12].

2.3. Applications of RoBERTa in Sentiment Analysis

RoBERTa’s ability to handle multilingual and informal language makes it highly relevant for sentiment analysis on platforms like Xiaohongshu, where users often use multiple languages and colloquial expressions [13]. Studies have shown that RoBERTa outperforms traditional models like Word2Vec and RNNs in terms of accuracy and F1-score when analyzing social media and e-commerce reviews, indicating its effectiveness in capturing complex linguistic patterns in user-generated content [13].

2.4. Advancements in Deep Learning Techniques: BiLSTM and Attention Mechanisms

Recent advancements in deep learning, especially the integration of Bidirectional Long Short-Term Memory (BiLSTM) networks and attention mechanisms, have significantly enhanced sentiment analysis models, improving accuracy and efficiency. BiLSTM networks process text in both forward and backward directions, capturing contextual information from preceding and succeeding tokens in a sequence, which is crucial in sentiment analysis where sentence meaning often depends on multiple parts [14,15,16]. Attention mechanisms allow models to focus on the most important words or phrases in a text by assigning weights to input elements, enhancing the model’s sensitivity to emotional cues in user-generated content [18,19,20].

2.5. Sentiment Analysis on Xiaohongshu

Sentiment analysis on Xiaohongshu, a social commerce platform, presents unique challenges due to the informal, brief, and multimedia-rich nature of user reviews. These characteristics complicate text-based sentiment extraction, yet Xiaohongshu analysis has great potential for insights into consumer preferences. Early studies mainly used lexicon-based or simple machine learning models, but these struggled with Xiaohongshu's complex context and language [21,22]. The RoBERTa-BiLSTM-Attention model addresses these challenges effectively by capturing context, adapting to language variability, and accurately identifying sentiment in diverse user-generated content.

3. Methodology

3.1. Dataset construction

Text data from Xiaohongshu notes, collected via web crawling from July to September 2024, was preprocessed to address its informal linguistic style. The preprocessing steps included:

1. Removing special characters: Noise such as user tags, hashtags, and emojis was filtered out using Python's re module.

2. Merging columns: Titles and main text were combined into one column to include all emotional cues using the pandas library.

3. Filtering abnormal data: Entries with fewer than three characters or duplicates were removed to improve data quality.

4. Manual labeling: Sentiments were labeled as positive (1) or negative (0).

A total of 10,747 notes were collected, encompassing lifestyle tips, product reviews, and shopping experiences. Of these, 5,290 entries were labeled as positive and 5,457 as negative. The dataset was split into training (80%), validation (10%), and test sets (10%), ensuring a balanced distribution of sentiment categories.

3.2. Experimental setup

3.2.1. Configure the experimental environment. The model was trained using an RTX 3080 GPU (10GB) in an Ubuntu-20.04 environment with Pytorch and Transformers. Key software and hardware parameters included:

CPU: Intel Xeon Platinum 8255C

Memory: 40GB

Python Version: 3.11

3.2.2. Hyperparameter configuration. In order to train a better model, the setting of model parameters is very important. After several rounds of experimental testing, the values of the relevant parameters in the sentiment analysis model in this chapter were finally determined. This is shown in Table 1.

Table 1. Hyperparameter Configuration Table

3.2.3. Baseline model. To further validate the effectiveness of the RoBERTa-BiLSTM-Attention sentiment analysis To validate the RoBERTa-BiLSTM-Attention model, its performance was compared with:

Word2Vec-LSTM: Word2Vec with LSTM for sentiment analysis.

Word2Vec-BiLSTM: Word2Vec with BiLSTM for improved contextual understanding.

BERT-BiLSTM: Combines BERT with BiLSTM for semantic feature extraction.

BERT-BiLSTM-Attention: Incorporates an attention mechanism for enhanced sentiment analysis.

3.2.4. Evaluation indicators. Model performance was measured using Accuracy and F1-score: Accuracy evaluates overall prediction correctness. F1-score balances precision and recall, highlighting robustness in imbalanced datasets.

The formula for calculating Accuracy is shown in Eq. (3.1).

\( Accuracy= \frac{TP+TN}{TP+TN+FP+FN}\ \ \ (3.1) \)

F1 is calculated as shown in Eq. (3.2).

\( F1= \frac{2* precision* recall}{precision* recall}\ \ \ (3.2) \)

Thereo:

\( precision= \frac{TP}{TP+FP}\ \ \ (3.3) \)

\( recall= \frac{TP}{TP+FN}\ \ \ (3.4) \)

4. Results

To evaluate the effectiveness of the proposed RoBERTa-BiLSTM-Attention model, comparative experiments were conducted with baseline models, including Word2vec-LSTM, Word2vec-BiLSTM, BERT-BiLSTM, and BERT-BiLSTM-Attention. Accuracy and F1-score were used as primary evaluation metrics, as they measure prediction correctness and the balance between precision and recall. Table 2 summarizes the results, showing that the proposed RoBERTa-BiLSTM-Attention model outperforms all baseline models with the highest accuracy (91.91%) and F1-score (91.39%).

Table 2. Comparison of experimental results for different models

The RoBERTa-BiLSTM-Attention model excels due to its advanced word embeddings from RoBERTa, which capture richer semantic information. Adding BiLSTM enhances forward and backward semantic understanding, while the attention mechanism emphasizes key emotional features. Further analysis examined the impact of sentence embedding. By removing the sentence embedding component (RBLA-Seq variant), the model showed reduced accuracy (91.26%) and F1-score (90.67%), highlighting the importance of sentence embedding in improving performance. The comparative results are summarized in Table 3.

Table 3. Comparative experimental results

The proposed model processes Xiaohongshu's text by generating word and sentence embeddings with RoBERTa. These embeddings are input into a BiLSTM layer to capture contextual semantics, and the attention mechanism enhances focus on key sentiment cues. Concatenating word embeddings with sentence embeddings further improves sentiment classification performance, particularly for Xiaohongshu's informal content.

5. Discussion

The comparative analysis highlights the unique advantages of the RoBERTa-BiLSTM-Attention model for Xiaohongshu sentiment analysis. Integrating RoBERTa with BiLSTM and attention mechanisms enables a more nuanced understanding of user sentiments. While Word2vec models struggled with polysemy, and BERT-based models showed contextual improvements, only the RoBERTa-BiLSTM-Attention model offered both deep semantic capture and weighting of sentiment-critical words, making it highly effective for Xiaohongshu's informal content. The inclusion of sentence embeddings further enhanced semantic coherence and performance. Experiments without sentence embedding confirmed their contribution to model accuracy, demonstrating their importance in handling complex social media texts. These findings validate the potential of advanced language models and attention mechanisms in addressing linguistic challenges of user-generated content and enhancing commercial sentiment analysis applications.

6. Conclusion

This study proposed a RoBERTa-BiLSTM-Attention model that significantly improves sentiment analysis accuracy for Xiaohongshu's informal content. By capturing nuanced semantic relationships and leveraging bidirectional semantic and attention mechanisms, the model outperformed traditional approaches. These findings highlight the importance of advanced techniques in processing user-generated content. Future research could extend this model to multilingual sentiment analysis, further refining its ability to capture and understand user sentiments across diverse social commerce platforms.

Acknowledgments

I would like to extend my gratitude to Professor Osman Yagan from Carnegie Mellon University for his insightful course on Social and Information Networks. The knowledge and techniques gained from this workshop, especially on topics like information propagation and misinformation analysis, greatly inspired and shaped the framework of this study. His guidance through the course's final project has been invaluable in enriching my understanding of the complex dynamics of social networks and their applications.