Sentiment Classification on tweets —Based on LSTM with variants and BERT finetune

Research Article
Open access

Sentiment Classification on tweets —Based on LSTM with variants and BERT finetune

Haoyuan Yang 1* , Wenyu Cai 2 , Qizheng Liu 3
  • 1 Wuhan University    
  • 2 University of Sydney    
  • 3 Beijing Normal University    
  • *corresponding author 2018302030118@whu.edu.cn
Published on 14 June 2023 | https://doi.org/10.54254/2755-2721/6/20230336
ACE Vol.6
ISSN (Print): 2755-273X
ISSN (Online): 2755-2721
ISBN (Print): 978-1-915371-59-1
ISBN (Online): 978-1-915371-60-7

Abstract

Covid-19 is a new type of epidemic, we performed sentiment classification tasks on Covid-19 tweets using different machine learning models. The famous pre-training models are not trained with the text relevant to COVID-19, a new kind of virus appearing at December 2019. The twitter posts with such a topic also have not been applied to test the performance of existing pre-training models and neural networks well. In our experiment, we used LSTM and Transformer to do the sentiment analysis (quinary classification) with a dataset including those twitter posts and tried different hyperparameters and models to improve the performance of classification. We finally found that the Transformer performs better than LSTM with an extra softmax layer in the encoder part, and the bidirectional transformer with 4 hidden layer and dropout 0.2 provides the best results among all hyperparameters we have tested. With the finetune of BERT,we got the best performance with the accuracy over 85%.

Keywords:

Sentiment Classification, LSTM, BERT

Yang,H.;Cai,W.;Liu,Q. (2023). Sentiment Classification on tweets —Based on LSTM with variants and BERT finetune. Applied and Computational Engineering,6,1533-1543.
Export citation

References

[1]. Galiatsatos, P. (2020, April 13). What Coronavirus Does to the Lungs. Www.hopkinsmedicine.org. https://www.hopkinsmedicine.org/health/conditions-and-diseases/coronavirus/what-coronavirus-does-to-the-lungs

[2]. Coronavirus tweets NLP - Text Classification. (n.d.). Www.kaggle.com. https://www.kaggle.com/datasets/datatattle/covid-19-nlp-text-classification

[3]. Schuster, Mike & Paliwal, Kuldip. (1997). Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on. 45. 2673 - 2681. 10.1109/78.650093.

[4]. S. Hochreiter and J. Schmidhuber, "Long Short-Term Memory," in Neural Computation, vol. 9, no. 8, pp. 1735-1780, 15 Nov. 1997, doi: 10.1162/neco.1997.9.8.1735.

[5]. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Brain, G., Research, G., Jones, L., Gomez, A., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. https://arxiv.org/pdf/1706.03762.pdf

[6]. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv:1810.04805 [Cs].

[7]. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv:1810.04805 [Cs]. https://arxiv.org/abs/1810.04805v2

[8]. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv.org. https://arxiv.org/abs/1907.11692

[9]. MrDoghead. LSTM&Bi-LSTM[EB/OL]. https://blog.csdn.net/mch2869253130/article/details/89150043, 2021-04-19

[10]. Mathor. The application of BiLSTM with PyTorch[EB/OL]. https://wmathor.com/index.php/archives/1447/, 2020-06-28

[11]. Cskywit. [LSTM NOTE5]How to Develop Stacked LSTMs[EB/OL]. https://blog.csdn.net/cskywit/article/details/87460704, 2019-02-16

[12]. TensorFlow. Neural machine translation with a Transformer and Keras[EB/OL]. https://www.tensorflow.org/text/tutorials/transformer

[13]. Rosenbaum, Paul R.; Rubin, Donald B. (1983). "The Central Role of the Propensity Score in Observational Studies for Causal Effects". Biometrika. 70 (1): 41–55. doi:10.1093/biomet/70.1.41.

[14]. Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7).

[15]. Kingma D, Ba J, 2015. Adam: A method for stochastic optimization[C]//Proceedings of International Conference on Learning Representations.

[16]. Loshchilov I, Hutter F. Decoupled weight decay regularization[J]. arXiv preprint arXiv:1711.05101, 2017.

[17]. Zeiler, M. (n.d.). ADADELTA: AN ADAPTIVE LEARNING RATE METHOD. https://arxiv.org/pdf/1212.5701.pdf

[18]. Kingma, D., & Lei Ba, J. (2017). ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION. https://arxiv.org/pdf/1412.6980.pdf

[19]. Rui Chen, Jie Cao & Dan Zhang. "Probabilistic Prediction of Photovoltaic Power Using Bayesian Neural Network - LSTM Model", 2021 IEEE 4th International Conference on Renewable Energy and Power Engineering (REPE), 2021

Cite this article

Yang,H.;Cai,W.;Liu,Q. (2023). Sentiment Classification on tweets —Based on LSTM with variants and BERT finetune. Applied and Computational Engineering,6,1533-1543.

Data availability

The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.

Disclaimer/Publisher's Note

The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of EWA Publishing and/or the editor(s). EWA Publishing and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

About volume

Volume title: Proceedings of the 3rd International Conference on Signal Processing and Machine Learning

ISBN：978-1-915371-59-1(Print) / 978-1-915371-60-7(Online)
Editor：Omer Burak Istanbullu
Conference website: http://www.confspml.org
Conference date: 25 February 2023
Series: Applied and Computational Engineering
Volume number: Vol.6
ISSN：2755-2721(Print) / 2755-273X(Online)

© 2024 by the author(s). Licensee EWA Publishing, Oxford, UK. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license. Authors who publish this series agree to the following terms:
1. Authors retain copyright and grant the series right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this series.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the series's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this series.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See Open access policy for details).

References

[1]. Galiatsatos, P. (2020, April 13). What Coronavirus Does to the Lungs. Www.hopkinsmedicine.org. https://www.hopkinsmedicine.org/health/conditions-and-diseases/coronavirus/what-coronavirus-does-to-the-lungs

[2]. Coronavirus tweets NLP - Text Classification. (n.d.). Www.kaggle.com. https://www.kaggle.com/datasets/datatattle/covid-19-nlp-text-classification

[3]. Schuster, Mike & Paliwal, Kuldip. (1997). Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on. 45. 2673 - 2681. 10.1109/78.650093.

[4]. S. Hochreiter and J. Schmidhuber, "Long Short-Term Memory," in Neural Computation, vol. 9, no. 8, pp. 1735-1780, 15 Nov. 1997, doi: 10.1162/neco.1997.9.8.1735.

[5]. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Brain, G., Research, G., Jones, L., Gomez, A., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. https://arxiv.org/pdf/1706.03762.pdf

[6]. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv:1810.04805 [Cs].

[7]. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv:1810.04805 [Cs]. https://arxiv.org/abs/1810.04805v2

[8]. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv.org. https://arxiv.org/abs/1907.11692

[9]. MrDoghead. LSTM&Bi-LSTM[EB/OL]. https://blog.csdn.net/mch2869253130/article/details/89150043, 2021-04-19

[10]. Mathor. The application of BiLSTM with PyTorch[EB/OL]. https://wmathor.com/index.php/archives/1447/, 2020-06-28

[11]. Cskywit. [LSTM NOTE5]How to Develop Stacked LSTMs[EB/OL]. https://blog.csdn.net/cskywit/article/details/87460704, 2019-02-16

[12]. TensorFlow. Neural machine translation with a Transformer and Keras[EB/OL]. https://www.tensorflow.org/text/tutorials/transformer

[13]. Rosenbaum, Paul R.; Rubin, Donald B. (1983). "The Central Role of the Propensity Score in Observational Studies for Causal Effects". Biometrika. 70 (1): 41–55. doi:10.1093/biomet/70.1.41.

[14]. Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7).

[15]. Kingma D, Ba J, 2015. Adam: A method for stochastic optimization[C]//Proceedings of International Conference on Learning Representations.

[16]. Loshchilov I, Hutter F. Decoupled weight decay regularization[J]. arXiv preprint arXiv:1711.05101, 2017.

[17]. Zeiler, M. (n.d.). ADADELTA: AN ADAPTIVE LEARNING RATE METHOD. https://arxiv.org/pdf/1212.5701.pdf

[18]. Kingma, D., & Lei Ba, J. (2017). ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION. https://arxiv.org/pdf/1412.6980.pdf

[19]. Rui Chen, Jie Cao & Dan Zhang. "Probabilistic Prediction of Photovoltaic Power Using Bayesian Neural Network - LSTM Model", 2021 IEEE 4th International Conference on Renewable Energy and Power Engineering (REPE), 2021

For authors
Guide for authors Policies for authors Aims and scope Subscription Open access policy
For editors
Policies for editors Editorial board
For reviewers
Policies for reviewers Privacy policy

Cookies are used by this site.
Copyright © 2024 EWA Publishing, its licensors, and contributors. All rights reserved, including text and data mining, AI training, and similar technologies.