Download pdf
Comparative study of sequence-to-sequence models: From RNNs to transformers
- 1 University of California
* Author to whom correspondence should be addressed.
Abstract
In this comprehensive exploration of sequence-to-sequence models in Natural Language Processing (NLP), we have traced the trajectory of their evolution and contributions. Starting from foundational Recurrent Neural Networks (RNNs) to the revolutionary capabilities of Long Short-Term Memory (LSTM), In this comprehensive exploration of sequence-to-sequence models in Natural Language Processing (NLP), we have meticulously traced the trajectory of their evolution and impactful contributions. From the foundational Recurrent Neural Networks (RNNs) to the revolutionary capabilities of Long Short-Term Memory (LSTM), as well as the transformative innovations brought forth by Transformers and BERT, this review eloquently highlights the monumental advancements that have fundamentally reshaped our understanding and generation of language. The crux of our comparative analysis lies in its ability to spotlight the distinctive strengths and limitations inherent in each model. Through an intricate examination, we uncover their nuanced applications across a diverse spectrum of NLP tasks. Particularly noteworthy is the pivotal role played by Transformers and the transformative Bidirectional Encoder Representations from Transformers (BERT). The paper concludes with a summary and outlook of the entire paper.
Keywords
Sequence-to-Sequence, Recurrent Neural Networks, LSTMs, Transformers
[1]. Cho, Kyunghyun, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint (2014)
[2]. Li, Shuai, et al. Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN. Computer Vision and Pattern Recognition, (2018)
[3]. Schuster, Mike, and Kuldip K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing 45.11 (1997)
[4]. Pascanu, Razvan, Caglar Gulcehre, and Yoshua Bengio. How to construct deep recurrent neural networks. arXiv preprint (2013)
[5]. Gers, Felix A., Jurgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with LSTM. Neural Computation 12.10 (2000).
[6]. Cho, Kyunghyun, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv: https://arxiv.org/abs/1406.1078 (2014)
[7]. Zhang, Linhao, et al. Graph LSTM with Context-Gated Mechanism for Spoken Language Understanding. Association For the Advancement of Artificial Intelligence (2020)
[8]. Sanh, Victor, et al. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. NeurIPS Workshop 2019.
[9]. Lan, Zhenzhong, et al. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv preprint arXiv:1909.11942, 2019
[10]. Lynn, Htet Myet, et al. A Deep Bidirectional GRU Network Model for Biometric Electrocardiogram Classification Based on Recurrent Neural Networks. IEEE Access, 7, 145395-405 (2019).
[11]. Schwenk, Holger, et al. Very Deep Convolutional Networks for Text Classification. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: 1, 1107-1116, (2017).
[12]. Linzen, Tal, et al. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4, 521-35, (2016).
[13]. Zeyer, Albert, et al. A comprehensive study of deep bidirectional LSTM RNNs for acoustic modeling in speech recognition. 2017. https://dx.doi.org/10.1109/ICASSP.2017.7952599
Cite this article
Zhu,J. (2024). Comparative study of sequence-to-sequence models: From RNNs to transformers. Applied and Computational Engineering,42,67-75.
Data availability
The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.
Disclaimer/Publisher's Note
The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of EWA Publishing and/or the editor(s). EWA Publishing and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
About volume
Volume title: Proceedings of the 2023 International Conference on Machine Learning and Automation
© 2024 by the author(s). Licensee EWA Publishing, Oxford, UK. This article is an open access article distributed under the terms and
conditions of the Creative Commons Attribution (CC BY) license. Authors who
publish this series agree to the following terms:
1. Authors retain copyright and grant the series right of first publication with the work simultaneously licensed under a Creative Commons
Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this
series.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the series's published
version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial
publication in this series.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and
during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See
Open access policy for details).