Download pdf
Comparative analysis and prospect of RNN and Transformer
- 1 Tianjin Chengjian University
* Author to whom correspondence should be addressed.
Abstract
Deep learning techniques have emerged as a key catalyst for innovation in the rapid advancement of artificial intelligence and machine learning. Deep learning has significantly transformed the field of natural language processing (NLP) and voice processing. It has revolutionized the processing and comprehension of linguistic data, leading to advancements in various applications, ranging from simple text categorization to intricate speech recognition. The progress made in this area would not have been feasible without the utilization of two crucial neural network models: the Recurrent Neural Network (RNN) and the Transformer. These models, with their distinctive processing skills, have attained significant accomplishments in the domains of NLP, computer vision, and several other sectors. Nevertheless, although they demonstrate exceptional performance in their specific application scenarios, there are notable distinctions in processing techniques, performance enhancement, and application range. This study aims to thoroughly investigate and compare the efficacy of two models, namely RNN and Transformer, in processing natural language and speech data. This will be achieved using the research methods of literature analysis and literature review. Through the analysis of their structures, strengths, flaws, and performance in practical applications, it offers a more comprehensive viewpoint and benchmark for future study and applications. As technology advances, we anticipate the emergence of more novel models and methodologies in the field of natural language processing and voice processing. This will further facilitate the development of these technologies.
Keywords
Recurrent Neural Networks, Transformer, Deep Learning, Performance Comparison
[1]. Li Huaxu. A review of research on natural language processing based on RNN and Transformer models [J]. Information Recorded Materials,2021,22(12):7-10.DOI:10.16009/j.cnki.cn13-1295/tq.2021.12.081.
[2]. Liu JW,Song ZY. A review of recurrent neural network research [J]. Control and Decision Making, 2022, 37(11): 2753-2768. DOI:10.13195/j.kzyjc.2021.1241.
[3]. Junyi Fan, Jibin Yang, Xiongwei Zhang et al. A review of single-channel speech enhancement models based on Transformer [J]. Computer Engineering and Application,2022,58(12):25-36.
[4]. WANG Xin,WU Ji,LIU Chao et al. Fault time series prediction based on LSTM recurrent neural network [J]. Journal of Beijing University of Aeronautics and Astronautics, 2018, 44 (04): 772-784. DOI:10.13700/j.bh.1001-5965.2017.0285.
[5]. Zheng YH,Zhou J,Huang KD et al. Research and design of machine translation based on pre-training model [J]. Technology and Innovation,2023(21):34-37.DOI:10.15913/j.cnki.kjycx.2023.21.010.
[6]. Y.L. Tian,Y.T. Wang,J.K. Wang et al. Key Issues in Vision Transformer Research:Status and Prospects[J]. Journal of Automation, 2022, 48 (04): 957-979.DOI:10.16383/j.aas.c220027.
[7]. W.T. Liu,X.M. Lu. Research progress of Transformer based on computer vision [J]. Computer Engineering and Application, 2022, 58 (06): 1-16.
Cite this article
Li,X. (2024). Comparative analysis and prospect of RNN and Transformer. Applied and Computational Engineering,75,178-184.
Data availability
The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.
Disclaimer/Publisher's Note
The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of EWA Publishing and/or the editor(s). EWA Publishing and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
About volume
Volume title: Proceedings of the 2nd International Conference on Software Engineering and Machine Learning
© 2024 by the author(s). Licensee EWA Publishing, Oxford, UK. This article is an open access article distributed under the terms and
conditions of the Creative Commons Attribution (CC BY) license. Authors who
publish this series agree to the following terms:
1. Authors retain copyright and grant the series right of first publication with the work simultaneously licensed under a Creative Commons
Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this
series.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the series's published
version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial
publication in this series.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and
during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See
Open access policy for details).