1. Introduction

Many domains, including language, time series, etc., frequently use sequence data. In natural language processing (NLP), understanding and generating sentences require models that can deal with sequences. Although early approaches like n-gram models and Hidden Markov Models were useful, they had drawbacks including the requirement for large amounts of data and their inability to adequately incorporate long-term dependencies [1].

With the rise of deep learning, new models were created to solve these problems. In 1986, inspired by the coarticulation in speech research, Jordan created one of the first deep learning models designed for sequence data, the Recurrent Neural Network (RNN) [2]. Later, the Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) were developed to solve vanishing gradients problem in RNN [3,4]. These models became popular for tasks such as speech recognition and language modeling. However, these models still had limits in dealing with very long sequences. better the development of the Attention mechanism and then the Transformer model, which allowed much better performance and faster training [5,6]. Ultimately, pre-trained models like BERT, which is based on the bidirectional Transformer model, emerged as a new NLP standard [7].

The goal of this paper is to review the development of six important models: RNN, LSTM, GRU, Attention mechanism, transformer, and BERT. We will first introduce the architecture of each network, and then describe their strengths and weaknesses, and finally show how they connect to each other in the history of sequence modeling.

2. Development

2.1. Recurrent Neural Networks (RNN)

Recurrent Neural Networks (RNN) are proposed by Michael I. Jordan in 1986. They are a class of artificial neural networks that are used to analyze sequential input. These networks are inspired by sequential behaviors of humans, where humans’ speech system predicts the future actions and changes the shape of lip before pronouncing, which allows people to speak fast [2]. The RNN introduces recurrence (i.e., one or more cycles) to the neural network, which enables the networks’ hidden state to retain contextual information from earlier time steps.

In Elman network (1990), the hidden state at time step t, h_t, is

$$$ \begin{array}{c}{h}_t=\sigma \left(W_{xh}{x}_t+W_{hh}{h}_{t-1}+b_h\right),\end{array}$$$(1)

where σ is a non-linear activation function, usually tanh; W_hh and W_xh are weight matrices for the recurrent and input neuron, respectively; x_t is the input vector; and b_h is the bias for the hidden state. Moreover, the output at time step  $$ t$$ ,  $$ y_t$$ , is

$$$ \begin{array}{c}{y}_t=W_{hy}{h}_t+b_y,\end{array}$$$(2)

where $$ W_{hy} $$ is the output weight matrix, and $$ b_y $$ is the bias for the output [8].

There are several uses for RNNs, such as character-level and word-level language modeling [9], speech recognition [10], etc. In the language modeling, RNNs successfully generate predictive texts and probabilistic modeling of sequences; and in speech recognition, RNNs can effectively model the temporal patterns in audio signals. However, the RNNs encounter problems, such as they can only use the previous context [10], and the gradient vanishes or explodes when processing long sequences, because repeated multiplication of weight matrices causes gradients to decay or grow exponentially [11]. Hence, later networks like Long Short-Term Memory (LSTM), are created to address these limitations of RNNs.

2.2. Long Short-Term Memory (LSTM)

Hochreiter and Schmidhuber proposed the LSTM model in 1997. It was aimed at fixing the exponentially decaying gradient error in RNNs. LSTM achieves this by introducing memory cells and gated units (input, output, and forget gates). The input gate limits the data being added to the memory cells, the forget gate removes unimportant sequences from the memory cells, and the output gate decides the outcome from memory cells. To prevent conflicts between input and output weights, each cell is constructed around a central linear unit with a constant error carousel (CEC) [3].

These structures make LSTM possible to learn tasks that contain extensive time intervals, while the RNNs are hard to handle. LSTM’s ability to capture long-range dependencies led to a variety of applications, including speech recognition, time-series predicting, language modeling, etc. Graves et al in 2013 compared the ability of RNN and LSTM in speech recognition and reached a result that LSTM performs much better than the traditional  $$ \mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}$$  model [10], Sundermeyer et al applied LSTM in two language modeling tasks [12], and Ma et al predicted traffic speed in Beijing using LSTM [13].

Due to its advantages, it has been one of the most popular networks after it was proposed. However, LSTM has its own limitations – compared to RNN, LSTM’s calculations are much more complex, making it expensive to use. This was not improved until 2014, when further models are put forward.

2.3. Gated Recurrent Unit (GRU)

Gated Recurrent Unit (GRU) was a streamlined variant of LSTM proposed by Cho et al. in 2014, initially called RNN Encoder-Decoder. It creates an update gate by combining input and forget gates:

$$$ \begin{array}{c}z=\sigma \left(W_z·\left[h_{t-1}, x\right]\right),\end{array}$$$(3)

and creates a reset gate by combining the cell state and hidden state:

$$$ \begin{array}{c}r=\sigma \left(W_r·\left[h_{t-1}, x\right]\right),\end{array}$$$(4)

where  $$ \sigma $$  is the logistic sigmoid function,  $$ W_r, W_z$$  are the weight matrices,  $$ h_{t-1}$$  is the hidden state at last time step, and  $$ x$$  is the input. This leads to fewer parameters, making GRUs quicker to train, and more computationally efficient, particularly in environments with limited resources [4].

Preliminary experiments show that GRU and LSTM have similar performance in the tasks of sequence modeling, while both outperform the traditional tanh unit RNNs [14]. Empirical evaluations show that while LSTMs may outperform GRUs on very long sequences or highly complex strings such as language modeling, GRUs perform better than LSTMs for low-complexity dependencies [15,16].

The compact design of GRUs makes them more reliable when training data is limited, since it reduces the risk of overfitting. Hence, GRUs have been widely adopted in the tasks that both performance and efficiency are critical, including time series forecasting [17] and machine translation, which also inspired the invention of attention mechanism [5].

2.4. Attention mechanism

Presented by Bahdanau et al. in 2015, a novel architecture called the attention mechanism was to enhance the efficiency of traditional encoder–decoder networks such as GRU, which degenerates quickly when the length of sentences increases. This mechanism uses the bidirectional RNN as the encoder, and simulates the searching process when decoding [5]. The encoder maps the input to a sequence of hidden states  $$ \left(h_1,\dots ,h_{T_x}\right)$$ , called annotations, and then the decoder computes a weighted sum of the annotations as the context vector  $$ c_t$$ :

$$$ \begin{array}{c}{c}_t={\sum }_{i=1}^{T_x}{\alpha }_{t,i}{h}_i, \end{array}$$$(5)

where  $$ {\alpha }_{t,i}$$ ​ are the attention weight, computed via a score function based on the hidden state  $$ s_{i-1}$$  of RNN and the j-th annotation  $$ h_j$$ :

$$$ \begin{array}{c}{\alpha }_{t,i}=\frac{\exp \left(\mathrm{score}\left(s_{t-1},h_i\right)\right)}{{\sum }_{j=1}^{T_x}\exp \left(\mathrm{score}\left(s_{t-1},h_j\right)\right)}.\end{array}$$$(6)

By enabling selective access to different input positions, attention mechanisms significantly improve sequence to sequence (Seq2Seq) performance, especially for long-range dependencies. Today, attention mechanisms are widely applied not only in machine translation but also in image captioning and time series forecasting [17,18]. The attention mechanism’s innovation also laid the foundation for the Transformer architecture, where self-attention mechanisms completely replace the recurrence architecture and enable highly parallelizable computation [6].

2.5. Transformers

Vaswani et al. presented the Transformer architecture in their article “Attention is All You Need.” in 2017. The novel architecture completely removes the recurrent structures in previous networks, and solely relying on self-attention mechanisms. It enables the Transformer to deal with highly parallelizable computations and can capture global dependencies more effectively. The model introduces three matrices: query matrix  $$ Q$$ , key matrix  $$ K$$ , and value matrix  $$ V$$ , which compute the matrix of output as

$$$ \begin{array}{c}Attention\left(Q,K,V\right)=softmax\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V.\end{array}$$$(7)

The self-attention architecture avoids the leftward information flow that previous recurrent mechanisms used. The decoder of the Transformers pay attention to all positions in a sentence, and thus can capture richer contextual information at different semantic levels [6].

The Transformer’s ability to process data in parallel and capture global context makes it much more efficient and powerful than older sequence models. In jobs like machine translation, English constituency parsing, and text generation, it has demonstrated remarkable performance, since it can notice the connections between words in one sentence such as the pronouns’ reference [6]. More importantly, it became the foundation for later pre-trained models like BERT [7].

2.6. Bidirectional Encoder Representations from Transformers (BERT)

BERT, proposed by Devlin et al. in 2019, is one of the most influential models in NLP. While the earlier Transformers included both an encoder and a decoder, the BERT is built on a multi-layer bidirectional Transformer that only has an encoder. The bidirectional refers to that when dealing with sentences, the model will read the context both from left to right and from right to left simultaneously, while the basic models can only process the information in one direction.

The success of BERT introduced the pre-training and fine-tuning paradigm. BERT pre-training uses big text corpus like Wikipedia and BooksCorpus, and requires two tasks: the Masked Language Model (MLM) and the Next Sentence Prediction (NSP). The MLM task randomly hides several words in the input sentence, and the BERT must anticipate these components according to the context. NSP gives BERT two sentences, and the model must decide whether the second sentence really follows the previous one, which improves BERT’s understanding of sentence relationships. In fine-tuning, the BERT is adapted to specific tasks like sentiment analysis and question answering.

These approaches led BERT to achieve excellent results on many benchmarks, such as GLUE and SQuAD [7]. It also inspired many later models, such as RoBERTa and ALBERT, which improved efficiency and performance [19,20]. Overall, BERT shows the effectiveness of large-scale pre-trained language models.

3. Conclusion

This paper reviews the development of sequence learning models, in their development order from RNN to LSTM and GRU, and later to the Attention mechanism, Transformer, and BERT. The later models solved important problems of the previous ones. RNNs introduced recurrent structures for sequence data, LSTMs and GRUs improved the learning of long-term dependencies, while Attention mechanisms proposed important architectures that enable the later Transformer model to replace recursion with attention, and BERT brought pre-training methods that changed NLP research.

However, the BERT model still has limits, as it requires huge resources to be pre-trained. In the future, we expect more efficient architectures, lighter models for deployment, and a stronger focus on ethics and fairness. These trends will guide the next stages of sequence modeling research.