1. Introduction

Predicting the price of stock market is a key point to provide investors with the basis for an optimal decision in trading but also a challenging task to achieve. Due to its chaotic and high volatility character, the movement is considered to be a stochastic process influenced by many uncertainties from reality, such as macroeconomic factors, the market anticipation and confidence.

There have been many previous studies involving the prediction of stock market, mostly divided into fundamental and technical analysis. Fundamental analysis estimates intrinsic value by analyzing internal and external variables, meanwhile technical analysis finds a pattern in the stock price based on which we can make predictions.

Before the appearance of machine learning methods, early researchers use statistical methods such as time series and multivariable statistics, as well as different kinds of econometric methods, to build the model and make the prediction. However, because of the characteristics of being chaotic, high volatility and nonparametric, the better methods of machine learning are applied. In this situation, it is still difficult to predict the exact values to predict the stock prices, so it is better to treat it as a classification problem rather than a prediction problem. Many machine learning methods are applied based on these conditions, such as support vector machine (SVM), logistic regression, and random forest.

Deep neural networks (DNNs) is an important kind of machine learning due to its unique advantages compared to those conventional methods: The highly complex nonlienearity of DNN can approximately simulate and describe the complexity of the influencing factors, as deep nonlinear topologies can be built to represent complex high-dimensional functions by stacking the hidden layers of nonlinear activation functions. Besides, they are numeric, datadriven and adaptive to be able to analyze inaccurate and noisy data and have been extensively used to predict time series.

The above advantages ensure a good performance of DNN for prediction problems, therefore it is feasible to predict financial problems with deep learning. In this paper, we mainly show its effect of the prediction and how it appears its advantages. (If possible, we may enforce the capability of decision-making by combining reinforcement learning.) In comparison, we also use some traditional methods in machine learning, such as SVM and logistic regression, and show the differences in the results.

The main part of this paper is organized as follows. In part 2 we introduce some related works of this topic in the previous time. In part 3 we discuss the reinforcement learning, including its characters and advantages. The comparison between it and other traditional machine learning methods will also be mentioned to show its improvement. Finally we make the conclusions about the problem and talk about the future development.

2. Literature review

In the early ages, it was widely believed that predicting the future trends in stock market prices contradicted a basic rule known as the Efficient Market Hypothesis (Fama and Malkiel) [1]. However, later more researchers chose to reject this controversial and disputed theory by using more advanced algorithms to model more complex dynamics of the financial system. In this field, more representative work has been studies by Lo, by Mackinlay [2]. Some example studies on the predictability of returns of a long-term investment are those by DeBondt and Thaler [3-5].

With a brilliant development of computer, the machine learning methods based on statistics have contributed a lot to the prediction of stock price. The machine learning methods have a better performance in handling complex nonlinear analysis, which enables various approaches compared with other studies using traditional statistics only. Those representative methods include logistic regression, genetic algorithm, fuzzy theory, SVM, decision tree, and adaptive boosting (AdaBoost) [6-9]. For example, L Khaidem et al. successfully used ensemble learning to improve the performance of random forest for stock prediction. As for logistic regression, J Gong and S Sun introduced innovative feature index variables into the prediction model and proposed a special optimization process to select optimizing regression parameters [10-11]. In 2012, A Upadhyay et al. used seven independent financial ratiosto construct a Multinomial Logistic Regression method [12]. The method of SVM is also widely considered. J Heo and JY Yang evaluated the stock price predictability of it and the time to keep a good performance to support the efficient market hypothesis; meanwhile F Wen et al. proposed the singular spectrum analysis to improve the performance of SVM [13-14].

In 1980S, NNs had been applied by IBM to predict the changes of daily stock prices as well as its returns. An autoregressive integrated moving average (ARIMA) model was added to an artificial neural network(ANN) for predicting the results of time series in stock markets. Later in recent years, deep learning have experienced a fantastic development, as well as its wide applications in prediction prices of stock market. According to different methods and usages of deep learning, for different problems we have certain choices, such as convolutional neural network (CNN), Long Short-Term Memory (LSTM) and Recurrent Neural network (RNN), but here we mainly focus on the most frequently used deep learning methods, namely the deep neural network (DNN) and Reinforcement learning (RL).

DNN can be used to predict stock trends because of its fantastic performance for prediction problems with numerous data and nonlinear mapping relations. Due to this reason, much work have been done recently. Shen et al constructed a deep belief network using a continuous restricted Boltzmann machine and used it to predict exchange rates. Through a comparison, they found that the DL network was better than the conventional feedforward NNs [15]. Song, Y. proposed a plunge filtering technique for a DNN model to improve the accuracy of it. This proposed model had great profitability [16]. Naik, N. proposed a DNN using the Boruta feature selection technique to solve the problem of selecting indicator feature and identificating the relevant indicators , which performed much better than the ANN and SVM models [17]. Chatzis, S.P. proposed a DNN model with Boosted approaches to predict stock market crisis episodes [18]. His research showed that it was meaningful for the stock market crisis to predict the price. Nakagawa, K. proposed a deep factor model and a shallow model with DNN, and the former performed better than the linear model, which implied the nonlinear relationship between the stock returns and the factors in the financial market [19]. The deep factor model also had a better performance than other machine learning methods including SVR and random forest. Chong, E. examined the effects of DNN with three feature extraction methods including, PCA, auto-encoder, and the restricted Boltzmann machine, to predict future market behavior [20]. It was showed that additional information could be extracted from the residuals of the auto-regressive model to improve the final performance in DNN.

RL learns the local optimal timing trading action from the response changes of the stock market by viewing the neighboring information of the transaction, which could be regarded as the environment of reinforcement learning, and there have been much work in this field. Shin, H.-G. proposed a RL model combined with LSTM and CNN, which generated various charts from stock trading data and used them as input layers [21]. The features extracted from the CNN were used to construct the LSTM layer. The RL defined the agents’ policy neural network structure, reward, and provided the final output. Jia, W. proposed a RL model with an LSTM-based agent to sense the dynamics of the stock market and decrease the difficulty of designing indicators from massive data [22]. Carapuço, J. proposed a RL-Q network model, in which three hidden layers of ReLU neurons were trained as RL agents through the Q-learning algorithm [23]. The framework could consistently induce stable learning that generalized to out-of-sample data. Kang, Q. proposed to apply the Asynchronous Advantage Actor-Critic algorithm (A3C algorithm) to solve the portfolio management problem, and later designed a deep RL model [24]. H Yang proposed a novel ensemble strategy combining three deep reinforcement learning algorithms to find the optimal trading strategy in a complex and dynamic stock market. The strategy can adjust to maximize return subject to risk constraint [25]. There are also methods that combine DNN and RL together, namely Deep Reinforcement Learning (DRL), which combines the perception ability of DNN with the decision-making ability of RL. For example, Y Li proposed to introduce DRL into the application of finance and proved its advantages of improving prediction ability [26].

3. Discussion

3.1. characters and advantages

The basic view of traditional research methods could be mainly divided by the viewpoint of time series and factor analysis. The former consider the evolution and changing trend of a single stock as time goes by, and in this way make the prediction of the future trend of this single stock. In this process the main characters in consider are those significantly influenced by time. The latter, on the other hand, mainly explore the deep influence factors that decides the value of stocks. This target is achieved by comparing the value of different stocks as well as their main influence factors given a fixed time.

By comparing different traditional views above, it is concluded that the researching methods from the view of time series may be short of mining the hidden information from the related characters; meanwhile from the view of factor analysis,it is not convenient to precisely control the develop trend following the time. Based on this situation, the DRL could combine the advantages of these two methods and avoid their disadvantages at the same time.

3.2. deep and reinforcement learning

Deep learning could uncover complex, intrinsic, and deep-seated data information. When dealing with problems about stock prediction, the complex characters and factors concerned with the stock price could be well described by DNN due to its complex nonlinearity. Because of the goodness and advantages above, the DNN have a quite fantastic accuracy for complex prediction problems in many fields in practice, such as image classification and natural language process. The research results above also functions well for data analysis concerning time series and prediction with a deep learning algorithm. Overall, deep learning models perform excellent in broad and diverse research fields, which makes it feasible to predict the future trend of stock markets with deep learning.

However, although DNN have many advantages as stated, there also exists limitations and we cannot blindly increase depth. Firstly, increasing the depth will increase the complexity of the model, and the required number of training samples will also increase accordingly. Secondly, the single-layer feedforward operation complexity of the prediction model is relatively high, and increasing the number of layers will lead to an overall high complexity. Thirdly, gradient explosion and gradient disappearance may occur. By combining it with another machine learning method, the DNN method could have some developments.

Reinforcement learning could be applied to learn an optimal policy by making interactions with the environment and repeated experiments. The process above is widely applied for sequential decision making problems. For such models, the key conception is the value funtion which decides the learning targets, the control algorithms are applied to find optimal policies, and the final target is to learn behavior strategies in multiple stages. Reinforcement learning enhances the model's ability to learn from data and improves feature dimensions. The method have the advantages to minimum the transaction cost.

Combining the two methods above, it is obvious that each method has its goodness and shortages. In this case, we use DNN to simulate components of reinforcement learning, including value function, policy, and the models. Although reinforcement learning can process problems of making decisions, it is short in expressing perception, which prompts the combination of reinforcemet learning and DNN.

DRL integrates the perception of DNN with the decision-making ability of reinforcement learning to achieve its goals of simulating the cognition and learning mode of human being. In this method, different sorts of information can be input, and then actions will be directly output through the simulation of DNN, which is controlled directly according to the input data without other outside supervision.

4. Conclusion and future development

Generally speaking, models usually have their shortages in real applications. The DRL also has its shortages and limitations. One key point is that the current DRL mainly use the historical data of one stock and process it as a single mode, which means that the model has a significant shortage of been in lack of the exploration of other dimensions which may potentially influence the final data. In this case, it is a highly considerable view to make the model been more compatible of the data structure with multimodal and promote the performance in general situations. The improvement of performance in generalization is important for the model, because this is closed related to the universality of it.

Apart from that, the current DRL model still can not handle the problems about portfolio strategy. This is because that the DRL model mainly focus on the problems about prediction. But in reality, on the other hand, it is more common to pursue the portfolio strategy according to the predicting result, which may have different results between by dividing it into two problems and by regarding the whole problem as an integral. How to connect the portfolio strategy to the current model and regard the problem as a whole may be a meaningful topic for prediction models, including the DRL model.

Finally, attention becomes popular these years, and it is especially suitable for stock prediction problems. As the prices of the stock have different influence on the future at different points of time in the past, it is really reasonable to add the attention mechanism to the current prediction model in the field of stock prediction.