1. Introduction
With the acceleration of urbanization and the continuous improvement of per capita vehicle ownership rate, traffic flow monitoring and forecasting has become a key link in today's intelligent transportation system. In the existing transportation system, traffic congestion, frequent accidents and environmental pollution are increasingly prominent problems, both developed and developing countries are facing these challenges. Therefore, to address these challenges, many cities are beginning to explore more efficient traffic management strategies and smarter transportation services. Therefore, accurate monitoring and forecasting of traffic flow is particularly important. [1] Real-time understanding of traffic conditions can not only help traffic management departments to formulate more flexible traffic control strategies, deploy traffic resources and guide traffic flow in advance, and reduce traffic congestion, but also provide scientific basis for traffic planning departments to carry out road reconstruction and traffic facility deployment.
Therefore, a machine learning-based approach has become an effective means to solve this challenge. [2-3] Machine learning technology can better deal with complex nonlinear problems, and can comprehensively consider the historical regularity of traffic flow data and the spatial correlation of road network, so as to improve the accuracy and performance of traffic flow prediction. This article will explore the potential of cloud data warehousing and machine learning in solving traffic flow monitoring and forecasting, as well as their application prospects in intelligent transportation systems. Through in-depth analysis and case studies of these technologies, we can gain a better understanding of how modern technology can be used to improve urban traffic management and services to address growing traffic challenges.
2. Related work
2.1. Traditional traffic flow monitoring
Traditional traffic flow monitoring usually adopts manual direct observation or video analysis. Due to Urban traffic flow can be analyzed from two dimensions of time and space through big data [3]. In terms of time, we can better grasp the periodic rules of traffic flow and the evolution characteristics of traffic flow under sudden situations by observing the changes of traffic flow over time from more than one year to as short as a second. In terms of space, this paper analyzes the route selection and starting and ending distribution of traffic in the flow process, and at the same time understands the quantity fluctuation of traffic flow in different sections, which enables researchers to examine urban traffic operation from a global perspective. [4] The system captures the traffic situation on the city's main roads through surveillance cameras, converts the video information into digital data, and records key information such as the driving path, speed and time of each vehicle. This data is stored in cloud data warehouses, forming a vast dataset of traffic flows. By analyzing these data, it is possible to understand the traffic conditions in different time periods and regions, identify congestion points and accident locations, and provide decision support for traffic management departments.
Figure 1. Development of traffic flow monitoring methods
Although the traditional traffic flow monitoring method adopts direct observation or video analysis, it has certain advantages. First of all, accurate traffic flow information [5] can be obtained through manual direct observation or video analysis. Secondly, due to the targeted and flexible way of manual monitoring, a specific area can be monitored according to needs and the traffic situation in the area can be effectively mastered. In addition, the efficiency of manual data processing is low, which can not meet the rapidly changing traffic flow information processing needs. Therefore, it is necessary to introduce advanced technologies such as machine learning to make up for the shortcomings of traditional traffic flow monitoring methods and achieve a more comprehensive, accurate and efficient monitoring and analysis of urban traffic flow.
2.2. Machine learning and traffic flow monitoring
Traffic flow prediction is an important part of intelligent transportation system and has been widely used in intelligent transportation subsystem. For example, Advanced Traveler Information Systems (ATIS), Advanced Traffic Management Systems (ATMS), etc. [6]. On the one hand, accurate traffic flow prediction can provide travelers with accurate road information, so as to effectively avoid congested sections and save travel time. On the other hand, the traffic management department can use the results of traffic flow prediction to conduct traffic guidance in advance to avoid too much congestion in a certain section. Therefore, in recent years, traffic flow forecasting has become one of the research hotspots in the field of traffic.
In recent years, with the improvement of computer computing power, the field of artificial intelligence and deep learning has been rapidly developed, and many applications have been produced in the field of traffic flow prediction. Common deep learning algorithms include deep residual network, recurrent neural network, convolutional neural network, etc. Example is China's traffic management system. In many Chinese cities, traffic surveillance cameras and on-board sensors are widely used on urban roads to monitor the movement of vehicles in real time. This data is aggregated to a central server, analyzed and processed to generate traffic flow heat maps and congestion warnings. [7-8] Through this information, traffic management departments can formulate traffic control measures, guide traffic flow, ease congestion and improve traffic efficiency. Therefore, the accumulation and analysis of real-time traffic monitoring data provides important support for urban traffic management, while the integration of machine learning and cloud data warehouses further improves the efficiency and accuracy of data processing and analysis.
2.3. Data warehousing and machine learning
Data warehouses play an integral role in machine learning to predict traffic flow, and their importance is reflected in several ways. First of all, the data warehouse integrates various traffic data sources, such as traffic flow detection equipment, vehicle sensors, GPS [9] systems, etc., and gathers scattered data in one place to build a complete data set. This integration greatly improves the accessibility and availability of data, making it easier for analysts and decision makers to access the data they need for analysis and decision-making. Machine learning models based on data warehouses can be trained and optimized using historical data to make accurate predictions about future traffic flows. [10-11] This forecasting model based on historical data can not only help traffic management departments better understand the changing trend of traffic flow, but also provide scientific basis for traffic planning and decision-making, and improve the operating efficiency and service quality of urban traffic system.
The role of data warehouse in machine learning to predict traffic flow is multifaceted, including data integration and storage, data management and cleaning, data analysis and modeling. By making full use of the advantages of data warehouse, we can realize the comprehensive perception and accurate prediction of traffic flow data, and provide important support for the intelligent governance of urban traffic.
2.4. Relevant matrix
The relation matrix is crucial to the convolution of graphs because it determines how node features are clustered and updated. In order to reduce the computational cost and improve the efficiency, we chose to include only the first 20 rows and 20 columns of data, which represent the first 20 most important features in the station. Through the comprehensive use of similarity matrix and adaptive matrix, we can grasp the spatial dependence and dynamic change of traffic flow data more comprehensively, and provide more effective support for traffic management and optimization. The definition is as follows:
\( A_{ij}^{0}=exp(-\frac{\\x_{i}^{5}-x_{j}^{5}{\\^{2}}}{{ε^{2}}} \) )(1)
Adaptive matrix: Two learnable node embedding matrices E1 and E2 are randomly initialized, and the spatial dependence between the two embedding layers is calculated by multiplying E1 and E2, and the adaptive matrix is obtained:\( \bar{{A_{ada}}=ReLU({E_{1}}E_{2}^{T})} \) (2)
\( \bar{{A_{ada}}=norm({A_{ada}})} \) (3)
Normalization of the adaptive matrix:
\( D=\sum _{j}{A_{ada}} \) (4)
\( {D_{1}}=\frac{1}{D} \) (5)
\( \widetilde{A}={D_{1}}\bar{{A_{ada}}} \) (6)
The definition of traffic flow prediction involves learning from historical traffic graph signal data with P time steps to develop a mapping function F. This function enables the prediction of future traffic graph signal data with Q time steps. In essence, the process entails analyzing the historical traffic data to understand patterns and trends, thereby enabling the accurate prediction of future traffic flow dynamics. This predictive capability is crucial for informing traffic management strategies and optimizing resource allocation in transportation systems:
\( {X_{t-p+1:t}}∈{R^{p×N×d}} \) , \( {X_{t+1:t+Q}}∈{R^{Q×N×d}} \) (7)
References
[1]. May, Adolf Darlington. Traffic flow fundamentals. 1990.
[2]. Treiber, Martin, and Arne Kesting. "Traffic flow dynamics." Traffic Flow Dynamics: Data, Models and Simulation, Springer-Verlag Berlin Heidelberg (2013): 983-1000.
[3]. Tian B, Yao Q, Gu Y, et al. Video processing techniques for traffic flow monitoring: A survey[C]//2011 14th international IEEE conference on intelligent transportation systems (ITSC). IEEE, 2011: 1103-1108.
[4]. Williams, Nigel, Sebastian Zander, and Grenville Armitage. "A preliminary performance comparison of five machine learning algorithms for practical IP traffic flow classification." ACM SIGCOMM Computer Communication Review 36.5 (2006): 5-16.
[5]. Gui, G., Zhou, Z., Wang, J., Liu, F., & Sun, J. (2020). Machine learning aided air traffic flow analysis based on aviation big data. IEEE Transactions on Vehicular Technology, 69(5), 4817-4826.
[6]. Phan, Trung V., et al. "DeepGuard: Efficient anomaly detection in SDN with fine-grained traffic flow monitoring." IEEE Transactions on Network and Service Management 17.3 (2020): 1349-1362.
[7]. Lv, Yisheng, et al. "Traffic flow prediction with big data: A deep learning approach." Ieee transactions on intelligent transportation systems 16.2 (2014): 865-873.
[8]. Kumar, K., Parida, M., & Katiyar, V. K. (2015). Short term traffic flow prediction in heterogeneous condition using artificial neural network. Transport, 30(4), 397-405.
[9]. Wu, Y., Tan, H., Qin, L., Ran, B., & Jiang, Z. (2018). A hybrid deep learning based traffic flow prediction method and its understanding. Transportation Research Part C: Emerging Technologies, 90, 166-180.
[10]. Mitchell, Tom M. "Machine learning and data mining." Communications of the ACM 42.11 (1999): 30-36.
[11]. Kumar, Arun, Matthias Boehm, and Jun Yang. "Data management in machine learning: Challenges, techniques, and systems." Proceedings of the 2017 ACM International Conference on Management of Data. 2017.
[12]. Vartak, M., Subramanyam, H., Lee, W. E., Viswanathan, S., Husnoo, S., Madden, S., & Zaharia, M. (2016, June). ModelDB: a system for machine learning model management. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics (pp. 1-3).
[13]. Polson, Nicholas G., and Vadim O. Sokolov. "Deep learning for short-term traffic flow prediction." Transportation Research Part C: Emerging Technologies 79 (2017): 1-17.
Cite this article
Yang,P.;Chen,Z.;Su,G.;Lei,H.;Wang,B. (2024). Enhancing traffic flow monitoring with machine learning integration on cloud data warehousing. Applied and Computational Engineering,77,238-244.
Data availability
The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.
Disclaimer/Publisher's Note
The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of EWA Publishing and/or the editor(s). EWA Publishing and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
About volume
Volume title: Proceedings of the 2nd International Conference on Software Engineering and Machine Learning
© 2024 by the author(s). Licensee EWA Publishing, Oxford, UK. This article is an open access article distributed under the terms and
conditions of the Creative Commons Attribution (CC BY) license. Authors who
publish this series agree to the following terms:
1. Authors retain copyright and grant the series right of first publication with the work simultaneously licensed under a Creative Commons
Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this
series.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the series's published
version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial
publication in this series.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and
during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See
Open access policy for details).
References
[1]. May, Adolf Darlington. Traffic flow fundamentals. 1990.
[2]. Treiber, Martin, and Arne Kesting. "Traffic flow dynamics." Traffic Flow Dynamics: Data, Models and Simulation, Springer-Verlag Berlin Heidelberg (2013): 983-1000.
[3]. Tian B, Yao Q, Gu Y, et al. Video processing techniques for traffic flow monitoring: A survey[C]//2011 14th international IEEE conference on intelligent transportation systems (ITSC). IEEE, 2011: 1103-1108.
[4]. Williams, Nigel, Sebastian Zander, and Grenville Armitage. "A preliminary performance comparison of five machine learning algorithms for practical IP traffic flow classification." ACM SIGCOMM Computer Communication Review 36.5 (2006): 5-16.
[5]. Gui, G., Zhou, Z., Wang, J., Liu, F., & Sun, J. (2020). Machine learning aided air traffic flow analysis based on aviation big data. IEEE Transactions on Vehicular Technology, 69(5), 4817-4826.
[6]. Phan, Trung V., et al. "DeepGuard: Efficient anomaly detection in SDN with fine-grained traffic flow monitoring." IEEE Transactions on Network and Service Management 17.3 (2020): 1349-1362.
[7]. Lv, Yisheng, et al. "Traffic flow prediction with big data: A deep learning approach." Ieee transactions on intelligent transportation systems 16.2 (2014): 865-873.
[8]. Kumar, K., Parida, M., & Katiyar, V. K. (2015). Short term traffic flow prediction in heterogeneous condition using artificial neural network. Transport, 30(4), 397-405.
[9]. Wu, Y., Tan, H., Qin, L., Ran, B., & Jiang, Z. (2018). A hybrid deep learning based traffic flow prediction method and its understanding. Transportation Research Part C: Emerging Technologies, 90, 166-180.
[10]. Mitchell, Tom M. "Machine learning and data mining." Communications of the ACM 42.11 (1999): 30-36.
[11]. Kumar, Arun, Matthias Boehm, and Jun Yang. "Data management in machine learning: Challenges, techniques, and systems." Proceedings of the 2017 ACM International Conference on Management of Data. 2017.
[12]. Vartak, M., Subramanyam, H., Lee, W. E., Viswanathan, S., Husnoo, S., Madden, S., & Zaharia, M. (2016, June). ModelDB: a system for machine learning model management. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics (pp. 1-3).
[13]. Polson, Nicholas G., and Vadim O. Sokolov. "Deep learning for short-term traffic flow prediction." Transportation Research Part C: Emerging Technologies 79 (2017): 1-17.