1. Introduction

The Markov chain is known for interpreting a sequence of possible events where the probability of each stage is solely dependent on the current state. This function has made it convenient for estimating the results of certain probabilistic events. Some applications involve investigating treatment programs and health care protocols for chronic diseases, analyzing package delivery schedules, or visualizing the spread and progression of infectious diseases using the Markov chain model [1,2]. These investigations can be carried out by using the Markov chain and transition matrix models. This paper will conduct a literature review on how Markov chain can impact a range of domains, and how the Markov chain model can be necessary in predicting certain outcomes. By carrying out this investigation, this study will create a model used for examination, affirm that Markov chain would benefit research in different approaches, analyze the limitations, and most importantly, discuss the differences between the Markov chain model under different scenarios.

2. Related concepts

The study of stochastic processes is without a doubt the subject of interest for mathematicians and researchers working in all different sectors trying to analyze possible outcomes of an event. Among all the stochastic models used for attack graph analysis, the Markov chain model stands out. This is generally because of its wide applicability. Markov chain can model a variety of stochastic events, making it relevant for a wide range of applications in computer science, artificial intelligence, finance, natural language processing and so on. Moreover, Markov chain is also known for its simplicity and scalability, as it is relatively easy to understand and implement or modify, primarily because of its memoryless property. The Markov chain makes decisions that only depend on the current state and does not refer to the entire history of past states, which will greatly reduce the complexity of the model. This allows Markov chain models to be more manageable compared to other models, which may require the full history of state transition, and it can be applied to both small and large-scale problems. An important question is whether the investigation would have similar procedures for other applications. To begin with, each separate stage of the Markov chain model should be identified and given its probability of occurring.

3. Analysis of the application

Markov chains come in different forms, such as discrete-time and continuous-time Markov chains, first-order and higher-order Markov chains and Hidden Markov Models. Discrete-time Markov chains model systems change states at discrete time intervals, while continuous-time Markov chain model systems change states continuously over time. First-order Markov chain assume that the probability of transitioning from one state to another depends only on the current state, while higher-order Markov chains consider more than one previous state in determining the probability of transitioning. Hidden Markov Models, on the other hand, model systems where the states are not directly observable but are inferred from observed outputs.

Markov chain models have many applications in everyday life, from speech recognition on our smartphones to recommendation systems on e-commerce platforms. For example, speech recognition algorithms use Hidden Markov Models to predict spoken words based on observed sound signals. Recommendation systems use Markov chain to predict the next item a user is likely to purchase based on their previous purchases.

Markov chain models can also be used in public health, such as to model the spread of infectious diseases and predict the effectiveness of vaccination programs. In transportation, Markov chain models can be used to model traffic flow and optimize transportation networks.To visualize the Markov model in action, a model will be used to investigate how the Markov chain would provide the estimated length of the political career through calculation.

To begin with, each separate stage of the Markov chain model should be identified and given a probability of occuring. This investigation is about estimating the approximate number of years it will take for the politician “X” to retire after running for the US Congress. If politician “X” was never elected before, the chance of him being elected will be ½, he will only be allowed to run for the ballot again in 2 years if he loses the ballot. If “X” is already in office, his chance of getting reelected will be 9/10. However, if he loses the ballot, he will retire from politics. With this information, the three stages of the Markov chain model would be “in office”, “not in office” and “Retired”. Knowing that the model only has 3 states, a 3x3 transition matrix would be created to display all possible results. Another investigation into the presence and absence of squirrels used a similar method. In this investigation, 4 states exist to create a distribution map of squirrels: R = Only red squirrels were recorded this year G = Only grey squirrels were recorded this year B = Both species were recorded this year O = Neither species were recorded this year. With the 4 existing states, a 4x4 transition matrix can be created listing the frequencies of the above states. The idea of listing out frequency numbers can be applied to making the transition matrix in my investigation, and will correspond to the matrix entry. Another common example is the random walk, where a particle moves randomly in a one-dimensional space [3]. As mentioned in the article, the probability of the particle moving left or right at each step depends only on its current position and not on any previous positions.

The next step of the investigation is looking at how the Markov chain model would help the user acquire a result. To get the expected result of how many years is politician “X” expected to retire, there are two options available. The first is getting the average number of years from several randomized results generated by an algorithm. To get a random result from the listed states, the model uses the random.choice() method in Python. The random choice function will assign a “weight” to each variable in an array, which is equivalent to the probability of picking the variable. There will be 2 arrays that correspond to the two situations, each containing 2 variables. The first array contains the possible stages when politician “X” is currently out of office but not retired: in office (wins ballot) and out of office (loses the ballot), the weight is 50 and 50 (equivalent to 50%). When “In office” is selected through the random function, an IF statement will trigger stage 2. Stage 2 includes: In office and retired, the weights are 90 and 10. Stage 2 will be run continuously through a WHILE loop until “retired” is selected as the result of stage 2. Each time the second stage repeats, a counter will add 2 to the total number of years. This algorithm may also be used to obtain the frequency used to create the transition matrix. Using the random.choice method is feasible, however, there is an alternative. A common way to represent Markov chains in Python is through NumPy. NumPy defines each separate state and the probability for it to occur after another state. The possible downside to displaying a Markov chain through NumPy would be that compared to using random.choice, it is more complicated to modify conditions for repeating the loop.

While Markov chain models have many advantages, they are not suitable for all applications. Alternative models, such as neural networks, decision trees, and regression models, can provide better accuracy and performance in some cases. For example, neural networks can capture complex patterns and dependencies in data that Markov chains may not capture, while decision trees can handle categorical and continuous data simultaneously. Another limitation of Markov chains is their assumption of memory-lessness, which means that the probability of transitioning to a future state depends only on the current state and not on any previous states. Markov chains are sensitive to the initial conditions of the system, which means that small changes in the initial state can lead to large changes in the predicted outcomes [3].

However, Markov chains are still relevant and widely used due to their simplicity, scalability, and analytical tractability. One advantage of Markov chains is their ability to model complex systems using relatively simple mathematical models. This is because Markov chains only require knowledge of the current state of the system to predict future states, rather than requiring detailed information about the system's history [4]. This makes Markov chains particularly useful in situations where there is a large amount of uncertainty or randomness involved, such as in weather forecasting or stock market prediction [5]. In many cases, Markov chains provide adequate results with minimal computational resources and time, making them an attractive option for real-time applications. By using Markov chains to model these complex systems, researchers and analysts can gain insights into how they operate and make more accurate predictions about future outcomes [6].

4. Conclusion

In conclusion, Markov chains are powerful mathematical models that have numerous applications across a wide range of fields. The main advantage of Markov chains is their ability to model complex systems using relatively simple mathematical models. This makes them particularly useful in situations where there is a large amount of uncertainty or randomness involved. However, Markov chains also have some limitations, such as their assumption of memorylessness and their sensitivity to the initial conditions of the system. Examples such as attack graph technology can directly reflect the security situation of the network. Therefore, the attack graph modeling of the target network topology is of great significance for security researchers to analyze the possible attack paths of attackers and formulate defense measures in time. Moreover, the modelling of Markov chains will also include Python programming. There are many libraries available in Python, such as NumPy and SciPy, that can be used to perform calculations and simulations with Markov chains. Understanding the properties and applications of Markov chains can be a valuable tool for modeling and analyzing complex systems. By using Python to represent and simulate Markov models, researchers and analysts can gain insights into how these systems operate and make more accurate predictions about future outcomes. While Markov chains have some limitations, their ability to model complex systems using relatively simple models makes them a powerful tool for data analysis. How to evaluate unknown vulnerabilities and zero-day vulnerabilities, and give a reasonable method to measure their utilization probability, will be thw focus of future research.

Acknowledgement

I am grateful for all the support that my family and professors of the course has given me, and I look forward to their continued accompany.