1. Introduction

In contemporary games, the importance of NPCs (non-player role) are almost indispensable. NPCs can be any form of virtual characters, from businessmen and teammates to enemies and mission targets, playing a wide variety of roles, interacting with players, enhancing the fidelity and complexity of the game. The appearance, behavior and dialogue of NPCs are programmed by game developers, which also require a lot of time, manpower and financial resources. With the improvement of AI (artificial intelligence) technology, people's requirements for games have gradually increased, and the requirements for game AI are also increasing. This also makes AI systems more complex and diverse in the game [1]. As a result, more and more game developers began to explore the use of AI to create NPCs. This method can not only save developers' time, but also AI can make NPCs achieve more functions, make the game richer and more playable. However, after a literature search, no one has reviewed the current situation in this area in recent years. Therefore, this paper aims to review the related research and application of using AI to create game NPCs in recent years, and sort out the current research status.

2. Historical development

With the gradual growth of the game industry, AI technology has gradually developed and has made great strides in game development. Zhao Lexuan proposed that AI directly and significantly reduced the threshold of the game, and players could play the game more easily [2]. Moreover, in many kinds of games, there will be various NPCs. In the course of development, two methods have been the most popular in the past, the first is the finite state machine (FSM), the second is the behavior tree.

Shi Boxuan mentioned that the state machine appeared in about the 1970s, when AI controlled hostile NPCs [3]. After that, many game manufacturers began to pay attention to the application and quality of AI in games. For example, arcade game 'Qwak', 'Speed Race', and PC-side game 'Star Trek'. NPCs in these games are controlled by AI, but they no longer simply move, but are unpredictable because they process more information. After that, developers added more changes to NPCs, such as 'PAC-MAN', and the AI strategy in this game became more complex. Then came 'Karate Champ', a role-playing game that featured the first AI combat character in game history. The behavior tree model was first proposed by Next-Gen AI in the game field. In 2001, they also published some key ideas about behavior trees [4]. Under the method of behavior tree, many popular games have been created, such as 'Red Dead Redemption'.

3. Research statues

3.1. A review of the limitations of AI game NPCs

With the rapid development of game engines, game companies are able to create many beautiful pictures. Therefore, the playability and innovation of the game itself has also become the key to standing out from many games [5]. The NPCs in the game directly or indirectly determine the fun and playability of the game [6]. Because of this, many game developers already have many theories and methods to create AI that meets the needs of players. But contrary to expectations, Jeff suggested that even if you went to some game developer conferences with many powerful companies at the time, using a finite state machine was still a game solution that most people could give [7]. This statement has also been confirmed by domestic companies. Shi shared his own experience, in 2016, he attended the Unity developer conference, and communicated with the personnel of NetEase, Tencent and other companies. The solution still coincided with each other and still used state machines or behavior trees [3]. This practice was broken only in 2018. At that time, a rare AI component appeared in the test version, which had the function of machine learning.

K Tupe et al. also pointed out the limitations of many in-game NPCs^[8]. NPCs act in limited scope and act in limited ways. These limitations still exist in a large number of games today. As a result, players can often quickly determine where NPCs’ limitations are. In order to pursue better and more credible NPCs, the article mentioned, including but not limited to optimizing offline automation behavior technology, adjusting actions and learning better countermeasures online automation technology, real-time control mechanism and so on. This paper highlights the shortcomings and optimization directions of NPCs, and also mentions the relatively basic principle of AI and the prospect of the future. The limitations are worth thinking about. As for how to realize these in detail, it also needs further research, which is not mentioned in this paper.

Mao Xiangyu proposed two methods for the implementation of NPC intelligent behavior based on Uinyt3D, which are still finite state machines and behavior trees [9]. Mao only summarized these two methods in the paper, and did not propose implementation schemes and specific experiments, and the limitations of such NPCs should still be within the scope of K Tupe [8]. It can be seen that at that time in the same year, there was still no direction for this in China. Shi [3] conducted research on behavior trees and finite state machines, and found that in the game industry at home and abroad, neither method can cope with the AI control of multiple character NPCs. Among them, the advantage of the state machine is that it performs a very fast process, simple operation and high flexibility. But admittedly, it scales poorly. Because of this drawback, many game AI behaviors are restricted, especially detailed movements. As more states increase, the number of transitions between states becomes difficult to control. At the same time, different NPCs have their own different state machines, and then add scenes, difficulty and other aspects in this situation, it is estimated that such a game will need hundreds of state machines, and the development and maintenance work brought by this, as well as the corresponding costs, are completely unrealistic. However, although behavior trees are highly reusable, the decision-making speed and readability of this method rapidly decreases as complexity increases. In summary, behavior trees also encounter the same bottlenecks as state machines, slowing down game development. This summary clearly explains the limitations of in-game NPCs using two common methods.

But for so long, the reasons why most game makers still choose to use finite state machines and behavior trees remain to be investigated. First of all, machine learning requires fault tolerance, but in most games, there is basically no fault tolerance and it is very prone to failure. Second, the process of machine learning is not easily manipulated, and there are many uncertainties. Third, AI training and learning require high training costs [10]. Finally, the game update will most likely lead to the loss of the training results, so such an AI can not give a definitive result at all, which is very detrimental to the development of the game, and requires more manpower, which also cannot bring more economic benefits [11].

3.2. A review of breakthrough research based on AI game NPC

The method proposed by Shi Yuan is different from the above two. Based on the reinforcement learning theory, the proximal strategy algorithm is optimized and designed, and the non-player character AI system is designed based on the hierarchical architecture, which is divided into perception layer, strategy layer, request layer and behavior layer [12]. This method reduces the coupling of system code, increases the flexibility of the system, and facilitates later maintenance. After designing and testing, Shi found that the system has significantly improved in scalability and development difficulty in a Rogue-lite game designed by himself. All in all, the optimized algorithm is used instead of the behavior tree, which enhances the action force of NPCs and increases the playability of the game. However, the study only targets NPCs in combat behavior, which has certain limitations and is currently only suitable for small game groups.

In addition, Shi also proposed a game AI design method for multi-NPCs and applied it to NPCs in competitive mobile game(Sohu Changyou "Agent Squad") [3]. In his design structure, the dimension and behavior abstraction layer is used to abstract the behavior of NPCs and the factors that can determine the behavior, and then design the appropriate model for each dimension, configure the file, parse and classify the file, then transfer the relevant data to the computational model, and finally select the most expected execution in the behavior. Through testing, this model can meet the needs of multi-NPC game AI, and the performance and consumption cost are also within a reasonable range. Compared with the previous two initial design methods, this construction not only shows high scalability, but also improves the efficiency of development and reduces maintenance and debugging costs. Nevertheless, there are still some functional deficiencies in this structure, the workload is preserved, it needs to be optimized, and some limitations are still present compared to the study of stone [12]. But it is not difficult to see that the current game has many algorithm optimizations for competitive AI hostile NPCs, which makes it possible for these NPCs to challenge humans.

Compared with the above proposed methods, Tang Zhentao et al. described the problems of real-time fighting games and elaborated a series of methods to achieve the combination of AI and fighting games [14]. They are heuristic rule type, statistical forward planning type (including Monte Carlo tree search algorithm and rolling time domain calculus method) and deep reinforcement learning type. Compared with the consistent heuristic rule-based strategy search method, the statistical forward planning method and the deep reinforcement learning method have good environmental adaptability and strategy model optimization. However, its generalization and efficiency need to be enhanced. If it is possible to combine several of them, such as statistical forward planning and deep reinforcement learning, it may have more potential.

3.3. A review of research based on the perspective of game developers

The functions of artificial intelligence systems in shooting games summarized by Yu Kechun are divided into three modules, among which they can sense enemies, assign teammates and initiate decision-making, which are very detailed and comprehensive [13]. It can be seen that in terms of combat, the functionality of artificial intelligence is very strong. Based on some game cheats that use code to make the character strong, AI can also do it, especially in competitive games such as shooting and implementing strategies. However, too strong enemies will frustrate players, resulting in bad gaming experience. So making AI make mistakes appropriately in games, or simulating different levels of human technology, may enrich the game.

In addition to these types of NPCs, the behavior of some neutral or friendly NPCs will be more technical, especially when dialogue is required. At this time, the single way of dialogue has considerable fixation. But there are actually few related studies, because of its complexity, and the reality of combining with games, are not easy to achieve. Zhu Peng mentioned in the article that it will take a long time for the breakthrough in algorithms, about 2 to 5 years, AI will present a pyramid structure [15]. The existing OpenAI and ChatGPT models can already provide a natural and comfortable continuous conversation experience, as well as the ability to write personal documents, and so on. So whether NPCs in the game can also use this kind of dialogue AI to integrate or not, perhaps in the future, it will be a way to make NPCs more vivid and more playable.

Throughout the current research status, a single computational intelligence method has its own unique advantages, but there are also corresponding shortcomings. How to organically integrate the existing intelligent methods to form complementary advantages is a direction of current game AI. In addition, in the case of more comprehensive development of hostile AI, how to make players more experiential and playful is also a problem. When AI is neutral or friendly, making dialogue and decision-making more flexible and humane is also a trend to make games richer and more intelligent. Finally, how to solve the problems caused by the application of AI in the game is also one of the difficulties that need to be broken.

4. Conclusion

This paper reviews the research and application of using AI to create game NPCs. Although AI faces some challenges in creating NPCs, it is still a very promising field in game development. With the development of AI and game industry, it is believed that this will get more attention and research. In the future, more intelligent and more playable NPC designs are expected to enrich the game experience of players.