1. Introduction

Software testing, a crucial component of software engineering, can consume up to half of the time and cost in the development process [1]. While continuous integration and continuous delivery/deployment (CI/CD) can enhance testing efficiency and reduce time costs through well-designed automated testing and effective development process management, these approaches still demand ongoing investment from skilled software test engineers.

In recent years, software development has evolved significantly with advancements in cloud native technology, big data, and the Internet of Things (IoT). Simultaneously, new fields like autonomous driving, augmented reality (AR), and virtual reality (VR) have created unprecedented demand for innovative software solutions. These shifts have dramatically altered software products' form, design, development processes, user base, and usage scenarios. Consequently, software testing has become more intricate and multifaceted than ever before. Furthermore, as agile development and DevOps practices gain widespread adoption among development teams, quality assurance has expanded beyond the traditional post-development testing phase. It now encompasses the entire lifecycle of software products—from initial design through development, release, and launch. This expansion necessitates improved test efficiency, broader testing methodologies, and enhanced test scale and coverage to address the increasingly complex challenges of modern software development.

To address the numerous challenges in software testing mentioned earlier, applying artificial intelligence (AI) technology and methods has emerged as a promising solution. AI-based software testing introduces innovative approaches to enhance test efficiency, accuracy, and effectiveness, enabling better management of software testing issues in various new and complex scenarios. With these advancements, developers can construct more comprehensive tests and ensure the reliability of software systems.

Traditional software testing often relies heavily on manual labor to complete complex tasks, which comes with several drawbacks: lengthy test cycles, inadequate code coverage across all possible branches, and the ever-present risk of human error. While automating some repetitive tasks can be an effective solution, there's still significant room for improvement in overall testing efforts. AI and machine learning-based software testing offers a more sophisticated approach. These technologies can leverage vast amounts of data to automatically generate, construct, execute, and analyze tests. By predicting potential defects based on historical data, AI-driven testing enables faster execution, shorter cycles, improved coverage, and enhanced issue detection. This not only streamlines the testing process but also addresses many of the limitations inherent in traditional manual testing methods.

This article will examine the recent advancements in AI applications for software testing in emerging fields, discussing their benefits, limitations, and future potential through specific case studies. Furthermore, it will categorize the current research focus areas within testing classifications and tasks, evaluate testing effectiveness, identify areas for future coverage, and explore new potential AI applications in software testing for emerging fields.

2. Background and related work

2.1. Overview of software testing

The ANSI/IEEE 1059 standard defines software testing as the process of examining software to identify discrepancies between its current state and its intended state (commonly referred to as defects, errors, bugs, or faults) and evaluating various aspects of the software [2]. As a vital process in the lifecycle of software development, the primary objective of software testing can be concisely summarized as identifying discrepancies, errors, defects, and missing features between the software product and its originally specified requirements.

Figure 1: Testing levels

Software testing can be categorized based on the test task type, testing level, and its association with specific stages in the software development process. It is typically divided into four distinct levels: unit testing, integration testing, system testing, and acceptance testing, as illustrated in Figure 1.

Software testing can also be categorized into two main types: manual testing and automated testing, in which the content of manual testing can be subdivided into many types according to the specific situation. Automated testing, on the other hand, is a programmatic treatment of repetitive work in some manual tests, as shown in Figure 2.

Figure 2: Types of software testing

In the specific classification of testing tasks, white-box testing allows testers to see all the software's code. In contrast, black-box testing conceals the code from testers during the test. This approach prevents testers from designing test cases with preconceived notions, enabling them to identify software issues more effectively. Black-box testing further divides into two parts: functional and non-functional testing. Functional testing verifies the software's functional requirements, while non-functional testing evaluates its performance, usability, stability, and security. For these various categories of testing tasks, automated testing can offer distinct benefits. Likewise, applying AI technology to specific software testing tasks presents multiple opportunities for implementation and improvement.

Additionally, from the perspective of the software testing lifecycle, testing can be divided into distinct phases: requirements analysis, test planning, test case design, test environment setup, test execution, and test cycle closure. Figure 3 illustrates this process.

From these perspectives, AI-driven software testing can enhance the testing process across multiple levels and angles. These enhancements include, but are not limited to, test design, use case generation, test planning, test environment deployment, test execution, defect prediction, and analysis.

Figure 3: Software testing lifecycle

2.2. Related work

In recent years, significant practical research has advanced the application and development of AI in software testing. Since 2019, IEEE has hosted five international academic conferences focused on AI Testing, exploring topics of "the usage of artificial intelligence for software testing" and "the testing of artificial intelligence methods themselves" [3]. These conferences have yielded 111 relevant papers.

There are some examples of specific research literature: Ehab Ebrahim, Mazen Sayed, et al. proposed an AI-based chatbot to improve the organization of release planning and resource management in the process of software development and testing [4]; Faqeer Ur Rehman, Madhusudan Srinivasan, et al. analyzed and studied the application prospects and challenges of transformation testing for machine learning [5]; Yue Ding, Qian Wu, Yinzhu Li, et al. proposed a method based on deep learning to improve the detection of cross-function null pointer risk in unit testing [6]; Yejun He, Muslim Razi, et al. designed a testing framework for autonomous vehicle using AI semantic models [7]; Caglar Osman, Taskin Furkan, et al. developed a software testing platform based on cloud and AI, called ChArIoT [8].

Overall, these research directions primarily focus on specific AI applications in various aspects of software testing. They rarely discuss necessary improvements or adjustments at the algorithmic level for AI in software testing scenarios. Moreover, much of the research is based on actual production and analyzed in conjunction with specific practical scenarios, making these findings more valuable for practical implementation. Additionally, several studies have explored recent advances in applying large deep learning models, including ChatGPT, convolutional neural networks (CNNs) for image recognition and object detection, and generative AI.

Currently, there is no shortage of raw data for testing in real production scenarios, but model training requires processed data. Machine learning training datasets for different scenarios are still insufficient. This part of the work needs to be further developed and improved in the future.

3. Research Methodology

This article employs the SLR (systematic literature review) methodology developed by Angela Carrera-Rivera, William Ochoa, et al. [9].

The review centers on the keyword "Artificial Intelligence in Software Testing," focusing on developments in emerging fields from 2021 to 2024. The study's data sources are confined to five prominent databases: Scopus, Web of Science, EI Compendex, IEEE Digital Library, and ACM Digital Library. Table 1 provides detailed information about these databases.

Table 1: Details of databases

Based on this topic, the paper presents the following research questions (RQs):

• RQ-1: What are the applications of Artificial Intelligence in software testing?

• RQ-2: What methods of artificial intelligence are commonly used?

• RQ-3: What are the pros and cons of AI testing?

• RQ-4: How to evaluate the effectiveness of AI testing?

• RQ-5: What are the challenges of applying AI methods to software testing in emerging fields?

The search strategy for this article is illustrated in Figure 4. First, the entry and exit criteria for the search are defined and the initial search is carried out within the scope of the selected database. Then, the preliminary results obtained from the search are manually reviewed, and the corresponding Quality Assessment Checklist (QA Checklist) is established to fine screen the selected results. In the end, about 30 articles are obtained for review analysis.

Figure 4: Search strategy

The various criteria used for the search are shown in Table 2.

Table 2: Inclusion and exclusion criteria

*The x value is adjusted according to the search.

Through the above criteria, a preliminary search was finally carried out in various databases. Although the search strings differ slightly in format and content, it can be ascertained that each database follows the same search criteria. According to the content of the preliminary screening, the retrieved data files are exported from each database. After converting it to a CSV file, it can be used for further fine screening and data processing. The QA Checklist is shown in Table 3. It is used to assess the quality of the obtained literature and to conduct a fine selection based on the examination results.

Table 3: QA Checklist

Eventually, 30 articles were finally obtained from the databases as the objects of the review study. In the selected literature, the data were extracted and summarized around the range of indicators mentioned in Table 4. The content of each article is read and analyzed in detail to form relevant report content.

Table 4: Indicators of data extraction

4. Results

4.1. Report

The findings of these studies are summarized as shown in Table 5 through the reading and analysis of the selected research papers.

Table 5: Findings of selected studies

4.2. Responses to RQs

4.2.1. RQ-1: What are the applications of Artificial Intelligence in software testing?

For this question, the following aspects can be summarized through the study of the selected literature as the specific application of AI in software testing:

• Test Case Generation - This topic has been addressed in several articles and research studies. Machine learning and deep learning techniques enable the use of large datasets as a foundation for test case generation. Furthermore, generative AI shows promise in automating test case creation for more complex scenarios.

• GUI Test - This direction allows AI to demonstrate its capabilities more tangibly. As computer vision and object detection technologies continue to advance, machines are becoming increasingly proficient at analyzing and recognizing images. Consequently, GUI testing—a task that once heavily depended on human effort—can now be effectively managed by AI.

• Defect Prediction - Several studies have highlighted AI's role in defect prediction. By analyzing vast amounts of historical problem data, AI—particularly through machine learning methods—can significantly aid in predicting software defects. These studies also discuss various defect prediction solutions, such as DePaaS [31], that integrate DevOps and cloud technologies. These integrated approaches offer more comprehensive and reliable defect prediction support through broader and more efficient data collection.

In addition, the studies also cover the use of AI in the following sections: test planning, test prioritization, test case optimization, test execution, mutation test, test coverage, unit test, and embedded test.

4.2.2. RQ-2: What methods of artificial intelligence are commonly used?

For this question, the following specific methods or directions of AI can be summarized from the study of the selected literature: machine learning, deep learning, reinforcement learning, large language models, computer vision, behavioral cloning, generative AI, and natural language processing.

Within the scope of the selected articles, most of the articles focus on exploring the use of AI technology in different aspects of software testing. There is little discussion about specific methods or algorithm improvements in AI, with only a few articles on this:

• Paper [18] discusses AI-driven CI/CD, which mentions the application and improvement of several algorithms, including NLP (Natural Language Processing), CSA (Cuckoo Search Algorithm), ROA (Randomized Optimization Algorithms), and PSO (Particle Swarm Optimization).

• In paper [26], the application of WSO and KELM algorithms for defect prediction on the IoT is discussed.

• A test case generation scheme based on the Behavioral Cloning method is proposed in the paper [28].

4.2.3. RQ-3: What are the pros and cons of AI testing?

The study summarizes some of the advantages and disadvantages of AI testing, as shown in Table 6.

Table 8: The pros and cons of AI Testing

4.2.4. RQ-4: How to evaluate the effectiveness of AI testing?

In fact, for applications in the testing field, the validation of the validity of AI tests is no different from the validation of the validity of general testing behaviors, and it is sufficient to verify them using the same or similar methods. For most simple test scenarios or test tasks, the validity verification of AI testing can be demonstrated by a simple comparison of results—that is, the test task is manually completed with traditional testing methods, and if the results are consistent, the test process and conclusions are valid.

For relatively complex scenarios, it is necessary to consider constructing some specific metrics for data statistics and analysis to determine the effectiveness of AI testing. Specific metrics include the following: accuracy, precision, specificity, recall, F1 score, ROC, AUC, ACC, and confusion matrix.

4.2.5. RQ-5: What are the challenges of applying AI methods to software testing in emerging fields?

Emerging areas here include, but are not limited to, the following: DevOps, Cloud-Native, Autonomous Vehicle, IoT, and Games.

In these areas, the requirements and tasks of software testing differ from those of traditional software testing in the past. In more and more scenarios, software testing emphasizes quality assurance throughout the software development lifecycle. Internet companies that are on the front line of testing left and testing right are often mentioned topics. As a result, AI faces some specific challenges in software testing in these emerging areas:

• In the cloud scenario, how to better solve the test problem of distributed systems?

• Sometimes the failure of SDN (software-defined networking) systems is not necessarily a bug in the code at the functional level, but a problem or limitation of hardware resources that causes the failure at the software level. What better solution to this problem is software testing or quality assurance?

• What can AI do in performance testing? What are the advantages of AI technology in the face of system stress testing scenarios with huge traffic volumes?

• Legal and ethical issues still need addressing in the adoption of AI technology for safety and stability testing. This is especially crucial for autonomous driving, as it directly impacts human lives.

5. Analysis and discussion

5.1. General discussion

In the realm of software testing, AI shows great development potential and promising applications. The fresh topics presented annually at IEEE's AI Test conference, along with the steady growth of research interest in various specific directions, demonstrate that this field still has significant room for expansion and innovation.

Through the analysis of research papers in recent years, this paper can summarize some parts of the testing field that are of concern but have not yet been involved in AI testing:

• By leveraging AI for traffic playback, the effectiveness and timeliness of performance testing can be improved and enhanced, which will provide more reliable and rich performance test data.

• By realizing automatic periodic fault injections in large-scale clusters based on machine learning, the concept of chaos engineering can be practiced improving the stability and reliability of the system, and the pain points of self-recovery of large-scale distributed systems can be explored.

• By analyzing and summarizing the LOG information more abstractly based on LLMs, and combining deep learning to summarize past defect cases, the defect location function at the code level is faster and more convenient. This can help developers and testers complete the debug process better and faster.

• By building a more complete automated test cluster based on cloud-native tools such as Kubernetes and Tracing, AI-driven test environment resource management can be implemented to provide more intelligent TaaS (Test as a Service).

Moreover, current research lacks emphasis on enhancing AI algorithms for testing purposes. There remains ample opportunity for further exploration and study in this area, particularly for specific real-world applications.

5.2. AI Testing in different fields

A comparison of research content over time reveals that many scholars and industry professionals have increasingly focused on emerging fields and new concepts in recent years. According to statistics, over 50% of studies relate to content in these emerging areas.

In the realm of DevOps, AI's role in testing primarily centers on process optimization and improvement. Additionally, new approaches may positively impact AI testing efforts in specific scenarios where testing is shifted left or right.

For Cloud Native environments, current research utilizes cloud native and cloud technologies to support AI testing design and implementation. However, there's a notable lack of research on AI testing for cloud services themselves.

Autonomous driving and the Internet of Things have emerged as hot topics in recent years. AI software testing in these fields emphasizes safety and stability. Testing in these areas also focuses on embedded systems tasks. Unlike traditional computer-based tests, tests on small mobile terminals are more performance-oriented. Furthermore, differences in hardware architecture introduce additional challenges to testing.

5.3. Limitations and deficiencies

Despite extensive efforts to search numerous topic-related databases within the given time constraints, some omissions in the search were inevitable. Additionally, space limitations necessitated selecting only a small number of publications closely aligned with the research topic during the fine selection process. In recent years, AI testing has seen abundant research, though many valuable articles were not included in this discussion. Consequently, the inability to analyze a more comprehensive set of reference data represents one of this paper's limitations.

Furthermore, time constraints prevented the verification of experimental validity and authenticity for the articles included in this paper's discussion. Many of these studies provide detailed source code and data for readers to examine and analyze. Therefore, the inability to reproduce and verify some methods represents another limitation of this paper. This aspect of the work can be pursued in the future when conditions permit.

6. Conclusion

Software testing, a critical component of software engineering, plays a pivotal role in ensuring quality throughout the software development lifecycle. As industry evolves, software testing in emerging fields faces increasingly complex and diverse challenges. The application of AI methods to optimize processes and enhance efficiency offers innovative approaches to software testing and quality assurance.

This paper reviews and analyzes research papers from the past four years (2021-2024) on this topic, summarizing the specific distribution and development trends of research content in this field. It also proposes ideas for future research directions and technological development paths.

In contrast to similar review articles, this paper focuses more on the application and development of AI technology in emerging fields. It also analyzes the strengths and weaknesses of specific case studies.

This review offers valuable insights and references for future research. Additionally, the questions raised, and areas identified for further research in the analysis and discussion sections can serve as a foundation for future work.