1. Introduction

With the development of technology, the relationship between technology and education is mutually reinforcing [1]. The use of ChatGPT in web-based autonomous learning has become increasingly prevalent among students [2]. With ChatGPT’s involvement in the education ecosystem, the education system will undergo a new transformation [3]. Opinions regarding ChatGPT’s impact on learning show extreme polarization, with concerns about academic integrity and the potential for improved work efficiency [4, 5]. The ethical considerations and concerns about artificial intelligence have also attracted widespread attention in the industry [6]. For the study of self-efficacy, current scholars have conducted research on the relationship between innovation and entrepreneurship and self-efficacy, as well as research on self-efficacy in specific disciplines [7, 8]. But it is still a research gap on whether artificial intelligence is related to learning efficacy. This study will build on Bandura’s theory of self-efficacy. Self-efficacy is defined as a belief that an individual has in his or her own ability to produce desired behaviors, and it can also be understood as the subjective judgment of whether people can successfully engage in a certain achievement behavior [9]. This study combines quantitative and qualitative research methods to explore the relationship between the usage of ChatGPT and the learning efficacy of Chinese college students. Quantitative research is based on corresponding theories and combined with qualitative research to make the results more comprehensive. The exploration of this issue is of great significance for how to effectively improve students’ learning efficiency in the future.

2. Research Methods

Quantitative and qualitative research methods were incorporated into this study. The quantitative approach of questionnaires is apt to evaluate the relation between the usage of ChatGPT and the learning efficacy among Chinese students, while the qualitative approach of interviews is used to gather more open data that is related to their real emotions and real experiences in the process of using ChatGPT.

2.1. Participants

The participants of this study consisted of 100 university students from China (55 males and 45 females), including undergraduate and postgraduate students who are still enrolled. These participants come from different majors, including the Department of Education, the Department of Chinese Language and Literature, the Department of Information System and Information Management, the Department of Artificial Intelligence, and the Department of Law, etc.

2.2. Measures

2.2.1. Questionnaire

Participant’s personal information was gathered from 5 items, which included gender, grade, the features of ChatGPT that students prefer to use, the frequency of the usage of ChatGPT among students and the Efficacy-ChatGPT Scale (ECGS). A total of 113 questionnaires were returned, and 100 valid questionnaires were obtained through screening eventually. All of the students who answered 100 questionnaires confirmed that they had the experience of using ChatGPT to varying extents.

2.2.2. Efficacy-ChatGPT Scale (ECGS)

The study adopted the Academic Self-efficacy Scale (ASS) mentioned in Angelo Reyes Dullas’s paper, including the concepts and variables that are highly related to self-efficacy. These factors consist of Self-Regulated Learning (SRL), Perceived Control (PC) or Internal Locus of Control, Persistence, and Competence [10]. The concepts of SRL, Persistence, and Competence are regarded as part of self-efficacy. Furthermore, PC or Internal Locus of Control is strongly related to self-efficacy, and it has an obvious influence on students’ academic self-efficacy. This scale is suitable for most situations in this study. However, since the focus of this study is to explore the impact of the usage of ChatGPT on the learning efficacy of Chinese college students, the scale has been screened and modified to make it more suitable for our research. The remaining 20 items were placed on a 5-point Likert scale (1=strongly disagree, 2=disagree, 3=neutral, 4=agree, 5=strongly agree). As the score increased, students agreed more with what the item described as well. And then, to assure the availability of this slightly modified questionnaire, the Analysis of Cronbach Alpha and the Analysis of KMO and Bartlett were conducted to illustrate its high reliability and high validity (Cronbach alpha=0.870, KMO=0.896). Table 1 and table 2 below show the results of the analysis of Cronbach alpha and KMO.

2.2.3. Open-ended Interview

To figure out the correlation between the usage of ChatGPT and the participants’ learning efficacy more comprehensively, 3 volunteers from the 100 students who participated in the questionnaire were invited to attend an open-ended interview.

Interview questions included their experiences and the benefits of using ChatGPT, their anxiety and confidence in using ChatGPT, the future of ChatGPT in teaching and learning, and the changes in learning efficacy.

2.3. Procedure

This study was conducted in June and July 2023, including 100 volunteers who finished the questionnaire with 24 items. The questionnaire was presented in Chinese, and it was produced through a Chinese questionnaire platform dubbed Wenjuanxing and distributed among the student groups. Every returned questionnaire was carefully checked to ensure its validity, and 13 questionnaires were judged invalid and were discarded.

After finishing the questionnaire data collection, 3 participants out of 100 students were invited to participate in the interview. Since the participants in the interviews were scattered across the country, the authors chose to conduct the interviews in the form of online conversations rather than face-to-face offline. Participants were given certain topics individually and wrote down their ideas, which were collected and analysed at the end.

2.4. Hypothesis

The expected assumption for the results is that Chinese college students would use ChatGPT more in course assignments, report presentations, and paper writing. A few students would have a high frequency of using its related functions, while the majority of students would have a relatively low frequency of use. The learning efficacy of Chinese college students would be significantly positively correlated with their usage of ChatGPT. Compared with the students who do not use ChatGPT, the students who use ChatGPT would have a stronger learning efficiency, and as the frequency of its use increases, the learning efficiency would gradually increase. Appropriate usage of ChatGPT could bring many benefits to students, such as increasing their learning motivation, expanding their horizons, and improving their learning effectiveness; but excessive reliance on ChatGPT would bring them some disadvantages like limited creativity and limited innovative thinking.

3. Data Analysis and Research Results

Descriptive Analysis, Non-parametric Test (Chi-square test), Parametric Tests (Independent-Samples T-Test, One-Way ANOVA) and Correlation Analysis were used to visualize data results and answer research questions through clear charts.

3.1. Research Questions

RQ1. What features of ChatGPT do Chinese students prefer to use, and how often?

RQ2. Is there a correlation between the usage of ChatGPT and learning efficacy of students in China? If so, what is the relationship?

RQ3. To what extent, how does ChatGPT help them?

3.2. Research Results

Figure 1: The results of multiple choice questions.

As figure 1 shows, there were seven important items provided for students to choose from, including looking up information, doing coursework, practising language, assisting with dissertation writing, reviewing for the exam, reporting speech, and conducting research related to ChatGPT. The bar chart compares the number of students who accomplished each activity with ChatGPT in their daily study and its proportion.

It was clear that college students prefer to use ChatGPT to look up information, do coursework and assist with dissertation writing. The function of looking up information was the most popular feature of ChatGPT, with 82 users. Then there were functions for doing coursework and assisting with dissertation writing, which ranked second (with 64 users) and third (with 53 users) respectively.

3.2.1. Gender, Grade and Frequency

Gender and grade were the independent variable and frequency was the dependent variable, both of which were categorical variables. According to table 3 and table 4, the chi-square test in the non-parametric test was used for difference analysis. The results showed that there were significant differences in the frequency of using ChatGPT among college students of different genders and grades.

3.2.2. Gender and Four Types of Impact Factors

As table 5 shows, the three items of self-regulated learning (SRL), persistence and perceived control (PC) did not show significant results (p>0.05), which means that different genders could not cause obvious differences in these three types of impact factors. Whereas the factor of competence showed a significant difference based on different gender (p<0.05).

3.2.3. Grade and Four Types of Impact Factors

Table 6 indicates that none of the four types of impact factors showed significant differences due to different grades.

Table 6: Analysis Results of One-Way ANOVA. 

3.2.4. Frequency and Four Types of Impact Factors

As illustrated in table 7, In the case of using ChatGPT with different frequencies, all four types of impact factors showed significant differences.

3.2.5. Correlation Between Gender, Grade, Frequency and Each Impact Factor

Spearman’s correlation coefficient was used to detect and analyse the correlation between gender, grade, frequency and each impact factor.

As can be seen from table 8 and table 9, all four impact factors were positively correlated with the frequency of using ChatGPT. However, the factors of perceived control (PC) and persistence did not correlate with gender and grade (The correlation coefficient was less than 0.2). In addition, the factor of competence had a significant negative correlation with gender but had no correlation with grade; while the factor of self-regulated learning had an obvious negative correlation with grade but was not correlated with gender.

In terms of the competence influencing factor, the score of male students among the surveyed subjects was generally higher than that of female students. The higher the grade of college students, the less positive impact on the self-regulated learning of learning efficacy when using ChatGPT.

Table 8: Analysis Results of Spearman’s Correlation Coefficient- Standard Format. 

Table 9: Analysis Results of Spearman’s Correlation Coefficient-Detailed Format.

3.2.6. The Analysis of Interview Results

In this study, 3 volunteers from the 100 students who frequently use ChatGPT were invited to attend an open-ended interview.

They believed that ChatGPT was widely used and could help them solve some problems in their daily study and daily life and they also agreed that the usage of ChatGPT was more conducive to success.

ChatGPT could enrich their cognition to build a more complete learning system, and it could help students complete basic work like the production of PPT, thus freeing people’s time to do more mental work. It could also help them improve their independent learning ability and develop their habit of active learning. By using ChatGPT, a lot of time was saved to consider deeper issues related to academic aspects. At the same time, inspiration emerged, and innovative thinking was enhanced, so that the comprehensive ability of the individual was significantly improved. In addition, with the help of ChatGPT, they were more willing to choose more challenging courses, their enthusiasm for scientific research also increased, and their confidence in achieving their goals had become stronger.

4. Discussion

The benefits that ChatGPT brought to them were based on appropriate use. Once overused, there would be adverse effects such as exam anxiety, insufficient grasp of the internal logic of knowledge, and academic misconduct. Therefore, it should be extremely important to master the degree of its use.

Based on the analysis of research results, college students should also increase their frequency of using ChatGPT, so that their learning efficiency will be more significantly and positively affected. Female college students should be more confident in their ability to use the most cutting-edge Internet products and should be encouraged to use ChatGPT to help themselves with their studies. Senior college students should make more use of ChatGPT to help develop their self-learning ability.

5. Conclusion

The research highlights a research gap regarding the relationship between artificial intelligence, self-efficacy, and learning efficacy. The authors designed a survey questionnaire and conducted interviews by browsing literature from Chinese and foreign scholars. Through research, this article found that ChatGPT was mainly used by students in daily learning and daily life to look up information, do coursework and assist with dissertation writing; The impact of ChatGPT on the learning efficiency of domestic college students was relatively positive, but it was not yet significant; In terms of learning efficiency, there was a significant difference in the frequency of using ChatGPT among college students of different genders, with male students overall scoring higher than female students in the survey subjects; Meanwhile, the authors found that the higher the grade of college students, the smaller the positive impact on self-directed learning of learning efficiency when using ChatGPT. In the interview, the authors found that students generally believed that ChatGPT had a positive effect on learning efficiency, but students also mentioned that ChatGPT was a double-edged sword.

In learning and life, people need to control the use of time and do not have excessive dependence, which may lead to a lack of deep understanding of knowledge, and thus Test anxiety, which is also a topic worthy of future scholars to study. The limited number of domestic college students currently using ChatGPT and the scope of data collection pose limitations to the study. Future scholars can also focus more on analyzing ChatGPT and students’ learning habits and initiative, moving beyond a mere examination of the advantages and disadvantages of artificial intelligence.

Authors Contribution

All the authors contributed equally and their names were listed in alphabetical order.