1. Introduction

Nowadays, technology has an unpredictable potential for changing the world, including the educational landscape. Before the existence of advanced technology, textbooks might be the only material for study, and students only can sit in the classroom to listen and write notes. Compared to traditional instruction, the utilization of technologies provides a more innovative possibility that is not only beneficial for providing a more personalized educational experience but also helping students to deepen the understanding of knowledge and attract their attention for higher engagement.

With the rapid technological advancements, various technologies have been introduced into education. For instance, online learning platforms, Virtual Reality (VR), Augmented Reality (AR), animation, interactive dashboards, and gamification can be utilized for students’ learning by creating a more flexible educational experience. According to the 2024 EDUCAUSE Horizon Report, it is significant for education to adapt to a technology-based world to meet the demands of the labor market, including the integration of emerging technologies [1].

Augmented Reality, as one of the significant assisting tools in educational settings, is defined as “the computer-generated information in a real environment”, combining the virtual objects into the physical environments [2]. The first AR was introduced as a tool for the training of airline and air force pilots in the 1990s [3], and it has been over 25 years since the first AR application was utilized in the educational area [4]. At present, more academic institutions are starting to use AR as one of their teaching tools, and becoming a popular technology with educational purposes. For instance, Silver Lake Elementary utilized an AR sandbox that was developed by UCLA to teach their fifth-grade students about topography, erosion, and water flow by observing changes in the rainbow-reflecting sand [5]. The AR sandbox provides a more direct way to introduce complex theories to students in elementary school where students can create different sand mountains and compare the difference between each landscape.

Over the years, lots of research has focused on analyzing the advantages of AR and its limitations which can serve as an efficient tool to enhance the engagement of students and performance [6]. Beyond that, the researchers noticed that AR could help students understand individuals’ knowledge more clearly and increase their learning motivation [7]. However, most teaching faculty members do not have prior experience on the usage of AR, such as no practical knowledge on the integration of AR into education [7]. Hence, understanding the impact of AR on students' performance is important since it might help teachers to understand in what ways to maximize the efficiency of AR in study.

Although existing research has demonstrated that educational technologies, such as AR, have a positive influence on student's academic performance on learning science, medical and healthcare, biology, and others [8-10], this research aims to provide a further investigation into the impacts of interactive AR and Animation on students' performance in their English vocabulary learning. By applying ordinal logistic regression and logistic regression for two groups of datasets respectively, the study aims to contribute to the evaluation of students’ learning behavior under the intervention of educational technologies in language education. Meanwhile, the results may be an indicator of policy development regarding the integration of emerging technology in educational settings.

2. Method

2.1. Dataset preparation and preprocessing

The original dataset, the AR-Based English Vocabulary Learning Dataset, utilized for the research is from Kaggle [11]. The dataset contains information from 200 users between the ages of 10 and 17 years old and 12 variables, capturing their demographics of users, engagement metrics, learning performance with various AR features and activities, and user feedback.

To understand the efficiency of different AR features on language learning, the original dataset has been divided into two groups, the Interactive AR group, and the Animation group, for further investigation. The author selected parts of the dataset, such as Duration, Engagement_Score, Accuracy, Completion_Rate, AR_Features_Used, and Post_Test_Category (Table 1). The age of the participants and their grade level, the type of activity, the test score before the intervention of AR features, and the score of feedback were not considered in the research since they might only have minimal association with the learning outcomes of English vocabulary which can be ignored.

Meanwhile, data cleaning is conducted beforehand to provide higher accuracy and reliability of results by filtering out data of selected features with zero observations. For instance, the author considered it unreasonable to have the value of zero on the completion rate while data records are available for the variable of duration, engagement score, and final learning outcomes.

Furthermore, the author kept the original categorical outcomes in Post_Test_Category in the dataset of the Interactive AR group, while redefined the outcomes in Post_Test_Cateogry into two levels by combining “Low” and “Medium” into “Not High” in the dataset of the Animation group. The redefinition can avoid convergence failure in the model since only one participant is showing “Low” outcomes after the intervention of using animation.

2.2. Algorithm

2.2.1. Logistic regression

Logistic regression is commonly used in research when there are only two possible values for the response variable [12]. In this study, the response variable contains two values, “Not High” and “High”. Therefore, a binary logistic regression model is developed to examine the relationship between four independent variables and the post-test performance with the intervention of an animation feature. The logistic function that is used to fit the model in the logistic regression is

$$ p\left(X\right) = \frac{e^{{\beta }_0+{\beta }_i{X}_i}}{1+ e^{{\beta }_0+{\beta }_i{X}_i}}$$(1)

There will be an S-shaped curve produced by this logistic function (1). Compared to linear regression, the advantage of logistic regression is that it better captures the range of possibilities [13]. By taking the logarithm of both sides, the binary logistic regression model will be changed to

$log(\frac{p(X)}{1-p(X)})={\beta }_0+{\beta }_i{X}_i$(2)

The left-hand side of the equal sign is called “a logit the log of the odds that an event occurs” [14].

For the animation group, the author chose logistic regression to analyze and evaluate the most significant variable in vocabulary learning outcomes.

2.2.2. Ordinal logistic regression

Ordinal Logistic Regression is taken as the extension of binary logistic regression which is the special model of multinomial regression with the ordinal categorical response variable, and multinomial regression contains the categorical response that has more than two classes [12]. It is usually used to determine whether there is a statistically significant effect between independent and dependent variables or not [14]. For instance, the dataset of interactive AR has three categories of learning outcomes on English vocabulary: Low, Medium, and High. By applying ordinal logistic regression, the author aims to investigate the impacts of independent variables on learning outcomes for users who utilized Interactive AR for their language education.

3. Results and discussion

Table 2 shows the results of the ordinal logistic regression model for the interactive AR group to examine the relationship between the post-test performance and four independent variables. From Table 2, it is clear to understand that Duration is a statistically significant predictor for having better learning outcomes after the usage of interactive AR since the t-value is larger than 1.96, while the rate of completion does not have a significant effect on post-test performance. Meanwhile, in the animation group shown in Table 3, logistic regression has been applied since there are only two post-test outcomes, “Not High” and “High. Experimental results indicate that Duration is also a strong and significant predictor with positive impacts (p = 0.00327 < 0.05). In other words, spending more time learning English vocabulary via animation can lead to a higher possibility of better learning outcomes. Surprisingly, Accuracy plays a significant negative predictor in the Animation group where the z-value is -2.777 and the p-value is 0.00549 (p < 0.05), suggesting that students with higher accuracy on tasks are less likely to get a higher level in the post-test. However, the AIC value increased from approximately 59 to around 67 (Table 3 and Table 4) without considering Accuracy as one of the predictors, indicating a worse model fit. Hence, it can be concluded that Accuracy needs to be considered in the model.

To further understand the unusual results shown for the Accuracy predictor, the author conducted a Variance Inflation Factor (VIF) analysis and generated a marginal effects plot. All VIF values were below the conventional threshold of 5 (Table 5) which means all predictors contribute independently to the model, and the model did not have the multicollinearity issue. In addition, the marginal effects plot (Figure 1) demonstrated a clear downward trend, supporting the result that the accuracy of tasks during learning has negative impacts on learning outcomes. As the rate of accuracy increases, the probability of being classified as High for the post-test score on English vocabulary learning is lower.

On the one hand, the unusual result about the relationship between the accuracy of tasks and the learning outcome for the animation group might be impacted by speculation with a high accuracy rate made by students. However, it did not help users enhance their ability of vocabulary because students may just memorize the words without understanding their meaning. On the other hand, the author assumed that the extraordinary finding is possibly caused by oversimplified tasks. Although students are engaged in learning, they may be limited by opportunities for deep thinking because of simple tasks. Therefore, to investigate the specific reasons for the negative influence of accuracy on English learning, further research should be conducted. For instance, it is necessary to clarify the types of final tests and engage an AB test between different difficulties of tests.

In addition, despite lots of research indicating that animation and interactive AR both can enhance the efficiency of learning, one explanation for showing the opposite effects of the accuracy rate on learning performance in the group of users who utilized interactive AR is an essential difference between animation and interactive AR. High interactivity is one of the advantages of interactive AR that encourages students to have independent exploration and active learning. Interactive AR creates a unique and distinctive learning experience for each user by filling in the gaps of unknown knowledge and understanding knowledge in more detail [15]. Animation can be effective for introducing and helping students to understand complex concepts, but it lacks active interaction since the position, angle, and action in animation are designed by the animator.

Moreover, Duration has significant impacts in both groups, indicating that longer exposure to learning with interactive AR and animation can lead to a higher probability of showing better performance. Many prior studies evidenced the positive relationship between students who spend more time on learning using educational technologies and their test scores outcomes [16]. Hence, it can be concluded that learning time is an important factor in the study, no matter which digital tools they are using. The educators should consider providing enough space and time for students to learn in the future.

4. Conclusion

In this study, the author examined how four variables (Duration, Engagement_Score, Accuracy, and Completion_Rate) can affect the learning outcome of English vocabulary of students who utilized interactive AR and animation during the learning. The research utilized ordinal logistic regression for the interactive AR group and logistic regression for the animation group. The author found a positive statistically significant influence of the duration of students’ learning with both digital tools on the learning performance, while the accuracy of tasks in the animation group had negative effects on the learning outcomes. Meanwhile, the author indicates that accuracy is an important factor in the model because of the lower AIC value in the model with the variable of accuracy.

The author highlights the challenges of uncommon findings between accuracy and learning outcomes in the animation group. This challenge may be caused by oversimplified tasks and a high accuracy rate of guessing made by students. However, it is unclear about the specific reasons for this situation; thus, further exploration should pay attention to the difficulty of tasks and alleviate the effects caused by guessing the word's meaning during the test. It is also important to add a control group that has not used digital tools for the comparison in the future, which could provide more valuable insights into the effectiveness of integrating educational technologies into language education. Besides, a larger sample size is recommended in further research which could be beneficial in providing more reliable and accurate results.