1. Introduction

In recent years, the rapid development of educational technology worldwide has induced changes in teaching models, especially in K-12 education. In the teaching process, homework correction is a vital link. Zang, Cao, Zhou, and Zhang pointed out that when teachers correct a large amount of homework, they are easily affected by subjective factors such as fatigue, which will lead to a decrease in the efficiency and accuracy of the correction [1]. With the further integration of artificial intelligence (AI) into education, AI tools can enhance the student learning experience through personalized learning paths and real-time feedback, saving teachers and students a great deal of time [2]. However, despite the growing use of AI in education, the effectiveness of these tools in grading needs to be further explored [3]. Especially when dealing with complex subjective problems, the specific application effect still needs further discussion [4].

AI automatic grading systems generally fall into two broad categories: rule-based scoring systems and machine learning-based scoring systems. According to previous research, rule-based scoring systems rely on preset answer templates, are suitable for standardized questions, and perform poorly on complex mathematical subjective questions [5]. The scoring system based on machine learning can automatically generate assessment criteria by learning a large amount of homework data, analyzing students' problem-solving ideas, and dealing with complex problems with greater flexibility and accuracy [6]. Both MathGPTPro and Zuoyebang, are machine learning-based systems. The important concerns of this study are how effective they are in practical application and whether they can accurately identify error types.

This paper aims to assess the accuracy of machine learning-based AI scoring systems, MathGPTPro and Zuoyebang, in addressing high school mathematical function problems through experimental methodologies. Specific research inquiries include:

1. How accurate are the scores generated by MathGPTPro and Zuoyebang when addressing high school math function-solving problems? Which system exhibits higher accuracy? Can students' problem-solving steps be accurately assessed? Is there a discernible difference?

2. What are AI scoring systems' potential benefits and limitations in practical teaching scenarios?

This investigation not only provides valuable insights for future educational technology research and development, but also offers practical guidance for mathematics educators to effectively utilize AI tools in classroom instruction, while presenting enhanced ideas for software development designers.

2. Research Methodology

2.1. Sample Selection

2.1.1. Scoring system selection

MathGPTPro can handle relatively complex math topics using natural language processing and deep learning techniques [7]. Research has shown that it can provide accurate scoring and instant feedback when dealing with logical reasoning and multi-step problem solving, and is a highly intelligent AI software focused on assessing math problems [8]. Zuoyebang is widely used in China's basic education. It is the only software in China that can do homework correction for all subjects in high school, with relatively mature AI technology and a large stock of questions [9]. Both AI systems support identifying problem-solving steps, final answers, ensuring sufficient grading dimensions for analysis and comparison.

Both softwares are based on machine learning scoring algorithms, and when dealing with subjective questions, both rely on big data training and deep learning models to make scoring decisions, and build up the ability to recognize solving patterns by training a massive math question bank. MathGPTPro scores questions based on the reasonableness and logic of the solving steps. Zuoyebang scores by analyzing students' solution steps and comparing answers. They both use NLP technology to parse students' problem-solving processes. The similarity of core algorithms ensures the comparability of the system's scoring logic.

2.1.2. Selection of Question Samples

The experimental method will be used to collect the national one-volume college entrance examination papers of the past fifteen years and select the function answer questions from them to ensure that the sample is representative. Thirty questions will be collected to ensure the statistical significance of the results.

For selecting function questions, it is more suitable for assessing AI's logical and analytical ability in problem-solving, for example, a question in which students need to use multiple combinations of knowledge points such as the definition and value domains of a function, monotonicity of a function and so on. Function questions have multiple steps and go on to assess the AI scoring system's recognition of steps. In addition, the questions involve many symbols, which can add dimension to assess the AI system. Function questions are also mandatory for many questions in the entrance exam every year.

2.2. Experimental Variables

• Independent variables: AI scoring systems (MathGPTPro and Zuoyebang), function problems.

• Dependent variables: Scoring accuracy (accuracy, precision, recall, F1 score)

2.3. Data Collection and Processing

First, the confusion matrix was utilized to record the scoring results of each question in the two AI systems. The confusion matrix includes:

• TP: recognized correct answer

• TN: recognized wrong answer

• FP: the number of times the wrong answer was judged by the correct answer

• FN: the number of times the correct answer was judged as the wrong answer

Secondly, according to the high school mathematics scoring rules, four scoring dimensions, namely, logic and steps, final answer correctness, expression and notation, and error analysis and feedback, were selected to be recorded for each question, as Table 1 shows.

Table 1: On the development of accuracy scoring rules.

The scoring of the AI system was recorded in detail based on the four scoring dimensions mentioned above. A scale of 1-5 is set for each dimension and scores are given for each dimension, as Table 2 shows.

Table 2: Scoring rules for each dimension.

2.4. Data Analysis

2.4.1. Accuracy Analysis

First, the scores of the four scoring dimensions for each question were weighted by the weighting coefficients to produce a weighted total score that reflects the importance of different dimensions and avoids scoring bias. Further, the accuracy details such as correct identification, incorrect identification, and omission are analyzed using a confusion matrix.

The following key metrics are calculated through the confusion matrix:

• Accuracy: the overall correctness of the system’s scoring.

• Precision: The percentage of scores predicted by the system to be “excellent answers.”

• Recall: The proportion of excellent answers correctly identified by the system.

• F1 Score: The average of the precision rate and the recall rate.

2.4.2. Analysis of Variance

A t-test was used to compare the difference in performance between MathGPTPro and Zuoyebang in terms of scoring accuracy. This is done as follows:

• Hypothesis formulation:

Accuracy Hypothesis: H0 (Original Hypothesis): There is no significant difference in scoring accuracy between MathGPTPro and Zuoyebang.

H1 (Alternative Hypothesis): The scoring accuracy of MathGPTPro is significantly higher than that of Zuoyebang.

• Calculate the T-test:

First, calculate each AI system’s average weighted mean over multiple topics.

Second, the standard deviation of the scoring accuracy of each AI system is calculated to measure the volatility of the accuracy.

Then, the t-test formula was used, and the p-value was calculated. When the p-value was less than 0.05, it was considered that there was a significant difference between the two groups.

3. Results

3.1. Results of Weighted Scoring

According to the criteria developed in the methodology, the formula for calculating the total weighted score is as follows: total weighted score = (Logic and Steps Score × 0.40) + (Correctness of Final Answer Score × 0.30) + (Expression and Symbol Score × 0.20) + (Error Analysis and Feedback Score × 0.10).

The weighted total scores of MathGPTPro and Zuoyebang Learning Machine after testing on 30 subjective math questions are shown in Table 3 below:

Table 3: Calculation Result Chart.

Based on the above data, the t-test calculates a t-value of:

\( t=\frac{4.12−3.39}{\sqrt{\frac{{0.76}^2}{30}+\frac{{1.04}^2}{30}}}= \) 3.11 (1)

The corresponding p-value of 0.0025 was obtained, and since the p-value is less than 0.05, we reject the H0 and accept the H1. That is, MathGPTPro is significantly more accurate in scoring than Zuoyebang. MathGPTPro is significantly higher than Zuoyebang Learning Machine in terms of the average weighted total score, which indicates that the overall score of MathGPTPro is better than that of Zuoyebang Learning Machine in several dimensions, such as Logic and Steps, Correctness of the Final Answer, Expression and Symbolism, and Error Analysis and Feedback.

3.2. Accuracy Analysis

Based on the average criteria shown in the data, a weighted score of ≥ 4.0 was set as an excellent answer, a weighted total score between 3.0 and 4.0 as a qualified answer, and a weighted total score of < 3.0 as a failed answer. The confusion matrix of the two systems can be calculated and compared. The confusion matrix for MathGPTpro is shown in Table 4, and the confusion matrix for the Zuoyebang is shown in Table 5.

Table 4: MathGPTPro's confusion matrix.

Table 5: Zuoyebang's confusion matrix.

Further calculating the values of the MathGPTPro.

Through the formula:

\( Accuracy=\frac{TP+TN}{TP+TN+FP+FN} \) (2)

The accuracy is equal to 83.33%.

Through the formula:

\( Precision=\frac{TP}{TP+FP} \) (3)

The precision is equal to 87.5%.

Through the formula:

\( Recall=\frac{TP}{TP+FN} \) (4)

The recall is equal to 91.3%.

Through the formula:

\( F1=2\times\frac{Precision\ast R e c a l l}{Precision+Recall} \) (5)

The F1 Score is equal to 89.36%.

The values for each of the Zuoyebang, calculated by the above formula, were 70.0% accuracy, 77.27% precision, 80.95% recall, and 79.07% F1 score. The results show the overall scoring correctness of the MathGPTPro system. A higher percentage of predicted and actual excellent answers were correctly recognized by the system, and the reconciled mean of precision and recall was higher. This indicates that it performs better than the Zuoyebang learning machine in terms of accuracy, precision, recall, and F1 score.

3.3. Feedback Quality Analysis

Through data analysis, combined with weighted scoring and accuracy analysis results, MathGPTPro outperforms the Zuoyebang learning machine in handling subjective high school math problems. Error and feedback data indicate that MathGPTPro provides more specific and constructive feedback, effectively assisting students in correcting thinking errors and incorrect expressions during problem-solving processes. Conversely, Zuoyeabng learning machine's feedback is more standardized, accurately displaying final results without analyzing process steps or providing overall result feedback.

4. Discussion

The average weighted total score of MathGPTPro and Zuoyebang exceeds 4.0, while the average weighted total score of Zuoyebang is lower than 4.0, when dealing with function Problems of Standard Difficulty from the National College Entrance Examination (Gaokao) Mathematics Paper. This shows that MathGPTPro is highly able to identify correct answers and give accurate scores, but the applicability of the Zuoyebang needs further consideration.

MathGPTPro also has a definite advantage over Zuoyebang in dealing with complex problems. The reason may be that MathGPTPro uses more advanced natural language processing and deep learning techniques to understand better and analyze the student's problem-solving process.

5. Conclusion

MathGPTPro has great potential in practical teaching, especially in reducing the burden on teachers and providing timely correction to students. However, there may still be misjudgment or insufficient feedback on some highly subjective topics and require high detail control. Although AI systems can efficiently handle most mathematical subjective problems, whether they can effectively replace manual feedback in the face of open-ended problems needs further consideration. AI may have specific limitations in actual teaching. For example, the analysis of Zuoyebang data showed that some AI systems still could not fully understand the complex solution process, and could not properly deal with inconsistencies in symbols and expressions and give targeted feedback. AI flexibility must be enhanced, especially when teacher experience and judgment are required.

Based on the overall study, the author can analyze its limitations and future direction. This research focuses on function questions, especially Chinese college entrance examination questions. The application of the findings needs to be extended to other areas of mathematics or non-Chinese education systems. Comprehensively evaluate other question types such as inequalities, probabilities. In addition, the scoring process of the AI scoring system needs to be more transparent, and the reasoning behind the system's scoring needs explanation. Future research should explore ways to make the scoring process more transparent and explainable. Longitudinal studies on students' learning effects and teachers' experience of using AI can be studied, and the help of AI feedback to students' problem-solving can be investigated from students' perspective. From teachers' perspective, explore how to combine AI with manual labor better to bring more effective grading results.