Download pdf
Evaluating ChatGPT's Chinese-English Translation Quality of Tender Documents: A Research of MTPE with MQM Scoring Models
- 1 Beijing Normal University
* Author to whom correspondence should be addressed.
Abstract
Since Google introduced the Transformer model into natural language processing (NLP) in 2017, AI-aided translation has rapidly advanced. At the same time, translation is evolving from a solitary endeavor into a cooperative activity between human translators and machine translation systems, epitomized by the emergency of platforms with the Machine Translation Post Editing (MTPE) function. The advent of new translation modes also leads to increased research evaluating the effectiveness and quality of machine translation, for example, studies on the translation quality under the Multidimensional Quality Metrics (MQM) error typology framework. Involving AI-based translators and MTPE in their translation enables human translators to prepare the engineering documents efficiently. However, researchers notice that it is difficult for most machine translators to figure out the semantic and cultural differences in the source language and generate coherent structural translation in the target language. This research opens up ChatGPT’s application in tender document translation under the MQM framework, hoping to cast light on assessment on ChatGPT's translation quality, identification of ChatGPT's errors in translating such documents and suggestions on human translators' performance throughout MTPE.
Keywords
Machine Translation, Translation Error Typology, MTPE, MQM
[1]. Soydaner, Derya. "Attention mechanism in neural networks: where it comes and where it goes." Neural Computing and Applications 34.16 (2022): 13371–85.
[2]. Vieira, Lucas Nunes, Minako O'Hagan, and Carol O'Sullivan. "Understanding the societal impacts of machine translation: a critical review of the literature on medical and legal use cases." Information, Communication & Society 24.11 (2021): 1515–32.
[3]. Lommel, Arle, et al. "The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control." arXiv preprint arXiv:2405.16969, 2024.
[4]. Yang, Yanxia, et al. "Performance and perception: machine translation post-editing in Chinese-English news translation by novice translators." Humanities and Social Sciences Communications 10.1 (2023): 1–8.
[5]. Gao, Yuan, Ruili Wang, and Feng Hou. "How to design translation prompts for ChatGPT: An empirical study." arXiv preprint arXiv:2304.02182, 2023.
[6]. Gao, Ruiyao, et al. "Machine translation of Chinese classical poetry: a comparison among ChatGPT, Google Translate, and DeepL Translator." Humanities and Social Sciences Communications 11.1 (2024): 1–10.
[7]. Liu, Huey-Ing, and Wei-Lin Chen. "X-transformer: a machine translation model enhanced by the self-attention mechanism." Applied Sciences 12.9 (2022): 4502.
[8]. Ren, Jinghui. "Mining Lexical Co-occurrence Features Based Research On Construction Engineering English." IOP Conference Series: Earth and Environmental Science. Vol. 267. No. 6. IOP Publishing, 2019.
[9]. Gou, Miao. "Syntactic Features and Translation Strategies of English for Construction: A Review of English for Construction." Building Structure, vol. 51, no. 14, 2021, pp. Appendix 4–5 (in Chinese).
[10]. Wang, Haifeng, et al. "Progress in machine translation." Engineering 18 (2022): 143–53.
[11]. Bahdanau, Dzmitry. "Neural machine translation by jointly learning to align and translate." arXiv preprint arXiv:1409.0473, 2014.
[12]. Wang, Shanshan, and Xiaohui Wang. "Cross-Language Translation and Comparative Study of English Literature Using Machine Translation Algorithms." Journal of Electrical Systems 20.6s (2024): 2221–29.
[13]. Feng, Zhiwei. "Term Translation in Era of Large Language Models." China Terminology, vol. 26, no. 3, 2024, p. 93 (in Chinese).
[14]. Mossop, Brian. "Lynne Bowker and Jairo Buitrago Ciro. Machine Translation and Global Research: Towards Improved Machine Translation Literacy in the Scholarly Community. Bingley, Emerald Publishing, 2019, 111 p." TTR: traduction, terminologie, rédaction 32.2 (2019): 250–55.
[15]. Yang, Yanxia, and Xiangling Wang. "Predicting student translators' performance in machine translation post-editing: interplay of self-regulation, critical thinking, and motivation." Interactive Learning Environments 31.1 (2023): 340–54.
[16]. Pang, Yaxin, and Xiaoyan Wang. "A Study on the Translation Quality of ChatGPT in the Context of Large Language Model." Modern English, vol. 22, 2023 (in Chinese).
[17]. Al-Khalifa, Hend, Khaloud Al-Khalefah, and Hesham Haroon. "Error Analysis of Pretrained Language Models (PLMs) in English-to-Arabic Machine Translation." Human-Centric Intelligent Systems, 2024: 1–14.
[18]. Popović, Maja. "Error classification and analysis for machine translation quality assessment." Translation Quality Assessment: From Principles to Practice, 2018: 129–58.
[19]. Tsuji, Kayo. "Identifying MT Errors for Higher-Quality Second Language Text." International Journal of Translation, Interpretation, and Applied Linguistics (IJTIAL) 6.1 (2024): 1–12.
Cite this article
Zhou,L. (2025). Evaluating ChatGPT's Chinese-English Translation Quality of Tender Documents: A Research of MTPE with MQM Scoring Models. Communications in Humanities Research,61,53-62.
Data availability
The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.
Disclaimer/Publisher's Note
The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of EWA Publishing and/or the editor(s). EWA Publishing and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
About volume
Volume title: Proceedings of the 4th International Conference on Literature, Language, and Culture Development
© 2024 by the author(s). Licensee EWA Publishing, Oxford, UK. This article is an open access article distributed under the terms and
conditions of the Creative Commons Attribution (CC BY) license. Authors who
publish this series agree to the following terms:
1. Authors retain copyright and grant the series right of first publication with the work simultaneously licensed under a Creative Commons
Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this
series.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the series's published
version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial
publication in this series.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and
during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See
Open access policy for details).