Evaluation of character creation of large language models

Yuhui Wang; An Yang; Yifan Zhou; Chao Yao; Xiuxiu Sun; Wandi Sun

doi:10.54254/2977-3903/2025.20514

Research Article

Open access

Published on 15 January 2025

Download pdf

Wang,Y.;Yang,A.;Zhou,Y.;Yao,C.;Sun,X.;Sun,W. (2025). Evaluation of character creation of large language models. Advances in Engineering Innovation,15,14-20.

Export citation

Evaluation of character creation of large language models

Yuhui Wang *^,1, An Yang ², Yifan Zhou ³, Chao Yao ⁴, Xiuxiu Sun ⁵, Wandi Sun ⁶

¹ Academy of Cyber
² Academy of Cyber
³ Academy of Cyber
⁴ Academy of Cyber
⁵ Academy of Cyber
⁶ Academy of Cyber

* Author to whom correspondence should be addressed.

https://doi.org/10.54254/2977-3903/2025.20514

Abstract

In intelligent scenarios, large language models (LLMs) are used to create characters that interact with users, providing guidance and relevant information. The higher the degree of anthropomorphism of these roles, the better the emotional experience they provide, which is beneficial for user interaction and enhances user experience. Therefore, evaluating the character-creation capabilities of LLMs is essential. This study used a questionnaire and used another LLM (ChatGPT-4o) to assess the impact of emoji usage and language style on the anthropomorphism and emotional expression of content generated by LLMs. The results indicate that when using emojis, the characters exhibit higher levels of anthropomorphism and emotional expression. Additionally, informal language styles contribute to enhancing both anthropomorphism and emotional expression.

Keywords

large language model, anthropomorphism, emotional expression, questionnaire

View pdf

References

[1]. Tseng, Y. M., Huang, Y. C., Hsiao, T. Y., Hsu, Y. C., Foo, J. Y., Huang, C. W., & Chen, Y. N. (2024). Two tales of persona in llms: A survey of role-playing and personalization. arXiv preprint arXiv:2406.01171.

[2]. Wu, W., Wu, H., Jiang, L., Liu, X., Hong, J., Zhao, H., & Zhang, M. (2024). From Role-Play to Drama-Interaction: An LLM Solution. arXiv preprint arXiv:2405.14231.

[3]. Zhang, W., Deng, Y., Liu, B., Pan, S. J., & Bing, L. (2023). Sentiment analysis in the era of large language models: A reality check. arXiv preprint arXiv:2305.15005.

[4]. Ugare, S., Suresh, T., Kang, H., Misailovic, S., & Singh, G. (2024). Improving llm code generation with grammar augmentation. arXiv e-prints, arXiv-2403.

[5]. Nandkumar, C., & Peternel, L. (2024). Enhancing Supermarket Robot Interaction: A Multi-Level LLM Conversational Interface for Handling Diverse Customer Intents. arXiv preprint arXiv:2406.11047.

[6]. Dalal, M., Chiruvolu, T., Chaplot, D., & Salakhutdinov, R. (2024). Plan-seq-learn: Language model guided rl for solving long horizon robotics tasks. arXiv preprint arXiv:2405.01534.

[7]. Li, J., Mehrabi, N., Peris, C., Goyal, P., Chang, K. W., Galstyan, A., ... & Gupta, R. (2023). On the steerability of large language models toward data-driven personas. arXiv preprint arXiv:2311.04978.

[8]. Choi, H. K., & Li, Y. PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learning. In Forty-first International Conference on Machine Learning.

[9]. Shanahan, M., McDonell, K., & Reynolds, L. (2023). Role play with large language models. Nature, 623(7987), 493-498.

[10]. Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., ... & Zhang, X. (2024). Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680.

[11]. Poria, S., Hazarika, D., Majumder, N., & Mihalcea, R. (2020). Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research. IEEE transactions on affective computing, 14(1), 108-132.

[12]. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., ... & Zhang, Y. (2023). Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.

[13]. Wang, Y., & Luo, Z. (2023, December). Enhance multi-domain sentiment analysis of review texts through prompting strategies. In 2023 International Conference on High Performance Big Data and Intelligent Systems (HDIS) (pp. 1–7). IEEE.

[14]. Samuel, V., Zou, H. P., Zhou, Y., Chaudhari, S., Kalyan, A., Rajpurohit, T., ... & Murahari, V. (2024). Personagym: Evaluating persona agents and llms. arXiv preprint arXiv:2407.18416.

[15]. Shao, Y., Li, L., Dai, J., & Qiu, X. (2023). Character-llm: A trainable agent for role-playing. arXiv preprint arXiv:2310.10158.

[16]. Zhou, J., Chen, Z., Wan, D., Wen, B., Song, Y., Yu, J., ... & Huang, M. (2023). Characterglm: Customizing chinese conversational ai characters with large language models. arXiv preprint arXiv:2311.16832.

[17]. Li, C., Zhang, M., Mei, Q., Wang, Y., Hombaiah, S. A., Liang, Y., & Bendersky, M. (2023). Teach LLMs to Personalize--An Approach inspired by Writing Education. arXiv preprint arXiv:2308.07968.

[18]. Li, C., Leng, Z., Yan, C., Shen, J., Wang, H., Mi, W., ... & Sun, H. (2023). Chatharuhi: Reviving anime character in reality via large language model. arXiv preprint arXiv:2308.09597.

[19]. Al-Thubaity, A., Alkhereyf, S., Murayshid, H., Alshalawi, N., Omirah, M., Alateeq, R., ... & Alkhanen, I. (2023, December). Evaluating ChatGPT and bard AI on Arabic sentiment analysis. In Proceedings of ArabicNLP 2023 (pp. 335–349).

[20]. Zhang, W., Deng, Y., Liu, B., Pan, S. J., & Bing, L. (2023). Sentiment analysis in the era of large language models: A reality check. arXiv preprint arXiv:2305.15005.

[21]. Huang, S., Mamidanna, S., Jangam, S., Zhou, Y., & Gilpin, L. H. (2023). Can large language models explain themselves? a study of llm-generated self-explanations. arXiv preprint arXiv:2310.11207.

[22]. Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology.

[23]. Edwards, A. W. (2005). RA Fischer, statistical methods for research workers, (1925). In Landmark writings in western mathematics 1640–1940 (pp. 856–870). Elsevier Science.

Cite this article

Wang,Y.;Yang,A.;Zhou,Y.;Yao,C.;Sun,X.;Sun,W. (2025). Evaluation of character creation of large language models. Advances in Engineering Innovation,15,14-20.

Data availability

The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.

Disclaimer/Publisher's Note

The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of EWA Publishing and/or the editor(s). EWA Publishing and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

About volume

Journal：Advances in Engineering Innovation

Volume number: Vol.15

ISSN：2977-3903(Print) / 2977-3911(Online)

© 2024 by the author(s). Licensee EWA Publishing, Oxford, UK. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license. Authors who publish this series agree to the following terms:
1. Authors retain copyright and grant the series right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this series.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the series's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this series.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See Open access policy for details).