Translation from spoken Arabic digits to sign language based on deep learning approach

Research Article
Open access

Translation from spoken Arabic digits to sign language based on deep learning approach

Mothanna Almahmood 1 , Sayel Abualigah 2 , Rehab M. Duwairi 3 , Laith Abualigah 4* , Raed Abu Zitar 5 , Anas Ratib Alsoud 6 , Sathishkumar V. E. 7
  • 1 Jordan University of Science and Technology    
  • 2 Jordan University of Science and Technology    
  • 3 Jordan University of Science and Technology    
  • 4 Al-Ahliyya Amman University    
  • 5 Sorbonne University-Abu Dhabi    
  • 6 Al-Ahliyya Amman University    
  • 7 Jeobuk National University    
  • *corresponding author aligah.2020@gmail.com and sathish@jbnu.ac.kr
Published on 31 January 2024 | https://doi.org/10.54254/2755-2721/30/20230064
ACE Vol.30
ISSN (Print): 2755-273X
ISSN (Online): 2755-2721
ISBN (Print): 978-1-83558-285-5
ISBN (Online): 978-1-83558-286-2

Abstract

Deaf-and-dumb humans make up about 5% of the world's population, and they need special care by providing them alternative methods that help them to communicate with the outside world, whereas the sense of hearing is the main element of human communications, which is indispensable. From the standpoint of introducing helpful applications that help deaf-and-dumb population, the idea of this research aimed used deep learning techniques to create a model based on the principle of converting Arabic spoken digits to sign language images, through a study of two different datasets that were freely taken from open-source websites. The first one contains audio records of Arabic spoken digits that was used to train on-dimensional CNN model to generate a text translation of any Arabic spoken digit record. The second one contains sign language images of Arabic digits, where used to build IF-THEN rules system that can generate the sign language image as a translation of given Arabic digit text. The whole idea conducted through using both systems in one prediction model that can generate the sign language image of any giving spoken Arabic digits’ record, where it had accurate results with 86.85% accuracy value and 0.5039 loss value. The goal of this research is to add a new technology based on deep learning, in order to help this group of people with a simple idea that opens the researchers’ minds to produce a model of all Arabic spoken speech, which in turn can be a complete technology that helps deaf-and-dumb humans’ to easily communicate with the outside world.

Keywords:

one-dimensional CNN, sign language, spoken Arabic digits; deep learning

Almahmood,M.;Abualigah,S.;Duwairi,R.M.;Abualigah,L.;Zitar,R.A.;Alsoud,A.R.;E.,S.V. (2024). Translation from spoken Arabic digits to sign language based on deep learning approach. Applied and Computational Engineering,30,25-31.
Export citation

References

[1]. T. Tinsley and K. Board, “Languages for The Future,” Br. Counc., 2013.

[2]. “Arabic - Wikipedia” [Online]. Available: https://en.wikipedia.org/wiki/Arabic/ [Accessed: 12-May-2020].

[3]. W. Tin, Z. Lin, - Swe, and N. K. Mya, “Deaf mute or Deaf,” Asian J. Med. Biol. Res., vol. 3, no. 1, pp. 10–19, 2017.

[4]. M. M. El-Gayyar, A. S. Ibrahim, and M. E. Wahed, “Translation from Arabic speech to Arabic Sign Language based on cloud computing,” Egypt. Informatics J., vol. 17, no. 3, pp. 295–303, 2016.

[5]. K. Yousaf et al., “A Novel Technique for Speech Recognition and Visualization Based Mobile Application to Support Two-Way Communication between Deaf-Mute and Normal Peoples,” Wirel. Commun. Mob. Comput., vol. 2018, pp. 1–12, 2018.

[6]. L., H., & M., S., A., Automatic translation of Arabic text-to-Arabic sign language. Universal Access in the Information Society, 2019, 18(4), 939–951.

[7]. D., P., Hermawan Nugroho Centre for Intelligent Signal and Imaging Research Universiti Teknologi PETRONAS, Bandar Sri Iskandar Perak. 2015, 1–5.

[8]. K. El-darymli, O. O. Khalifa, H. Enemosah, K. Lumpur, and K. K. El-darymli, “The citation of this paper is as follows: Khalid El-Darymli , Othman O . Khalifa, and Hassan Enemosah , " Speech to Sign Language Speech to Sign Language Interpreter System ( SSLIS ),” no. May, 2006.

[9]. A. S. Mahfoudh Ba Wazir and J. Huang Chuah, “Spoken Arabic Digits Recognition Using Deep Learning,” 2019 IEEE Int. Conf. Autom. Control Intell. Syst. I2CACIS 2019 - Proc., no. June, pp. 339–344, 2019.

[10]. “The Arabic Speech Corpus for Isolated Words” [Online]. Available: http://www.cs.stir.ac.uk/~lss/arabic/ [Accessed: 12-May-2020].

[11]. “Numbers in sign language” [Online]. Available: http://easyenglishforme.blogspot.com/2013/03/numbers-in-sign-language.html. [Accessed: 12-May-2020].

[12]. B. McFee et al., “librosa: Audio and Music Signal Analysis in Python,” Proc. 14th Python Sci. Conf., no. Scipy, pp. 18–24, 2015.

[13]. S. Abdoli, P. Cardinal, and A. Lameiras Koerich, “End-to-end environmental sound classification using a 1D convolutional neural network,” Expert Syst. Appl., vol. 136, no. September, pp. 252–263, 2019.

[14]. A. Kipnis, A. J. Goldsmith, and Y. C. Eldar, “Optimal trade-off between sampling rate and quantization precision in Sigma-Delta A/D conversion,” 2015 Int. Conf. Sampl. Theory Appl. SampTA 2015, pp. 627–631, 2015.

[15]. “Label Encoder” [Online]. Available: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html [Accessed: 12-May-2020].

[16]. S. Kiranyaz, O. Avci, O. Abdeljaber, T. Ince, M. Gabbouj, and D. J. Inman, “1D Convolutional Neural Networks and Applications: A Survey,” pp. 1–20, 2019.

[17]. F. Emmert-Streib, Z. Yang, H. Feng, S. Tripathi, and M. Dehmer, “An Introductory Review of Deep Learning for Prediction Models with Big Data,” Front. Artif. Intell., vol. 3, no. February, pp. 1–23, 2020.

[18]. G. Montavon et al., “Neural Networks: Tricks of the Trade,” Springer Lect. Notes Comput. Sci., no. MAY 2000, p. 432, 2012.

[19]. H. Chen, S. Lundberg, and S.-I. Lee, “Checkpoint Ensembles: Ensemble Methods from a Single Training Process,” no. October 2017, 2017.

Cite this article

Almahmood,M.;Abualigah,S.;Duwairi,R.M.;Abualigah,L.;Zitar,R.A.;Alsoud,A.R.;E.,S.V. (2024). Translation from spoken Arabic digits to sign language based on deep learning approach. Applied and Computational Engineering,30,25-31.

Data availability

The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.

Disclaimer/Publisher's Note

The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of EWA Publishing and/or the editor(s). EWA Publishing and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

About volume

Volume title: Proceedings of the 2023 International Conference on Machine Learning and Automation

ISBN：978-1-83558-285-5(Print) / 978-1-83558-286-2(Online)
Editor：Mustafa İSTANBULLU
Conference website: https://2023.confmla.org/
Conference date: 18 October 2023
Series: Applied and Computational Engineering
Volume number: Vol.30
ISSN：2755-2721(Print) / 2755-273X(Online)

© 2024 by the author(s). Licensee EWA Publishing, Oxford, UK. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license. Authors who publish this series agree to the following terms:
1. Authors retain copyright and grant the series right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this series.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the series's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this series.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See Open access policy for details).

References

[1]. T. Tinsley and K. Board, “Languages for The Future,” Br. Counc., 2013.

[2]. “Arabic - Wikipedia” [Online]. Available: https://en.wikipedia.org/wiki/Arabic/ [Accessed: 12-May-2020].

[3]. W. Tin, Z. Lin, - Swe, and N. K. Mya, “Deaf mute or Deaf,” Asian J. Med. Biol. Res., vol. 3, no. 1, pp. 10–19, 2017.

[4]. M. M. El-Gayyar, A. S. Ibrahim, and M. E. Wahed, “Translation from Arabic speech to Arabic Sign Language based on cloud computing,” Egypt. Informatics J., vol. 17, no. 3, pp. 295–303, 2016.

[5]. K. Yousaf et al., “A Novel Technique for Speech Recognition and Visualization Based Mobile Application to Support Two-Way Communication between Deaf-Mute and Normal Peoples,” Wirel. Commun. Mob. Comput., vol. 2018, pp. 1–12, 2018.

[6]. L., H., & M., S., A., Automatic translation of Arabic text-to-Arabic sign language. Universal Access in the Information Society, 2019, 18(4), 939–951.

[7]. D., P., Hermawan Nugroho Centre for Intelligent Signal and Imaging Research Universiti Teknologi PETRONAS, Bandar Sri Iskandar Perak. 2015, 1–5.

[8]. K. El-darymli, O. O. Khalifa, H. Enemosah, K. Lumpur, and K. K. El-darymli, “The citation of this paper is as follows: Khalid El-Darymli , Othman O . Khalifa, and Hassan Enemosah , " Speech to Sign Language Speech to Sign Language Interpreter System ( SSLIS ),” no. May, 2006.

[9]. A. S. Mahfoudh Ba Wazir and J. Huang Chuah, “Spoken Arabic Digits Recognition Using Deep Learning,” 2019 IEEE Int. Conf. Autom. Control Intell. Syst. I2CACIS 2019 - Proc., no. June, pp. 339–344, 2019.

[10]. “The Arabic Speech Corpus for Isolated Words” [Online]. Available: http://www.cs.stir.ac.uk/~lss/arabic/ [Accessed: 12-May-2020].

[11]. “Numbers in sign language” [Online]. Available: http://easyenglishforme.blogspot.com/2013/03/numbers-in-sign-language.html. [Accessed: 12-May-2020].

[12]. B. McFee et al., “librosa: Audio and Music Signal Analysis in Python,” Proc. 14th Python Sci. Conf., no. Scipy, pp. 18–24, 2015.

[13]. S. Abdoli, P. Cardinal, and A. Lameiras Koerich, “End-to-end environmental sound classification using a 1D convolutional neural network,” Expert Syst. Appl., vol. 136, no. September, pp. 252–263, 2019.

[14]. A. Kipnis, A. J. Goldsmith, and Y. C. Eldar, “Optimal trade-off between sampling rate and quantization precision in Sigma-Delta A/D conversion,” 2015 Int. Conf. Sampl. Theory Appl. SampTA 2015, pp. 627–631, 2015.

[15]. “Label Encoder” [Online]. Available: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html [Accessed: 12-May-2020].

[16]. S. Kiranyaz, O. Avci, O. Abdeljaber, T. Ince, M. Gabbouj, and D. J. Inman, “1D Convolutional Neural Networks and Applications: A Survey,” pp. 1–20, 2019.

[17]. F. Emmert-Streib, Z. Yang, H. Feng, S. Tripathi, and M. Dehmer, “An Introductory Review of Deep Learning for Prediction Models with Big Data,” Front. Artif. Intell., vol. 3, no. February, pp. 1–23, 2020.

[18]. G. Montavon et al., “Neural Networks: Tricks of the Trade,” Springer Lect. Notes Comput. Sci., no. MAY 2000, p. 432, 2012.

[19]. H. Chen, S. Lundberg, and S.-I. Lee, “Checkpoint Ensembles: Ensemble Methods from a Single Training Process,” no. October 2017, 2017.

For authors
Guide for authors Policies for authors Aims and scope Subscription Open access policy
For editors
Policies for editors Editorial board
For reviewers
Policies for reviewers Privacy policy

Cookies are used by this site.
Copyright © 2024 EWA Publishing, its licensors, and contributors. All rights reserved, including text and data mining, AI training, and similar technologies.