Sensory Feedback Improvement of BCI Robotics for Movement Control

Zekai Jiang; Haocheng Lyu; Yuecen Wan; Haochen Qian

doi:10.54254/2755-2721/2024.20548

Research Article

Open access

Published on 24 January 2025

Download pdf

Jiang,Z.;Lyu,H.;Wan,Y.;Qian,H. (2025). Sensory Feedback Improvement of BCI Robotics for Movement Control. Applied and Computational Engineering,131,85-97.

Export citation

Sensory Feedback Improvement of BCI Robotics for Movement Control

Zekai Jiang *^,1, Haocheng Lyu ², Yuecen Wan ³, Haochen Qian ⁴

¹ College of Science, Shanghai University, Shanghai, 200444, China
² Powder Metallurgy Institute, Central South University, Changsha, 410083, China
³ Wuhan No.2 High School, Wuhan, 430010, China
⁴ Shanghai Hongwen School, Shanghai, 200127, China

* Author to whom correspondence should be addressed.

https://doi.org/10.54254/2755-2721/2024.20548

Abstract

Brain-computer interfaces have great potential in motor control and rehabilitation. In related research fields, how to effectively monitor users has always been a research focus. Many studies have found that the performance of brain-computer interfaces can be effectively improved by improving and integrating feedback methods. This article reviews the four main types of feedback currently available, including visual feedback, auditory feedback, vibration, and electrical stimulation in tactile feedback, and introduces their principles and applications. This article summarizes the improvements in experimental accuracy and efficiency brought about by these sensory feedbacks in research and finally proposes limitations and future development trends.

Keywords

Sensory feedback, BCI, movement control

View pdf

References

[1]. Clerc, M. Brain Computer Interfaces, Principles and Practise. BioMed Eng OnLine 12, 22 (2013). https://doi.org/10.1186/1475-925X-12-22

[2]. Lim, J. H., Lee, J. H., Hwang, H. J., Kim, D. H., & Im, C. H. (2015). Development of a hybrid mental spelling system combining SSVEP-based brain-computer interface and webcam-based eye tracking. Biomedical Signal Processing and Control, 21, 99–104.https://doi.org/10.1016/j.bspc.2015.05.012

[3]. Millán JdR, Rupp R, Müller-Putz GR, Murray-Smith R, Giugliemma C, Tangermann M, Vidaurre C, Cincotti F, Kübler A, Leeb R, Neuper C, Müller K-R and Mattia D (2010) Combining brain–computer interfaces and assistive technologies: state-of-the-art and challenges. Front. Neurosci. 4:161. doi:10.3389/fnins.2010.00161

[4]. Nijboer, F., Furdea, A., Gunst, I., Mellinger, J., McFarland, D. J., Birbaumer, N., & Kübler, A. (2008). An auditory brain-computer interface (BCI). Journal of Neuroscience Methods, 167(1), 43–50. https://doi.org/10.1016/j.jneumeth.2007.02.009

[5]. Abdelrahman, Y., Bennington, M., Huberts, J., Sebt, S., Talwar, N., & Cauwenberghs, G. (2021). Sensory Substitution for Tactile Feedback in Upper Limb Prostheses. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 7519–7525. https://doi.org/10.1109/EMBC46164.2021.9629539

[6]. Halme, H. L., & Parkkonen, L. (2022). The effect of visual and proprioceptive feedback on sensorimotor rhythms during BCI training. PLOS ONE, 17(2), e0264354. https://doi.org/10.1371/JOURNAL.PONE.0264354

[7]. McCreadie, K. A., Coyle, D. H., & Prasad, G. (2014). Is sensorimotor BCI performance influenced differently by mono, stereo, or 3-D auditory feedback? IEEE Transactions on Neural Systems and Rehabilitation Engineering, 22(3), 431–440. https://doi.org/10.1109/TNSRE.2014.2312270

[8]. AdOscari, F., Secoli, R., Avanzini, F., Rosati, G., & Reinkensmeyer, D. J. (2012). Substituting auditory for visual feedback to adapt to altered dynamic and kinematic environments during reaching. Experimental Brain Research, 221(1), 33–41. https://doi.org/10.1007/s00221-012-3144-2

[9]. Moore, C. H., Corbin, S. F., Mayr, R., Shockley, K., Silva, P. L., & Lorenz, T. (2021). Grasping Embodiment: Haptic Feedback for Artificial Limbs. Frontiers in Neurorobotics, 15. https://doi.org/10.3389/fnbot.2021.662397

[10]. Guémann, M., Halgand, C., Bastier, A., Lansade, C., Borrini, L., Lapeyre, É., Cattaert, D., & de Rugy, A. (2022). Sensory substitution of elbow proprioception to improve myoelectric control of upper limb prosthesis: experiment on healthy subjects and amputees. Journal of NeuroEngineering and Rehabilitation, 19(1). https://doi.org/10.1186/s12984-022-01038-y

[11]. Zhu, B., Chu, Y., & Zhao, X. (2019). Sheng wu yi xue gong cheng xue za zhi = Journal of biomedical engineering = Shengwu yixue gongchengxue zazhi, 36(6), 1048–1054. https://doi.org/10.7507/1001-5515.201904064

[12]. Guemann, M., Bouvier, S., Halgand, C., Paclet, F., Borrini, L., Ricard, D., Lapeyre, E., Cattaert, D., & Rugy, A. (2019). Effect of vibration characteristics and vibror arrangement on the tactile perception of the upper arm in healthy subjects and upper limb amputees. Journal of neuroengineering and rehabilitation, 16(1), 138. https://doi.org/10.1186/s12984-019-0597-6

[13]. Markus Ploesser;Mickey Ellis Abraham; Marike Lianne Daphne Broekman;Miriam Tanja Zincke;Craig Aaron Beach;Nina Beatrix Urban;Sharona Ben-HaimStereotact Funct Neurosurg 308–324.https://doi.org/10.1159/000539757

[14]. Ong Sio, Lady Christine, Brian Hom, Shuchita Garg, and Alaa Abd-Elsayed. 2023. "Mechanism of Action of Peripheral Nerve Stimulation for Chronic Pain: A Narrative Review" International Journal of Molecular Sciences 24, no. 5: 4540. https://doi.org/10.3390/ijms24054540

[15]. Azeem, N., Attias, M.D. (2018). Neuromodulation: Mechanisms of Action. In: Diwan, S., Deer, T. (eds) Advanced Procedures for Pain Management. Springer, Cham. https://doi.org/10.1007/978-3-319-68841-1_8

[16]. Flesher, S. et al. (2017). Intracortical Microstimulation as a Feedback Source for Brain-Computer Interface Users. In: Guger, C., Allison, B., Lebedev, M. (eds) Brain-Computer Interface Research. SpringerBriefs in Electrical and Computer Engineering. Springer, Cham. https://doi.org/10.1007/978-3-319-64373-1_5

[17]. Birbaumer N, Cohen L G. Brain–computer interfaces: communication and restoration of movement in paralysis[J]. The Journal of Physiology, 2007, 579(3): 621-636. https://doi.org/10.1113/jphysiol.2006.125633

[18]. McFarland D.J., Wolpaw J.R. Sensorimotor rhythm-based brain-computer interface (BCI): model order selection for autoregressive spectral analysis[J]. Journal of Neural Engineering,2008,5(2):155-162. https://doi.org/10.1088/1741-2560/5/2/006

[19]. Fodor, M. A., Herschel, H., Cantürk, A., Heisenberg, G., & Volosyak, I. (2024). Evaluation of Different Visual Feedback Methods for Brain—Computer Interfaces (BCI) Based on Code-Modulated Visual Evoked Potentials (cVEP). Brain Sciences, 14(8). https://doi.org/10.3390/brainsci14080846

[20]. Blankertz B., Dornhege G., Krauledat M., Müller K.R., Curio G. The non-invasive Berlin Brain-Computer Interface: fast acquisition of effective performance in untrained subjects[J].NeuroImage,2007,37(2):539-550.https://doi.org/10.1016/j.neuroimage.2007.01.051

[21]. Fan, Z., Lin, C., & Fu, C. (2020). A Gaze Signal Based Control Method for Supernumerary Robotic Limbs. 2020 3rd International Conference on Control and Robots, ICCR 2020, 107–111. https://doi.org/10.1109/ICCR51572.2020.9344272

[22]. Wolpaw J.R., Wolpaw E.W. Brain-computer interfaces: principles and practice[M]. Oxford University Press,2012. https://doi.org/10.1016/B978-0-444-52901-5.00006-X

[23]. Pinardi, M., Longo, M. R., Formica, D., Strbac, M., Mehring, C., Burdet, E., & di Pino, G. (2023). Impact of supplementary sensory feedback on the control and embodiment in human movement augmentation. Communications Engineering, 2(1), 64. https://doi.org/10.1038/s44172-023-00111-1

[24]. Neuper C., Müller-Putz G.R., Scherer R., Pfurtscheller G. Motor imagery and EEG-based control of spelling devices and neuroprostheses[J]. Progress in Brain Research,2006,159:393-409. https://doi.org/10.1016/S0079-6123(06)59025-9

[25]. Pfurtscheller G., Neuper C. Motor imagery and direct brain-computer communication[J]. Proceedings of the IEEE,2001,89(7):1123-1134. https://doi.org/10.1109/5.939829

[26]. Girbes-Juan, V., Schettino, V., Demiris, Y., & Tornero, J. (2021). Haptic and Visual Feedback Assistance for Dual-Arm Robot Teleoperation in Surface Conditioning Tasks. IEEE Transactions on Haptics, 14(1), 44–56. https://doi.org/10.1109/TOH.2020.3004388

[27]. Koritnik, T., Koenig, A., Bajd, T., Riener, R., & Munih, M. (2010). Comparison of visual and haptic feedback during training of lower extremities. Gait and Posture, 32(4), 540–546. https://doi.org/10.1016/j.gaitpost.2010.07.017

[28]. Sigrist, R., Rauter, G., Marchal-Crespo, L., Riener, R., & Wolf, P. (2015). Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning. Experimental Brain Research, 233(3), 909–925. https://doi.org/10.1007/s00221-014-4167-7

[29]. Cutler, N., Balicki, M., Finkelstein, M., Wang, J., Gehlbach, P., McGready, J., Iordachita, I., Taylor, R., & Handa, J. T. (2013). Auditory force feedback substitution improves surgical precision during simulated ophthalmic surgery. Investigative Ophthalmology and Visual Science, 54(2), 1316–1324. https://doi.org/10.1167/iovs.12-11136

[30]. Sabes P. N. (2011). Sensory integration for reaching: models of optimality in the context of behavior and the underlying neural circuits. Progress in brain research, 191, 195–209. https://doi.org/10.1016/B978-0-444-53752-2.00004-7

[31]. Raveh, E., Portnoy, S., & Friedman, J. (2018). Myoelectric Prosthesis Users Improve Performance Time and Accuracy Using Vibrotactile Feedback When Visual Feedback Is Disturbed. Archives of physical medicine and rehabilitation, 99(11), 2263–2270. https://doi.org/10.1016/j.apmr.2018.05.019

[32]. Guemann, M., Bouvier, S., Halgand, C., Paclet, F., Borrini, L., Ricard, D., Lapeyre, E., Cattaert, D., & Rugy, A. (2019). Effect of vibration characteristics and vibror arrangement on the tactile perception of the upper arm in healthy subjects and upper limb amputees. Journal of neuroengineering and rehabilitation, 16(1), 138. https://doi.org/10.1186/s12984-019-0597-6

[33]. Brunner, I., Lundquist, C. B., Pedersen, A. R., Spaich, E. G., Dosen, S., & Savic, A. (2024). Brain computer interface training with motor imagery and functional electrical stimulation for patients with severe upper limb paresis after stroke: a randomized controlled pilot trial. Journal of NeuroEngineering and Rehabilitation, 21(1). https://doi.org/10.1186/s12984-024-01304-1(2024)

[34]. Jones, Matt., Schmidt, Albrecht., Palanque, Philippe., & Grossman, Tovi. (2014). A Visual Feedback Design based on a Brain-Computer Interface to Assist Users Regulate their Emotional State. Association for Computing Machinery.

[35]. Gonzalez, J., Soma, H., Sekine, M., & Yu, W. (2012). Psycho-physiological assessment of a prosthetic hand sensory feedback system based on an auditory display: A preliminary study. Journal of NeuroEngineering and Rehabilitation, 9(1), 1–14. https://doi.org/10.1186/1743-0003-9-33/FIGURES/9

[36]. Wang, D., Huang, Y., Liang, S., Meng, Q., & Yu, H. (2023). The identification of interacting brain networks during robot-assisted training with multimodal stimulation. Journal of Neural Engineering, 20(1). https://doi.org/10.1088/1741-2552/acae05

[37]. Choi, S., & Kuchenbecker, K. J. (2013). Vibrotactile display: Perception, technology, and applications. In Proceedings of the IEEE (Vol. 101, Issue 9, pp. 2093–2104). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/JPROC.2012.2221071

[38]. Tidoni, E., Gergondet, P., Kheddar, A., & Aglioti, S. M. (2014). Audio-visual feedback improves the BCI performance in the navigational control of a humanoid robot. Frontiers in Neurorobotics, 8(JUN). https://doi.org/10.3389/fnbot.2014.00020

[39]. Papaleo, E. D., D'Alonzo, M., Fiori, F., Piombino, V., Falato, E., Pilato, F., De Liso, A., Di Lazzaro, V., & Di Pino, G. (2023). Integration of proprioception in upper limb prostheses through non-invasive strategies: a review. Journal of neuroengineering and rehabilitation, 20(1), 118. https://doi.org/10.1186/s12984-023-01242-4

[40]. Hoffmann, R., Valgeirsdóttir, V. V., Jóhannesson, Ó. I., Unnthorsson, R., & Kristjánsson, Á. (2018). Measuring relative vibrotactile spatial acuity: effects of tactor type, anchor points and tactile anisotropy. Experimental brain research, 236(12), 3405–3416. https://doi.org/10.1007/s00221-018-5387-z

[41]. Alfihed, Salman, Majed Majrashi, Muhammad Ansary, Naif Alshamrani, Shahad H. Albrahim, Abdulrahman Alsolami, Hala A. Alamari, Adnan Zaman, Dhaifallah Almutairi, Abdulaziz Kurdi, and et al. 2024. "Non-Invasive Brain Sensing Technologies for Modulation of Neurological Disorders" Biosensors 14, no. 7: 335. https://doi.org/10.3390/bios14070335

[42]. González-Franco, M., Peck, T.C., Rodríguez-Fornells, A. et al. A threat to a virtual hand elicits motor cortex activation. Exp Brain Res 232, 875–887 (2014). https://doi.org/10.1007/s00221-013-3800-1

[43]. Lotte F., Congedo M., Lécuyer A., Lamarche F., Arnaldi B. A review of classification algorithms for EEG-based brain-computer interfaces[J]. Journal of Neural Engineering,2007,4(2): R1-R13. https://doi.org/10.1088/1741-2560/4/2/R01

[44]. Wentink, E. C., Mulder, A., Rietman, J. S., & Veltink, P. H. (2011). Vibrotactile stimulation of the upper leg: effects of location, stimulation method and habituation. Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference, 2011, 1668–1671. https://doi.org/10.1109/IEMBS.2011.6090480

[45]. Gonzalez-Rodriguez, A., Ramon, J. L., Morell, V., Garcia, G. J., Pomares, J., Jara, C. A., & Ubeda, A. (2019). Evaluation of Optimal Vibrotactile Feedback for Force-Controlled Upper Limb Myoelectric Prostheses. Sensors (Basel, Switzerland), 19(23), 5209. https://doi.org/10.3390/s19235209

[46]. Rosenbaum-Chou, T., Daly, W., Austin, R., Chaubey, P., & Boone, D. A. (2016). Development and Real World Use of a Vibratory Haptic Feedback System for Upper-Limb Prosthetic Users.https://doi.org/10.1097/JPO.0000000000000107

[47]. Zhang, S., Qin, Y., Wang, J., Yu, Y., Wu, L., & Zhang, T. (2023). Noninvasive Electrical Stimulation Neuromodulation and Digital Brain Technology: A Review. In Biomedicines (Vol. 11, Issue 6). https://doi.org/10.3390/biomedicines11061513

[48]. Matheus Nordi, T., Augusto Ginja, G., Gounella, R., Talanoni Fonoff, E., Colombari, E., Moreira, M. M. A., Afonso, J. A., Monteiro, V., Afonso, J. L., & Carmo, J. P. (2023). Wireless Device with Energy Management for Closed-Loop Deep Brain Stimulation (CLDBS). Electronics (Switzerland), 12(14). https://doi.org/10.3390/electronics12143082

Cite this article

Jiang,Z.;Lyu,H.;Wan,Y.;Qian,H. (2025). Sensory Feedback Improvement of BCI Robotics for Movement Control. Applied and Computational Engineering,131,85-97.

Data availability

The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.

Disclaimer/Publisher's Note

The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of EWA Publishing and/or the editor(s). EWA Publishing and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

About volume

Volume title: Proceedings of the 2nd International Conference on Machine Learning and Automation

Conference website: https://2024.confmla.org/

ISBN：978-1-83558-939-7(Print) / 978-1-83558-940-3(Online)

Conference date: 21 November 2024

Editor：Mustafa ISTANBULLU

Series: Applied and Computational Engineering

Volume number: Vol.131

ISSN：2755-2721(Print) / 2755-273X(Online)

© 2024 by the author(s). Licensee EWA Publishing, Oxford, UK. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license. Authors who publish this series agree to the following terms:
1. Authors retain copyright and grant the series right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this series.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the series's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this series.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See Open access policy for details).