1. Introduction

In recent years, research on the combination of brain-computer interface (BCI) and robotics has made significant progress in the field of motion control and rehabilitation. Starting from the basic principles of BCI robots and their applications in motion control, this paper focuses on the impact of improved sensory feedback on motion control.

The core of BCI robotics technology is to convert the brain's neural signals into motion commands and achieve motion control through external devices (such as robotic arms, exoskeletons, etc.). The main goal of this technology is to help patients whose motor function is impaired due to neurological damage or other diseases recover or enhance their motor ability. However, it is difficult to obtain precise and natural motion control by relying solely on BCI signal transmission. This is because traditional BCI systems often ignore the importance of sensory feedback in motion control when decoding brain signals.

Sensory feedback refers to the information obtained by the human body through senses such as vision, hearing, and touch when performing movements. This feedback helps the brain adjust and accurately control movements, especially in complex motor tasks. The lack of this feedback in BCI robotic systems leads to poor precision and naturalness of movement. Therefore, in recent years, research has begun to introduce multimodal sensory feedback into BCI robot systems to improve the effect of motion control by transmitting information such as touch, force, and position perception.

In the current research trend, more and more scholars are concerned about how to improve the motion control performance of BCI robots by improving sensory feedback mechanisms. For example, hybrid BCI systems (combining brain signals and non-brain signals) have been used to better decode motion intentions while enhancing the construction of multimodal sensory feedback pathways to improve the flexibility and accuracy of BCI control. In addition, the introduction of wearable devices also provides new possibilities for sensory feedback of BCI robots.

The introduction of this review will focus on the combination of BCI and robotics in motion control, explain the principles of sensory feedback, and analyze the current research directions and technical means to improve sensory feedback. By reviewing the existing literature, this review aims to provide a theoretical basis and technical reference for the design and application of future BCI robot systems.

2. Literature Review

The purpose of this research is to study the literature on the role of sensory feedback in BCI for movement control. Then this research will summarize whether several current sensory feedbacks can effectively and feasibly meet the daily use needs of patients, analyze the limitations and shortcomings of existing studies, and make suggestions for future research in this field.

2.1. Selection of Papers

In the preliminary literature search stage, since BCI technology has developed rapidly in the last ten years, we mainly studied the literature from 2012 to 2024 to get a timely conclusion. As for the selection of article content, this study divides the feedback process of BCI into four ways, hoping to get a more detailed comparison and discussion. Therefore, we decided to search with the following keywords (Table 1).

Table 1. The search string

2.2. Sensory Feedback Modalities

In this paper, the research on the feedback process in BCI is divided into three modalities: Visual Feedback, Auditory Feedback, and Tactile Feedback. Among them, auditory feedback may also include Audio-visual feedback, and tactile feedback can be subdivided into Vibration Tactile Feedback and Electrical Stimulation Tactile Feedback feedback. (A detailed introduction to each type of feedback will be mentioned in the following part.) For each type of feedback, we reviewed about 8 articles. Table 2 below shows the number of articles we selected for each type of feedback.

Table 2. The selected sensory feedback modalities

2.3. Assessments

When we searched the literature, we focused on reading the literature that included the following two assessment methods in the results and conclusions, so that we could conduct subsequent analysis.

We counted the number of studies that included both methods of assessment within our reference (Table 3).

Table 3. The methods of assessment

2.4. Behavioral Performance and Accuracy

In many kinds of literature, researchers designed some simulation experiments to study the performance of a certain feedback mode. Some participants were invited to carry out a designed task, such as grasping an object, in a variety of feedback modes. During the course, researchers would record their behavioral performance, such as the time to complete a movement, the accuracy of reaching a position, and the time they took to realize and correct the error.

These results correlate with participants' performance on the task under different feedback modes and directly reflect whether replacing or adding one mode leads to better performance. In this paper, we will analyze the feasibility and practicability of several feedback modalities by analyzing the results and conclusions obtained from experiments in numerous literatures.

2.5. Psychophysiological Measurements

Also, in some literature, researchers recorded the Psychophysiological data of the participants. Physiological data may include EEG, EMG, Heart Rate, fMRI, fNIRS, etc., and psychological data are assessed by directly asking participants how they feel or by the NASATLX questionnaire.

The data from the Psychophysiological Measurements correlated with the participants’ physical state at the time of using a certain feedback. These results can assess the workload and mental burden on participants under different sensory feedback modalities. In this paper, the results and conclusions of this part will be used to analyze whether these feedback modalities are suitable for patients' daily long-term use, as well as the efficiency of patients' learning and training to use a certain feedback mode.

3. Methodology

3.1. Types of Sensory Feedback

3.2. Visual Feedback

Visual feedback in a brain-computer interface (BCI) is a key component of user-system interaction. Essentially, it involves displaying information (usually in real-time) that reflects the user's brain activity and the system's interpretation of those signals. The user generates specific brain signals that are then detected and interpreted by the BCI system. These signals can be motor imagery, visual evoked potentials, or other patterns of neural activity.

The role of visual feedback is to provide immediate, clear, and accurate information about the state of the BCI system and the user's performance. This feedback can take various forms, such as changes in the graphical user interface, the movement of virtual objects, or changes in visual stimuli. Users can adjust their mental strategies based on the feedback they receive, allowing them to control the BCI more effectively. Studies have shown that visual feedback can significantly improve the accuracy and speed of BCI operation[1]. For example, in a study by Lim et al., participants who received real-time visual feedback were able to improve their control of the BCI system faster than participants who did not receive such feedback[2].

In addition, visual feedback helps alleviate some of the problems faced in BCI use, such as the variability of brain signal patterns and the learning curve for mastering brain control. By providing continuous and intuitive feedback, users can better understand how their brain signals are being interpreted and make the necessary adjustments to improve their performance[3].

3.3. Auditory Feedback

Amputees using robotic arms lose the feeling of touch and proprioception. So they need other feedback modalities to help them control the robotic arms. For people with normal visual function, they use visual feedback of course, but it can not provide enough information. Moreover, Many CLIS (complete locked-in state) patients have vision loss and can not use a visually-based BCI[4]. They need a sensory substitution or cross-modal feedback. Thus, BCI based on other sensory feedback needs to be researched. Usually, amputees have an unbroken auditory system, which means auditory feedback is a good choice for sensory substitution[5].

Visual feedback alone is not sufficient for good control and many amputees suffer from phantom limb pain and prosthetic rejection. The application of auditory feedback helps patients to carry out somatosensory integration, thus improving proprioception[6].

Auditory feedback can be presented in various forms. For example, mono sound, stereo sound, or 3-D sound[7]. The volume and pitch of the sound can be linear. Also, simulating the sound of an instrument is a good solution for auditory feedback.

In whatever form, sound is used to deliver information to patients. For example, 3D sound can receive position-related information. Through different volume tones of sound, patients can represent the distance, the size of the force, etc., and can also indicate positive and negative feedback such as right or wrong. Therefore, for patients whose hearing is not impaired or intact, auditory substitution enables them to complete tasks of pointing, reaching, and object localization[8].

However, it is still unknown if auditory feedback can sufficiently help patients reach altered dynamics and adapt to kinematic environments[8]. And, compared with visual feedback, how is the performance of this sensory feedback?

3.4. Tactile Feedback for Vibration

Vibrotactile feedback is an excellent option for reenabling the tactile feedback of artificial limbs[9].

The restitution of the sensation’s loss is important for amputee rehabilitation. Even though visual feedback remains important for performance, many subjects told researchers that the information visual feedback provided was limited. Hence, it is good to relay and integrate tactile feedback [10].

Vibrotactile feedback is to fix vibration devices such as motors on the subject's bodies, to give the wearer different sensory feedback methods with different vibration frequencies or intensities[11].

Inspiringly, vibrotactile feedback can provide excellent information about the sense of limb position[10] and realize good spatial discrimination even with short stimulus time[12].

3.5. Tactile Feedback for Electrical Stimulation

A key component of BCI is electrical stimulation (ES), which has a long history in neuromodulation. ES has its roots in bioelectrical research in the late 18th century. In 1791, Italian scientist Luigi Galvani discovered that frog legs twitched when exposed to an electric current, marking the first confirmed effect of electrical stimulation on biological tissue and laying the foundation for exploring the therapeutic use of electricity in the human body. Since then, ES has evolved from basic experiments in muscle activation to sophisticated applications in BCI, which helps restore motor function and provide sensory feedback for prosthetic limbs. Throughout the 19th and early 20th centuries, ES began to be applied to medical treatments. Early devices were rudimentary and their use was often experimental. However, by the mid-20th century, researchers began to apply ES to neurological conditions such as epilepsy and chronic pain, paving the way for modern technologies such as transcutaneous electrical nerve stimulation (TENS). The development of functional electrical stimulation (FES) for patients with spinal cord injuries in the 1960s marked a turning point. FES uses electrical current to stimulate paralyzed muscles, helping patients regain some movement. In the 1990s, as neuroimaging and computing technologies advanced, researchers began to explore integrating electrical stimulation into BCIs to more precisely control artificial limbs and prostheses through brain signals. This era saw the first demonstrations of BCIs restoring motor function by directly linking brain activity to the electrical stimulation of muscles.

In BCI, electrical stimulation promotes neuromodulation by altering neural activity (either through peripheral nerve stimulation or directly in the brain). This modulation can restore function in patients with movement disorders[13].

Peripheral stimulation is used in situations where muscles are paralyzed but neural pathways remain intact[14], Central nervous system stimulation, on the other hand, is done by intracortical electrical stimulation, which directly stimulates neurons in the brain to elicit motor responses or sensory feedback[15].

Three forms of electrical stimulation are used in BCI systems: Functional electrical stimulation (FES) helps paralyzed people regain motor control by stimulating specific muscles based on signals decoded from the brain. Transcranial electrical stimulation (TES), including transcranial direct current stimulation (TDCS), is a non-invasive way to modulate brain activity with the potential to improve cognition and restore motor abilities[13].

Intracortical microstimulation (ICMS) is an invasive stimulation applied directly to the motor or sensory cortex to control a prosthesis or provide sensory feedback[16].

4. Comparison and Discussion

4.1. Behavioral Performance and Accuracy

In terms of evaluating methods for BCI visual feedback based on code-modulated visual evoked potentials, an important study compared various c-VEP-based visual feedback methods and found that the use of binary phase-shift keying (BPSK) modulation significantly improved the signal-to-noise ratio (SNR) of the evoked potential[17]. This makes the control of the BCI system more precise and efficient, allowing users to execute commands more easily with minimal cognitive load. In addition, the study also emphasized the importance of stimulus design in optimizing the performance of c-VEP-based BCIs. For example, high-contrast visual stimulation and fast flicker rates can produce stronger and more distinguishable evoked potentials.

Another key aspect of evaluating c-VEP-based visual feedback methods is user adaptability. Research showed that users can quickly adapt to c-VEP stimulation and achieve a high level of control accuracy in a short period [18]. This rapid adaptation is crucial for practical applications because it reduces the time and effort required for users to master the BCI system. The dynamic interface designed by the study highlights the potential of personalized interfaces in enhancing BCI user experience and performance[19].

In terms of different types of visual feedback, which investigated the effects of static and dynamic visual feedback on user performance[20]. The results showed that dynamic visual feedback can adapt to the user's brain signals in real time, significantly improving the user's ability to maintain attention and produce more consistent responses. This improves the overall accuracy and efficiency of the BCI system.

The concept of additional robotic limbs (SRLs) refers to the integration of additional robotic limbs into the user's perception and control system. By controlling the additional limbs with BCIs, the user's physical interaction capabilities can be extended beyond the limitations of the biological body. This technology is particularly beneficial in rehabilitation and assistive technology, where enhanced motor function can significantly improve the quality of life of people with disabilities. A key challenge in the implementation of SRL is to achieve seamless integration so that users can perceive the extra limbs as part of their body schema. Fan et al. proposed an SRL control system based on gaze information. The study found that the total mean absolute error limit between the gaze position and the actual execution position was low, indicating that visual feedback is conducive to improving the control efficiency of SRL and expanding the application possibilities of SRL[21]. Another study emphasized the importance of supplementary sensory feedback (such as tactile and visual cues) in achieving this implementation[22]. The researchers found that providing coherent sensory feedback significantly enhanced users' control over SRL and increased their sense of ownership of these limbs. Virtual reality (VR) has also been gradually applied in extra mechanical limbs (SRL). A questionnaire assessment was used to investigate whether users can perceive extra limbs as part of their own body in a VR environment[23]. In the study, it was believed that introducing VR in SRL can create a more immersive and functional user experience[24].

In addition, combining visual feedback with other sensory feedback (such as touch or hearing) is expected to further improve BCI performance. For example, Pfurtscheller G. et al. explored the combination of visual and tactile feedback in dual-arm robot teleoperation tasks and found that multimodal feedback significantly improved task performance and user satisfaction[25]. The study showed that tactile and visual feedback assistance can promote the performance of dual-arm robot teleoperation in surface processing tasks[26].

However, in this study, the researchers found that in a virtual reality environment, tactile feedback performed better than visual feedback, while tactile feedback combined with visual cues did not improve tracking performance, which shows that multimodal feedback is not suitable for all situations[27].

Roland Sigrist and his(her) teammates did some research in their paper [28]. They explored whether adding auditory feedback to visual feedback could promote or hinder motor control and learning. They invited some people to take part in a simulated rowing experiment. Participants should control their boat and move straight forward. Expect using visual feedback, they would hear sound in different directions and volume. This kind of auditory feedback could help them determine if there is a deviation from the regulated direction and by how much.

Finally, they found that VAF had smaller velocity errors and spatial errors than VF, and the error rate of VAF decreased faster during the three-day continuous experiment. These results indicated that VAF had stronger information feedback performance and higher learning efficiency.

Nathan Cutler and his() teammates designed a new type of surgical instrument that used auditory feedback as a sensory substitution for force feedback [29]. They designed an auditory force feedback system to alert surgeons if too much force is being applied.

After using this kind of instrument to simulate surgical operations, surgeons found that the time to complete the task was shorter, the amplitude of force change was smaller, and the number of triggered excessive force alarms was also reduced after multiple training. These results indicated that the addition of AF also promoted the user's learning and training of motion control.

Therefore, as a multimodal feedback, VAF can provide patients with more accurate and faster perception and facilitate the learning process as well.

According to the Bayesian principles of multisensory integration, a shared representation offers better performance because of the reduced variability, etc. when multimodal feedback is executed. Hence, theoretically, the addition of vibrotactile feedback to the visual feedback is believed to improve the performance of the subjects compared to the visual feedback alone[30].

Some studies proved such a prediction. Eitan et al. conducted experiments with some trans-radial amputees, using different methods of feedback and accomplishing tasks in different rooms with different scales of light. They measured the time subjects moved the blocks and the accuracy by figuring out the number of empty grips etc. The results showed that adding vibrotactile feedback has positive effects on performance time and accuracy when visual feedback is disturbed[31].

Also, in the previous study of Matthieu et al., they found that vibrotactile feedback could provide better spatial resolution[32].

However, when they conducted experiments for online control, using a novel method, the results were different. Their novel vibrotactile feedback doesn’t enhance or deteriorate the performance and the accuracy of visual feedback[10].

Functional electrical stimulation (FES) within BCIs is widely used in motor recovery, particularly for stroke or spinal cord injury patients. By translating brain signals into electrical impulses that activate muscles, BCIs enable paralyzed individuals to regain voluntary movement. This approach fosters neural plasticity and aids long-term rehabilitation.

One of the significant challenges in BCIs is providing sensory feedback to users of prosthetic limbs. Electrical stimulation can bridge this gap by stimulating sensory nerves to send feedback to the brain. This enables users to "feel" their prosthetic limbs, improving control, precision, and the overall user experience. Early applications of sensory feedback through ES in prosthetics were demonstrated in the early 2000s, and ongoing advancements continue to refine this technology.

Electrical stimulation is also used for neurorehabilitation and cognitive enhancement. Techniques like **transcranial direct current stimulation (tDCS) have shown promise in improving cognitive functions by enhancing neural plasticity. These techniques are being studied for their potential to aid patients with neurodegenerative diseases or traumatic brain injuries[33].

4.2. Psychophysiological Indicators

Visual feedback can provide a visual method for emotion regulation. Jones et al. designed a tangible interface consisting of a series of colored LEDs to provide users with real-time emotional feedback to enhance their ability to regulate emotions more effectively in stressful situations[34].

Jose Gonzalez and his() teammates researched the performance of AF, VF, and VAF[35]. Each modality was used in controlling the robotic arm to catch and hold an item. During the participants doing the test, their physiological data like EEG, EMG, and heart rate were recorded. They were also asked to finish a NASATLX questionnaire.

These physiological data showed a clear correlation with patients' performance on the tests, such as lower Alpha band EEG and heart rate in VAF mode. Results suggested that using auditory feedback as sensory substitution made attention demand lower when controlling the robotic arm, rather than reducing the cognitive effort. The results of the questionnaire suggested that although the psychological assessment varied among individuals, most people thought that the psychological pressure brought by VAF was the lowest. This meant that this sensory feedback mode was more suitable for patients to use for a long time when considering the fatigue effect.

Duojin Wang and his(her) teammates also did similar research[36]. In their study, they used a continuous-wave fNIRS system to detect the hemodynamic responses in the brain area of participators. Participants were asked to control a link mechanism to play a video game in AV, AV, and VAF.

They analyzed networks of brain regions and cerebral cortical activation levels. Results showed that, when adding auditory feedback, the connection of interacting networks was stronger and the cerebral cortex was more active.

Most of the studies about vibrotactile feedback concentrated on the accuracy and the best forms of vibrations etc. Hence, NASA-TLX, a widely-accepted and multidimensional questionnaire was used. In the previous study of Matthieu et al., they found that using vibrotactile feedback only could increase the mental workload of the subjects compared to the visual feedback, while visual plus vibratory feedback didn’t put more burdens. However, when the researchers asked the subjects to rank those methods, they found that most of the subjects ranked visual plus vibratory as the first one[10].

5. Conclusion

Integrating visual feedback into BCI systems is essential for their functionality and user-friendliness. Not only does it enhance user control and accuracy, but it also provides a more engaging and intuitive experience, making BCI more accessible and more effective for a wider range of applications.

In summary, visual feedback plays a vital role in the development and optimization of BCI systems, as well as in brain signal recognition.

Auditory feedback is an excellent transmembrane feedback scheme with low training difficulty, and after being familiar with it, patients can perform well and feel less psychophysiological pressure, which can support long-term use. In future research, audiovisual feedback has a large space for development.

In summary, there is evidence to support the use of vibrotactile feedback for the restitution of proprioception and improvement of performance. Also, the vibrators are quite cost-effective compared to some machines for electrical feedback and can convey sufficient information with relatively low complexity[37].

Electrical stimulation plays a crucial role in BCIs, particularly in restoring motor functions and providing sensory feedback. Despite current challenges such as invasiveness and long-term efficacy, ongoing innovations in closed-loop systems and wireless technology promise to enhance the integration of electrical stimulation in BCI applications. Future research will likely continue to refine these techniques, bringing us closer to more effective, non-invasive BCI solutions.

6. Limitations and Future Outlook

6.1. Limitations

Despite all the progress, there are still some challenges to be addressed in developing visual feedback methods. One of the main issues is that users react differently to visual stimuli, which can be affected by factors such as fatigue, attention, and individual differences in visual processing.

In addition, controlling an additional limb using visual feedback creates a huge cognitive load. Users have to manage not only their biological limbs but also their robotic limbs, which require high cognitive abilities. Therefore, it is crucial to design intuitive control interfaces and feedback systems.

While sound feedback is available in general conditions, it has a few limitations. Patients with hearing loss cannot use auditory feedback as a sensory substitution. They cannot accurately receive the information conveyed by sound and this may further increase their psychophysiological burden.

Moreover, noise from complex application environments will lead to a great distraction in auditory feedback. For example, when lower limb amputees use robotic legs, the sound of their footsteps will be mixed with the [38]. Also, the electrical noise of the electronic device itself will make the sound difficult to discern.

However, there were several flaws due to the mechanism of the machines and our bodies or the difficulties when it comes to the studying process.

For example, some people may get accustomed to the stimulus, which makes it barely perceivable after some time. A low spatial resolution could be found[39].

The reason is that in our bodies, Pacinian corpuscles and hair follicle receptors are greatly stimulated by rapid skin indentation and hair motion, such as from vibration, but they tend to produce lower resolution, due to their large and small quantity of receptive fields. Also, propagation to the surrounding skin contributes to the low spatial resolution[39-40].

Also, some subjects had difficulties telling between different perceptions of feedback levels. Most of the users who found it easy to keep up with the learning process of the experiments were just accustomed to the usage of vibrotactile feedback in their daily lives. Hence, the learning cost of vibrotactile feedback needs to be considered[10].

Despite the significant advancements in electrical stimulation within BCIs, several challenges remain:

Achieving precise stimulation without side effects remains a challenge. Customizing stimulation parameters, such as frequency, amplitude, and duration, for individual users is necessary to avoid unintended neural activation or fatigue[41].

Invasive methods, such as intracortical stimulation, require surgical implantation of electrodes, which poses risks such as infection or device failure. Developing non-invasive methods that offer similar precision remains a critical area of research[13].

6.2. Future Research Directions

To better solve the many problems in visual feedback, the current research focus and future research trend is to develop adaptive algorithms to customize visual feedback according to the unique characteristics and current state of each user. In addition, integrating visual feedback that simulates natural limb movement can also reduce the user's cognitive burden on visual feedback and improve control efficiency. Studies have shown that when users receive real-time visual feedback consistent with their expected actions, their performance in controlling additional robotic limbs (SRL) will be significantly improved[42].

In terms of user experience, the study explored the impact of different visual feedback on motor imagery control during BCI user training[43]. The results showed that immersive and interactive visual feedback can promote better learning and adaptation, thereby improving the control ability and accuracy of BCI tasks.

To further research the feasibility and practicality of auditory feedback, several details can be optimized in future research. Compared with using speakers to make sound in many experiments, using headphones can make the sound more clear and efficient[7]. The coding strategy that maps force signals to sound signals can be optimized to make the information more comprehensible[5]. Force sensors can be combined with wearable devices like gloves or robotic arms, which can directly enhance the practicability of the devices[5]. After making the above improvements, the simulation experiment will be closer to daily life, and the data and results will be more meaningful for the development of real products.

Despite the accuracy and the mental workload of the patients, the most appropriate patterns, amplitude, or frequency of the vibrations, the placements of the vibrators, etc. were also under research.

Andrea et al. used different sets of vibrations with various pattern shapes as well as pattern amplitudes. They aimed to figure out the best patterns of vibration.

However, as E C Wentink et al. conducted an experiment and found that high frequencies are better perceived than low frequencies significantly[44], the studies of Andrea et al. who used a similar method to conduct experiments showed that there are modest differences in perceptions of feedback levels between different patterns[45].

The experimentation on tractor placement demonstrated that the proximal location was better for vibration discrimination, although not significantly different. They thought the reason was probably that each remainder limb had different locations of scar tissue or less sensitive areas. Hence, tractor placement will likely need to remain customizable[46].

Future advancements in electrical stimulation for BCI systems are likely to focus on increasing the precision, efficacy, and accessibility of the technology.

In closed-loop BCI systems, real-time feedback is used to adjust stimulation parameters dynamically, improving the system's adaptability and performance. This could significantly enhance rehabilitation outcomes and prosthetic control[47].

Wireless stimulation systems will enhance the user experience by eliminating the need for cumbersome wires and making the devices more portable and user-friendly, particularly for long-term use[48].

Acknowledgment

Zekai Jiang and Haocheng Lyu contributed equally to this work and should be considered co-first authors.