1. Introduction

In recent years, intelligence and AVs have gotten great concern and have been studied all over the world. According to the Society of Automotive Engineers (SAE), vehicles can be divided into six levels L0-L5 according to the level of intelligence. Level 0 refers to fully manual driving and Level 5 is fully automated driving. At present, most of the intelligent vehicles are semi-AVs, which are in the range of Level 2 – Level 3. However, the improvement of AVs is still limited by technology, trust, and legislation. It can be predicted that in the future for a long time, human-machine co-driving will be the main driving type.

In human-machine co-driving conditions, the study of human-machine trust and efficient take-over strategy is very important, which is related to whether the human-machine trust can be established properly, the performance of intelligent driving of the vehicle can be fully utilized, and whether the driver can take over the control of the vehicle in time and react quickly in emergencies to avoid accidents. However, the current research on human-machine trust mainly focuses on theoretical research. For example, human-machine trust based on transparency studies the establishment of human-machine trust and the related factors, while there are few studies on real-time detection of human-machine trust in automated driving conditions [1]. At the same time, with the continuous development of MSIF, it is also applied to the field of intelligent driving, which is used to improve the intelligence level of the auxiliary driving system to achieve better auxiliary driving effects. At present, the application of MSIF in the field of automated driving mainly focuses on the perception of the external environment and map construction, using lidar, millimeter-wave radar and binocular cameras, etc. [2]. However, there are few discussions on MSIF for in-car HMI. Therefore, this paper will focus on the analysis of the current application of MSIF in the HMI of intelligent driving conditions, discuss the possibility of applying MSIF to monitor human-machine trust and realize efficient take-over, and put forward some suggestions for the HMI design of AVs in the future.

2. MSIF applied in the intelligent cockpit

With the development of automobile cockpit intelligence and network connection, HMI has gradually changed from the traditional single modality to multiple modalities. Compared with the traditional interaction mode, multi-modal HMI can provide a more efficient interactive experience by engaging multiple human senses. At present, the HMI modes applied in the automobile cockpit can be divided into two types: explicit interaction and implicit interaction. Explicit interaction is based on visual, auditory, haptic and other senses, which can be directly perceived by the driver. Implicit interaction refers to extracting the driver's facial expression, eye movement and other biological features through biological sensors to judge the driver's driving state [1].

In recent years, many enterprises researching and developing intelligent vehicles have independently designed a series of HMI systems. In the HMI process, MSIF plays a crucial role. By virtue of the complementary advantages between different sensors, it has shown great advantages in HMI. Some interactive instruction recognition technologies are shown in Table 1 [3].

Table 1. Interactive instruction recognition technology.

As can be seen from Table 1, cameras, pressure sensors, voice interaction, and other technologies are mainly applied in the intelligent cockpit to mobilize human senses such as vision, hearing, and touch to achieve better HMI. However, the current HMI mainly focuses on navigation, communication, and entertainment. There are few applications in trust detection and driving monitoring.

3. Analysis of human-machine trust

3.1. Definition of human-machine trust

Marsh and Dibben propose that human-machine trust consists of three levels: dispositional trust, situational trust, and learned trust [4]. Dispositional trust is an inherent tendency, which is mainly determined by people's age, gender, personality and some other factors. Situational trust refers to human-machine trust in a specific situation, which is not only affected by the driver's emotions and health conditions but also influenced by environmental factors such as visibility and road conditions. Learned trust refers to the driver's trust in the system acquired by training and learning [5]. For example, according to current research, experienced drivers tend to have higher trust in the automation system [6].

In the field of automated driving, trust refers to the overall standard of trust the driver has in the vehicle's automation system. It is important to note that higher trust is not always better. Ideally, the driver's trust in the vehicle is supposed to be positively correlated with the overall capability of the driving system. Insufficient trust will lead to the increase of driver’s workload or cause failure to give full play to the vehicle’s ability. Excessive trust may lead to excessive reliance on the vehicle, which may cause the driver to fail to take over the vehicle in time [5]. Therefore, it is necessary to establish appropriate human-machine trust during intelligent driving, not only to reduce the driver's workload, but also to ensure driving safety.

3.2. MSIF applied in trust detection and calibration

At present, research on human-machine trust of automated driving mainly focuses on the influence of steady factors on trust, such as driver's age and gender. However, there is little research on dynamic trust, mainly because dynamic human-machine trust is easily affected by multiple factors, dependent variables are difficult to set, and it is difficult to obtain data related to dynamic trust during driving. Therefore, it is difficult to realize real-time human-machine trust detection. However, the application of MSIF can solve the above problems to some extent.

By setting up different environment states, He et al. simulated two driving scenarios, driving during the day and driving at night. They detected physiological indicators such as the average saccadic time by head-mounted eye tracker [7]. The results show that the cognitive state of the environment has a significant impact on the average saccadic time and other physiological indicators, which reflect the change of the driver's trust in the vehicle. Specific results are shown in Figure 1 [7].

The above data shows that the average saccadic time of drivers in a fuzzy environment is higher than that in a clear environment. It indicates that drivers have higher trust in vehicles in a fuzzy environment.

According to the results above, a real-time monitoring method of human-machine trust based on MSIF can be proposed. It is possible to set up appropriate sensors in the vehicle to monitor the driver's physiological indicators. For example, using the depth camera to monitor the driver's eye movement, or installing some other sensors to monitor other physiological indicators. Thus, the real-time human-machine trust can be detected.

In order to ensure that the human-machine trust is in the proper range during automated driving, it is necessary to calibrate the human-machine trust in real time. The study of Gao et al. pointed out that warning and reliability information can be provided to the driver through visual or auditory ways to calibrate the human-machine trust when the driver has insufficient or excessive trust in the driving system [8].

4. Analysis and evaluation of take-over strategy

4.1. Definition of take-over

When the automated driving system breaks down or is in certain circumstances, the driver needs to take over the driving control and take emergency measures to ensure driving safety. Actually, driving take-over refers to the process of a driver's transition from a non-working state to a driving state. The interaction between the driver and the vehicle is a crucial part of driving take-over. Reasonable and appropriate HMI is an important prerequisite for improving take-over efficiency and ensuring driving safety.

Evaluating the driver's driving ability is an important part of designing the driving take-over strategy. At present, there are three main methods to evaluate drivers' driving ability: physiological index evaluation method, behavioral index evaluation method and psychological index evaluation method [9]. Physiological index evaluation method can only be used to detect the driver's stable driving ability, while behavioral index evaluation and psychological index evaluation can be used to detect the driver's real-time driving ability by applying MSIF.

4.2. The influence of trust in take-over performance

According to the human-machine trust in vehicles, Chu et al. divided the drivers into two groups, the trust group and the doubt group, then tested their average take-over frequency and decision reaction time in dangerous situations respectively [10]. Specific results are shown in Figure 2 and Figure 3 [10].

The results show that the average take-over frequency of drivers who trust the driving system is lower than those who doubt the driving system, and the average reaction time of drivers who trust the driving system is higher than that of the doubting group. Thus, drivers in the trust group are more optimistic about the automated driving system than the doubt group. It indicates that the driver's trust in the driving system has a significant impact on the take-over decision and take-over performance.

4.3. Take-over strategy applied MSIF

Visual, auditory, and haptic take-over methods are common warning ways in driving take-over. Visual interaction is usually achieved through the lighting of the dashboard icon or the steering wheel. Auditory interaction can be realized by alarm warnings or voice reminders. Haptic interaction can be achieved through seat vibration [11]. The take-over method has an influence on human-machine trust and driver’s take-over performance. The research of You Fang et al. shows that by improving the design of steering wheel light can enhance man-machine trust and take-over efficiency to some extent [12].

In more complex automated driving conditions, the single-mode take-over warning may be ignored by the driver or cause omitting crucial information [7]. However, the multimodal interaction based on MSIF can complement the advantages between different sensors, in order to improve the take-over efficiency.

Frederik Naujoks et al. 's research shows that the take-over warning method has an impact on both take-over time and take-over quality [13]. Compared with a single visual take-over warning, the take-over time under visual-auditory take-over warning is shorter. At the same time, the audiovisual joint take-over method also shows great advantages in lateral vehicle control. Specific data is shown in Table 2 [13].

Table 2. Means of the parameter.

The above results show that it is more beneficial for the driver to take over the vehicle in time if the driver is provided with appropriate take-over guidance in both auditory and visual aspects when the take-over request is given.

Yun et al. 's study also shows that the multimodal take-over interaction based on MSIF is more conducive to improving the take-over efficiency [14]. In automated driving conditions, they respectively detect the driver’s take-over effect under visual take-over warning, auditory take-over warning, haptic take-over warning, visual-auditory take-over warning, visual-haptic take-over warning, auditory-haptic take-over warning, and visual-auditory-haptic take-over warning. The test results are shown in Table 3 [14].

Table 3. Means of the dependent variables.

In this experiment, the driver's comprehensive take-over effect is the best under a visual-auditory-haptic take-over warning, while the comprehensive take-over effect is the worst under a single visual take-over warning.

Many types of research on this problem have similar results, indicating that multimodal HMI has better comprehensive effect than single-modal HMI in a take-over warning. Therefore, the future HMI design should fully consider the application of multimodal HMI, giving the driver concise prompts without frightening them when necessary. That is supposed to improve the take-over efficiency and comprehensive take-over effect.

5. Discussion

Based on the above discussion, it is obvious that the application of MSIF has great advantages and feasibility in HMI design of automatic vehicles. Because of the reflection of human-machine trust on physiological indicators such as eye movement, real-time human-machine trust can be detected by an eye movement camera equipped in the cockpit. Then the analysis results are given to the ECU, adjusting the HMI dynamically. For example, the take-over interaction strategy can be adjusted by changing the auditory warning advance time and volume, visual light brightness, flashing frequency, or seat vibration amplitude and frequency. The dynamic adjustment of HMI strategy will adapt to the driver's driving state better, and have a more reliable and efficient effect. In automated driving conditions, the HMI effect will significantly affect the take-over performance, thus affecting driving safety. Therefore, MSIF can be further applied to the HMI of AVs, which can both improve the HMI performance and enhance the driving safety and the system’s reliability.

However, there are still some problems in the application of MSIF in HMI. Firstly, the normal operation of the camera has high demand for environmental conditions such as light. Thus, it is difficult to ensure high stability and reliability in complex driving conditions. Secondly, human-machine trust and the optimal take-over strategy have a complicated corresponding relationship. That means it is difficult to achieve linear adjustment. The possible solution is to apply MSIF to improve the overall reliability of the system, avoiding the system’s breakdown due to certain sensor’s failure. Besides, it is necessary to further study the relationship between human-machine trust and take-over performance, design optimal driving take-over methods in different trust intervals respectively, and maintain the human-machine trust at a proper level by optimizing the HMI design.

6. Conclusion

In human-machine co-driving conditions, the application of MSIF has impact both on human-machine trust and take-over performance. This paper lists some HMI methods used in the intelligent cockpit and points out the advantages and prospects of multimodal interaction. Many types of research show that compared with a single-mode HMI, multimodal HMI has more advantages in interaction effect for mobilizing a variety of human senses. This paper also explains the concept and composition of human-machine trust and analyses trust detection and calibration methods based on MSIF. This paper explains the concept of take-over, shows that human-machine trust has a significant impact on take-over performance, and puts forward MSIF applied in take-over strategies.

In the future design of intelligent cockpit, it is considerable to further optimize the MSIF, in order to establish appropriate and reliable human-machine trust and monitor real-time human-machine trust. At the same time, adjusting the take-over strategy in real time according to the changes of human-machine trust can help to realize a more concise, intuitive and efficient HMI, achieving a better interaction effect, reducing the drivers’ load and improving their interactive experience.