1. Introduction

An important field in image processing and human activity recognition is drowsiness detection. A person's reaction time is greatly slowed down during the period between wakefulness and sleep, which is commonly referred to as drowsiness. The main causes of fatal traffic accidents include driver negligence, fatigue, and sleepiness. These elements have a big effect on road safety because they cause drivers to pay far less attention to the road and have less control over their vehicles. Driving while weary or drowsy is the practice of operating a vehicle while experiencing exhaustion or sleepiness.

Fortunately, it is now possible to detect signs of driver fatigue and drowsiness at an early stage and alert them, which can significantly reduce the occurrence of accidents. This is achieved by identifying various symptoms of drowsiness, such as frequent eye closures and prolonged yawning. Depending on the area of detection, these signs of fatigue can be categorized into four main types. These signs can be detected through images or video sequences captured by cameras monitoring the driver's facial expressions.

Furthermore, biological signals can be recorded by sensors attached to the driver, and the vehicle's movement and behavior can be monitored. For the same objective, hybrid approaches are also employed. Based on how they are used, Ramzan et al. categorized the state-of-the-art driver drowsiness detection (DDD) methods in a thorough analysis [1]. There exist three primary categories of these techniques: physiological signal-based, vehicle behavior-based, and behavioral parameter-based. The authors examined the most effective supervised methods for identifying sleepiness and talked about the benefits and drawbacks of the three methods they offered. A review research on the identification of driver weariness and drowsiness was carried out by Sikander et al. [2]. There are five types of DDD techniques: hybrid, vehicle-based, physical, biological, and subjective reporting. A study by Otmani et al. used deep learning methods to measure drowsiness and weariness, including recurrent neural networks (RNN) and convolutional neural networks (CNN) [3].

This paper makes significant contributions to the driver fatigue detection systems introduced in recent years, particularly those modern systems, as detailed below:

• Introduces modern deep learning techniques used to detect driver drowsiness for ensuring safe driving.

• Classifies driver fatigue detection systems into four groups based on the tools used to assess drowsiness: eye and facial detection, mouth detection, head detection, and hybrid detection.

• Focuses on the datasets primarily used for detecting drowsiness and introduces their characteristics.

• Compares the applicability and dependability of the four categories for detecting driver weariness.

• explains the drawbacks of drowsiness detection methods, their implications, and the challenges facing the field of detecting driver weariness.

2. Driver Drowsiness Detection (DDD) Systems

When a person is mentally fatigued, they may fall asleep and their attention to tasks decreases. It is crucial to understand that fatigue is not a disease but a state that can be restored through rest and sleep. However, when driving vehicles that require continuous attention, such as cars, buses, or trains, this recovery can become very difficult, potentially leading to serious accidents. Currently, numerous studies are exploring the physiological mechanisms of drowsiness and assessing the degree of fatigue. The literature confirms that there are numerous elements that influence weariness and drowsiness/lethargy in different ways, making it difficult to determine the exact degree of these states. As shown in Figure 1, in order to effectively and reliably depict the driver's condition, fatigue monitoring requires a system that can gather data from various sources and integrate it. Driver fatigue detection systems include several stages:

Data Collection: Data is acquired through convenient sensing devices, known as input modules.

Preprocessing Module: This module prepares the data by applying landmarks, dimensionality reduction, face detection, feature extraction, feature transformation, and feature selection.

Detection or Classification: Relies on machine learning and deep learning techniques for detection or classification.

Output Module: This module determines the driver's condition and indicates whether or not they are tired.

Alarm Activation Module: Activates an alarm when drowsiness is detected. As shown in Figure 1, this series of processes works in coordination to ensure the driver's safety.

Figure 1. Data flow and general block diagram in DDD systems [4]

2.1. Eye and Facial Detection

For many computer vision applications, including human-computer interface (HCI), facial emotion recognition (FER), and fatigue and sleepiness monitoring, automatic classification of eye states is essential. These applications are mentioned in Figure 2 and Table 2.

Saurav S. et al. concentrated on creating and implementing a vision-based real-time eye state identification system for embedded platforms with limited resources in order to overcome these problems [5]. To extract meaningful data from the ocular regions, the system makes use of two lightweight CNNs. Fine-tuned transfer learning (TL) was used to train the CNNs using a small sample eye state database without overfitting.

60-second picture sequences of the face were employed by Magán E. et al. They created two distinct techniques to determine whether the driver is sleepy in order to reduce false positives [6]. While the second method uses DL techniques to extract numerical features that are then fed into a fuzzy logic-based system, the first strategy relies on RNN and CNN.

Dua M. et al. introduced a driver fatigue detection system that utilizes a structure comprising four deep learning models: VGG-FaceNet, AlexNet, FlowImageNet, and ResNet. To identify drowsiness and weariness, they used RGB videos of the driver as input [7]. They took into account four main types of features while applying these techniques: hand and head movements, behavioral traits, and facial expressions.

Figure 2. Procedure for observing blinking of the eyes [4]

2.2. Mouth (Yawning) Detection

The several methods for determining tiredness based on mouth and yawning analysis are shown in Table 2 and Figure 3, along with the benefits and drawbacks of each method.

The goal of the work by Savaş BK et al. was to develop a yawning-based driver detection system that uses a CNN model for classification to detect fatigue [8]. The datasets YawDD, Nthu-DDD, and KouBM-DFD were used to train and evaluate the suggested model.

An enhanced algorithm that can recognize the mouth and face was used in the work by Omidyeganeh M. et al. To determine yawning, the back-projection technique was used to quantify the rate of mouth alterations [9]. Congivue's intelligent embedded Apex camera was employed in this project, fully using the high computational power and low memory of the embedded platform. It could only identify specific alertness levels, though.

In the study by Knapik M. et al., a yawning detection fatigue system based on thermal imaging was described. This device can function without disturbing the driver during the day or night because it makes use of thermal imaging [10]. Facial alignment is first accomplished by identifying the corners of the eyes, and yawning is then detected using the suggested thermal model.

Figure 3. Procedure for observing mouth and yawn [4]

2.3. Head Detection

Numerous techniques have been researched in recent years to detect driver drowsiness. The primary method used by these tactics to identify fatigue is driving behavior observation. Disturbances in a driver's driving style are frequently used to detect driver weariness. Numerous authors cited in Table 2 use gaze and head position variables to detect driver weariness, as illustrated in Figure 4.

Even when the driver wears sunglasses that cover their eyes, the study by Wijnands JS et al. provides an example explaining how to balance real-time inference requirements and good prediction accuracy using deeply separated 3D convolutions and early spatial and temporal feature fusion [11].

Figure 4. Head pose and gaze observation procedure [4]

2.4. Hybrid Eye, Mouth, and Head Detection

Table 2 presents driver fatigue detection systems that rely on and incorporate hybrid architectures with parameters from eye, mouth, and head movements.

De Lima Medeiros PA et al. used volunteers' blinking as a human-computer interaction interface and developed an advanced computer vision detector capable of processing information acquired by a standard camera in real-time [12]. Blink detection was carried out using the following processes as an improved technique for eye state recognition: face recognition, modeling, region of interest (ROI) extraction, and eye state categorization. Moving average filter, ROI evaluator, and rotation compensator were other components of this method. Moving average filter, ROI evaluator, and rotation compensator were other components of this method. Two other datasets were also generated: the Autonomous Blinking Dataset (ABD) and the YouTube Eye State Classification (YEC) dataset, which was produced by taking facial photos out of the AVSpeech collection. CNN and SVM models were trained on the YEC dataset, and experiments were carried out on multiple datasets, such as CEW, ZJU, Talking Face, Eyeblink, and ABD, to assess the algorithms' performance.

Moujahid A. et al. introduced a facial monitoring method that relies on compact facial texture descriptors, which can capture the most apparent drowsiness features [13]. This compactness was further improved by applying a feature selection procedure to the initial recovery features and using a multi-scale pyramid facial representation with fundamental sections comprising both local and global information. The model was separated into four stages: (i) facial description utilizing a multi-level multi-scale hierarchy for feature extraction; (ii) pyramid multi-level (PML) facial modeling; (iv) feature subset selection and classification. In the study by Ed-Doughmi Y. et al., RNNs were used to evaluate a sequence of driver facial images and predict the driver's fatigue status. The model was designed and evaluated using a dataset to detect driver drowsiness. A multi-layer model based on a recurrent neural network architecture that relies on 3D convolutional networks was constructed [14].

3. Results Analysis

3.1. Analysis of Commonly Used Datasets

Table 1 lists the most commonly used datasets in driver fatigue detection systems, providing descriptions for each. Various datasets have been utilized in multiple driver fatigue detection systems.

Table 1. Description of datasets

3.2. Evaluation Metrics

Various metrics have been adopted to evaluate the system's ability to recognize drowsy and fatigued participants, including precision, accuracy, F1 score, and sensitivity. Equation (1) provides the accuracy evaluation metric. Accuracy is the most widely used metric for evaluating driver fatigue detection systems. It provides a good indicator of the system's ability to distinguish between true positives (TP) and true negatives (TN). True positives, true negatives, false positives, and false negatives are denoted by the letters TP, TN, FP, and FN, in that order.

\( Accuracy= \frac{TP+TN}{TP+FP+TN+FN}\ \ \ (1) \)

3.3. Experimental Results Analysis

The work on driver drowsiness detection (DDD) that combines deep learning (DL) and machine learning (ML) is shown in Table 2. covering most algorithms based on eye, facial, mouth, and head measurements. The table clearly indicates the source, year, measurement area, algorithm classification, implementation environment, accuracy, and dataset for each algorithm.

Following data analysis with the NTHUDDD public dataset shown in Table 2, the following conclusions can be drawn:

The accuracy of deep learning is superior to that of traditional machine learning. This is because deep learning has more parameters, allowing it to learn from more data and features, thus enhancing its learning performance significantly compared to machine learning.

Measuring mouth features plays a crucial role in improving the model's accuracy. Yawning is a prominent and critical feature of driver fatigue, serving as an important criterion for determining whether a driver is fatigued. Therefore, mouth measurement holds significant weight in DDD systems.

Table 2. Image and video-based DDD systems

4. Discussion

In-depth literature research has revealed various strategies for detecting driver fatigue and mitigating potential issues. With advancements in technology and ongoing research in the field of artificial intelligence, the performance of these systems has significantly improved, successfully addressing many challenges. Considering factors such as invasiveness, interference, cost, ease of use, and accuracy, the practicality of these systems highlights their capability to accurately identify actual levels of drowsiness.

4.1. Image-Based Measurement Methods

Image-based measurement methods are considered practical because they are non-invasive, non-interfering, cost-effective, and do not require sensor setup each time the system is used. Common tools include webcams, smartphone cameras, and thermal cameras. Cameras that gather data are positioned at a specific distance from the driver in order to prevent obscuring their view.

4.2. Relevant Parameters and Accuracy

These systems are dependent on factors associated with weariness, like eye closures, head motions, and yawning. Their accuracy typically ranges from 80% to 99%. However, as mentioned earlier, these systems are influenced by various parameters. Typically, controlled conditions are used for their construction and evaluation, or video records of drivers detecting weariness are used. Therefore, the response to fatigue and its symptoms is mainly simulated, with drivers being prompted to mimic certain symptoms during the data collection phase. In simulated test environments or when using actual databases, the accuracy of image-based systems can exceed 90%. When tiredness manifests visually, it is easy to identify.

4.3. Advantages of Hybrid Systems

Overall, using an appropriate fatigue detection system is the most important step in minimizing car accidents caused by exhaustion. Furthermore, hybrid systems are seen to be the best option for detecting tiredness because of their outstanding track record of dependability.

5. Conclusion

This work offers a thorough analysis and current assessment of systems for detecting driver fatigue, with an emphasis on research findings from 2012 to 2024. Given that this field directly impacts human safety, extensive research is currently being conducted. Given the direct impact on human safety, extensive research is currently being conducted in this field. Driver carelessness or inattention is the primary cause of most traffic accidents. In this review, a comprehensive classification of driver fatigue detection systems is provided, discussing four fundamental methods based on different features for identifying fatigue and drowsiness. These four methods are: eye or facial detection, mouth detection, head detection, and hybrid detection systems. This study presents a comprehensive review of all the systems that are involved, along with the datasets that are utilized, the deep learning models that are implemented, the parameters that are used, and the accuracy of each system's detection. Additionally, typical datasets used for detecting driver fatigue are introduced. Finally, the paper proposes new research directions and potential solutions to address the current issues. Through continuous innovation and improvement, driver fatigue detection systems will be able to more effectively prevent fatigue-related road accidents, ensuring the safety of human lives.