ArticlesAll Issue
ArticlesResearch on a Driver Fatigue Detection Model Based on Image Processing
• Zhendong Mu1,*, Ling Jin1, Jinghai Yin1, and Qingjun Wang2,3

Human-centric Computing and Information Sciences volume 12, Article number: 17 (2022)
https://doi.org/10.22967/HCIS.2022.12.017

Abstract

In recent years, the continuous increase of private car ownership has led to an ever increasing number of road traffic accidents caused by driver fatigue. This paper, which is focused on image processing technology and research on driver fatigue, proposes a driver fatigue detection model based on image processing, and introduces various methods of identifying driver fatigue. It also discusses various algorithms, such as the Hough transform, AdaBoost, and PERCLOS algorithms, which can recognize a driver’s eyes and skin color. The experimental section of this paper explains the proposed driver fatigue detection model in many aspects. The analysis section analyzes eye closure detection, PERCLOS detection, and detection rate with regard to different algorithms used to detect driver behavior, the influence of spectacles on detection, and the success rate of different algorithms under different degrees of illumination. The PERCLOS value of the tester obtained in the “awakened” state is significantly smaller than the PERCLOS value obtained in the “fatigued” state. For example, the value of Tester 1 in time period 1 is 0.173 when awake, and the value in the same time period when fatigued is 0.523. The change in the percentage of eye closure is closely related to the degree of fatigue of the human body, and usually shows a positive correlation. Comprehensive analysis shows that the proposed image processing technology can effectively realize facial recognition. Therefore, the state of the driver’s eyes, limbs and other parts can be recognized by this technology, and the driver’s state of fatigue can be accurately assessed.

Keywords

Image Processing, Driver Fatigue, PERCLOS, State Extraction, Human Eye Positioning

Introduction

In the field of transportation, human body fatigue has a great influence on the stability, reliability and safety of drivers, and is a known cause of traffic accidents. Therefore, research on technology capable of detecting human body fatigue is a hot topic in the industry. Many scholars at home and abroad have conducted in-depth research on a variety of fatigue detection technologies based on drivers’ physiological signals and facial features, and car motion information. Each method of fatigue detection has its own shortcomings, such as invasive testing, false detection, and missed detection among others. A driver’s attention span, feelings, perceptions, thinking, judgment, will, ability to make decisions, and engage in sports activities are all affected by driver fatigue. If one could implement timely and effective measures, such as early warning, before a driver appears fatigued, then it would be possible to effectively prevent traffic accidents, which would be a contribution of great significance to traffic safety.
The concept of image processing consists in processing image information to meet the needs of people’s visual psychology and practical applications. Using image processing technology, collected videos or images can be digitized in order to detect and assess human behavior. Human behavior can be categorized into the following broad types: group behavior, individual behavior, normal behavior, and abnormal behavior under normal circumstances. At present, image processing technology is used in a wide range of applications in the aerospace, military, industrial automation detection, security identification, and entertainment fields among others.
With the focus on image processing technology, this paper studies driver fatigue and designs a driver fatigue detection model based on image processing. In Section 2, this article summarizes and discusses various researches by foreign scholars on image processing technology, driver fatigue and EEG signals. In Section 3.1, it introduces the definition and causes of driver fatigue, and introduces some methods of identifying fatigue. In Section 3.2, the paper introduces the Hough transform, AdaBoost, and PERCLOS algorithms, which can recognize a driver’s eyes and skin color. Section 4 explains the proposed driver fatigue detection model paper in many aspects. In Section 5, this paper analyzes the number of eye closure detection, PERCLOS detection, the detection rate of different algorithms on driver behavior, the influence of spectacles on detection, and the success rate of different algorithms under different degrees of lighting. The innovative aspects of this paper include use of image processing technology to detect the different states of a driver while driving, the design of an improved algorithm to improve the recognition rate of drivers’ fatigued state, and the division of driver fatigue into several levels in connection with image processing and testing in distinct time periods.

Related Work

Many scholars at home and abroad have conducted studies on driver fatigue detection models based on image processing. Furthermore, numerous scholars have adopted machine learning methods to analyze the driving behavior of drivers [1], such as detecting driving fatigue by analyzing facial features such as the driver’s eyes and mouth [24]. For example, Ed-Doughmi et al. [5] proposed a sequence of frame driver’s face, he applied recurrent neural network to analyze and predict the driver’s drowsiness, the accuracy rate was 92%. Meanwhile, Savas and Becerikli [6] proposed a multi-task convolutional neural network (ConNN*) model by calculating eye closure duration/percentage of eye closure (PERCLOS) and yawning frequency, i.e., Mouth Frequency (FOM), in order to detect driver sleepiness and fatigue with a recognition rate of 98%. Yet other scholars have studied methods of detecting driver fatigue based on physiological signals [7], such as the use of image processing techniques to analyze electoencephalography (EEG) signals [8]. Neural networks have been used to analyze the EEG signal to detect driver fatigue [9, 10]. However, EEG signals are easily affected by the actual environment, such as weather conditions and a driver’s physical state, so the recognition rate is very unstable. Due to the development of in-vehicle devices, it is relatively easy to obtain the driving behavior data of drivers, so methods of fatigue detection based on driving behavior have the advantages of higher stability and easy detection. Therefore, for the purposes of this paper, a fatigue driving detection model was designed and developed based on image processing to detect drivers’ state of fatigue by monitoring their driving behavior.
Generally speaking, driving fatigue refers to the generalization of physiological changes, psychological fatigue, and reduced driving ability in order to facilitate objective detection of changes caused by a driver’s continuous and prolonged driving. It is the normal law of change regarding both physiological and psychological activities. When driving a car, a driver is subject to both a physical load and a mental load. The physical load is similar to general labor, whereas the mental load reflects the sense of tension and pressure felt by the driver due to the external road traffic environment and his own inner mental cognitive state when performing driving or non-drivingrelated tasks [11, 12]. The main cause of driving fatigue is an excessive mental load. Because the driver sits in the main driving position, maintains the same posture for a long time, and is in a state of high mental stress, constantly receiving and processing external information, driving fatigue is extremely likely to occur.
Most of the factors that cause driving fatigue are attributable to a lack of self-awareness on the part of the driver. The so-called fatigued state is mainly caused by fatigue of the sense organs and nerves, as well as the long-term and continuous driving posture, which inhibits smooth circulation of the blood. When the driver sits in a fixed seat for a long time, his or her body movement will be restricted. Moreover, because the driver should maintain a high degree of concentration and observe the information outside the car in real time, it is difficult to avoid being overly nervous in the psychological state, resulting in blurred vision, lower back pain, slow response, and other driving fatigue phenomena. The main causes of fatigue originate in the eyes, neck, shoulders and waist. The main feeling of fatigue is caused by the eyes and brain. However, there are other factors that cause different degrees of driver fatigue. In summary, they mainly include the following:
(1) Driver fatigue caused by lack of sleep: The driver’s sleep quality directly affects his or her driving experience.
(2) Fatigue caused by driving a vehicle for a long time: When a driver is sat in the cab for a long time, with the eyes always staring at other vehicles on the road, soreness of the head and waist soon shows, simultaneously causing the blood circulation to be unsmooth.
(3) Physiological diseases and fatigue caused by alcohol: Although slight drinking behavior does not cause drunk driving, alcohol is a kind of drink that stimulates the nerves. After drinking, the brain and nervous system will gradually become unresponsive due to alcohol, which will cause the driver to drive in a fatigued state.
(4) Fatigue caused by improper diet: Some drivers eat too much before driving; some drivers eat foods that are too greasy, too difficult to digest, or too irritating; others drink strong tea and espresso directly before driving or while on the way. In addition, smoking is a common cause of fatigue [13, 14]. As shown in Table 1, there are several methods of identifying driver fatigue.

Table 1. Fatigue recognition technology

Detection technology Description Accuracy Practicality Scalability
Based on physiological signals Detects brain waves, heart rate, etc. Good Very poor General
Based on body response Detects the frequency of the driver's head tilt, eye closure, etc. Well Good Well
Based on driving behavior Detects the transformation of various manipulators. Good Well Very bad
Based on car behavior Detects the behavior of the car itself. Good Well Very bad
Based on driver response Checks the driver’s reaction every once in a while. Good Very poor Well
Based on driving conditions Detects driving time and driving conditions. Very poor Good Well

Research Method of Driver Fatigue Detection Model Based on Image Processing

Image Processing
In the process of collecting and transmitting images, they are likely to be interfered with by noise, and the brightness of the collected images will also be affected by external ambient light [15]. Fig. 1 shows the image collection system mode while driving. Image preprocessing refers to the filtering and denoising of an image, adjustment of the light balance before further processing, and analysis of the image. Its purpose is to eliminate interference factors such as noise and uneven lighting, improve image quality, and save useful details in the image, in order to enhance the convenience of subsequent image feature recognition and analysis [16].

Fig. 1. Image gathering system (pictures from a Baidu,https://image.baidu.com).

Image filtering is usually the first step in the image processing flow, which refers to a method of eliminating interference information in the image while retaining useful information such as edge information. When an image is disturbed by a noise signal generated in its acquisition or transmission, the quality of that image will be reduced to a significant extent, and even the integrity of the image information will be destroyed [17, 18]. Therefore, it is necessary to use image filtering methods to reduce the influence of noise signals and restore the details of the image as much as possible.
Common image noises include Gaussian noise and impulse noise. Gaussian noise is mainly caused by a high sensor temperature or insufficient light. Impulse noise generally refers to pixel dead pixels, including bright and dark pixels. Because it looks like salt and pepper particles, impulse noise is also called salt and pepper noise. Depending on the filtering method used, it can be divided into spatial filtering and frequency domain filtering. For (To + VERB INFINITIVE?) an image, the pixel value is regarded as a two-dimensional spatial function k(m, n), and (m, n) represents the position of the pixel. The spatial domain filtering method consists in setting a filter template, as parameters are required to calculate the parameters of the pixel neighborhood, and in moving the template pixel by pixel in the two-dimensional image, calculating the neighborhood response, and replacing the pixel in the template with it [19].
Traditional digital image filtering methods include mean filtering, median filtering and Gaussian filtering. Mean filtering is a linear filter whose template output response is the average value of pixels in the neighborhood of the template [20]. Median filtering is a nonlinear filter, whose basic function is to sort the statistics of pixels in the template area, that is, to arrange the gray values of all pixels in the neighborhood of the template in an orderly manner, and output the median as the response of the template. It has been shown that the median filter has a good processing effect on salt and pepper noise. Gaussian filtering is a linear smoothing filter, the main principle of which is to perform a weighted average in the neighborhood, in order to retain useful edge detail information in the image, increase the weight of the center parameter of the filter template, and reduce the weight of the edge pixels of the template. Based on this consideration, the Gaussian filter template can be obtained. However, it is necessary to avoid too big a difference between the pixel values of two adjacent pixels [21, 22].

Human Eye State Recognition
The eyes can most intuitively describe and reflect a driver’s state of fatigue. Based on completed human eye positioning, a direct and effective method needs to be established to identify and assess the state of the human eye and, ultimately, to detect driver fatigue. There are many methods of human eye state recognition, each with its own advantages and disadvantages. The following is a research taken from a study on several common eye state recognition methods.

Hough transform
Hough transform can effectively identify geometric figures in images. It was proposed by Paul Hough in 1962 and was first used to detect straight lines in binary images [23]. Later, the Hough transform was extended to detect a variety of shapes, such as arcs, ellipses, circles, and so on. Based on the characteristic that the pupil of the human eye can be regarded as a circle, this paper uses the Hough transform to identify the state of the human eye. If the circle is detected, it means that the eye is in an open state, and if the circle is not detected, it means that the eye is in a closed state [24].
The general expression for circles is as follows:

$(x-m)^2 + (y-n)^2 = r$(1)

This can be reduced to

$\begin{cases} m = x - r × cosθ \cr n = y - r × sinθ \end{cases}$(2)

To map the point $(x, y)$ in the image space to the parameter space $(m, n, r)$, one first needs to create a three-dimensional accumulation array $M$. The element coordinates can be written as $M(m, n, r)$. The straight line parameter space has one more dimension. When the coordinate point moves on the image to be detected, the coordinates $(m, n)$ of other pixel points on the circle with a radius of $r$ are calculated, and the cumulative calculation is performed. One then finds the largest $M(m0, n0, r)$ through the cumulative traversal method, and detects that the center of the circle in the original image is $(m0, n0)$ and the radius is r.

AdaBoost is a robust and adaptable/adaptive algorithm. It is a strong classifier sequence set and has a good classification effect. It is composed of a cascade of many weak classifiers. When used to detect human faces in video sequence images, the AdaBoost algorithm chooses to use Harr-like features and extended features, including edge features, linear features, and center features for training, and finally forms a classifier group. In the first stage, different weak classifiers will train different image examples and belong to the same classifier set, and then form an independent strong classifier. In the second stage, multiple strong classifiers form a sequence classifier set to recognize and detect faces. After the harr-like feature is used for statistical classification of human faces, each strong face classifier is trained and formed. For this classifier set, each node is a strong classifier. When an image sample cannot pass the current classifier, it will be moved to the next strong classifier for reclassification, so that iterative classification processor integration can improve the accuracy and detection rate of the final classification success [25].
First, select $k$ training samples $(i_1,j_1 ),…,(i_k,j_k)$, where $j_m=1$ corresponds to a positive samples $(faces) and j_m=0$ corresponds to b negative samples (non-faces), and $a+b=c$.
Initialize the weight value:

$l_{a,m} = \begin{cases} \frac {1}{2a},y_m = 1 \cr \frac {1}{2b},y_m = 0 \end{cases}$(3)

For the s-th classification training, the normalized weights are as follows:

$l_{s,m} ← l_{s,m} / \displaystyle\sum_{m=1}^{c}l_{s,m}$(4)

For each classification feature n, train its weak classifier v_((s,n)) (i), and calculate the classification error rate of the weak classifier on the sample set:

$σ_{s,n} = \displaystyle\sum_{m=1}^{c} 1_{s,n}(v_{(s,n)}(i_m)-j_m), n = 1,2,...,t$(5)

Find a weak classifier with the smallest error rate σ_s of classification from the set of weak classifiers trained in the previous step, and add it to the resulting strong classifier, denoted as v_s. The updated sample weight is:

$l_{s+1,m}=l_{s,m}r_s^{1-e_{m}}$(6)

The final training to form a strong classifier is:

(7)

Where

(8)

PERCLOS
The PERCLOS parameter was originally proposed by foreign experts when studying the relationship between eye change and fatigue in optics. There are three measurement methods for this measurement: P80, P70, and em (i.e., P50). These three are defined as follows: P80 represents the percentage the time in which the eyeball is covered by the eyelid for more than 80% of the time; P70 represents the ratio of the time in which the eyeball is covered by the eyelid for more than 70% of the time; and P50 represents the proportion of the time that the eye is covered by the eyelid for more than 50% [26]. PERCLOS represents the time of eye closure as a ratio of unit time to total unit time. Since the human eye is in a state of moderate fatigue, it is constantly changing from closed to slightly open. It is not until a state of deep fatigue sets in that the human eye closes completely and no longer opens. Therefore, the value of PERCLOS is calculated based on the series of states when the human eye is closed and opened. Fig. 2 is a diagram of PERCLOS measurement principle.

Fig. 2. Diagram of PERCLOS measurement principle.

The calculation formula for the value of PERCLOS is:

$k = \frac{t_3 - t_2}{t_r - t_1} × 100%$(9)

Among them, $k$ represents the size of the PERCLOS value, that is, the ratio of eye closure time in unit time. $t_1$ means the time it takes for the eyes to close from the largest pupil to 80% of the remaining pupil; $t_2$ means the time it takes for the eyes to close from 80% of the pupils to 20% of the pupils; $t_3$ represents the time taken for the remaining 20% of the pupils to complete full closure of the eyes to their opening again to the remaining 20% of the pupils; and $t_4$ is the time taken from the opening of the remaining 20% of the pupils to the remaining 80% of the pupils [27].
Because the time in the video and the image sequence correspond to each other, the ratio of the eye closure time to the total time is calculated by the number of frames:

$PERCLOS = \frac{FNsum}{Fsum}$(10)

In this regard, the sum $F$ is the number of effective freeze frames per unit of time, and the sum $FN$ is the number of frames per unit of time in terms of eyeball closure.

Skin Tone Model
Similar to other mathematical models, the skin color model uses an analytical method to indicate which pixels belong to skin color, that is, to calculate the similarity between the pixels in the image and the skin color. Moreover, since the sampling points are discrete, it cannot reliably determine whether a color is skin-colored by the $𝐴𝑐$ and $𝐴𝑑$ components of the pixel directly, which would require a mathematical model applicable to different skin colors and lighting conditions, in which the input represents the 𝐴𝑐 and 𝐴𝑑 values of the sampled pixels, and the output represents the probability of using color or setting a template to determine the threshold output [28].
Skin color approximately obeys a two-dimensional Gaussian distribution $𝐻(𝜐, 𝐴)$ in the color space, and its probability density function can be expressed as follows:

$t(\overline m) = \frac{1}{\sqrt{2π}|A|^{\frac{1}{2}}}exp \left\{ -\frac{1}{2}(\overline m - v)^k A^{-1}(\overline m - v) \right\}$(11)

Among them, $\overline m = (𝐴𝑐, 𝐴𝑑)$ is the two-dimensional chromaticity vector, A represents the mean vector and covariance matrix, and $n$ is the number of statistical samples. The calculation formula is:

$v = \frac{1}{r}\displaystyle\sum_{t=1}^{r}\overline {m_t}$(12)

$A = \displaystyle\sum_{t=1}^{r}(\overline {m_t}-v)(\overline {m_t}-v)^K$(13)

After statistical calculation, the mean and variance of the two-dimensional Gaussian model of skin
color distribution are:

(14)

After determining the skin color Gaussian model, the similarity 𝐷(𝐴𝑐, 𝐴𝑑) between each pixel in the image to be detected and the skin color can be calculated, i.e., the probability value that the pixel belongs to the facial skin, as follows:

$D(AC,AD)=exp[-0.5 × (\overline m-v)^K A^{-1}(\overline m-v)]$(15)

Among them, $\overline m = (𝐴𝑐, 𝐴𝑑)$ is the two-dimensional chromaticity vector of the pixels in the image [29], $𝜐$ and $𝐴$ is the value calculated by the above formula.

Research Experiment on a Driver Fatigue Detection Model Based on Image Processing

Experimental Design
The status of a driver’s facial features, such as the eyes and mouth, are important clues when analyzing whether a driver is in a state of fatigue. Therefore, it is necessary to accurately detect the eyes and mouth, and then analyze their status. Based on the results of analysis, it can be judged whether a driver is fatigued ornot. When analyzing the video image of a driver, the driver’s eyes and mouth are detected and located, and thedriver’s state is evaluated. For the detection and positioning of the eyes and mouth of a driver driving at night, infrared lighting and infrared imaging cameras can be used to collect video images of the driver, in order to detect, identify the positioning, and assess the state of the driver’s eyes and mouth whatever the weather conditions.
According to the existing experimental conditions, this paper mainly studies the detection and positioning of a driver’s eyes and mouth and the problem of evaluating a driver’s state by analyzing the video image of a driver under natural lighting conditions. For the detection and positioning of the eyes and mouth of a driver driving at night, infrared lighting and infrared imaging cameras can be used to collect video images of the driver, and further research can be conducted on the basis of this paper, in order to be able to detect and identify the positioning of the eyes and mouth of a driver working all-weather and state judgment.
The system is composed of three parts: video collection, fatigue detection, and alarm display. Fig. 3 is a diagram of the system structure. Video capture uses a color camera device fixed on the front dashboard of the cab to shoot a video of the driver in different postures and mental states in a variety of lighting environments. The driver’s face is located in the middle of the entire image and occupies a larger part of the area. Fatigue detection mainly includes three parts: video stream decompression and frame grabbing, eye state detection, and statistical discrimination. The alarm display consists of red and yellow alarm lights that indicate “warning” and “normal” states, respectively.

Fig. 3. Schematic diagram of the structure of the driver fatigue detection system.

Data Collection
The facial area in the image is mainly divided into the skin part and the non-skin part. The non-skin part includes the hair, eyes, mouth, nostrils, non-human objects/ornaments, background, etc. The skin area is the largest area of the face. Based on this feature, a skin color segmentation algorithm can be used to binarize the image of the facial area. The skin area becomes white, and the other areas are black, so the eye area must be black, which greatly reduces the detection range.
It is necessary to complete the collection of experimental data and to use a standard face database as a benchmark to form the training sample and test sample set for the experiment. This study selected the FERET face database, which contains 14,051 multi-posture and illuminated gray-scale face images, making it one of the most extensive face databases in the field of facial recognition. All the videos and images of drivers used in this study were selected from this database.

Experimental Environment
The proposed fatigue driving state detection system uses Microsoft Visual Studio 2010 as the development environment, and also uses the Windows 7.0 operating system and the open source computer vision library OpenCV 2.4.9 to pre-process the collected experimental data and configure the development environment.

Experimental Method
The calculation algorithms involved in the driver fatigue detection system are the AdaBoost algorithm and the PERCLOS algorithm, which are combined to process the collected data.
The collected data parameters include the number of closed eyes, the duration of eye closure, the PERCLOS value, the detection rate, and the number of frames. In addition, for these videos and images, certain actions of the driver were selected, i.e., normal driving, turning of the head, head down, eye closure, etc., and the variables were controlled to detect the difference between wearing spectacles and not wearing spectacles, and variations in light.

Driver Fatigue Detection Model Analysis Based on Image Processing

Detection of the Number of Eye Closures
In order to verify the method of detecting driver fatigue, four experiments were designed to simulate driver fatigue, each of which lasted for 60 seconds, and also used a researcher for manual detection.
Table 2 shows that the average rate of accuracy of the proposed detection algorithm in processing these five images is 85.19%, and that the degree of accuracy is not particularly stable, which is attributable to the fact that the image will sometimes be unstable. However, because the proposed detection system judges the total number of frames within 60seconds, the unstable situation of a single image does not affect the detection of driver fatigue.

Table 2. Detection of the number of eye closures
Sample Actual number of eye closures Detection of number of eye closures Accuracy (%)
Video 1 35 43 82.87
Video 2 21 23 89.45
Video 3 43 49 85.34
Video 4 76 88 83.09
Sum 175 203 85.19

PERCLOS Detection
For the purposes of this study, the video sequence used a simulated driving environment in a laboratory, and was collected by a COMS color camera with a sampling rate of 15 frames per second. There was no large-scale rotation. The head rotation angle did not exceed 15°, and the experiment passed two test subjects, obtained two sets of video images, and performed the duty cycle PERCLOS detection calculation. The images include both non-fatigue videos and fatigue videos.
It can be seen from Fig. 4 that the PERCLOS value of the tester in the awakened state is significantly smaller than the PERCLOS value in the fatigued state. For example, the value of Tester 1 in the time period 1 is 0.173 when awake; and the value in the same time period when the tester was fatigued is 0.523. The reason for this phenomenon is that in the awake state, the driver’s eyes closed for a short time and blinked faster, whereas, in the fatigued state, the driver’s blinking speed slowed down and the eyes closed for longer periods of time. We can see that when the driver is detected, the outline of the human eye becomes blurred, and there are obvious boundaries in the illumination part, which will cause great interference to the subsequent integral projection. This error needs to be reduced.

Fig. 4. PERCLOS value of the tester in different states.

Table 3 shows the assessment result of six videos. In the detection of these six videos, most of the systems discussed in this paper can be used to determine correctness. Therefore, PERCLOS can ideally determine the driving state of the driver, and provide a real-time warning of driver fatigue.

Table 3. Result of using PERCLOS to determine the status of the driver
Sample Actual state of sample Total video frames Number of frames with eyes closed PERCLOS value (%) Judgment state
Video 1 Mild fatigue 205 31 15.23 Mild fatigue
Video 2 Moderate fatigue 205 42 20.71 Moderate fatigue
Video 3 Severe fatigue 205 58 26.38 Severe fatigue
Video 4 Fatigue 775 25 3.29 Fatigue
Video 5 Awaken 205 5 2.32 Awake
Video 6 Keep eyes closed for more than 2 seconds 205 39 22.87 Moderate fatigue

Detection Rates of Driver Behavior of Different Algorithms
In order to compare the superiority of the algorithm proposed in this paper, experiments were conducted used different algorithms in order to detect the different actions of drivers during driving and thereby test the efficiency of different algorithms. This study used the image processing method, the model matching method, and the proposed method. The gray-scale processing method detects the eyes of the driver. The different actions of the driver include “normal driving,” “talking,” and “bowing and turning the head.”
Fig. 5 compares the results of eye detection by the three algorithms. It can be seen that the proposed algorithm based on image processing has a higher detection rate than both the gray projection method and the template matching method. When a driver drives a vehicle normally, the rate of accuracy of the proposed algorithm is 92.5%, while that of the model matching method is 88.7%, and that of the gray projection method is just 73.5%. When detecting the ‘head turning’ behavior, the accuracy of the three algorithms was observed to be lowest, because when the amplitude of the head rotation increased, the area visible to the eye during image detection was very small, causing a drop in accuracy.

Fig. 5. Results of comparison of eye test.

Impact of Spectacles on Detection
It can be seen from Table 4 that the proposed detection model based on image processing has a high rate of detection accuracy when a driver is driving normally, regardless of whether spectacles are worn or not. The accuracy rate is 92.23% when the driver is wearing spectacles, and accurate when not wearing spectacles. The rate is 94.98%. When a driver without spectacles turns or lowers his head, a high rate of accuracy can be maintained; however, the rate of accuracy is significantly reduced for drivers who are wearing spectacles. This is because when a driver wears spectacles, the angle of the head will shift, resulting in false detection. However, in actual situations, drivers generally face forward, so the proposed system can still maintain stable accuracy when detecting driver fatigue.

Table 4. Result statistics of human eye positioning
Wearing spectacles Not wearing spectacles Wearing spectacles Not wearing spectacles Wearing spectacles Not wearing spectacles
Total number of frames collected by the camera 458 487 439 488 459 483
Number of frames for successful human eye positioning 422 458 354 449 378 422
Detection accuracy (%) 92.23 94.98 81.36 91.88 79.03 93.91

Success Rate of Different Algorithms under Different Lighting
When a driver is driving a vehicle, the eyes will also show different open and closed states depending on differences in the light. In this experiment, 100 sample pictures were selected from the database.
Fig. 6 shows the success rate of the different detection methods. It can be seen that the three methods maintained a good success rate during normal illumination and when the driver’s posture remain unchanged. However, the success rate of two methods, the model matching method and the grayscale processing method, was observed to decrease when the light or the state of the driver’s body changed. Under abnormal light conditions, the proposed method showed a success rate of 92.34%, the model matching method, 81.03%, and the gray processing method, 82.95%; while, with regard to changes in the driver’s posture, the proposed method showed a success rate of 91.04%, the model matching method, 83.19%, and the gray processing method, 84.18%. These two sets of results clearly show that the success rate of the algorithm proposed inn this study is significantly higher than that of the other two methods. In summary, the proposed method can maintain a good success rate under different degrees of illumination and different driver postures.

Fig. 6. Diagram showing the success rate of different methods.

Fig. 7 shows the number and duration of eye closures under different conditions. Analysis shows that a change in the percentage of eye closures is closely related to the degree of fatigue of the human body, and generally shows a positive correlation. In general, the percentage of eye closures of drivers in the ‘awake’ state in each time period also shows the basic feature differences. Although there are differences, the overall stay within the controllable range, so the degree of driver fatigue can be determined. It can be reported that this method is capable of judging driver fatigue to a certain extent and provide an early warning of driver fatigue, which will be of great practical value and significance.

Fig. 7. Degree of eye closure in different states.

Conclusion

Regarding driving fatigue, if certain measures are taken to detect a driver’s status in real time, and an alarm is issued to achieve the effect of warning the driver, traffic accidents could be reduced significantly. As image processing technology can effectively realize facial recognition, the status of a driver’s eyes, limbs and other body parts can be recognized by this technology, making it possible to assess the driver’s state of fatigue. In order to solve the major damages and harm resulting from traffic accidents caused by driving fatigue, this paper proposes a driver fatigue detection model based on image processing, which can promptly and accurately send alarm signals to drivers who are fatigued or inattentive. However, this paper has the following shortcomings: because an image processing method is used, the algorithm is only effective with drivers who wear ordinary spectacles, and is invalid completely ineffective with drivers who wear sunglasses. At the same time, the eye detection algorithm cannot be used in serious side-face situations. Areas related to driver fatigue other than the eyes have not yet been detected, such as the opening and closing of the mouth; and since all the experiments and algorithm procedures in this study were carried out on a PC, real-time detection of driver fatigue will require embedded programming for the program; therefore, further design and perfection are needed to resolve the migration problem of the follow-up program in order to ensure accuracy. As for future research, due to the relatively complex detection algorithms and the huge amount of data, it will be necessary to study the real-time problem of the system, starting from two aspects: The first consists in simplifying the processing algorithms as much as possible from the software aspect, so as to optimize the calculation structure, and in using parallel algorithms instead of serial algorithms. The second relates to hardware, and consists in improving hardware performance or optimizing hardware structure, such as the use of multiple computing centers.

Author’s Contributions

Writing of the original draft, ZM. Software, JY. Data curation, LJ. Writing of the review and editing, QW.

Funding

This work was supported by the Jiangxi Education and Teaching Reform Project (No. JXJG-17-24-12), Jiangxi Natural Science Foundation (No. 20202BABL202031), Science and Technology Project of the Jiangxi Provincial Department of Education (No. GJJ202001).

Competing Interests

The authors declare that they have no competing interests.

Author Biography

Name : Zhendong Mu
Affiliation : The Center of Collaboration and Innovation, Jiangxi University of Technology, Nanchang 330098, Jiangxi, China.
Biography : Zhendong Mu (1975.11), male, graduated from Nanchang University with a master's degree. Working in Jiangxi University of Technology, professor.Mainly engaged in intelligent computing and brain-computer interface research.

Name : Ling Jin
Affiliation : The Center of Collaboration and Innovation, Jiangxi University of Technology, Nanchang 330098, Jiangxi, China.
Biography : Ling Jin (1979.03), famale, graduated from Nanchang University with a master's degree. Working in Jiangxi University of Technology, associate professor.Mainly engaged in intelligent computing and computer teaching.

Name : Jinghai Yin
Affiliation : The Center of Collaboration and Innovation, Jiangxi University of Technology, Nanchang 330098, Jiangxi, China.
Biography : Jinghai Yin received his master’s degree from Nanchang University in 2005. He is currently a Professor at Jiangxi University of Technology. His research interests are mainly in brain computer interface and artificial intelligence.

Name : Qingjun Wang
Affiliation : College of Economics and Management, Shenyang Aerospace University, Shenyang, 110136, Liaoning, China.Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China.
Biography : Qingjun Wang, eceived his M.S. degree from the Northeast University, in 2009. He is currently a graduate student studying for Ph.D. degree in the College of Automation Engineering, Nanjing University of Aeronautics and Astronautics. His research interests include Pattern Recognition, Artificial Intelligent.

References

[1] R. Bhalarao and M. Raval, “Automated tabla syllable transcription using image processing techniques,” Multimedia Tools and Applications, vol. 79, no. 39, pp. 28885-28899, 2020.
[2] R. Galindo, W. G. Aguilar, and R. P. R. Ch, “Landmark based eye ratio estimation for driver fatigue detection,” in Intelligent Robotics and Applications. Cham, Switzerland: Springer, 2019, pp. 565-576.
[3] R. Huang, Y. Wang, Z. Li, Z. Lei, and Y. Xu, “RF-DCM: multi-granularity deep convolutional model based on feature recalibration and fusion for driver fatigue detection,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 1, pp. 630-640, 2020.
[4] J. Wang, X. Yu, Q. Liu, and Z. Yang, “Research on key technologies of intelligent transportation based on image recognition and anti-fatigue driving,” EURASIP Journal on Image and Video Processing, vol. 2019, article no. 33, 2019.https://doi.org/10.1186/s13640-018-0403-6
[5] Y. Ed-Doughmi, N. Idrissi, and Y. Hbali, “Real-time system for driver fatigue detection based on a recurrent neuronal network,” Journal of Imaging, vol. 6, no. 3, article no. 8, 2020.https://doi.org/10.3390/jimaging6030008
[6] B. K. Savas and Y. Becerikli, “Real time driver fatigue detection system based on multi-task ConNN,” IEEE Access, vol. 8, pp. 12491-12498, 2020.
[7] C. Zhang, L. Sun, F. Cong, T. Kujala, T. Ristaniemi, and T. Parviainen, “Optimal imaging of multi-channel EEG features based on a novel clustering technique for driver fatigue detection,” Biomedical Signal Processing and Control, vol. 62, article no. 102103, 2020.https://doi.org/10.1016/j.bspc.2020.102103
[8] N. S. Karuppusamy and B. Y. Kang, “Multimodal system to detect driver fatigue using EEG, gyroscope, and image processing,” IEEE Access, vol. 8, pp. 129645-129667, 2020.
[9] T. Tuncer, S. Dogan, F. Ertam, and A. Subasi, “A dynamic center and multi threshold point based stable feature extraction network for driver fatigue detection utilizing EEG signals,” Cognitive Neurodynamics, vol. 15, pp. 223-237, 2021.
[10] Y. Yang, Z. Gao, Y. Li, Q. Cai, N. Marwan, and J. Kurths, “A complex network-based broad learning system for detecting driver fatigue from EEG signals,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 51, no. 9, pp. 5800-5808, 2021.
[11] G. Wu, B. Masia, A. Jarabo, Y. Zhang, L. Wang, Q. Dai, T. Chai, and Y. Liu, “Light field image processing: an overview,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 7, pp. 926-954, 2017.
[12] I. R. Kamel, C. Georgiades, and E. K. Fishman, “Incremental value of advanced image processing of multislice computed tomography data in the evaluation of hypervascular liver lesions,” Journal of Computer Assisted Tomography, vol. 27, no. 4, pp. 652-656, 2003.
[13] X. Y. Cui, Z. G. Gui, Q. Zhang, H. Shangguan, and A. H. Wang, “Learning-based artifact removal via image decomposition for low-dose CT image processing,” IEEE Transactions on Nuclear Science, vol. 63, no. 3, pp. 1860-1873, 2016.
[14] K. Aoki, S. Yanagi, M. Kokubo, M. Matsuda, K. Ohara, and M. Miyazawa, “Server for implementing image processing functions requested by a printing device,” U.S. Patent 9509863, Nov 29, 2016
[15] G. Cheung, E. Magli, Y. Tanaka, and M. K. Ng, “Graph spectral image processing,” Proceedings of the IEEE, vol. 106, no. 5, pp. 907-930, 2018.
[16] T. Matsuyama, “Image processing device, image processing method, and computer-readable storage medium,” U.S. Patent 9292240, Mar 22, 2016.
[17] S. A. Tsaftaris, M. Minervini, and H. Scharr, “Machine learning for plant phenotyping needs image processing,” Trends in Plant Science, vol. 21, no. 12, pp. 989-991, 2016.
[18] F. Erden, S. Velipasalar, A. Z. Alkar, and A. E. Cetin, “Sensors in assisted living: a survey of signal and image processing methods,” IEEE Signal Processing Magazine, vol. 33, no. 2, pp. 36-44, 2016.
[19] M. Bergounioux, “Mathematical analysis of a inf-convolution model for image processing,” Journal of Optimization Theory and Applications, vol. 168, no. 1, pp. 1-21, 2016.
[20] Q. Yang, D. Chen, T. Zhao, and Y. Chen, “Fractional calculus in image processing: a review,” Fractional Calculus and Applied Analysis, vol. 19, no. 5, pp. 1222-1249, 2016.
[21] R. Chai, G. R. Naik, T. N. Nguyen, S. H. Ling, Y. Tran, A. Craig, and H. T. Nguyen, “Driver fatigue classification with independent component by entropy rate bound minimization analysis in an EEG-based system,” IEEE Journal of Biomedical and Health Informatics, vol. 21, no. 3, pp. 715-724, 2017.
[22] J. Xu, J. Min, and J. Hu, “Real-time eye tracking for the assessment of driver fatigue,” Healthcare Technology Letters, vol. 5, no. 2, pp. 54-58, 2018.
[23] N. S. Karuppusamy and B. Y. Kang, “Driver fatigue prediction using EEG for autonomous vehicle,” Advanced Science Letters, vol. 23, no. 10, pp. 9561-9564, 2017.
[24] A. Adam, Z. Ibrahim, N. Mokhtar, M. I. Shapiai, P. Cumming, and M. Mubin, “Evaluation of different time domain peak models using extreme learning machine-based peak detection for EEG signal,” SpringerPlus, vol. 5, article no. 1036, 2016.https://doi.org/10.1186/s40064-016-2697-0
[25] F. Zubler, C. Koenig, A. Steimer, S. M. Jakob, K. A. Schindler, and H. Gast, “Prognostic and diagnostic value of EEG signal coupling measures in coma,” Clinical Neurophysiology, vol. 127, no. 8, pp. 2942-2952, 2016.
[26] V. Laprevote, L. Bon, J. Krieg, T. Schwitzer, S. Bourion-Bedes, L. Maillard, and R. Schwan, “Association between increased EEG signal complexity and cannabis dependence,” European Neuropsychopharmacology, vol. 27, no. 12, pp. 1216-1222, 2017.
[27] W. X. Wang, S. Q. Sun, C. Li, and Z. C. Tang, “Recognition of upper limb motion intention of EEG signal based on convolutional neural network,” Journal of Zhejiang University (Engineering Science), vol. 51, no. 7, pp. 1381-1389, 2017.
[28] Y. Mitsukura, “EEG signal processing for real applications,” Journal of Signal Processing, vol. 20, no. 1, pp. 1-7, 2016.
[29] A. Broniec, “Analysis of EEG signal by flicker-noise spectroscopy: identification of right-/left-hand movement imagination,” Medical & Biological Engineering & Computing, vol. 54, no. 12, pp. 1935-1947, 2016.

Zhendong Mu1,*, Ling Jin1, Jinghai Yin1, and Qingjun Wang2,3, Research on a Driver Fatigue Detection Model Based on Image Processing, Article number: 12:17 (2022) Cite this article 1 Accesses