Enhancing Driver Stress Detection through Multimodal Integration of Eye Tracking and Physiological Signals
DOI:
https://doi.org/10.61978/logistica.v3i3.1147Keywords:
Driver Stress, Eye Tracking, Physiological Monitoring, Multimodal Integration, Driver Monitoring System, Cognitive Workload, Real-Time ClassificationAbstract
Driver stress poses significant risks to traffic safety, impairing attention, decision-making, and reaction time. Traditional monitoring methods often lack sensitivity. This study proposes and validates a novel multimodal framework that integrates synchronized eye-tracking and physiological data to significantly enhance the sensitivity and real-time accuracy of driver stress detection, addressing limitations of earlier unimodal approaches. Thirty licensed drivers participated in simulated driving tasks under baseline and stress-induced conditions. Eye-tracking metrics (pupil diameter, fixation duration, blink rate) and physiological signals (heart rate, skin conductance, heart rate variability) were collected. Data were synchronized and analyzed using Linear Discriminant Analysis (LDA) and other machine learning models to classify stress conditions. Under stress, pupil dilation increased by 20%, blink rate rose by 35%, and gaze spread narrowed, indicating visual tunneling. Physiologically, heart rate increased by 17%, skin conductance by 31%, and HRV decreased by 19%. The combined multimodal model achieved 91.4% classification accuracy, outperforming unimodal approaches. These results align with previous research showing that multimodal systems provide more reliable stress detection by integrating visual and autonomic markers. The findings highlight the system’s potential for real-time applications in Driver Monitoring Systems (DMS). Multimodal integration of eye-tracking and physiological signals enhances the sensitivity and reliability of driver stress detection. This approach offers a foundation for intelligent, adaptive DMS capable of improving road safety. Future work should focus on real-world validation and ethical implementation strategies. These findings demonstrate that multimodal integration provides a more comprehensive understanding of driver stress through complementary visual and autonomic indicators. The proposed framework forms a foundation for intelligent, adaptive Driver Monitoring Systems (DMS) capable of real-time stress recognition and proactive safety intervention.
References
Ahmad, M., Keller, I., Robb, D. A., & Lohan, K. S. (2020). A Framework to Estimate Cognitive Load Using Physiological Data. Personal and Ubiquitous Computing, 27(6), 2027–2041. https://doi.org/10.1007/s00779-020-01455-7 DOI: https://doi.org/10.1007/s00779-020-01455-7
Ahmadi, N., Sasangohar, F., Yang, J., Yu, D., Danesh, V., Klahn, S., & Masud, F. (2022). Quantifying Workload and Stress in Intensive Care Unit Nurses: Preliminary Evaluation Using Continuous Eye-Tracking. Human Factors the Journal of the Human Factors and Ergonomics Society, 66(3), 714–728. https://doi.org/10.1177/00187208221085335 DOI: https://doi.org/10.1177/00187208221085335
Balakrishnan, A., Rodríguez, S., & Reynaud, R. (2020). Integrity Monitoring of Multimodal Perception System for Vehicle Localization. Sensors, 20(16), 4654. https://doi.org/10.3390/s20164654 DOI: https://doi.org/10.3390/s20164654
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2021). A Survey on Deep Multimodal Learning for Computer Vision: Advances, Trends, Applications, and Datasets. The Visual Computer, 38(8), 2939–2970. https://doi.org/10.1007/s00371-021-02166-7 DOI: https://doi.org/10.1007/s00371-021-02166-7
Brouwer, A.-M., Hogervorst, M. A., Oudejans, B., Ries, A. J., & Touryan, J. (2017). EEG and Eye Tracking Signatures of Target Encoding During Structured Visual Search. Frontiers in Human Neuroscience, 11. https://doi.org/10.3389/fnhum.2017.00264 DOI: https://doi.org/10.3389/fnhum.2017.00264
Dogan, D., Boğosyan, S., & Acarman, T. (2022). Evaluation of Takeover Time Performance of Drivers in Partially Autonomous Vehicles Using a Wearable Sensor. Journal of Sensors, 2022, 1–31. https://doi.org/10.1155/2022/7924444 DOI: https://doi.org/10.1155/2022/7924444
Fan, L., Wang, J., Li, Q., Song, Z., Dong, J., Bao, F., & Wang, X. (2023). Eye Movement Characteristics and Visual Fatigue Assessment of Virtual Reality Games With Different Interaction Modes. Frontiers in Neuroscience, 17. https://doi.org/10.3389/fnins.2023.1173127 DOI: https://doi.org/10.3389/fnins.2023.1173127
Feng, J., Zhang, R., & Liang, J. (2022). A Survey to Applications of Deep Learning in Autonomous Driving. https://doi.org/10.1117/12.2645702 DOI: https://doi.org/10.1117/12.2645702
Gabrielli, L., Migliorelli, L., Cantarini, M., Mancini, A., & Squartini, S. (2024). An Advanced Multimodal Driver-Assistance Prototype for Emergency-Vehicle Detection. Integrated Computer-Aided Engineering, 31(4), 381–399. https://doi.org/10.3233/ica-240733 DOI: https://doi.org/10.3233/ICA-240733
Gall, C. v., Muth, T., & Angerer, P. (2023). Sleep Duration on Workdays Is Correlated With Subjective Workload and Subjective Impact of High Workload on Sleep in Young Healthy Adults. Brain Sciences, 13(5), 818. https://doi.org/10.3390/brainsci13050818 DOI: https://doi.org/10.3390/brainsci13050818
Gao, H., & Kasneci, E. (2022). Eye‐Tracking‐Based Prediction of User Experience in VR Locomotion Using Machine Learning. Computer Graphics Forum, 41(7), 589–599. https://doi.org/10.1111/cgf.14703 DOI: https://doi.org/10.1111/cgf.14703
Guo, Y., Freer, D., Deligianni, F., & Yang, G. (2022). Eye-Tracking for Performance Evaluation and Workload Estimation in Space Telerobotic Training. Ieee Transactions on Human-Machine Systems, 52(1), 1–11. https://doi.org/10.1109/thms.2021.3107519 DOI: https://doi.org/10.1109/THMS.2021.3107519
Han, L., Du, Z., & Wu, K.-L. (2023). Evaluation of Traffic Signs Information Volume at Highway Tunnel Entrance Zone Based on Driver’s Visual Characteristics. Transportation Research Record Journal of the Transportation Research Board, 2678(6), 804–823. https://doi.org/10.1177/03611981231200228 DOI: https://doi.org/10.1177/03611981231200228
Hebbar, A., Bhattacharya, K., Prabhakar, G., Pashilkar, A. A., & Biswas, P. (2021). Correlation Between Physiological and Performance-Based Metrics to Estimate Pilots’ Cognitive Workload. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.555446 DOI: https://doi.org/10.3389/fpsyg.2021.555446
Huang, J., Zhang, Q., Zhang, T., Wang, T., & Tao, D. (2024). Assessment of Drivers’ Mental Workload by Multimodal Measures During Auditory-Based Dual-Task Driving Scenarios. Sensors, 24(3), 1041. https://doi.org/10.3390/s24031041 DOI: https://doi.org/10.3390/s24031041
Hughes, A. M., Hancock, G. M., Marlow, S. L., Stowers, K., & Salas, E. (2019). Cardiac Measures of Cognitive Workload: A Meta-Analysis. Human Factors the Journal of the Human Factors and Ergonomics Society, 61(3), 393–414. https://doi.org/10.1177/0018720819830553 DOI: https://doi.org/10.1177/0018720819830553
Jung, Y., Hwang, J., & Lee, J.-H. (2024). The Effects of Emotional Distress on Attentional Bias Toward Cigarette Warnings According to Smokers’ Anxiety Levels. Frontiers in Psychology, 15. https://doi.org/10.3389/fpsyg.2024.1411747 DOI: https://doi.org/10.3389/fpsyg.2024.1411747
Lee, Y., & Winston, F. K. (2016). Stress Induction Techniques in a Driving Simulator and Reactions From Newly Licensed Drivers. Transportation Research Part F Traffic Psychology and Behaviour, 42, 44–55. https://doi.org/10.1016/j.trf.2016.06.019 DOI: https://doi.org/10.1016/j.trf.2016.06.019
Liang, P. P., Lyu, Y., Chhablani, G., Jain, N., Deng, Z., Wang, X., Morency, L., & Salakhutdinov, R. (2022). MultiViz: Towards Visualizing and Understanding Multimodal Models. https://doi.org/10.48550/arxiv.2207.00056
Liu, T., Liu, L., Gou, G., Fang, Z., Sun, J., Chen, J., Cheng, J., Han, M., Ma, T., Liu, C., & Xue, N. (2023). Recent Advancements in Physiological, Biochemical, and Multimodal Sensors Based on Flexible Substrates: Strategies, Technologies, and Integrations. Acs Applied Materials & Interfaces, 15(18), 21721–21745. https://doi.org/10.1021/acsami.3c02690 DOI: https://doi.org/10.1021/acsami.3c02690
Lyu, Z., Chun-hua, Q., Zhu, S., & Wang, H. (2022). The Visual Scanning Behavior and Mental Workload of Drivers at Prairie Highway Intersections With Different Characteristics. Ieee Access, 10, 123043–123056. https://doi.org/10.1109/access.2022.3223653 DOI: https://doi.org/10.1109/ACCESS.2022.3223653
Ma, Y., Qi, S., Zhang, Y., Lian, G., Lu, W., & Chan, C. (2020). Drivers’ Visual Attention Characteristics Under Different Cognitive Workloads: An on-Road Driving Behavior Study. International Journal of Environmental Research and Public Health, 17(15), 5366. https://doi.org/10.3390/ijerph17155366 DOI: https://doi.org/10.3390/ijerph17155366
Nweke, H. F., Wah, T. Y., Mujtaba, G., Alo, U. R., & Al-Garadi, M. A. (2019). Multi-Sensor Fusion Based on Multiple Classifier Systems for Human Activity Identification. Human-Centric Computing and Information Sciences, 9(1). https://doi.org/10.1186/s13673-019-0194-5 DOI: https://doi.org/10.1186/s13673-019-0194-5
Ramirez, J. P., & Hamaza, S. (2023). Multimodal Locomotion: Next Generation Aerial–Terrestrial Mobile Robotics. Advanced Intelligent Systems, 7(7). https://doi.org/10.1002/aisy.202300327 DOI: https://doi.org/10.1002/aisy.202300327
Rothkegel, L. O. M., Schütt, H. H., Trukenbrod, H. A., Wichmann, F. A., & Engbert, R. (2019). Searchers Adjust Their Eye-Movement Dynamics to Target Characteristics in Natural Scenes. Scientific Reports, 9(1). https://doi.org/10.1038/s41598-018-37548-w DOI: https://doi.org/10.1038/s41598-018-37548-w
Sauerbeck, F., Huch, S., Fent, F., Karle, P., Kulmer, D., & Betz, J. (2023). Learn to See Fast: Lessons Learned From Autonomous Racing on How to Develop Perception Systems. Ieee Access, 11, 44034–44050. https://doi.org/10.1109/access.2023.3272750 DOI: https://doi.org/10.1109/ACCESS.2023.3272750
Shi, Y., Maskani, J., Caruso, G., & Bordegoni, M. (2019). Explore User Behaviour in Semi-Autonomous Driving. Proceedings of the Design Society International Conference on Engineering Design, 1(1), 3871–3880. https://doi.org/10.1017/dsi.2019.394 DOI: https://doi.org/10.1017/dsi.2019.394
Sim, S., & Kim, C. (2024). Proposal of a Cost-Effective and Adaptive Customized Driver Inattention Detection Model Using Time Series Analysis and Computer Vision. World Electric Vehicle Journal, 15(9), 400. https://doi.org/10.3390/wevj15090400 DOI: https://doi.org/10.3390/wevj15090400
Song, D., Wu, J., Yang, D., Chen, H., & Zeng, X. (2022). An Active Multiobjective Real-Time Vibration Control Algorithm for Parallel Hybrid Electric Vehicle. Proceedings of the Institution of Mechanical Engineers Part D Journal of Automobile Engineering, 237(1), 21–33. https://doi.org/10.1177/09544070221130122 DOI: https://doi.org/10.1177/09544070221130122
Stocco, A. (2021). Mind the Gap! A Study on the Transferability of Virtual vs Physical-World Testing of Autonomous Driving Systems. https://doi.org/10.48550/arxiv.2112.11255
Wu, C., Liu, Y., Guo, X., Ma, W.-F., & Zhu, T. (2021). Mental Workload Analysis Associated With Emotional Design in E-Learning Contexts: Combining EEG and Eye-Tracking Data. https://doi.org/10.31234/osf.io/aryud DOI: https://doi.org/10.31234/osf.io/aryud
Zhang, G., & Etemad, A. (2021). Capsule Attention for Multimodal EEG-EOG Representation Learning With Application to Driver Vigilance Estimation. Ieee Transactions on Neural Systems and Rehabilitation Engineering, 29, 1138–1149. https://doi.org/10.1109/tnsre.2021.3089594 DOI: https://doi.org/10.1109/TNSRE.2021.3089594
Zhao, C., Liu, N., Li, S., & Zhao, X. (2024). Investigation of Eye Movement Characteristics During Free Throws at Varying Intensities Among Basketball Players and Its Correlation With Free Throw Percentage. Plos One, 19(8), e0299938. https://doi.org/10.1371/journal.pone.0299938 DOI: https://doi.org/10.1371/journal.pone.0299938
