Online Eye Status Detection in the Wild with Convolutional Neural Networks

Essa R. Anas, Pedro Henriquez, Bogdan J. Matuszewski


A novel eye status detection method is proposed. Contrary to the most of the previous methods, this new method is not based on an explicit eye appearance model. Instead, the detection is based on a deep learning methodology, where the discriminant function is learned from a large set of exemplar images of eyes at different state, appearance, and 3D position. The technique is based on the Convolutional Neural Network (CNN) architecture. To assess the performance of the proposed method, it has been tested against two techniques, namely: SVM with SURF Bag of Features and Adaboost with HOG and LBP features. It has been shown that the proposed method outperforms these with a considerable margin on a two-class problem, with the two classes defined as “opened” and “closed”. Subsequently the CNN architecture was further optimised on a three-class problem with “opened”, “closed”, and “partially-opened” classes. It has been demonstrated that it is possible to implement a real-time eye status detection working with a large variability of head poses, appearances and illumination conditions. Additionally, it has been shown that an eye blinking estimation based on the proposed technique is at least comparable with the current state-of-the-art on standard eye blinking datasets.


  1. number 611516 Fazli, S., Esfehani, P., 2012. Automatic Fatigue Detection Based On Eye States. In: Proc. of Int. Conf. on Advances in Computer Engineering.
  2. Danisman, T., Bilasco, I., Djeraba, C., Ihaddadene, N., 2010. Drowsy driver detection system using eye blink patterns. In: Machine and Web Intelligence (ICMWI), 2010 International Conference on, pp. 230-233.
  3. Divjak, M., Bischof, H., 2009. Eye blink based fatigue detection for prevention of computer vision syndrome. In: IAPR Conference on Machine Vision Applications, pp. 350-353.
  4. Drutarovsky, T., Fogelton, A., 2014. Eye blink detection using variance of motion vectors. In: Computer Vision - ECCV 2014 Workshops, Zurich, Switzerland. Springer, Cham, Switzerland, pp. 436-448.
  5. Du, Y., Ma, P., Su, X., Zhang, Y., 2008. Driver Fatigue Detection based on Eye State Analysis. In: Proceedings of the 11th Joint Conference on Information Sciences.
  6. Fogelton, A., Benesova W., 2016. Eye blink detection based on motion vectors analysis. Computer Vision and Image Understanding, 148, 23-33.
  7. Joshi, KV., Kangda, A., Patel, S., 2016. Real Time System for Student Fatigue Detection during Online Learning”, Inter. Jour of Hybrid Info. Tech.
  8. Królak, A., Strumillo, P., 2012. Eye-blink detection system for human computer interaction. Universal Access Inf. Soc. 11 (4), 409-419.
  9. Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T., 2012. Interactive facial feature localization. ECCV'12, pp.679-692.
  10. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P, 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, pp. 2278-2324.
  11. Lee, WO., Lee, EC., Park, KR., 2010. Blink detection robust to various facial poses. In: J. Neurosci. Methods 193 (2), 356-372.
  12. Liu, X., Tan, X., Chen, S., 2012. Eyes closeness detection using appearance based methods. Springer, International Conference on Intelligent Information Processing, 398-408.
  13. Malik, K., Smolka, B., 2014. Eye blink detection using local binary patterns. In: 2014 International Conference on Multimedia Computing and Systems (ICMCS), pp. 385-390.
  14. Marcos-Ramiro, A., Pizarro-Perez, D., Marron-Romera, M., Gatica-Perez., D., 2014. Automatic Blinking Detection Towards Stress Discovery. In: Proc. of the 16th Int. Conf. on Multimodal Interaction, pp. 307-310.
  15. Mohammed, A., Anwer, S., 2014. Efficient Eye Blink Detection Method for disabled-helping domain. Inter. Jour. of Adv. In: Computer Science and Applications.
  16. Pan, G., Sun, L., Wu, Z., Lao, S., 2007. Eyeblink-based anti-spoofing in face recognition from a generic webcamera. In: 11th IEEE International Conference on Computer Vision. ICCV 7807, pp. 1-8.
  17. Rezaei, M., Klette, R., 2012. Novel Adaptive Eye Detection and Tracking for Challenging Lighting Conditions. Asian Conference on Computer Vision, 427-440.
  18. Portello, JK., Resenfield, M., Chu, CA., 2013. Blink Rate, Incomplete Blinks and Computer Vision Syndrome. Optometry and Vision Science, 90 (5), 4 82-4 87.
  19. Saragih, JM., Lucey, S., Cohn, JF., 2011. Deformable model fitting by regularized landmark mean-shift, International Journal of Image and Vision Computing, Vol. 91, Issue 2, pp 200-215.
  20. Shin, HC., Roth, HR, Gao, M., Lu, L. Xu, Z., Nogues, I., Yao, J., Mollura, D., Summers, R., 2016. Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning.
  21. Sun, Y., Zafeiriou, S., Pantic, M., 2013. A Hybrid System for On-line Blink Detection. In: Hawaii International Conference on System Sciences.
  22. Szwoch, M., Pieniazek, P., 2012. Eye blink based detection of liveness in biometric authentication systems using conditional random fields. In: Computer Vision and Graphics. In: Lecture Notes in Computer Science, 7594. Springer, pp. 669-676.
  23. “Talking Face Video”, Face & Gesture Recognition Working Group, IST-2000-26434 http://www-prima. html.
  24. Tomasi, C., Kanade, T., 1991. Detection and Tracking of points features. Computer Science Department, Carnege Mellon University, Tech. rep.
  25. Viola, P., Jones, M., Polatsek, P., 2004. Robust real-time face detection. International. Journal of Computer vision. 57 (2), 137-154.
  26. Wascher, E., Heppner, H., Möckel, T., Kobald, SO., Getzmann, S., 2015. Eye-blinks in choice response tasks uncover hidden aspects of information processing. In: Jour. Citation Reports, pp. 1207-1218.
  27. Yosinski, J., Clune, J., Nguyen, A., Fucks, T., Lipson, H., 2015. Understanding Neural Networks Through Deep Visualization. In: Deep Learning Workshop, 31st International Conference on Machine Learning.

Paper Citation

in Harvard Style

Anas E., Henriquez P. and Matuszewski B. (2017). Online Eye Status Detection in the Wild with Convolutional Neural Networks . In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 6: VISAPP, (VISIGRAPP 2017) ISBN 978-989-758-227-1, pages 88-95. DOI: 10.5220/0006172700880095

in Bibtex Style

author={Essa R. Anas and Pedro Henriquez and Bogdan J. Matuszewski},
title={Online Eye Status Detection in the Wild with Convolutional Neural Networks},
booktitle={Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 6: VISAPP, (VISIGRAPP 2017)},

in EndNote Style

JO - Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 6: VISAPP, (VISIGRAPP 2017)
TI - Online Eye Status Detection in the Wild with Convolutional Neural Networks
SN - 978-989-758-227-1
AU - Anas E.
AU - Henriquez P.
AU - Matuszewski B.
PY - 2017
SP - 88
EP - 95
DO - 10.5220/0006172700880095