Automated Classification of Therapeutic Face Exercises using the Kinect

Cornelia Lanz, Birant Sibel Olgay, Joachim Denzler, Horst-Michael Gross


In this work, we propose an approach for the unexplored topic of therapeutic facial exercise recognition using depth images. In cooperation with speech therapists, we determined nine exercises that are beneficial for therapy of patients suffering from dysfunction of facial movements. Our approach employs 2.5D images and 3D point clouds, which were recorded using Microsoft’s Kinect. Extracted features comprise the curvature of the face surface and characteristic profiles that are derived using distinctive landmarks. We evaluate the discriminative power and the robustness of the features with respect to the above-mentioned application scenario. Using manually located face regions for feature extraction, we achieve an average recognition accuracy of about 91% for the nine facial exercises. However in a real-world scenario manual localization of regions for feature extraction is not feasible. Therefore, we additionally examine the robustness of the features and show, that they are beneficial for a real-world, fully automated scenario as well.


  1. Besl, P. and Jain, R. (1986). Invariant surface characteristics for 3d object recognition in range images. Computer Vision, Graphics, and Image Processing, 33(1):33- 80.
  2. Chang, C.-C. and Lin, C.-J. (2011). LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1-27:27. Software available at url cjlin/libsvm.
  3. Chua, C.-S., Han, F., and Ho, Y.-K. (2000). 3d human face recognition using point signature. In Proceedings of the 4th Int. Automatic Face and Gesture Recognition Conf., pages 233-238.
  4. Chua, C. S. and Jarvis, R. (1997). Point signature: a new representation for 3d object recognition. In Int. Journal of Computer Vision, volume 25, pages 63-85.
  5. Colombo, A., Cusano, C., and Schettini, R. (2006). 3d face detection using curvature analysis. Pattern Recognition, 39(3):444-455.
  6. Cootes, T., Edwards, G., and Taylor, C. (2001). Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(6):681-685.
  7. Cootes, T. and Taylor, C. (2001). Constrained active appearance models. In Proceedings of the 8th Int. Conf. on Computer Vision, volume 1, pages 748-754.
  8. Gebhard, A., Paulus, D., Suchy, B., and Wolf, S. (2000). A system for diagnosis support of patients with facialis paresis. Kl, 3/2000:40-42.
  9. Grosse, M., Schaffer, M., Harendt, B., and Kowarschik, R. (2011). Fast data acquisition for three-dimensional shape measurement using fixed-pattern projection and temporal coding. Optical Engineering, 50:100503.
  10. Haase, D. and Denzler, J. (2011). Anatomical landmark tracking for the analysis of animal locomotion in xray videos using active appearance models. In Image Analysis, volume 6688 of Lecture Notes in Computer Science, pages 604-615.
  11. Hartley, R. and Zisserman, A. (2000). Multiple View Geometry in Computer Vision. Cambridge University Press.
  12. Hsu, C., Chang, C., and Lin, C. (2009). A practical guide to support vector classification. TR available at http://www. csie. ntu. edu. tw/ cjlin/papers/guide/guide. pdf.
  13. Khoshelham, K. (2011). Accuracy analysis of kinect depth data. In ISPRS Workshop Laser Scanning, volume 38.
  14. Nakamura, K., Toda, N., Sakamaki, K., Kashima, K., and Takeda, N. (2003). Biofeedback rehabilitation for prevention of synkinesis after facial palsy. Otolaryngology-Head and Neck Surgery, 128(4):539-543.
  15. Salomon, D. (2004). Data compression: the complete reference. Springer-Verlag New York Inc.
  16. Viola, P. and Jones, M. (2004). Robust real-time face detection. Int. Journal of Computer Vision, 57(2):137-154.
  17. Wang, J., Yin, L., Wei, X., and Sun, Y. (2006). 3d facial expression recognition based on primitive surface feature distribution. Int. Conf. on Computer Vision and Pattern Recognition, 2:1399-1406.
  18. Wang, Y., Chua, C.-S., and Ho, Y.-K. (2002). Facial feature detection and face recognition from 2d and 3d images. In Pattern Recognition Letters, volume 23, pages 1191-1202.
  19. Webb, A., Copsey, K., and Cawley, G. (2011). Statistical pattern recognition. Wiley.

Paper Citation

in Harvard Style

Lanz C., Olgay B., Denzler J. and Gross H. (2013). Automated Classification of Therapeutic Face Exercises using the Kinect . In Proceedings of the International Conference on Computer Vision Theory and Applications - Volume 1: VISAPP, (VISIGRAPP 2013) ISBN 978-989-8565-47-1, pages 556-565. DOI: 10.5220/0004294005560565

in Bibtex Style

author={Cornelia Lanz and Birant Sibel Olgay and Joachim Denzler and Horst-Michael Gross},
title={Automated Classification of Therapeutic Face Exercises using the Kinect},
booktitle={Proceedings of the International Conference on Computer Vision Theory and Applications - Volume 1: VISAPP, (VISIGRAPP 2013)},

in EndNote Style

JO - Proceedings of the International Conference on Computer Vision Theory and Applications - Volume 1: VISAPP, (VISIGRAPP 2013)
TI - Automated Classification of Therapeutic Face Exercises using the Kinect
SN - 978-989-8565-47-1
AU - Lanz C.
AU - Olgay B.
AU - Denzler J.
AU - Gross H.
PY - 2013
SP - 556
EP - 565
DO - 10.5220/0004294005560565