Recognizing Hand and Finger Gestures with IMU based Motion and EMG based Muscle Activity Sensing

Marcus Georgi, Christoph Amma, Tanja Schultz

2015

Abstract

Session- and person-independent recognition of hand and finger gestures is of utmost importance for the practicality of gesture based interfaces. In this paper we evaluate the performance of a wearable gesture recognition system that captures arm, hand, and finger motions by measuring movements of, and muscle activity at the forearm. We fuse the signals of an Inertial Measurement Unit (IMU) worn at the wrist, and the Electromyogram (EMG) of muscles in the forearm to infer hand and finger movements. A set of 12 gestures was defined, motivated by their similarity to actual physical manipulations and to gestures known from the interaction with mobile devices. We recorded performances of our gesture set by five subjects in multiple sessions. The resulting datacorpus will be made publicly available to build a common ground for future evaluations and benchmarks. Hidden Markov Models (HMMs) are used as classifiers to discriminate between the defined gesture classes. We achieve a recognition rate of 97.8% in session-independent, and of 74.3% in person-independent recognition. Additionally, we give a detailed analysis of error characteristics and of the influence of each modality to the results to underline the benefits of using both modalities together

References

  1. Alibali, M. W. (2005). Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information. Spatial Cognition Computation, 5(4):307-331.
  2. Amma, C., Georgi, M., and Schultz, T. (2014). Airwriting: a wearable handwriting recognition system. Personal and Ubiquitous Computing, 18(1):191-203.
  3. Benbasat, A. Y. and Paradiso, J. A. (2001). An inertial measurement framework for gesture recognition and applications. In Wachsmuth, I. and Sowa, T., editors, Revised Papers from the International Gesture Workshop on Gesture and Sign Languages in Human-Computer Interaction, volume LNCS 2298 of Lecture Notes in Computer Science, pages 9-20. Springer-Verlag.
  4. Chen, X., Zhang, X., Zhao, Z., Yang, J., Lantz, V., and Wang, K. (2007). Hand gesture recognition research based on surface emg sensors and 2d-accelerometers. In Wearable Computers, 2007 11th IEEE International Symposium on, pages 11-14.
  5. Cho, S.-J., Oh, J.-K., Bang, W.-C., Chang, W., Choi, E., Jing, Y., Cho, J., and Kim, D.-Y. (2004). Magic wand: a hand-drawn gesture input device in 3-d space with inertial sensors. In Frontiers in Handwriting Recognition, 2004. IWFHR-9 2004. Ninth International Workshop on, pages 106-111.
  6. Hartmann, B. and Link, N. (2010). Gesture recognition with inertial sensors and optimized dtw prototypes. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Istanbul, Turkey, 10- 13 October 2010, Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Istanbul, Turkey, 10-13 October 2010, pages 2102- 2109. IEEE.
  7. Hauptmann, A. G. and McAvinney, P. (1993). Gestures with speech for graphic manipulation. International Journal of ManMachine Studies, 38(2):231-249.
  8. Kim, D., Hilliges, O., Izadi, S., Butler, A. D., Chen, J., Oikonomidis, I., and Olivier, P. (2012). Digits: Freehand 3d interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, UIST 7812, pages 167-176, New York, NY, USA. ACM.
  9. Kim, J., Mastnik, S., and André, E. (2008). Emg-based hand gesture recognition for realtime biosignal interfacing. In Proceedings of the 13th International Conference on Intelligent User Interfaces, volume 39 of IUI 7808, pages 30-39, New York, NY, USA. ACM Press.
  10. Li, Y., Chen, X., Tian, J., Zhang, X., Wang, K., and Yang, J. (2010). Automatic recognition of sign language subwords based on portable accelerometer and emg sensors. In International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI 7810, pages 17- 1, New York, NY, USA. ACM.
  11. Mistry, P., Maes, P., and Chang, L. (2009). WUW-wear Ur world: a wearable gestural interface. Proceedings of CHI 2009, pages 4111-4116.
  12. Oviatt, S. (1999). Ten myths of multimodal interaction. Communications of the ACM, 42(11):74-81.
  13. Rabiner, L. (1989). A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceedings of the IEEE, 77(2):257-286.
  14. Rekimoto, J. (2001). GestureWrist and GesturePad: unobtrusive wearable interaction devices. Proceedings Fifth International Symposium on Wearable Computers.
  15. Samadani, A.-A. and Kulic, D. (2014). Hand gesture recognition based on surface electromyography. In Engineering in Medicine and Biology Society (EMBC), 2014 36th Annual International Conference of the IEEE.
  16. Saponas, T. S., Tan, D. S., Morris, D., and Balakrishnan, R. (2008). Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 7808, pages 515- 524, New York, New York, USA. ACM Press.
  17. Wolf, M. T., Assad, C., Stoica, A., You, K., Jethani, H., Vernacchia, M. T., Fromm, J., and Iwashita, Y. (2013). Decoding static and dynamic arm and hand gestures from the jpl biosleeve. In Aerospace Conference, 2013 IEEE, pages 1-9.
  18. Zhang, X., Chen, X., Li, Y., Lantz, V., Wang, K., and Yang, J. (2011). A framework for hand gesture recognition based on accelerometer and emg sensors. Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on, 41(6):1064-1076.
Download


Paper Citation


in Harvard Style

Georgi M., Amma C. and Schultz T. (2015). Recognizing Hand and Finger Gestures with IMU based Motion and EMG based Muscle Activity Sensing . In Proceedings of the International Conference on Bio-inspired Systems and Signal Processing - Volume 1: BIOSIGNALS, (BIOSTEC 2015) ISBN 978-989-758-069-7, pages 99-108. DOI: 10.5220/0005276900990108


in Bibtex Style

@conference{biosignals15,
author={Marcus Georgi and Christoph Amma and Tanja Schultz},
title={Recognizing Hand and Finger Gestures with IMU based Motion and EMG based Muscle Activity Sensing},
booktitle={Proceedings of the International Conference on Bio-inspired Systems and Signal Processing - Volume 1: BIOSIGNALS, (BIOSTEC 2015)},
year={2015},
pages={99-108},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005276900990108},
isbn={978-989-758-069-7},
}


in EndNote Style

TY - CONF
JO - Proceedings of the International Conference on Bio-inspired Systems and Signal Processing - Volume 1: BIOSIGNALS, (BIOSTEC 2015)
TI - Recognizing Hand and Finger Gestures with IMU based Motion and EMG based Muscle Activity Sensing
SN - 978-989-758-069-7
AU - Georgi M.
AU - Amma C.
AU - Schultz T.
PY - 2015
SP - 99
EP - 108
DO - 10.5220/0005276900990108