MULTI-MODAL HANDS-FREE HUMAN COMPUTER INTERACTION: A PROTOTYPE SYSTEM

Frangiskos Frangeskides, Andreas Lanitis

2006

Abstract

Conventional Human Computer Interaction requires the use of hands for moving the mouse and pressing keys on the keyboard. As a result paraplegics are not able to use computer systems unless they acquire special Human Computer Interaction (HCI) equipment. In this paper we describe a prototype system that aims to provide paraplegics the opportunity to use computer systems without the need for additional invasive hardware. The proposed system is a multi-modal system combining both visual and speech input. Visual input is provided through a standard web camera used for capturing face images showing the user of the computer. Image processing techniques are used for tracking head movements, making it possible to use head motion in order to interact with a computer. Speech input is used for activating commonly used tasks that are normally activated using the mouse or the keyboard. Speech input improves the speed and ease of executing various HCI tasks in a hands free fashion. The performance of the proposed system was evaluated using a number of specially designed test applications. According to the quantitative results, it is possible to perform most HCI tasks with the same ease and accuracy as in the case that a touch pad of a portable computer is used. Currently our system is being used by a number of paraplegics.

References

  1. Assistive Technology Solutions. Retrieved October 4, 2005, from http://www.abilityhub.com/mouse/
  2. CameraMouse: Hands Free Computer Mouse. Retrieved October 4, 2005, from http://cameramouse.com/
  3. EyeTech Digital Systems-Eye Tracking Device. Retrieved October 4, 2005, from http://www.eyetechds.com/
  4. Hands Free Mouse - Assistive Technology Solution. Retrieved October 4, 2005, from http://eyecontrol.com/smartnav/
  5. Gorodnichy D.O. and Roth G, 2004. Nouse 'Use Your Nose as a Mouse' - Perceptual Vision Technology for Hands-Free Games and Interfaces, Image and Vision Computing, Vol. 22, No 12, pp 931-942.
  6. Kettebekov S, and Sharma R , 2001. Toward Natural Gesture/Speech Control of a Large Display, M. Reed Little and L. Nigay (Eds.): EHCI 2001, LNCS 2254, pp. 221-234, Springer-Verlag Berlin Heidelberg.
  7. Krahnstoever N, Kettebekov S, Yeasin M, Sharma R. 2002. A Real-Time Framework for Natural Multimodal Interaction with Large Screen Displays. Proc. of Fourth Intl. Conference on Multimodal Interfaces (ICMI 2002).
  8. Mateos G.G. 2003. Refining Face Tracking With Integral Projections. Procs. Of the 4th International Conference on Audio and Video-Based Biometric Person Identification, Lecture Notes in Computer Science, Vol 2688, pp 360-368.
  9. Microsoft Speech - Speech SDK5.1 For Windows Applications. Retrieved July 10, 2005, from http://www.microsoft.com/speech/download/sdk51/
  10. Mouse Vision Assistive Technologies, Retrieved October 4, 2005, from http://mousevision.com/
  11. Origin Instruments Corporation. Retrieved October 4, 2005, from http://orin.com/index
  12. O'Shaughnessy, D., 2003. Interacting with computers by voice: automatic speech recognition and synthesis. 1272- 1305.
  13. Potamianos G., Neti C., Gravier G., Garg A., Senior A.W. 2003. Recent advances in the automatic recognition of audiovisual speech. IEEE Proceedings, Vol 91, Issue 9, pp 1306- 1326.
  14. Toyama K. 1998. Look, Ma - No Hands! - Hands Free Cursor Control with Real Time 3D Face Tracking, Procs. Of Workshop on Perceptual User Interfaces, pp. 49-54.
Download


Paper Citation


in Harvard Style

Frangeskides F. and Lanitis A. (2006). MULTI-MODAL HANDS-FREE HUMAN COMPUTER INTERACTION: A PROTOTYPE SYSTEM . In Proceedings of the Eighth International Conference on Enterprise Information Systems - Volume 5: ICEIS, ISBN 978-972-8865-45-0, pages 19-26. DOI: 10.5220/0002443900190026


in Bibtex Style

@conference{iceis06,
author={Frangiskos Frangeskides and Andreas Lanitis},
title={MULTI-MODAL HANDS-FREE HUMAN COMPUTER INTERACTION: A PROTOTYPE SYSTEM},
booktitle={Proceedings of the Eighth International Conference on Enterprise Information Systems - Volume 5: ICEIS,},
year={2006},
pages={19-26},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0002443900190026},
isbn={978-972-8865-45-0},
}


in EndNote Style

TY - CONF
JO - Proceedings of the Eighth International Conference on Enterprise Information Systems - Volume 5: ICEIS,
TI - MULTI-MODAL HANDS-FREE HUMAN COMPUTER INTERACTION: A PROTOTYPE SYSTEM
SN - 978-972-8865-45-0
AU - Frangeskides F.
AU - Lanitis A.
PY - 2006
SP - 19
EP - 26
DO - 10.5220/0002443900190026