NeuRow: An Immersive VR Environment for Motor-Imagery Training with the Use of Brain-Computer Interfaces and Vibrotactile Feedback

Athanasios Vourvopoulos, André Ferreira, Sergi Bermudez i Badia

2016

Abstract

Motor-Imagery offers a solid foundation for the development of Brain-Computer Interfaces (BCIs), capable of direct brain-to-computer communication but also effective in alleviating neurological impairments. The fusion of BCIs with Virtual Reality (VR) allowed the enhancement of the field of virtual rehabilitation by including patients with low-level of motor control with limited access to treatment. BCI-VR technology has pushed research towards finding new solutions for better and reliable BCI control. Based on our previous work, we have developed NeuRow, a novel multiplatform prototype that makes use of multimodal feedback in an immersive VR environment delivered through a state-of-the-art Head Mounted Display (HMD). In this article we present the system design and development, including important features for creating a closed neurofeedback loop in an implicit manner, and preliminary data on user performance and user acceptance of the system.

References

  1. Adams, E., Ip, B., n.d. From Casual to Core: A Statistical Mechanism for Studying Gamer Dedication [WWW Document]. URL http://www.gamasutra.com/view/fea ture/131397/from_casual_to_core_a_statistical_.php (accessed 1.5.15).
  2. Allison, B.Z., Neuper, C., 2010. Could Anyone Use a BCI?, in: Tan, D.S., Nijholt, A. (Eds.), Brain-Computer Interfaces, Human-Computer Interaction Series. Springer London, pp. 35-54.
  3. Bangor, A., Kortum, P., Miller, J., 2009. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. J Usability Stud. 4, 114-123.
  4. Berka, C., Levendowski, D.J., Lumicao, M.N., Yau, A., Davis, G., Zivkovic, V.T., Olmstead, R.E., Tremoulet, P.D., Craven, P.L., 2007. EEG Correlates of Task Engagement and Mental Workload in Vigilance, Learning, and Memory Tasks. Aviat. Space Environ. Med. 78, B231-B244.
  5. Bermudez i Badia, S., Cameirao, M.S., 2012. The Neurorehabilitation Training Toolkit (NTT): A Novel Worldwide Accessible Motor Training Approach for At-Home Rehabilitation after Stroke. Stroke Res. Treat. 2012. doi:10.1155/2012/802157.
  6. Boostani, R., Moradi, M.H., 2004. A new approach in the BCI research based on fractal dimension as feature and Adaboost as classifier. J. Neural Eng. 1, 212-217. doi:10.1088/1741-2560/1/4/004.
  7. Brooke, J., 1996. SUS-A quick and dirty usability scale. Usability Eval. Ind. 189, 194.
  8. Cincotti, F., Kauhanen, L., Aloise, F., Palomäki, T., Caporusso, N., Jylänki, P., Mattia, D., Babiloni, F., Vanacker, G., Nuttin, M., Marciani, M.G., Millán, J. del R., 2007. Vibrotactile Feedback for Brain-Computer Interface Operation. Comput. Intell. Neurosci. 2007, e48937. doi:10.1155/2007/48937.
  9. Delorme, A., Makeig, S., 2004. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 134, 9-21. doi:10.1016/j.jneumeth .2003.10.009.
  10. Dickstein, R., Deutsch, J.E., Yoeli, Y., Kafri, M., Falash, F., Dunsky, A., Eshet, A., Alexander, N., 2013. Effects of Integrated Motor Imagery Practice on Gait of Individuals With Chronic Stroke: A Half-Crossover Randomized Study. Arch. Phys. Med. Rehabil. 94, 2119-2125. doi:10.1016/j.apmr.2013.06.031.
  11. Dobkin, B.H., 2007. Brain-computer interface technology as a tool to augment plasticity and outcomes for neurological rehabilitation. J. Physiol. 579, 637-642. doi:10.1113/jphysiol.2006.123067.
  12. Eaves, D.L., Haythornthwaite, L., Vogt, S., 2014. Motor imagery during action observation modulates automatic imitation effects in rhythmical actions. Front. Hum. Neurosci. 8. doi:10.3389/fnhum.2014.00028.
  13. Friedman, D., 2015. Brain-Computer Interfacing and Virtual Reality, in: Nakatsu, R., Rauterberg, M., Ciancarini, P. (Eds.), Handbook of Digital Games and Entertainment Technologies. Springer Singapore, pp. 1-22.
  14. Garcia, G.N., Ebrahimi, T., Vesin, J., 2003. Support vector EEG classification in the Fourier and time-frequency correlation domains, in: First International IEEE EMBS Conference on Neural Engineering, 2003. Conference Proceedings. Presented at the First International IEEE EMBS Conference on Neural Engineering, 2003. Conference Proceedings, pp. 591-594. doi:10.1109/C NE.2003.1196897.
  15. Girouard, A., Solovey, E.T., Hirshfield, L.M., Chauncey, K., Sassaroli, A., Fantini, S., Jacob, R.J.K., 2009. Distinguishing Difficulty Levels with Non-invasive Brain Activity Measurements, in: Gross, T., Gulliksen, J., Kotzé, P., Oestreicher, L., Palanque, P., Prates, R.O., Winckler, M. (Eds.), Human-Computer Interaction - INTERACT 2009, Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp. 440-452.
  16. Gomez-Rodriguez, M., Peters, J., Hill, J., Schölkopf, B., Gharabaghi, A., Grosse-Wentrup, M., 2011. Closing the sensorimotor loop: haptic feedback facilitates decoding of motor imagery. J. Neural Eng. 8, 036005. doi:10.1088/1741-2560/8/3/036005.
  17. Gwak, K., Leeb, R., Millan, J.D.R., Kim, D.-S., 2014. Quantification and reduction of visual load during BCI operation, in: 2014 IEEE International Conference on Systems, Man and Cybernetics (SMC). Presented at the 2014 IEEE International Conference on Systems, Man and Cybernetics (SMC), pp. 2795-2800. doi:10.1109/ SMC.2014.6974352.
  18. Hanakawa, T., 2015. Organizing motor imageries. Neurosci. Res. doi:10.1016/j.neures.2015.11.003.
  19. Hart, S.G., Staveland, L.E., 1988. Development of NASATLX (Task Load Index): Results of Empirical and Theoretical Research, in: Meshkati, P.A.H. and N. (Ed.), Advances in Psychology, Human Mental Workload. North-Holland, pp. 139-183.
  20. Hattie, J., Timperley, H., 2007. The Power of Feedback. Rev. Educ. Res. 77, 81-112. doi:10.3102/003465430 298487.
  21. Hinterberger, T., Neumann, N., Pham, M., Kübler, A., Grether, A., Hofmayer, N., Wilhelm, B., Flor, H., Birbaumer, N., 2004. A multimodal brain-based feedback and communication system. Exp. Brain Res. 154, 521-526. doi:10.1007/s00221-003-1690-3.
  22. IJsselsteijn, W., Poels, K., de Kort, Y.A., 2008. The Game Experience Questionnaire: Development of a selfreport measure to assess player experiences of digital games. TU Eindh. Eindh. Neth.
  23. Jeunet, C., Vi, C., Spelmezan, D., N'Kaoua, B., Lotte, F., Subramanian, S., 2015. Continuous Tactile Feedback for Motor-Imagery Based Brain-Computer Interaction in a Multitasking Context, in: Abascal, J., Barbosa, S., Fetter, M., Gross, T., Palanque, P., Winckler, M. (Eds.), Human-Computer Interaction - INTERACT 2015, Lecture Notes in Computer Science. Springer International Publishing, pp. 488-505.
  24. Klem, G.H., Lüders, H.O., Jasper, H.H., Elger, C., 1999. The ten-twenty electrode system of the International Federation. The International Federation of Clinical Neurophysiology. Electroencephalogr. Clin. Neurophysiol. Suppl. 52, 3-6.
  25. Lange, B., Koenig, S., Chang, C.-Y., McConnell, E., Suma, E., Bolas, M., Rizzo, A., 2012. Designing informed game-based rehabilitation tasks leveraging advances in virtual reality. Disabil. Rehabil. 34, 1863-1870.
  26. Leeb, R., Gwak, K., Kim, D.-S., del R Millán, J., 2013. Freeing the visual channel by exploiting vibrotactile BCI feedback. Conf. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Annu. Conf. 2013, 3093-3096. doi:10.1109/EMBC.2013.6610195.
  27. Leonardis, D., Frisoli, A., Solazzi, M., Bergamasco, M., 2012. Illusory perception of arm movement induced by visuo-proprioceptive sensory stimulation and controlled by motor imagery, in: 2012 IEEE Haptics Symposium (HAPTICS). Presented at the 2012 IEEE Haptics Symposium (HAPTICS), pp. 421-424. doi:10.1109/HAPTIC.2012.6183825.
  28. Lledo, P.-M., Alonso, M., Grubb, M.S., 2006. Adult neurogenesis and functional plasticity in neuronal circuits. Nat. Rev. Neurosci. 7, 179-193. doi:10.1038 /nrn1867.
  29. Lotte, F., 2012. On the need for alternative feedback training approaches for BCI. Presented at the Berlin Brain-Computer Interface Workshop.
  30. Lotte, F., Faller, J., Guger, C., Renard, Y., Pfurtscheller, G., Lécuyer, A., Leeb, R., 2013a. Combining BCI with Virtual Reality: Towards New Applications and Improved BCI.
  31. Lotte, F., Larrue, F., Mühl, C., 2013b. Flaws in current human training protocols for spontaneous BrainComputer Interfaces: lessons learned from instructional design. Front. Hum. Neurosci. 7, 568. doi:10.3389/fn hum.2013.00568.
  32. Mulder, T., 2007. Motor imagery and action observation: cognitive tools for rehabilitation. J. Neural Transm. 114, 1265-1278. doi:10.1007/s00702-007-0763-z.
  33. Neuper, C., Scherer, R., Reiner, M., Pfurtscheller, G., 2005. Imagery of motor actions: differential effects of kinesthetic and visual-motor mode of imagery in singletrial EEG. Brain Res. Cogn. Brain Res. 25, 668-677. doi:10.1016/j.cogbrainres.2005.08.014.
  34. Neuper, C., Scherer, R., Wriessnegger, S., Pfurtscheller, G., 2009. Motor imagery and action observation: Modulation of sensorimotor brain rhythms during mental control of a brain-computer interface. Clin. Neurophysiol. 120, 239-247. doi:10.1016/j.clinph.20 08.11.015.
  35. Obermaier, B., Guger, C., Neuper, C., Pfurtscheller, G., 2001. Hidden Markov models for online classification of single trial EEG data. Pattern Recognit. Lett., Selected Papers from the 11th Portuguese Conference on Pattern Recognition - RECPAD2000 22, 1299- 1309. doi:10.1016/S0167-8655(01)00075-7.
  36. Pfurtscheller, G., Neuper, C., Müller, G.R., Obermaier, B., Krausz, G., Schlögl, A., Scherer, R., Graimann, B., Keinrath, C., Skliris, D., Wörtz, M., Supp, G., Schrank, C., 2003. Graz-BCI: state of the art and clinical applications. IEEE Trans. Neural Syst. Rehabil. Eng. Publ. IEEE Eng. Med. Biol. Soc. 11, 177-180. doi:10.1109/TNSRE.2003.814454.
  37. Pope, A.T., Bogart, E.H., Bartolome, D.S., 1995. Biocybernetic system evaluates indices of operator engagement in automated task. Biol. Psychol., EEG in Basic and Applied Settings 40, 187-195. doi:10.1016/0301-0511(95)05116-3.
  38. Renard, Y., Lotte, F., Gibert, G., Congedo, M., Maby, E., Delannoy, V., Bertrand, O., Lécuyer, A., 2010. OpenViBE: an open-source software platform to design, test, and use brain-computer interfaces in real and virtual environments. Presence Teleoperators Virtual Environ. 19, 35-53.
  39. Roberts, R., Callow, N., Hardy, L., Markland, D., Bringer, J., 2008. Movement imagery ability: development and assessment of a revised version of the vividness of movement imagery questionnaire. J. Sport Exerc. Psychol. 30, 200-221.
  40. Rossini, P.M., Calautti, C., Pauri, F., Baron, J.-C., 2003. Post-stroke plastic reorganisation in the adult brain. Lancet Neurol. 2, 493-502.
  41. Shute, V.J., 2008. Focus on Formative Feedback. Rev. Educ. Res. 78, 153-189. doi:10.3102/003465430731 3795.
  42. Sigrist, R., Rauter, G., Riener, R., Wolf, P., 2013. Augmented visual, auditory, haptic, and multimodal feedback in motor learning: a review. Psychon. Bull. Rev. 20, 21-53. doi:10.3758/s13423-012-0333-8.
  43. Slater, M., 2009. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philos. Trans. R. Soc. Lond. B Biol. Sci. 364, 3549- 3557. doi:10.1098/rstb.2009.0138.
  44. Solhjoo, S., Moradi, M., 2004. Mental task recognition: A comparison between some of classification methods, in: BIOSIGNAL 2004 International EURASIP Conference. pp. 24-26.
  45. Taylor, R.M., II, Hudson, T.C., Seeger, A., Weber, H., Juliano, J., Helser, A.T., 2001. VRPN: A Deviceindependent, Network-transparent VR Peripheral System, in: Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST 7801. ACM, New York, NY, USA, pp. 55-61. doi:10.1145/505008.505019.
  46. Trompetto, C., Marinelli, L., Mori, L., Pelosin, E., Currà, A., Molfetta, L., Abbruzzese, G., 2014. Pathophysiology of Spasticity: Implications for Neurorehabilitation. BioMed Res. Int. 2014. doi:10.1155/2014/354906.
  47. Vidaurre, C., Blankertz, B., 2009. Towards a Cure for BCI Illiteracy. Brain Topogr. 23, 194-198. doi:10.100 7/s10548-009-0121-6.
  48. Vourvopoulos, A., Bermúdez i Badia, S., 2016. Usability and Cost-effectiveness in Brain-Computer Interaction: Is it User Throughput or Technology Related?, in: Proceedings of the 7th Augmented Human International Conference, AH 7816. ACM, Geneva, Switzerland. doi:10.1145/2875194.2875244.
  49. Vourvopoulos, A., Cardona, J.E.M., Badia, S.B. i, 2015. Optimizing Motor Imagery Neurofeedback through the Use of Multimodal Immersive Virtual Reality and Motor Priming. Presented at the 2015 International Conference on Virtual Rehabilitation (ICVR).
  50. Vourvopoulos, A., Faria, A.L., Cameirao, M.S., Bermudez i Badia, S., 2013. RehabNet: A distributed architecture for motor and cognitive neuro-rehabilitation, in: 2013 IEEE 15th International Conference on E-Health Networking, Applications Services (Healthcom). Presented at the 2013 IEEE 15th International Conference on e-Health Networking, Applications Services (Healthcom), pp. 454-459. doi:10.1109/He althCom.2013.6720719.
  51. Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M., 2002. Braincomputer interfaces for communication and control. Clin. Neurophysiol. Off. J. Int. Fed. Clin. Neurophysiol. 113, 767-791.
  52. Yao, L., Meng, J., Zhang, D., Sheng, X., Zhu, X., 2014. Combining motor imagery with selective sensation toward a hybrid-modality BCI. IEEE Trans. Biomed. Eng. 61, 2304-2312. doi:10.1109/TBME.2013.2287 245.
Download


Paper Citation


in Harvard Style

Vourvopoulos A., Ferreira A. and Badia S. (2016). NeuRow: An Immersive VR Environment for Motor-Imagery Training with the Use of Brain-Computer Interfaces and Vibrotactile Feedback . In Proceedings of the 3rd International Conference on Physiological Computing Systems - Volume 1: PhyCS, ISBN 978-989-758-197-7, pages 43-53. DOI: 10.5220/0005939400430053


in Bibtex Style

@conference{phycs16,
author={Athanasios Vourvopoulos and André Ferreira and Sergi Bermudez i Badia},
title={NeuRow: An Immersive VR Environment for Motor-Imagery Training with the Use of Brain-Computer Interfaces and Vibrotactile Feedback},
booktitle={Proceedings of the 3rd International Conference on Physiological Computing Systems - Volume 1: PhyCS,},
year={2016},
pages={43-53},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005939400430053},
isbn={978-989-758-197-7},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 3rd International Conference on Physiological Computing Systems - Volume 1: PhyCS,
TI - NeuRow: An Immersive VR Environment for Motor-Imagery Training with the Use of Brain-Computer Interfaces and Vibrotactile Feedback
SN - 978-989-758-197-7
AU - Vourvopoulos A.
AU - Ferreira A.
AU - Badia S.
PY - 2016
SP - 43
EP - 53
DO - 10.5220/0005939400430053