Developing a Novel fMRI-Compatible Motion Tracking System

for Haptic Motor Control Experiments

M. Rodríguez

, A. Sylaidi

and A. A. Faisal

1,2

Brain and Behaviour Lab, Department of Bioengineering, Imperial College London, London, U.K.

Brain and Beahvioural Lab, Department of Computing, Imperial College London, London, U.K.

Keywords: fMRI, Motion Tracking, Low Cost, Object Manipulation, Neuroimaging.

Abstract: Human neuroimaging can play a key role in addressing open questions in motor neuroscience and embodied

cognition by linking human movement experiments and motor psychophysics to the neural foundation of

motor control. To this end we designed and built fMOVE, an fMRI-compatible motion tracking system that

captures 3DOF goal-directed movements of human subjects within a neuroimaging scanner. fMOVE

constitutes an ultra-low-cost technology, based on a zoom lens high-frame rate USB camera and, our

adaptation library for camera-based motion tracking and experiment control. Our motion tracking algorithm

tracks the position of markers attached to a hand-held object. The system enables to provide the scanned

subjects a closed-loop real time visual feedback of their motion and control of complex, goal-oriented

movements. The latter are instructed by simple speed-accuracy tasks or goal-oriented object manipulation.

The system’s tracking precision was tested and found within its operational parameters comparable to the

performance levels of a scientific grade electromagnetic motion tracking system. fMOVE thus offers a low-

cost methodological platform to re-approach the objectives of motor neuroscience by enabling ecologically

more valid motor tasks in neuroimaging studies.

1 INTRODUCTION

On a daily basis, humans acquire new motor skills or

enhance their performance on already encountered

motor tasks. Occasionally, they may also re-acquire

skills, which are difficult to perform or cannot be

executed because of injury or disease. The motor

learning experience in all these cases involves a

number of different processes, which support motor

behaviour through interacting and/or hierarchical

roles.

At the level of information extraction, skilled

performance is based on the efficient gathering of

information from the environment. Humans use task

dependent attentional mechanisms to actively select

(Friston, Daunizeau, and Kiebel, 2009; Friston,

2010) and integrate multisensory stimuli. They

uncover the dynamics of a task by learning key

properties of their body and the world (Faisal Syladi

and Faisal, 2012; Brayanov, Press, and Smith,

2012). Crucially, they determine the necessary

motor commands so as to optimize their

performance according to task goals, e.g. to

minimise variability and noise (Todorov and Jordan,

2002; Todorov, 2004; Faisal and Wolpert 2009;

Faisal et al, 2008). This process ranges from high-

level decision making that dictates the structure of

an effective control policy to low-level optimization

of the execution of the selected policy.

Although motor behaviour has been

systematically studied for more than 100 years, the

mechanisms that underlie motor learning and the

formation of control policies remain unclear. A large

body of research has examined motor behaviour

through motor psychophysical experiments, which

provide insight into the patterns of adaptive

responses in tasks that introduce changes in the

target, the workspace or the force-field (Shadmehr

and Moussavi, 2000; Shadmehr and Mussa-Ivaldi,

1994; Wolpert, Diedrichsen, and Flanagan, 2011;

Faisal and Wolpert, 2009). Such studies have relied

primarily on high-resolution behavioural data and

have inspired a number of computational approaches

that describe abstract mechanisms of the interplay

between perception and action, as well as

mechanisms of generalisation of learned activity

(e.g. optimal feedback control, reinforcement

learning, Bayesian inference).

Rodríguez M., Sylaidi A. and Faisal A..

Developing a Novel fMRI-Compatible Motion Tracking System for Haptic Motor Control Experiments.

DOI: 10.5220/0005094700250030

In Proceedings of the 2nd International Congress on Neurotechnology, Electronics and Informatics (NEUROTECHNIX-2014), pages 25-30

ISBN: 978-989-758-056-7

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

A novel strand of research in the field has focused

on ecologically more valid tasks shifting away from

strict lab protocols and thereby enabling subjects to

move freely (within the confines of their task and

calibration) in naturalistic settings (e.g. flint stone

tool making). Such settings were used in motor

studies carried out in parallel with neuroimaging

approaches, which investigated the related demands

on the brain (e.g. Faisal et al, 2010; Hecht et al.,

2014). This work provided insight into human

natural movement statistics (Ingram et al 2008,

Faisal et al. 2010), as well as into the predictable

structure and sequence of movements with

immediate implications for Brain-Machine Interface

and prosthetic control (Thomik et al, 2013; Haber et

al 2014).

Yet despite this substantial progress in the study

and understanding of motor control processes and in

the predictability of movements, less methodological

advancement has been achieved in linking motor

psychophysics and computational models of

behaviour to their underlying neurophysiological

correlates. Brain imaging based on fMRI, one of the

predominant technological paradigms to access the

neural implementation level, has been primarily used

in studies that examine purely cognitive tasks. In the

less common cases, in which fMRI has been

employed in motor neuroscience research, the

examined functions refer to very simple, lab-

constrained movements (e.g. finger-tapping) and the

designed experiments instruct non-realistic open-

loop tasks, which do not provide any sensory

feedback of performance so as to encourage

learning. The main reason for this restriction lies in

the technical constraints, which are interwoven with

the fMRI function and which often make its use

incompatible to most advanced motion tracking

systems.

Here we designed and built an fMRI-compatible

motion tracking system that allows us to examine

how humans learn complex motor tasks. Our system,

fMOVE, constitutes a technology capable of

acquiring information about 3D motion inside an

fMRI scanner in a three-degrees-of-freedom context.

The designed platform can host closed-loop motion

studies by establishing continuous motion tracking

and providing human subjects with online virtual

feedback of their motor behaviour and performance.

fMOVE thus provides an expansion of conventional

motion tracking methods used in fMRI studies,

which are trying to improve the analysis of fMRI

data (compensation for head and/or body motion), or

to adjust the block design to the actual motion start

and pause.

Figure 1: Comparison of fMOVE and commercial motion

tracking systems. fMOVE corresponds to the lowest cost

level and possesses motion tracking performance within

the range covered by established motion trackers (e.g.

Polhemus, Vicon). The processing power of the displayed

systems is estimated as the product of DOF and sampling

frequency (both of which should be maximised, but are

often treated as trade-offs in conventional motion tracking

systems).

At the same time fMOVE possesses motion

tracking performance in the same range as

established fMRI-incompatible motion tracking

methods (e.g. Vicon with motion tracking at 1DOF

and 250Hz, Intersense 900 at 6DOF and 180Hz,

Polhemus Liberty at 6DOF and 240Hz) and is

significantly cheaper than other fMRI-incompatible

technologies with better information processing

features (e.g. Hiball with motion tracking at 6DOF

and 2000Hz, Optotrack at 6DOF and 4600Hz, see

Fig. 1). The development of our software that

supports this platform makes use of ARToolkit, a

software library for building Augmented Reality

applications. The motion tracking setup was

developed inside a simulated fMRI environment to

match the deployment in the Clinical Imaging

Facility at Hammersmith Hospital, London.

2 MATERIALS AND METHODS

2.1 Hardware Development

Our system consists of the motion tracking

installation inside the fMRI environment and the

software that runs the experiment by adapting its

phases and provided feedback to the subject's

captured performance. In particular, subjects lay

inside the fMRI-scanner holding the fMOVE object

in their dominant hand. The fMOVE adjusted

camera is installed at the distal end of the scanner

room with its telezoom lens facing the foot-end view

of the scanner cylinder with the hand-held object

NEUROTECHNIX2014-InternationalCongressonNeurotechnology,ElectronicsandInformatics

clearly visible (Fig. 2B). Camera images are

processed in real-time by a laptop-based system

which controls the experiment and can present real-

time feedback to the patient via the scanner’s visual

display setup. We used a PlayStation 3 Eye camera

(SONY, Tokyo) to track the motion of markers

attached to a hand-held object. This camera is able

to work with frame rates of 120Hz at a 320 x 240

pixel resolution and can also work with frame rates

of 60 Hz at 640 x 480 if more resolution is needed.

In addition, this device can be set for close up

framing at 56º field of view or 75º for long shot

framing. All these features provide the camera a

satisfying image acquisition quality for the needs of

our motion tracking setup. Its single component

price of 23£ (at time of printing) constitutes the sole

cost of fMOVE and therefore establishes the latter as

the cheapest 3DOF motion tracking technology

amongst a number of commercially available

systems (Fig. 1).

fMOVE’s motion tracking accuracy was tested

for 3 different camera lenses in order to examine

whether the markers are captured successfully both

for smaller and larger distances to the camera. We

selected the variable lens focal lengths taking into

account three different distances to the camera

(0.5m, 1m, 2.5m) based on:







∙



(1)

where  denotes the focal length, 



the width

of the CMOS sensor (3.98mm),  the Field Of

View (400mm),  the distance between the camera

and the tracked marker. The focal lengths for the

different distances were estimated as 6.35mm,

12mm and 35mm.

Figure 2: (A) Setup for the one-time calibration for any camera system to be used with fMOVE system. Rigid aluminium

frames allow mounting of camera and target calibration pattern at defined position and orientation. (B) fMOVE system in

use: healthy subject lying inside the fMRI-scanner holding the fMOVE object in the dominant hand. Camera installed at the

distal end of the scanner room with its telezoom lens facing the foot-end view of the scanner cylinder with the hand-held

object clearly visible. Camera images are processed in real-time by laptop-based system which controls the experiment and

can present real-time feedback to the patient via the scanner’s visual display setup. (C) Set-up to measure the motion

tracking performance of the fMOVE system against a commercial electromagnetic (i.e. non-optical) motion tracking system

used as reference gold-standard (LIBERTY polhemus). A reference fMOVE object (black cube) which contains the

magnetic motion-tracking sensor is moved at defined distances and orientations (see text for details). (D) The coordinate

systems used by fMOVE include the camera coordinates (X_C, Y_C, Z_C in red), the coordinate system of the camera

view (x_C, y_C in green) and the relative coordinates of the fMOVE object surface(s) (X_m, Y_m, Z_m in blue).

DevelopingaNovelfMRI-CompatibleMotionTrackingSystemforHapticMotorControlExperiments

2.2 Calibration

The camera was calibrated in a setup, which consists

of two reference planes; one marker based and one

camera based (Fig. 2A). The marker plane remained

fixed throughout the calibration whereas the camera

plane could be rotated around a reference point and

translated away from or closer to the marker plane.

The camera was positioned at a fixed height on the

camera plane and at variable distances to the camera

plane’s rotation axis. Altogether in each calibration

we tested three different camera positions with

regard to the camera plane rotation axis (11.5cm,

18.5cm, and 25.5cm), three different camera plane

rotation angles (29º, -17º, 19º or/and 65º) and three

different camera plane distances to the marker plane

(either 33cm, 49.5cm, 41cm or 56.5cm, 45cm,

37cm).

2.3 Coordinate systems

fMOVE takes three coordinate systems into account:

a camera screen based (2D), a camera based (3D)

and a marker based (3D) (Fig. 2D). The marker

coordinate system uses as reference the centre of the

marker, having 



and 



parallel to the borders of

the marker, and 



pointing away from the marker.

The marker centre is defined as (



, 



,



) =

(0,0,0). The relationship between the camera and

marker coordinate system is determined through

rotational and translational operations. In particular,

we used the following transformation, which reflects

a rotation followed by a translation:







































































∙

/2

 (2)

where 



and 



determine the values of the rotation

and translation matrices respectively (Kato &

Billinghurst, 1999) and V denotes an edge of the

cubic component of the object; it has a negative sign

due to the orientation of the axes in the camera

based coordinate system.

Figure 3: (A) Spatial and temporal alignment of fMOVE and Polhemus data during translational movements in the X, Y and

Z axis. (B) fMOVE and ground truth comparison based on R

and Root Mean Squared Error (RMSE) for translational

movements in the X, Y and Z axis. (C) Spatial and temporal alignment of fMOVE and Polhemus data during rotational

movements in the X, Y and Z axis. (D) fMOVE and ground truth (Polhemus, Liberty) comparison based on R

and RMSE

for rotational movements in the X, Y and Z axis.

NEUROTECHNIX2014-InternationalCongressonNeurotechnology,ElectronicsandInformatics

2.4 Testing the Tracking Accuracy

In order to assess the tracking accuracy of fMOVE,

the system was compared with Polhemus Liberty, a

commercial electromagnetic motion tracker (Fig.

2C). One Polhemus sensor was positioned exactly

in the centre of an exemplary multi-marker object.

The designed object was free of metal so as to suit

usage inside the fMRI scanner. It consisted of a

wooden handle and a plastic cube at the surfaces of

which we attached four different ARToolkit

compatible markers. These markers can be

identified by the image-processing algorithm of our

system and assigned to a different label according

to the pattern they display. Labelling the different

patterns enabled the identification of rotational

movements.

In this setup, the object’s 3D position could be

tracked simultaneously by the calibrated camera of

fMOVE and the electromagnetic sensor. The two

streams of motion data were subsequently

compared after their respective reference coordinate

systems were aligned. The estimated error between

them was used as a performance measure for

fMOVE’s motion tracking accuracy in translational

and rotational movements of the object.

Translational movements were constrained

between the borders of a specified workspace.

During translation, the camera was always tracking

the same face and consequently the same marker on

the object. On the other hand, during rotational

movements within the same workspace borders, the

system’s motion tracking switched between the

different markers positioned on the object surfaces.

3 RESULTS

We assessed the tracking accuracy of fMOVE by

testing its position measurements against a widely

used electromagnetic motion tracker (Polhemus

Liberty). The comparison was performed for three

different distances between camera and object plane

(0.5m, 1m and 2.5m). For each of these three cases

we tested 15 trials of pre-specified motor tasks. In

10 of these the object was translated and in the

remaining 5 it was rotated.

After temporally and spatially aligning the

position measurements of the marker-based fMOVE

and the sensor-based Polhemus Liberty, we noticed

that our motion tracking system acquires data

streams that accurately match our ground truth. This

matching is evident in the overlaid position plots for

both translational and rotational trials (Fig. 3A,C).

In fact, during rotational movements fMOVE

manages to efficiently avoid tracking omissions, by

switching from one marker to another (as is evident

in the color-coding of Fig. 3).

Two measures of comparison of the acquired

data streams (R

and RMSE) verify the efficiency

of fMOVE in motion tracking (Fig. 3B,D). In

translational movements R

reflects over 84%

accuracy for all tested distances between camera

and tracked object. The lowest R

level (R

= 0.846)

is estimated for the largest distance (2.5 m) in the y-

dimension. The same case produces the highest

RMSE (RMSE = 1.6181 cm). Throughout all cases,

the y-dimension produces the highest error levels,

which reflects the fact that fMOVE is mostly

sensitive along the axis that connects the camera

centre with the marker centre. Even these instances

however, do not significantly affect the R

levels as

displayed in Fig. 3B.

Similarly, in rotational movements our

assessment verifies a matching between the

measurements of our system and the ground truth.

The lowest R

level (R

= 0.8214) is estimated again

for the largest distance (2.5 m) in the y-dimension

(Fig. 3C) for which the corresponding RMSE =

1.6181cm (Fig. 3D). As in the translational

movements, the largest tracking errors here are

noted again in the y-dimension. It is thus evident

that fMOVE displays its highest sensitivity in the y-

direction for a broader range of movements.

4 DISCUSSION

Based on our experience in designing extremely

affordable Neurotechnology for eye-tracking

(Abbott and Faisal, 2012, Abbott et al, 2013),

wearable motion tracking and muscle signal

systems (Gavriel et al. 2013, Fara et al. 2013), we

now designed and built fMOVE, a first low-cost

fMRI-compatible marker-based motion tracking

system capable of capturing 3 DOF movement. The

system acquires behavioral data from subjects,

while they manipulate a marked object inside an

fMRI scanner and it provides to them online visual

feedback of motion and task performance.

We tested the efficiency of the system against

Polhemus Liberty, a commercial electromagnetic

motion tracker, which operates with high accuracy

at 240 updates per second. We found that fMOVE

achieves high tracking accuracy for both

translational and rotational movements of the

markers and preserves this accuracy for variable

distances of the camera to the moving object.

DevelopingaNovelfMRI-CompatibleMotionTrackingSystemforHapticMotorControlExperiments

fMOVE poses technical advantages since it allows

high frequency data acquisition inside the fMRI

environment which is commonly incompatible to all

widely used motion tracking technologies, due to

the applied magnetic field. Our system is amenable

to further customization depending on the needs of

the experimental study, designed to be carried out

inside the scanner. Such customization can include

developing a multiple-marker tracking algorithm, so

as to increase the motion tracking accuracy, avoid

false marker detections and cover the motion of

multiple body parts or more complex behavioural

tasks.

Importantly, apart from its compatibility to the

fMRI environment, fMOVE constitutes an ultra-

low-cost motion tracking technology, that limits

expenses to the price of the used camera. At the

same time, the methodological platform it supports,

offers promising advantages for future studies of

motor behaviour (Wolpert, et.al., 2011, Wolpert and

Flanagan, 2010). It namely enables a tight

integration of psychophysical and functional

imaging studies and can thereby guide

investigations of the still unknown neural

foundation of cortical action selection and motor

learning rules.

REFERENCES

Abbott, W. W.; Faisal, A. A. (2012) Ultra-low-cost 3D

gaze estimation: an intuitive high information

throughput compliment to direct brain–machine

interfaces, Journal of Neural Engineering, 9

(4),046016

Abbott, W. W.; Zucconi, A.; Faisal, A. A.; Large-field

study of ultra low-cost, non-invasive task level bmi

(2013) IEEE/EMBS Conf. Proc. on Neural Eng.

(NER),723-726

Brayanov, J. B., Press, D. Z., and Smith, M. A. (2012).

Motor Memory Is Encoded as a Gain-Field

Combination of Intrinsic and Extrinsic Action

Representations. Journal of Neuroscience.

Faisal, A. A. , Selen, L. P. J. and Wolpert, D. M. (2008)

Noise in the nervous system, Nature Reviews

Neuroscience, 9 (4),292-303

Faisal, A. A. , Stout, D., Apel, J. and Bradley, B. (2010)

The manipulative complexity of Lower Paleolithic

stone toolmaking,PloS one, 5(11): e13718,2010

Fara, S.; Vikram, C. S. , Gavriel, C.; Faisal, A. A. Robust,

ultra low-cost mmg system with brain-machine-

interface applications (2013) IEEE/EMBS Conf. Proc.

on Neural Eng. (NER),723-726

Gavriel, C.; Faisal, A. A.; ",Wireless kinematic body

sensor network for low-cost neurotechnology

applications “in-the-wild” (2013) IEEE/EMBS Conf.

Proc. on Neural Eng. (NER), 1279-1282

Thomik, A. A. C.; Haber, D. ,Faisal, A. A. (2013); Real-

time movement prediction for improved control of

neuroprosthetic devices, IEEE/EMBS Conf. Proc. on

Neural Eng. (NER), 625-628

Hecht, E. E.; Gutman, D. A.; Khreisheh, N.; Taylor, S.

V.; Kilner, J.; Faisal, A. A.; Bradley, B .A.;

Chaminade, T.; Stout, D.; (2014) Acquisition of

Paleolithic toolmaking abilities involves structural

remodeling to inferior frontoparietal regions, Brain

Structure and Function 1863-2653, 1-17

Haber, D.; Thomik, A. A. C.; Faisal, A. A. (2014).

Unsupervised time series segmentation for high-

dimensional body sensor network data streams, IEEE

Conf Wearable and Implant. Body Sensor Networks

(BSN),121-126,

Friston, K. (2010). The free-energy principle: a unified

brain theory? Nature Reviews. Neuroscience, 11(2),

127–138.

Friston, K. J., Daunizeau, J., and Kiebel, S. J. (2009).

Reinforcement learning or active inference? PLoS

ONE, 4. doi:10.1371/journal.pone.0006421

Ingram, J., Körding, K. P. , Howard, I. S. and Wolpert, D.

M. (2008) The statistics of natural hand movements,

Experimental Brain Research 188 (2), 223-236

Kato, H. and Billinghurst, M., 1999. Marker Tracking and

HMD Calibration for a Video-based Augmented

Reality Conferencing System. In Augmented Reality,

1999.(IWAR'99) Proceedings. 2nd IEEE and ACM

International Workshop on, pp. 85-94.

Shadmehr, R., and Moussavi, Z. M. (2000). Spatial

generalization from learning dynamics of reaching

movements. The Journal of neuroscience : the official

journal of the Society for Neuroscience (Vol. 20, pp.

7807–7815). doi:20/20/7807 (pii)

Shadmehr, R., and Mussa-Ivaldi, F. A. (1994). Adaptive

representation of dynamics during learning of a motor

task. The Journal of Neuroscience : The Official

Journal of the Society for Neuroscience, 14, 3208–

3224. doi:8182467

Syladi A., Faisal A.A. (2012). What is the hierarchical

representation of tasks involving objects with

complex internal dynamics? Frontiers in

Computational Neuroscience.doi:10.3389/conf.fncom

.2012.55.00129

Todorov, E. (2004). Optimality principles in sensorimotor

control. Nature Neuroscience, 7, 907–915. doi:10.

1038/nn1309

Todorov, E., and Jordan, M. I. (2002). Optimal feedback

control as a theory of motor coordination. Nature

Neuroscience, 5, 1226–1235. doi:10.1038/nn963

Wolpert, D. M., Diedrichsen, J., and Flanagan, J. R.

(2011). Principles of sensorimotor learning. Nature

Reviews. Neuroscience, 12(12), 739–51. doi:10.1038/

nrn3112

Wolpert, D. M. and Flanagan, J. R., 2010. Motor

learning. Curr Biol, pp. 467-472.

NEUROTECHNIX2014-InternationalCongressonNeurotechnology,ElectronicsandInformatics