A New 2D Interaction-based Method for the Behavioral

Analysis of Instrumental Activities of Daily Living

Eulalie Verhulst

1,2

, D

eborah Foloppe

1,2

, Paul Richard

, Fr

eric Banville

and Philippe Allain

Laboratoire Angevin de Recherche en Ing

enierie des Syst

emes (LARIS), Universit

e d’Angers, Angers, France

Laboratoire de Psychologie des Pays de Loire (LPPL), Universit

e d’Angers, Angers, France

epartement de Psychologie, Universit

e de Montr

eal, Montr

eal, Canada

Keywords:

Behavioral Analysis, 2D Interaction, Activity of Daily Living.

Abstract:

In neuropsychology, many computerized solutions have been proposed in order to assess patients’ function-

ing in activities of daily living, via realistic interactive simulation. In this context, most developed systems

are based on simple devices, real time 2D interaction, and monoscopic 3D computer graphics environment.

Behavioral analysis has drawn the interest of many domains, such as neuropsychology, ergonomics, web de-

sign, or virtual reality. However, advances on this topic remains fragmented in their respective areas. Thus,

in computerized solutions applied to neuropsychology, the behavioral analysis does not take into account the

data from interaction. The potential interest of computerized solutions is hence underexploited. In this paper,

we propose a transdisciplinary solution, based on a ﬁner analysis of 2D interaction data, such as stop duration.

This method could reveal interesting aspects of users’ behaviors.

1 INTRODUCTION

Intrumental Activities of Daily Living (IADL; e.g.,

cooking, shopping, driving) are assessed to help the

diagnosis of dementia in clinical practice, as a recom-

mended primary outcome in several kinds of research

in dementia. Indeed, the IADL assessment permits

to propose relevant solutions to help patients deal-

ing with their everyday difﬁculties (Seligman et al.,

2014).

In clinical practice, IADL assessment is gener-

ally done using questionnaires (e.g., IADL scale,

Lawton and Brody, 1969). However, for an early

detection or rehabilitation, they lack of objectivity

and they are insufﬁcient to provide a comprehensive

view of the patients’ everyday difﬁculties. Other tra-

ditional tests, which focus on different speciﬁc as-

pects of perception, cognition, or motricity, are in-

sufﬁcient to detect everyday difﬁculties for some pa-

tients (Lewis et al., 2011) and may fail to reﬂect the

real measurements (Burgess et al., 2006; Chaytor and

Schmitter-Edgecombe, 2003). In other words, the tra-

ditional tests lack of ecological validity (Campbell

et al., 2009).

In response, occupational therapy and neuropsy-

chology have developed various methods based on the

performance in IADL activities, such as cooking in

front of the experimenter. This real-world approach

aims to provide a direct and comprehensive behav-

ioral analysis of the patient on the IADL he/she has

to perform. Nevertheless, such assessments are often

difﬁcult to set up, because of time consuming, perish-

able storage and cost, or potential risks (e.g., burn).

Several computer-based assessments have also

been developed to submit patients to controlled

situations with which they can act without risks.

This approach typically uses traditional desktop

human-computer interfaces (e.g., monitor, mouse).

In the most basic conﬁguration, the patient selects

2D graphics using 2D interaction techniques (e.g.,

mouse or touch-screen). However, these solutions

lack of realism and do not allow a ﬁne behavior

analysis. Consequently, several studies using such

solutions have reported a little predictive value

of everyday functioning (Armstrong et al., 2013;

Schultheis et al., 2002). For these reasons, Virtual

Reality (VR) techniques have been proposed to

assess IADL (Louisy et al., 2003). VR provides

3D pseudo-natural real-time multimodal virtual

environments (Arnaldi et al., 2003), and 3D inter-

action techniques based on real-world interaction

(e.g., pointing, walking-in-place). In this context,

the use of wireless motion capture devices (e.g.,

Microsoft Kinect

T M

) allows pseudo-natural users’

146

Verhulst E., Foloppe D., Richard P., Banville F. and Allain P.

A New 2D Interaction-based Method for the Behavioral Analysis of Instrumental Activities of Daily Living.

DOI: 10.5220/0006255901460151

In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theor y and Applications (VISIGRAPP 2017), pages 146-151

ISBN: 978-989-758-229-5

interaction. However, several limitations compromise

the use of VR techniques, especially for for elderly or

vulnerable patients. For example, 3D gestures-based

selection or manipulation techniques require rela-

tively high energy expenditure for them and therefore

cannot be used in long experiments (Sousa Santos

et al., 2009; Verhulst et al., 2016). In addition, some

VR systems lack of portability, and are not easy to

setup at home. Moreover, a recent study about VR

has pointed out the lack of ecological validity of the

fully immersive systems (Negut¸ et al., 2016). The

authors have suggested that VR adds complexity and

difﬁculty, and cognitive load, compared to the real

world.

A good compromise between these 2 approaches

has been used, as a kind of technically non-immersive

system. Such systems are VR-based, because they

simulate 3D IADL environments, propose a real-time

interaction, and use VR as science to develop realimic

experience of tasks. However, like in the most basic

solution, they use a 2D desktop conﬁguration. When

the manipulation of several objects is a key element

of the evaluated tasks (e.g., cooking, laundry, small

home repairs), mice or touch screens are typically

used. In other words, these systems use a 2D inter-

action interface, to interact with 3D computer graph-

ics environments, displayed monoscopically (Richard

et al., 2010; Allain et al., 2014; Zhang et al., 2003).

Traditional desktop interfaces offer a good porta-

bility for clinical practice. The monoscopic dis-

play avoids discrepancy and eye fatigue (Teather and

Stuerzlinger, 2015) for assessed patients. Moreover,

several studies have compared the use of the mouse

in 2D and 3D environments, showing that the use of

mouse is easier and more accurate for 2D interac-

tion than for 3D interaction (Teather and Stuerzlinger,

2015; Ware and Lowther, 1997). Mice and touch

screens are commonly used 2D interaction interfaces.

However, even if mice and touch screens have a com-

parable efﬁciency, users make more errors when using

a touch screen (Forlines et al., 2007), especially for

the elderly, due to bad wrist posture (Motti and Caine,

2016). Importantly, research on IADL assessment us-

ing such RV-based systems has shown adequate re-

liable and valid results, although these systems were

technically non-immersive. Moreover, they can pro-

vide explicative data about the origins of this limita-

tion (Banville and Nolin, 2012).

A remaining problem with 3D computer graph-

ics environment based on 2D interaction techniques

is the difﬁculty to control the mouse pointer mou-

vements, especially for the elderly or some disabled

people. This problem may lead to artiﬁcial unwanted

errors. Consequently, the ecological validity of ob-

tained measured could be affected. Taking into ac-

count the wide use of this kind of systems, there is

a crucial need to develop solutions to distinguish us-

ability errors from cognitive errors. To achieve a

better comprehension of users abilities in computer-

based assessment, we propose a new method to in-

vestigate behavior of patients with IADL limitations

due to cognitive deﬁcits by analyzing mouse activities

such as pathways and mouse events (e.g. click).

The following Section provides a short survey of

the related works concerning real-world assessments

of IADL (types of errors), and in mouse data anal-

ysis (mouse interaction with 2D content). Then, we

propose an alternative method to analyze users perfor-

mance, taking into account usability errors and cogni-

tive errors. The paper ends by a conclusion and tracks

for future works.

2 RELATED WORKS

2.1 Types of Errors

Using the approach based on performance in the

real-world, neuropsychological research on IADL

difculties has yielded observable taxonomies of

errors. In this context, (Giovannetti et al., 2008) have

proposed and formulated the commission-omission

model. This model postulate that global cognitive

deﬁcit and episodic memory deﬁcit lead patients

to forget the steps associated with management

of the task (i.e., produce omissions errors). Thus,

a deﬁcit in executive functions leads patients to

incorrectly perform pertinent actions or to perform

no relevant actions (i.e., produce commissions

omissions). The commissions errors do not prevent

the tasks completion, but can make more difﬁcult

the goals achievement. There are several types

of commission errors such as perseveration where

participants realize the same action sequence more

than one time, the omission-anticipation where

they make an action step in a non-conventional

order and ﬁnally substitution where they select a

given object semantically related to the expected

object. For instance, turning on the coffee ma-

chine before ﬁlling it with water is a commission

error, while to not put water at all is an omission error.

In line with the commission-omission

model, Seligman et al. (2014) have proposed to

take account of the inefﬁcient actions, but not overtly

erroneous, called micro-errors. The micro-errors

include too many steps in the realization of the

A New 2D Interaction-based Method for the Behavioral Analysis of Instrumental Activities of Daily Living

147

task, wrong sequencing and microslips which are

initialization of an unﬁnished action. Micro-errors

are pertinent for detecting cognitive decline in

pathological ageing (Seligman et al., 2014).

2.2 2D Interaction with 2D Content

The use of mouse is known by a majority of peo-

ple and does not cause fatigue for the user (Besanc¸on

et al., 2016). Several studies have investigated the in-

terest of the mouse movements analysis. Although

mainly conducted in a context of free web infor-

mation searching (Shapira et al., 2006; Claypool

et al., 2001; Guo and Agichtein, 2008), they pro-

vide useful results. For instance, when participants

focus on website reading, they produce more mouse

movements (Shapira et al., 2006), and more mouse

wheels (Claypool et al., 2001). Guo and Agichtein

(2008) have found that, when users want to switch

to another internet link, they make direct movements

into the target. When they are reading or searching

for information, they spent more time on the web-

site, and make longer and random movements with

the mouse (Guo and Agichtein, 2008). Overall, users

make long mouse movements when they are cog-

nitively involved, whereas they make quick move-

ments, doing a motor task (e.g., clicking on a link),

when they are focusing on some information (Guo

and Agichtein, 2008).

Other mouse movements analyses have shown the

usefulness of this approach, to extract behavioral pat-

terns, and to discriminate proﬁle groups. For exam-

ple, Seelye et al. (2015) have pointed out that a group

of people with mild cognitive impairment was less ef-

ﬁcient to use the mouse than a group of healthy el-

derly. The impaired people needed more time to click

on icons during web searching, make large mouse

movements, and make fewer movements, compared

to the healthy elderly. More generally, the elderly

could have issues with the mouse, especially for the

most complex actions, like double click. Smith et al.

(1999) have shown that the elderly spent more time

to move the mouse, and make longer path to reach

a target, because they are less precise compared to

younger people.

3 PROPOSAL

We propose to add the analysis of data from inter-

action, in the context of the IADL assessment by

”non-immersive VR” systems. The steps of the task

which are compromised can be detected according to

the omission-commission model (Giovannetti et al.,

2008, 2012). The analysis of data from interaction

could distinguish usability errors from cognitive er-

rors. We hence could improve the validity of assess-

ment by ”non-immersive VR” systems. Moreover,

we could specify new behavioral patterns, similarly

to web searching study and in the Second Life (Harris

et al., 2009). We hence could provide new data about

how/why some steps are compromised.

3.1 Proposed Data Analysis

Data from interaction can be collected while the par-

ticipant performs the computerized task. These data

include mouse path, halt time, click(s) on item, wrong

click, drag and drop, and movement velocity. Dif-

ferent kinds of data could be displayed and replayed,

in the experimented environment, for individual or

group synthesis (see Fig. 1).

Figure 1: Example of visualization of some mouse data for

a coffee-making task: Mouse path (full line), halt time (cir-

cle), click (cross), drag and drop (dotted line).

The mouse path includes an origin, an end, and

several sub-movements. Sub-mouvements can be de-

ﬁned by their length and direction (Hourcade, 2006)

separated by pauses (Almanji et al., 2014). Gener-

ally, the ﬁrst sub-movement has a higher velocity and

others are slower (Thompson et al., 2007). We ex-

pect that, during the phases of cognitive planing, the

mouse movements will be slow and random, whereas

they will be faster during a planned action.

Several clicks on an object could reveal a lack of

control-display ratio between click and objects ani-

mation, or a micro-error (Seligman et al., 2014).

A wrong click is a click outside a point of interest.

This kind of data could reveal bad system usability or

cognitive perturbations.

3.2 Planned Experiment

The data analysis that we propose can be used in var-

ious contexts. In our case, we project to use it in

HUCAPP 2017 - International Conference on Human Computer Interaction Theory and Applications

148

Figure 2: Experimental setup (left) and screenshot of the coffee task (right) from our previous work.

line with our previous work on IADL assessment in

impaired people (Allain et al., 2014; Richard et al.,

2010). More clearly (see Fig.2), we will use a sys-

tem based on VR, but non-immersive, which pro-

poses to perform a coffee-making task, in a virtual

kitchen (Richard et al., 2010). Coffee-making tasks

are commonly used in the ecological assessment and

rehabilitation of IADL (Allain et al., 2014; Foloppe

et al., 2015; Zhang et al., 2003). In our task, patient

has to pick and place virtual objects, in order to pre-

pare a cup of coffee with milk and sugar. The mouse

allows controlling the position of the virtual objects

on the vertical and horizontal axis. The virtual ob-

jects are automatically moved in depth. A behavioral

analysis according the omission-commission is avail-

able (Allain et al., 2014). We plan to integrate the

analysis of mouse data, and complete the behavioral

analysis.

We wonder whether the patterns obtained from

this kind of data are similar to those obtained from

our current assessment (Allain et al., 2014). We will

conduct an experiment in young people, old people,

and cognitively impaired people. We expect to ﬁnd

different behavioral patterns in accordance with the

age of the population. Indeed, the elderly could make

more pauses and more unwanted clicks, especially

if they have mild cognitive impairment, than young

adults. Behavioral pattern could help to make a dis-

tinction between proﬁle groups. Indeed, monitoring

data will permit to detect pattern in the use of a virtual

kitchen to prepare a cup of coffee. The data recorded

by the mouse could help us to detect moment where

participant are thinking (reﬂections steps), where un-

timely clicks are made and actions sequence realiza-

tion could help to detect where mistakes occurs and

what kind of errors it is according the commission-

omission model (Giovannetti et al., 2008, 2012). Fur-

thermore; we could make in relation monitoring data

and actions sequence realization to detect in which

case participant needs more time to prepare the cup

of coffee. Is it after an error or in the same moment

for several participants?

4 CONCLUSION AND FUTURE

WORK

3D computer graphics environments enable to assess

patients behaviors in ecological contexts. The aim of

this proposal is to suggest a complementary method

to analyze participants performance during cognitive

tasks. Actually, error analyzes help to understand par-

ticipants performance, but it is difﬁcult to discrimi-

nate usability errors from errors due to cognitive per-

turbations. For instance, some variables (e.g., time

completion) can inform on participants performance

and on their ability to interact with the virtual envi-

ronments. In order to differentiate cognitive errors

from usability errors we propose to analyze mouse

movements. In this context, we will try to identify be-

havioral patterns during the simulated task and com-

pare them with those found in a web searching activ-

ity (Seelye et al., 2015; Shapira et al., 2006; Claypool

et al., 2001). Our analyzes could allow discriminating

usability errors from cognitive perturbations errors. In

addition, the mouse movements analysis could help

us to have a better understanding of the participants

performance and could be adapted in several virtual

environments with 2D interaction. Moreover as mice

do not cause fatigue to the users (Besanc¸on et al.,

2016), we could think about workload associated to

interaction technique in VR. Indeed, a recent study

has pointed out that the lack of ecological validity of

the fully immersive systems (Negut¸ et al., 2016). The

authors have suggested that VR adds complexity and

difﬁculty, and cognitive load, compared to the real

world. This suggestion is comforted by Allain et al.

(2014), who found more errors in a virtual coffee task

than during the same task in reality.

A New 2D Interaction-based Method for the Behavioral Analysis of Instrumental Activities of Daily Living

149

REFERENCES

Allain, P., Foloppe, D. A., Besnard, J., Yamaguchi, T.,

Etcharry-Bouyx, F., Le Gall, D., Nolin, P., and

Richard, P. (2014). Detecting everyday action deﬁcits

in alzheimer’s disease using a nonimmersive virtual

reality kitchen. Journal of the International Neuropsy-

chological Society (JINS), 20(5):468–477.

Almanji, A., Davies, T. C., and Stott, N. S. (2014). Using

cursor measures to investigate the effects of impair-

ment severity on cursor control for youths with cere-

bral palsy. International Journal of Human-Computer

Studies, 72(3):349–357.

Armstrong, C. M., Reger, G. M., Edwards, J., Rizzo, A. A.,

Courtney, C. G., and Parsons, T. D. (2013). Validity

of the virtual reality stroop task (vrst) in active duty

military. Journal of Clinical and Experimental Neu-

ropsychology, 35(2):113–123.

Arnaldi, B., Fuchs, P., and Tisseau, J. (2003). Introduction

la r

ealit

e virtuelle (chapitre 1). Les Presses de l’Ecole

des Mines, Paris, France.

Banville, F. and Nolin, P. (2012). Using virtual reality to as-

sess prospective memory and executive functions after

traumatic brain injury. Journal of Cybertherapy & Re-

habilitation, 5(1):45–55.

Besanc¸on, L., Issartel, P., Ammi, M., and Isenberg, T.

(2016). Usability Comparison of Mouse, Touch and

Tangible Inputs for 3D Data Manipulation.

Burgess, P. W., Alderman, N., Forbes, C., Costello, A.,

Coates, L. M.-A., Dawson, D. R., Anderson, N. D.,

Gilbert, S. A. M. J., Dumontheil, I., and Channon, S.

(2006). The case for the development and use of “eco-

logically valid” measures of executive function in ex-

perimental and clinical neuropsychology. Journal of

the International Neuropsychological Society (JINS),

12(2):”194–209”.

Campbell, Z., Zakzanis, K., D.Jovanovski, Joordens, S.,

Mraz, R., and Graham, S. J. (2009). Utilizing vir-

tual reality to improve the ecological validity of clini-

cal neuropsychology: an FMRI case study elucidating

the neural basis of planning by comparing the tower of

london with a three-dimensional navigation task. Ap-

plied Neuropsychology, 16:295–306.

Chaytor, N. and Schmitter-Edgecombe, M. (2003). The

ecological validity of neuropsychological tests: A re-

view of the literature on everyday cognitive skills.

Neuropsychology Review, 13(4):”181–197”.

Claypool, M., Le, P., Wased, M., and Brown, D. (2001). Im-

plicit interest indicators. In Proceedings of the 6

In-

ternational Conference on Intelligent user Interfaces,

pages 33–40.

Foloppe, D. A., Richard, P., Yamaguchi, T., Etcharry-

Bouyx, F., and Allain, P. (2015). The potential of

virtual reality-based training to enhance the functional

autonomy of Alzheimer’s disease patients in cooking

activities: A single case study. Neuropsychological

Rehabilitation, (October):1–25.

Forlines, C., Wigdor, D., C.Shen, and Balakrishnan, R.

(2007). Direct-touch vs. mouse input for tabletop dis-

plays. In Proceedings of the 2007 Conference on Hu-

man Factors in Computing Systems (CHI 2007), San

Jose, California, USA, April 28 - May 3, 2007, pages

647–656.

Giovannetti, T., Bettcher, B. M., Brennan, L., Libon, D.,

Burke, M., Duey, K., Nieves, C., and Wambach, D.

(2008). Characterisation of everyday functioning in

mild cognitive impairment: A direct assessment ap-

proach. Dementia and Geriatric Cognitive Disorders,

25:359–365.

Giovannetti, T., Britnell, P., Brennan, L., Siderowf, A.,

Grossman, M., Libon, D. J., Bettcher, B. M., Rouzard,

F., Eppig, J., and eidel, G. A. (2012). Everyday action

impairment in parkinson’s disease dementia. Jour-

nal of the International Neuropsychological Society

(JINS), 18(5):787–98.

Guo, Q. and Agichtein, E. (2008). Exploring mouse move-

ments for inferring query intent. In 31

annual in-

ternational ACM SIGIR conference on Research and

development in information retrieval, pages 707–708.

Harris, H., Bailenson, J. N., Nielsen, A., and Yee, N. (2009).

The evolution of social behavior over time in sec-

ond life. Annual Review of Ecology and Systematics,

5(6):325–383.

Hourcade, J. P. (2006). Learning from preschool children’s

pointing sub-movements. In Proceedings of the 2006

conference on Interaction design and children, pages

65–72. ACM.

Lawton, M. P. and Brody, E. M. (1969). Assessment of

older people: Selfmaintaining and instrumental activ-

ities of daily living. Gerontologist, 9(3).

Lewis, M. W., Babbage, D. R., and Leathem, J. M.

(2011). Assessing executive performance during cog-

nitive rehabilitation. Neuropsychological Rehabilita-

tion, 21(2):145–163.

Louisy, T., Richard, P., Morin, D., Gall, D. L., and Al-

lain, P. (2003). Virtual kitchen: A virtual environment

for assessment and rehabilitation of impaired mem-

ory. In Proceedings of the 5

Virtual Reality Interna-

tional Conference (VRIC’03), pages 113 –117, Laval,

France.

Motti, V. G. and Caine, K. (2016). Smart wearables or dumb

wearables?: Understanding how context impacts the

UX in wrist worn interaction. In Proceedings of the

ACM International Conference on the Design of

Communication, SIGDOC, Silver Spring, MD, USA,

September 23-24, 2016, page 10.

Negut¸, A., Matu, S.-A., Sava, F. A., and David, D. (2016).

Task difﬁculty of virtual reality-based assessment

tools compared to classical paper-and-pencil or com-

puterized measures. Computers in Human Behavior,

54(C):414–424.

Richard, P., Massenot, L., Besnard, J., Richard, E., Gall,

D. L., and Allain, P. (2010). A virtual kitchen to

assess the activities of daily life in alzheimer’s dis-

ease. In Proceedings of the 5

International Confer-

ence on Computer Graphics Theory and Applications

(GRAPP 2010), Angers, France, May 17-21, pages

378–383.

Schultheis, M. T., Himelstein, J., and Rizzo, A. A. (2002).

Virtual reality and neuropsychology: upgrading the

HUCAPP 2017 - International Conference on Human Computer Interaction Theory and Applications

150

current tools. The Journal of Head Trauma Rehabili-

tation, 17(5):378–394.

Seelye, A., Hagler, S., Mattek, N., Howieson, D. B., Wild,

K., Dodge, H. H., and Kaye, J. A. (2015). Com-

puter mouse movement patterns: A potential marker

of mild cognitive impairment. Alzheimer’s & Demen-

tia: Diagnosis, Assessment & Disease Monitoring,

1(4):472–480.

Seligman, S. C., Giovannetti, T., Sestito, J., and Libon, D. J.

(2014). A new approach to the characterization of sub-

tle errors in everyday action: Implications for mild

cognitive impairment. The Clinical Neuropsycholo-

gist, 28(1):97–115.

Shapira, B., Taieb-Maimon, M., and Moskowitz, A. (2006).

Study of the usefulness of known and new implicit in-

dicators and their optimal combination for accurate in-

ference of users interests. In Proceedings of the 2006

ACM Symposium on Applied Computing, pages 1118–

1119.

Smith, M. W., Sharit, J., and Czaja, S. J. (1999). Aging, mo-

tor control, and the performance of computer mouse

tasks. Human Factors, 41(3):389–96.

Sousa Santos, B., Dias, P., Pimentel, A., Baggerman, J. W.,

Ferreira, C., Silva, S., and Madeira, J. (2009). Head-

mounted display versus desktop for 3D navigation in

virtual reality: A user study. Multimedia Tools and

Applications, 41(1):161–181.

Teather, R. and Stuerzlinger, W. (2015). Factors affecting

mouse-based 3d selection in desktop vr systems. In

Proceedings of the ACM Symposium on Spatial User

Interaction (SUI), pages 10–19.

Thompson, S. G., McConnell, D. S., Slocum, J. S., and Bo-

han, M. (2007). Kinematic analysis of multiple con-

straints on a pointing task. Human movement science,

26(1):11–26.

Verhulst, E., Richard, P., Richard, E., Allain, P., and Nolin,

P. (2016). 3D interaction techniques for virtual shop-

ping: Design and preliminary study. In Proceedings of

the 11

Joint Conference on Computer Vision, Imag-

ing and Computer Graphics Theory and Applications

(GRAPP 2016), pages 271–279.

Ware, C. and Lowther, K. (1997). Selection using a one-

eyed cursor in a ﬁsh tank vr environment. ACM Trans-

actions on Computer-Human Interaction (TOCHI),

4(4):309–322.

Zhang, L., Abreu, B. C., Seale, G. S., Masel, B., Chris-

tiansen, C. H., and Ottenbacher, K. J. (2003). A vir-

tual reality environment for evaluation of a daily liv-

ing skill in brain injury rehabilitation: Reliability and

validity. 84(8):1118–1124.

A New 2D Interaction-based Method for the Behavioral Analysis of Instrumental Activities of Daily Living

151