Feature Extraction based on Touch Interaction Data in Virtual

Reality-based IADL for Characterization of Mild Cognitive

Impairment

Yuki Kubota

, Takehiko Yamaguchi

, Takuya Maeta

, Yosuke Okada

, Yoshihito Miura

Niken Prasasti Martono

, Hayato Ohwada

and Giovannetti Tania

Department of Applied Electronics, Graduated of Tokyo University of Science, Katsushika-ku, Tokyo, Japan

Department of Applied Electronics, Tokyo University of Science, Katsushika-ku, Tokyo, Japan

Department of Industrial Administration, Tokyo University of Science, Noda-shi, Chiba, Japan

Department of Psychology, Temple University, 1701 N. 13

Street, Philadelphia, U.S.A.

Keywords: Dementia, MCI, Screening, VR-IADK, Characterization.

Abstract: The aim of this study was to explore the feature pattern of Mild Cognitive Impairment (MCI) in Virtual Reality

based Instrumental Activities of Daily Living (VR-IADL) which runs on a tablet PC as well as requires

participants a touch interaction to complete the task. Twelve participants (MCI: 4, history of MCI: 2, healthy

elderly: 6) were recruited from the region of Philadelphia in USA to perform a VR-IADL task. We found that

Non Touch Time (NTT) which is time interval during not touching screen on tablet was longer than that of

MCI patients as well as healthy older adults with having history of MCI. Several types of feature patterns

were extracted from the NTT such as … Based on the feature pattern, Support Vector Machine (SVM) was

performed to calculate the accuracy of the feature patter for characterization of MCI. As the result, the

identification rate was 75%.

1 INTRODUCTION

The population of older adults is increasing, placing

a burden on health care resources. According to

World Alzheimer’s Report (2016), the world’s

dementia population included approximately 47

million in 2016 and is expected to increase to 131

million by 2050. Dementia is an irreversible

condition that includes symptoms that disturb daily

living due to degeneration of the brain and impaired

cognitive abilities. Mild Cognitive Impairment (MCI)

is the prodromal stage of dementia that is associated

with mild cognitive and functional difficulties. MCI

may be reversible; therefore, screening for MCI may

promote prevention of Alzheimer’s disease through

early intervention. Recent research has turned to the

characterization of early behavioral dysfunction in

MCI and cognitive aging (Seligman et al., 2013).

People with MCI have been shown to perform more

inefficiently than healthy older adults on

performance-based tests of instrumental activities of

daily living (IADL; i.e., learned, sequential, object-

oriented behaviour in the service of everyday goals

like meal preparation) (Schmitter-Edgecomb et al.,

2012). In several studies, neuropsychologists view

video recordings of the everyday action performance

of participants and code behaviour according to an

error taxonomy that includes micro-error and overt

error subtypes, such as omission error and various

types of commission (Giovannetti et al., 2008). The

method has been used with the Naturalistic Action

Test (NAT), a performance-based test of IADL.

IADL performance analysis of MCI participants must

include indicators of subtle errors or inefficient

behaviour. Some investigators have shown that time

to complete IADL tasks distinguish people with MCI

from healthy older adults (Wadley et al., 2008). Long

completion times in people with MCI may be the

result of slowed thinking and/or slowed movements;

however, to date investigators have not evaluated

motor vs. cognitive IADL speed in MCI. Virtual

Reality (VR)-based technology has been introduced

for environmental assistive technology and to assist

in diagnostic decision making. Yamaguchi et al.

developed the Virtual Kitchen application with VR-

based technology for assessment Alzheimer’s disease

152

Kubota Y., Yamaguchi T., Maeta T., Okada Y., Miura Y., Martono N., Ohwada H. and Tania G.

Feature Extraction based on Touch Interaction Data in Virtual Reality-based IADL for Characterization of Mild Cognitive Impairment.

DOI: 10.5220/0006265201520157

In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2017), pages 152-157

ISBN: 978-989-758-229-5

(Yamaguchi et al., 2012, Allain et al., 2015). The

Virutal Kitchen includes VR-based IADL tasks (VR-

IADL) to assess functional ability of people with

Alzheimer’s disease by comparing frequency of

omission error and commission error. Martono et al.

classified the error pattern based on the acceleration

changes of finger movement in a touch-screen version

of the Virtual Kitchen (Martono et al., 2016).

However, effective feature patterns that could clearly

classify MCI vs. healthy elderly remain elusive. The

aim of this study was to investigate the feature pattern

of MCI with behavioral data such as touching

interaction data in VR-IADL task. Because previous

studies have shown that people with MCI

demonstrate longer decision making times and slower

cognitive processing speed, we hypothesized that

MCI participants would show greater cognitive

slowing than older adults on IADL. By contrast, we

expected motor speed would be relatively spared.

Thus, we predicted that MCI participants would

spend more time not touching the screen (i.e., Non-

Touch Time; NTT) than healthy elderly.

2 EXPERIMENT AND METHOD

2.1 Participants

Table 1: Neuropsychological test.

Domain Test

Language

Category fluency;

Boston Naming Test

Executive

function

WAIS-R digit span backward;

Trail Making Test version B

Memory-

Immediate

Rey Figure in recall; WMS-R:

logical memory Ⅱ Story A

Episodic

Memory ability

Rey Figure copy; WAIS-R:

picture completion

Attention

TMT part A, WAIS-R:

digit span forward

Processing

speed

TMT part A, crossing-off test

The participants aged 60 years and older were

recruited from the region of Philadelphia in the USA.

Participants included 6 healthy elderly, 4 elderly with

MCI, and 2 elderly with a history of MCI but who no

longer met MCI diagnostic criteria at the time of

testing. Participants were classified with several test

such as MMSE (score > 24), GDS: (Geriatric

Depression Scale < 10) and neuropsychological tests

of specific abilities (Seelye, 2015) as shown in Table

1 (Kaye, 2011).

2.2 Apparatus

2.2.1 Experimental Setup

Figure 1: The experimental setup for the VR-IADL task.

Figure 1 shows the experimental setup for the task.

It included one tablet device, web camera for video

recording during the task, and the Virtual Kitchen

Challenge application, which is VR-IADL

assessment tool. Participants were seated in front of

the tablet device, which was placed on the table. They

were asked to use their finger to complete the task.

The finger enabled them to move virtual objects by

touching on the object with their finger and then

dragging the object to the desired position in the plane.

In this study, we used the Virtual Kitchen

Challenge (VKC), which is visually developed 3D

environment in order to improve reality. VKC could

record several types of data such as the objects the

participant touched and dragged in time series. Firure

2 shows an example of the structure of each task. This

application consists of two IADL tasks: (1) Toast and

coffee task, (2) Lunch box task. Each task has two

or three main tasks, and main task has several sub task

which is action to complete main task. For example,

Toast and coffee task has 2 main task: (1) Toast task,

(2) Coffee task. Toast task has 8 sub task, and coffee

task has 8 sub task, too.

Before performing the task, we introduced the

goal of task, and operation of VK and we were sure

that participants understood (Giovannetti, 2008).

2.2.2 Toast and Coffee Task

The toast and coffee task included 14 objects, 10

target objects and 4 distractor objects. Two of the

distractors were related (salt shaker, ice-cream scoop)

to the target objects and the other two distractors were

unrelated to the targets (paintbrush, ash tray). The

placement of the distractors was evenly distributed

within right/left and proximal/distal positions on the

Feature Extraction based on Touch Interaction Data in Virtual Reality-based IADL for Characterization of Mild Cognitive Impairment

153

screen. Figure 3 shows the screen shot of toast and

coffee task.

Figure 2: The example of the task structure.

Figure 3: Screen shot of the Toast and Coffee Task.

Participants were required to perfom 13 steps to

make toast with butter and jelly and instanct coffee

with cream and sugar. The steps included: (1) place a

slice of bread into the toaster; (2) turn on the switch

to toast; (3) remove toast and place on the plate; (4)

take butter; (5) spread butter on the toast; (6) open the

jelly jar; (7) place jelly on the toast with knife; (8)

open the coffee jar; (9) take a spoonful of coffee

powder with spoon and into the cup; (10) open the

suger pot jar;(11) place suger into cup with spoon;

(12) pour milk into the cup; (13) stir cup of coffee

with spoon.

2.2.3 Lunch Box Task

Lunch box task included 13 objects, 9 target objects

and 4 distractor objects. Two of the distractors were

related (fork, cup) to the target objects and the other

two distractors were unrelated to the targets (razor,

spray bottle). The placement of the distractors was

evenly distributed within right/left and

proximal/distal positions on the screen. Figure 4

shows the screen shot of the lunch task. The lunch

task required participants to make, wrap and pack a

peanut butter and jelly sandwich, prepare a thermos,

and wrap and pack cookies. The task included 16

steps: (1) take bread; (2) open the jelly jar; (3) place

jelly on the bread with knife; (4) open the peanut

butter jar; (5) place peanut butter on bread with knife;

(6) take another piece of bread to close sandwich; (7)

take a sheet of foil to wrap the sandwich; (8) pack the

wrapped sandwich into the lunchbox; (9) take cookies

(one by one); (10) take a sheet of foil then wrap the

cookies; (11) pack the wrapped cookies into the

lunchbox; (12) pour the juice into the themos; (13)

seal the thermos with lid; (14) seal the thermos with

cap; (15) pack thermos into the lunch box; (16) close

the lunch box (Martono, 2016).

Figure 4: Screen shot of the Lunch Box Task.

3 DATA AND METHODS

3.1 Movie Analysis

Video recordings of performances were reviewed for

error coding. As a result, we realized difference

between the Touch Time (TT) and Non Touch Time

(NTT). Figure 5 shows the flow of acquire several

data and definition of dragging time and not dragging

time.

Figure 5: Classification of behaviour.

In this study, we defined TT as the time

participants had contact with the screen. NTT was the

time when participants were not touching the screen;

see Figure 5. TT and NTT behaviour could be

HUCAPP 2017 - International Conference on Human Computer Interaction Theory and Applications

154

classified as shown in

Figure 6.

When participants

touched the screen, they make two types of actions:

(1) move object appropriately to complete task (2)

move object inappropriately (i.e., error). NTT

included three types of action: (1) no movement

(possibly resting or thinking), (2) movements above

tablet (possibly scanning and searching for object),

(3) reaching error.

Figure 6: Classification of behaviour.

We hypothesized that MCI participants would

show larger NTT times than healthy elderly, because

we hypothesized that people with MCI demonstrate

longer decision making times and slower processing

speed.

3.2 Touch Time Data

We hypothesized that the rate of touching time

(  ∈





) would be higher in healthy elderly

people compared to people with MCI, including those

with a history of MCI. We calculated RTT, using the

following mathematical formula (1).

 

∑









∑











∑









(1)

RTT was calculated with  ∈





\ and∈





. The Rate of Not Touching Time ( ∈ 



)

was calculated with the following mathematical

formula (2).

  1  

(2)

4 RESULT AND DISCUSSION

4.1 VR-IADL Analysis

The number of times the participant touched the

screen, the sum of TT and NTT, mean and variance

of TT and NTT, RTT and RNTT, which is showed

formula (1) and (2), were calculated for both IADL

tasks for the healthy elderly group and MCI groups

(i.e, MCI + history of MCI). Mann-Whitney tests for

the toast and coffee task showed significant

differences between the healthy and MCI participants

for variance of NTT (p < 0.05), mean of NTT (p <

0.1), the sum of NTT (p < 0.05), and RNTT and RTT

(p < 0.05). Also in lunch box task, there were

significant differences on two types of variance of

NTT (p < 0.05), the sum of NTT (p < 0.05) and RNTT

and RDT (p < 0.05).The mean NTT in the toast and

coffee task was longer for MCI participants than the

healthy group. Also, the sum of NTT and RNTT of

MCI group in both tasks is larger than that of healthy

group.

The results suggest that NTT could reflect slowed

cognitive speed and may be a behavioural feature that

reliably distinguishes MCI from healthy elders. The

variance of NTT was larger for the MCI group;

therefore, it is possible that the MCI group may be

experiencing more cognitive slowing at specific

points in the IADL (i.e., main task vs. subtask). In

next section, we analysed NTT and TT in different

IADL task segments.

4.2 Main Task Analysis

The lunch box task consists of three main tasks:

sandwich, snack, and juice. The toast and coffee task

consists of two main tasks: toast and coffee. We

observed how these main tasks were conducted by

analysing timing of subtasks in these main tasks.

Figure 7 shows the sequence in which the main tasks

were conducted. The timing of the sub tasks under

each the main tasks is represented by a different red

marker. This graph was generated automatically with

participant data. The blue, yellow, and green area

respectively represents the section of the main tasks

(i.e. sandwich task, snack task, and juice task).

Figure 8 shows the sequence of the toast and

coffee t main tasks and sub tasks under the main task.

The blue, and green area respectively represents the

section of the main tasks (i.e. toast task, and coffee

task).

In this study, the main task was defined as a set of

completion times of each subtask. The mean of NTT,

variance of NTT, the sum of NTT, the number of

Feature Extraction based on Touch Interaction Data in Virtual Reality-based IADL for Characterization of Mild Cognitive Impairment

155

Figure 7: The sequence of the conducted main task in

lunchbox task.

Figure 8: The sequence of the conducted main task in toast

and coffee task.

touching screen, RTT and RNTT for each task.

Mann-Whitney tests were performed on these

parameters. In lunchbox’s main task, there were

significant differences for NTT variance (p < 0.05,

sandwich and juice task), NTT mean (p < 0.05, only

sandwich task), the sum of NTT (p < 0.05, sandwich

task, p < 0.1, juice task) and RNTT (p < 0.05,

sandwich task). In toast task and coffee task, there

were significant difference for NTT variance (p <

0.05, both task), NTT mean (p < 0.1, both task), the

sum of NTT (p < 0.05, toast task, p < 0.1, coffee task)

and RNTT (p < 0.05, both task). There were no

significant differences for the number of times

participants touched the screen, though there were

significant differences for all data in toast and coffee

task.

The result indicates that NTT of MCI group is

longer and more variable than that of healthy group.

The sandwich and juice tasks were especially difficult

for the MCI group. From the task structure point of

view, the toast and coffee tasks are performed easily

in parallel; therefore, the causal relationship of

subtask of toast and coffee task could be more

complicated to perform.

Summarizing the above, (1) Sandwich and juice

task more difficult for the MCI group. (2) Task

structure could make participant confused. In next

section, we attempted to perform the NTT and TT

analyses on smaller IADL subtasks.

4.3 Sub Task Analysis

We tried to divided main tasks into sub tasks and

calculate NTT in order to find which sub task is

difficult for MCI group. However, it is difficult to

analyse because NTT data is small. We conclude that

we should analyse with another data to explore the

behaviour such as finger velocity data during VR-

IADL.

4.4 Classification with SVM

In order to investigate whether the features is useful

for classifying MCI, Support Vector Machine (SVM)

was performed on eleven parameters: variance and

mean of NTT in both tasks, RNTT in both tasks, NTT

variance in sandwich task, juice task, NTT mean in

sandwich task and variance of toast task and coffee

task. The accuracy of MCI including history of MCI

is 75%. Figure 9 showed that the result of ROC

analysis, which indicate that FPT is 0.5 and FPF is

0.67 and AUC is 0.71. This result suggests that NTT

and several parameters calculated with NTT can

classify MCI group.

Figure 9: ROC analysis.

HUCAPP 2017 - International Conference on Human Computer Interaction Theory and Applications

156

5 CONCLUSIONS

This study explored the features of IADL behaviours

that effectively distinguish MCI from healthy older

adults. The study focused on identifying cognitive

processing speed and motor processing speed using

time spent touching (TT) or not touching (NTT) the

screen on a VR-based IADL task. We conclude that

NTT was greater in participants with MCI or a history

of MCI. NTT may reflect slowed cognitive

processing speed; however, we were unable to

identify reliable behaviours during NTT did not see a

reliable relation between NTT and IADL subtask.

Therefore, future work is needed to understand why

NTT are longer in MCI. Future studies will include

larger participant samples and analysis with other

methods, including analysis of finger movements

during NTT.

REFERENCES

Seligman, S. C., Giovannetti, T., Sestito, J., & Libon, D. J.,

2013. “A new approach to the characterization of subtle

errors in everyday action: implications for mild

cognitive impairment,” The Clinical

Neuropsychologist, pp. 97-115..

Belchior, P. D. C., Holmes, M., Bier, N., Bottari, C., Mazer,

B., Robert, A., & Kaur, N., 2015. “Performance-based

tools for assessing functional performance in

individuals with mild cognitive impairment,” The Open

Journal of Occupational Therapy, vol. 3.

Schmitter-Edgecombe, M., McAlister, C., & Weakley, A.,

2012. “Naturalistic assessment of everyday functioning

in individuals with mild cognitive impairment: The

day-out task,” Neuropsychology, pp 631 – 641.

Giovannetti, T., Bettcher, B. M., Brennan, L., Libon, D. J.,

Burke, M., Duey, K., Nieves C & Wambach, D., 2008.

“Characterization of everyday functioning in mild

cognitive impairment: A direct assessment

approach.” Dementia and geriatric cognitive

disorders, pp 359-365.

Wadley, Virginia G., Ozioma Okonkwo, Michael Crowe,

and Lesley A. Ross-Meadows., 2008. "Mild cognitive

impairment and everyday function: evidence of reduced

speed in performing instrumental activities of daily

living." The American Journal of Geriatric Psychiatry

16, no. 5, 416-424.

Schwartz, Myrna F, Buxbaum, Laurel J, Veramonti Tracy,

Ferraro Mary, Segal Mary., 2001. “Instruction manual

for the naturalistic action test,” Interpretation A Journal

Of Bible And Theology, pp 24-28.

Richard, P., Massenot, L., Besnard, J., Richard, E., Le Gall,

D., & Allain, P., 2010. “A virtual kitchen to assess the

activities of daily life in Alzheimer’s disease,”

Proceedings of the International Conference on

Computer Graphics Theory and Applications, pp 378–

383.

Martono, Niken Prasasti, Takehiko Yamaguchi, and Hayato

Ohwada., 2016. "Utilizing Finger Movement Data to

Cluster Patients with Everyday Action Impairment.".

Yamaguchi, T., Foloppe, D. A., Richard, P., Richard, E., &

Allain, P., 2012. “A dual-modal virtual reality kitchen

for (re) learning of everyday cooking activities in

alzheimer's disease” Presence: Teleoperators and

Virtual Environments, 21(1), 43-57.

Allain, P., Foloppe, D. A., Besnard, J., Yamaguchi, T.,

Etcharry-Bouyx, F., Le Gall, D., ... & Richard, P., 2014.

Detecting everyday action deficits in Alzheimer’s

disease using a nonimmersive virtual reality

kitchen. Journal of the International

Neuropsychological Society, 20(05), 468-477.

Seelye, A., Hagler, S., Mattek, N., Howieson, D. B., Wild,

K., Dodge, H. H., & Kaye, J. A., 2015. “Computer

mouse movement patterns: A potential marker of mild

cognitive impairment,” Alzheimer's & Dementia:

Diagnosis, Assessment & Disease Monitoring, pp 472-

480.

Kaye, J. A., Maxwell, S. A., Mattek, N., Hayes, T. L.,

Dodge, H., Pavel, M., ... & Zitzelberger, T. A., 2011.

“Intelligent systems for assessing aging changes:

home-based, unobtrusive, and continuous assessment

of aging,” The Journals of Gerontology Series B:

Psychological Sciences and Social

Sciences,66(Suppl 1), pp i180-i190.

Feature Extraction based on Touch Interaction Data in Virtual Reality-based IADL for Characterization of Mild Cognitive Impairment

157