A Method to Detect Keystrokes using Accelerometry to Quantify

Typing Rate and Monitor Neurodegenerative Progression

Ana Londral

, Mafalda Câmara

, Hugo Gamboa

, Mamede de Carvalho

1,3

Anabela Pinto

1,3

and Luís Azevedo

Instituto de Medicina Molecular, Faculdade de Medicina, Universidade de Lisboa, Lisbon, Portugal

Faculdade de Ciências e Tecnologia, Universidade Nova de Lisboa, Lisbon, Portugal

Hospital de Santa Maria, Lisbon, Portugal

Instituto Superior Técnico, Lisbon, Portugal

Keywords: Progressive Neurological Conditions, Motor Performance, Assistive Technologies, Accelerometer,

Amyotrophic Lateral Sclerosis.

Abstract: Progressive motor neurodegenerative diseases, as ALS, cause progressive loss of motor function in upper

limbs. Motor involvement, also affecting speech at some stage of disease, cause increasing difficulties in

accessing to computer devices (and internet tools) that allow communication with caregivers, and healthy

professionals. Thus, monitoring progression is important to anticipate new assistive technologies (AT), e.g.

computer interface. We present a novel methodology to monitor upper limb typing task functional

effectiveness. In our approach, an accelerometer is placed on the index finger allows to measure the number

of keystrokes per minute. We developed algorithm that was accurate when tested in three ALS patients and

in three control subjects. This method to evaluate communication performance explores the quantification

of movement as an early predictor of progression.

1 INTRODUCTION

Quantitative assessment of the motor performance of

human body has raised important questions and

scientific findings in last decades. Modern

technology allows to study movement using

miniaturized and wearable equipment. Namely,

accelerometers used as clinical tools, are broadly

used to monitor daily activity and tremor, specially

in movement disorders (Bonato, 2003; Godfrey et

al., 2008). In context of progressive diseases,

monitoring tools that can be used in the daily living

are important to track progression and adjust

treatments and interventions (Shany et al., 2012;

Bustamante et al., 2011).

The use of computer devices as assistive

technologies (ATs) for Communication is very

important concerning quality of life in some

neuromuscular diseases that affect speech or writing

abilities. Particularly in Amyotrophic Lateral

Sclerosis (ALS/MND), patients experiment

progressive loss of speech and limbs motor function

and consequent difficulties in communicating

without ATs (Korner et al., 2013; Beukelman et al.,

2000). Access to computer devices is also important

to give access to eHealth services for patients.

Although speech progression in ALS has been

studied for the purpose of monitoring

communication needs (Ball et al., 2002), writing

function (as upper limb motor progression) has been

underestimated as a variable for monitoring

communication abilities of ALS patients. ATs for

communication are commonly based on electronic

devices and typing tasks (considering the use of text-

to-speech technologies), either on a physical

keyboard or a virtual keyboard accessed via

touchscreen. Due to the neurodegenerative

characteristics, it is important to follow symptoms of

progression in upper limb motor control to identify

periods to adapt or introduce ATs, aiming at

augmenting users’ functionality in communication

(Beukelman et al., 2011; Bongioanni, 2012).

In this study, we aim to investigate the potential

of monitoring progression of upper limb motor

functionality needed to perform typing tasks on a

keyboard. For this purpose, we captured data from a

3D accelerometer placed on the index finger (finger

used to press the keys) and captured a 10-word

typing task. In this paper, we describe the

Londral A., Câmara M., Gamboa H., de Carvalho M., Pinto A. and Azevedo L..

A Method to Detect Keystrokes using Accelerometry to Quantify Typing Rate and Monitor Neurodegenerative Progression.

DOI: 10.5220/0004638800540059

In Proceedings of the International Congress on Neurotechnology, Electronics and Informatics (NEUROTECHNIX-2013), pages 54-59

ISBN: 978-989-8565-80-8

 2013 SCITEPRESS (Science and Technology Publications, Lda.)

methodology and the developed algorithm to detect

and quantify keystrokes events from accelerometer

signals captured during the experiments. As first

results, we present data from three ALS patients and

from three control subjects (with no diagnosed

neurological disease).

The rest of the paper is organized as follows:

Section 2 describes methodology used to acquire

data from accelerometer; Section 3 describes the

proposed algorithm for typing detection; in Sections

4 results from data analysis are presented; Section 5

and 6 present discussion and main conclusions on

the first results of this study.

2 METHODS

2.1 Participants

We present results from 3 patients with ALS and 3

healthy control subjects. ALS patients (2 women and

1 man) had a mean age of 53 years old (37, 59 and

63 years old). At baseline assessment, all

participants with ALS had clinical evaluation of

ALSFRS-r (Cedarbaum et al., 1999) speech

subscore less or equal to 2, though all were able to

use upper limbs to type on a keyboard. Patients had

no dementia. Healthy control subjects (2 women and

1 man) had mean age of 32 (23, 35 and 37) years

old.

2.2 Equipment

For accelerometry acquisition a BiopluxResearch

system (PLUX SA) was used. In our research

settings, we used the system with a 3-axial MEMS

accelerometer sensor (±3g measurement range).

Sensor was placed in exterior part of index finger of

the functional hand, as depicted in Fig.1. The three

axes were measured according to Fig.1 in directions:

anterior-posterior(X), distal(Y) and lateral(Z). Data

was sampled at 1KHz. Data was acquired via

Bluetooth to a laptop computer to be later processed

using Python tools. A second laptop was used by the

participants to perform the typing tasks.

Figure 1: Index finger for typing.

2.3 Procedure

Subjects were asked to type a 10-word sentence

using just the index finger. Accelerometer was

placed in the finger and data from the accelerometer

was saved. The same accelerometer sensor was

always used. Patients were evaluated in 3 sessions,

in 3 months intervals. Control subjects just

performed one trial in one session, as no progression

is expected for control participants. A camera was

also used to capture typing task, for results

validation.

Figure 2: Photo from an evaluation.

2.3.1 Outcome Measures

based on Accelerometer

For simplification, we used Y axis (distal

movements of index finger) to characterize typing

function, as this is the direction related to the

movement of pressing keys (as illustrated in Fig 1).

As outcome variables we wanted to have number of

keystrokes, typing rate, time between keystrokes,

time duration of each keystroke, amplitude of

acceleration signal (amplitude of acceleration of

finger movements in distal direction), magnitude of

acceleration signal (calculated as the Euclidean

vector for the 3-axis acceleration signal).

2.4 Algorithm for Typing Detection

We analysed accelerometer signals from patients

and control subjects. Key types from the 10-word

sentence were analysed. Fig.3 shows the plot from

the acceleration signal of one of the control subjects,

illustrating a set of keystrokes and one isolated

keystroke.

An algorithm was developed to extract outcome

variables from each typing task.

AMethodtoDetectKeystrokesusingAccelerometrytoQuantifyTypingRateandMonitorNeurodegenerativeProgression

Figure 3: Plots from keytyping from a control subject. (up)

Set of 13 keystrokes (down) One keystroke: first

acceleration pulses correspond to press action and the

second acceleration pulses correspond to release action.

2.5 Characterization of a Typing Action

For the setup suggested in this paper, a keystroke

(for simplicity, we consider signal from Y-Axis to

analyze movements in distal direction) is

characterized by two events of acceleration (Figure

3). The first event is related to pressing action and

the second event is related to releasing action.

2.5.1 Signal Processing for Detection

of Typing Actions

Proposed algorithm first removes DC component

from acquired data, then uses a moving average

algorithm to smooth the signal’s module (Figure 4).

From the processed signal, peaks are detected (from

a threshold value calculated as a factor of maximum

amplitude) as events of pressing and releasing keys

– each group of two near peaks corresponds to the

signal of a keystroke.

Due to erroneous movements (from video

analysis we could identify hand gestures performed

during typing task or touching a key with no

pressing action) two kinds of peaks were detected as

frequent false typing events:

- isolated event: isolated peaks or a peak close to

a pressing event (video analyses show that a single

peak may occur when user touches a key but doesn’t

press it – this is caused by a hesitation (commonly

caused by low experience in the use of a qwerty

keyboard);

- third event: we could observe in the acquired

signals frequent low amplitude acceleration impulses

prior to a typical keystroke signal. From the video

analysis we could conclude that these acceleration

impulses are due to slow typing (an evident delay

between touching the key and pressing it).

To guarantee a pair number of peaks

(press/release keys), previously described false

typing events were removed: signal was analysed

near each detected peak of acceleration. Isolated

events were eliminated and, in groups with more

than two close events, only the two with higher

amplitude values were kept.

Keystroke events detected with the proposed

algorithm are depicted in Fig 5.

Figure 4: (top left) raw signal; (top right) signal’s module; (bottom left) filtered signal using smooth algorithm; (bottom

right) bottom left signal and vertical lines representing peaks detection. All schematic representations are of 20s of samples

from one patient performing a typing task.

NEUROTECHNIX2013-InternationalCongressonNeurotechnology,ElectronicsandInformatics

Figure 5: Representation of keystrokes detected from

approximately 10s of accelerometer signal acquisition (a

patient performs 15 keystrokes, typing a 10-word

sentence).

3 RESULTS

All collected signals were processed with the

proposed algorithm. Features from typing task were

calculated and are represented in Table 1. We also

measured performance of typing task, presented in

second column of Table 1 as words per minute

(wpm), using visual video analysis.

We applied the algorithm to the signals of

control subjects and patients (different users and

three different periods during disease progression).

As can be observed in Table 1, in spite all

subjects were asked to do the same typing task, the

number of keystrokes performed in each typing task

(#ks) is not always the same. This is due to

variability among subjects experience. For example,

some of the patients forgot to add spaces between

two different words, some added more than one

space or used the backspace to correct a mistake.

One of the patients (patient P1, in Table 1) had no

experience in using computer devices, the reason

why we reduced the sentence to 5 words (half of the

sentence). In fact, as it is not the aim of this study to

evaluate the written text, participants had no specific

constraints (related to text) when performing the

typing task.

From visual observation of accelerometer plots, a

validation of the algorithm was performed, based on

a manual adjustment of the algorithm variables (i.e.

threshold value for peak detection and time window

to search for false events). Number of keystrokes

detected by the algorithm was confirmed with the

number counted in video observation. Results are

presented in Table 1. We calculated a Pearson

correlation of 0.98484 between words per minute

(obtained from video analysis) and keyrate (result of

proposed algorithm from accelerometer analysis) for

the set of all analyzed data. These results validate

the used methodology to performance evaluation.

4 DISCUSSION

Finding methodologies that can early predict

progression allows a faster and customized response

of interventions and care. We hypothesized there is a

relation between user performance on typing tasks

(measured as communication speed - wpm) and

respective kinematic analysis of typing and that this

can provide a more sensitive tool to detect

progression in upperlimbs motor function. A

methodology to collect data from a 3-axis

accelerometer placed on the index finger was

developed for typing task.

A set of data captured from healthy subjects and

ALS patients (in different time periods) was

analyzed. Presented results show that keystrokes

detected from the developed algorithm had high

correlation with performance measured by video

analysis. Although, high accuracy was due to fine

adjustment of two parameters, which had to be

manually adjusted for each data set. It was not

possible to establish a peak threshold common to all

studied signals – it had to be manually adjusted

within different users or along different samples

from the same user. Also, using the suggested

algorithm, it was difficult to distinguish very low

amplitude peak from noise or involuntary movement

– window size to remove false events had to be

manually adjusted for each sample.

From a preliminary analysis of the results

presented in this paper, we could observe that, for

ALS patients, in spite performance values in

different periods of evaluation are variable (we can’t

always find evident decrease in time), the maximum

amplitude of y-axis signal (V

máx, Table 1) and the

mean value of the amplitude of magnitude signal

(last column in Table 1) always decrease along the

different time periods of evaluation (3 months

interval, approximately). Although careful analysis

is part of future work for the presented study, these

preliminary results suggest a new surrogate marker

of typing function deterioration, potentially more

accurate than simple typing performance

observation.

5 CONCLUSIONS

In this paper we present an algorithm for studying

AMethodtoDetectKeystrokesusingAccelerometrytoQuantifyTypingRateandMonitorNeurodegenerativeProgression

Table 1: Results from the analysis of acquired data with the algorithm proposed for typing detection. Each row describes

the results from control subjects (C

) and patients (P

) in the different evaluation times (T

#ks – number of keystrokes; µ tbt_key – average of time interval between keystrokes; µ tkey – average time duration of

keystrokes; VYmáx – maximum amplitude of Y-axis of accelerometer signal; Magn. Máx – maximum amplitude of

magnitude of accelerometer signal; Magn.Mean – mean magnitude of the accelerometer signal).

Particip.

wpm

(video)

keyrate # ks

total

time

(s)

µ tbt_key µ tkey

máx

(mV)

Magn.

Máx

(mV)

Magn.

Mean

µV

C1 10 43 43 60 331.8 75 623.79 220.12 60.99

C2 10 44 44 60 852.1 244 253.44 151.91 32.65

C3 10 44 44 60 1011.4 144.5 299.53 192.05 31.18

P1.T0 3.04 13.17 18 82 3952.2 255.1 623.43 149.26 40.68

P1.T1 3.1 11.46 17 89 4819.4 186.6 555.29 314.36 33.6

P1.T2 3.87 12.00 13 65 4129 241.5 412.22 195 31.44

P2.T0 16.66 67.89 43 38 527.6 168.1 624.19 147.34 38.34

P2.T1 16.61 62.67 47 45 669.9 188.5 475.47 225.08 36.62

P2.T2 17.3 61.82 44 42.7 666.1 252.5 427.32 129.08 33.72

P3.T0 7.69 28.53 39 82 1282.3 712.8 495.77 212.19 42.64

P3.T1 9.38 33.75 45 80 736.3 847.1 480.26 177.54 32.35

typing performance through accelerometry.

A 3-axis accelerometer was placed in the index

finger of 6 participants (3 with progressive

neuromuscular disease and 3 healthy participants).

Signal processing of the accelerometer signals

showed high correlation between independent

measures of performance: words per minute (from

video analysis) and keystrokes per minute (from

accelerometer).

Presented algorithm should be improved to

automatically adjust all the parameters for different

users and different stages of progressive disease. As

future work, a detailed analysis of other parameters

of accelerometry, independent from performance

measures, should be done.

REFERENCES

Ball, L. J., Beukelman, D. R. and Pattee, G. L., 2002.

Timing of speech deterioration in people with

amyotrophic lateral sclerosis, Journal of Medical

Speech-Language Pathology, 10(4), 231–235.

Beukelman, D., Fager, S. and Nordness, A., 2011.

Communication Support for People with ALS. In

Neurology Research International, Article ID 714693.

Beukelman, D. R., Yorkston, K. M., Reichle, J., 2000.

Augmentative and Alternative Communication for

Adults with Acquired Neurologic Disorders. Brookes

H. Paul Publishing, Baltimore.

Bonato, P., 2003. Wearable sensors/systems and their

impact on biomedical engineering. In IEEE

Engineering in Medicine and Biology Magazine,

22(3), 18–20.

Bongioanni, P., 2012. Communication Impairment in ALS

Patients: Assessment and Treatment. In Maurer, M.

(Ed.) Amyotrophic Lateral Sclerosis. Available from:

http://www.intechopen.com/books/amyotrophic-

lateral-sclerosis.

Bustamante, P., Solas, G. and Grandez, K., 2011.

Neurodegenerative Disease Monitoring Using a

Portable Wireless Sensor Device. In Chang, R. (Ed)

Neurodegenerative Diseases - Processes, Prevention,

Protection and Monitoring, InTech Publisher.

Available from: http://www.intechopen.com/books/

neurodegenerative-diseases-processes-prevention-

protection-and-monitoring.

Cedarbaum, J. M., Stambler, N., Malta, E., Fuller, C., Hilt,

D., Thurmond, B., Nakanishi. A., 1999. The ALSFRS-

R: a revised ALS functional rating scale that

incorporates assessments of respiratory function.

BDNF ALS Study Group (Phase III). In Journal of the

Neurological Sciences,169(1-2),13-21.

Godfrey, A., Conway, R., Meagher, D. and ÓLaighin, G.,

2008. Direct measurement of human movement by

accelerometry, In Medical Engineering & Physics, 30,

1364–1386.

Korner, S., Siniawski, M., Kollowe, K., Rath, K.J.,

Krampfl, K., Zapf, A., Dengler, R., Petri, S., 2013.

Speech therapy and communication device: Impact on

NEUROTECHNIX2013-InternationalCongressonNeurotechnology,ElectronicsandInformatics

quality of life and mood in patients with amyotrophic

lateral sclerosis. In Amyotrophic Lateral Sclerosis and

Frontotemporal Degeneration, 14, 20–25.

Shany, T., Redmond, S. J., Narayanan, M. R. and Lovell,

N. H., 2012. Sensors based wearable systems for

monitoring of human movement and falls. In IEEE

Sensors Journal, 12(3), 658–670.

AMethodtoDetectKeystrokesusingAccelerometrytoQuantifyTypingRateandMonitorNeurodegenerativeProgression