A Method to Detect Keystrokes using Accelerometry to Quantify
Typing Rate and Monitor Neurodegenerative Progression
Ana Londral
1
, Mafalda Câmara
2
, Hugo Gamboa
2
, Mamede de Carvalho
1,3
,
Anabela Pinto
1,3
and Luís Azevedo
4
1
Instituto de Medicina Molecular, Faculdade de Medicina, Universidade de Lisboa, Lisbon, Portugal
2
Faculdade de Ciências e Tecnologia, Universidade Nova de Lisboa, Lisbon, Portugal
3
Hospital de Santa Maria, Lisbon, Portugal
4
Instituto Superior Técnico, Lisbon, Portugal
Keywords: Progressive Neurological Conditions, Motor Performance, Assistive Technologies, Accelerometer,
Amyotrophic Lateral Sclerosis.
Abstract: Progressive motor neurodegenerative diseases, as ALS, cause progressive loss of motor function in upper
limbs. Motor involvement, also affecting speech at some stage of disease, cause increasing difficulties in
accessing to computer devices (and internet tools) that allow communication with caregivers, and healthy
professionals. Thus, monitoring progression is important to anticipate new assistive technologies (AT), e.g.
computer interface. We present a novel methodology to monitor upper limb typing task functional
effectiveness. In our approach, an accelerometer is placed on the index finger allows to measure the number
of keystrokes per minute. We developed algorithm that was accurate when tested in three ALS patients and
in three control subjects. This method to evaluate communication performance explores the quantification
of movement as an early predictor of progression.
1 INTRODUCTION
Quantitative assessment of the motor performance of
human body has raised important questions and
scientific findings in last decades. Modern
technology allows to study movement using
miniaturized and wearable equipment. Namely,
accelerometers used as clinical tools, are broadly
used to monitor daily activity and tremor, specially
in movement disorders (Bonato, 2003; Godfrey et
al., 2008). In context of progressive diseases,
monitoring tools that can be used in the daily living
are important to track progression and adjust
treatments and interventions (Shany et al., 2012;
Bustamante et al., 2011).
The use of computer devices as assistive
technologies (ATs) for Communication is very
important concerning quality of life in some
neuromuscular diseases that affect speech or writing
abilities. Particularly in Amyotrophic Lateral
Sclerosis (ALS/MND), patients experiment
progressive loss of speech and limbs motor function
and consequent difficulties in communicating
without ATs (Korner et al., 2013; Beukelman et al.,
2000). Access to computer devices is also important
to give access to eHealth services for patients.
Although speech progression in ALS has been
studied for the purpose of monitoring
communication needs (Ball et al., 2002), writing
function (as upper limb motor progression) has been
underestimated as a variable for monitoring
communication abilities of ALS patients. ATs for
communication are commonly based on electronic
devices and typing tasks (considering the use of text-
to-speech technologies), either on a physical
keyboard or a virtual keyboard accessed via
touchscreen. Due to the neurodegenerative
characteristics, it is important to follow symptoms of
progression in upper limb motor control to identify
periods to adapt or introduce ATs, aiming at
augmenting users’ functionality in communication
(Beukelman et al., 2011; Bongioanni, 2012).
In this study, we aim to investigate the potential
of monitoring progression of upper limb motor
functionality needed to perform typing tasks on a
keyboard. For this purpose, we captured data from a
3D accelerometer placed on the index finger (finger
used to press the keys) and captured a 10-word
typing task. In this paper, we describe the
54
Londral A., Câmara M., Gamboa H., de Carvalho M., Pinto A. and Azevedo L..
A Method to Detect Keystrokes using Accelerometry to Quantify Typing Rate and Monitor Neurodegenerative Progression.
DOI: 10.5220/0004638800540059
In Proceedings of the International Congress on Neurotechnology, Electronics and Informatics (NEUROTECHNIX-2013), pages 54-59
ISBN: 978-989-8565-80-8
Copyright
c
2013 SCITEPRESS (Science and Technology Publications, Lda.)
methodology and the developed algorithm to detect
and quantify keystrokes events from accelerometer
signals captured during the experiments. As first
results, we present data from three ALS patients and
from three control subjects (with no diagnosed
neurological disease).
The rest of the paper is organized as follows:
Section 2 describes methodology used to acquire
data from accelerometer; Section 3 describes the
proposed algorithm for typing detection; in Sections
4 results from data analysis are presented; Section 5
and 6 present discussion and main conclusions on
the first results of this study.
2 METHODS
2.1 Participants
We present results from 3 patients with ALS and 3
healthy control subjects. ALS patients (2 women and
1 man) had a mean age of 53 years old (37, 59 and
63 years old). At baseline assessment, all
participants with ALS had clinical evaluation of
ALSFRS-r (Cedarbaum et al., 1999) speech
subscore less or equal to 2, though all were able to
use upper limbs to type on a keyboard. Patients had
no dementia. Healthy control subjects (2 women and
1 man) had mean age of 32 (23, 35 and 37) years
old.
2.2 Equipment
For accelerometry acquisition a BiopluxResearch
system (PLUX SA) was used. In our research
settings, we used the system with a 3-axial MEMS
accelerometer sensor (±3g measurement range).
Sensor was placed in exterior part of index finger of
the functional hand, as depicted in Fig.1. The three
axes were measured according to Fig.1 in directions:
anterior-posterior(X), distal(Y) and lateral(Z). Data
was sampled at 1KHz. Data was acquired via
Bluetooth to a laptop computer to be later processed
using Python tools. A second laptop was used by the
participants to perform the typing tasks.
Figure 1: Index finger for typing.
2.3 Procedure
Subjects were asked to type a 10-word sentence
using just the index finger. Accelerometer was
placed in the finger and data from the accelerometer
was saved. The same accelerometer sensor was
always used. Patients were evaluated in 3 sessions,
in 3 months intervals. Control subjects just
performed one trial in one session, as no progression
is expected for control participants. A camera was
also used to capture typing task, for results
validation.
Figure 2: Photo from an evaluation.
2.3.1 Outcome Measures
based on Accelerometer
For simplification, we used Y axis (distal
movements of index finger) to characterize typing
function, as this is the direction related to the
movement of pressing keys (as illustrated in Fig 1).
As outcome variables we wanted to have number of
keystrokes, typing rate, time between keystrokes,
time duration of each keystroke, amplitude of
acceleration signal (amplitude of acceleration of
finger movements in distal direction), magnitude of
acceleration signal (calculated as the Euclidean
vector for the 3-axis acceleration signal).
2.4 Algorithm for Typing Detection
We analysed accelerometer signals from patients
and control subjects. Key types from the 10-word
sentence were analysed. Fig.3 shows the plot from
the acceleration signal of one of the control subjects,
illustrating a set of keystrokes and one isolated
keystroke.
An algorithm was developed to extract outcome
variables from each typing task.
AMethodtoDetectKeystrokesusingAccelerometrytoQuantifyTypingRateandMonitorNeurodegenerativeProgression
55
Figure 3: Plots from keytyping from a control subject. (up)
Set of 13 keystrokes (down) One keystroke: first
acceleration pulses correspond to press action and the
second acceleration pulses correspond to release action.
2.5 Characterization of a Typing Action
For the setup suggested in this paper, a keystroke
(for simplicity, we consider signal from Y-Axis to
analyze movements in distal direction) is
characterized by two events of acceleration (Figure
3). The first event is related to pressing action and
the second event is related to releasing action.
2.5.1 Signal Processing for Detection
of Typing Actions
Proposed algorithm first removes DC component
from acquired data, then uses a moving average
algorithm to smooth the signal’s module (Figure 4).
From the processed signal, peaks are detected (from
a threshold value calculated as a factor of maximum
amplitude) as events of pressing and releasing keys
– each group of two near peaks corresponds to the
signal of a keystroke.
Due to erroneous movements (from video
analysis we could identify hand gestures performed
during typing task or touching a key with no
pressing action) two kinds of peaks were detected as
frequent false typing events:
- isolated event: isolated peaks or a peak close to
a pressing event (video analyses show that a single
peak may occur when user touches a key but doesn’t
press it – this is caused by a hesitation (commonly
caused by low experience in the use of a qwerty
keyboard);
- third event: we could observe in the acquired
signals frequent low amplitude acceleration impulses
prior to a typical keystroke signal. From the video
analysis we could conclude that these acceleration
impulses are due to slow typing (an evident delay
between touching the key and pressing it).
To guarantee a pair number of peaks
(press/release keys), previously described false
typing events were removed: signal was analysed
near each detected peak of acceleration. Isolated
events were eliminated and, in groups with more
than two close events, only the two with higher
amplitude values were kept.
Keystroke events detected with the proposed
algorithm are depicted in Fig 5.
Figure 4: (top left) raw signal; (top right) signal’s module; (bottom left) filtered signal using smooth algorithm; (bottom
right) bottom left signal and vertical lines representing peaks detection. All schematic representations are of 20s of samples
from one patient performing a typing task.
NEUROTECHNIX2013-InternationalCongressonNeurotechnology,ElectronicsandInformatics
56
Figure 5: Representation of keystrokes detected from
approximately 10s of accelerometer signal acquisition (a
patient performs 15 keystrokes, typing a 10-word
sentence).
3 RESULTS
All collected signals were processed with the
proposed algorithm. Features from typing task were
calculated and are represented in Table 1. We also
measured performance of typing task, presented in
second column of Table 1 as words per minute
(wpm), using visual video analysis.
We applied the algorithm to the signals of
control subjects and patients (different users and
three different periods during disease progression).
As can be observed in Table 1, in spite all
subjects were asked to do the same typing task, the
number of keystrokes performed in each typing task
(#ks) is not always the same. This is due to
variability among subjects experience. For example,
some of the patients forgot to add spaces between
two different words, some added more than one
space or used the backspace to correct a mistake.
One of the patients (patient P1, in Table 1) had no
experience in using computer devices, the reason
why we reduced the sentence to 5 words (half of the
sentence). In fact, as it is not the aim of this study to
evaluate the written text, participants had no specific
constraints (related to text) when performing the
typing task.
From visual observation of accelerometer plots, a
validation of the algorithm was performed, based on
a manual adjustment of the algorithm variables (i.e.
threshold value for peak detection and time window
to search for false events). Number of keystrokes
detected by the algorithm was confirmed with the
number counted in video observation. Results are
presented in Table 1. We calculated a Pearson
correlation of 0.98484 between words per minute
(obtained from video analysis) and keyrate (result of
proposed algorithm from accelerometer analysis) for
the set of all analyzed data. These results validate
the used methodology to performance evaluation.
4 DISCUSSION
Finding methodologies that can early predict
progression allows a faster and customized response
of interventions and care. We hypothesized there is a
relation between user performance on typing tasks
(measured as communication speed - wpm) and
respective kinematic analysis of typing and that this
can provide a more sensitive tool to detect
progression in upperlimbs motor function. A
methodology to collect data from a 3-axis
accelerometer placed on the index finger was
developed for typing task.
A set of data captured from healthy subjects and
ALS patients (in different time periods) was
analyzed. Presented results show that keystrokes
detected from the developed algorithm had high
correlation with performance measured by video
analysis. Although, high accuracy was due to fine
adjustment of two parameters, which had to be
manually adjusted for each data set. It was not
possible to establish a peak threshold common to all
studied signals – it had to be manually adjusted
within different users or along different samples
from the same user. Also, using the suggested
algorithm, it was difficult to distinguish very low
amplitude peak from noise or involuntary movement
– window size to remove false events had to be
manually adjusted for each sample.
From a preliminary analysis of the results
presented in this paper, we could observe that, for
ALS patients, in spite performance values in
different periods of evaluation are variable (we can’t
always find evident decrease in time), the maximum
amplitude of y-axis signal (V
Y
máx, Table 1) and the
mean value of the amplitude of magnitude signal
(last column in Table 1) always decrease along the
different time periods of evaluation (3 months
interval, approximately). Although careful analysis
is part of future work for the presented study, these
preliminary results suggest a new surrogate marker
of typing function deterioration, potentially more
accurate than simple typing performance
observation.
5 CONCLUSIONS
In this paper we present an algorithm for studying
AMethodtoDetectKeystrokesusingAccelerometrytoQuantifyTypingRateandMonitorNeurodegenerativeProgression
57
Table 1: Results from the analysis of acquired data with the algorithm proposed for typing detection. Each row describes
the results from control subjects (C
i
) and patients (P
i
) in the different evaluation times (T
i
).
#ks – number of keystrokes; µ tbt_key – average of time interval between keystrokes; µ tkey – average time duration of
keystrokes; VYmáx – maximum amplitude of Y-axis of accelerometer signal; Magn. Máx – maximum amplitude of
magnitude of accelerometer signal; Magn.Mean – mean magnitude of the accelerometer signal).
Particip.
wpm
(video)
keyrate # ks
total
time
(s)
µ tbt_key µ tkey
V
Y
máx
(mV)
Magn.
Máx
(mV)
Magn.
Mean
µV
C1 10 43 43 60 331.8 75 623.79 220.12 60.99
C2 10 44 44 60 852.1 244 253.44 151.91 32.65
C3 10 44 44 60 1011.4 144.5 299.53 192.05 31.18
P1.T0 3.04 13.17 18 82 3952.2 255.1 623.43 149.26 40.68
P1.T1 3.1 11.46 17 89 4819.4 186.6 555.29 314.36 33.6
P1.T2 3.87 12.00 13 65 4129 241.5 412.22 195 31.44
P2.T0 16.66 67.89 43 38 527.6 168.1 624.19 147.34 38.34
P2.T1 16.61 62.67 47 45 669.9 188.5 475.47 225.08 36.62
P2.T2 17.3 61.82 44 42.7 666.1 252.5 427.32 129.08 33.72
P3.T0 7.69 28.53 39 82 1282.3 712.8 495.77 212.19 42.64
P3.T1 9.38 33.75 45 80 736.3 847.1 480.26 177.54 32.35
typing performance through accelerometry.
A 3-axis accelerometer was placed in the index
finger of 6 participants (3 with progressive
neuromuscular disease and 3 healthy participants).
Signal processing of the accelerometer signals
showed high correlation between independent
measures of performance: words per minute (from
video analysis) and keystrokes per minute (from
accelerometer).
Presented algorithm should be improved to
automatically adjust all the parameters for different
users and different stages of progressive disease. As
future work, a detailed analysis of other parameters
of accelerometry, independent from performance
measures, should be done.
REFERENCES
Ball, L. J., Beukelman, D. R. and Pattee, G. L., 2002.
Timing of speech deterioration in people with
amyotrophic lateral sclerosis, Journal of Medical
Speech-Language Pathology, 10(4), 231–235.
Beukelman, D., Fager, S. and Nordness, A., 2011.
Communication Support for People with ALS. In
Neurology Research International, Article ID 714693.
Beukelman, D. R., Yorkston, K. M., Reichle, J., 2000.
Augmentative and Alternative Communication for
Adults with Acquired Neurologic Disorders. Brookes
H. Paul Publishing, Baltimore.
Bonato, P., 2003. Wearable sensors/systems and their
impact on biomedical engineering. In IEEE
Engineering in Medicine and Biology Magazine,
22(3), 18–20.
Bongioanni, P., 2012. Communication Impairment in ALS
Patients: Assessment and Treatment. In Maurer, M.
(Ed.) Amyotrophic Lateral Sclerosis. Available from:
http://www.intechopen.com/books/amyotrophic-
lateral-sclerosis.
Bustamante, P., Solas, G. and Grandez, K., 2011.
Neurodegenerative Disease Monitoring Using a
Portable Wireless Sensor Device. In Chang, R. (Ed)
Neurodegenerative Diseases - Processes, Prevention,
Protection and Monitoring, InTech Publisher.
Available from: http://www.intechopen.com/books/
neurodegenerative-diseases-processes-prevention-
protection-and-monitoring.
Cedarbaum, J. M., Stambler, N., Malta, E., Fuller, C., Hilt,
D., Thurmond, B., Nakanishi. A., 1999. The ALSFRS-
R: a revised ALS functional rating scale that
incorporates assessments of respiratory function.
BDNF ALS Study Group (Phase III). In Journal of the
Neurological Sciences,169(1-2),13-21.
Godfrey, A., Conway, R., Meagher, D. and ÓLaighin, G.,
2008. Direct measurement of human movement by
accelerometry, In Medical Engineering & Physics, 30,
1364–1386.
Korner, S., Siniawski, M., Kollowe, K., Rath, K.J.,
Krampfl, K., Zapf, A., Dengler, R., Petri, S., 2013.
Speech therapy and communication device: Impact on
NEUROTECHNIX2013-InternationalCongressonNeurotechnology,ElectronicsandInformatics
58
quality of life and mood in patients with amyotrophic
lateral sclerosis. In Amyotrophic Lateral Sclerosis and
Frontotemporal Degeneration, 14, 20–25.
Shany, T., Redmond, S. J., Narayanan, M. R. and Lovell,
N. H., 2012. Sensors based wearable systems for
monitoring of human movement and falls. In IEEE
Sensors Journal, 12(3), 658–670.
AMethodtoDetectKeystrokesusingAccelerometrytoQuantifyTypingRateandMonitorNeurodegenerativeProgression
59