Inferring Smartphone Users’ Handwritten Patterns by using Motion

Sensors

Wei-Han Lee

, Jorge Ortiz

, Bongjun Ko

and Ruby Lee

Princeton University, U.S.A.

IBM Research, U.S.A.

Keywords:

Smartphone Sensors, Handwritten Pattern, Dynamic Timing Warping, Majority Voting.

Abstract:

Mobile devices including smartphones and wearable devices are increasingly gaining popularity as platforms

for collecting and sharing sensor data, such as the accelerometer, gyroscope, and rotation sensor. These

sensors are used to improve the convenience of smartphone users, e.g., supporting the mobile UI motion-

based commands. Although these motion sensors do not require users’ permissions, they still bring potential

risks of leaking users’ private information reﬂected by the changes of sensor readings. In this paper, we

investigate the feasibility of inferring a user’s handwritten pattern on a smartphone touchscreen by using the

embedded motion sensors. Speciﬁcally, our inference attack is composed of two key steps where we 1) ﬁrst

exploit the dynamic time warping (DTW) technique to differentiate any pair of time-series sensor recordings

corresponding to different handwritten patterns; and 2) develop a novel sensor fusion mechanism to integrate

information contained in multiple motion sensors by exploiting the majority voting strategy. Through extensive

experiments using real-world data sets, we demonstrate the effectiveness of our proposed attack which can

achieve 91.4% accuracy for inferring smartphone users’ handwritten patterns.

1 INTRODUCTION

The ubiquity of mobile devices such as smart-

phones and wearable devices together with their ever-

growing computing, networking, and sensing powers

have been increasingly changing the landscape of our

daily lives. These devices are often equipped with

various embedded sensors including the Global Po-

sitioning System (GPS) sensor, camera, microphone,

the environmental sensor (e.g., the ambient light sen-

sor and the barometer), and the motion sensors (e.g.,

the accelerometer, gyroscope, rotation sensor). These

sensors have been effectively utilized to improve the

convenience of smartphone users. For instance, the

GPS can be used for positioning and the motion sen-

sors can be leveraged for mobile gaming.

Within these built-in sensors, some require users’

permission to obtain access (such as the GPS, cam-

era, microphone) because these sensors are explic-

itly utilized for collecting users’ private information

(such as location, image and speech). In comparison,

motion sensors such as the accelerometer, gyroscope

and rotation sensor do not require users’ permissions,

probably due to the assumption that data collected by

these sensors is not sensitive. Motion sensors can pro-

vide recordings of acceleration, rotation and direction

with high precision and accuracy, which can enable

applications that provide convenient functions for the

users. For example, a game can utilize the gravity sen-

sor data of a smartphone to infer users’ different ges-

tures, such as tilt, shake, rotation, or swing (?). All the

measurements of the accelerometer, gyroscope, and

rotation sensor in smartphones running on the iOS

system and the Android system can be accessed with-

out requiring any user permission (Xu et al., 2012).

However, even motion sensors that do not require

explicit permissions are still vulnerable to privacy at-

tacks since their measurements are closely correlated

with users’ sensitive behavior patterns. With the in-

creasing development of motion sensors in smart-

phones, the risks of leaking a user’s sensitive in-

formation through an installed third-party applica-

tion exploring motion sensors have raised more pri-

vacy and security concerns. For instance, Marquardt

et al. (Marquardt et al., 2011) utilize the vibra-

tions detected by smartphone accelerometer to infer

the user’ inputs to a nearby keyboard. Michalevsky

et al. (Michalevsky et al., 2014) show that exist-

ing gyroscopes on smartphones are sufﬁciently sen-

sitive to measure acoustic signals in the vicinity of

Lee, W-H., Ortiz, J., Ko, B. and Lee, R.

Inferring Smartphone Users’ Handwritten Patterns by using Motion Sensors.

DOI: 10.5220/0006650301390148

In Proceedings of the 4th International Conference on Information Systems Security and Privacy (ICISSP 2018), pages 139-148

ISBN: 978-989-758-282-0

139

the smartphone to identify the speaker’ private infor-

mation and even parse speech. These security and

privacy breaches demonstrate that the motion sensors

are possible side channels for attackers that aim to in-

fer users’ sensitive behavior patterns (Xu et al., 2012;

Marquardt et al., 2011; Michalevsky et al., 2014).

In this work, we aim to infer users’ handwritten

patterns by exploiting the motion sensors embedded

in the smartphones, which has not been investigated

in the literature to the best of our knowledge. The

fundamental intuition of our attack is that the users’

handwritten behaviors and the changes of the smart-

phone motions are closely correlated with each other.

During a writing event, the force from the users’ ﬁn-

ger on the touchscreen would cause changes of the

motion sensor measurements, which would follow

certain patterns corresponding to different contexts

of handwritten behaviors. By utilizing the changes

of motion sensor readings, the attacker can therefore

infer users’ handwritten patterns on the smartphone

touchscreen.

More speciﬁcally, our proposed attack is com-

posed of two key steps where we 1) ﬁrst exploit the

dynamic time warping (DTW) (Berndt and Clifford,

1994) technique to evaluate the similarity between

any pair of time-series sensor recordings, in order

to differentiate users’ various handwritten patterns.

This DTW-based similarity evaluation technique can

be utilized for constructing template sensor signals

corresponding to different handwritten patterns (un-

der the training mode), as well as inferring the incom-

ing handwritten event by matching the observed sen-

sor measurement with the constructed template sen-

sor signals (under the testing mode); and 2) develop

a novel sensor fusion mechanism to integrate infor-

mation contained in multiple motion sensors by ex-

ploring the majority voting strategy (Lam and Suen,

1997). In summary, our key contributions include:

• We investigate the unique patterns of users’ hand-

written events in terms of measurement changes

of motion sensors embedded in the smartphones.

Our observations raise the awareness of motion

as a signiﬁcant channel that may leak smartphone

users’ handwritten patterns.

• We exploit the dynamic time warping technique

to measure the distance between any pair of time-

series sensor recordings, in order to distinguish

users’ handwritten patterns on the smartphone

touchscreen.

• We further propose a novel sensor fusion mech-

anism by leveraging the majority voting strategy

to integrate information recorded by multiple sen-

sors, in order to enhance the overall accuracy of

inferring users’ handwritten patterns.

• We present the design of our attack that uti-

lizes observed motion sensor readings to stealthily

record the user’s inputs on the touchscreen.

Extensive experiments on real-world data sets

demonstrate the effectiveness of our attack which

can infer the contexts of users’ secret inputs with

up to 91.4% accuracy.

2 BACKGROUND AND RELATED

WORK

In this section, we will ﬁrst describe the motion sen-

sors embedded in the smartphones and then discuss

existing attacks that aim to infer users’ private in-

formation by exploiting measurements recorded by

smartphone sensors.

2.1 Motion Sensors

Since motion sensors such as the accelerometer, gyro-

scope and rotation sensor are integrated into a smart-

phone, they bring the opportunity to assist navigation,

location, etc., with knowledge about the motion of the

smartphone users.

Accelerometer: the accelerometer measures the ac-

celeration in m/s

, which is the rate of change of ve-

locity with time, of a smartphone along three axes:

x-axis (lateral or left-right), y-axis (longitudinal or

forward-backward), and z-axis (vertical or up-down)

(Xu et al., 2012).

Linear Accelerometer: the linear accelerometer

measures the acceleration effect of the smartphone

movement, excluding the effect of the Earth’s grav-

ity on the device. It is typically derived from the ac-

celerometer, where other sensors (e.g. the gyroscope)

can help to remove the linear acceleration from the

data. Linear acceleration units are shown in m/s

sim-

ilar to the accelerometer.

Gyroscope: the gyroscope measures the rate of rota-

tion in rad/s around a device’s x, y, and z axis. The

gyroscope is used to maintain and control the posi-

tion, level or orientation of the smartphone based on

the principle of angular momentum.

Rotation Sensor: the rotation (orientation) sensor

measures the change of direction of the smartphone

along three dimensions: x-axis (Azimuth), y-axis

(Pitch), and z-axis (Roll) (Xu et al., 2012).

2.2 Inferences Derived from

Smartphone Sensor Data

Mobile devices which are often equipped with sen-

sors such as the accelerometer, gyroscope, rotation

ICISSP 2018 - 4th International Conference on Information Systems Security and Privacy

140

sensor, camera, microphone, GPS and so on, are be-

ing used by mobile sensing systems to make sophis-

ticated inferences about users. These inferences have

enabled an entire ecosystem of context-aware appli-

cations such as trafﬁc and environmental monitor-

ing (Azizyan et al., 2009; Templeman et al., 2013;

Mohan et al., 2008; Tung and Shin, 2015), behavior-

based user authentication (Lee and Lee, 2017; Frank

et al., 2013; Mare et al., 2014; Zhu et al., 2013; Lee

et al., 2017; Riva et al., 2012; Conti et al., 2011), ac-

tivity mode detection (Reddy et al., 2010; Bao and

Intille, 2004; Luxton et al., 2011), and speech transla-

tion (Michalevsky et al., 2014; Lei et al., 2013).

While the smartphone sensory data has enabled

context-aware applications, the same data can also

be used by an adversary to make inferences about

the private information of the users. Therefore,

there exist fundamentally conﬂicting requirements

between protecting privacy of users’ sensitive infor-

mation recorded by smartphone sensors and preserv-

ing utility of the same data for authorized context-

aware applications. These private inferences include

the identiﬁcation of emotional state (Chang et al.,

2011; Rachuri et al., 2010), speaker identity recog-

nition (Nirjon et al., 2013; Liu et al., 2012), location

tracking (Han et al., 2012; Brouwers and Woehrle,

2011; Nirjon et al., 2013; Kim et al., 2010), on-

screen taps recognition (Miluzzo et al., 2012), onset

of stress (Lu et al., 2012; Chang et al., 2011) and

keystroke detection (Miluzzo et al., 2012; Xu et al.,

2012; Liu et al., 2015; Marquardt et al., 2011; Owusu

et al., 2012; Cai and Chen, 2011). Furthermore, many

applications have access to data recorded by the mo-

tion sensors that do not require users’ permissions,

combinations of which can be maliciously used to

predict more private information than what the appli-

cations advertise.

3 ATTACK OVERVIEW

3.1 Attack Goals

The objective of our attack is to infer the users’ hand-

written patterns by exploiting the motion sensors em-

bedded in the smartphones. Since these motion sen-

sors are usually considered as collecting insensitive

information, our attack does not require any user per-

mission to access the recordings of motion sensors

such as the accelerometer, linear accelerometer, gy-

roscope and rotation sensor.

3.2 Attack Workﬂow

We explain the workﬂow of our attack which works

under the training mode and the testing mode as fol-

low:

In the training mode, when the user is interact-

ing with the smartphone, we can record the handwrit-

ten characters entered on the touchscreen, and corre-

late these ground-truth information with the measure-

ments of motion sensors collected during the hand-

written events to generate the user’s interaction pat-

terns/templates.

In the testing mode, our attack keeps monitoring

the measurements of motion sensors. When the user

is performing sensitive inputs on the touchscreen, the

acquired interaction patterns/templates in the training

mode can be leveraged to infer the user’s handwrit-

ten characters based on the measurements of motion

sensors.

4 EXPERIMENTAL SETUP

Android Application Implementation: We develop

an Android application to implement our privacy at-

tack of inferring smartphone users’ handwritten pat-

terns. Speciﬁcally, we collect recordings from all

the motion sensors including the accelerometer, lin-

ear accelerometer, gyroscope, and rotation sensor in

a Google Nexus 5 (with 2.3GHz, Krait 400 proces-

sor, 16GB internal storage and 2GB RAM on Android

7.0 operating system), corresponding to the scenarios

where the user entered 26 characters from A to Z on

the touchscreen. The sampling frequency of our ap-

plication is set to 50 Hz.

Sensor Data Collection: In our experiments, we

collect sensor recordings of 10 users’ handwritten

events corresponding to the 26 characters, and we re-

peated this process for 10 times. Therefore, we collect

10 × 26 × 10 = 2600 time-series sensor recordings in

our data set. Furthermore, we use 10-fold cross vali-

dation in our experiments to generate the training and

testing data, i.e., 9/10 of our collected data is used as

training data and the remaining 1/10 is used as testing

data. We repeated this process for 1000 iterations and

reported the averaged results.

5 PROPOSED APPROACH

Our proposed inference attack is composed of two

key techniques. First, we propose to exploit the dy-

namic time warping (DTW) algorithm (Berndt and

Clifford, 1994) to quantify similarities between two

Inferring Smartphone Users’ Handwritten Patterns by using Motion Sensors

141

time-series sensor recordings, in order to distinguish

users’ handwritten patterns (detailed process will be

discussed in Section 5.1). More speciﬁcally, under

the training mode, we construct the template sensor

recordings using DTW technique, by selecting the

most representative sensor measurement correspond-

ing to each handwritten pattern of the user. Under

the testing mode, we evaluate the similarity between

the incoming sensor signal and all the template sen-

sor recordings by using DTW algorithm, from which

we identify the closest template sensor recording and

label the incoming sensor signal accordingly for each

sensor dimension. Second, we develop a novel sen-

sor fusion mechanism to generate the ﬁnal inference

result which integrates information contained in mul-

tiple motion sensors by leveraging the majority voting

strategy (Lam and Suen, 1997) (as will be discussed

in Section 5.2).

5.1 Evaluating Similarity of Sensor

Recordings by using DTW

DTW is a well-known technique (Berndt and

Clifford, 1994) to ﬁnd the optimal alignment

between two given (time-dependent) sequences

x := (x

,...,x

) of length N ∈ N and y

y :=

,...,y

) of length M ∈ N under certain restric-

tions. It has been successfully applied to compare

different speech patterns in automatic speech recog-

nition and other applications in the data mining com-

munity. While there is a surfeit of possible distance

measures for time-series data, empirical evidence has

shown that DTW is exceptionally difﬁcult to beat.

Ding et al. in (Ding et al., 2008) tested the most

cited distance measures on 47 different data sets, and

no method consistently outperforms DTW. Therefore,

we aim to exploit the DTW technique to carefully

measure the distance between any pair of time-series

sensor recordings which may vary in time or speed.

DTW calculates the distance of two sequences us-

ing dynamic programming (Bertsekas, 1995), where

the sequences are warped in a nonlinear fashion to

match each other. It constructs an N-by-M matrix,

where the (i, j)-th element is the minimum distance

(called local distance) between the two sequences

that end at points x

and y

respectively. An (N,M)-

warping path p = (p

, p

,··· , p

) is a contiguous set

of matrix elements which deﬁnes an alignment be-

tween two sequences x

x and y

y by aligning the element

of x

x to the element y

of y

y. The boundary condi-

tion enforces that the ﬁrst elements of x

x and y

y as well

as the last elements of x

x and y

y are aligned to each

other. The total distance d

x,y

y) of a warping path

p between x

x and y

y with respect to the local distance

measure d is deﬁned as

x,y

y) =

∑

l=1

d(x

) (1)

Therefore, the DTW distance for two time-series

data can be computed as

DTW (x

x,y

y) = mind

x,y

y) (2)

Constructing Template Sensor Recording:

Under the training mode of our attack, we

aim to construct the template sensor record-

ing corresponding to each character and sen-

sor dimension, i.e., t

character, sensor, axis

, where

character ∈ C

C = {A,...Z}, sensor ∈ S

S =

{accelerometer, linear accelerometer, gyroscope ,

rotation sensor} and axis ∈ A

A = {x, y, z}. Our

objective is to identify the most representative

sensor signal that is the closest to all the senor

recordings of the same character. Speciﬁcally,

for each character ∈ C

C, we compute the DTW

distance between any pair of sensor recordings in

character, sensor, axis

= {r

character, sensor, axis

} and select

as template the one that achieves the smallest DTW

distance, i.e.,

character, sensor, axis

= argmin

∈R

character, sensor, axis

∑

∈R

character, sensor, axis

DTW (r

)

(3)

Inferring Handwritten Pattern Corresponding to

Each Sensor Dimension: Under the testing mode

of our attack, we aim to infer the handwritten char-

acter corresponding to an incoming sensor recording

D = {d

sensor, axis

}

sensor∈S

S,axis∈A

. Our inference attack

consists of two steps: 1) identify the input character

from data recorded by each sensor dimension; and 2)

infer the input character by integrating the informa-

tion contained in all the sensors. For the ﬁrst step, we

can calculate the DTW distance Dist

character, sensor, axis

between the input data d

sensor, axis

and each template

sensor recording t

character, sensor, axis

, from which we

can identify the handwritten character corresponding

to each sensor dimension as

In f er(d

sensor, axis

)

=argmin

character

Dist

character, sensor, axis

=argmin

character

DTW (d

sensor, axis

character, sensor, axis

)

(4)

The second step of integrating the information

contained in multiple sensors is described as follows.

ICISSP 2018 - 4th International Conference on Information Systems Security and Privacy

142

5.2 Majority Voting based Sensor

Fusion Mechanism

In order to achieve enhanced inference performance,

we aim to exploit the majority voting strategy to inte-

grate information recorded by all the sensors. Ma-

jority voting is previously applied in the classiﬁca-

tion domain (Lam and Suen, 1997), where it repre-

sents a mapping function that maps multiple classi-

ﬁers’ decisions into a single decision, i.e., it maps

Class × Class × ··· × Class to Class, where Class =

{A ,B,...Z} in our setting. The beneﬁt of using ma-

jority voting is that it can reduce the noise and bias

caused by a single classiﬁer. Therefore, the deci-

sion made by majority voting is more robust and re-

liable. Speciﬁcally, assuming that we have m clas-

siﬁers Class

,Class

,...,Class

, the majority voting

function Ma jorityVote(·) can be described as

Class = Ma jorityVote(Class

,Class

,··· ,Class

)

(5)

where Ma jorityVote(Class

,Class

,··· ,Class

) re-

turns the most frequent result appearing within

Class

,Class

,··· ,Class

With the knowledge of the inferred char-

acter corresponding to each sensor dimension

In f er(d

sensor, axis

) in Eq. 4, we construct the set of

inferred characters for all the sensors as

L(D

D) = {In f er(d

sensor, axis

)}

sensor∈S

S, axis∈A

(6)

Then, we can infer the handwritten character cor-

responding to the incoming sensor recordings D

D =

sensor, axis

}

sensor∈S

S, axis∈A

as follows:

In f er(D

=Ma jorityVote(L

L(D

D))

=T he Most Frequent Character in L

L(D

(7)

Based on our analysis above, we therefore sum-

marize our attack in Algorithm 1.

6 EVALUATION

6.1 Evaluation Metrics

In our experiments, we quantify the performance of

our attack by using the metric of accuracy, which is

computed as the ratio of correctly-inferred characters.

More speciﬁcally, we can compute the accuracy of

Algorithm 1: Our Attack of Inferring Users’ Handwritten

Patterns by Using Motion Sensors.

Input : The input sensor signal D

D =

sensor, axis

}

sensor∈S

S, axis∈A

and

the template sensor recordings

character, sensor, axis

}

character∈C

C, sensor∈S

S, axis∈A

where C

C = {A,...Z}, S

S = {accelerometer,

linear accelerometer, gyroscope,rotation

sensor} and A

A = {x, y, z};

Output: The detected character corresponding to the

input sensor data In f er(D

D);

Construct the set of inferred characters for all the sen-

sors L

L(D

D) as an empty set;

for sensor in S

S do

for axis in A

A do

for character in C

C do

According to Eq. 2, calculate the DTW

distance Dist

character, sensor, axis

between

sensor, axis

and t

character, sensor, axis

;

end

Label the identiﬁed character for each sensor

dimension as

In f er(d

sensor, axis

)

= argmin

character

Dist

character, sensor, axis

;

Update L

L(D

D) as

L(D

= [L

L(D

D),In f er(d

sensor, axis

)];

end

Label the detected character corresponding to D

D as

In f er(D

D) = Ma jorityVote(L

L(D

D))

=T he Most Frequent Character in L

L(D

D);

return In f er(D

D);

our attack by using measurement of each sensor di-

mension as

Accuracy

sensor,axis

∑

sensor, axis

I(In f er(d

sensor, axis

) = Char(D

D))

∑

sensor, axis

(8)

where I(event) is the indicator function and

I(event) = 1 if event holds otherwise I(event) = 0.

Char(D

D) represents the ground-truth character that is

entered on the touchscreen corresponding to the in-

put sensor signal D

D. In f er(d

sensor, axis

) is the inferred

Inferring Smartphone Users’ Handwritten Patterns by using Motion Sensors

143

0 0.5 1 1.5 2

Time (Second)

-0.8

-0.6

-0.4

-0.2

0.2

0.4

0.6

Amplitude

(a) x axis of Accelerometer

0 0.5 1 1.5 2

Time (Second)

7.4

7.6

7.8

8.2

8.4

Amplitude

(b) y axis of Accelerometer

0 0.5 1 1.5 2

Time (Second)

4.5

5.5

6.5

Amplitude

0 0.5 1 1.5 2

Time (Second)

-1

-0.5

0.5

Amplitude

(d) x axis of Linear Accelerometer

0 0.5 1 1.5 2

Time (Second)

-0.6

-0.4

-0.2

0.2

0.4

Amplitude

(e) y axis of Linear Accelerometer

0 0.5 1 1.5 2

Time (Second)

-1.5

-1

-0.5

0.5

Amplitude

(f) z axis of Linear Accelerometer

0 0.5 1 1.5 2

Time (Second)

-0.4

-0.3

-0.2

-0.1

0.1

0.2

Amplitude

(g) x axis of Gyroscope

0 0.5 1 1.5 2

Time (Second)

-0.3

-0.2

-0.1

0.1

0.2

0.3

Amplitude

(h) y axis of Gyroscope

0 0.5 1 1.5 2

Time (Second)

-0.3

-0.2

-0.1

0.1

0.2

Amplitude

(i) z axis of Gyroscope

0 0.5 1 1.5 2

Time (Second)

0.18

0.185

0.19

0.195

0.2

Amplitude

(j) x axis of Rotation Sensor

0 0.5 1 1.5 2

Time (Second)

0.42

0.422

0.424

0.426

0.428

0.43

0.432

0.434

Amplitude

(k) y axis of Rotation Sensor

0 0.5 1 1.5 2

Time (Second)

0.796

0.798

0.8

0.802

0.804

0.806

Amplitude

(l) z axis of Rotation Sensor

Figure 1: The visualization of handwritten signals extracted from the accelerometer, linear accelerometer, gyroscope and

rotation sensor under the three different dimensions. We randomly select two handwritten signals from the same character A

(red and blue lines) and a handwritten signal from another character B (green lines). We observe that the distance between

two handwritten signals corresponding to the same character is smaller than that from a different character, which lays the

foundation of our attack.

character by using data collected by each sensor di-

mension as shown in Eq. 4.

After applying the majority voting based sensor

fusion mechanism in Section 5.2, the overall accuracy

of our attack can be computed as

Accuracy

∑

sensor, axis

sensor∈S

axis∈A

I(In f er(d

sensor, axis

) = Char(D

D))

∑

sensor, axis

sensor∈S

axis∈A

(9)

ICISSP 2018 - 4th International Conference on Information Systems Security and Privacy

144