Integrating Knowledge Artifacts and Inertial Measurement Unit Sensors

for Decision Support

Stefano Pinardi, Fabio Sartori and Riccardo Melen

Department of Computer Science, Systems and Communication, University of Milano-Bicocca,

viale Sarca 336, Milan, Italy

Keywords:

Inertial Measurement Unit, Android, Wearable Devices, Recommendation Systems, Knowledge Artifacts.

Abstract:

Modern wearable devices provide new opportunities for the development of knowledge artifacts and decision

support systems. In this paper we present a recent development of KAFKA, a knowledge engineering method-

ology based on knowledge artifact notion, that make it able to manage real-time data detected and analyzed

by means of Inertial Measurement Units sensors, mounted on Android wearables. This improvement makes

KAFKA suitable to deal with many domains where real-time data are necessary, in particular the health-care

and rehabilitation ones.

1 INTRODUCTION

Today, Wearable devices and Smartphones offer a

great variety of sensors on board: ambient and

body-temperature sensors, heart beat rate sensors,

light presence sensors, barometric pressure sensors,

and also Inertial Measurement Units (IMUs) sensors.

IMUs have shown to be of particular importance in

the ﬁeld of situations assessment to determine body

movements and type of executed actions (Bao, 2003;

Avci et al., 2010). However, a lot must be still done

in real case scenarios when the context may require

either real-time abilities of analysis, and algorithms

at the state of the art, or a neat and akin knowledge

of the problems speciﬁc to the application domain.

In this case the deﬁnition of information quality is

a parameter not independent from the contingency

of the situations: the quality of the data, the desired

parameters, the “degree of freedom” given by actu-

ally used technology, and the speciﬁcities of the ap-

plication domain, are all components that inﬂuence

the ﬁnal result. In this paper, we propose the inte-

gration of IMUs and Knowledge Artifacts for the de-

velopment of time-evolving expert systems (Sartori

and Melen, 2015), in order to analyze the data stream

collected by wearables and transform it in valuable

suggestions for the user. The result is a unique con-

ceptual and computational framework that can be ex-

ploited in many domains. In particular, we are inter-

ested in developing recommendation systems for the

promotion of physical activity in people risking car-

diovascular diseases. In this context, the hybrid ap-

proach proposed in this paper should allow to moni-

tor the accomplishment of a given activity, evaluating

the quality of it and proposing a personalized training

program for the future. The rest of the paper is or-

ganized as follows: section 2 brieﬂy review literature

about knowledge artifacts and IMUs, according to pa-

per scopes. Sections 3 and 4 introduce the motivation

of the presented work, that is improving the perfor-

mance of the KAFKA KA-User component. Section

5 and 6 explain how this point has been from both the

conceptual and computational point of view. Section

7 presents the typical application scenario of the im-

proved KAFKA framework. Finally, conclusions and

future works are brieﬂy pointed out in section 8.

2 RELATED WORK

Holsapple and Joshi (2001) described knowledge ar-

tifacts as objects that convey or hold usable repre-

sentations of knowledge. Accordingly,Salazar-Torres

et al. (2008) argued that KAs are artifacts which rep-

resent executable encodings of knowledge, which can

be suitably embodied as computer programs. Cab-

itza and Locoro (2014) grouped KA experiences into

ﬁve conceptual clusters, where different KAs are used

with different scopes, on the basis of objectivity and

situativity dimensions: Artiﬁcial Intelligence (AI-

KAs from now on), Knowledge Management, Com-

puter Supported Cooperative Learning, Information

Pinardi, S., Sartori, F. and Melen, R.

Integrating Knowledge Artifacts and Inertial Measurement Unit Sensors for Decision Support.

DOI: 10.5220/0006091203070313

In Proceedings of the 8th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2016) - Volume 3: KMIS, pages 307-313

ISBN: 978-989-758-203-5

307

Systems and Computer Supported Cooperative Work.

Here, we are mainly interested in AI-KAs, devoted to

design and implement decision support systems, ex-

pert systems and ontologies. In that classiﬁcation, AI-

KAs where lacking from the situativity point of view,

meaning that they are less able to adapt to the context

and situation at hand. This means that they cannot

be applied in domains characterized by huge amounts

of real-time data, where other kinds of KAs have

been successfully applied in the past, like Health-Care

(Cabitza et al., 2015). In the following, we propose

the integration of IMUs and AI-KAs to overcome this

limitation. IMUs can be used either i) to determine

biomechanical parameters of the body movements or

ii) to detect information about movements, with the

aim to devise the state of the person. Using new re-

cent approaches IMUs can also be used iii) to un-

derstand more complex aspects of the body move-

ments, like the motion grammar (Pinardi and Bisiani,

2010; Mileo et al., 2009). In general, type and quality

of movements are recognized by means of classiﬁca-

tion and pattern recognition methods, like Clustering,

Bayesian rules, Hidden Markov Models (Bao, 2003;

Avci et al., 2010), and even using reasoning rules

(Bisiani et al., 2013; Mileo et al., 2010). In the follow-

ing, we focus on the ﬁrst and second aspects of body

movement recognition, to detect the pre-determinants

useful to understand the situation. In particular, we

will describe an n-tier application which acquires bio-

mechanical parameters, transforms them and graphi-

cally represent them at run-time. These data are then

analyzed and, ﬁnally,stored for future use by AI-KAs,

to develop decision support systems in the Health-

Care application domain, where the suggestions are

tailored on the user behavior.

3 MOTIVATION

Figure 1 shows the architecture of KAFKA, namely

Knowledge Acquisition Framework based on Knowl-

edge Artifacts. It is developed according to the client-

server paradigm, where the client, i.e. the KA-User, is

responsible for detecting data and observations about

the environmentand the server, i.e. the KA-Developer

is responsible for the interpretation of these data on

the basis of a three-tier knowledge representation

model (Sartori and Melen, 2015). Till now, greater ef-

forts have been spent on the design of KA-Developer,

being able to automatically create a rule-based sys-

tem from scratch in case of signiﬁcant changes in

the knowledge domain (Melen et al., 2015). These

changes are mainly due to new observations made

by the KA-User: in the current implementation of

KAFKA, these changes are manually detected by

the user on the ﬁeld and sent to the KA-Developer

through a JSON interface. In this paper we present

a rethinking of the client side of KAFKA, where the

data observations come from wearables. Doing so,

the KA-User is provided with “intelligence” too, dif-

ferent from the past. Possible values of system in-

puts are the result of wearable sensors querying: the

data are then interpreted and stored to be exploited

by the KA-Developer in the decision making process.

In this way, KAFKA is able to manage new kinds

of data, related to the real-time status of the person

equipped with wearable devices. These data can then

be transformed into useful inputs for recommendation

systems, depending on the application domain.

Figure 1: The multi–layered model of Knowledge Artifact

in KAFKA.

4 KA-User IMPROVEMENT: THE

IMU ROLE

As brieﬂy introduced above, the aim of this paper

is exploiting wearable devices to collect data about

a given application domain, increasing signiﬁcantly

the role of KA-User in the development of recom-

mendation systems through KAFKA. The challenge

is demonstrating how a KA-User provided with sim-

ple and relatively low-cost technology can become a

crucial source of information in potentially complex

decision making processes. To this aim, it is impor-

tant to point out that the human body is not merely

a virtual mass-point in a physical model, it is a com-

plex articulated-body with joints and body segments,

with related masses and degrees of freedom. Via the

IMUs sensors, it is possible to determine: i) the angu-

lar movements of a joint of the body and its geomet-

rical trajectories, having some a-priori determinants,

i.e. knowing which portion of the body the sensor is

KITA 2016 - 2nd International Workshop on the design, development and use of Knowledge IT Artifacts in professional communities and

aggregations. Knowledge Artifacts as resources in the maker and DIY communities

308

applied to and which exact position it is placed on; ii)

the dynamics of the related motion, i.e. the acceler-

ations and angular velocities to which the segment is

subjected. These data are measured at run-time, and

all information can be stored with their timestamps,

for successive analysis and tests. In particular, we are

interested in four aspects of human body motion: i)

angles and accelerations of the joints, ii) movements

and accelerations of the CoM (Center of Mass) of the

human body, iii) orientation of the spine cord with re-

spect to the vertical position, iv) direction (heading)

of the person, while in motion or not. Semantically

speaking, the motion of the human body is made of

concurrent events (see Figure 2). For example, a per-

son can stay still, turning of 90 degrees on the left, if

he/she is sitting on a swivel chair; or he/she can move

and rotate of 90 degrees on the left, if he/she rounds

the corner walking down a corridor. In this case, mov-

ing and rounding are considered concurrent aspects of

an action. The system should be able to detect all the

concurrent aspects of the body movement to further

combine them into a more complex event. An action

is then composed by a time series of elementary con-

current movement-information and facts.

Figure 2: Determination of the personal state: movement

section.

5 HOW DOES IT WORK?

To devise the motion determinants, IMUs use two ref-

erence systems: one inertial with respect to Earth,

usually called frame G, and a second reference system

co-moving with the sensor, called frame S. In general,

given a reference system S

where i indicates the i-th

sensor, it is possible to identify: the angular velocities

around the three axes of S

; the rotation angles Φ, χ,

anmd Ψ, namely Yaw, Pitch and Roll, which indicate,

instant by instant, the rotation of the system S

with

respect to G, and the instantaneous accelerations of S

on the three axes. These raw data can be then used

to precisely determine the motion of the human body,

deriving the motion parameters of its CoM and seg-

ments. Note that the purpose is not to determine the

exact positions of the body segments, which would

require to place a sensor on every body segment (ﬁn-

gers included), but to devise the general state of the

person. In particular, we want to determine the ac-

tivities and actions carried out in a speciﬁc context

(drinking, running, climbing the stairs, sitting, etc.).

To this aim, the KA-User is designed according to

a semantically driven rather than geometrically driven

approach: understanding the action semantics is an

important aspect in many application domain, espe-

cially in the medical one, in which the ADLs (Ac-

tivities of Daily Living) are considered an affordable

measure of the capacity of the person to be indepen-

dent.

Figure 3: The data ﬂow architecture.

6 KA-User COMPUTATIONAL

MODEL

Today it is possible to seamlessly use any kind of sen-

sor available on the market mounted in mobile phones

and wrist bands or smart-watches. Market’s low costs

IMUs are valid instruments of measure, with their in-

herent precision and accuracy. Figure 3 shows the

data ﬂow of the KA-User.

The three key components of the architecture are:

1) a set of sensors (each equipped with three-axial

accelerometers, gyroscopes and magnetometers, i.e.

a three-axial IMU) to capture body data movements,

and 2) an Android Smartphone to channel the data

and eventually transport them via Wi-Fi to a Server;

3) a Server that i) stores the data for future analysis, ii)

shows the data for an intuitive visual analysis, and iii)

carries out body information at runtime (or during the

ofﬂine analysis). The purpose is the determination of

movement semantic artifacts, useful for situation as-

sessment.

Note that the data and the derived information

of the sensors can be sent to a concentrator or any

other net-component in two possible ways: using

a WSN (Wireless Sensor Network) and, for exam-

Integrating Knowledge Artifacts and Inertial Measurement Unit Sensors for Decision Support

309

Figure 4: Server Side View, On top the tabs of the graphs view, At the center the data (e.g. magnetometer) of both the phone

and smartwatch IMUs, of the given view; bottom-down the state of the person at the current time.

ple, a ZigBee protocol, or using a socket connection

(TCP/IP) over Wi-Fi. Our application was originally

written as a full component of a grid of communi-

cation of a WSN, but the increasing preference of

Wi-Fi/Bluetooth protocol by wearable devices, made

mandatory to use a Wi-Fi/Bluetooth channel for the

communication between components. In the current

KA-User implementation, data are acquired by us-

ing two IMUs respectively present in two commer-

cially available devices: an Android Smartphone with

a three axial IMU, which is usually placed on a belt,

i.e. in the proximity of the CoM of the body; and a LG

W110 G Watch R SmartWatch, worn by the user on

his/her left wrist (but a Microsoft Band or an Apple S

watch will be perfectly usable, too). The code of the

Android smartphone is written in Java and developed

using Android Studio; the code of the smart-watch

is an Android Wear component and contains also a

Bluetooth Energy component for the bands. A com-

ponent for Microsoft Bend was also developed in C#.

The server used for the runtime visualization of the

resulting data and for the data analysis, the most valu-

able component, was developed in C# using Visual

Studio 2015. Communication between the Android

Smartphone and the Server use a TCP/IP socket over

a Wi-Fi channel.

7 AN APPLICATION SCENARIO

A typical example of acquisition is described in the

following: given a person wearing a smartphone on

the belt and the smart-watch on the left wrist, the pur-

pose is to detect in real time, instant by instant, the

position of the body (standing, sitting, laying), the di-

rection (heading) of the person, and the quantity of

movement (the person is still or moving); we want

also to know the path followed by the person from

a speciﬁc point, for a few minutes, without having

other sources of information (see Figure 4). Further-

more, we want to know the Hearth Beat Rate of the

person, instant by instant, while the actions are per-

formed. These data are considered pre-determinants

for a more complex description of the person status.

Figure 5 shows the output provided by the KA-

User in response to the ﬁrst question. Inclination of g

vector with respect to the spinal cord is used to deter-

mine if the person is standing, sitting or laying. The

direction, or heading, is determined using Euler an-

gles, typically the Yaw data, while the quantity of

movement is calculated as a proportion of standard

deviation of the magnitude of the accelerometer. Note

that direction and position of the body requires the a

priori knowledge of the position of the sensor with re-

KITA 2016 - 2nd International Workshop on the design, development and use of Knowledge IT Artifacts in professional communities and

aggregations. Knowledge Artifacts as resources in the maker and DIY communities

310

Figure 5: A KA-User Server Side Data-Example: movements quality and quantity.

spect to the body (i.e. with respect to the G reference

system), while the magnitude is independent by rota-

tions, being a scalar quantity.

To reconstruct the movements of the person in the

space we use a dead reckoning approach: it is im-

portant to remember that dead reckoning approaches,

without any further correction, are subject to errors,

so the reconstructed position tends to separate and to

diverge with respect to the actual position. Hence, any

useful reckoning method requires an external source

of information of position to reconcile the prediction

to the actual position. A Bayesian ﬁlter, or an Ex-

tended Kalman Filter is normally used to maintain

consistent the prediction to the real position.

Figure 6 shows the output provided by the KA-

User in response to the second question. Heart Beat

Reat (HBR) simply requires to read the relative data

from the smartwatch that constantly monitor the HBR

of the wrist. This piece of information ﬂows from the

smartwatch to the smartphone, and then goes to the

server where is coupled with the motion information.

Figure 7 show a sketch the ﬁnal KA-User archi-

tecture on the basis of new improvements. The most

interesting feature is its transformation into client-

server module: the client is made up of hardware and

software components to collect and analyze data, the

server is made of software components to make cor-

rect interpretation of them according to the applica-

tion domain. The client receives real-time data from

sensors placed on wearable devices. The IMU mod-

ule is then responsible for the extraction of useful in-

formation from them: this information is then sent to

the server for next interpretation. Given that the most

interesting feature of the current KA-User implemen-

tation is its capability to automatically query wear-

able devices, it has been provided with a graphical

user interface for manual input by user. This feature

allows recording data like personal user information

(e.g. name, surname and so on), security data (like

user-name and password for server connection) and so

on. The data interpreted by the IMU module are then

stored by the server into a proper MySql database, for

future use by KAFKA. For example, a KA-Developer

can exploit them to build up knowledge artifacts for

recommendation systems development, as shown in

Melen et al. (2015).

Integrating Knowledge Artifacts and Inertial Measurement Unit Sensors for Decision Support

311

Figure 6: A KA-User Server Side Data-Example: Heart Beat Rate detection.

Figure 7: The ﬁnal KA-User architecture.

8 CONCLUSIONS AND FUTURE

WORKS

This paper has presented an improvement of the KA-

User role in the KAFKA framework, aiming at wear-

able device exploitation to detect and analyse new ob-

servations on the ﬁeld. These observations can be

proﬁtably used in many domains: in particular, we

are collaborating with psychologists of the Univer-

sity of Milano-Bicocca in the design and development

of mobile apps for the promotion of physical activity

in people potentially affected by cardiovascular dis-

eases (Baretta et al., 2016), taking care of both quan-

titative (like HBR) and qualitative (like self-efﬁcacy)

variables.

Physical activity (PA) is considered one of the

most important factors for the prevention and man-

agement of non-communicable diseases (NCDs).

Mobile technologies offer several opportunities for

supporting PA, especially if combined with psycho-

logical aspects, model-based reasoning systems and

personalized human computer interaction. Given

that people carry smartphones and can access data

anywhere and anytime, physical activity behaviour

change promotion apps offer the opportunity to pro-

vide tailored feedback and advice at the appropriate

time and place. Therefore, apps offer new opportuni-

ties to deliver individually tailored interventions, in-

cluding real-time assessment and feedback that are

KITA 2016 - 2nd International Workshop on the design, development and use of Knowledge IT Artifacts in professional communities and

aggregations. Knowledge Artifacts as resources in the maker and DIY communities

312

more likely to be effective.

The integration of IMUs and KAFKA in this con-

text will allow expanding the range of recommenda-

tions suggested to the user, analyzing from both the

quantitative (e.g. detecting and elaborating his/her

HBR) and qualitative (e.g. understanding how the

physical activity is conducted) points of view his/her

physical and psychological status, with the ﬁnal aim

to build up a complete and innovative framework for

personalized training.

REFERENCES

Avci, A., Bosch, S., Marin-Perianu, M., Marin-Perianu, R.,

and Havinga, P. (2010). Activity recognition using

inertial sensing for healthcare, wellbeing and sports

applications: A survey. In Architecture of computing

systems (ARCS), 2010 23rd international conference

on, pages 1–10. VDE.

Bao, L. (2003). Physical activity recognition from accel-

eration data under semi-naturalistic conditions. PhD

thesis, Massachusetts Institute of Technology.

Baretta, D., Sartori, F., Greco, A., Melen, R., Stella, F.,

Bollini, L., D’addario, M., and Steca, P. (2016). Wear-

able devices and ai techniques integration to promote

physical activity. In Proceedings of the 18th In-

ternational Conference on Human-Computer Interac-

tion with Mobile Devices and Services Adjunct, pages

1105–1108. ACM.

Bisiani, R., Merico, D., Pinardi, S., Dominoni, M., Cesta,

A., Orlandini, A., Rasconi, R., Suriano, M., Umbrico,

A., Sabuncu, O., et al. (2013). Fosteringsocial interac-

tion of home-bound elderly people: the easyreach sys-

tem. In International Conference on Industrial, Engi-

neering and Other Applications of Applied Intelligent

Systems, pages 33–42. Springer.

Cabitza, F. and Locoro, A. (2014). Made with knowledge:

disentangling the it knowledge artifact by a qualita-

tive literature review. In Proceedings of the 6th Inter-

national Conference on Knowledge Management and

Information Sharing, Rome, pages 64–75.

Cabitza, F., Simone, C., and De Michelis, G. (2015). User-

driven prioritization of features for a prospective in-

terpersonal health record: Perceptions from the italian

context. Computers in biology and medicine, 59:202–

210.

Holsapple, C. W. and Joshi, K. D. (2001). Organiza-

tional knowledge resources. Decision support sys-

tems, 31(1):39–54.

Melen, R., Sartori, F., and Grazioli, L. (2015). Model-

ing and understanding time-evolving scenarios. In

WMSCI 2015 - 19th World Multi-Conference on Sys-

temics, Cybernetics and Informatics, Proceedings,

volume 1, pages 267–272.

Mileo, A., Merico, D., Pinardi, S., and Bisiani, R. (2009).

A logical approach to home healthcare with intelligent

sensor-network support. The Computer Journal.

Mileo, A., Pinardi, S., and Bisiani, R. (2010). Movement

recognition using context: a lexical approach based

on coherence. MRC 2010, page 37.

Pinardi, S. and Bisiani, R. (2010). Movement recogni-

tion with intelligent multisensor analysis, a lexical

approach. In Intelligent Environments (Workshops),

pages 170–177.

Salazar-Torres, G., Colombo, E., Da Silva, F. C., Noriega,

C., and Bandini, S. (2008). Design issues for knowl-

edge artifacts. Knowledge-based systems, 21(8):856–

867.

Sartori, F. and Melen, R. (2015). Time evolving expert sys-

tems design and implementation: The KAFKA ap-

proach. In KEOD 2015 - Proceedings of the Inter-

national Conference on Knowledge Engineering and

Ontology Development, part of the 7th International

Joint Conference on Knowledge Discovery, Knowl-

edge Engineering and Knowledge Management (IC3K

2015), Volume 2, Lisbon, Portugal, November 12-14,

2015, pages 84–95.

Integrating Knowledge Artifacts and Inertial Measurement Unit Sensors for Decision Support

313