An Online Balance Training Application using Pose Estimation and

Augmented Reality

Amirhossein Etaat

, Negar Haghbin

1 a

and Marta Kersten-Oertel

1,2 b

Gina Cody School of Computer Science and Engineering, Concordia University, Montr

eal, Canada

PERFORM Centre, Concordia University, Montr

eal, Canada

Keywords:

Healthy Aging, Rehabilitation, Pose Estimation, Web Applications, Balance Training, Augmented Reality.

Abstract:

The evolution of digitally connected devices and artiﬁcial intelligence has opened the door for novel health

and ﬁtness applications that can be used by individuals at a time and in an environment convenient to them.

The purpose of our research was to develop a platform that requires no additional hardware to provide an

online home balance training program. Balance exercises are often prescribed for healthy aging to keep the

body active, improve balance and coordination, and prevent falls and injuries, as well as, for those doing

rehabilitation after injuries or diseases such as stroke. We developed a simple web application (BaART:

Balance Augmented Reality Trainer) that uses PoseNet to determine a user’s location and pose for balance

exercises. Furthermore, we looked at how augmented reality, and speciﬁcally adding a virtual chair, might

impact a user’s sense of balance. In a study of 20 participants with and without balance disorders, we found

that the developed system was easy to use, however, we also found that the virtual object (i.e. chair) was not

used by most people. Furthermore, those with balance issues felt they required a real chair for balance and

some even felt that the virtual object was distracting from the exercise. In the future, we will explore other

uses of augmented reality, such as feedback on exercise quality, gaming features, and a virtual avatar trainer.

1 INTRODUCTION

As smart mobile phones and internet connections

become more accessible than ever, mobile health

(mHealth) applications are becoming ever more pop-

ular with many seeking to take their health into their

own hands within their own environments. This has

led to growing availability of web and mobile appli-

cations for rehabilitation, virtual coaches for training

and exercise, online doctor visits, and mental health,

meditation, and ﬁtness applications.

In this paper, we focus speciﬁcally on rehabilita-

tion and balance exercises. Balance exercises which

work the core muscles, lower back, and legs are often

prescribed for healthy aging to keep the body active,

improve balance and coordination, and prevent falls

and injuries. They are also used in rehabilitation after

injuries or diseases such as stroke. New sensing (e.g.

wearables) and communication technologies (e.g. in-

ternet of things) are positively inﬂuencing the expan-

sion of training and rehabilitation programs outside

https://orcid.org/0000-0002-9598-9936

https://orcid.org/0000-0002-9492-8402

of standard healthcare facilities. It is now feasible to

do guided online exercises ensuring that they are done

correctly and even track a person’s progress over time

in the comfort of one’s own environment without di-

rect contact with a physiotherapist or trainer. This can

be done using different motion-tracking devices such

as wearable sensors, depth cameras and most recently

using pose estimation from simple web camera feeds.

Pose estimation is the task of using machine learning

models to estimate the pose of a person from an im-

age or a video by determining the spatial location of

keypoints (i.e. knee, elbow, nose, etc). Pose estima-

tion is beginning to be widely used in training robots,

motion tracking for game consoles, augmented real-

ity, animation, and health and ﬁtness applications.

Gamiﬁcation, i.e. the introduction of game ele-

ments to traditionally non-gaming situations, has also

proved to be an important aspect of mhealth applica-

tions, which can encourage and motivate users to not

only comply with their exercise regimes but also to

help them to enjoy doing so (Haghbin and Kersten-

Oertel, 2021). Augmented reality (combining virtual

and real elements) can also be used both for aiding

users to better understand exercises and the proper

168

Etaat, A., Haghbin, N. and Kersten-Oertel, M.

An Online Balance Training Application using Pose Estimation and Augmented Reality.

DOI: 10.5220/0010973800003188

In Proceedings of the 8th International Conference on Information and Communication Technologies for Ageing Well and e-Health (ICT4AWE 2022), pages 168-176

ISBN: 978-989-758-566-1; ISSN: 2184-4984

way of doing them and/or for adding gaming elements

to make exercising more enjoyable.

In this paper, we focus on delivering balance train-

ing exercises into the comfort of a user’s own home

using simple and low-cost technology (only a web-

cam and connected digital device are needed). Fur-

thermore, we explore the use of augmented reality

elements, i.e. adding a virtual chair for balance, to

see if this can aid users to perform balance exercises.

In order to study these questions, we developed a

balance training exercise prototype system, “Balance

Augmented Reality Trainer” (BaART) and tested it

with 20 subjects to determine the impact of the virtual

elements and in general the ease of use of the system.

2 RELATED WORK

2.1 Pose Estimation Health Applications

Encouraging people to exercise by exploiting tech-

nology at home can be a strategy that may be cost-

effective, particularly if consumer-level equipment is

used. A number of research groups have looked at us-

ing simple applications with webcams and using pose

estimation algorithms to determine the location of the

user’s joints in order to help guide and monitor exer-

cises. Pose estimation speciﬁcally refers to computer

vision techniques that can be used to detect human

ﬁgures in images and videos. By taking a processed

camera image as the input, pose estimation models

outputs information about the keypoints (e.g. elbow,

nose, knee, etc.) which can then further be used to

determine the pose of a person.

Moreira et al. (Moreira et al., 2020a) systemati-

cally reviewed mobile applications proposed for ana-

lyzing human posture. Their results showed that the

use of human pose estimation on mobile applications

is reliable, and can assist medical clinics, and physio-

therapists, especially in the case of evaluating physi-

cal treatments. Moreira et al. (Moreira et al., 2020b)

also proposed a prototype application using PoseNet

(a real-time pose detection library) and the Tensor-

Flow library for automatic identiﬁcation of Anatom-

ical and Segment Points (ASPs). Their results show

that PoseNet can be used to develop applications con-

cerning the physical assessment process and diag-

nosis of disorders related to postural and movement

changes. Herrera et al. (Herrera et al., 2020) devel-

oped a web application, using Posenet and Tensor-

Flow, for sedentary workers to help them take active

pauses (i.e. pausing work to do exercises) in order

to reduce the risk of job-related muscular and skele-

tal injuries and diseases. The application accurately

determined the position of the user and whether they

were doing the exercise correctly.

2.2 Balance Training

A number of previous research studies have employed

various technologies to help aging adults to train bal-

ance, with an aim to maintain health and reduce the

likely hood of falls (Jorgensen et al., 2013). Mosta-

jeran et al. (Mostajeran et al., 2019) presented an ap-

plication for balance training at home using the Mi-

crosoft Kinect Sensor V2. The application has a vir-

tual coach who gives balance training instructions and

demonstrated the exercises. The authors found that

adults have a more positive reaction towards using

a virtual coach for balance training compared to tra-

ditional health care approaches. In a similar work,

Kouris et al. (Kouris et al., 2018) proposed a system

that uses a virtual reality avatar and wearable sensors

for physiotherapy balance exercises. The wearable

sensors can monitor user activity and determine the

correctness of the balance exercises in real-time. Von-

stad et al. (Vonstad et al., 2020) developed a 2D cus-

tom balance training exergame (i.e. a type of exercise

that mixes exercise and video games) using a deep

learning based pose estimation system to detect hu-

man body parts and estimate three-dimensional (3D)

body positions. In their work, they compared three

systems: a Microsoft Kinect Sensor V2, a marker-

based three-dimensional motion capturing system and

a deep learning system using a digital camera. In

a study where participants played the balance ex-

ergame, the deep learning method had similar perfor-

mance to both the Kinect and the marker-based sys-

tem, demonstrating the feasibility of using less com-

plex hardware and sensors for these types of applica-

tions.

Ogonowski et al. (Ogonowski et al., 2016) devel-

oped a system that used individualized physical ﬁt-

ness training, gamiﬁcation and wearable sensors with

a Microsoft Kinect and simple TV. Their work illus-

trated the possibility of incorporating such systems

into the daily life of older adults. Hardy et al. (Hardy

et al., 2013) also developed an exergame to motivate

older adults to do balance exercises and gain training

based on the adaptation and exergame analysis. Their

ﬁndings suggest that elderly individuals accept mul-

timedia training and the adaptability notion improves

the system’s accessibility.

Smartphones are also beginning to be tested for

use for balance, rehabilitation and physiotherapy ex-

ercises. For example, Androutsou et al. (Androut-

sou et al., 2020) proposed a smartphone application

for patients with balance disorders that enables users

An Online Balance Training Application using Pose Estimation and Augmented Reality

169

to self-evaluate their activity and progress, communi-

cate with others using the system, and get real-time

feedback about their training, activities and progress

over time.

2.3 Augmented Reality in mHealth

A number of researchers have explored the use of

augmented reality in the context of ﬁtness and reha-

bilitation applications and speciﬁcally the use of vir-

tual coaches or trainers (Tsiourti et al., 2014). Virtual

coaches have been explored to help clinicians moni-

tor the use of medications and seniors’ adherence to

speciﬁc guidelines for medication use (F

elix et al.,

2019) or to keep the elderly physically active (Bick-

more et al., 2009; Albaina et al., 2009). For example,

Felberbaum et al. (Felberbaum et al., 2018) explored

aspects of an AR-based virtual coach that would im-

prove older adults’ mobility. Features of such a virtual

coach would include virtual friends to walk with, hav-

ing interactive guidance to help deﬁne and reach goals

and online monitoring to determine risks such as falls.

Bickmore et al. (Bickmore et al., 2009) demonstrated

that the combination of a virtual coach and pedome-

ter can increase the amount of walking among older

adults.

Virtual training assistants or coaches do not nec-

essarily have to take on the form of a human avatar.

Albaina et al. (Albaina et al., 2009) proposed an ani-

mated ﬂower that is used as a virtual trainer to boost

the motivation of individuals to walk. Their study

found that elderly users appreciated the ﬂower vir-

tual trainer and had a positive tendency to use it over

a longer period of time. The results of their studies

also demonstrated that the virtual ﬂower trainer was

effective in boosting the acceptance of such a system

among older adults, despite its uncommon form.

Other studies have incorporated AR technology

into the daily lives of older adults. Ku et al. (Ku et al.,

2019) used the 3D-ARS (3D interactive augmented

reality) system to determine its impact on people with

balance disorders. A randomized controlled study

with 36 people who could walk on their own and

could stand on one leg was done to assess the effect of

the AR technology. In comparing the control group,

who did a conventional physical ﬁtness program three

times a week for 1 month, to the experimental group,

who used 3D-ARS training three times a week for 4

weeks, they found that although both groups experi-

enced improvements those who used of 3D-ARS had

better results in terms of stability index, weight distri-

bution index, fall risk index, and Fourier transforma-

tions index of posturography. Roy et al. (Roy et al.,

2017) developed a low-cost AR system using the Mi-

crosoft Kinect. The system guides users to perform

stability training and even calculates a real-time sta-

bility score.

3 SYSTEM DESCRIPTION

Using intervention and assessment tools at home can

considerably decrease the risk of falls and enhance

one’s quality of life through increased mobility and

in turn healthy aging (Society et al., 2001). Balance

training exercises and therapies are commonly used

to improve both balance and gait conditions. Accord-

ing to the American Geriatrics Society (Panel on Pre-

vention of Falls in Older Persons, 2011) a proper and

standard balance exercise plan can play a vital role

in health and rehabilitation. However, studies have

found that many patients and users do not do physio-

therapy exercises or training that can help or prevent

ailments due to a lack of motivation, time and/or the

high cost of physiotherapy, gyms, and hiring personal

trainers (Argent et al., 2018). Building on the previ-

ous works described above, our goal was to develop

a prototype system that requires no specialized equip-

ment but rather can be used in the comfort of one’s

desired environment by simply clicking on a url on

a computer, mobile phone, tablet, Smart TV or other

connected device. We also aimed to make the system

easy to use by providing a simple interface that in-

cludes voice commands and instructional videos. Our

developed home-based rehabilitation online app uses

(1) pose estimation to determine the users pose and

count the number or repetitions of a given exercise

and (2) explores the ability of augmented reality to

help with balance exercises. The motivation for the

latter, was to determine if a stationary object added

via a virtual object, which would provide a visual

static reference point, can improve balance. Although

this is the case for walking or balancing in the real

world it is unclear if virtual objects have the same im-

pact.

3.1 PoseNet

Our prototype uses PoseNet (Papandreou et al.,

2018), a library built on top of TensorFlow, which is

an open-source machine learning platform that pro-

vides an ecosystem of tools for developing machine

learning applications. PoseNet is trained in the Mo-

bileNet Architecture (a convolutional neural network

(CNN) developed by Google and trained on Ima-

geNet) and has pre-trained models that detect user

gestures and poses from given images. An advantage

of PoseNet in comparison to other API-dependent li-

ICT4AWE 2022 - 8th International Conference on Information and Communication Technologies for Ageing Well and e-Health

170

Figure 1: The user opens the application on a browser on any connected digital device. The web browser fetches data from

the server (written in HTML, CSS, JavaScript codes including Posenet Library and P5js library). The keypoints are detected

using the PoseNet model. The web application processes the captured inputs to determine exercises correctness and counts

repetitions, as well as, displays the AR object in the correct position relative to the user.

braries for pose estimation is that the pre-trained mod-

els run in a browser thus private data (i.e. images and

video stream from webcam) are protected and not sent

to a server for processing. Furthermore, anyone with a

webcam equipped desktop or TV, tablet or phone can

use PoseNet within their web browser. Taking advan-

tage of PoseNet allowed us to keep the system sim-

ple and highly accessible, without the need of devices

such as Kinect, wearables or motion capture systems

that can add complexity and may have high installa-

tion and preparation costs.

3.2 BaART: Balance AR Trainer

The BaART user interface was developed using

Javascript, HTML and CSS and PoseNet and Ten-

sorFlow’s models were used to recognize and esti-

mate the human poses. Pose estimation was speciﬁ-

cally used to count the number of repetitions that were

done by the end user and check if the exercise has

been done correctly. We also used p5.Speech (an ex-

tension of p5.js which is an open-source JavaScript

library) for implementing speech-to-text and voice

commands.

The application pipeline is shown in Figure 1. As

mentioned above, the application uses a simple web-

cam without any further specialized equipment. After

opening the application via the provided URL on a

computer, laptop, or mobile device, RGBA input is

captured from the web camera and the user’s pose is

estimated via PoseNet. After successful keypoint de-

tection, the exercise routine begins. The keypoint data

(e.g. knee, ankles, hips, shoulders, eyes, etc.) is used

to determine the location of the user to both place the

virtual object and determine if an exercise is being

done. While exercises are being performed, the key-

points and movements are tracked so that the system

can evaluate if an exercise is being executed correctly

and the number of repetitions the user has done. Re-

sults are displayed to the user in real-time through the

interface, which shows the count and or repetitions

(either number of seconds or repetitions successfully

completed). The user can interact with the applica-

tion using either voice commands: “next exercises”,

“re-do”, “refresh” or “previous” or with a mouse on a

computer or gestures on a mobile device.

3.2.1 Exercises

For the ﬁrst prototype of our system we selected three

exercises that improve balance and prevent falls based

on the clinical guidelines and recommendations of

the National Institute of Clinical Excellence (NICE)

and the joint American and British Geriatric Society

(ABGS) for the prevention and assessment of falls

in older people (National Institute for Health and

Care Excellence (NICE), 2013; Panel on Prevention

of Falls in Older Persons, 2011). Speciﬁcally, we

chose: heel raises, side stepping and single leg stand-

ing exercises (see Figure 2).

Tyagi et al. (Tyagi et al., 2018) studied the adop-

tion of technology-based telerehabilitation (TR) and

found that although younger users preferred TR, older

ones preferred daily visits to the rehabilitation center.

As in person visits are not always feasible particu-

larly for daily visits (or during a global pandemic),

Tyagi et al. recommended using videos in TR pro-

grams for older patients. Thus for each of the exer-

cises in the BaART application, an instructional video

(taken from www.Physitrack.com) is ﬁrst shown to

explain how to do the exercise. After the video, the

user then does the given exercise following along with

both verbal and onscreen written instructions. As well

as giving instructions, the voice of the “trainer” tries

to motivate the user with encouraging statements and

counts the repetitions the user has done. We brieﬂy

describe

Heel Raises: For heel raises, the keypoint informa-

tion is used to track the position of the hips of the

user while standing and uses the raising and lower-

ing of the hips to determine the number of repetitions.

Often times a chair is used for extra balance in heel

An Online Balance Training Application using Pose Estimation and Augmented Reality

171

raise exercises, thus we use a virtual chair to study if

“resting” one’s hands on a virtual chair can invoke a

sense of balance. The virtual chair is placed in front of

the determined location of the user. During the exer-

cise, the user raises both of their heels off the ground

to stand on their tiptoes. The user raises their heels

10 times, each time the application says “raise your

heels”, and then slowly lowers their heels back to the

ﬂoor when the application says “release”.

Side Step: For side stepping, the keypoint informa-

tion of the hips and shoulders are used to ensure the

user is stepping to the side correctly and to count the

repetitions. Again, a virtual chair is shown in front

of the user so that the user can “rest” their hands and

focus on this static point for better balance. This ex-

ercise starts with the feet together and knees slightly

bent, the user then moves one foot to the side and then

back to join the other foot. The instructions given are

“step right” to take a full right step and then “release”

to step back to the centre and bring the feet back to-

gether. The exercise is repeated 10 times.

Figure 2: The instructional videos demonstrating the three

exercises and user doing the exercises with the virtual chair.

The green line represents the position of the hips.

Single Leg Standing: For this exercise, the most

challenging of the three, keypoint information is used

to determine how high the leg is raised relative to the

distance to the pelvis. For this speciﬁc exercise, the

user is asked to rotate the body by 90 degrees to the

right or left in order to be able to best detect the knees.

The user is then instructed to stand upright with their

feet together, and try to slowly lift their foot off the

ground so their thigh is perpendicular to the ﬂoor. The

virtual chair is provided and the user is instructed to

do single leg standing for 15 seconds for the left and

then right leg.

3.2.2 User Interface

A simple user interface (UI) was developed so that

users of all ages can easily interact with BaART.

For instance, verbal communication and images were

used to convey information to the user. For those with

hearing problems, the words and sentences are tran-

scribed and displayed at the top of the interface. Also,

we tried to encourage and motivate the user after each

correct set using motivational and prompting words

and phrases, e.g. “good job!”, similar to a real trainer.

4 BaART EVALUATION

The BaART application was evaluated with 20 partic-

ipants aged 18 to 73 with 14 males and 6 females (see

Figure 3). Participants used the application online at

their convenience. Those who agreed to participate

were ﬁrst sent a consent form, a description of the ap-

plication, instructions on how to use the BaART ap-

plication and a link to the application that they could

run on the device of their choice at their convenience.

4.1 Questionnaire

After the study, participants ﬁlled out a questionnaire

with both background questions and questions about

their experience with the application. Note that the

pre-test and post-test questionnaires were combined

to facilitate navigation between instructions, applica-

tion and questionnaires. The summary of the analy-

sis of the questionnaire are presented in the following

sections.

4.1.1 Technology Use

To determine comfort with new technologies, partic-

ipants were questioned on how they felt about their

Figure 3: The BaART application was evaluated with 20

participants aged 18 to 73 with 14 males and 6 females.

ICT4AWE 2022 - 8th International Conference on Information and Communication Technologies for Ageing Well and e-Health

172

practical knowledge of technology, and how often

they used a smartphone, tablet, or computer. The ma-

jority of participants (75%) use some form of digital

device daily, whereas 25% use a device only weekly.

In terms of their practical knowledge of technology,

which was answered on a scale from 1 (I don’t know

anything about technology) to 5 (I’m very technolog-

ically savvy), the majority of participants rated their

knowledge as 4 or 5, thus our sample participants

were technologically savvy.

We also asked how keen the participants are on

trying out new technologies from 1 (Not at all) and 5

(Yes I’m usually very keen) and found that 35% were

very keen, 40% were quite keen, 10% were neither

keen or not keen and 15% were not keen. Lastly,

we asked participants about the degree to which it

was easy and simple for them to use new technolo-

gies such as mobile apps and found that the majority

ﬁnd it either very easy or somewhat easy.

4.1.2 Sport and Health Activities

The second part of the questionnaire queried users

about health and ﬁtness activities. Speciﬁcally, partic-

ipants were asked if they considered themselves phys-

ically active as well as how often they participated in

ﬁtness activities such as sports, aerobics, training, etc.

In a range of 1 (Not at all physically active) to 5 (Yes

I’m very physically active), only 1 person (5%) con-

sidered themselves very active, 30% considered them-

selves somewhat active and 40% considered them-

selves neither very active or inactive. Lastly, 25%

of participants considered themselves somewhat inac-

tive. In terms of how often participants exercised 40%

said 1-2 times a week, 40% 3-4 times a week, and

20% answered that they hardly ever exercise. None

of our participants exercised 5-6 times per week or

everyday. In terms of users ideal location for exercis-

ing, 30% prefer exercising at home, 35% at the gym,

30% prefer exercising outside and 5% said they didn’t

have a preference.

We also questioned participants about the main

factors that lead them to exercise less frequently than

they would like to. Here participants could select mul-

tiple answers. Of our population, 95% (19 people)

answered “I don’t have time”, 6 answered “I forget to

exercise”, one person answered “I don’t like exercis-

ing”, one other said “I don’t feel I need to exercise”,

and one person answered that they did not exercise

because they did not have someone else to exercise

with.

We also asked participants about their use of on-

line exercise training (e.g. Apple Fit+, Wii, YouTube

online class) and found that most participants (70%)

had not previously done online ﬁtness classes or ex-

ercises whereas as 30% had. Lastly, participants

were asked if they have ever done any rehabilita-

tion, physiotherapy or balance training. For reha-

bilitation and/or physiotherapy 20% had done this,

whereas 80% had not. For balance training which

included balance exercises within yoga and rehabil-

itation/physiotherapy routines, we had 25% who had

done balance training in yoga or uni-cycling routines

and 75% who previously had not done any balance

exercises and one person (5%) who was not sure.

4.1.3 Balance Disorders

As our application concerns balance exercises, we

queried the participants about their experience of

dizziness, falls and loss of balance. Speciﬁcally, we

asked if participants had experienced a loss of balance

or had a balance problem leading to falls or near falls

in the last six months. The majority of participants

(65%) did not have any balance problems. More than

one-third of participants reported that they suffered

from dizziness and loss of balance. Speciﬁcally, 10%

had experienced a fall and 20% had nearly fallen. One

person had experienced both a near fall and had fallen

in the last six months.

4.1.4 Ease of Use

Participants were asked to rate the application on a

scale of 1 (Hard to understand) to 5 (Easy to under-

stand), for the video instructions and auditory coach-

ing instructions. We found that 60% found the video

instructions easy to understand (rated with 5), 35%

found the video instructions somewhat easy to under-

stand and 5% found them neither hard or easy. For

the spoken instructions, 55% found them easy to un-

derstand, 40% found them somewhat easy to under-

stand and 5% found them difﬁcult to understand. We

also asked if the subjects used the voice commands

(next, repeat, previous) while doing the study. Most

of the participants, 80% used voice commands, 15%

did not and one person (5%) answered maybe. Of

those that used the voice commands, 11.5% found

them very easy to use, 47% found them somewhat

easy to use, 11.5% found them difﬁcult to use, and

30% found them very difﬁcult to use.

In terms of the overall system ease of use 50%

of the participants found it very easy, 35% found it

somewhat easy to use, 10% found it neither hard or

easy and one participant (5%) found it difﬁcult to use.

The majority (80%) tested the application on laptops,

15% tested on workstation computers and 5% (1 per-

son) used a mobile device.

An Online Balance Training Application using Pose Estimation and Augmented Reality

173

4.1.5 Augmented Reality

As one of our goals was to determine if augmented re-

ality (provided by virtual objects introduced into the

video images) could aid in improving a sense of bal-

ance, we speciﬁcally asked users about their impres-

sion of this. First users were asked if they tried using

the virtual chair for the different exercises, we found

that 60% of participants did use the virtual chair, 20%

did not, and 20% answered maybe. Of those that used

the virtual object, we asked if it gave them an impres-

sion of helping them with their balance as they per-

formed their exercises. On a scale of 1 (It didn’t help

at all) to 5 (It helped me to balance), we found that

21% found it did not help at all, 10.5% found it did

not help, 32% were not sure, 26% thought it helped

a little to balance and 10.5% found it helped them to

balance. Furthermore, we asked participants if they

had any comments about the use of the virtual objects

for exercises, most of the participants didn’t have any

issue with using virtual objects, but we received some

comments including “I do not know how to use a vir-

tual object to maintain my balance”, “The virtual ob-

jects didn’t improve my performance”, as well as, “It

did not help much and sometimes made me lose my

focus on the exercise itself just because I wanted to

keep my hands on it”.

4.1.6 System Usability Scale

Finally, we provided participants with a version of

the System Usability Scale (SUS) (Brooke, 1996).

The questionnaire has ten questions positive and neg-

ative questions relating to a system’s usability. The

questions are answered on a scale of 1 to 5, with

1 representing Strongly Disagree and 5 representing

Strongly Agree with the question’s statement. The

BaART application received a SUS score of 80. The

average SUS score is 68, and given that a SUS value

of 75 is considered good, and over 85 is excellent, our

system was rated very highly with participants ﬁnd-

ing who found it easy to use, easy to learn, and felt

conﬁdent using it. The results of the individual SUS

questions are represented in Table 1.

4.1.7 Using BaART in the Longer Term

We asked participants to answer on a scale from 1

(very unlikely ) to 5 (very unlikely) if they would use

BaART if the prototype was extended with more exer-

cises and features. Thirty percent of participants were

not sure if they would use such a system, 20% percent

and 50% said they would very likely use it.

We also queried speciﬁcally if the participants felt

that such an application can motivate them to exer-

cise whenever they did not feel like it or were busy,

by sending notiﬁcations to their phone, on a scale of

1 (No would not help at all) to 5 (Yes it would very

much help). We found that 5% said it would not help

at all and another 5% said it would not help and 15%

said they were not sure. However, 35% thought it

would help and 40% thought it would help very much.

Lastly, we had an open question to gather the par-

ticipant’s overall opinions about the prototype system.

Some commented on the exercises, e.g. ”Increase ex-

ercises”, ”I think the second exercise which was step

siding could be replaced to be a more effective exer-

cise”. A few users had difﬁculties with some aspects

of the app such as their pose not being recognized al-

though they mentioned they were able to resolve the

issues by moving the camera or their position relative

to the camera but this made them rate the usability

lower. We also had the following comments in terms

of the platform in general: ”It was good for me be-

cause I’m 77 years old and it’s hard for me to go to the

gym. But I needed a real chair or someone to help me

with doing exercises.” and ”Since I am a housewife

and I am usually at home, I think it is a good idea if I

can do exercises at home with this app”. In general,

we found that it was are older participants, who tend

to spend more time at home, who were more excited

about using such a system.

5 DISCUSSION AND FUTURE

WORK

The results of our work suggest that online home ex-

ercise routines for balance training may be desirable

for some users, though it may not be the ideal plat-

form for all. Speciﬁcally, we were happy to see that

our older population was excited about such a plat-

form. Although our participant pool was very tech

savvy, surprisingly 70% had not used online training

applications before, but did rate the BaART applica-

tion as easy and convenient to use.

In terms of the augmented reality visualization, al-

though our sample population did not have a signif-

icant portion of users with balance disorders, those

that had them, believed that they would require a real

object to help them for balance rather than a virtual

object. Furthermore, only 60% used the virtual ob-

ject and one person even mentioned it was distracting

from the exercises. We believe this still may be an in-

teresting avenue of future research, however, it might

require a better synchronization between verbal in-

structions and the visualization. For example, asking

the users to imagine having their hands on a real chair

while they step to the side. In future work, we plan to

ICT4AWE 2022 - 8th International Conference on Information and Communication Technologies for Ageing Well and e-Health

174

Table 1: System Usability Scale (SUS) average scores from 1 (strongly disagree) to 5 (strongly agree).

System Usability Scale Questions Average

I think that I would like to use this system frequently 3.65

I found the system unnecessarily complex 1.85

I thought the system was easy to use 4.3

I think that I would need the support of a technical person to be able to use this system 1.55

I found the various functions in this system were well integrated 4.55

I thought there was too much inconsistency in this system 1.6

I would imagine that most people would learn to use this system very quickly 4.25

I found the system very cumbersome to use 1.85

I felt very conﬁdent using the system 4.2

I needed to learn a lot of things before I could get going with this system 2.05

add more AR elements not necessarily for balancing

but to guide the exercises and add gaming elements.

One such example is an apple to pick from a tree when

raising on ones tiptoes. Adding gamiﬁcation elements

such as this may make the application more engaging.

Furthermore, a virtual trainer or avatar that does the

exercises with you and adjusts their pace to the end

user may also be interesting.

Given this was a ﬁrst prototype, we only had three

exercises and found users would be interested in hav-

ing more. In the future, we plan to extend the platform

to have more exercises and include other features as

suggested by the participants. Additional features in-

cluding: being able to either watch the tutorials again

or skip them, improving the voice commands which

some users had issues with (perhaps due to micro-

phone issues or accents) and into different languages

to accommodate non-native English speakers, enlarge

the text instructions, and send notiﬁcations to the user

to ensure they perform the exercises regularly.

Lastly, although we capture the pose and deter-

mine that the user is doing the exercise correctly in

order to count the repetition, the user does not receive

any visual or auditory feedback on the quality of the

exercise and how to improve it if they are doing it

wrong. In future work, we plan to relay this infor-

mation to the user. In general, collecting, analyzing,

and delivering performance-related data of users in

charts could motivate users further by showing them

the progress they have made.

As well as, the future directions described above it

will be important to do a longer term study with more

participants to look at not only how we can encourage

these healthy habits in the short term but change user

habits for sustained use.

6 CONCLUSION

In this paper, we describe a developed prototype for

doing balancing exercises at home based on the Na-

tional Institute of Clinical Excellence (NICE) guide-

lines for fall prevention and British Geriatric Soci-

ety (ABGS) set of clinical instructions to prevent and

evaluate older adults falls. We conducted a study with

20 users, both younger and older, and with and with-

out balance disorders. The results of our study indi-

cate that the participants liked the concept in general

and found the application easy to use. Furthermore,

they are interested in using this application, especially

if it were extended to have more exercises and fea-

tures such as notiﬁcations.

In terms of the augmented reality aspect, most

users found the virtual object did not help them to bal-

ance. Although this is the case, we posit that there are

additional possibilities to be explored for using AR in

such a system, including adding visual feedback on

the quality and correctness of the exercise, adding a

virtual coach, and adding gaming elements. The par-

ticipants of the study also felt that they do not exer-

cise as much as they like because it is time-consuming

and costly. Online ﬁtness and health applications are

helping to mitigate these issues by allowing users to

have personal trainers to work with them whenever

and wherever they are.

REFERENCES

Albaina, I. M., Visser, T., Van Der Mast, C. A., and Vas-

tenburg, M. H. (2009). Flowie: A persuasive virtual

coach to motivate elderly individuals to walk. In In-

ternational Conference on Pervasive Computing Tech-

nologies for Healthcare, pages 1–7.

Androutsou, T., Kouris, I., Anastasiou, A., Pavlopoulos,

S., Mostajeran, F., Bamiou, D.-E., Genna, G. J.,

Costafreda, S. G., and Koutsouris, D. (2020). A

smartphone application designed to engage the elderly

in home-based rehabilitation. Frontiers in Digital

Health, 2.

Argent, R., Daly, A., Caulﬁeld, B., et al. (2018). Patient in-

volvement with home-based exercise programs: can

An Online Balance Training Application using Pose Estimation and Augmented Reality

175

connected health interventions inﬂuence adherence?

JMIR mHealth and uHealth, 6(3):e8518.

Bickmore, T., Silliman, R., Nelson, K., Cheng, D., Winter,

M., Henault, L., and Paasche-Orlow, M. K. (2009). A

randomized controlled trial of an automated exercise

coach for older adults. Journal of the American Geri-

atrics Society.

Brooke, J. (1996). SUS: A “quick and dirty” usability scale.

Usability evaluation in industry, 189(3).

Felberbaum, Y., Lanir, J., and Weiss, P. L. (2018). Chal-

lenges and requirements for technology to support

mobility of older adults. In Extended Abstracts of the

2018 CHI Conference on Human Factors in Comput-

ing Systems, pages 1–6.

elix, I. B., Guerreiro, M. P., Cavaco, A., Cl

audio, A. P.,

Mendes, A., Balsa, J., Carmo, M. B., Pimenta, N.,

and Henriques, A. (2019). Development of a com-

plex intervention to improve adherence to antidiabetic

medication in older people using an anthropomorphic

virtual assistant software. Frontiers in Pharmacology,

10.

Haghbin, N. and Kersten-Oertel, M. (2021). Multimodal

cueing in gamiﬁed physiotherapy: A preliminary

study. In Proceedings of the 7th International Con-

ference on Information and Communication Technolo-

gies for Ageing Well and e-Health (ICT4AWE), pages

137–145. INSTICC, SciTePress.

Hardy, S., G

obel, S., and Steinmetz, R. (2013). Adaptable

and personalized game-based training system for fall

prevention. In Proceedings of the 21st ACM interna-

tional conference on Multimedia, pages 431–432.

Herrera, F., Ni

no, R., Montenegro-Mar

ın, C. E., Gaona-

Garc

ıa, P. A., de Mend

ıvil, I. S. M., and Crespo, R. G.

(2020). Computational method for monitoring pauses

exercises in ofﬁce workers through a vision model.

Journal of Ambient Intelligence and Humanized Com-

puting, pages 1–9.

Jorgensen, M. G., Laessoe, U., Hendriksen, C., Nielsen, O.

B. F., and Aagaard, P. (2013). Efﬁcacy of nintendo wii

training on mechanical leg muscle function and pos-

tural balance in community-dwelling older adults: a

randomized controlled trial. Journals of Gerontology

Series A: Biomedical Sciences and Medical Sciences,

68(7):845–852.

Kouris, I., Saraﬁdis, M., Androutsou, T., and Koutsouris,

D. (2018). Holobalance: an augmented reality virtual

trainer solution forbalance training and fall preven-

tion. In 2018 40th Annual International Conference

of the IEEE Engineering in Medicine and Biology So-

ciety (EMBC), pages 4233–4236. IEEE.

Ku, J., Kim, Y. J., Cho, S., Lim, T., Lee, H. S., and Kang,

Y. J. (2019). Three-dimensional augmented reality

system for balance and mobility rehabilitation in the

elderly: A randomized controlled trial. Cyberpsy-

chology, Behavior, and Social Networking, 22(2):132–

141.

Moreira, R., Teles, A., Fialho, R., Baluz, R., Santos, T. C.,

Goulart-Filho, R., Rocha, L., Silva, F. J., Gupta, N.,

Bastos, V. H., et al. (2020a). Mobile applications for

assessing human posture: A systematic literature re-

view. Electronics, 9(8):1196.

Moreira, R., Teles, A., Fialho, R., Dos Santos, T. C. P.,

Vasconcelos, S. S., de S

a, I. C., Bastos, V. H., Silva,

F., and Teixeira, S. (2020b). Can human posture

and range of motion be measured automatically by

smart mobile applications? Medical hypotheses,

142:109741.

Mostajeran, F., Katzakis, N., Ariza, O., Freiwald, J. P., and

Steinicke, F. (2019). Welcoming a holographic vir-

tual coach for balance training at home: two focus

groups with older adults. In 2019 IEEE Conference

on Virtual Reality and 3D User Interfaces (VR), pages

1465–1470. IEEE.

National Institute for Health and Care Excellence (NICE)

(2013). Falls: Assessment and prevention of falls in

older people.

Ogonowski, C., Aal, K., Vaziri, D., Rekowski, T. V., Ran-

dall, D., Schreiber, D., Wieching, R., and Wulf, V.

(2016). Ict-based fall prevention system for older

adults: qualitative results from a long-term ﬁeld study.

ACM Transactions on Computer-Human Interaction

(TOCHI), 23(5):1–33.

Panel on Prevention of Falls in Older Persons (2011). Sum-

mary of the updated american geriatrics society/british

geriatrics society clinical practice guideline for pre-

vention of falls in older persons. Journal of the Amer-

ican Geriatrics Society, 59(1):148–157.

Papandreou, G., Zhu, T., Chen, L.-C., Gidaris, S., Tompson,

J., and Murphy, K. (2018). Personlab: Person pose es-

timation and instance segmentation with a bottom-up,

part-based, geometric embedding model. In Proceed-

ings of the European conference on computer vision

(ECCV), pages 269–286.

Roy, S., Mazumder, O., Chatterjee, D., Chakravarty, K., and

Sinha, A. (2017). Quantiﬁcation of postural balance

using augmented reality based environment: A pilot

study. In 2017 IEEE SENSORS, pages 1–3. IEEE.

Society, A. G., Society, G., Of, A. A., and On Falls Preven-

tion, O. S. P. (2001). Guideline for the prevention of

falls in older persons. Journal of the American Geri-

atrics Society, 49(5):664–672.

Tsiourti, C., Joly, E., Wings, C., Moussa, M. B., and Wac,

K. (2014). Virtual assistive companions for older

adults: qualitative ﬁeld study and design implications.

In Proceedings of the 8th International Conference

on Pervasive Computing Technologies for Healthcare,

pages 57–64.

Tyagi, S., Lim, D. S., Ho, W. H., Koh, Y. Q., Cai, V., Koh,

G. C., and Legido-Quigley, H. (2018). Acceptance of

tele-rehabilitation by stroke patients: perceived barri-

ers and facilitators. Archives of physical medicine and

rehabilitation, 99(12):2472–2477.

Vonstad, E. K., Su, X., Vereijken, B., Bach, K., and Nilsen,

J. H. (2020). Comparison of a deep learning-based

pose estimation system to marker-based and kinect

systems in exergaming for balance training. Sensors,

20(23):6940.

ICT4AWE 2022 - 8th International Conference on Information and Communication Technologies for Ageing Well and e-Health

176