Exploring Different User Interfaces for Automatic Tracking of Free

Weight Exercises Using Computer Vision

unter Alce

1 a

, Axel Mulder

2 b

and Jakob H

akansson

2 c

Design Sciences, Lund University, Lund, Sweden

Sony Nordic Europe B.V, Lund, Sweden

Keywords:

Training, Health, User Interface, Tracking Free Weight.

Abstract:

Several research studies have shown the importance of physical exercise. The global gym and ﬁtness industry

faced a transition from traditional gyms to virtual ﬁtness training during the pandemic lockdowns. A prolifer-

ation of applications is available to provide different digital gym experiences. Advagym is one example that

uses sensors to track and give feedback to gym goers. Sony’s R&D Center Lund Laboratory has developed a

Camera-based Tracking System (CTS) which aims to offer an automatic free weight exercise tracking solution

with Advagym. The main goal of this paper is to align the Advagym application combined with CTS for free

weight exercise (FWE) tracking and to conduct a comparative study of four different user interfaces presenting

FWE tracking to increase the user experience. The main contribution of this paper is to elucidate knowledge

about which UI feedback of FWE given for the “gym-goer” was preferred.

1 INTRODUCTION

Physical exercise is of signiﬁcant importance for all

adults and adolescents. It can, for example, improve

their mental health, sleep, physical ability, weight

control, decrease the risk of chronic diseases and ail-

ments (Piercy et al., 2018). Therefore, it is in society’s

economic and social interests to help and encourage

people to exercise and stay healthy. A growing data-

and technology-based industry is looking into new

ways to assist with this matter.

The global gym and ﬁtness industry has seen

its market size grow by over 43% from 2009 to

2019 ($67B to $97B) (Statista, 2020). However, the

COVID-19 pandemic led to nationwide lockdowns

and social distancing regulations and norms. There-

fore, the transition from traditional gyms to virtual

ﬁtness accelerated instead. This transition has led

explicitly to increased downloads and subscriptions

of ﬁtness applications. The global ﬁtness applica-

tion market size is valued at $1.1B as of 2021 and

is expected to see a compound annual growth rate

of 17.6% from 2022 to 2030 (Grand View Research,

2022).

https://orcid.org/0000-0001-9112-2414

https://orcid.org/0000-0002-1380-4572

https://orcid.org/0000-0002-8309-561X

Different companies and entities have a clear in-

centive to support this growth and market expansion.

There are currently a vast amount of ﬁtness applica-

tions on the market that satisﬁes speciﬁc niches or

needs of the user. Such conditions encompass every-

thing from simple motivational applications to mon-

itoring, planning, and logging applications to help

users track their workouts’ progress. However, some

companies are aiming to offer the user a more “com-

plete” solution, i.e., digitizing the entire gym-going

experience. An example of such a company is Sony

Advagym

Advagym (2015) is a commercial solution already

available in the market, which aims to use Internet of

Things (IoT) devices mounted on existing gym ma-

chines to track performance data from users’ work-

outs and digitize the gym experience. The IoT sen-

sors serve to collect data, push the data, and share

the data with a whole network of connected devices.

IoT sensors are used to track the user’s performance

and movements (Gubbi et al., 2013). More recently,

Advagym added velocity-based training to their ap-

plication, which allows the user to easier keep a cer-

tain pace while lifting weights (Alce et al., 2021).

However, Advagym does not currently support auto-

matic tracking and logging of an essential aspect of

https://advagymsolutions.com/

Alce, G., Mulder, A. and Håkansson, J.

Exploring Different User Interfaces for Automatic Tracking of Free Weight Exercises Using Computer Vision.

DOI: 10.5220/0011827400003476

In Proceedings of the 9th International Conference on Information and Communication Technologies for Ageing Well and e-Health (ICT4AWE 2023), pages 143-150

ISBN: 978-989-758-645-3; ISSN: 2184-4984

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

143

the gym-going experience, i.e., the free weight exer-

cises (FWE). Recent advances in computer vision and

machine learning have presented possibilities to solve

this problem.

Sony’s R&D Center Lund Laboratory has devel-

oped a Camera-based Tracking System (CTS) that

enables three-dimensional (3D) object tracking and

can be trained to detect speciﬁc movement patterns.

Through the use of this system, Advagym and Sony’s

R&D Center Lund Laboratory are jointly exploring

implementations of an automatic free weight exercise

tracking (FWET) solution. Since the CTS itself is “in-

visible” to the user, i.e., with no means of input or in-

teraction. One can ask, how is the user supposed to

use the system?

The main goal of this paper is to align the Ad-

vagym application combined with CTS for FWE

tracking and to conduct a comparative study of four

different user interfaces (UIs) presenting FWET to in-

crease the user experience.

The main contribution of this paper is to eluci-

date knowledge about which UI of FWE given for the

“gym-goer” was preferred.

The following section presents relevant related

work. Then the Advagym system is described, fol-

lowed by the evaluation, results, discussion, and con-

clusions.

2 RELATED WORK

Previous approaches for exercise detection and track-

ing include using wearables and instrumenting equip-

ment. For example, wearables such as IMUs have

been used to track user’s exercise movements by

Seeger et al. 2012. Example of instrumenting gym

equipment to monitor its use, Velloso et al. 2013 can

be mentioned who instrumented both the users and

the equipment with IMUs and used the system to train

a novice user. However, more recently research is be-

ing done in the ﬁeld of computer vision and machine

learning. For example, there have been numerous

computer vision-based systems that have focused on

rehabilitation. Ar et al. 2014 used a depth camera to

recognize in-home physiotherapy exercises, and An-

ton et al. 2015 used a depth camera to track exercises

for telerehabilitation. More recently, systems have

employed deep learning-based approaches to track

pose from an RGB feed by Cao et al. 2017, and Mehta

et al. 2017. These results have slowly begun to be-

come commercial products available to consumers.

In 2018, Amazon launched a new physical shop-

ping concept called Amazon Go. With Amazon

Go, customers ﬁrst enter through a turnstile where

their smartphone’s screen with the Amazon smart-

phone application opened is scanned. Then, they walk

around the store and put food items into their shop-

ping cart and when they are ready they leave the store.

The total purchase amount is automatically charged

through the credit card registered with their Amazon

account. No further interaction is required of the cus-

tomer. This is done through multiple cameras uti-

lizing computer vision and deep learning algorithms

together with “sensor fusion,” i.e., different sensors

such as weight- and proximity sensors working in

symbiosis (Wankhede et al., 2018).

GymCam is a proposed solution that detects, rec-

ognizes, and tracks exercises in a gym through com-

puter vision and machine learning algorithms. The

system utilizes a single-camera solution to track mo-

tion and assumes that most repetitive movements over

time in a gym are regarded as exercises in progress,

thereby differentiating random movements from ex-

ercises (Khurana et al., 2018). Khurana et al. (2018)

claim to be able to track hundreds of users simulta-

neously while still being able to detect and track the

individual exercises and the corresponding repetitions

being carried out by the users. The system can also

handle occlusion due to its focus on detecting repeti-

tive movements in favor of accurately estimating body

key points (i.e., skeletons) as with typical multiple-

camera solutions. However, it does not focus e.g. on

the user interface ﬂow and experience.

3 ADVAGYM

Currently, users of Advagym’s system interact with it

by use of an application on their smartphones. By

implementing a smartphone-based prototype appli-

cation that can interact and communicate with Ad-

vagym’s hardware (IoT devices) and Application Pro-

gramming Interfaces (APIs), along with Sony R&D

Center Lund Laboratory’s CTS system, it will be pos-

sible to evaluate an FWET system.

To provide a complete digitization solution for the

gym experience. One would simplify it down into the

following required parts:

• Hardware: to somehow translate the physical

world into the digital, you would need devices and

sensors placed and mounted differently through-

out a gym.

• Software Platform: interpreting/parsing the data

recorded by the hardware requires different types

of software to be written. Transmitting the pro-

cessed data to where it is needed and handling

a large number of users requires some form of

ICT4AWE 2023 - 9th International Conference on Information and Communication Technologies for Ageing Well and e-Health

144

a ﬂexible software platform to build various fea-

tures/solutions.

• Client/Application: for a user to interact with the

software platform and hardware, a client/an appli-

cation of some form is needed, such as a smart-

phone application.

3.1 Sony’s Camera Tracking System

Using both machine learning and computer vision,

Sony’s R&D Center Lund Laboratory has developed

a Camera-based Tracking System (CTS) that enables

3D object-tracking, and that is trained to detect spe-

ciﬁc movement patterns.

The CTS employs a multi-camera solution to track

multiple objects within a space accurately. This

is done to minimize the issue of occlusion present

in many single-camera solutions for object track-

ing (Zhu, 2019). Multiple cameras with tempo-

rally consistent video feeds allow matching of two-

dimensional (2D) inputs, which in turn makes it pos-

sible to achieve fast individual 3D pose estimations of

all humans present in the view of the cameras (Chen

et al., 2020). The system can detect and track several

exercises, including bench presses, bicep curls, dead-

lifts, situps, and squats.

Sony’s R&D Center Lund Laboratory has set up a

gym space to test and evaluate the CTS. The gym has

many machines that Advagym uses to test its hard-

ware and software solutions. There are also several

types of free weights equipment available: barbells,

barbell weights, and dumbbells.

3.2 User Interface for Onboarding

After testing and exploring the CTS, multiple brain-

storming sessions took place to develop an overall

concept for the smartphone application. The follow-

ing constraints of the CTS were noted:

• The system does not have 100% accuracy in terms

of repetition detection.

• Users need to carry out exercises within a speciﬁc

gym area, covered by the cameras used.

• In certain use cases, the system can lose track of a

user even though they are within the tracked area.

• Only the exercises the system can detect are

tracked and logged, i.e., there is no possibility of

custom exercises unless the system is re-trained.

From these constraints, it was determined that

three main areas needed to be taken into account when

devising the initial concept:

• Connection: How would a user connect to the

tracking system?

• User Interface Flow: What steps would the user

need to take to interact and use the system?

• Feedback: What modes of feedback would need

to be in place for the user to be able to respond to

errors?

3.2.1 Connection

For a user to connect to the FWET system and to start

using it, one would somehow need to bind (link) the

“skeleton” shown in the visualization of the CTS with

the user. Three different UIs regarding how a user

would bind were proposed:

1. Gesture: if the user performs a speciﬁc gesture

such as holding both arms above their head. The

system can see which skeleton is making that ex-

act gesture and bind the user.

2. Waypoint: if the user presses a connect button

on their smartphone application while standing at

a speciﬁc location in the tracking area. The sys-

tem would know that the skeleton at that speciﬁc

location, at that very instant, is the user.

3. Puck: works identically to the Waypoint method,

but the user taps a Puck with their phone instead of

pressing the connect button on their smartphone

app.

The prototype’s goal is to be integrated with the cur-

rent Advagym application. Therefore, we used the

current version as a baseline for the user interface

ﬂow.

3.2.2 User Interface Flow

The User Interface Flow (UIF) was created in a digital

Lo-Fi version. Due to the COVID-19 pandemic and

its subsequent restrictions and regulations in Sweden,

it was impossible to meet with potential usability test

participants in person. Therefore, the goal of the digi-

tal Lo-Fi prototype was to create a digital smartphone

prototype that test participants could use and interact

with over the internet. Since three different connec-

tion methods were proposed, three distinct prototypes

were evaluated, each with varying connection meth-

ods.

The UI was inspired by the current Advagym ap-

plication. The different screens constructed were in-

teractable in terms of being able to click/tap on spe-

ciﬁc UI components, scroll through list views, and

move and navigate back and forth throughout the ap-

plication. The screens and screen transitions were

Exploring Different User Interfaces for Automatic Tracking of Free Weight Exercises Using Computer Vision

145

also animated to resemble the current Advagym ap-

plication closely.

The FWET UI was integrated with the Advagym

UIF. The Gesture connection method implies that the

user performs a speciﬁc gesture, which the CTS can

detect and link the tracked skeleton with the user per-

forming the gesture.

The second connection prototype was referred to

as Waypoint and implies that the user walks over to

a speciﬁc location in the gym that is clearly marked

with the help of signage, paint, or some other material

or objects. Since the CTS has the exact X, Y, and

Z coordinates of the Waypoint stored in its system,

the CTS can determine that the skeleton is inside the

speciﬁed Waypoint boundary and can be bound to the

speciﬁc user.

The third connection prototype was referred to as

Puck and implies that the user taps a speciﬁc Puck in

the gym with their phone. To differentiate the FWET

Puck from the other Pucks which are placed on the

gym machines, there should be additional signage ac-

companying the free weight connection Puck or af-

fording the Puck a different color signifying the dif-

ferent use case.

3.2.3 Lo-Fi Usability Test of the User Interface

Flow

The goal of the usability test was to evaluate which

of the three proposed connection methods showed the

most promise regarding quantitative and subjective

metrics.

Setup. Due to the COVID-19 pandemic and its re-

strictions and regulations in Sweden, all test session

parts were conducted online using Google Meet.

Participants. Twelve participants (three females,

nine males) conducted the online test. The average

age of the participants was M = 27.0 and ranged from

25 to 30 years.

Procedure. At the start of the test session, the par-

ticipants were asked to imagine being in a gym envi-

ronment and to place a smartphone in front of them.

Depending on what prototype was used, a further

description of the gym area and what objects/things

were available was presented. When the participants

were ready, they were asked to perform speciﬁc tasks

using one of the proposed Lo-Fi prototypes, 1) Ges-

ture, 2) Waypoint, and 3) Puck. The order in which

the prototypes were presented to the test participant

was counterbalanced. After all test scenarios had

been completed, a semi-structured interview was con-

ducted, and the participants were asked to rank the

prototypes. Each session lasted about 30 min.

Results. All twelve participants managed to ac-

complish the usability test. A summary of the pre-

Table 1: Preferred prototypes.

Prototype 1st place 2nd place 3rd place

Gesture 3 1 3

Waypoint 4 1 2

Puck 5 4 0

ferred prototype can be found in Table 1. The semi-

structured interviews found that the UIF of start-

ing an exercise was not straightforward; there was

more than one way of getting there. Many non-

interactable/functional user interface elements and

components made navigation confusing.

3.3 Hi-Fi Prototype

Based on the results found during the Lo-Fi proto-

type, it was decided not to proceed with the Gesture

connection and only focus on Waypoint and Puck.

Moreover, it was decided to focus on creating pure

free weight training rather than getting stuck with in-

tegration to Advagym. However, still, the same color

scheme and fonts were used. It was decided to de-

velop the application in React Native

, since it allows

cross-platform development.

3.3.1 State Handling and Application Settings

For the application to understand what information to

display to the user or what feedback to provide in re-

sponse to system errors or actions taken by the user.

The application would need to handle the following

states: 1) LoggedIn: is the user logged in or not? 2)

UserID: what is the logged-in user ID? 3) UserType:

is the user an admin/tester? 4) Connection: this state

consists of the following child-states: - Connecting:

is the user attempting a connection? - Connected: is

the user connected to the CTS? 5) Error: an error of

some kind occurred.

The state handling was implemented through the

use of React Redux

. Redux allows you to read and be

notiﬁed of any changes made to the state(s) through

the use of “Hooks” from anywhere inside the appli-

cation. This was used to update the appearance or

functionality of components or the layout of differ-

ent application parts. The states were stored either as

boolean, string, or number types.

Some data needed to be stored in memory that the

application could read and write to. Redux allows for

a “store” which can save and handle data of different

kinds. This was used extensively throughout the ap-

plication. Some of the data stored were as follows:

1) skeletonID: the user’s current id according to the

https://reactnative.dev/

https://react-redux.js.org/

ICT4AWE 2023 - 9th International Conference on Information and Communication Technologies for Ageing Well and e-Health

146

CTS (saved as a string). 2) latestMessage: the lat-

est MQTT message received from the CTS (saved as

a string). 3) repNumber: the current number of rep-

etitions in the set (saved as a number). 4) exercises:

an array of all exercises performed by the user, and

the accompanying data of each exercise, including the

number of sets, repetitions, and weight lifted.

3.4 CTS Connection

For the user to connect and interact with the CTS, the

prototype smartphone application needs to be able to:

1) Connect and listen to the CTS broker’s modiﬁed

topic and parse its MQTT messages. 2) Know when

to listen for certain MQTT message types. 3) Know

what Skeleton ID(s) are within the Waypoint bound-

aries in the tracked area.

This was the key part of the solution for connect-

ing and using the CTS. For the CTS broker to send out

the closest MQTT message type, it needs to know two

things: 1) The X, Y, and Z coordinates of the Way-

point. 2) Detect a skeleton (tracked human) within a

certain distance of the Waypoint, i.e., within the Way-

point boundary.

The Waypoint location could be anywhere as long

as it was inside the CTS’s tracked area. To deﬁne a

Waypoint, the following was done: 1) Make sure the

tracked area has no other humans in it. 2) Stand at

the exact location of where you would like the Way-

point to be. 3) Record the X, Y, and Z coordinates of

the Skeleton IDs present in the scene reported by the

CTS. 4) The recorded X, Y, and Z coordinates of the

single, stationary “Skeleton” is your Waypoint loca-

tion.

To determine that a skeleton is within the Way-

point boundary, the formula for calculating the dis-

tance, d between two points P

= (x

, y

, z

) and P

, y

, z

) in 3D-space (xyz-space) was used (Equa-

tion 1):

d =

− x

)

+ (y

− y

)

+ (z

− z

)

(1)

If the distance d between the Waypoint and a skeleton

was calculated as less than 0.5m a closest message

was sent out by the CTS broker containing the Skele-

ton ID of the tracked/detected skeleton.

After the development and implementation had

concluded, a fully functional Hi-Fi prototype smart-

phone application was developed.

Four different connection states of the application

make the top of the Exercise screen change its ap-

pearance (Figure 1). Additionally, two different feed-

back alternatives were added when the user was suc-

cessfully connected to the CTS, 1) tactile feedback in

the form of vibration or 2) auditory feedback in the

Figure 1: The Hi-Fi prototype’s four connection states.

form of a human voice stating “Connection success-

ful!”. When the user starts to perform the exercise,

the application changes the user interface and shows

a counter counting the number of completed repeti-

tions.

Four different prototypes were developed:1) Way-

point with auditory feedback, 2) Waypoint with tac-

tile feedback, 3) Puck with auditory feedback, and 4)

Puck with tactile feedback.

4 EVALUATION

A user study was conducted to evaluate the different

prototypes by letting all participants perform tasks in

the ofﬁce gym.

4.1 Setup

The evaluation was conducted in Sony’s ofﬁce gym.

The gym was already equipped with Advagym tech-

nology. It also has a wide range of gym machines,

along with free weights such as barbells, dumbbells,

and different benches. Throughout the gym and along

the ceiling, small cameras were placed that created the

tracked area of the CTS. Both quantitative and qual-

itative data were collected. The active session was

documented through video recordings from a Sony

smartphone as an overview camera. Participants who

use an Android phone daily conducted the test with

a Google Pixel 2 XL, and those who use iOS tested

with an iPhone 13 Pro. An orange Puck was placed

in the middle of the room, adhered to a larger gym-

machine assembly. On the ﬂoor beneath the Puck, a

large circle was outlined with the help of tape. The

circle functioned as the Waypoint area.

4.2 Participants

Advagym is an application with a very broad user

group that consists of both young and elderly users.

It can be beginners as well as elite trainers. The one

thing they have in common is that they are training

at a gym. However, we had some restrictions, such

Exploring Different User Interfaces for Automatic Tracking of Free Weight Exercises Using Computer Vision

147

as being able to perform the test physically, i.e., the

participant should be able to perform squats exercise

without any weight and without any pain.

An online questionnaire using Google Forms was

used to recruit participants more easily. The online

questionnaire served two purposes; to gather relevant

demographic data about the participant and book an

available time slot. The sign-up questionnaire was

spread through different channels, both in digital and

physical form. The physical form consisted of a

poster with QR-code links that were placed in several

crowded areas within the campus of Lund University.

The digital form consisted of a link and an accompa-

nying description of the test and its purpose, that was

distributed through Sony’s social media groups.

In total, 32 participants (17 female, 15 male) were

recruited. The age of the participants ranged from 19

to 60 years (M = 31.1, SD = 10.42). To estimate and

grade the training skill of the participants, a sequence

of calculations was made based on their sign-up ques-

tionnaire answers. The following parameters: weekly

training frequency and time kept with current training

frequency. This was graded on a scale of one to ﬁve,

where the interval of one to three was graded as be-

ginner/novice training skill and 4 to 5 was graded as

advanced training skill. 32 test participants meant that

the participants would be evenly divided between the

four Hi-Fi prototypes, meaning each prototype would

be evaluated eight times.

4.3 Procedure

The test session was divided into three parts: The

preparation stage (Pre Test), the Test session, and

Post-test. The Preparation stage includes the initial

contact with the participants. Participants signed up

for the test with the help of an online questionnaire.

The COVID-19 pandemic regulations and restrictions

had been removed in Sweden by the time of the eval-

uation of the Hi-Fi prototype. Therefore, all test par-

ticipants could conduct the evaluation in person at

Sony’s ofﬁce gym. Participants were welcomed and

escorted to the ofﬁce gym when they arrived at the

test location. In the ofﬁce gym, they were given a

brief introduction to what Advagym is and the pur-

pose of the evaluation. Moreover, they signed a Non-

Disclosure Agreement (NDA) and an informed con-

sent form. Next, they were asked to start with the

test scenario, which included seven tasks: 1) Carry

out two sets of Squats, ﬁve repetitions each, with

the help of the application and the automatic tracking

feature, 2) Go over to a table outside of the tracked

area, to get some water and then return to the exer-

cise area, 3) Carry out one set, ten repetitions of Bi-

ceps curls, with the help of the application and the

automatic tracking feature, 4) Carry out two sets, ﬁve

repetitions of Bench Press with the help of the ap-

plication and the automatic tracking feature, 5) Carry

out a couple of Biceps curl repetitions but do it with-

out the application automatically tracking the repeti-

tions/exercise, 6) Carry out a couple of additional Bi-

ceps curl repetitions with the help of the application

and the automatic tracking feature and ﬁnally 7) Fin-

ish/end the workout. All tasks were designed to un-

derstand whether the participant will know how to use

the application and whether they are being tracked.

After all tasks had been completed, the test partici-

pants were asked to ﬁll out a SUS questionnaire re-

garding that speciﬁc prototype, followed by a semi-

structured interview including questions about if they

have used any gym application before and their ini-

tial thoughts regarding the FWE system. Each session

lasted about 45 minutes, and the participant was given

a movie ticket as a reward. The whole procedure of

the test session is visualized in block diagrams.

4.4 Results

In the following section, the results from the SUS

scale and the structured interview are presented. All

32 participants managed to accomplish the exercises.

We used an alpha level of 0.05 for all statistical tests.

4.4.1 SUS Score

The results obtained from the SUS questionnaire for

the Puck-NoSound present a mean score of M =

81.3, SD = 4.23, with a minimum score of 75.0 and

a maximum score of 87.5. For the Puck-Sound,

a mean score of M = 73.8, SD = 11.10, with a

minimum score of 50.0 and a maximum score of

87.5. For the Waypoint-NoSound, a mean score

of M = 72.8, SD = 9.86, with a minimum score

of 60.0 and a maximum score of 90.0. For the

Waypoint-Sound, a mean score of M = 70.6, SD =

11.78 with a minimum score of 52.5 and a maxi-

mum score of 90.0. A one-way ANOVA for depen-

dent measures between Puck-NoSound, Puck-Sound,

Waypoint-NoSound and Waypoint-Sound showed no

signiﬁcant relation: F(3,28) = 1.81, p = .169.

4.4.2 Semi-Structured Interview

During the semi-structured interview, 15 out of 32 test

participants said that they would like to use a system

such as the one proposed during the test. 10 out of 32

test participants said they would like to use a system

like the one presented during the evaluation if some

changes were made, such as more stable tracking, if

ICT4AWE 2023 - 9th International Conference on Information and Communication Technologies for Ageing Well and e-Health

148

more exercises were added, and if they had more in-

terest in tracking their exercises. Seven out of 32 test

participants said they would not like to use a system

like the one proposed during the test. However, 25 of

32 were positive in regards to using an automatic free

weight exercise tracking solution.

The test participants provided a lot of feedback

regarding the prototypes’ features/functionality and

user interface. Examples of such feedback include:

- It was not clear that you could step out of the “Way-

point” when performing exercises. - It was not clear

where and what the “Waypoint” was supposed to be

in the gym. - It was not clear if you had to bring your

smartphone with you in order to be tracked by the sys-

tem. - It was not clear where the orange “Puck” was

in the gym. - The solution needed to be even more

“automatic,” i.e., less interaction would be required

by its users. - The repetition tracking was sometimes

not accurate, i.e., the system missed some repetitions,

and at other times the tracking system did not work.

- The ability for the application to “guide” the user

in the sense of offering programs with different exer-

cises that the application would then “ﬁll in” as the

user was performing the different sets and repetitions.

- In the case of the “Puck” prototype, when asked

to reconnect to the system after having left the free

weight training area to get a drink of water, it was not

clear which Puck that the test participants needed to

tap with their phone to reconnect. Test participants

(correctly) observed that tapping whichever Puck was

available to them in the gym, such as Pucks found on

gym machines, did not initiate an automatic FWET

connection attempt. Some form of animation of the

illustrations when it came to instructing the user on

how to connect to the system would help with under-

standing.

5 DISCUSSION

In this section, we will discuss the “takeaways” of the

evaluation and the limitations of the prototypes.

5.1 SUS Score

There was no signiﬁcant difference regarding the SUS

score. All four had a SUS score above 68, which is

considered above average (Brooke, 2014). The Puck-

NoSound had a mean slightly larger than the other

ones. The SUS score measures cognitive attributes

such as learnability and perceived ease of use. The

result indicates that all four UIs are considered to be

easy to use and easy to learn.

Both Puck prototypes had the highest mean SUS

score. People, in general, are already quite experi-

enced with tapping on things in different contexts,

such as when tapping one’s credit card or smartphone

using Apple/Google pay to a register when paying at

a store or when tapping one’s transit card when us-

ing public transport. This perhaps made this form

of interaction feel more intuitive and natural to the

user. Building upon already-established interactions

could be helpful when users need to interact with a

new piece of technology.

5.2 Semi-Structured Interview

One of the main reasons for not being interested in the

FWET system is the need for more stable tracking and

more exercises. The CTS’s human tracking and exer-

cise detection do not have 100% accuracy. When the

CTS did not detect repetitions, the repetition counter

displayed on the prototype or communicated to the

user by way of audio was not incremented. One of

the usability test tasks instructed the user to go to the

far corner of the gym and do sets of Bench Press ex-

ercises. This activity was speciﬁcally chosen since

prior knowledge of the CTS in the gym had shown

that the far corner combined with the Bench Press ex-

ercise led to the system losing track of the user and,

therefore, not registering their repetitions. The non-

registered repetitions and loss of tracking frustrated

several test participants. This might have led them to

believe that the prototype and system were difﬁcult

to use and complex, which was different from the in-

tended effect.

Adding more feedback in the form of auditory and

haptic feedback in the prototypes did not correlate to

higher mean SUS scores. One possibility could be

that the modes of feedback needed higher affordance

or be sufﬁciently explained/put into context for the

user. On the other hand, several test participants that

used the prototypes without the additional feedback

explicitly asked for the addition of such forms of feed-

back during their Post Test short interviews, which in-

dicated that such additions would be positive.

It was also clear that the small outlined circle

on the ﬂoor in the gym, representing the Waypoint,

needed more discoverability and could be afforded

more mapping to free weights. This could be done

with additional signage or ensuring that the exercise

screen’s illustration was mapped to the circle on the

ﬂoor. A lot of test participants did not ﬁnd the Way-

point and some participants also chose to stand within

the circle on the ﬂoor during their exercises. The pro-

totype needed to sufﬁciently explain to the user that

they were free to move around the gym area when

performing exercises. Several users suggested that it

Exploring Different User Interfaces for Automatic Tracking of Free Weight Exercises Using Computer Vision

149

would be a good idea to have a more thorough and

comprehensive tutorial on how the system functioned

before using it for the ﬁrst time.

6 CONCLUSIONS

The ﬁndings presented in this paper expand the

knowledge base of HCI research in the context of us-

ing a mobile application to support free weight exer-

cise tracking. The result of the prototypes has been

very impressive overall. All of the prototypes have

performed very well regarding the usability scores.

All of the prototypes were above the average score

of 68 for the SUS-based questionnaire, which indi-

cates that the proposed user interfaces are easy to un-

derstand and use. The majority of participants would

also prefer to use one of the proposed prototypes in

their daily training with free weight exercises. This is

a good indication that the feature itself is interesting

for users.

ACKNOWLEDGEMENTS

Advagym team for supporting this research and the

participants.

REFERENCES

Advagym (2015). Advagym - for a connected gym experi-

ence. http://advagymsolutions.com.

Alce, G., H

akansson, J., and Espinoza, A. (2021). Explor-

ing different user interfaces for velocity based training

using smart gym machines: Pilot study. In ICT4AWE,

pages 113–120.

Ant

on, D., Goni, A., and Illarramendi, A. (2015). Exercise

recognition for kinect-based telerehabilitation. Meth-

ods of information in medicine, 54(02):145–155.

Ar, I. and Akgul, Y. S. (2014). A computerized recognition

system for the home-based physiotherapy exercises

using an rgbd camera. IEEE Transactions on Neural

Systems and Rehabilitation Engineering, 22(6):1160–

1171.

Brooke, J. (2014). Sus—a quick and dirty usability

scale. 1996. URL: http://cui. unige. ch/isi/icle-

wiki/ media/ipm: test-suschapt. pdf [accessed 2015-

10-14][WebCite Cache ID 6cGs6wGeu].

Cao, Z., Simon, T., Wei, S.-E., and Sheikh, Y. (2017). Real-

time multi-person 2d pose estimation using part afﬁn-

ity ﬁelds. In Proceedings of the IEEE conference on

computer vision and pattern recognition, pages 7291–

7299.

Chen, L., Ai, H., Chen, R., Zhuang, Z., and Liu, S. (2020).

Cross-view tracking for multi-human 3d pose estima-

tion at over 100 fps. In Proceedings of the IEEE/CVF

conference on computer vision and pattern recogni-

tion, pages 3279–3288.

Gubbi, J., Buyya, R., Marusic, S., and Palaniswami, M.

(2013). Internet of things (iot): A vision, architec-

tural elements, and future directions. Future genera-

tion computer systems, 29(7):1645–1660.

Khurana, R., Ahuja, K., Yu, Z., Mankoff, J., Harrison, C.,

and Goel, M. (2018). ”gymcam: Detecting, recog-

nizing and tracking simultaneous exercises in uncon-

strained scenes”. Proc. ACM Interact. Mob. Wearable

Ubiquitous Technol., 2(4).

Mehta, D., Sridhar, S., Sotnychenko, O., Rhodin, H.,

Shaﬁei, M., Seidel, H.-P., Xu, W., Casas, D., and

Theobalt, C. (2017). Vnect: Real-time 3d human pose

estimation with a single rgb camera. Acm transactions

on graphics (tog), 36(4):1–14.

Piercy, K. L., Troiano, R. P., Ballard, R. M., Carlson, S. A.,

Fulton, J. E., Galuska, D. A., George, S. M., and Ol-

son, R. D. (2018). The physical activity guidelines for

americans. JAMA, 320(19).

Research, G. V. (2022). Fitness app market size, share &

trends analysis report by type (exercise & weight loss,

diet & nutrition, activity tracking), by platform (an-

droid, ios), by device, by region, and segment fore-

casts. https://www.grandviewresearch.com/industry-

analysis/ﬁtness-app-market [Accessed: 2022-04-19].

Seeger, C., Buchmann, A., and Van Laerhoven, K. (2012).

Adaptive gym exercise counting for myhealthassis-

tant. In 6th International ICST Conference on Body

Area Networks.

Statista (2020). Market size of the global

health club industry from 2009 to 2019.

https://www.statista.com/statistics/275035/global-

market-size-of-the-health-club-industry/ [Accessed:

2022-04-19].

Velloso, E., Bulling, A., and Gellersen, H. (2013). Mo-

tionma: motion modelling and analysis by demonstra-

tion. In Proceedings of the SIGCHI Conference on

Human Factors in Computing Systems, pages 1309–

1318.

Wankhede, K., Wukkadada, B., and Nadar, V. (2018).

Just walk-out technology and its challenges: A case

of amazon go. In 2018 International Conference

on Inventive Research in Computing Applications

(ICIRCA), pages 254–257. IEEE.

Zhu, C. (2019). Multi-camera people detection and track-

ing.

ICT4AWE 2023 - 9th International Conference on Information and Communication Technologies for Ageing Well and e-Health

150