Physiological Computing Gaming

Use of Electrocardiogram as an Input for Video Gaming

Adam Chęć

1,2

, Dominika Olczak

1,2

, Tiago Fernandes

and Hugo Alexandre Ferreira

Lodz University of Technology, Łódź, Poland

Faculty of Science, University of Lisbon, Lisbon, Portugal

Keywords: Electrocardiogram, Physiological Computing, Video Gaming, Electrolycra, Bitalino.

Abstract: There are several ways of creating a human-computer interaction (HCI). One of those is physiological

computing (PC) i.e. the use of body signals as a real-time input to control a user interface. In this paper one

describes the development of a new solution in which electrocardiography (ECG) signals are used as input

for video gaming. The solution includes: a tailored belt with conductive textiles as ECG electrodes; a

specialized data acquisition board (Bitalino); signal processing algorithms implemented in Python for signal

filtering, QRS complex detection and heart rate calculation; and use of Unity 3D, a game development

engine, in which the heart rate is used as an input of a proof-of-concept PC video game – FlappyHeartPC.

With this project we conclude that nowadays it is possible to build tools that can make the bridge between

the machine and the human body in order to respond to innovations required in the gaming business.

1 INTRODUCTION

Innovative technologies are increasing their presence

in every field of living. Spending free time in front

of a computer or a television is far more common

than several years ago and has become an essential

part of modern lifestyle. Of particular relevance in

this context is physiological computing (PC) which

has enormous potential to innovate human–computer

interaction (HCI) by extending the communication

bandwidth between humans and computers, enabling

the development of “smart” technologies (Hettinger,

2003). Here, HCI is achieved by analyzing and

responding to covert psychophysiological activity

from the user in real-time (Fairclough, 2009).

Probably one of the most common biosignals that

can be used as an input for PC is the

Electrocardiogram, which is commonly used to

access the overall cardiac condition. In new-born

infants, the resting heart rate is commonly 120 beats

per minute (bpm) or higher, and it declines with age

to 72-80 bpm in young adult females and 64-72 bpm

in young adult males. Nonetheless, heart rate

changes during physical activity, fatigue or when

experiencing emotions. It can be also changed due to

cardiovascular diseases (Ostchega, 2011).

Factors

that raise the heart rate are called positive

chronotropic agents, and factors that lower it are

known as negative chronotropic agents. Although

the nervous system does not initiate the heartbeat, it

can modulate its force and rhythm (Goldberger,

1991). Some neurons in cardiac centers have a

cardiostimulatory effect transmitting signals to the

heart via the sympathetic pathway, others

communicated to the heart via the vagus nerves

(Pokrovskii, 2005).

Sensory and emotional stimuli can then act on

the heart rate via the cerebral cortex, limbic system,

and hypothalamus. Hence, heart rate can rapidly

change during “fight or fly” situations or during

workout, and is influenced by emotions, such as love

or anger (Porges, 1991; Wiltbank, 2008).

During workout, proprioceptors in the muscles

and chemoreceptors (De Burgh Daly, 1958; Hutton,

1992) transmit signals to the cardiac centers,

translating muscle activity; thus sympathetic output

from the cardiac centers increases cardiac output to

meet the expected demands. Endurance athletes can

have resting heart rates as low as 40 to 60 bpm, even

though, due to the higher stroke volumes, their

resting cardiac output is about the same as that of an

untrained person. Such athletes have a bigger

cardiac reserve, so they can tolerate more exertion

than a sedentary person can (Sandercock, 2005).

Electrocardiography (ECG) measurements can

be divided into “fixed-on-body” measurements with

157

Ch˛e

c A., Olczak D., Fernandes T. and Ferreira H..

Physiological Computing Gaming - Use of Electrocardiogram as an Input for Video Gaming.

DOI: 10.5220/0005244401570163

In Proceedings of the 2nd International Conference on Physiological Computing Systems (PhyCS-2015), pages 157-163

ISBN: 978-989-758-085-7

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

silver/silver chloride (Ag/AgCl) electrodes, and

fixed-in-the-environment electrodes. Although

fixed-on-body electrodes are reliable and give good

signal quality, fixed-in-the-environment electrodes’

nature makes them an attractive option for daily

monitoring (Lim, 2006).

Advances in computation and integrated circuits

capabilities allow videogames to be more

emotionally engaging. Still, at the moment, these

interactions are based purely on the input the player

consciously decides to use in the game world (i.e.

actions executed through the game controller).

However, there are unseen physiological responses

(e.g. heart rate variations) taking place within the

player's body as well as various behavioral responses

(e.g. gestures, facial expressions, body postures).

Such responses are useful in identifying the current

emotional state of the player. The form of gameplay

where this information is collected is commonly

referred to as affective gaming, in which the player's

emotional state is used to influence gameplay.

From the perspective of emotion recognition

technology, affective gaming is related to

biofeedback systems, where people ‘learn’ how to

control physiological activity such as heart rate,

muscular action, or brain waves by being provided

with real-time graphical representations of their

biometric state. The main goal of this work was to

test the usability of an ECG signal as an input for

gaming. For that purpose a game called

FlappyHeartPC was developed.

2 MATERIALS AND METHODS

A. Belt Design

Bearing in mind experiences of other researchers

(Holley, 1990; Milis, 1979; Misczynski, 2005;

Muhlsteff, 2004), we decided to use

electroconductive textiles instead of conventional

Ag/AgCl electrodes. Textile electrodes are a new

potential choice for biomedical measurements and

their medical usage was proven during motion

analysis with

great success (Lourenço, 2012; Pola,

2007). Using electro-textile electrodes as the

interface between the sensor and the skin, it allows

the improvement of signal acquisition methods for

ECG biometrics, targeting wearable, and

unobtrusive and continuous applications (Silva,

2011). This electroconductive textile is a highly

conductive lycra made from silver wires woven with

nylon (MedTex

P-130). Up to date, there is no

proof that electrolycra may cause any skin irritation.

The prototype belt is also composed of: elastic

rubber-textile material (allows the belt to adhere

strictly to the body, which is the condition for an

accurate measurement), non-elastic knitted fabric

(assures the same position of the electrodes) and

Velcro® (fixing to the body and locking).

Thanks to its construction, the belt can be used

by a widely range of people, maintaining the same

position during breathing or workout (Figure 1).

B. Signal Acquisition – Bitalino

Board

Inspired by the Maker and DIY movements we

choose not to use commercially available Bluetooth

Figure 1: A. Experimental setting of the belt position, with one electrode at the sternum and the other two in the two sides

of the heart, with the belt connected to Bitalino®. B. Schematic of the belt positioning. The yellow dot represents the

“ground” electrode, and black dots correspond to the positive electrode (placed at the 4

intercostal space near the sternum)

and the negative electrode (placed at the 4

intercostal space midclavicular line). The green rectangle corresponds to non-

elastic knitted fabric; the remaining black strips correspond to rubber-like elastic material.

PhyCS2015-2ndInternationalConferenceonPhysiologicalComputingSystems

158

Figure 2: Electrical circuit used in Bitalino for ECG measurement.

ECG medical hardware. Therefore, for acquiring the

ECG signals we used the Bitalino

, a low-cost

platform, targeted at multimodal biosignal

acquisition, that can interface with other devices, or

perform rapid prototyping of end-user

applications/in the field of PC (Guerreiro, 2013). Its

electrical system is represented in Figure 2. The

board includes the electrical ECG circuit represented

in Figure 2. Since this was a proof-of-concept study

we used the conjugation of the electrolycra belt with

the Bitalino

to measure the ECG of only two

subjects (subject 1 – Male, 22 years old, 85 kg;

subject 2 –Male, 22 years old, 73kg). All signals

were acquired with a sampling rate 1 kHz. During

testing of the electrolycra belt, ECG measurements

were also done using conventional Ag/AgCl

electrodes for qualitative performance comparison.

C. Algorithm Explanation

Software for signal acquisition and processing was

developed in Python language (Enthought Canopy

Distribution), which includes numerous already

written packages designed for numerical data

analysis, signal processing or graphics design. There

are several possible algorithms for detecting the

QRS complex (Afonso, 1993) that could be used

with this platform. After testing with

different

algorithms we chose the following approach:

First, the signals were recorded and opened by

Python software, in which each 10 s of the data were

processed and analyzed. The DC component and

power-line interference was removed using a Notch

filter at 50Hz. Then, the signal was filtered by a

series of Finite Impulse Response (FIR) filters.

These filters require more memory than Infinite

Impulse Response (IIR) filters, but have no feedback

elements, which makes them stable.

Subsequently, a lowpass filter (cutoff 20 Hz,

Blackman & Harris window, 61 order), followed by

a highpass filter (cutoff 15 Hz, Blackman & Harris

window, 61 order) (Pan, 1985) were applied. Digital

filters frequency response is presented in Figure 3

The next step of the algorithm was ‘thresholding’

the unwanted part of the rectified signal. The

threshold is set up to 30% of the maximum

magnitude of the 10 s piece of the signal (Christov,

2004), which removes unwanted components of the

signal. Before detection of the peaks, an additional

smoothing with a lowpass filter with cutoff

frequency equal to 2 Hz and order 100 was done.

The last step of the signal processing program is

peak detection. In this work two methods for peak

detection were studied: differentiation and wavelet-

based peak detector from the scipy.signal python

library (Du, 2006). The general approach of the

latter method is to perform a continuous wavelet

transform on data vector. In this work the Ricker

wavelet, also known as Mexican Hat wavelet was

used with an array of integers range from 70 to 100

as widths. Regarding the differentiation method, a

function that generates a nonzero array, detects the

difference between consecutive array values, and

adds that position to the array if the sign changed

from positive to negative, was implemented. This

method easily detects the local maxima of the signal

(Afonso, 1993).

The accuracy and the computational performance

of peak detection algorithms was then evaluated and

compared using two 3-minute recordings for each

subject.

Finally, computing a heart rate was simply

measuring the time between each peak and

converting it to bpm. The heart rate value was then

used as a game input.

PhysiologicalComputingGaming-UseofElectrocardiogramasanInputforVideoGaming

159

Figure 3: Designed filters: lowpass filter – cutoff 20 Hz (left) and highpass filter (right) – cutoff 15 Hz.

D. Unity – 3D Game Engine and PC

Game Development –

FlappyHeartPC

Unity 3D is a game development ecosystem, a

rendering engine integrated with a set of intuitive

tools and workflows to create interactive 3D and 2D

content. Additionally, it offers the possibility of

physical simulations, working upon mechanical

laws.

For the proof-of-concept of this work we decided

to design a PC game named FlappyHeartPC in

which the heart rate is translated into the speed of

the heart game character (Figure 4). The inspiration

for this game was the FlappyBird™ game.

Figure 4: Screenshot of the FlappyHeartPC game.

Background and rocks cc0 licence from Kenney Vleugels.

Thanks to the low complexity of the game, we

were able to focus more on the implementation of

the PC interface and playability. Due to

incompatibility of Python language with Unity 3D

(which supports C# and JavaScript) the output of the

processing algorithm was in the form of a .txt file,

which can be imported by the game engine. In order

to enhance the PC aspect of the game, we assumed

that the players’ heart rate was between 50-100 bpm

(Ostchega, 2011). The mass of the game character

was set up as 1 [unit of mass in Unity 3D space] and

gravity as 1 [unit of gravity in Unity 3D space].

Then the player’s heart rate was assigned to the

following thresholds; determining the speed

(Units/Frame) of the game character, see Table 1.

Table 1: Player’s heart rate and following thresholds for

the Units/Frame.

Hearth rate <50 50<60 60<70 70<80

Units/Frame 0.5 1 2 3

Hearth rate 80<90 90<100 =>100

Units/Frame 4 5 6

The goal of the game is to survive as long as it is

possible, avoiding collision with obstacles. The gaps

between obstacles were chosen randomly. Here, the

higher the heart rate is, the faster the game character

moves. This makes the game more difficult and

therefore it induces the player to maintain a low

heart rate level to increase the chances of survival.

3 RESULTS

Figure 5 show ECG signals recorded using

conventional Ag/AgCl electrodes and using the

electrolycra electrodes. Although electrolycra

signals seem to have smaller signal-to-noise ratio

than the ones obtained using conventional

electrodes, the QRS complex can be easily

identified. Figure 6 shows that after signal

processing, the QRS complex detection is achieved

using electrolycra signals. Both wavelet and

PhyCS2015-2ndInternationalConferenceonPhysiologicalComputingSystems

160

Figure 5: ECG signal recorded by Ag/AgCl electrodes (left) and the belt with electrolycra electrodes (right).

Figure 6: ECG signal after filtering (left) and final example of QRS complex detection (right).

differentiation based peak detection algorithms were

tested: the wavelets algorithm obtained 91,4%,

accuracy and 95,9 % precision. The differentiation

algorithm, on the other hand, obtained 90,6% of

accuracy and 90,6 % precision but was

computationally faster (10 s of ECG data processed

in 1.2 s vs 7.8 s using the wavelets-based method).

Therefore, the differentiation algorithm was chosen

for the final version of the game.

4 DISCUSSION

A. Belt Design

One of the most important challenges in belt design

was to make it suitable for several people and

minimize the noise during signal acquisition. The

first one was solved using elastic rubber-like

material with Velcro®. The second challenge

required using electroconductive thread, instead of

traditional cotton one, for connecting the

electrolycra electrodes to the Bitalino® cables. Then

the belt, as a first implementation, showed that with

low cost material it is possible to obtain suitable

ECG signals.

B. Game Design

Due to being just a proof-of-concept study we

decided to design the game as simple as possible.

Still the FlappyHeartPC game plays well its role as

an implementation of physiological signal input into

a game environment.

However, it has to be mentioned that people with

tachycardia (increased heart rate) or with

cardiovascular diseases (such as arrhythmia) may

have difficulties playing the game due to harder

peak detection.

According to subject 1, FlappyHeartPC game

was “an interesting experience”. The most important

factors in the game mentioned were “simplicity of

PhysiologicalComputingGaming-UseofElectrocardiogramasanInputforVideoGaming

161

the gameplay” and the fact that the game “promotes

competition between players”.

Subject 2 focused mostly on the PC side of the

game. According to him, after “becoming familiar

with game properties” it promotes players who are

able to “control heart rate, by meditation”.

C. Analysis of Signal Processing

Results

Signal acquisition and preprocessing appeared to be

rather well developed; still the most challenging part

was related to the detection of the QRS complexes

and measuring the duration between them.

As mentioned before, the wavelet-based

algorithm obtained 91,4% accuracy, whereas the

differentiation algorithm was slightly less accurate

reaching 90,6% of accuracy. However, this method

was computationally simpler to implement and was

less memory demanding than the one based on

wavelets. It has to be mentioned, though, that in both

algorithms the number of false-positive peaks

detected can be minimized using a false positive

peaks removal algorithm (Du, 2006).

From the signal processing point-of-view both

methods for detecting the QRS complex

could be taken into account. Nonetheless, for the

gaming industry time of response and dynamics of

the program are more important than accuracy. Due

to the fact that wavelet transform processing times

(10 s of data in 7.8 s) is almost 8-fold larger than for

the differentiation algorithm (10 s of data in 1.2 s),

this latter algorithm was preferred for use in the final

version of the game.

D. Possibilities and Future

Improvements

During this project we focused on creating an

additional dimension in gaming. In general we hope

that this work contributes to the widespread use of

PC in the gaming industry, which is thought to

continue to be one of the most profitable businesses

in years to come. Now we are going to discuss

briefly about the future possible usages for our belt

and the PC platform, which are:

Fatigue detector: when computing heart rate and

breathing it is possible to calculate and predict the

level of fatigue for the current player. Similar

approaches are used nowadays in modern sports

laboratories, but with the use of electrodes and

wires. Combining this tool with Microsoft

Kinect

technology, will allow the creation of a real virtual

representation of the human body, including the real

stamina of the person. Thus, the possibility of using

our belt in a game would be an improvement over

existing games, allowing for a much more realistic

experience. Additionally, the belt and PC platform

could be used also in the medical field, as a feedback

for workout or rehabilitation. In that case, games

would complement the role of the physiotherapist or

trainer, both motivating and controlling correctness

of the movement (Marozas, 2011).

PC gaming with Unity 3D or other platforms:

one of the most obvious possible improvements is to

create an add-on or plugin to Unity 3D with the

integrated heart rate detection algorithm, which will

make the FlappyHeartPC, or any designed game

available for the market. This approach would focus

on increasing the playability of the games, making

them more enjoyable, interesting and fitting each

player. This would also make games more diverse

and different each time the game is played.

Long term, portable monitoring of human body

signals: people with cardiac or breathing problems

usually have to go to a hospital for long and

exhausting monitoring. By making use of portable

monitors, patients could regain their autonomy. Our

belt and PC platform could be one good solution to

address this issue if given phone or internet

communication capabilities. Thus, data could be

recorded and sent to medical doctors for evaluation.

This approach could also be used for monitoring the

elderly and isolated patients and inform the hospital

about arising problems. This could be a means to

increase the chances to survival for these patients.

5 CONCLUSIONS

This project is a starting point for futures studies in

the field of HCI. It is already possible to infer that

with information from our physiology as input (heart

rate) it is possible to change the dynamics of a game

environment using a low-cost tool such as

Bitalino®.

For future investigations more tests are required,

e.g. more data has to be acquired and analyzed. For

further work the most reasonable upgrade is

integrating signal processing within Unity 3D

software, which will make the game more stable,

and versatile. Additionally, in the long run, this type

of game applications can have an interesting role in

rehabilitation and physiotherapy.

PhyCS2015-2ndInternationalConferenceonPhysiologicalComputingSystems

162

REFERENCES

Afonso, V. X. (1993) "ECG QRS Detection." Biomedical

Digital Signal Processing. Englewood Cliffs, NJ:

Prentice-Hall.

Christov, I. I. (2004) "Real time electrocardiogram QRS

detection using combined adaptive threshold."

BioMedical Engineering OnLine 3 (1): 28.

De Burgh Daly, M., and Scott, M. J. (1958) "The effects

of stimulation of the carotid body chemoreceptors on

heart rate in the dog." The Journal of Physiology 144

(1): 148-166.

Du, P. Kibbe, W. A., and Lin, S. M. (2006) "Improved

peak detection in mass spectrum by incorporating

continuous wavelet transform-based pattern

matching”, Bioinformatics 22 (17): 2059-2065.

Fairclough, S. H. (2009) "Fundamentals of physiological

computing." Interacting with Computers 21 (1): 133-

145.

Goldberger, A. L. (1991) "Is the normal heartbeat chaotic

or homeostatic." News Physiol Sci 6: 87-91.

Guerreiro, J., et al. (2013). BITalino-A Multimodal

Platform for Physiological Computing. In ICINCO (1)

(pp. 500-506).

Hettinger, L. J., et al. (2003) Neuroadaptive technologies:

Applying neuroergonomics to the design of advanced

interfaces. Theoretical Issues in Ergonomic Science 4

(1-2): 220–237.

Holley, G. H., and Salem, R. J. (1990) "Portable belt

monitor of physiological functions and sensors

therefor." U.S. Patent No. 4,909,260.

Hutton, R. S., and Atwater, S. W. (1992) "Acute and

chronic adaptations of muscle proprioceptors in

response to increased use." Sports Medicine 14 (6):

406-421.

Lefter, I., et al. (2010) "Emotion recognition from speech

by combining databases and fusion of classifiers."

Text, Speech and Dialogue. Springer Berlin

Heidelberg.

Lim, Y. G., et al. (2006) "ECG measurement on a chair

without conductive contact." IEEE Transactions on

Biomedical Engineering 53 (5): 956-959.

Lourenço, R., et al. (2012). Experimental Apparatus for

Finger ECG Biometrics. In BIODEVICES: 196-200.

Ostchega, Y., et al. (2011) Resting pulse rate reference

data for children, adolescents, and adults: United

States, 1999-2008. US Department of Health and

Human Services, Centers for Disease Control and

Prevention, National Center for Health Statistics.

Marozas, V., et al. (2011) "A comparison of conductive

textile-based and silver/silver chloride gel electrodes

in exercise electrocardiogram recordings." Journal of

Electrocardiology 44 (2): 189-194.

Misczynski, D. J., et al. (2005) "Electrode belt for

acquisition, processing and transmission of cardiac

(ECG) signals." U.S. Patent Application 10/901,588.

Mills, H., et al. (1979) "The precordial ECG BELT for

obtaining rapid reproducible precordial leads." Journal

of Electrocardiology 12 (4): 407-410.

Muhlsteff, J., et al. (2004) "Wearable approach for

continuous ECG-and activity patient-monitoring".

IEMBS'04. 26th Annual International Conference of

the IEEE. Engineering in Medicine and Biology

Society, 2004: 2184-2187.

Pan, J. and Tompkins, W. J. (1985) "A real-time QRS

detection algorithm." IEEE Transactions on

Biomedical Engineering 3: 230-236.

Pokrovskii, V. M. (2005) "Integration of the heart

rhythmogenesis levels: heart rhythm generator in the

brain." Journal of Integrative Neuroscience 4 (2): 161-

168.

Pola, T., and Vanhala, J. (2007). "Textile electrodes in

ECG measurement." Intelligent Sensors, Sensor

Networks and Information, 2007. ISSNIP 2007. 3rd

International Conference on. IEEE, 2007.

Porges, S. W. (1991) "Vagal tone: An autonomic

mediator of affect." in The development of emotion

regulation and dysregulation. Cambridge studies in

social and emotional development, (pp. 111-128).

New York, NY, US: Cambridge University Press.

Sandercock, G. R., Bromley, P. D., and Brodie, D. A.

(2005). "Effects of exercise on heart rate variability:

inferences from meta-analysis." Med Sci Sports Exerc

37 (3): 433-439.

Silva, H. et al. (2011) "Study and evaluation of a single

differential sensor design based on electro-textile

electrodes for ECG biometrics applications." IEEE

Sensors 2011: 1764-1767.

Wagner, S., et al. (2002) "Electrotextiles: concepts and

challenges." International Journal of High Speed

Electronics and Systems 12 (2): 391-399.

West, B. J., et al. (1985). "Nonlinear dynamics of the

heartbeat: I. The AV junction: Passive conduit or

active oscillator?." Physica D: Nonlinear Phenomena

17 (2): 198-206.

Wiltbank, T. B., et al. (2008) "Faint and prefaint reactions

in wholeblood donors: an analysis of predonation

measurements and their predictive value." Transfusion

48 (9): 1799-1808.

PhysiologicalComputingGaming-UseofElectrocardiogramasanInputforVideoGaming

163