A Touch-based Configurable Gamepad

for Gamers with Physical Disabilities

Davide Gadia

, Marco Granato

, Dario Maggiorini

, Matteo Marras

and Laura Anna Ripamonti

Department of Computer Science, University of Milan, via Comelico 39, I-20135, Milano, Italy

Key

words: Video Game, Gaming Devices, Physical Disability.

Abstract: In modern videogames, interfaces and interaction design play a major role in user experience. As of today,

in-game interaction is mainly performed through industry-standard devices. These devices can be either

general purpose (e.g., mouse and keyboard) or specific for gaming (e.g., a gamepad). However, gaming

interaction devices are not usually designed for people with physical disabilities. In this paper, we first explore

issues related to the use of standard gaming devices from gamers with physical disabilities and then we

propose a solution by means of an innovative game controller device. This game controller is build using a

touch screen interface. The touch screen interface can be configured based on the user needs and will be

accessible by gamers which are missing fingers or are lacking control in hands movement.

1 INTRODUCTION

In these last years, the role of video games in our lives

is changing. As a matter of fact, they are now also

regarded as viable digital artefacts to deliver

interactive stories, teach new skills (edugames),

perform physical exercise (exergames), and much

more. Given this new role for games, the way we use

to interact with the media (and its design) is of

paramount importance for an optimal user

experience. While for the design part we already have

a consolidated literature, the devices used to perform

the interaction itself are – sometime legacy –

industrial standards. These devices range from

general purpose tools, inherited from office

automation activities (e.g., mouse and keyboard), to

gaming specific tools (e.g., gamepads). These devices

have been designed with limited consideration for

gamers with physical disabilities. As an example, it is

almost impossible for a player missing the left hand

to effectively use a standard gamepad. A keyboard is

a more viable solution, but nonetheless it might put a

serious disadvantage on the player.

In this paper, we tackle the problem of providing

a good gaming experience to players with physical

disabilities. In particular, we focus our research on

gamers which are missing – or have difficulties

using – a hand or part of it. The contribution of this

manuscript is twofold: firstly, we provide insights for

a deep understanding of the problem; secondly, we

propose a novel gamepad, accessible to gamers with

physical disabilities of the hands. The controller we

propose is designed to also take advance of a touch

device to be easily reconfigurable to cope with the

user’s needs.

The remainder of this paper is organized as

follows: In Sec. 2 we analyze existing literature

addressing the same problem, while in Sec. 3 current

commercial solutions to support physically impaired

gamers are presented. Section 4 describes the One-

Hand Controller, our proposed solution, discussing its

hardware and software architectures. In Sec. 5 results

from user tests are summarized and Sec. 6 concludes

the paper.

2 RELATED WORK

Game development was born into research labs in the

late ‘50s, with the game Tennis for Two by Willy

Higinbotham of the Brookhaven National Lab. It is

no surprise that even the developers of Spacewar!

(1961) declared that controls available on the DEC-

PDP-1 were not adequate for a game (Graetz, 1981).

As a result, Spacewar! has been the first game

augmented with an external ad-hoc control system

(Commings, 2007).

Moving now to recent literature, there is a vivid

interest in game accessibility (Yuan et all, 2011)

Gadia D., Granato M., Maggiorini D., Marras M. and Ripamonti L.

A Touch-based Conﬁgurable Gamepad for Gamers with Physical Disabilities.

DOI: 10.5220/0006508800670074

In Proceedings of the International Conference on Computer-Human Interaction Research and Applications (CHIRA 2017), pages 67-74

ISBN: 978-989-758-267-7

(Westin et all, 2011) (Rowland et all, 2016) and the

Independent Game Developers Association (IGDA)

published in 2004 a set of guidelines for gaming

accessibility. Unfortunately, these guidelines focused

mainly on visual impairments. The guidelines have

been then extended in 2006 by (Ossmann and

Miesenberger, 2006).

The scientific community is contributing to this

important topic in two ways: devising guidelines for

accessible game design, and providing innovative

devices for disabled gamers.

With respect to guidelines for accessible game

design, in (Ossmann et all, 2008), we can find a study

about making classic games accessible to disabled

gamers. In this work, a middleware between the game

and the I/O subsystem has been created using a

descriptive language. This result aims to demonstrate

that – technically – any game can be made compatible

with any kind of device. Unfortunately, this approach

seems to be too invasive for the game core

architecture because it requires for the (assistive) I/O

device to have a direct interface with the hosting

game engine. This interface requires the development

of a game engine specific to the device and, in turn,

to the supported disability. Another contribution

along this same line (Grammenos et all, 2009) led to

the creation of a game that, with specific changes to

game mechanic and logic, can be adapted to any kind

of player’s disability. A very specific scientific

contribution is presented in (Hernandez et all, 2013),

where the design of fast-paced action-oriented games

for children with cerebral palsy is discussed. In this

paper, a participatory design process is used to prove

the feasibility of the approach and provide a set of

recommendations to achieve action-orientation and

playability.

To address the introduction of innovative devices,

the research here is mainly focused on making

standard controllers available to impaired people

through physical adaptation and integration with

additional sensors, such as in (Iacopetti et all, 2008)

and (Fanucci et all, 2011). Unfortunately, their

approach seems to be a bit intrusive and cumbersome

to setup for the average player due to the additional

wiring and sensors unsupported by the console

vendor. Nevertheless, these contributions proved to

be an interesting solution to support some cognitive

disabilities.

Other researchers are working to completely

exclude the physical interaction with a controller. To

remove the physical interaction, it is possible to adopt

BCI (Lopetegui et all, 2011) or EMG (Watanabe at

all, 2010, Kawala et all, 2015) technologies.

The combination of the two approaches above,

may result in using a sensor system in place of a

controller like in the case of the VoodooIO Gaming

Kit (Villar et all, 2007). In the VoodooIO Gaming Kit,

players are allowed to place physical inputs as they

need on a conductive fabric.

Solutions provided from outside the scientific

community are also available, we address them in the

next section.

3 CONTROLLERS SUITABLE

FOR PHYSICAL DISABILITIES

A physical disability is defined as a limitation on an

individual's physical functioning. This limitation may

regard mobility, dexterity or stamina. A gamer

suffering from mobility or dexterity limitations may

experience issues in interacting with a video game. As

an example, the player may not be able to provide

specific inputs (or combinations of them) due to

inability to press multiple buttons at once or move a

finger between two positions in a timely manner.

These limitations can make the gaming experience

unbalanced at least, if not even completely frustrating

(Tollefsen and Loude, 2004) (Brown et all, 2015) for

the gamer.

To overcome the aforementioned limitations,

impaired players usually look for specific input

devices. On the market, many devices have been

proposed by gaming-oriented companies to improve

(or make possible) gaming experience of physically

impaired players.

The ASCII Grip controller (1996), is a device

originally designed for the fourth generation of game

consoles. This controller is conceived to let gamers

play using only one hand while merging all

interaction under the control of 5 fingers (Fig. 1).

A similar solution to the ASCII Grip is proposed

by the DragonPlus RPG DuoCon (2008), where all

the controls of a standard gamepad are placed on a

tabletop case and the player can set the hand on top of

an ergonomic support.

The One-Handed Ergonomic Palm Game

Controller (2010) is also available on the market.

This controller adopts the concept of tabletop case

and all controls are located at the end of a palm-

resting platform to avoid straining the hand (see Fig.

2). While this proved to be just a variation of a gaming

keypad, the interesting feature of this product is its

compatibility. As a matter of fact, it supports PCs as

well as several modern consoles up to Playstation 4

and Xbox One.

Figure 1: ASCII Grip controller.

Figure 2: One-Handed Ergonomic Palm Game Controller.

An interesting step forward to support one-handed

gamers is represented by the eDimensional Access

Controller (2008). As we can see in Fig. 3, like the

previous one, also this controller extends the concept

of tabletop case but, differently from the Palm

Controller, is providing a modular architecture. Each

control set can be plugged in any socket to tailor the

controller to the player’s specific disability.

Most probably, the best commercial example of

controller to support disabilities is the NES Hands

Free Controller (1989) by Nintendo. This controller

was designed for gamers totally unable to move the

hands. This device must be strapped to the chest and

hooked to the neck of the player (Fig. 4). For

movements, the gamer can use her chin to move a

joystick while buttons are simulated by blowing in a

small pipe.

Other opportunities for disabled players lie on the

use of motion tracking devices such as Microsoft

Kinect or Wii Remote. These control systems can be

useful, but are usually tight to specific game

mechanics. Mainstream games are either specifically

designed to use them (e.g., Just Dance for Xbox) or

require the use of a standard controller. As a result, a

user cannot just play any game but only a specific

subset, which may not be compatible with her specific

disability.

Figure 3: eDimensional Access Controller.

Figure 4: NES Hands Free Controller.

Along with big companies, we can also observe

the constant growth of non-profit organizations

helping disabled players, such as Ablegamers and

OneSwitch. These organizations offer an interesting

ground for homebrewer to create tailored solutions

for single users.

4 OHC: A NOVEL AND

ACCESSIBLE GAMEPAD

Designing a game or a device to cope with all possible

disabilities is very difficult, but not impossible, as

discussed by (Ossman et all., 2008). In this paper we

are going to focus on gamers suffering from mobility

issues in the upper limbs. In particular, our research

targets players which are missing one hand (or part of

it) or have severe limitations in the mobility of their

dominant hand.

Starting from the above considerations, and

taking inspiration from existing devices on the

market, we designed the One-Hand Controller

(OHC). OHC is similar in philosophy to the

eDimensional Access Controller, but it leverages on

a mix of analog and digital inputs. Analog inputs are

providing legacy compatibility with mass-market

video games, while digital inputs can be built on top

of a configurable touch interface to provide a

configurable buttons layout.

A touch interface has been selected because of

three important advantages over current hardware

solutions. First, it allows a flexible and fine-tuned re-

configuration. This is the same goal of the

eDimension Controller but without any constraint in

controls size and location. Second, in a next stage of

the project, it might be feasible to substitute the touch

surface with a tablet or smartphone already owned by

the player, thus achieving a consistent cost reduction.

Last, as a prototype, a touch interface allows fast

prototyping of controls layouts and easy data

collection about hand posture and touch misses.

The digital buttons layout of OHC can be finely

tuned to the gamer’s requirements; moreover, it can

also be specific to each video game. A concept design

of the OHC device can be seen in Fig. 5. As it can be

observed, the device is completely symmetrical; this

way it can be easily flipped to support both right- and

left-handed players.

Figure 5: One-Hand Controller Concept Design.

In the following subsections, we are going to

describe hardware and software architectures of the

OHC prototype we built.

4.1 Hardware Architecture

OHC is built around a 7” LCD multi-touch

touchscreen equipped with an analog stick and two

pushbuttons. The touchscreen supports up to five

fingers at once and the analog stick is two-axis

thumbstick. The controller hardware is managed by

two interoperating microcontrollers: a Raspberry PI 2

and an Arduino Leonardo. A scheme of the hardware

setup is reported in Fig. 6.

The Raspberry PI microcontroller is in charge to

drive the LCD touchscreen using the onboard Display

Serial Interface (DSI). Using the DSI, the 7” LCD can

be driven at 25 frames per second. The Arduino

Leonardo alone would not provide enough bandwidth

to drive such a large display. Moreover, the

Raspberry is also in charge to collect user inputs.

These user inputs include both touches on the screen

and the external thumbstick with pushbuttons.

Unfortunately, there are no analog inputs on the

Raspberry GPIO bus. Therefore, we were forced to

use an Analog to Digital Converter (ADC) to convert

the thumbstick position. To perform this conversion,

we used an MCP3008, but any 10 bits converter

supporting 5V digital outputs can be used.

The Arduino Leonardo oversees managing the

communication between OHC and the gaming PC.

Leonardo has been used since it natively supports

USB slave mode, and libraries to emulate mouse and

keyboard are ready available. This way, the

Raspberry PI can decode user input and use the

Arduino to remap the result to legacy inputs for the

game. Raspberry PI alone is not able to perform this

task because it – as a system-on-chip computer – can

only work as a USB Host.

Figure 6: One-Hand Controller board scheme.

Figure 7: OHC prototype.

A picture of the physical prototype we used for

experiments can be seen in Fig. 7.

4.2 Software Interface

The software architecture is designed with the user

experience in mind. As a matter of fact, its main task

is to virtualize a legacy controller starting from the

touchscreen and thumbstick inputs. This task must

also be performed with minimal delay.

When first turned on, the OHC software will

perform a calibration. As a first step, the user is asked

if she wants to (or can) use the buttons and

thumbstick. Then, the user will be prompted to lay her

hands on the touchscreen in a comfortable position.

Starting from the detected touches, a default layout

will be proposed, based on the number of available

fingers. This layout will be aligned and stretched

based on fingers’ position. The layout can also be

modified later for a better gaming experience.

The internal software supports multiple profiles.

This allows several users to share a single device.

Moreover, each user can store multiple layouts under

her profile. As a result, each player can store a

specific layout for each game, depending on personal

taste and disability. For severe disabilities, or

difficulties in coordinating fingers, macros are

supported and a single tap can be associated to

multiple inputs (see also next section).

The configuration menu can be accessed at any

time from the controller itself. The controller

touchscreen will provide menus to calibrate,

configure controls, and save/load configuration

without a requesting additional software on the

PC/console (see Fig. 8). This feature allows OHC to

be more flexible and portable. Currently, there is no

direct feedback provided to the game. This means

that, during configuration, the game is not going to

pause automatically, but it is possible to switch

configuration/layout in any game while playing.

Switching configuration automatically based on

game status is not yet supported.

Figure 8: Configuration menu layout.

Finally, OHC can support gestures. Gestures are

programmable combinations of command the user

can associate to given touch patterns. When enabled,

an area of the touchscreen can be reserved to gesture

recognition. Gestures will be detected by the

Raspberry PI and translated into a sequence of

keystrokes and movements. Currently, we support

single, double, and triple touch, rotation, swipes, and

two-fingers scrolling.

4.3 User Interaction

The Graphical User Interface is the most critical part

of the OHC software. As a matter of fact, the GUI

must be flexible enough to meet a huge range of user

requirements. Moreover, it should also be easy to

maintain and extend. For these reasons, we decided to

leverage on the computational functionality of

Raspberry PI and to implement it using Python and

the Kivy framework (Kivy, 2011). Kivy supports the

Tangible User Interface Objects (TUIO) paradigm to

manage inputs from the touchscreen in a standard

way.

The OHC GUI is implemented by composing

visual widgets in a hierarchical way. Each widget is

taking care of a specific kind of input. An overview

of the implemented input widgets is reported in

Fig. 9. When performing calibration, one or more

widgets are assigned to each finger. Based on selected

finger and feedback from the user, the widget will be

rotated and stretched to maximize comfort and

encompass any movement constraints the gamer may

have. Fingers and hand discomfort are reduced by

Figure 9: Basic GUI widgets.

Figure 10: GUI buttons deformation based on fingers.

deforming the widget in a way to place each button very

close to each finger landing point. Moreover, widgets

position should help an easy switch between controls.

Figure 10 shows default finger-based transformations

applied to primary and secondary buttons.

Nevertheless, additional considerations are

required to address missing or not usable fingers. To

cope with every possible kind of disability, we

designed specific default interfaces for each case. For

every variant, a default widget-finger association is

proposed and the actual interface is the result of the

deformation and the relocation of widgets basing on

the information collected during calibration. Extra

care must be devoted to understanding and detecting

situations where widgets are too close to each other,

hence becoming cumbersome, or, due to mobility

constraints on one or more fingers, the widgets

position may cause strain. When two widgets are too

close the controls may not be effective, especially if

the assigned finger has a reduced mobility. Strain may

be caused by physical conditions, considering that

different fingers are sharing muscle and nerve

connections. Default interfaces proposed to users

having only four or three fingers are reported in Fig.

11 and Fig. 12, respectively. In the figures, the greyed

areas indicate the working area for the touchscreen,

while the large dark dots are fingers positions

detected during calibration. The working area and the

interface position are calculated starting from these

dots.

Figure 11: Default interface for 4 fingers.

Figure 12: Default interface for 3 fingers.

Figure 13: Default interface for one finger or no hand.

In the case of players having only one finger or

missing the hand, the interface must provide a gesture

area and should be customized by the user. In this

case, we are proposing a dialpad instead of primary

and secondary buttons. The result is shown in Fig. 13.

This last case is where macros definition can be very

useful. Macros can be used to associate a single

gesture or a dialed number to complex movements or

to a sequence of inputs on a standard controller.

5 USER ACCEPTANCE TEST

To evaluate the usability of OHC, we performed a test

on a sample of fourteen players, of both sexes, both

with and without disabilities, in the age range 20-25

years (they were students graduating in Computer

Science). We included testers with no disabilities, not

only as a control group, but also and mainly because

we wanted to understand whether a device designed

with a very specific audience in mind, could be

perceived as an effective alternative to commercial

controllers also by the average user. To each tester,

we asked to setup the controller on a PC and play One

Piece: Pirate Warriors 3 (developed by Omega Force

and released in 2015), an action-based game featuring

simple hack’n’slash mechanics. This game was

selected because its difficulty is quite low. As a

matter of fact, we wanted to reduce the possibility to

collect negative feedbacks on the controller

performances, due to a frustrating user-experience

deriving from the game.

After the test, the users have been asked to fill a

feedback form and to sustain an unstructured

interview, aimed at collecting both qualitative and

quantitative data. The form was composed by five

sections, aimed at collecting demographical data,

feedbacks about the hardware and the software, the

users’ acceptance rate and problems arise during the

testing session.

The part of the test about the device setup was

intended to verify the hardware compatibility and

Figure 14: Easiness in setting up the device.

Figure 15: GUI intuitiveness.

integration with legacy drivers. In particular, we had

to check the Arduino acting as a USB slave device on

different hardware configurations. Results are

reported in Fig. 14. From the feedback, it seems no

user had any major problem in connecting OHC to her

computer. Actually, more than half of the group

reported an installation without any problem. This is

an interesting indication about OHC connecting

easily with current retail hardware.

The following section of the test was about the

effectiveness of the interface. This is a critical point

for the whole project. Results are plotted in Fig. 15.

From this standpoint, users are hinting that there is

still space left for improvement. Most of the testers

found the interface intuitive, but still improvable.

After performing some interviews on the topic, many

users felt positive about the calibration process and

the reason for the relative lack of usability may sit on

the widget graphical appearance. This is quite

understandable, since in its current state, the

prototype is proposing a very basic black and white

interface, with no help menu.

The next part of the test investigated how useful

is the device for the user. Our aim was to understand

Figure 16: Usefulness of the device.

Figure 17: Willingness to adopt OHC as main gaming

device.

to what extent users (both with and without

disabilities) would like to have a controller which can

be configured following the user’s preferences.

Surprisingly enough, as reported in Fig. 16, not only

the users with disabilities claimed OHC was useful,

but more than 85% of the test group replied that OHC

was very useful for their needs. No one was negative

about the controller.

The last part of the testing session investigated the

willingness of the users to substitute with OHC their

current retail PC gaming controller. It has been

clearly stated in the feedback form to avoid

comparison with a console controller. As Fig. 17,

shows, there is a general interest in adopting OHC.

The outcomes of the feedback surprised us: on

one hand, we did not expect non-impaired users to

like so much our prototype, while on the other hand,

we find it curious that, such a small fraction of the

sample affirmed that they would substitute their

controller with OHC. For sure, further investigation

is required on this specific aspect.

6 CONCLUSION AND FUTURE

WORK

In this paper, we addressed the problem of

accessibility of gaming devices to gamers with

physical disabilities. In particular, we have been

focusing on disabilities affecting upper limbs. As a

viable solution, we designed and prototyped OHC -

One-Hand Controller: a highly configurable

controller mixing analog and digital inputs and

leveraging on a multi-touch display. By using a touch

interface, we can finely adapt to the gamer’s

requirements. After building a prototype, we

performed tests on a group of students. The prototype

received favorable feedbacks, despite the fact a

minority of users is willing to adopt it as their main

gaming device.

In the future, we are planning two main

improvements: a better graphical interface and the

addition of haptic feedback. The interface will be

reworked from a graphical standpoint, but we are also

going to provide a better widgets placement

considering the FFitts law [sic] (Bi et all, 2013). For

the haptic feedback, since OHC is a tabletop device,

we are considering adding external actuators.

REFERENCES

Bi, X., Li, Y., Zhai, S. (2013). 2013. FFitts law: modeling

finger touch with fitts' law. In Proceedings of the

SIGCHI Conference on Human Factors in Computing

Systems (CHI '13). ACM, pages 1363-1372.

Brown, M., Kehoe, A., Kirakowski, J., Pitt, I. (2015).

Beyond the Gamepad: HCI and Game Controller

Design and Evaluation. In Game User Experience

Evaluation, Springer International Publishing, pages

263-285.

Commings, A. H. (2007). The evolution of game controllers

and control schemes and their effect on their games. In

17th Annual University of Southampton Multimedia

Systems Conference. Vol. 21.

Fanucci, L., Iacopetti, F., Roncella, R. (2011). A console

interface for game accessibility to people with motor

impairments. In IEEE International Conference on

Consumer Electronics -Berlin (ICCE-Berlin), pages

206-210.

Graetz, J. M. (1981). The origin of spacewar. In Creative

Computing Issue 39.

Grammenos, D., Savidis, A., Stephanidis, C. (2009).

Designing universally accessible games. In Computers

in Entertainment, 7(1), 29 pages.

Hernandez, H. A., Ye, Z., Graham, T. C. Fehlings, D.,

Switzer, L. (2013). Designing action-based exergames

for children with cerebral palsy. In Proceedings of the

SIGCHI Conference on Human Factors in Computing

Systems (CHI), pages. 1261-1270.

Iacopetti, F., Fanucci, L., Roncella, R., Giusti, D., Scebba,

A. (2008). Game Console Controller Interface for

People with Disability. In International Conference on

Complex, Intelligent and Software Intensive Systems,

pages 757-762.

Kawala-Janik, A., Podpora, M., Gardecki, A., Czuczwara,

W., Baranowski, J., Bauer, W. (2015). Game controller

based on biomedical signals. In proc. 20

International

Conference on Methods and Models in Automation and

Robotics (MMAR), pages 934-939.

Kivy, 2011. http://www.kivy.org/.

Lopetegui, E., Zapirain, B.G., Mendez, A. (2011). Tennis

computer game with brain control using EEG signals.

In 16

International Conference on Computer Games

(CGAMES), pages 228-234.

Ossmann, R., Miesenberger, K. (2006). Guidelines for the

Development of Accessible Computer Games. In:

ICCHP 2006, Computers Helping People with Special

Needs. Lecture Notes in Computer Science, vol 4061.

Springer, Berlin, Heidelberg.

Ossmann, R., Miesenberger, K., Archambault, D. (2008). A

Computer Game Designed for All. In: ICCHP 2008,

Computers Helping People with Special Needs. Lecture

Notes in Computer Science, vol 5105. Springer, Berlin,

Heidelberg.

Rowland, J. L., Malone, L. A, Fidopiastis, C. M.,

Padalabalanarayanan, S., Thirumalai, M., Rimmer, J.

H. (2016). Perspectives on active video gaming as a

new frontier in accessible physical activity for youth

with physical disabilities. Physical therapy

96(4):521-532.

Tollefsen, M., Lunde, M. (2004). Entertaining Software for

Young Persons with Disabilities. In ICCHP 2004

Computers Helping People with Special Needs. Lecture

Notes in Computer Science, vol 3118. Springer, Berlin,

Heidelberg.

Villar, N., Gilleade, K. M., Ramdunyellis, D. Gellersen, H.

(2007). The VoodooIO Gaming Kit: A Real-time

Adaptable Gaming Controller. In Computers in

Entertainment, 5(3), 16 pages.

Watanabe, M., Yamamoto, T., Kambara, H. Koike, Y.

(2010). Evaluation of a game controller using human

stiffness estimated from electromyogram. In proc. 2010

Annual International Conference of the IEEE

Engineering in Medicine and Biology, pages 4626-

4631.

Westin T., Bierre K., Gramenos D., Hinn M. (2011).

Advances in Game Accessibility from 2005 to 2010. In

Proc. Universal access in human-computer interaction.

Users Diversity: 6

International Conference

(UAHCI). pages 400-409.

Yuan B., Folmer E., Harris F. C. (2011). Game

accessibility: a survey. In Universal Access in the

Information Society 10(1):81-100.