Stuck in Limbo with Magical Solutions: The Testers’ Lived Experiences

of Tools and Automation

Isabel Evans

1 a

, Chris Porter

1 b

, Mark Micallef

2 c

and Julian Harty

3 d

Department of Computer Information Systems, University of Malta, Malta

Department of Computer Science, University of Malta, Malta

School of Computing and Communications, Open University, Milton Keynes, U.K.

Keywords:

Software Testing, Automation, Tools, HCI, User Experience, UX, Lived Experience, Usability, Industry.

Abstract:

The automation of people’s roles at work brings changes to their lives and work, bringing advantages of in-

creased effectiveness and efﬁciency, yet potentially life-changing effects, including redundancy. The software

industry’s purpose is to automate people’s tasks and activities, and this applies also to jobs within the soft-

ware industry, including teams who specialise in testing software. Test automation projects are not always

successful, and our research initially set out to discover whether the challenges were usability-related, and

whether HCI methods could help improve tools. We discovered a much richer story, which told of emotional

stresses and life experiences within the software testing community. We discuss how automation, with all its

beneﬁts, affects motivation, causing disassociation of testers from their roles, and affecting their job-task mix.

We show reasons why software test automation affects testers. Finally, we set out our position for our research

about the lived experience of software testers using automation, which we are calling TX: The Testers’ Lived

Experiences of Tools and Automation, and argue that the effect of automation and tooling on testers’ lived

experience and its effect on their motivation is an area of study worthy of research.

1 INTRODUCTION

The automation of people’s roles at work brings

changes to their lives (Groover, 2019). These include

increased effectiveness and efﬁciency in their work,

potentially saving money and time for the employing

organisation, as well as providing gains in productiv-

ity, and consistency of output. Should we addition-

ally (even instead) focus on the effect on the person’s

job, how it changes that, and the experience of the au-

tomation as it affects people’s lives? When those jobs

are within the software industry, where the purpose of

the industry is to automate people’s tasks and activi-

ties, the effects of automation on the individual may

still be the same as for any other worker. One group

of workers within the software industry are the teams

who specialise in testing software. Aspects of their

roles are subject to automation projects.

In this paper we set out our position for our re-

https://orcid.org/0000-0003-3913-4463

https://orcid.org/0000-0003-3813-2941

https://orcid.org/0000-0003-4159-6923

https://orcid.org/0000-0003-4052-0054

search about the lived experience of software testers

of automation, which we are calling TX: The Testers’

Lived Experiences of Tools and Automation, and ar-

gue that the effect of automation and tooling on

testers’ lived experience and its effect on their mo-

tivation is an area of study worthy of research.

2 BACKGROUND

Software is ubiquitous and relied upon. It is essen-

tial that software is tested adequately before it is used.

This testing is done by developers and by users of the

software, however testing is also done by specialist

individuals and teams who focus on testing.

2.1 Software Testing

Software testing is an activity intended to give stake-

holders information about the quality of software or

services, and includes the planning for, design of,

execution of, and analysis of the results of inves-

tigation into the behaviour and characteristics of a

Evans, I., Porter, C., Micallef, M. and Harty, J.

Stuck in Limbo with Magical Solutions: The Testers’ Lived Experiences of Tools and Automation.

DOI: 10.5220/0009091801950202

In Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 2: HUCAPP, pages

195-202

ISBN: 978-989-758-402-2; ISSN: 2184-4321

195

product or part of a product (ISTQB, 2018; Reck-

less, 2017a). Testing is hard to do well, expensive

and time-consuming but takes place in the context

of projects where quality must be maintained in the

face of reduced timescales and budgets (F. Lehner and

Abran, 2013; Jones, 2015; Tassey, 2002). Testing is

hard because it includes challenging cognitive activ-

ities that require creativity such as threat modelling

within complex domain and system spaces (Bach and

Bolton, 2016; Bolton, 2016; Toledo, 2017).

2.2 Software Test Automation

Aspects of testing are repetitious or difﬁcult for peo-

ple to do well and these are candidates for automation

(ISTQB, 2018). Test automation is the use of tools

to aid parts of the test activity. That can be for exam-

ple, repetitious execution of tests, checking the results

of testing, aspects of test planning and test reporting

(ISTQB, 2018).

Test automation projects are not always success-

ful, causing both practitioners and academics to de-

bate the reasons and potential solutions for automa-

tion project failures (Fewster and Graham, 1999;

Graham and Fewster, 2012; Kaner, 1988; Wiklund,

2015). A debate within the software testing commu-

nity asks whether the causes of failure include testers’

lack of adequate knowledge and technical skills, or

whether the tools are inadequately designed, requir-

ing unreasonable cognitive effort from the testers and

slowing them carrying out their challenging tasks

(Hendrikson, 2010; Lambert, 2014; Reckless, 2017b).

2.3 Usability, User Experience and

Lived Experience

The interaction between a person and the technology

she is using is not just about the design of the user

interface (UI), or even about the usability of the tech-

nology. The UI provides the necessary affordances

which support interaction between a person and the

technology. The UI and interaction design contribute

to usability. Whether a tool meets the needs of a user

(its utility) combines with its usability to make the

tool useful.

For the purposes of this paper we deﬁne usabil-

ity as in ISO 9241 (ISO, 2018) in terms of user

goals, user effectiveness, user efﬁciency, and user

satisfaction in a speciﬁed context of use. Usability

contributes to user experience, which also includes

the quality of all the interactions between the soft-

ware provider and the customer: “[User experience]

encompasses all aspects of the end-user’s interac-

tion with the company, its services, and its prod-

ucts [so that the product meets the needs of the

customer] without fuss and bother” (Norman and

Nielsen, 2019). User experience attributes are human-

focused, including trust, ﬂow and credibility, and

these contribute to the lived experience of the user. In

considering ‘lived experience’, we look at “repercus-

sions extending into the users’ daily lives and away

from the technology itself ” (Porter, 2015), and this in-

cludes the emotional impact of the technology. Thus,

the UI design contributes to usability, and usability of

software contributes to but does not encompass user

experience. User experience in turn contributes to and

is part of the lived experience. The emotional content

of the experience can be supported by a great interface

design, but all the circumstances of a person’s inter-

actions with the software, its provider and the people

around them will also contribute to their emotional

response.

2.4 Our Research to Date

Based on the research question: “What are the experi-

ences of testers with automation?”, we are exploring

the interactions of testers and their automation tools,

seeking to understand what problems hinder success-

ful tool adoption.

Following a literature review and industry expe-

rience, our hypothesis at the start of the study was

that the challenges would fall into technical, organ-

isational or usability themes (Fewster and Graham,

1999; Graham and Fewster, 2012; Wiklund, 2015).

Examples of technical challenges include issues with

the IT environment, IT security, the tool’s perfor-

mance, and execution of tests. Examples of organ-

isational challenges include process issues, manage-

ment support, staff availability, and testers’ skill sets

(Gamba and Graham, 2018b). Examples of usability

challenges include the automation not supporting the

testing workﬂow, poor learnability and poor operabil-

ity for the end user (in this case, the end user is the

tester using the automation).

2.4.1 Mixed Methods Approach

A mixed-method approach was adopted to collect

primary data from practitioners in the ﬁeld. Semi-

structured interviews and surveys were used to learn

more about testers, their skill-sets and experience

with test tools and automation. Qualitative meth-

ods were adopted to deconstruct personal experiences

or stories with test tools and automation, allowing

for a richer understanding of the phenomena being

investigated. We asked testers to “tell us a story

about your experiences with test tools and automa-

tion.” We expected in our study to uncover issues

HUCAPP 2020 - 4th International Conference on Human Computer Interaction Theory and Applications

196

with skill sets and with usability to help us examine

the skills/usability debate, provide evidence for which

was the most important contributory factor to automa-

tion success, and to allow us to suggest potential solu-

tions. Responses collected so far from from 117 peo-

ple via surveys and interviews were used to feed the-

matic analysis initially based on the ﬁndings from our

literature review, while additional themes were added

as they emerged from data analysis. The data we

have gathered so far indicates patterns and themes that

conﬁrm ﬁndings from other industry writing and aca-

demic research on technical, organisational and us-

ability impediments to test automation (Graham and

Fewster, 2012) and (Wiklund, 2015) but also our data

indicates that the relationship that testers have with

their tools is more diverse, richer, and affords more

emotional experiences than previously reported. This

has led us to think about the purposes and effects of

automation, not just in terms of productivity gains, but

in terms of the lived experience of software testers of

their tool set and automation as a signiﬁcant challenge

to successful test automation projects, and ultimately

to the sustainable well-being of those testers.

3 DISCUSSION

The purpose of the software industry is to provide

beneﬁt to its stakeholders by automating activities

of all kinds. When we start to apply tools and au-

tomation to activities within the software develop-

ment sphere, then the focus of the automation is the

change or removal of tasks carried out by software

professionals. This includes software testing; when

testing is automated, then the people who test soft-

ware are subject to the same beneﬁts and threats as

anyone else whose role is subject to an automation

(software) project. We argue that automation, as it

affects the people whose tasks are being automated,

is worthy of discussion, and that this includes the

roles within the software industry. It is reasonable

to ask questions about the lived experience of staff

within organisations, and that executives, managers,

owners and shareholders be concerned with the emo-

tional welfare of their staff, as well as with proﬁt and

productivity. In examining the automation and tool-

ing used by software testers, concern for their moti-

vation and happiness, as well as their efﬁciency and

effectiveness, is humane as well as contributing to a

healthy organisation. Motivation studies (Reid, 2015;

Warden and Nicholson, 1996) show that testers are

more productive when motivated.

The data we have collected so far based on the

question “Tell me a story about an experience with

test automation” indicates a richer and more nuanced

relationship between testers and their tools than we

had expected. We received responses that were more

emotional, more about the user experience of the

tools, and more about the lived experience than we

had expected. In this paper, reporting interim results,

and claiming that the lived experience of testers is an

important factor to consider when attempting to un-

derstand the success of and impediments to adoption

of test automation and test tools, we divide our discus-

sion into ﬁve areas. (1) We conﬁrm that there are ben-

eﬁts to automation and tools for software testing. (2)

We consider the effect of automation and tools on mo-

tivation of software testers. (3) We discuss whether

tester skill sets or usability improvements are the so-

lution to tools implementation problems. (4) We con-

ﬁrm that there are other problems that affect the suc-

cess of automation and tools adoption for testing. (5)

We demonstrate that the lived experience of testers is

important, at the least to the testers, but also we sug-

gest to their managers, team mates, and organisations,

and this is affected by their encounters with automa-

tion and tools.

3.1 Beneﬁts of Tools and Automation

The Cambridge English dictionary deﬁnes the term

‘tool’ as a piece of equipment used by a person, often

handheld, to enable them to carry out a task. The per-

son learns to use the tool, is in control of it, and uses

it to enhance the way they carry out their activities

and to improve the way they can deliver their craft.

‘Automation’ on the other hand is deﬁned as the use

of tools by machines to complete activities without

or with minimal human intervention. An example of

very simple tools are the needles and scissors used by

an embroiderer. A simple sewing machine is still a

tool. The sewer is in in charge of the machine, and

guides the fabric through the needles to make the em-

broidery. A more complex modern machine includes

software and automation; an embroidery pattern can

be selected, a button pressed, and then the sewing ma-

chine guides the fabric through the needles and com-

pletes the embroidery with no human intervention.

Similarly, laundry can be done by hand using sim-

ple tools such as a washboard and scrubbing brush, or

using a simple washing machine, such as a twin-tub.

This requires a lot of manual intervention and heavy,

repetitive work. An automatic washing machine, once

plumbed in, loaded and started with appropriate set-

tings, removes the need for heavy and repetitive ac-

tivities.

When tasks are automated successfully, we can

measure productivity gains. We could also measure

Stuck in Limbo with Magical Solutions: The Testers’ Lived Experiences of Tools and Automation

197

the effect on people who formerly carried out that

task. This might be a positive effect, if tedious, repet-

itive, difﬁcult or dangerous tasks are eliminated, free-

ing the person to do more interesting things. This au-

tomation could be life-enhancing and beneﬁcial. One

might think of the automatic washing machine remov-

ing the drudgery of laundry day, the modern sewing

machine allowing long seams on heavy fabric to be

sewn securely and also precise reproduction of the

same embroidery pattern easily and quickly, or of in-

dustrial production lines being made safer with re-

duced industrial accidents.

Software testing is an essential part of software

development and increasingly a focus for automation.

Testing is hard to do well, time-consuming and expen-

sive. It includes activities that are repetitive, such as

executing tests on multiple occasions and also activi-

ties that are cognitively expensive, such as data com-

parisons. Some testing is not possible without the use

of tools, either because it is too difﬁcult or because

it is too time consuming: “no matter how valuable

in-person testing is, effective automation is able to in-

crease the value of overall testing by increasing its . . .

range” (Harty, 2011). These activities are candidates

for automation, and many organisations are pursuing

projects to automate some or all of their testing.

3.2 Effect of Automation on Motivation

Successful automation, where a person’s work and

life are enhanced may be a motivating factor for that

person. However, there are circumstances where au-

tomation of all or part of a job could be demotivating.

These include (1) the fear of redundancy (2) the ef-

fect of an unbalanced task mix (3) the effect of the

automation being ﬂawed.

3.2.1 Motivation, Redundancy and Dissociation

When a role or job can be completely automated, then

the experience of the person who formerly had the

role is of redundancy and this can have a massive

negative impact on the individual, their families, and

whole communities. The community of people shar-

ing a task can build a sense of belonging and of self-

worth. One’s work-role can be the way one deﬁnes

oneself. When we meet people, we commonly ask,

“What do you do?” and the reply is often a job title,

or even a former job title: “I’m retired but I used to

be...”.

In software testing, the dream of automatic test-

ing is talked of by academics (Bertolino, 2007) and

refuted by testing community experts, with software

testing expert Michael Bolton rapping at a number

of conferences “Just don’t tell me you can automate

the testing” (Toledo, 2017) and other testers publish-

ing blogs, books, and industry conference papers on

the limits of automation (Fewster and Graham, 1999;

Johnson, 2011; Martin, 2017; Rachel, 2017). During

conversations at industry conferences during 2019,

software testers reported that their organisations had

“sacked all the testers” as part of their desire to have

all tests automated and replace people with technol-

ogy.

Testers have a strong community (evidenced by

the number of conferences, meetups and slack chan-

nels worldwide) which are mutually supporting and

full of lively debate, but there is a fear in the commu-

nity, noted for example by Bach and Bolton (2016)

that “the term ‘test automation’ threatens to dissoci-

ate people from their work.”

Bach and Bolton distinguish between testing, an

activity that by deﬁnition can only be done by a

human because it involves high cognitive skill, and

checking, a subset of the testing activity that can be

automated because it is routine. Their view is: “the

basic problem is a shallow, narrow, and ritualistic ap-

proach to tool use. This is encouraged by the pan-

demic, rarely examined, and absolutely false belief

that testing is a mechanical, repetitive process. Good

testing, like programming, is instead a challenging in-

tellectual process.”

The threat posed by automation to testers’ self-

perception as worthy, intellectual, highly skilled in-

dividuals may be a real threat, or a perceived threat,

and in either case, the attitude of testers to automation,

and their experiences of automation projects are an in-

teresting area of investigation, worthy of research, to

help us understand whether the testers’ experiences

affects the automation project, as well as how the

automation project affects the testers. Understand-

ing the threat to testers’ self-perception will require

a multi-disciplinary understanding of the world of the

testers, including insights from sociology, psychology

and organisational engineering, among possible inﬂu-

ences, and as part of our future work, we want to ex-

plore and use insights from those disciplines.

3.2.2 Motivation and Job Task Mix

In studies during the 1990’s of the job design and mo-

tivation of IT workers, Warden and Nicholson (1996)

found that the quality and testing roles are remark-

able in being both the most boring and the most over-

stimulating and stressful of all the jobs in IT. Later

motivation studies by Reid (2015) indicate that Hack-

man and Oldham’s job design measures (Hackman

and Oldham, 1974) are still a reasonable predictor of

testers’ motivation. Hackman and Oldham noted that

if the task mix within a job is not optimized then that

HUCAPP 2020 - 4th International Conference on Human Computer Interaction Theory and Applications

198

can lead to de-motivation, because jobs need to in-

clude a mix of the routine with the more challenging

aspects of a role, providing a balance. It is possible

for jobs to be stressful and demotivating if they are too

boring, and also if they are too stimulating. If all the

tedious and repetitive parts of a role are removed, then

only the more stressful aspects are retained by the hu-

man, an overstimulating role remains. Therefore, the

job mix for a tester needs to include routine as well as

more challenging tasks to be maximally motivating.

3.2.3 Motivation and Flaws in Automation

All the above assumes that automation is successfully

implemented and works well. If the implementation

is not successful, and the automation is ﬂawed - not

working at all, giving the wrong results, working in-

consistently - and is still being used, perhaps because

of organisational factors, then whoever is responsible

for the completion of the activity is left with both the

more challenging and stressful, non-automated parts

of their roles, together with stress of dealing with

the fallout from ﬂawed automation. Test automation

is proving problematic for many teams and organi-

sations, with reports of shelfware (that is tools ac-

quired but not used) and reports of difﬁculties in using

and maintaining tools (Gamba and Graham, 2018b;

Graham and Fewster, 2012; Wiklund, 2015). Difﬁ-

culties – what Wiklund refers to as impediments –

that prevent successful testing via automation are of-

ten divided into technical and organizational imped-

iments (Gamba and Graham, 2018b), and addition-

ally researchers and practitioners report usability is-

sues with tools and automation (Wiklund, 2015). If

testers have to handle the ﬂaws in automation, they

may be demotivated by having to repeat tasks man-

ually that should have completed in the automation,

double-check results, and deal with increased uncer-

tainty. Approximately one third of the testers we have

interviewed and surveyed reported issues with their

automation and tools. This, together with the ﬁndings

from our literature review, indicate that investigating

the experience of and attitude to problems in tools and

automation is worth further investigation, particularly

to look at how ﬂaws in automation affect testers’ mo-

tivation, and how it affects the UX of the testing tools

themselves.

The scope of automation and tools for testing is

also problematic. While Bertolino (2007) discusses

100% automatic testing as a dream for academia, and

some tools vendors claim to reduce the time needed

for people to be testing

or propose automation can

Automate Workday Testing, https://www.

kainosworksmart.com/testing-workday/ (accessed Oc-

cover up to 90% of testing

, test experts and test

automation experts make opposing claims, for exam-

ple Bach and Bolton (2016). If organisations attempt

to automate 90% of their testing (whether to refocus

staff onto different work, increase the workﬂow, or to

reduce staff numbers), or if an attempt is made to au-

tomate activities that cannot be automated, then peo-

ple will be frustrated by a disconnect between what

is expected of them, and what they are able to ac-

complish. The ironies of automation, noted by Bain-

bridge in 1983 (Bainbridge, 1983) and revisited more

recently by Baxter et al. (2012) could be argued to

particularly affect software testers; by the very nature

of their role, they monitor, check, and challenge au-

tomation rather than trusting it, and the automation

of part of their tasks can present them with the addi-

tional challenge of testing the test automation. Fur-

ther, the complexities and varieties of the various en-

vironment and ecosystems within which software un-

der test could be used is both a reason to automate,

and yet a challenge to understanding whether the test

automation can be trusted to accurately reﬂect real-

life software use, especially with an increased empha-

sis in the industry on the UX of software over and

above its functional accuracy.

3.3 Usability Flaws or Skills Gaps?

Testers are under increasing pressure to improve their

technical skills, in order to be able to build and main-

tain test automation. A debate has started in the

testing comunity about whether tester skill sets need

to be improved, or whether improving the usability

of tools would improve adoption (Hendrikson, 2010;

Lambert, 2014; Reckless, 2017b). Some tools ven-

dors are starting to address the perceived shortcom-

ings in terms of tool usability, speciﬁcally in terms

of learnability and interface design [based on per-

sonal observations at industry conferences and expos

and in projects with a tool vendor]. Additionally,

work is happening within the community to increase

skill sets, for example the Test Automation University

(Applitools)

. This discussion of tester skill sets was

the initial impulse for the research study described in

this paper. We started to look at whether the inci-

dence of shelfware could be reduced, and the beneﬁts

of tools more clearly realized if tools were designed

tober 2019)

The Buyer’s Guide for Selecting Software Test Au-

tomation Tools, https://www.microfocus.com/media/white-

paper/the buyers guide for selecting software test

automation tools wp.pdf (accessed October 2019)

The Automation University, https://testautomationu.

applitools.com (accessed October 2019)

Stuck in Limbo with Magical Solutions: The Testers’ Lived Experiences of Tools and Automation

199

with usability for the existing testers, rather than

testers needing to upskill to use the existing tools.

Findings from our literature review (e.g. (Wiklund,

2015)) indicate that usability is an impediment to suc-

cessful test automation, but not the only impediment.

Our own ﬁndings indicate that improving usability

may not necessarily result in a better experience for

testers. We are investigating this idea further in our

existing data, and in follow up data collection.

Our interim hypothesis is that usability is a neces-

sary but not a sufﬁcient condition for tools and au-

tomation to be successfully implemented and used.

Furthermore, we suggest that if usability techniques

are applied superﬁcially this may add to the shelfware

problem, by causing tools suppliers to provide attrac-

tive interfaces that make tools saleable, but which dis-

guise other issues which manifest under long-term

use, for example maintenance problems. This is a fo-

cus for our current research.

We also believe it is worth investigating the effect

of the pressure on testers to improve their skills, and

whether this challenge is more motivating or demoti-

vating for testers. We propose this as a focus for fu-

ture research, as motivation factors at work form part

of people’s lived experience. Based on personal ex-

perience and discussions at software testing industry

conferences over the last two years, for many peo-

ple their life at work signiﬁcantly affects their overall

wellbeing, health and lived experience. This was par-

ticularly observed during conversations at events such

as ‘Women Who Test’ events in 2018 and 2019, work-

shops presented by one of the authors on ‘People Fac-

tors for Test Automation’ in 2018 and 2019, a work-

shop at the UK Software Testers’ Retreat Weekend,

and discussions during the ‘Test Automation Face

Off’ Panel at STARCanada in October 2019.

3.4 Other Impediments to Automation

Our literature review to date conﬁrms a number of

other technical and organisational impediments to test

automation. Literature identiﬁes security and user ac-

cess issues, environmental set up issues, and inter-

operability among groups of tools, among the tech-

nical impediments. Organisational problems include

lack of management support, lack of budget for train-

ing, misunderstanding the scope of tools, and strate-

gic/tactical direction changes which mean the chosen

automation solution is discarded (Bach and Bolton,

2016; Gamba and Graham, 2018b; Graham and Few-

ster, 2012; Wiklund, 2015). These are major prob-

lems in tool and automation adoption, and various so-

lutions to them are suggested by those authors.

Although we might treat these as simply technical

or organisational issues to be overcome objectively,

we also ﬁnd evidence of a more human and emo-

tional side to how those problems manifest. Bach and

Bolton use the word “pain” to describe the interaction

of testers and tools, while Gamba and Graham wrote

a novel about test automation patterns, with the char-

acters expressing their emotions about their work as

well as their objective solutions to challenges (Gamba

and Graham, 2018a).

In our own data, we have discovered more emo-

tions, and more emotional language that we had ex-

pected, about both technical and organisational chal-

lenges. “Stuck in limbo” was a phrase used in one

survey response to describe encountering security and

access problems when trying to use a mandated au-

tomation tool, while the word “magical” was used by

two respondents describing management expectations

of automation as a solution to their perceived problem

of testing being hard, expensive and time consuming.

A technical issue does not just lead to an objective

search for a solution to the problem; the situation is

not emotionless. Respondents to our survey expressed

both frustration and anger at their being blocked by

technical defects in automation and its environment,

as well as pride and joy in being able to resolve those

bugs in automation and tools. Strong emotions were

also expressed around organisational issues, including

frustration with managers and tool providers. Some

respondents also expressed fear and self-doubt around

their skill sets needed for using tools, using words

such as “scary”. Emotions were shown in the body

language of interviewees and also in the written sur-

vey responses with negative emojis, strong language,

and even a level of despair:

“I think I should leave my job and look for

company who actually values testing.”

“...if we make this crap again, what’s the

point?”

What Wiklund (2015) describes as impediments

to test automation were primarily technical and orga-

nizational, with a large proportion of usability issues.

In our data, testers’ responses to all their challenges

and successes are emotional and related to their lived

experiences. We therefore argue that these emotional

responses, together with the effect on testers’ work

and lives makes lived experience of test automation a

worthwhile area of research. Technical ﬂaws - bugs

in the automation and tools - and organisational is-

sues are impediments to successful automation, and

also directly affect the testers themselves, causing a

degraded UX of the tool, negative emotions and de-

motivating factors affecting their lived experience.

Actual and perceived skill sets problems, lack of

self-conﬁdence, and a perception of the tools as dif-

HUCAPP 2020 - 4th International Conference on Human Computer Interaction Theory and Applications

200

ﬁcult to use lead both to the need to improve the UX

and usability of tools and automation, and also to the

need to improve testers’ self-conﬁdence and learning

opportunities. Flawed attempts to improve usability

leads to tools that look appealing, but which provide

a poor UX on prolonged use, because of issues with

maintenance, interoperability and security.

4 CONCLUSIONS

The lives of testers whose organisations want to auto-

mate part, or all of the testing activities are affected

in a similar way to any other job when automation

is introduced in an imposed manner. Along with the

beneﬁts which improve people’s lives, come the ac-

tual and perceived threats to motivation: redundancy,

sub-optimal task mix in a job, and the stress of dealing

with ﬂaws in the automation and tools. The latter may

affect even testers who are self-motivated to adopt au-

tomation. For academia, the possibility of automatic

testing may be considered a desirable goal (Bertolino,

2007), however, this does not match the lived experi-

ence of testers, and their possible perception of auto-

matic testing as either impossible or threatening their

redundancy.

5 PROPOSED RESEARCH

DIRECTION

Bringing together the discussion above, and our cur-

rent ﬁndings, we argue that lived experience of tools

and automation is a speciﬁc topic for further research,

and includes their user experience. Our research in-

cludes analysis of our existing data, and interviewing

and surveying more test practitioners about their lived

and user experiences of tools and automation. We

continue to collect and analyse testers’ stories, and are

also examining in the data about testers’ backgrounds,

skills and aspirations in order to better understand the

tester personas.

We might ask ”what is special about software

testers?” Why do they merit a study of the automa-

tion of their tasks? Are testers different to other IT

or knowledge workers - in their skills, the intellectual

content of their work, or in their attitude to automa-

tion and tools? We acknowledge that future research

could compare software testers with other knowledge

workers to see if the phenomena we collectively ob-

served over several decades are part of a general pat-

tern or are speciﬁc to this group.

We present the concept of TX: The Testers’ Lived

Experiences of Tools and Automation which we re-

gard as important both to individual testers and to

the software testing community, and important to the

software industry as a whole. It is a concept built

from a complex mix of technical, organisational, us-

ability, UX and human factors, which informs our un-

derstanding of the outcomes of attempts to automate

testing, and provides a potentially rich seam of re-

search in the HCI area and includes multidisciplinary

strands across sociology, psychology and organisa-

tional change management.

ACKNOWLEDGEMENTS

The authors thank the HUCAPP paper reviewers for

their useful and stimulating comments. We also

thank the expert testers and automators we inter-

viewed, and the test practitioners who took part in

our surveys. We thank the organisers of the fol-

lowing conferences for allowing us to collect data

at their events: UKSTAR, EuroSTAR, STARWEST,

STAREAST, STARCanada, Women Who Test. We

thank the ANZTB for enabling us to run a partic-

ipants workshop in New Zealand, and also the UK

Software Testing Retreat for discussing our research

and interim results.

REFERENCES

Bach, J. and Bolton, M. (2016). A context

driven approach to automation in testing.

https://www.satisﬁce.com/download/a-context-

driven-approach-to-automation-in-testing.

Bainbridge, L. (1983). Ironies of automation. In Analysis,

design and evaluation of man–machine systems, pages

129–135. Elsevier.

Baxter, G. D., Rooksby, J., Wang, Y., and Khajeh-Hosseini,

A. (2012). The ironies of automation: still going

strong at 30? In ECCE, pages 65–71.

Bertolino, A. (2007). Software testing research: Achieve-

ments, challenges, dreams. In 2007 Future of Software

Engineering, pages 85–103. IEEE Computer Society.

Bolton, M. (2016). A context driven

approach to automation in testing.

https://www.developsense.com/blog/2016/01/a-

context-driven-approach-to-automation-in-testing.

F. Lehner, R. D. and Abran, A. (2013). Software Met-

rics: Research and practice in software measurement.

Springer.

Fewster, M. and Graham, D. (1999). Software test automa-

tion: Effective use of test execution tools. Addison-

Wesley.

Gamba, S. and Graham, D. (2018a). A Journey through Test

Automation Patterns: One teams adventures with the

Stuck in Limbo with Magical Solutions: The Testers’ Lived Experiences of Tools and Automation

201

Test Automation Patterns wiki. CreateSpace Indepen-

dent Publishing Platform.

Gamba, S. and Graham, D. (2018b). Test automation pat-

terns. http://testautomationpatterns.org.

Graham, D. and Fewster, M. (2012). Experiences of test

automation: Case studies of software test automation.

Addison-Wesley.

Groover, M. (2019). Automation.

https://www.britannica.com/technology/automation.

Hackman, J. R. and Oldham, G. R. (1974). The job diag-

nostic survey: An instrument for the diagnosis of jobs

and the evaluation of job redesign projects.

Harty, J. (2011). Finding usability bugs with automated

tests. Communications of the ACM, 54(2):44–49.

Hendrikson, E. (2010). Do testers have to write code?

http://testobsessed.com/2010/10/testers-code/.

ISO (2018). Iso 9241-11:2018 ergonomics

of human-system interaction - part 11:

Usability: Deﬁnitions and concepts.

https://www.iso.org/obp/ui/iso:std:iso:9241:-11:ed-

2:v1:en.

ISTQB (2018). Downloads - istqb

 interna-

tional software testing qualiﬁcations board.

https://www.istqb.org/downloads/send/51-

ctﬂ2018/208-ctﬂ-2018-syllabus.html.

Johnson, D. (2011). Is automated test-

ing replacing the software tester?

https://searchsoftwarequality.techtarget.com/tip/Is-

automated-testing-replacing-the-software-tester.

Jones, C. (2015). Wastage: The impact of poor quality on

software economics. Software Quality Professional,

18(1).

Kaner, C. (1988). Avoiding shelfware: A

managers’ view of automated gui testing.

http://www.kaner.com/pdfs/shelfwar.pdf.

Lambert, R. (2014). Why testers really should learn to

code. http://thesocialtester.co.uk/why-testers-really-

should-learn-to-code/.

Martin, J. (2017). Using automation to assist –not replace

– manual testing. https://smartbear.com/blog/test-

and-monitor/using-automation-to-assist-not-replace-

manual-test.

Norman, D. and Nielsen, J. (2019). The

deﬁnition of user experience (ux).

https://www.nngroup.com/articles/deﬁnition-user-

experience/.

Porter, C. (2015). Designing for experience-a requirements

framework for enrolment based and public facing e-

government services. PhD thesis, University College

London.

Rachel, L. (2017). Test automation can’t replace

manual testing. https://dzone.com/articles/why-test-

automation-cannot-replace-manual-testing.

Reckless, C. (2017a). So what is software testing?

https://www.ministryoftesting.com/dojo/lessons/so-

what-is-software-testing.

Reckless, C. (2017b). Software testing

shouldn’t require expert coding skills.

https:// www.ministryoftesting.com/ dojo/ lessons/

software-testing-shouldn-t-require-expert-coding-

skills.

Reid, S. (2015). Motivated or motivating? what sort

of tester are you? http://www.stureid.info/home-

page/white-papers/motivation-testing-roles.

Tassey, G. (2002). The Economic Impacts of Inadequate In-

frastructure for Software Testing: Final Report. Diane

Publishing Company.

Toledo, F. (2017). Don’t try to tell me you can automate

the testing. https://dzone.com/articles/dont-try-to-tell-

me-you-can-automate-the-testing.

Warden, R. and Nicholson, I. (1996). Motivational survey

of it staff. Software Futures.

Wiklund, K. (2015). Impediments for Automated Software

Test Execution. PhD thesis, M

alardalen University.

HUCAPP 2020 - 4th International Conference on Human Computer Interaction Theory and Applications

202