Evaluating Healthcare Automation: A Multi-Case Study on the

Utilization of Automation Initiatives in Healthcare Operations

Jani Kaitosalmi

and Milla Ratia

Department of Industrial Engineering and Management, Aalto University, Otakaari 1B, Espoo, Finland

Keywords: Healthcare Automation, Impact Assessment, Robotic Process Automation, Intelligent Automation.

Abstract: Automation technologies such as robotic process automation (RPA) and intelligent automation (IA) are

essential for managing rising healthcare costs and ensuring sustainable health services. Although these

solutions have been implemented in several Finnish healthcare organizations, their overall impact has not

been systematically evaluated. This research investigates the impact and evaluation of healthcare automation

through a multi-case study conducted in two Finnish healthcare organizations. While automation has

improved resource utilization, process efficiency, and standardization across units, the findings highlight the

need for a more comprehensive evaluation and continuous monitoring of automation benefits. Future research

should focus on developing a specific evaluation framework tailored to healthcare automation technologies.

The adoption of holistic evaluation methods could allow healthcare organizations to better understand the

impact of automation and further enhance operational efficiency and patient care.

1 INTRODUCTION

Mestres (2017) posits that healthcare faces three key

challenges: rising costs, a decrease in available

physicians, and an increase in patients. Similar trends

are noted in Finland, with increasing healthcare costs

and declining availability of personnel (Kirkonpelto,

Mäntyranta, et al., 2023). Digitalization offers

solutions by improving productivity, transforming

care delivery, and simplifying administration (Sony,

Antony & Tortorella, 2023). Although digitalization

can help healthcare organizations meet their

objectives, the outcomes of digital interventions often

take time to materialize and can be challenging to

measure (Cresswell, 2023).

This paper focuses on one specific avenue of

digitalization in healthcare – the impact of automating

digital workflows. Automation initiatives can deliver

a wide range of benefits to healthcare organizations

through cost savings, standardized and always-

available processes, and freeing employees from

repetitive manual tasks (Kedziora and Smolander,

2022; Ratia et al., 2021; Kedziora and Kiviranta,

2018).

Although automation solutions are recognized for

their potential, it remains unclear how these back-

https://orcid.org/0009-0009-1800-5137

https://orcid.org/0000-0002-3360-9555

office solutions generate value in the public

healthcare sector (Ratia et al., 2021). In particular,

there is a lack of relevant research on the impact

assessment of automation (Meironke and Kuehnel,

2022), with discussions being "ad-hoc and scattered,

with minimal empirical and theoretical support"

(Denagama Vitharanage et al., 2020). The research on

automation solutions often focuses on the technical

dimensions, giving less emphasis on the benefits and

value creation (Ratia et al., 2021). Various patient-

centric attempts have been made to create an overall

impact assessment framework for digitalization

efforts (e.g. WHO, 2016; Lillrank et al., 2019;

Parviainen et al., 2017; Karunasena & Deng, 2012).

However, research on automated back-office

solutions that do not directly affect patient outcomes

has been more limited. Especially in the Finnish

public sector, systematic evaluations of automation

technologies are limited (Kääriäinen et al., 2018).

This research aims to examine the outcomes of

automation solutions in healthcare and provide

information on how their impact can be measured.

The study focuses on two methods: robotic process

automation (RPA) and intelligent automation (IA),

applied within two Finnish public healthcare districts:

The Wellbeing Services County of Pirkanmaa

246

Kaitosalmi, J. and Ratia, M.

Evaluating Healthcare Automation: A Multi-Case Study on the Utilization of Automation Initiatives in Healthcare Operations.

DOI: 10.5220/0012947100003838

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2024) - Volume 3: KMIS, pages 246-255

ISBN: 978-989-758-716-0; ISSN: 2184-3228

(PIRHA) and the Helsinki and Uusimaa Hospital

District (HUS). Although both healthcare

organizations had implemented various automation

tools, they had not fully assessed the overall impact

of these solutions. The objective of this paper is two-

fold: to identify key automation benefits and potential

risks and to provide information on what should be

considered when evaluating automation outcomes.

The research questions are the following:

RQ1. What are the benefits and risks of

utilizing automation solutions in healthcare?

RQ2. What should be considered when

evaluating automation in healthcare?

2 BACKGROUND LITERATURE

2.1 Utilizing Automation to Enhance

Healthcare Operational Processes

Workflow automation, which involves identifying

sequences of tasks that can be streamlined with digital

tools, provides opportunities to address process

inefficiencies in healthcare (Zayas-Cabán, Okubo, &

Posnack, 2023). Automating simple workflows is

needed in healthcare, where professionals are

burdened by repetitive administrative duties like data

entry, documentation, and scheduling – tasks that

often distract from patient care. Automation

technologies can take over these routine tasks,

allowing healthcare providers to dedicate more time

to meaningful patient interactions. (Mohamed &

Frank, 2022.)

Often automating business processes require the

use of application programming interfaces (API) to

connect with legacy systems (Herm et al., 2023; Syed

et al., 2020). However, such solutions are not always

feasible in healthcare due to technical constraints,

legacy systems, or vendor limitations. In these cases,

lightweight and non-invasive automation solutions,

such as RPA and IA, can present an opportunity to

improve operational processes across care delivery

and administration (Ferris, Ackers & Borhani, 2022;

Zayas-Caban, Okubo & Posnack, 2023).

RPA automates repetitive digital tasks typically

performed by humans (Ivančić, Suša Vugec, and

Bosilj Vukšić, 2019; Kääriäinen et al., 2018;

Willcocks, Lacity, and Craig, 2015; Ratia et al.,

2021). These robots mimic human actions on system

interfaces (Herm et al., 2023). Due to its low cost,

quick implementation, and minimal system changes,

RPA is attractive for healthcare providers struggling

with system integration (Ratia et al., 2021;

Osmundsen et al., 2019; Kedziora & Kiviranta,

2018).

RPA is effective in automating routine processes,

such as physician credentialing, patient scheduling,

and billing (Jain and Bhatnagar, 2019). For example,

during COVID-19, RPA accelerated diagnoses,

distributed targeted health information, and updated

quarantine data (Doğuç, 2021). In Finland, RPA was

used to process up to 2,000 COVID-19 vaccinations

daily, a task previously done by 10-15 employees

(Adolfsson, 2021). Similarly, in HUS, RPA saved

over 13,000 workdays in 2021, equivalent to the work

of 65 full-time employees (HUS, 2021).

While RPA is a powerful tool, its applications are

primarily limited to tasks that are highly rule-based,

structured, standardized, and supported by well-

documented decision logic (Ng, 2021). IA enhances

RPA with AI features such as machine learning and

natural language processing to replicate human

cognitive skills (Kedziora and Hyrynsalmi, 2023;

Coombs et al., 2020). With these capabilities, IA can

handle unstructured data, make real-time decisions,

and perform content-aware computing, overcoming

many limitations of RPA (Ng et al., 2021).

IA can assist in complex scheduling, capacity

management, and process optimization in operating

rooms and emergency departments by managing

patient movement between diagnostics and wards

(Garcia et al., 2020). Additionally, IA can coordinate

patient information, issue health risk alerts, predict

health outcomes, and optimize logistics processes

(Secinaro et al., 2021). However, IA research is still

in its early stages with challenges and potential

applications yet to be fully explored (Ng, 2021).

2.2 Impact Evaluation of Automation

Initiatives

Porter (2013) defines healthcare value as

“maximizing value for patients,” meaning achieving

the best outcomes at the lowest cost. In other words,

healthcare value = patient outcomes / total costs.

Specifically, Porter (2014) argues that while process

measurement and improvements are valuable tactics,

they cannot replace the importance of measuring the

results of care. Thus, healthcare value is the result of

care, not the volume of services delivered. What is

then the impact of back-office automation solutions

that do not directly affect patient outcomes?

To measure the overall healthcare outcomes, the

primary metric used is effectiveness, which refers to

the impact of treatment on the health conditions of a

population under routine conditions (Ikonen, 2019;

Pitkänen et al., 2018). From effectiveness, several

Evaluating Healthcare Automation: A Multi-Case Study on the Utilization of Automation Initiatives in Healthcare Operations

247

other important metrics are derived, such as cost-

effectiveness, which evaluates the ratio of

effectiveness to costs, and productivity, which is the

ratio of output (the result of an operation) to input (the

resources or costs used to achieve that output)

(Sintonen et al., 2021). Productivity can be enhanced

by either reducing costs while keeping the output

constant or by increasing output while maintaining

the same costs. Consequently, improving

productivity also enhances the cost-effectiveness of

healthcare processes (Sintonen et al., 2021).

Automation is often implemented to improve

cost-effectiveness by enhancing productivity.

Automation benefits can be tangible, like reduced

costs and faster processing, or intangible, such as

better customer satisfaction and employee

motivation. Evaluating both types separately is

crucial since not all benefits, like financial gains, are

easily measurable (Axmann and Harmoko, 2021).

While the automation benefits are recognized,

their impact assessment is still lacking (Meironke &

Kuehnel, 2022; Kääriäinen et al., 2018). According to

Kääriäinen et al. (2018), organizations tend to assess

automation mainly using a narrow set of criteria, with

a focus on internal savings. Many of the assessment

frameworks focus on the requirements, feasibility,

and readiness of a company to implement new

technology, such as the 5D digital technology

assessment (Axmann and Harmoko, 2021), RPA

selection indicators by Kim (2023), or the method for

RPA process selection proposed by Wanner et al.

(2019). However, they do not consider how to

conduct continuous monitoring and evaluate the

impact after the implementation.

Moreover, Meironke and Kuehnel (2022)

identified 62 unique metrics in the literature to

evaluate the benefits of automation. Most of these

metrics focus on efficiency and costs, emphasizing

the number of transactions, work hours saved, and

process time. Accuracy or error rates are also

commonly used to measure quality and compliance

benefits. On the other hand, metrics related to

implementation effort, employee and customer

satisfaction, availability, and interoperability are less

commonly found. Thus, the authors conclude that the

assessment of automation benefits “shows a

tendency” to prioritize quantifiable economic metrics

over qualitative and non-economic ones. (Meironke

and Kuehnel, 2022.)

3 METHODOLOGY

This research aims to provide insights into evaluating

automation outcomes. To address the research

questions, the study has focused on the benefits, risks,

and evaluation challenges of automation in two

Finnish healthcare districts: PIRHA and HUS.

The research was conducted as a multi-case

qualitative study. This study employed a qualitative

design to explore a relatively novel topic, which is

ideal for gaining an in-depth understanding of

phenomena within specific contexts when little is

known about a topic (Saunders et al., 2019; Antwi &

Hamza, 2015). Moreover, the research employed an

embedded multi-case study design; it contained more

than one sub-unit of analysis to conduct an in-depth

examination of a current phenomenon (the ‘case’)

within its real-life setting (Yin, 2018). The method

allowed to investigate stakeholders' experiences with

various automation implementations across multiple

units. The two case organizations were ideal for the

study due to their size and previous experience with

multiple automation projects. Furthermore, while

both healthcare organizations had implemented

various automation solutions, they had not fully

assessed their overall impact.

3.1 Data Collection

The empirical data was gathered through 32 semi-

structured interviews with employees, administrative

staff, and stakeholders in PIRHA and HUS

experienced with RPA and IA. Participants included

department secretaries, nurses, doctors, pharmacists,

digitalization experts, and head physicians. External

IT companies providing automation solutions were

also interviewed to capture their perspectives. Table

1 lists all participants.

The interview data was gathered in two phases:

from PIRHA in Spring 2023 and HUS in Autumn

2023. PIRHA interviews focused on RPA solutions,

while HUS interviews covered IA processes.

Purposive sampling was used, meaning the

participants were selected based on characteristics

that matched the research objectives (Andrade, 2021;

Etikan et al., 2016). Specifically, participants had

prior experience with automation and represented a

diverse set of roles. The research team provided

preferences for roles and units but had limited control

over participant selection. Due to the policies of both

organizations, the final participant selection was

made within a tight timeframe by a designated contact

person. Consequently, convenience and availability

also played a significant role in recruitment.

KMIS 2024 - 16th International Conference on Knowledge Management and Information Systems

248

All interviews followed a similar structure and

were conducted remotely via Microsoft Teams.

Participants discussed automation benefits and risks

from the perspectives of employees, patients, costs,

and processes and shared their views on impact

assessment. The interviews were recorded,

transcribed, and pseudonymized with participants'

permission before analysis.

While the interviewees discussed several

automated processes, the primary focus was on two

processes: referral handling and medical dosage

building. Both organizations had implemented a

referral sorter to shift manual work from nurses and

department secretaries. “Before automation or the

electronic patient record system's XDS archive, the

paper referrals arriving at the unit were placed on the

doctor’s desk,” as one secretary from HUS describes

it (Ahlskog, 2022). Now, the robot continuously

processes new doctors' referral texts in a virtual

referral center. It categorizes them into the correct

queues, where doctors can access and review the

referrals for further evaluation. In PIRHA, the RPA-

based solution uses predefined logic to automate

repetitive referral handling. However, if the robot

misplaces a referral, an employee must correct it.

Moreover, HUS has enhanced the referral sorter with

AI, using machine learning to interpret symptoms and

diagnoses from referral texts.

HUS has also an automation solution for filling

patients' dosage information in the electronic health

record system, Apotti. This medical dosage builder,

used by nurses and pharmacists, transforms free-text

medication info into the required structured format. It

provides code suggestions, reducing the need for

manual entry of the patient’s medication list. The

structured medical information is standardized data

that is used to generate dosage instructions with a

similar structure for all patients, reducing the need for

manual input and minimizing errors.

3.2 Data Analysis

Data analysis was conducted using qualitative content

analysis with the QDA software ATLAS.ti. The study

followed a directed content analysis approach

presented by Hsieh and Shannon (2005), starting with

relevant research findings and preliminary theory-

based categories for automation benefits and risks.

The empirical data consisted of two datasets, one for

each case, with data collection divided among the

researchers. Consequently, none of the researchers

participated in all interviews, resulting in a final

dataset that was a mix of primary and secondary data

for all researchers.

Given the mix of primary and secondary data, the

study employed tactics for an abductive content

analysis as outlined by Vila-Henninger et al. (2024).

Firstly, a deductive theory-based codebook was

created and expanded with inductive codes. For

instance, in this step, the benefit dimensions provided

by Meironke and Kuehnel (2022) were utilized to

categorize the benefits. The “AI coding” feature of

ATLAS.ti was also used for exploratory coding.

Secondly, to reduce data volume, codes were

combined into broader categories. Here, the ‘Query

Tool’ was used to sort codes based on the four

perspectives that were investigated. Finally, a

detailed manual qualitative analysis was performed to

identify emerging themes and compare differences

between the two cases.

Table 1: Participant list from PIRHA and HUS.

Region Role

Number of

artici

ants

PIRHA

Department Secretar

Nurse 2

Chief Physician / Directo

Medical Docto

Digitalization Specialist 2

Head / De

Head Nurse 2

Product Owner (External) 1

Midwife 1

Service Provider (External) 1

HUS

Pharmacist / Senior

Pharmacist

italization S

ecialist 2

Department Secretar

Nurse 1

Product Owne

Deputy Chief Physician 1

Product Owner (External) 1

Data Scientist (External) 1

Total 32

Evaluating Healthcare Automation: A Multi-Case Study on the Utilization of Automation Initiatives in Healthcare Operations

249

4 FINDINGS

4.1 Perceived Benefits

The thematic interviews in both healthcare counties

expressed that the overall perception of automation

has been positive. The most mentioned direct benefits

were related to process efficiency, improved resource

utilization, and availability of services.

In HUS, one external expert describes that the

referral processing time has decreased significantly,

from 32-35 hours to approximately 3-4 hours.

However, while quicker referral handling provides

information to patients faster, it doesn't necessarily

lead to faster treatment. More precisely, participants

describe that the referral sorter has merely moved the

bottleneck from the referral handling to the next step

of the patient journey. Thus, speeding up one part of

the process does not necessarily reduce overall

treatment time. However, faster referral handling can

be impactful for urgent patients. Participants describe

how automation robots process urgent cases faster

and thus increase the accessibility to treatment: “For

urgent referrals that take 1-7 days, it makes a

significant difference if they can be processed in 2-4

hours instead of five working days.”

While interviewees often mention time savings,

they struggle to pinpoint where the extra time goes.

Nurses and pharmacists note a slight increase in

patient care time, which enhances the overall quality

of the patient experience. However, these time

savings per patient inquiry are deemed small and hard

to measure. In HUS, doctors have faced additional

workload from the referral sorter, as incorrect

referrals have been directed to them. On the contrary,

in PIRHA, the effect has been more moderate. For

secretaries, the possible benefits seem most direct, as

they have more time for more challenging

administrative tasks, such as appointment scheduling

and other phone-related work.

Moreover, participants in HUS described that IA

solutions has created standardization across the

organization. Before automation, there were large

variations in how medication lists were written. As

automation requires rule-based inputs, it has

standardized the medical information, leading to

consistent interpretations across the organization.

“The fact that information is the same for all users.

Perhaps that’s the best benefit here. It also guides us

in standardizing practices across different areas of

healthcare,” describes one pharmacist. “This is a

massive organization, and with automation, we have

achieved greater unity across specialties,”

summarizes another participant.

Other process-related benefits were related to

compliance and interoperability. The solutions have

helped to meet reporting requirements and facilitated

data and system linkage. “The smoother we can make

reporting, the better,” describes one participant.

“From the perspective of organizational

development, it supports knowledge management,”

continues another interviewee. Moreover, while

automation robots act fast, they also handle

information without biases, positively impacting

equal access to treatment. “It’s not affected by

whether someone is in a bad mood, had a rough

morning, or is running late for work. Automation is

consistent; it doesn’t get tired or have biases.”

Automation solutions have also created indirect

benefits in both organizations, such as enhanced

employee and patient experience. The participants

describe that automation can help “find meaningful

job roles” and provide time for “brain-intensive work

that professionals are trained for.”

While cost savings are discussed, the perception

of it varies across units and roles. The digitalization

experts highlight the fast payback time and scalability

of automation solutions. “The time savings

accumulate when creating such easily scalable

processes. There are no additional costs with

expansion,” describes one digitalization expert. On

the contrary, nurses and secretaries find it hard to

evaluate cost savings, one reason being that some of

the potential saved time goes to monitoring and

correcting the results: “Any freed-up time currently

goes into fixing and monitoring the results. However,

in the long term, when hopefully everything is running

smoothly, I would see cost savings occurring,”

Lastly, participants note that patients likely don't

notice automation since it works in the background.

However, automation has positively impacted

patients in various ways, such as delivering treatment

information faster and allowing doctors more time for

patient care.

4.2 Recognized Risks

In both cases, the two main recognized potential risks

of automation were additional work created by

technical errors and decreased process quality. The

risk of additional work appears to be significantly

higher during what participants describe as the

“infancy stage,” meaning the early stages of

development. PIRHA reported significantly more

issues with its early-stage RPA solutions than HUS,

which had both more experience and more advanced

automation solutions implemented.

KMIS 2024 - 16th International Conference on Knowledge Management and Information Systems

250

Table 2: An overview of the perceived main benefits and

risks in HUS and PIRHA.

Direct benefits Description

Process efficiency

Increase in volumes and

decrease in throughput times

and delays.

24/7 availability

The robots are always

available to work.

Increased resource

utilization

Time savings enable

employees to complete more

complex tasks.

Standardization

Automation provides similar

out

uts across units.

Faster access to

treatment (urgent

cases)

Automation can prioritize

urgent cases.

Compliance

Helps to meet reporting

uirements.

Interoperability

Easier data and system

linkage.

Scalability

The solutions are fast and

cheap to scale across units.

Equal access to

treatment

Robots handle information

without biases.

Indirect benefits

Increased employee

erience

Increased job satisfaction and

meanin

fulness of work.

Increased patient

experience

Provides information faster to

the patient and can speed up

treatment visits.

Cost-effectiveness

Increase in the value of time

gains (difference in cost of

process by a human vs

automation

)

Potential risks

Additional work

Automation solutions are not

error-free. The risk of

additional work is higher

durin

the "infanc

sta

e."

Decrease in quality

Humans perform tasks more

accurately than robots.

Risks in patient safety

Automation faces challenges

in interpreting complex data.

Outputs involving critical

patient information still need

to be reviewed by a

rofessional.

While additional work is a risk, it also matters to

whom and how much of it is created. In PIRHA, most

of the additional work was targeted to secretaries and

nurses, some of whom felt that “it has consumed work

hours and taken away time from patient care.” The

participants in PIRHA describe that the RPA robot

should be able to handle 50 percent of the referrals to

be beneficial. For example, one participant noted a

success rate of only 27 percent during a two-week test

period. Some other interviewed secretaries and nurses

share similar experiences of automation feeling like

an “additional burden.” “There is still a substantial

amount of manual checking required. In the last

report, more than 3/4 of cases had errors,” describes,

for example, another participant. In HUS, while the

intelligent referral sorter appears to be more effective

than the regular RPA solution in PIRHA, it has also

created some extra work for the doctors. However, on

the organizational level, the overall effect has been

positive: “The doctors don’t see the advantage yet in

the referral handling work. However, when I try to

consider it from the perspective of the clinic’s

operation, I see that it creates a positive impact”,

describes one participant in HUS.

While automation increases quantity, it does not

necessarily increase quality. Participants underline

that humans still perform tasks with higher accuracy

than robots. Thus, there appears to be a trade-off

between quality and efficiency: “I don’t believe it has

improved quality. But it has shifted mechanical work

away from humans,” concludes one doctor. “The

process is less precise; errors occur more frequently

than with a human. However, the robot performs

faster and around the clock. That’s the trade-off,”

continues one external expert.

Lastly, interpreting data with changing inputs

appears to be challenging. Several participants

described that the dosage builder struggles with

situations where the medication dosage changes

within a time period. “Often if there is a variable

dosage, like one tablet in the morning and two in the

evening… It cannot handle such situations properly,

so it just makes a guess,” describes one pharmacist.

Thus, to maintain patient safety, the outputs need to

be verified carefully by a professional. Moreover, the

referral sorter in both case organizations has had

issues interpreting complex data, for instance,

sending the patient to a clinic outside their regular

municipality.

4.3 Challenges Measuring the Impact

of Automation

Measuring the impact of automation initiatives is

perceived hard in both case organizations. The main

assessment challenges relate to a lack of holistic data,

baseline measurements, targets, and a regular

evaluation process.

Active evaluation and monitoring have been

missing from many implementations in both

organizations. “Not in any way, at least not in our

unit,” is how one participant describes the current

state. “I don’t know if there’s any monitoring at the

Evaluating Healthcare Automation: A Multi-Case Study on the Utilization of Automation Initiatives in Healthcare Operations

251

PIRHA level, but at the unit level, there isn’t,”

continues another employee.

The assessment has mainly focused on whether

the technology works as intended, not on how people

interact with it or how it improves outcomes: “We

have tried to gather genuine user feedback, but it has

not been very successful. Essentially, what we

document is the result of encountering bugs."

Evaluating the impact of automation is also

challenging because it's “just a small part of it all,”

making it unclear which outcomes are directly caused

by automation. Feedback has mostly been technical

error reports: “We get these cold reports that tell us

how many times the robot has run and how many

cases there are per month or week.”

Interviewees in both organizations note that

evaluating progress is difficult without a baseline for

comparison. “We didn’t do measurements before the

start of this referral processing, such as how much

time we spend now and how much we used to spend.

Therefore, it’s a bit challenging to assess time

savings,” describes one secretary. In some cases,

potential benefits have been calculated in advance

without collecting data afterward. We haven’t had

any active monitoring. It’s been more like we’ve

calculated in advance the potential benefit and how

quickly the process would pay for itself,” summarizes

one digitalization expert.

While participants acknowledge the time savings,

it seems unclear how the saved time is utilized. “What

would interest me is somehow measuring how the

saved work time is being used. Whether it means

being able to serve one more patient or making more

phone calls,” describes one digitalization expert.

Measuring time savings appears challenging because

automation replaces only small specific tasks, not

entire workflows. “Measuring how much time

various small tasks take is always challenging," as

one digitalization expert describes. Secondly,

assessment efforts would require an additional layer

of monitoring, which would be difficult both

technically and due to possible resistance: “Adding

extra monitoring to the busy daily work might not be

the most effective solution."

In some units, monitoring is seen as a sensitive

issue. The work culture in these units seems to affect

how openly the benefits of automation are discussed:

“If you have a difficult work atmosphere, you don’t

go tell your boss you have more bandwidth. You enjoy

the fact that you have more space to do things. You

don’t report that you saved another 5 hours of work

time this week,” as one participant describes.

However, collecting impact data could help in

motivating employees to use the solutions: It would

likely turn even sceptical individuals towards a more

positive outlook.

4.4 Evaluation Metrics

Process metrics like time savings, throughput

volumes, and error rates are frequently mentioned,

especially those affecting treatment delivery, such as

handled referrals for urgent patients. At HUS, greater

emphasis is placed on qualitative metrics like worker

wellbeing and satisfaction, reflecting their more

mature solutions. In contrast, PIRHA focuses more on

technical aspects, as their solutions are in earlier

stages of development.

Back-office automation benefits patients

indirectly, such as faster referral processing or clearer

prescription instructions. Key metrics from the

patient’s perspective include patient safety,

satisfaction, treatment efficiency, and service

accessibility. Pharmacists and nurses using the

medical dosage builder particularly emphasize

patient safety as a crucial metric. Regarding the

referral sorter, the main patient risk is treatment

delays due to incorrect classification: “The biggest

risk we've identified is a delay in treatment due to

incorrect classification. However, the risk is very low,

and the consequences should not be significant.”

Measuring cost efficiency appears challenging,

particularly due to difficulties in quantifying time

savings. Cost savings may differ across units

depending on whose tasks are automated. For

instance, automating a doctor's tasks is more valuable

than a secretary's. However, organizations can

estimate time gains by "converting the saved time into

work hours and the hourly rate,” as one participant

summarizes. Additionally, automation enhances

organizational capabilities by increasing

standardization and simplifying scalability. This

organizational impact could be measured by metrics

like the number of standardized processes, speed of

implementation, and the number of employees

trained in automation.

5 DISCUSSION

The results emphasize the need for an impact

assessment model, as no proper "template or tool"

exists. Despite some used metrics, employees found

the value of implemented solutions unclear: “The

robot works well, but very few feel it’s needed in their

process.” In some cases, not seeing the benefits has

also caused resistance to adoption: “Some employees

KMIS 2024 - 16th International Conference on Knowledge Management and Information Systems

252

feel that they don’t want to adopt automation because

they perceive it as an additional burden.”

Based on both PIRHA and HUS interviews,

evaluation has been inadequate because a process has

not been built around it. Essentially, assessing digital

transformation should answer two questions: is the

project doing things right and is the project doing the

right things (Pritchett et al., 2013)

More concretely, impact assessment includes two

steps: monitoring implementation activities and

evaluating the monitored outcomes. Monitoring,

which refers to routine data collection, review, and

analysis, is the most time-consuming part of the

process (WHO, 2016). Evaluation, the systematic and

objective assessment of the implemented solutions, is

only the final step in determining whether objectives

have been met and the impact has been achieved

(WHO, 2016). In other words, evaluation aims to

determine whether changes in the monitored metrics

are the result of the digital intervention.

The findings from the two case organizations

emphasize six key steps in impact assessment: 1)

defining clear objectives, 2) choosing suitable

metrics, 3) setting baselines and targets, 4)

implementing monitoring mechanisms, 5) gathering

feedback, and 6) conducting regular evaluations.

Firstly, when setting goals, it is essential to

consider both the potential direct and indirect benefits

of automation. In HUS and PIRHA, the primary

advantages include operational efficiencies such as

time savings and productivity boosts, while indirect

benefits include reduced costs, improved

transparency, and enhanced experiences for patients

and employees. Automation outcomes that improve

the health system's effectiveness are mostly indirect

second-order effects, such as freeing up time for

patient interaction or more complex administrative

work.

The maturity of the solution should also be

considered for in the evaluation (WHO, 2016). In

other words, consider if the automation solution is

developed and evaluated for the first time or if it is

undergoing scale-up. For instance, PIRHA’s newer

RPA solutions had more errors than HUS's mature

systems, which requires evaluating them differently

depending on their development stage.

Moreover, not all metrics are relevant to every

context. More specifically, the impact can vary based

on the context in which they are implemented. Thus,

a full acknowledgment of the different perspectives

and boundaries is necessary (Williams 2015). More

precisely, organizations must broadly consider their

specific stakeholders’ perspectives: what goals are

relevant for whom and what is needed to measure.

Automation goals may not be achieved if

stakeholders are not engaged in setting the objectives

(Zayas-Caban et al., 2021).

Baseline measurements are essential for setting

realistic targets. Both PIRHA and HUS faced

challenges due to a lack of baseline data, as no initial

measurements were taken before implementation.

Inconsistent monitoring was also a major obstacle.

While technical data was collected, qualitative

insights have been lacking. Employees stressed the

need for gathering feedback to fully understand

automation's impact. Collecting feedback is

important, as unanswered quality or safety concerns

can undermine the long-term success of automation

initiatives (Zayas-Caban et al., 2021).

Lastly, evaluation should occur regularly and be

based on evidence collected at across multiple time

points (WHO, 2016). While interviews identified cost

efficiency and resource utilization as key factors for

evaluation, both organizations would benefit from

considering other dimensions. Similarly, Axmann

and Harmoko (2024) argue that traditional cost-

benefit analysis overlooks many automation benefits,

possibly leading to poor decisions. They propose a

balanced scorecard (BSC) framework to evaluate

RPA projects, categorizing benefits into four areas:

financial, process improvement, customer

satisfaction, and learning. With adjustments, the

model could potentially be applied to healthcare. The

BSC, originally developed by Kaplan and Norton

(1992), has already been used in healthcare, both with

original and modified perspectives (Amer et al., 2022;

Betto et al., 2022). Further research is needed to

determine whether BSC framework could effectively

evaluate healthcare automation outcomes.

6 CONCLUSIONS

This research explored the impact of automation

solutions in two Finnish healthcare districts,

providing insights into the benefits, risks, and

evaluation of automation initiatives in healthcare.

However, several limitations must be acknowledged.

Firstly, this study focused only on two regions with

varying levels of automation maturity, which may

limit the generalizability of the findings to other

healthcare environments. Additionally, the research

primarily gathered insights from employees familiar

with automation without including patients. Although

participants were asked to reflect on the impact of

automation on patients, no direct feedback was

collected from patients themselves.

Evaluating Healthcare Automation: A Multi-Case Study on the Utilization of Automation Initiatives in Healthcare Operations

253

The findings contribute to future research on

developing an impact assessment framework that

could help healthcare organizations better understand

and enhance the use of automation. Future research

should explore whether successful evaluation

frameworks from other industries can be adapted for

the healthcare sector.

REFERENCES

Adolfsson, F. (2021). Västerbotten sparar 100 000 timmar

med RPA. Voister.

Ahlskog, B. (2022). Ohjelmistorobotit ja tekoäly

lähetekäsittelyn apuna. Husari, 4/2022.

Amer, F., Hammoud, S., Khatatbeh, H., Lohner, S., Boncz,

I., & Endrei, D. (2022). A systematic review: the

dimensions to evaluate health care performance and an

implication during the pandemic. BMC Health Services

Research, 22(1), 621.

Antwi, S. K., & Hamza, K. (2015). Qualitative and

quantitative research paradigms in business research: A

philosophical reflection. European journal of business

and management, 7(3), 217-225.

Andrade C. (2021). The Inconvenient Truth About

Convenience and Purposive Samples. Indian journal of

psychological medicine, 43(1), 86–88.

Axmann, B., & Harmoko, H. (2021). The five dimensions

of digital technology assessment with the focus on

robotic process automation (RPA). Tehnički glasnik,

15(2), 267-274.

Betto, F., Sardi, A., Garengo, P., & Sorano, E. (2022). The

evolution of balanced scorecard in healthcare: A

systematic review of its design, implementation, use,

and review. International Journal of Environmental

Research and Public Health, 19(16), 10291.

Coombs, C., Hislop, D., Taneva, S. K., & Barnard, S.

(2020). The strategic impacts of Intelligent Automation

for knowledge and service work: An interdisciplinary

review. The Journal of Strategic Information Systems,

29(4), 101600.

Cresswell, K., Anderson, S., Montgomery, C., Weir, C. J.,

Atter, M., & Williams, R. (2023). Evaluation of

Digitalisation in Healthcare and the Quantification of

the “Unmeasurable.” Journal of General Internal

Medicine, 38(16), 3610-3615.

Denagama, V., Manuraji, I., Bandara, W., Syed, R., &

Toman, D. (2020). An empirically supported

conceptualisation of robotic process automation (RPA)

benefits. Association for Information System,

Marrakech.

Doğuç, Özge (2021). “Robotic process automation (RPA)

applications in COVID-19”. In: Dincer, H. & Yüksel,

S. (Eds.) Management Strategies to Survive in a

Competitive Environment: How to Improve Company

Performance (pp. 233–247). Springer.

Etikan, I., Musa, S. A., & Alkassim, R. S. (2016).

Comparison of convenience sampling and purposive

sampling. American journal of theoretical and applied

statistics, 5(1), 1-4.

Ferris, T., Ackers, J., and Borhani, M. (2022). RPA in the

NHS: Guidance for designing, delivering and

sustaining RPA within the NHS. May. NHS.

Garcia, J.F, Spatharou, A., Hieronimus, S., Beck, J.P., and

Jenkins, J. (2020). Transforming healthcare with AI.

McKinsey & Company: Our Insights.

Herm, L. V., Janiesch, C., Helm, A., Imgrund, F., Hofmann,

A., & Winkelmann, A. (2023). A framework for

implementing robotic process automation

projects. Information Systems and e-Business

Management, 21(1), 1-35.

Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to

qualitative content analysis. Qualitative health

research, 15(9), 1277-1288.

HUS (2021). Tarkastuslautakunnan arviointikertomus 2021

Helsingin ja Uudenmaan sairaanhoitopiirin

valtuustolle. April. HUS.

Ikonen, T (2019). Asiakasryhmäkohtainen tieto laadusta ja

vaikuttavuudesta sosiaali- ja terveydenhuollon tiedolla

johtamisessa ja ohjauksessa. Sosiaali-ja

terveysministeriön raportteja ja muistioita, 70.

Ivančić, L., Suša Vugec, D., & Bosilj Vukšić, V. (2019).

Robotic process automation: systematic literature

review. In Business Process Management: Blockchain

and Central and Eastern Europe Forum: BPM 2019

Blockchain and CEE Forum, Vienna, Austria,

September 1–6, 2019, Proceedings 17 (pp. 280-295).

Springer International Publishing.

Jain, R and R Bhatnagar (2019). Robotic process

automation in healthcare- a review. International

Robotics & Automation Journal 5(1), 12–14.

Kääriäinen, J., Aihkisalo, T., Halén, M., Holmström, H.,

Jurmu, P., Matinmikko, T., Seppälä, T., Tihinen, M.,

and Tirronen, J., (2018). Ohjelmistorobotiikka ja

tekoäly–soveltamisen askelmerkkejä. Valtioneuvoston

selvitys- ja tutkimustoiminnan julkaisusarja, 65/2018.

Kaplan, R. S. (1992). The Balanced Scorecard-Measures

That Drive Performance. Harvard Business Review.

Karunasena, K., & Deng, H. (2012). Critical factors for

evaluating the public value of e-government in Sri

Lanka. Government information quarterly, 29(1), 76-

84.

Kedziora, D., and Smolander, K., 2022: Responding to

Healthcare Emergency Outbreak of COVID-19

Pandemic with Robotic Process Automation (RPA),

Proceedings of the 55th Hawaii International

Conference on System Sciences, 6207-6216

Kedziora, D., & Hyrynsalmi, S. (2023, May). Turning

robotic process automation onto intelligent automation

with machine learning. Proceedings of the 11th

International Conference on Communities and

Technologies, 1-5

Kedziora, D., & Kiviranta, H. (2018). Digital Business

Value Creation with Robotic Process Automation (rpa)

in Northern and Central Europe. Management

(18544223), 13(2), 64- 174.

KMIS 2024 - 16th International Conference on Knowledge Management and Information Systems

254

Kim, S.H., (2023). Development of Evaluation Criteria for

Robotic Process Automation (RPA) Solution Selection.

Electronics 12(4), 986.

Lillrank, P., Tenhunen, H., Hörhammer, I., Halminen, O.,

Lyly, T., Linna, M., Silander, K., Laurila, R., Hiltunen,

A-M, Riikonen, E., Miettinen, S., Tanila, T., Chen, A.

& Vesinurm, M. (2019). Terveydenhuollon

digitaalisten ratkaisujen vaikuttavuuden osoittaminen.

HEMA-instituutti.

Meironke, A., & Kuehnel, S. (2022). How to measure

RPA’s benefits. A review on metrics, indicators, and

evaluation methods of RPA benefit assessment. 17th

International Conference on Wirtschaftsinformatik.

Mestres, J. C. (2017). State of the art of health care: The

cubism period. The digitization of healthcare: New

challenges and opportunities, 3-21.

Ng, K. K., Chen, C. H., Lee, C. K., Jiao, J. R., & Yang, Z.

X. (2021). A systematic literature review on intelligent

automation: Aligning concepts from theory, practice,

and future perspectives. Advanced Engineering

Informatics, 47 (101246).

Osmundsen, K., Iden, O. and Bygstad, B. (2019).

Organizing Robotic Process Automation: Balancing

Loose and Tight Coupling. Proceedings of the 52nd

Hawaii International Conference on System Sciences

2019. 6918-6926.

Parviainen, P., Kääriäinen, J., Honkatukia, J., and Federley,

M., (2017). Julkishallinnon digitalisaatio–tuottavuus ja

hyötyjen mittaaminen. Valtioneuvoston selvitysja

tutkimustoiminnan julkaisusarja, 3/2017.

Pitkänen, L., Haavisto, I., Vähäviita, P., Torkki, P., Leskelä,

L., & Komssi, V. (2018). Vaikuttavuus SOTE:ssa.

Suoritteista tuloksiin. Nordic Healthcare Group NHG.

Pritchett, L., Samji, S., & Hammer, J. S. (2013). It's all

about MeE: Using Structured Experiential Learning

('e') to crawl the design space. Center for Global

Development working paper, (322).

Ratia, M., Myllärniemi, J. and Helander, N. (2018). Robotic

process automation-creating value by digitalizing work

in the private healthcare? Proceedings of the 22nd

International Academic Mindtrek Conference, 222-227

Ratia, M., Myllärniemi, J. and Helander, N. (2021).

Intellectual Capital and Robotic Process Automation

(RPA)-a Capability Perspective. International Forum

on Knowledge Asset Dynamics, 1810-1824

Saunders, M., Lewis, P., Thornhill, A., and Bristow, A.,

(2019). “Research Methods for Business Students"

Chapter 4: Understanding research philosophy and

approaches to theory development”. In: Research

Methods for Business Students. Pearson Education, pp.

128–171.

Secinaro, S., Calandra, D., Secinaro, A., Muthurangu, V.,

& Biancone, P. (2021). The role of artificial intelligence

in healthcare: a structured literature review. BMC

medical informatics and decision making, 21, 1-23.

Sintonen, H., Blom, M., Roine, R. P., Räsänen, P., &

Ryynänen, O. P. (2021). Terveystaloustiede on tärkeää

lääkärille. Lääkärilehti, 76(39)

Sony, M., Antony, J., & Tortorella, G. L. (2023). Critical

success factors for successful implementation of

healthcare 4.0: A literature review and future research

agenda. International Journal of Environmental

Research and Public Health, 20(5), 4669.

Syed R, Suriadi S, Adams M, Bandara W, Leemans SJ,

Ouyang C., Hofstede AH, van de Weerd I, Wynn MT,

Reijers HA (2020). Robotic process automation:

contemporary themes and challenges. Computers in

Industry, 115, 103162.

Vila-Henninger, L., Dupuy, C., Van Ingelgom, V., Caprioli,

M., Teuber, F., Pennetreau, D., Bussi, M., & Le Gall,

C. (2024). Abductive coding: Theory building and

qualitative (re) analysis. Sociological Methods &

Research, 53(2), 968-1001.

Wanner, J., Hofmann, A., Fischer, M., Imgrund, F.,

Janiesch, C., & Geyer- Klingeberg, J. (2019). Process

selection in RPA projects-towards a quantifiable

method of decision making. 40th International

Conference on Information Systems, Munich.

WHO (2016). Monitoring and evaluating digital health

interventions: a practical guide to conducting research

and assessment. World Health Organization:

Publications.

Willcocks, L., Lacity, M. and Craig, A. (2015a). The IT

Function and Robotic Process Automation. The

Outsourcing Unit Working Research Paper Series.

15(05), 1-39.

Willcocks, L., Lacity, M. and Craig, A. (2015b). Robotic

Automation at Xchanging. The Outsourcing Unit

Working Research Paper Series, 15(03), 1-26.

Willcocks, L.P., Lacity, M.C., Craig, A. (2017). Robotic

Process Automation: Strategic Transformation Lever

for Global Business Services? Journal of Information

Technology Teaching Cases. 7, 17–28

Williams, Bob (2015). “Prosaic or profound? The adoption

of systems ideas by impact evaluation”. IDS Bulletin

46(1), 7–16.

Yin, R. K. (2018). Case study research and applications:

Design and Methods. (6

ed.). Sage Publication.

Zayas-Cabán, T., Okubo, T., and Posnack, S., (2023).

Priorities to accelerate workflow automation in health

care. Journal of the American Medical Informatics

Association 30(1), 195–201.

Evaluating Healthcare Automation: A Multi-Case Study on the Utilization of Automation Initiatives in Healthcare Operations

255