Human-centered Artificial Intelligence: A Multidimensional

Approach towards Real World Evidence

Bettina Schneider, Petra Maria Asprion and Frank Grimberg

Competence Center Cyber Security and Resilience, Institute of Information Systems,

University of Applied Sciences and Arts Northwestern Switzerland FHNW, Peter Merian-Str. 86, Basel, Switzerland

Keywords: Real World Data, Real World Evidence, Artificial Intelligence, Systems Dynamics, Health-related Quality of

Life, Multidimensional Approach, Human-centered Perspective.

Abstract: This study indicates the significance of a human-centered perspective in the analysis and interpretation of

Real World Data. As an exemplary use-case, the construct of perceived ‘Health-related Quality of Life’ is

chosen to show, firstly, the significance of Real World Data and, secondly, the associated ‘Real World

Evidence’. We settled on an iterative methodology and used hermeneutics for a detailed literature analysis to

outline the relevance and the need for a forward-thinking approach to deal with Real World Evidence in the

life science and health care industry. The novelty of the study is its focus on a human-centered artificial

intelligence, which can be achieved by using ‘System Dynamics’ modelling techniques. The outcome – a

human-centered ‘Indicator Set’ can be combined with results from data-driven, AI-based analytics. With this

multidimensional approach, human intelligence and artificial intelligence can be intertwined towards an

enriched Real World Evidence. The developed approach considers three perspectives – the elementary, the

algorithmic and – as novelty – the human-centered evidence. As conclusion, we claim that Real World Data

are more valuable and applicable to achieve patient-centricity and personalization if the human-centered

perspective is considered ‘by design’.

1 INTRODUCTION

The life science and health care industry is striving

for a higher degree of patient-centricity and

personalization. The necessary investments are

significant and expose healthcare systems worldwide

to high cost-pressure (Marwaha et al., 2018). To

address the explosive investments/cost growth,

‘outcome-based payment’ has emerged – as a very

promising pricing model. The model requires that any

payment/pricing is associated with the effectiveness

of a product for a dedicated patient.

Measuring the results of a patient treatment is

commonly carried out with the help of Randomized

Clinical Trials (RCTs) that take place in a highly

controlled and regulated laboratory environment

(Mahajan, 2015). It has been under discussion for

about ten years that an increasing portion of such

measurements can be achieved alternatively through

the analysis of data from the ‘real world’, the so-

called Real World Data (RWD). One idea is to

process the vast amounts of digital patient data with

cutting-edge technologies like Artificial Intelligence

(AI).

Unfortunately, results based on AI-analytics are

worthless if industry-specific regulatory bodies (e.g.,

the European Medicines Agency (EMA), the U.S.

Food and Drug Administration (FDA)) do not accept

them as evident data sources. There are a few

references from the regulatory bodies, which discuss

the application of RWD and the associated data

evidence, the so-called Real World Evidence (RWE).

The influential FDA elaborated as one of the first

institutions a leading guidance on how to prove RWE

and launched a framework for a RWE program (FDA,

2017, 2018). In Europe, the PRIority MEdicines

(PRIME) scheme recognizes electronic data from

patient registries or health records in order to identify

unmet medical needs (Davis et al., 2018). Both are

promising indications that regulators may permit the

use of RWD if RWE is proven (Marwaha et al.,

2018).

By definition, (treatment) effectiveness is the

extent to which an intervention produces beneficial

outcomes under ordinary day-to-day circumstances

Schneider, B., Asprion, P. and Grimberg, F.

Human-centered Artiﬁcial Intelligence: A Multidimensional Approach towards Real World Evidence.

DOI: 10.5220/0007715503810390

In Proceedings of the 21st International Conference on Enterprise Information Systems (ICEIS 2019), pages 381-390

ISBN: 978-989-758-372-8

381

(Khan et al., 2011). Whereas some therapeutic effects

− the ‘hard factors’ can be quantified quite easily,

other effects are difficult to assess – we call them the

‘soft factors’. Patterns for hard factors are for

example pulse, blood sugar or blood pressure. For the

soft factors, it is more difficult to extract patterns but

there exists a general accepted concept, that will be

used in this study: perceived ‘Health-related Quality

of Life’ (HRQoL) − a human-centered factor defined

as patients' perception of their own health status

(Asadi-Lari et al., 2004; Jin et al., 2008).

From the affected patient’s perspective, soft

factors, like perceived HRQoL are crucial, because

they largely determine the essence of life (Asadi-Lari

et al., 2004; Jin et al., 2008). However, determining

perceived HRQoL is a challenging endeavour as it

can be driven by multiple individual sources and

factors, such as age, sex, type of disease or

personality (Bengtsson et al., 2018; Ekundayo et al.,

2018).

At this point, AI can be used to analyse sets of

RWD to quantify perceived HRQoL. However, can

an algorithm provide evidence about human´s

essence of life? From this question, we derive our

hypothesis that AI alone is not sufficient to create

evidence for perceived HRQoL. That is why we

postulate a multidimensional approach, which

combines AI with human-centered intelligence.

The objectives of this study are (1) as a pre-

condition to point out a suitable definition for RWD

and RWE, (2) to stress the relevance and to sketch a

multidimensional approach for analysing RWD to

determine perceived HRQoL, (3) to guide future

research by developing a systematic, method-based

procedure and putting forward a research agenda.

In terms of research methodology, first we settled

on an iterative approach and used hermeneutics for a

detailed literature review. For this, we used

recommendations of Boell and Cecez-Kecmanovic

(2014) as well as Tranfield et al. (2003). The repeated

steps of searching, acquiring, analysing, and

interpreting were focused on science databases like

Web of Science (all journals), and Google Scholar

(top journals). Then, we enriched the findings by

adapting knowledge, practical experience and work

from regulatory bodies. The main queries we

combined to find relevant sources related to the topics

‘Real World Data’, ‘Real World Evidence’, ‘Big

Data’ (as RWD is a specific set of big data) in

combination with ‘Health Care’, ‘Life Science’,

‘Pharmaceutical Industry’, ‘Artificial Intelligence’

and ‘Systems Dynamics’ . The last term is chosen as

technique to disclose human-centered perspectives.

The remainder of the paper is structured as

follows. In chapter 2, we discuss key terms and

causalities of RWD and RWE. Chapter 3 summarizes

challenges and risks associated with RWD. The need

for the multidimensional approach is outlined in

chapter 4. Chapter 5 sketches the components of the

developed procedure and the associated ‘Indicator

Set’ derived with Systems Dynamics techniques.

Lastly, chapter 6 concludes the results and shows

further research intentions.

2 BACKGROUND

Data may be regarded as factual, for example in the

form of figures, percentages or statistics. Evidence is

data of relevance, which additionally demonstrates

that it supports a particular conclusion. For RWD this

means that specific data sets may be relevant, but not

mandatorily sufficient to prove RWE. Therefore,

there is a need to separate RWD and RWE to ensure

exploitability, handling and compliance criteria of

RWE itself. This has important implications on how

RWD and RWE will be interpreted by regulators and

accepted in the course of RCTs.

In the context of life science and health care,

RWD can be defined as data relating to patients’

health status. In addition, RWD refers to data on the

delivery of healthcare that is commonly retrieved

from a variety of sources (FDA, 2017). This includes

data elements captured in a patient’s electronic health

record (EHR) in a hospital or in an insurance

company. It entails data on claims processes as well

as data collected directly from patients or various

providers in the course of an observational study.

Aside from clinical settings, the definition extends to

self-generated patient data (e.g., in-home monitoring

devices, wearable technologies, fitness trackers) and

data from registries that support various aspects of

care studies and research (FDA, 2017). It may also

include data on contextual metrics, such as patient’s

exposure and socio-economic indicators (WHO,

2010; Padilla et al., 2016). Importantly, this baseline

definition does not preclude the incorporation of

routinely collected data based on RCTs (Berger et al.,

2017a).

In contrast, Hubbard and Paradis (2015) defined

RWE as evidence derived from RWD through

application of research methods. RWE can further be

defined as clinical evidence regarding the use and

potential benefits or risks of a medical product

derived from RWD analysis (FDA, 2017).

According to Berger et al. (2017a) RWE is not

simply ‘anecdotes’ based on RWD – it involves data

ICEIS 2019 - 21st International Conference on Enterprise Information Systems

382

curation, validation, and standardization to ensure

that the data themselves are adequately ‘fit-for-

purpose’. It requires thoughtful study designs to

assess the effects of the treatments on the outcomes

of interest, and an understanding of the context, in

which the treatments are used.

Berger et al. (2017a) additionally emphasize that

the outlined definition of RWE reflects evidence

generation that is broader than passively collected

observational data and retrospective analytical

approaches. It conceptually enables the prospective

capture of a wider variety of data, and utilization of

study designs that are embedded in clinical practice

but retain randomization. This definition of RWE

does not characterize good versus bad evidence and

does not specify what ‘kind’ of RWE is suitable for

regulatory decisions. Therefore, rigorous RWE

should be able to provide insight into questions that

are difficult, infeasible, unethical, or cost-prohibitive

when addressed with traditional RCTs.

Once approval has been obtained from the

regulatory authorities, both RWD and the associated

RWE can contribute to a safer and more effective

patient profile. Such a profile with proven RWE is

increasingly valuable for patients and providers

compared to evidence only available from traditional

RCTs. Both – RWD and RWE – can be developed

through applications that capture information of

patient-related data and evidence for decision-making

and labelling (Berger et al., 2017a; Bipartisan Policy

Center, 2016).

To conclude the discussion, figure 1 visualizes the

considerations for generating RWE, which are ‘fit for

a specific (regulatory) purpose’.

3 CHALLENGES AND RISKS

Even though the application of RWD – as a

complementary or even substituting approach to

classical RCTs – holds great potential, numerous

hurdles need to be overcome. For example, the lack

of clear guidance on the inquiry and use of RWD and

associated evidence may lead to biased conclusions

with potential of adverse consequences for decision-

making regarding the efficacy and safety of new and

promising health technologies (Berger et al., 2017a).

RWD are based on real, everyday conditions of

individuals and their exploitation was inconceivable

just a few years ago. Today, for example sensor data

from fitness bands/apps or from social media

platforms are created daily in incredible quantity and

variety. The interest of the life science and health care

industry in pioneering for RWD is closely linked to

the search for alternative ways of developing and

approving new drugs, not least in order to have

methods that also allow research into drugs for rare

diseases.

Promising opportunities, however, come along

commonly with risks associated in the area of

patients’ interests with corresponding highly

regulated processes. Despite increasing recognitions

for the value of RWD and even though there are

definitions (chapter 2), a common understanding and

a harmonized body of language in the field of RWD

and RWE are lacking (Makady et al., 2017a).

As elaborated in chapter 2, a well-accepted

definition refers to RWD as data collected in a non-

RCT setting. A considerable number of definitions

diverge from this concept and frequently there is no

official or institutional definition for RWD in use.

This may lead to potential issues when decision-

making is based on RWD (Makady et al., 2017a).

Figure 1: Fit for purpose RWE (adopted from Berger et al., 2017a).

Human-centered Artiﬁcial Intelligence: A Multidimensional Approach towards Real World Evidence

383

Further, policies for the use of RWD notably

differ across contexts and agencies. Such variations

might discourage the application of RWD for drug

approval (Makady et al., 2017b). According to the

World Health Organization’s (WHO) global

observatories for eHealth from 2015, only 17% of the

member states surveyed (i.e. 21 of 125) enforced a

policy or strategy to regulate the use of big data in

their health sectors (WHO, 2016).

Currently, there are multiple public and private

efforts to digitize and aggregate health information

from e.g., administrative claims, EHR, or laboratory

tests. However, whereas these RWD promise insights

that are more robust into what works in health care,

there are various impediments. Most important in this

respect is the facilitation of greater openness among

public and private stakeholders to collaboration,

connecting information and data sharing, with the

goal of making robust data accessible to all

researchers (Berger et al., 2015).

There are a number of issues when collecting

RWD, for example the lack of good quality, sufficient

representative or complete databases, the presence of

many asymptomatic cases in RWD, more chances of

bias and confounding in prospective real-world

studies (Mahajan, 2015).

Not least, regulatory burdens from the European

Union (EU), the General Data Protection Regulation

(GDPR) must be taken into account (European

Union, 2016). This law entered into force in 2018 and

has global reach and implications with respect to how

companies manage and share personal data after

collection. This means companies need to establish

strict procedures for handling personal data; an active

data protection, e.g., to deal with the ‘right to be

forgotten’ (GDPR, article 17) needs to be established.

Nevertheless, good procedural practices are emerging

for RWD, which strengthen decision makers’

confidence in the related evidence (Berger et al.,

2017b).

Based on the discussions around RWD, we

aggregated the associated risks into three areas and

put them in relation to the expected extent of industry

transformation (figure 2). The aggregated risk areas

visualized in figure 2 are ‘Compliance

Controversies’, ‘Registration Failure’, and ‘Business

Model Disruption’. In the following, we discuss

significant differences and dependencies with regard

to the tree risk areas.

Compliance Risks. The inner layer of figure 2 deals

with the regulatory requirements − data-driven RWD

compliance. It is essential for the life science and

health care industry to familiarize itself with

upcoming requirements and recommendations of

regulatory bodies to prove, establish and audit the

company-specific conformity; this includes e.g.,

monitoring of potential contractors. The use of RWD

without sound knowledge and continuous monitoring

(governance) carry the risk of punishable compliance

violations.

Registration Risks. The middle layer of figure 2

emphasizes that the use of RWD can lead to

adjustments in the approval processes of drugs and in

the design of clinical trial setups. Companies

preparing to leverage RWD effectively, including

RWE's evidence will be able to respond proactively

to changes in the near future and surely gain

competitive advantage. In conclusion, the focus here

is set on process changes in the development and

approval of new drugs and/or therapies.

Business Model Risks. The outer layer of figure 2

shows that RWD have the potential to disrupt (not

only) the life science and health care industry. RWD

will develop into a critical success factor: Those in

possession of RWD and able to prove RWE and, in

addition, have the knowledge and competence for

their evaluation will probably dominate the market in

the near future. This area focuses on potential new

competitors (e.g., Apple and its Smart Watch)

entering the market as well as new disruptive business

models in the life sciences and health care industry.

Figure 2: Risk areas of RWD.

In conclusion, our focus is on regulatory and legal

requirements for RWD protection of personal data

and proof of data integrity to provide RWE. The latter

is subject to special attention by the FDA (FDA,

2016, 2017, 2018), United Kingdom Medicines and

Healthcare Products Regulatory Agency (MHRA),

and various other regulatory bodies. This means that

at the latest in an audit which is scoped to RWE,

companies need to provide information e.g., based on

ICEIS 2019 - 21st International Conference on Enterprise Information Systems

384

technical justification and corresponding scientific

rationale to prove that their analytical results based on

RWD are compliant (related to conceivably various

regulatory requirements).

4 MULTIDIMENSIONAL

APPROACH

The use of RWD promises enormous potential for

many areas of the life sciences and health care

industry (Greenfield, 2017); there are clear chances

that it may possibly disrupt the industry in the wake

of potential new competitors, like Apple (e.g., heart

study conducted by Apple (2018)).

Considerable research work has just been initiated

focusing on the use and analysis of RWD in health

care. AI is the technique of choice in many studies to

analyse the large amount of RWD. RWD resp. big

data in health care is generally regarded as crucial for

building (new) models of disease progression and

improved efficiency (cost effectiveness) of existing

clinical trial setups (Vayena et al., 2018).

In certain areas, the application of AI algorithms

has already outperformed experienced health care

professionals. One example is the identification of

skin cancer based on dermatologic image recognition

(Haenssle et al., 2018). Unfortunately, the use of AI

entails unsolved disadvantages. As an example, AI

algorithms are criticized for their ‘black box’ results,

not giving insights into the pathways/algorithms that

lead from input data to output results (Forbes, 2018).

Another, still unresolved challenge regarding AI

based results is the correct understanding of ‘soft

factors’ related to human´s behaviour: AI algorithms

are not foreseen to (properly) interpret figurative uses

of human´s language such as metaphors and irony.

Also the assignment of meaning to symbols or

behaviour is a challenge − there is a hidden side of

language and communication that requires at least

extra-linguistic knowledge (Moreno and Redondo,

2016; Lu et al., 2018).

According to Moreno and Redondo (2016),

figurative language is used to about 20% in social

media conversations. This means that about 20% of

the language is (at least) currently impossible to be

interpreted via AI techniques.

To conclude, while AI is extremely powerful in

extracting statistically significant patterns from data,

there are serious limitations to whole-brain functions

and associated underlying meanings and perceptions.

Despite this serious deficiency, many innovations

today rely heavily on automated data analytics, not

exclusively but more and more often, based on AI

algorithms, without considering the fact that data

evidence can be achieved only with a combination of

hard and soft factors. Figure 3 shows the

dependencies between hard and soft factors, in

particular the state of RWE (based mainly on hard

factors) and future perspectives with increasing

consideration of soft factors.

Figure 3: Hard and Soft factors of RWE.

For the life science and healthcare industry, the

abstract construct of perceived HRQoL is a

substantial criterion for evaluating the impact of

products and treatments. As an example, perceived

HRQoL relies on RWD and is strongly dependent on

RWE. It is mandatory to question the methodologies

and techniques used to collect RWD and to show

RWE, which lead to future decisions. Therefore, we

postulate that approaches that focus on the analysis of

RWD and associated RWE need to incorporate the

strengths of artificial and human intelligence ‘by

design’. Such a multidimensional method will

significantly enhance the use of RWDs for the

following reasons:

 Artificial Intelligence (AI) – is strong in

revealing correlations and extracting

statistically valid patterns, whereas

 Human Intelligence (HI) – is strong in

revealing causalities by creating system-related

sense-making and contextual scenarios,

considering symbols or behaviour.

The two complementary approaches have the

potential to cross-fertilize and replenish each other.

The results can be developed iteratively, so that

outcomes based on AI and HI are merged and finally

reach a level where it is possible to measure the

effectiveness of treatments beyond the hard factors

(related to AI), stretching out to a validated analysis

Human-centered Artiﬁcial Intelligence: A Multidimensional Approach towards Real World Evidence

385

of soft factors (related to HI). With this approach, for

example the concept of perceived HRQoL can be

supported with an increased validity of the analysed

RWD.

Finally, our multidimensional approach aims to

achieve three levels of evidence, which we refer to

‘elementary’, ‘algorithmic’ and ‘human-centered‘

evidence of RWD. The elementary level addresses

the challenges to ensure accuracy, consistency and

completeness of the data collected. The algorithmic

evidence is achieved with the help of AI algorithms,

whereas the human-centered evidence will rely on a

method, which puts humans in the center. For the last

one, we chose ‘System Dynamics’ as the appropriate

methodology which is introduced in the following

chapter. Figure 4 shows the emerging level of

evidence over time by applying our suggested

multidimensional or multilevel approach.

Figure 4: Levelled Multidimensional Approach.

In the following chapter we explain each

component of our approach, we enrich the use of AI

in combination with RWD providing a more systemic

and holistic perspective. Our novel multidimensional

approach will be reflected on the derived indicators

necessary to gather information about perceived

HRQoL.

5 COMPONENTS OF APPROACH

While the first level of our multidimensional

approach establishes the basis for further research

steps, the subsequent levels for building evidence

(algorithmic and human-centered levels) need to be

performed iteratively.

5.1 Elementary Evidence

The basis for establishing elementary evidence is to

assure a governed data quality. The prerequisite is the

access to trustworthy and comprehensible RWD.

It is of high relevance that the selected RWD is of

sufficient quality, with the consequence that data

assurance needs to be proven using procedures that

are subject to a recognised procedural guideline based

on regulatory requirements. This results in an

‘assurance quality seal‘, which verifies in particular

the accuracy, consistency and completeness of the

relevant RWD − a key when dealing with patient-

centered material. From an audit perspective, to

achieve an assurance quality seal for RWD, accurate

and traceable data management and related

governance procedures is a prerequisite. In a first

step, evidence criteria for the assurance quality seal

need to be determined. The evidence criteria will be

used for a RWE assessment. The results determine

whether the selected RWD source can be used (‘go’)

or if data quality improvement procedures must be

performed, or if another set of data needs to be

selected and audited (‘no go’).

The result is an assurance quality seal, the

elementary RWE that builds the foundation for

subsequent steps; we categorize this level of RWE as

‘maturity-level-1’ or elementary evidence (figure 4).

5.2 Algorithmic Evidence

The tempting idea of AI aims to simulate human (-

like) intelligence within machines, more specifically

computerized systems. This is termed as ‘general AI’,

which includes the replication of human emotions and

reasoning. By contrast, ‘narrow AI’ is used to

describe technologies that conduct specific tasks

similarly, or even better, than humans (Jones et al.,

2018). Even though AI has recently gained a lot of

attention, the idea and term was coined in 1955

(McCarthy et al., 1955).

A closely related approach is ‘Machine Learning’

(ML) which refers to ‘the study of computer

algorithms that can learn complex relationships or

patterns from empirical data and make accurate

decisions.’ (Jones et al., 2018). ML methods can be

divided in a.) supervised learning and b.)

unsupervised learning. While a.) implies the need to

train the rules and models based on existing

knowledge (e.g. training data, structures), b.) does not

rely on predefined data or structures (Moreno and

Redondo, 2016).

A very prominent group of AI algorithms are the

Neural Networks (NN) (Jones et al., 2018). The

example outlined in chapter 3 in which AI was used

for dermatologic image recognition is based on

convolutional NNs. There are currently numerous

software solutions available that offer numerical

high-performance calculations (e.g., the open source

ICEIS 2019 - 21st International Conference on Enterprise Information Systems

386

software library ‘TensorFlow’ and the application

programming interface ‘Keras’).

Another branch of AI is Natural Language

Processing (NLP), which supports to understand,

learn, interpret and produce human language content.

NLP draws from many disciplines, including

computer science and computational linguistics; it is

the ‘art’ to manage the understanding between human

communication and computers. NLP supports

human-human communication, human-machine

communication, or both by analysing learning and

producing content from a large quantity of data

(Hirschberg and Manning, 2015). In addition, NLP

can be used to extract information from unstructured

text such as clinical notes, or RWD from patient’s

interest groups (Murff et al., 2016; Jiang et al., 2017).

The initial part for algorithmic evidence is built

on indicators that analyse and interpret structured as

well as unstructured RWD. As previously elaborated,

the shortcomings of AI require the validation and

further elaboration of the indicators on which the

applied algorithms rely on. This is important to judge

if the AI-based results considered both - hard and soft

factors - to achieve the desired level of RWE.

Because our developed approach has an iterative

character, the indicators and data-driven explanations

discovered at level 2 - the algorithmic evidence - will

serve as initial input for level 3 (section 5.3);

consequently, the level-3-output will be returned for

subsequent iterations of (supervised) learning based

on AI algorithms. With this approach, we create a

complementary linkage of level-2- and level-3-

outcomes, which will leverage the RWE. Results of

the level-2- activities are the ‘maturity-level-2-’ or

algorithmic evidence: AI-generated indicators

subsequently can be used as input for the level-3-

exercise to achieve the desired level of human-

centered evidence (see figure 4).

5.3 Human-centered Evidence

As already outlined, the causalities of social

interactions are indispensable to understand complex

situations (e.g., language metaphors, irony, symbols,

signs, behaviour). Therefore, as a complementary

technique to the algorithmic evidence we considered

the ‘Systems Dynamics’ (SD) technique as sufficient

to collect human-centered indicators. The

methodology itself is generic and can be applied to

various other contexts (Van den Belt, 2004); for

example in management research and practice (Lane,

1992; Repenning, 2002; Rudolph et al., 2009).

The idea of the SD technique is a ‘systems thinking’

based analysis, which takes a step back from the level

of single events and attempts to develop structural

explanations of system behaviour.

So-called ‘Causal Loop Diagrams’ (CLDs) - shown

exemplarily in figure 5 - are used to describe feedback

loops; core building blocks of CLDs are variables and

causal relationships between them (von Kutzschen-

bach et al., 2018).

Every loop represents a feedback system, whereas

the loops can be categorized as either positive

/reinforcing (labelled as ‘R’) or negative/balancing

(labelled as ‘B’). The causal relationships between

the variables of a system are indicated as links -

visualized as arrows. Our example in figure 5 shows

that the variable ‘Growing Action’ is expected to

increase (+) variable ‘Results’, and vice versa.

However, the changes of ‘Results’ are expected to

increase (+) the third variable named ‘Slowing

Action’, which is expected to decrease (−) ‘Results’

again. The ‘II’ sign indicates assumed time delays

between causes and effects. The loops spoil the

distinction between the driver and the driven, cause

and effect, because, as time progresses, each variable

plays both roles. All loops together show the overall.

Figure 5: Generic Example of a CLD Diagram (adopted from Kutzschenbach et al., 2018).

Human-centered Artiﬁcial Intelligence: A Multidimensional Approach towards Real World Evidence

387

systems behaviour (von Kutzschenbach et al., 2018).

These SD modelling techniques can be applied to

validate the previously generated set of AI indicators.

The data-driven approach from level 2 is expected to

lead to explanations - indicators - that would be used

on level 3 to design the variables and links of a CLD.

This CLD would represent a system explaining the

behaviour of perceived HRQoL and its influencing as

well as influenced variables

Appreciating the complex and fuzzy nature of the

perceived HRQoL, the AI-approach (level 2) will be

validated and enhanced by ‘Group Model Building’

(GMB).

When applying SD, GMB is a proven way of

engaging multi-stakeholder perspectives in the

development of causal-loop diagrams and simulated

dynamic model (Scott et al., 2016).

With these activities, we create a dynamic model,

which will be refined and corrected until it is

saturated. This is a sense-making, human-centric and

collaborative action; with the GMB approach, we are

able to involve various stakeholders with different

perspectives and experiences. The outcome, the ‘SD

AI indicator set’, is grounded on a dynamic SD

model, which allows simulations of different

scenarios. The worked out ‘SD AI indicator set’ must

be regularly merged with the data-driven insights

based on AI algorithms (level 2); the associated

governance processes must be additionally defined.

The result of the level-3-activities is the human-

centered evidence, which means a set of AI-generated

indicators, which are complementarily and iteratively

enriched with the GMB-generated indicators of

HRQoL.

6 CONCLUSION AND OUTLOOK

At the beginning of this study, we claimed that the

exclusive use of AI to analyse and interpret RWD and

to achieve reasonable RWE for a selective dataset is

not sufficient. We showed (figure 1) that the ‘fit for

purpose’ to obtain RWE is depending on various

factors like regulatory, clinical, technology and

society perspectives as well as other considerations,

in particular data quality and the sufficiency of the

used and combined methods.

Thereafter, major challenges associated with the

use of RWD have been categorized by three areas -

compliance, registration and business model risks

(figure 2). Compliance risks are data-related and refer

to challenges along the assurance and governance of

RWD. Registration risks refer to product-related

challenges that might occur due to adapted approval

and development procedures once RWD become an

accepted means to prove drug effectiveness. Finally,

business model risks have been pointed out as RWD

have the potential to disrupt the life science and

healthcare industry.

Next, we addressed current trends to analyse

RWE with the help of AI techniques when trying to

demonstrate RWE. We concluded that, due to

shortcomings of current data-driven techniques, there

is a need for a multidimensional approach. We

selected and applied the concept of perceived HRQoL

to discuss our novel approach towards RWE.

Unlike pure hard factors (relatively easy to

analyse and measure with AI techniques), the

construct of perceived HRQoL is mostly determined

by soft factors - which are difficult to analyse and

quantify. Soft factors are not measurable via AI

exclusively; however, it is our claim that the soft

factors will be increasingly considered in future

decision-making related to RWE (see figure 3).

Thus, our developed and proposed approach has

the potential to contribute to one of the major

challenges of NLP - the ‘soft side’ of text and human

perceptions. Existing techniques mainly rely on text

fragments in which opinions/sentiments are explicitly

expressed (e.g., polarity terms and their co-

occurrence frequencies) (Cambria et al., 2016).

As an agenda for further research, a feasibility

study to test and apply the suggested procedures is

desired. As unit of analysis, patient interest group

data from a dedicated therapeutic area could be used.

In the scope of such a future study, the authors would

aim to limit the focus on assessing perceived HRQoL.

The design, development and validation of four

artefacts with relation to figure 4 is being considered:

1. Development of a quality assurance level (a

‘quality seal’) for the elementary evidence for

RWD resulting in a maturity-level-1-evidence.

2. AI-generated indicators to provide perceived

HRQoL algorithmic evidence (level 2) and as

input for human-centered evidence resulting in a

maturity-level-2-evidence.

3. ‘SD AI indicator set’ - a set of robust indicators

gathered via SD technique and passed back to the

AI-generated indicators resulting in a maturity-

level-3-evidence.

4. ‘RWE Framework’ - a prototypical frame-work,

which contains all relevant steps to achieve RWE

for a selective set of RWD – perceived HRQoL −

associated with the ‘SD AI indicator set’.

These results could trigger a multiplier effect and

form the basis for future research. First, the concept

of using a three-layered multidimensional procedure

ICEIS 2019 - 21st International Conference on Enterprise Information Systems

388

and an evidence providing quality seal can be used in

various AI related contexts to improve AI based

results, not at least in providing RWD evidence.

Second, the created artefacts could be generalized for

a broader use. More specifically, the concept of a

combined methodically sound set of AI indicators

based on hard- and soft factors could become a

standard approach for the AI discipline. Third, the

‘RWE Framework’ could be applied on further RWD

sources to show data evidence in other areas.

REFERENCES

Apple, 2018. Heart Study in collaboration with Stanford

Medicine. Available at http://med.stanford.edu/

appleheartstudy.html.

Asadi-Lari, M., Tamburini, M., Gray, D., 2004. Patients’

needs, satisfaction, and health related quality of life:

Towards a comprehensive model. Health and Quality

of Life Outcomes, 2(32).

Bengtsson, M., Wall, G., Larsson, P., Becktor, J.P.,

Rasmusson, L., 2018. Treatment outcomes and patient-

reported quality of life after orthognathic surgery with

computer-assisted 2- or 3-dimensional planning: A

randomized double-blind active-controlled clinical

trial. American Journal of Orthodontics and

Dentofacial Orthopedics, 153(6), pp. 786-796.

Berger, M., Craig, L., Gutteridge, A., Axelsen, K., Subedi,

P., Madigan, D., 2015. Optimizing the Leveraging of

Real-World Data to Improve the Development and Use

of Medicines. Value in Health, 18(1), pp. 127-130.

Berger, M., Overhage, M., Daniel, G., Platt, R., Frank, K.,

Romine, M., Hernandez, A., Tunis, S., McClelland, M.,

Wilson, M., Okun, S., 2017a. A Framework for

regulatory use of real-world evidence, White paper,

Center for Health Policy at Duke University. Available

at https://healthpolicy.duke.edu/sites/default/files/atom

s/files/rwe_white_paper_2017.09.06.pdf.

Berger, M.L., Sox, H., Willke, R., Brixner, D., Eichler, H.-

G., Goettsch, W., Mullins, C.D., 2017b. Recomm-

endations for Good Procedural Practices for Real-

World Data Studies of Treatment Effectiveness and/or

Comparative Effectiveness Designed to Inform Health

Care Decisions: Report of the Joint ISPOR-ISPE

Special Task Force on Real-World Evidence in Health

Care Decision Making, Pharmacoepidemiology &

Drug Safety, 26, pp. 1033-1039.

Bipartisan Policy Center, 2016. Using real-world evidence

to accelerate safe and effective cures. Advancing

Medical Innovation for a Healthier America, White

Paper. Available at https://cdn.bipartisanpolicy.org/

wp-content/uploads/2016/06/BPC-Health-Innovation-

Safe-Effective-Cures.pdf.

Boell, S.K., Cecez-Kecmanovic, D., 2014. A hermeneutic

approach for conducting literature reviews and

literature searches. Communications of the Association

for Information Systems, 34 (12), pp. 257-286.

Cambria, E., Poria, S., Bajpai, R., Schuller, B., 2016.

SenticNet 4: A Semantic Resource for Sentiment

Analysis Based on Conceptual Primitives. Proceedings

of COLING 2016, the 26th International Conference on

Computational Linguistics: Technical Papers, Osaka,

Japan, pp. 2666–2677.

Davis, B., Morgan, J., Shah, S., 2018. Mission Critical.

Biopharma companies are accelerating real-world

evidence adoption, investment, and application. A

report by the Center for Health Solutions. Available at

https://www2.deloitte.com/content/dam/ insights/ us/ar

ticles/4354_Real-World-Evidence/DI_Real-World-Evi

dence.pdf.

Ekundayo, O., Edwards, N., Bansal, A., Ali, A., Tang, E.,

Xu, A., Li, V., Novak, M., Mucsi, I., 2018. Patient

Reported Outcome Measures Information System

(PROMIS) domains explain a large portion of variance

in quality of life in advanced chronic kidney disease,

Journal of Psychosomatic Research, 109, pp. 100-101.

European Union, 2016. European Union General Data

Protection Regulation, GDPR, EU-Regulation

2016/679. Available at https://gdpr-info.eu/.

FDA, 2016. Use of real-world evidence to support

regulatory decision-making for medical devices: Draft

guidance for industry and Food and Drug

Administration staff. Paper published by U.S.

Department of Health and Human Services Food and

Drug Administration. Available at www.fda. gov/down

loads/medicaldevices/deviceregulationandguidance/gu

idancedocuments/ucm513027.pdf.

FDA, 2017. Use of Real-World Evidence to Support

Regulatory Decision-Making for Medical Devices.

Guidance for Industry and Food and Drug

Administration Staff. Paper published by U.S.

Department of Health and Human Services Food and

Drug Administration. Available at www.fda.gov/

BiologicsBloodVaccines/GuidanceComplianceRegulat

oryInformation/Guidances/default.htm.

FDA, 2018. Framework for FDA’s REALWORLD

EVIDENCE PROGRAM. Available at www.fda.gov/do

wnloads/ScienceResearch/SpecialTopics/RealWorldE

vidence/UCM627769.pdf.

Forbes, 2018. Don't Trust Artificial Intelligence? Time To

Open The AI 'Black Box'. Available at www.

forbes.com/sites/jasonbloomberg/2018/09/16/dont-

trust

-artificial-intelligence-time-to-open-the-ai-black-box/#

3353c7533b4a.

Greenfield, S., 2017. Making Real-World Evidence More

Useful for Decision Making. Value in Health, 20(8), pp.

1023-1024.

Haenssle, H. A., Fink, C., Schneiderbauer, R., Toberer, F.,

Buhl, T., Blum, A., Kalloo, A., Ben Hadj Hassen, A.,

Thomas, L., Enk, A., Uhlmann, L., 2018. Man against

machine: diagnostic performance of a deep learning

convolutional neural network for dermoscopic

melanoma recognition in comparison to 58

dermatologists. Annals of Oncology, 29(8), 1836-1842.

Hirschberg, J., Manning, C.D., 2015. Advances in natural

language processing. Science, 349(6245), pp. 261-266.

Human-centered Artiﬁcial Intelligence: A Multidimensional Approach towards Real World Evidence

389

Hubbard, T.E., Paradis, R., 2015. Real World Evidence: A

New Era for Health Care Innovation. Available at

www.nehi.net/writable/publication_files/file/rwe_issue

_brief_final.pdf.

Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang,

Y., Dong, Q., Shen, H., Wang, Y., 2017. Artificial

intelligence in healthcare: Past, present and future.

Stroke and vascular neurology, 2(4), pp. 230–243.

Jin, J., Sklar, G.E., Min Sen Oh, V., Chuen Li, S., 2008.

Factors affecting therapeutic compliance: A review

from the patient’s perspective. Therapeutics and

Clinical Risk Management, 4(1), pp. 269–286.

Jones, L.D., Golan, D., Hanna, S.A., Ramachandran, M.,

2018. Artificial intelligence, machine learning and the

evolution of healthcare. Bone & Joint Research, 7(3),

pp. 223–225.

Khan, K., Kunz, R., Kleijnen, J., Antes, G., 2011.

Systematic reviews to support evidence-based

medicine, 2nd edition. Medicine on the Move: Taylor

& Francis Group.

Lane, D.C., 1992. Modelling as learning: A consultancy

methodology for enhancing learning in management

teams. European Journal of Operational Research,

59(1), pp. 64–84.

Lu, H., Li, Y., Chen, M., Kim, H., Serikawa, S., 2018. Brain

Intelligence: Go beyond Artificial Intelligence. Mobile

Networks and Applications, 23(2), pp. 368–375.

Mahajan, R., 2015. Real world data: Additional source for

making clinical decisions. International journal of

applied & basic medical research, 5(2), p. 82.

Makady, A., de Boer, A., Hillege, H., Klungel, O.,

Goettsch. W., 2017a. What Is Real-World Data? A

Review of Definitions Based on Literature and

Stakeholder Interviews. Value in Health, 20(7), pp.

858–865.

Makady, A., Ham, R.T., de Boer, A., Hillege, H., Klungel,

O., Goettsch, W., 2017b. Policies for Use of Real-

World Data in Health Technology Assessment (HTA):

A Comparative Study of Six HTA Agencies. Value in

Health, 20(4), pp. 520–532.

Marwaha, S., Ruhl, M., Shorkey, P., 2018. Doubling

Pharma Value with Data Science. Boston Consulting

Group. Available at http://image-src.bcg.com /Images

/BCG-Doubling-Pharma-Value-with-Data-Science-Fe

b-2018_tcm9-184455.pdf.

McCarthy, J., Minsky, M.L., Rochester, N., Shannon, C.E.,

1955. A Proposal for the Dartmouth Summer Research

Project on Artificial Intelligence. Available at www-

formal.stanford.edu/jmc/history/Dartmouth/dartmouth.

html.

Moreno, A., Redondo, T., 2016. Text analytics: The

convergence of big data and artificial intelligence,

IJIMAI, 3(6), pp. 57-64.

Murff, H.J., FitzHenry, F., Matheny, M.E., Gentry, N.,

Kotter, K.L., Crimin, K., Dittus, R.S., Rosen, A.K.,

Elkin, P.L., Brown, S.H., Speroff, T., 2016. Automated

identification of postoperative complications within an

electronic medical record using natural language

processing. Available at www.semanticscholar.org/

paper/Automated-identification-of-postoperative-with

in-an-Murff-FitzHenry/7c299e9ffc6980283b119aff54

84834fcfe25aec.

Padilla, C.M., Kihal-Talantikit, W., Perez, S., Deguen, S.,

2016. Use of geographic indicators of healthcare,

environment and socioeconomic factors to characterize

environmental health disparities. Environmental

health: a global access science source, 15(1), p. 79.

Repenning, N.P., 2002. A simulation-based approach to

understanding the dynamics of innovation

implementation. Organization Science, 13(2), pp. 109–

127.

Rudolph, W.J., Morrison, B., Carroll, J.S., 2009. The

dynamics of action-oriented problem solving: Linking

interpretation and choice. Academy of Management

Review, 34(4), pp. 733–756.

Scott, R.J., Cavana, R.Y., Cameron, D., 2016. Recent

evidence on the effectiveness of group model building.

European Journal of Operational Research, 249(3), pp.

908-918.

Tranfield, D., Denyer, D., Smart, D., 2003. Towards a

methodology for developing evidence-informed

management knowledge by means of systematic

review. British Journal of Management, 14(3), pp. 207–

222.

Van den Belt, M. (Ed.), 2004. Mediated modeling: A system

dynamics approach to environmental consensus

building. Island Press. Washington, D.C.

Vayena, E., Dzenowagis, J., Brownstein, J.S., Sheikh, A.,

2018. Policy implications of big data in the health

sector. Bulletin of the World Health Organization,

96(1), pp. 66–68.

von Kutzschenbach, M., Schmid, A., Schoenenberger, L.,

2018. Using Feedback Systems Thinking to Explore

Theories of Digital Business for Medtech Companies.

In Business Information Systems and Technology 4.0.

Studies in Systems, Decision and Control, Springer.

Cham.

WHO, 2010. Environment and health risks: a review of the

influence and effects of social inequalities, World

Health Organization, Regional Office for Europe,

Geneva. Available at www.euro.who.int/data/assets/

pdf_file/0003/78069/E93670.pdf.

WHO, 2016. Global diffusion of eHealth: making universal

health coverage achievable. The third global survey on

eHealth [Internet], Geneva. Available at www.who.in

t/goe/publications.

ICEIS 2019 - 21st International Conference on Enterprise Information Systems

390