Human-centered Artificial Intelligence: A Multidimensional
Approach towards Real World Evidence
Bettina Schneider, Petra Maria Asprion and Frank Grimberg
Competence Center Cyber Security and Resilience, Institute of Information Systems,
University of Applied Sciences and Arts Northwestern Switzerland FHNW, Peter Merian-Str. 86, Basel, Switzerland
Keywords: Real World Data, Real World Evidence, Artificial Intelligence, Systems Dynamics, Health-related Quality of
Life, Multidimensional Approach, Human-centered Perspective.
Abstract: This study indicates the significance of a human-centered perspective in the analysis and interpretation of
Real World Data. As an exemplary use-case, the construct of perceived ‘Health-related Quality of Life’ is
chosen to show, firstly, the significance of Real World Data and, secondly, the associated ‘Real World
Evidence’. We settled on an iterative methodology and used hermeneutics for a detailed literature analysis to
outline the relevance and the need for a forward-thinking approach to deal with Real World Evidence in the
life science and health care industry. The novelty of the study is its focus on a human-centered artificial
intelligence, which can be achieved by using ‘System Dynamics’ modelling techniques. The outcome a
human-centered ‘Indicator Set’ can be combined with results from data-driven, AI-based analytics. With this
multidimensional approach, human intelligence and artificial intelligence can be intertwined towards an
enriched Real World Evidence. The developed approach considers three perspectives the elementary, the
algorithmic and as novelty the human-centered evidence. As conclusion, we claim that Real World Data
are more valuable and applicable to achieve patient-centricity and personalization if the human-centered
perspective is considered ‘by design’.
1 INTRODUCTION
The life science and health care industry is striving
for a higher degree of patient-centricity and
personalization. The necessary investments are
significant and expose healthcare systems worldwide
to high cost-pressure (Marwaha et al., 2018). To
address the explosive investments/cost growth,
‘outcome-based payment’ has emerged as a very
promising pricing model. The model requires that any
payment/pricing is associated with the effectiveness
of a product for a dedicated patient.
Measuring the results of a patient treatment is
commonly carried out with the help of Randomized
Clinical Trials (RCTs) that take place in a highly
controlled and regulated laboratory environment
(Mahajan, 2015). It has been under discussion for
about ten years that an increasing portion of such
measurements can be achieved alternatively through
the analysis of data from the ‘real world’, the so-
called Real World Data (RWD). One idea is to
process the vast amounts of digital patient data with
cutting-edge technologies like Artificial Intelligence
(AI).
Unfortunately, results based on AI-analytics are
worthless if industry-specific regulatory bodies (e.g.,
the European Medicines Agency (EMA), the U.S.
Food and Drug Administration (FDA)) do not accept
them as evident data sources. There are a few
references from the regulatory bodies, which discuss
the application of RWD and the associated data
evidence, the so-called Real World Evidence (RWE).
The influential FDA elaborated as one of the first
institutions a leading guidance on how to prove RWE
and launched a framework for a RWE program (FDA,
2017, 2018). In Europe, the PRIority MEdicines
(PRIME) scheme recognizes electronic data from
patient registries or health records in order to identify
unmet medical needs (Davis et al., 2018). Both are
promising indications that regulators may permit the
use of RWD if RWE is proven (Marwaha et al.,
2018).
By definition, (treatment) effectiveness is the
extent to which an intervention produces beneficial
outcomes under ordinary day-to-day circumstances
Schneider, B., Asprion, P. and Grimberg, F.
Human-centered Artificial Intelligence: A Multidimensional Approach towards Real World Evidence.
DOI: 10.5220/0007715503810390
In Proceedings of the 21st International Conference on Enterprise Information Systems (ICEIS 2019), pages 381-390
ISBN: 978-989-758-372-8
Copyright
c
2019 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
381
(Khan et al., 2011). Whereas some therapeutic effects
the ‘hard factors’ can be quantified quite easily,
other effects are difficult to assess we call them the
‘soft factors’. Patterns for hard factors are for
example pulse, blood sugar or blood pressure. For the
soft factors, it is more difficult to extract patterns but
there exists a general accepted concept, that will be
used in this study: perceived Health-related Quality
of Life (HRQoL) a human-centered factor defined
as patients' perception of their own health status
(Asadi-Lari et al., 2004; Jin et al., 2008).
From the affected patient’s perspective, soft
factors, like perceived HRQoL are crucial, because
they largely determine the essence of life (Asadi-Lari
et al., 2004; Jin et al., 2008). However, determining
perceived HRQoL is a challenging endeavour as it
can be driven by multiple individual sources and
factors, such as age, sex, type of disease or
personality (Bengtsson et al., 2018; Ekundayo et al.,
2018).
At this point, AI can be used to analyse sets of
RWD to quantify perceived HRQoL. However, can
an algorithm provide evidence about human´s
essence of life? From this question, we derive our
hypothesis that AI alone is not sufficient to create
evidence for perceived HRQoL. That is why we
postulate a multidimensional approach, which
combines AI with human-centered intelligence.
The objectives of this study are (1) as a pre-
condition to point out a suitable definition for RWD
and RWE, (2) to stress the relevance and to sketch a
multidimensional approach for analysing RWD to
determine perceived HRQoL, (3) to guide future
research by developing a systematic, method-based
procedure and putting forward a research agenda.
In terms of research methodology, first we settled
on an iterative approach and used hermeneutics for a
detailed literature review. For this, we used
recommendations of Boell and Cecez-Kecmanovic
(2014) as well as Tranfield et al. (2003). The repeated
steps of searching, acquiring, analysing, and
interpreting were focused on science databases like
Web of Science (all journals), and Google Scholar
(top journals). Then, we enriched the findings by
adapting knowledge, practical experience and work
from regulatory bodies. The main queries we
combined to find relevant sources related to the topics
‘Real World Data’, ‘Real World Evidence’, ‘Big
Data’ (as RWD is a specific set of big data) in
combination with ‘Health Care’, ‘Life Science’,
‘Pharmaceutical Industry’, ‘Artificial Intelligence’
and ‘Systems Dynamics’ . The last term is chosen as
technique to disclose human-centered perspectives.
The remainder of the paper is structured as
follows. In chapter 2, we discuss key terms and
causalities of RWD and RWE. Chapter 3 summarizes
challenges and risks associated with RWD. The need
for the multidimensional approach is outlined in
chapter 4. Chapter 5 sketches the components of the
developed procedure and the associated ‘Indicator
Set’ derived with Systems Dynamics techniques.
Lastly, chapter 6 concludes the results and shows
further research intentions.
2 BACKGROUND
Data may be regarded as factual, for example in the
form of figures, percentages or statistics. Evidence is
data of relevance, which additionally demonstrates
that it supports a particular conclusion. For RWD this
means that specific data sets may be relevant, but not
mandatorily sufficient to prove RWE. Therefore,
there is a need to separate RWD and RWE to ensure
exploitability, handling and compliance criteria of
RWE itself. This has important implications on how
RWD and RWE will be interpreted by regulators and
accepted in the course of RCTs.
In the context of life science and health care,
RWD can be defined as data relating to patients
health status. In addition, RWD refers to data on the
delivery of healthcare that is commonly retrieved
from a variety of sources (FDA, 2017). This includes
data elements captured in a patient’s electronic health
record (EHR) in a hospital or in an insurance
company. It entails data on claims processes as well
as data collected directly from patients or various
providers in the course of an observational study.
Aside from clinical settings, the definition extends to
self-generated patient data (e.g., in-home monitoring
devices, wearable technologies, fitness trackers) and
data from registries that support various aspects of
care studies and research (FDA, 2017). It may also
include data on contextual metrics, such as patient’s
exposure and socio-economic indicators (WHO,
2010; Padilla et al., 2016). Importantly, this baseline
definition does not preclude the incorporation of
routinely collected data based on RCTs (Berger et al.,
2017a).
In contrast, Hubbard and Paradis (2015) defined
RWE as evidence derived from RWD through
application of research methods. RWE can further be
defined as clinical evidence regarding the use and
potential benefits or risks of a medical product
derived from RWD analysis (FDA, 2017).
According to Berger et al. (2017a) RWE is not
simply ‘anecdotes’ based on RWD it involves data
ICEIS 2019 - 21st International Conference on Enterprise Information Systems
382
curation, validation, and standardization to ensure
that the data themselves are adequately fit-for-
purpose’. It requires thoughtful study designs to
assess the effects of the treatments on the outcomes
of interest, and an understanding of the context, in
which the treatments are used.
Berger et al. (2017a) additionally emphasize that
the outlined definition of RWE reflects evidence
generation that is broader than passively collected
observational data and retrospective analytical
approaches. It conceptually enables the prospective
capture of a wider variety of data, and utilization of
study designs that are embedded in clinical practice
but retain randomization. This definition of RWE
does not characterize good versus bad evidence and
does not specify what ‘kind’ of RWE is suitable for
regulatory decisions. Therefore, rigorous RWE
should be able to provide insight into questions that
are difficult, infeasible, unethical, or cost-prohibitive
when addressed with traditional RCTs.
Once approval has been obtained from the
regulatory authorities, both RWD and the associated
RWE can contribute to a safer and more effective
patient profile. Such a profile with proven RWE is
increasingly valuable for patients and providers
compared to evidence only available from traditional
RCTs. Both RWD and RWE can be developed
through applications that capture information of
patient-related data and evidence for decision-making
and labelling (Berger et al., 2017a; Bipartisan Policy
Center, 2016).
To conclude the discussion, figure 1 visualizes the
considerations for generating RWE, which are ‘fit for
a specific (regulatory) purpose’.
3 CHALLENGES AND RISKS
Even though the application of RWD as a
complementary or even substituting approach to
classical RCTs holds great potential, numerous
hurdles need to be overcome. For example, the lack
of clear guidance on the inquiry and use of RWD and
associated evidence may lead to biased conclusions
with potential of adverse consequences for decision-
making regarding the efficacy and safety of new and
promising health technologies (Berger et al., 2017a).
RWD are based on real, everyday conditions of
individuals and their exploitation was inconceivable
just a few years ago. Today, for example sensor data
from fitness bands/apps or from social media
platforms are created daily in incredible quantity and
variety. The interest of the life science and health care
industry in pioneering for RWD is closely linked to
the search for alternative ways of developing and
approving new drugs, not least in order to have
methods that also allow research into drugs for rare
diseases.
Promising opportunities, however, come along
commonly with risks associated in the area of
patients’ interests with corresponding highly
regulated processes. Despite increasing recognitions
for the value of RWD and even though there are
definitions (chapter 2), a common understanding and
a harmonized body of language in the field of RWD
and RWE are lacking (Makady et al., 2017a).
As elaborated in chapter 2, a well-accepted
definition refers to RWD as data collected in a non-
RCT setting. A considerable number of definitions
diverge from this concept and frequently there is no
official or institutional definition for RWD in use.
This may lead to potential issues when decision-
making is based on RWD (Makady et al., 2017a).
Figure 1: Fit for purpose RWE (adopted from Berger et al., 2017a).
Human-centered Artificial Intelligence: A Multidimensional Approach towards Real World Evidence
383
Further, policies for the use of RWD notably
differ across contexts and agencies. Such variations
might discourage the application of RWD for drug
approval (Makady et al., 2017b). According to the
World Health Organization’s (WHO) global
observatories for eHealth from 2015, only 17% of the
member states surveyed (i.e. 21 of 125) enforced a
policy or strategy to regulate the use of big data in
their health sectors (WHO, 2016).
Currently, there are multiple public and private
efforts to digitize and aggregate health information
from e.g., administrative claims, EHR, or laboratory
tests. However, whereas these RWD promise insights
that are more robust into what works in health care,
there are various impediments. Most important in this
respect is the facilitation of greater openness among
public and private stakeholders to collaboration,
connecting information and data sharing, with the
goal of making robust data accessible to all
researchers (Berger et al., 2015).
There are a number of issues when collecting
RWD, for example the lack of good quality, sufficient
representative or complete databases, the presence of
many asymptomatic cases in RWD, more chances of
bias and confounding in prospective real-world
studies (Mahajan, 2015).
Not least, regulatory burdens from the European
Union (EU), the General Data Protection Regulation
(GDPR) must be taken into account (European
Union, 2016). This law entered into force in 2018 and
has global reach and implications with respect to how
companies manage and share personal data after
collection. This means companies need to establish
strict procedures for handling personal data; an active
data protection, e.g., to deal with the ‘right to be
forgotten’ (GDPR, article 17) needs to be established.
Nevertheless, good procedural practices are emerging
for RWD, which strengthen decision makers’
confidence in the related evidence (Berger et al.,
2017b).
Based on the discussions around RWD, we
aggregated the associated risks into three areas and
put them in relation to the expected extent of industry
transformation (figure 2). The aggregated risk areas
visualized in figure 2 are Compliance
Controversies’, ‘Registration Failure’, and ‘Business
Model Disruption’. In the following, we discuss
significant differences and dependencies with regard
to the tree risk areas.
Compliance Risks. The inner layer of figure 2 deals
with the regulatory requirements − data-driven RWD
compliance. It is essential for the life science and
health care industry to familiarize itself with
upcoming requirements and recommendations of
regulatory bodies to prove, establish and audit the
company-specific conformity; this includes e.g.,
monitoring of potential contractors. The use of RWD
without sound knowledge and continuous monitoring
(governance) carry the risk of punishable compliance
violations.
Registration Risks. The middle layer of figure 2
emphasizes that the use of RWD can lead to
adjustments in the approval processes of drugs and in
the design of clinical trial setups. Companies
preparing to leverage RWD effectively, including
RWE's evidence will be able to respond proactively
to changes in the near future and surely gain
competitive advantage. In conclusion, the focus here
is set on process changes in the development and
approval of new drugs and/or therapies.
Business Model Risks. The outer layer of figure 2
shows that RWD have the potential to disrupt (not
only) the life science and health care industry. RWD
will develop into a critical success factor: Those in
possession of RWD and able to prove RWE and, in
addition, have the knowledge and competence for
their evaluation will probably dominate the market in
the near future. This area focuses on potential new
competitors (e.g., Apple and its Smart Watch)
entering the market as well as new disruptive business
models in the life sciences and health care industry.
Figure 2: Risk areas of RWD.
In conclusion, our focus is on regulatory and legal
requirements for RWD protection of personal data
and proof of data integrity to provide RWE. The latter
is subject to special attention by the FDA (FDA,
2016, 2017, 2018), United Kingdom Medicines and
Healthcare Products Regulatory Agency (MHRA),
and various other regulatory bodies. This means that
at the latest in an audit which is scoped to RWE,
companies need to provide information e.g., based on
ICEIS 2019 - 21st International Conference on Enterprise Information Systems
384
technical justification and corresponding scientific
rationale to prove that their analytical results based on
RWD are compliant (related to conceivably various
regulatory requirements).
4 MULTIDIMENSIONAL
APPROACH
The use of RWD promises enormous potential for
many areas of the life sciences and health care
industry (Greenfield, 2017); there are clear chances
that it may possibly disrupt the industry in the wake
of potential new competitors, like Apple (e.g., heart
study conducted by Apple (2018)).
Considerable research work has just been initiated
focusing on the use and analysis of RWD in health
care. AI is the technique of choice in many studies to
analyse the large amount of RWD. RWD resp. big
data in health care is generally regarded as crucial for
building (new) models of disease progression and
improved efficiency (cost effectiveness) of existing
clinical trial setups (Vayena et al., 2018).
In certain areas, the application of AI algorithms
has already outperformed experienced health care
professionals. One example is the identification of
skin cancer based on dermatologic image recognition
(Haenssle et al., 2018). Unfortunately, the use of AI
entails unsolved disadvantages. As an example, AI
algorithms are criticized for their ‘black box’ results,
not giving insights into the pathways/algorithms that
lead from input data to output results (Forbes, 2018).
Another, still unresolved challenge regarding AI
based results is the correct understanding of soft
factors’ related to human´s behaviour: AI algorithms
are not foreseen to (properly) interpret figurative uses
of human´s language such as metaphors and irony.
Also the assignment of meaning to symbols or
behaviour is a challenge there is a hidden side of
language and communication that requires at least
extra-linguistic knowledge (Moreno and Redondo,
2016; Lu et al., 2018).
According to Moreno and Redondo (2016),
figurative language is used to about 20% in social
media conversations. This means that about 20% of
the language is (at least) currently impossible to be
interpreted via AI techniques.
To conclude, while AI is extremely powerful in
extracting statistically significant patterns from data,
there are serious limitations to whole-brain functions
and associated underlying meanings and perceptions.
Despite this serious deficiency, many innovations
today rely heavily on automated data analytics, not
exclusively but more and more often, based on AI
algorithms, without considering the fact that data
evidence can be achieved only with a combination of
hard and soft factors. Figure 3 shows the
dependencies between hard and soft factors, in
particular the state of RWE (based mainly on hard
factors) and future perspectives with increasing
consideration of soft factors.
Figure 3: Hard and Soft factors of RWE.
For the life science and healthcare industry, the
abstract construct of perceived HRQoL is a
substantial criterion for evaluating the impact of
products and treatments. As an example, perceived
HRQoL relies on RWD and is strongly dependent on
RWE. It is mandatory to question the methodologies
and techniques used to collect RWD and to show
RWE, which lead to future decisions. Therefore, we
postulate that approaches that focus on the analysis of
RWD and associated RWE need to incorporate the
strengths of artificial and human intelligence ‘by
design’. Such a multidimensional method will
significantly enhance the use of RWDs for the
following reasons:
Artificial Intelligence (AI) is strong in
revealing correlations and extracting
statistically valid patterns, whereas
Human Intelligence (HI) is strong in
revealing causalities by creating system-related
sense-making and contextual scenarios,
considering symbols or behaviour.
The two complementary approaches have the
potential to cross-fertilize and replenish each other.
The results can be developed iteratively, so that
outcomes based on AI and HI are merged and finally
reach a level where it is possible to measure the
effectiveness of treatments beyond the hard factors
(related to AI), stretching out to a validated analysis
Human-centered Artificial Intelligence: A Multidimensional Approach towards Real World Evidence
385
of soft factors (related to HI). With this approach, for
example the concept of perceived HRQoL can be
supported with an increased validity of the analysed
RWD.
Finally, our multidimensional approach aims to
achieve three levels of evidence, which we refer to
‘elementary’, ‘algorithmic and ‘human-centered‘
evidence of RWD. The elementary level addresses
the challenges to ensure accuracy, consistency and
completeness of the data collected. The algorithmic
evidence is achieved with the help of AI algorithms,
whereas the human-centered evidence will rely on a
method, which puts humans in the center. For the last
one, we chose ‘System Dynamics’ as the appropriate
methodology which is introduced in the following
chapter. Figure 4 shows the emerging level of
evidence over time by applying our suggested
multidimensional or multilevel approach.
Figure 4: Levelled Multidimensional Approach.
In the following chapter we explain each
component of our approach, we enrich the use of AI
in combination with RWD providing a more systemic
and holistic perspective. Our novel multidimensional
approach will be reflected on the derived indicators
necessary to gather information about perceived
HRQoL.
5 COMPONENTS OF APPROACH
While the first level of our multidimensional
approach establishes the basis for further research
steps, the subsequent levels for building evidence
(algorithmic and human-centered levels) need to be
performed iteratively.
5.1 Elementary Evidence
The basis for establishing elementary evidence is to
assure a governed data quality. The prerequisite is the
access to trustworthy and comprehensible RWD.
It is of high relevance that the selected RWD is of
sufficient quality, with the consequence that data
assurance needs to be proven using procedures that
are subject to a recognised procedural guideline based
on regulatory requirements. This results in an
‘assurance quality seal‘, which verifies in particular
the accuracy, consistency and completeness of the
relevant RWD a key when dealing with patient-
centered material. From an audit perspective, to
achieve an assurance quality seal for RWD, accurate
and traceable data management and related
governance procedures is a prerequisite. In a first
step, evidence criteria for the assurance quality seal
need to be determined. The evidence criteria will be
used for a RWE assessment. The results determine
whether the selected RWD source can be used (‘go’)
or if data quality improvement procedures must be
performed, or if another set of data needs to be
selected and audited (‘no go’).
The result is an assurance quality seal, the
elementary RWE that builds the foundation for
subsequent steps; we categorize this level of RWE as
‘maturity-level-1’ or elementary evidence (figure 4).
5.2 Algorithmic Evidence
The tempting idea of AI aims to simulate human (-
like) intelligence within machines, more specifically
computerized systems. This is termed as ‘general AI’,
which includes the replication of human emotions and
reasoning. By contrast, ‘narrow AI’ is used to
describe technologies that conduct specific tasks
similarly, or even better, than humans (Jones et al.,
2018). Even though AI has recently gained a lot of
attention, the idea and term was coined in 1955
(McCarthy et al., 1955).
A closely related approach is ‘Machine Learning’
(ML) which refers to ‘the study of computer
algorithms that can learn complex relationships or
patterns from empirical data and make accurate
decisions.’ (Jones et al., 2018). ML methods can be
divided in a.) supervised learning and b.)
unsupervised learning. While a.) implies the need to
train the rules and models based on existing
knowledge (e.g. training data, structures), b.) does not
rely on predefined data or structures (Moreno and
Redondo, 2016).
A very prominent group of AI algorithms are the
Neural Networks (NN) (Jones et al., 2018). The
example outlined in chapter 3 in which AI was used
for dermatologic image recognition is based on
convolutional NNs. There are currently numerous
software solutions available that offer numerical
high-performance calculations (e.g., the open source
ICEIS 2019 - 21st International Conference on Enterprise Information Systems
386
software library ‘TensorFlow’ and the application
programming interface ‘Keras’).
Another branch of AI is Natural Language
Processing (NLP), which supports to understand,
learn, interpret and produce human language content.
NLP draws from many disciplines, including
computer science and computational linguistics; it is
the ‘artto manage the understanding between human
communication and computers. NLP supports
human-human communication, human-machine
communication, or both by analysing learning and
producing content from a large quantity of data
(Hirschberg and Manning, 2015). In addition, NLP
can be used to extract information from unstructured
text such as clinical notes, or RWD from patient’s
interest groups (Murff et al., 2016; Jiang et al., 2017).
The initial part for algorithmic evidence is built
on indicators that analyse and interpret structured as
well as unstructured RWD. As previously elaborated,
the shortcomings of AI require the validation and
further elaboration of the indicators on which the
applied algorithms rely on. This is important to judge
if the AI-based results considered both - hard and soft
factors - to achieve the desired level of RWE.
Because our developed approach has an iterative
character, the indicators and data-driven explanations
discovered at level 2 - the algorithmic evidence - will
serve as initial input for level 3 (section 5.3);
consequently, the level-3-output will be returned for
subsequent iterations of (supervised) learning based
on AI algorithms. With this approach, we create a
complementary linkage of level-2- and level-3-
outcomes, which will leverage the RWE. Results of
the level-2- activities are the ‘maturity-level-2- or
algorithmic evidence: AI-generated indicators
subsequently can be used as input for the level-3-
exercise to achieve the desired level of human-
centered evidence (see figure 4).
5.3 Human-centered Evidence
As already outlined, the causalities of social
interactions are indispensable to understand complex
situations (e.g., language metaphors, irony, symbols,
signs, behaviour). Therefore, as a complementary
technique to the algorithmic evidence we considered
the ‘Systems Dynamics’ (SD) technique as sufficient
to collect human-centered indicators. The
methodology itself is generic and can be applied to
various other contexts (Van den Belt, 2004); for
example in management research and practice (Lane,
1992; Repenning, 2002; Rudolph et al., 2009).
The idea of the SD technique is a ‘systems thinking’
based analysis, which takes a step back from the level
of single events and attempts to develop structural
explanations of system behaviour.
So-called ‘Causal Loop Diagrams’ (CLDs) - shown
exemplarily in figure 5 - are used to describe feedback
loops; core building blocks of CLDs are variables and
causal relationships between them (von Kutzschen-
bach et al., 2018).
Every loop represents a feedback system, whereas
the loops can be categorized as either positive
/reinforcing (labelled as ‘R’) or negative/balancing
(labelled as ‘B’). The causal relationships between
the variables of a system are indicated as links -
visualized as arrows. Our example in figure 5 shows
that the variable ‘Growing Action’ is expected to
increase (+) variable ‘Results’, and vice versa.
However, the changes of ‘Results’ are expected to
increase (+) the third variable named Slowing
Action’, which is expected to decrease (−) ‘Results’
again. The ‘II’ sign indicates assumed time delays
between causes and effects. The loops spoil the
distinction between the driver and the driven, cause
and effect, because, as time progresses, each variable
plays both roles. All loops together show the overall.
Figure 5: Generic Example of a CLD Diagram (adopted from Kutzschenbach et al., 2018).
Human-centered Artificial Intelligence: A Multidimensional Approach towards Real World Evidence
387
systems behaviour (von Kutzschenbach et al., 2018).
These SD modelling techniques can be applied to
validate the previously generated set of AI indicators.
The data-driven approach from level 2 is expected to
lead to explanations - indicators - that would be used
on level 3 to design the variables and links of a CLD.
This CLD would represent a system explaining the
behaviour of perceived HRQoL and its influencing as
well as influenced variables
Appreciating the complex and fuzzy nature of the
perceived HRQoL, the AI-approach (level 2) will be
validated and enhanced by Group Model Building’
(GMB).
When applying SD, GMB is a proven way of
engaging multi-stakeholder perspectives in the
development of causal-loop diagrams and simulated
dynamic model (Scott et al., 2016).
With these activities, we create a dynamic model,
which will be refined and corrected until it is
saturated. This is a sense-making, human-centric and
collaborative action; with the GMB approach, we are
able to involve various stakeholders with different
perspectives and experiences. The outcome, the ‘SD
AI indicator set’, is grounded on a dynamic SD
model, which allows simulations of different
scenarios. The worked out ‘SD AI indicator set’ must
be regularly merged with the data-driven insights
based on AI algorithms (level 2); the associated
governance processes must be additionally defined.
The result of the level-3-activities is the human-
centered evidence, which means a set of AI-generated
indicators, which are complementarily and iteratively
enriched with the GMB-generated indicators of
HRQoL.
6 CONCLUSION AND OUTLOOK
At the beginning of this study, we claimed that the
exclusive use of AI to analyse and interpret RWD and
to achieve reasonable RWE for a selective dataset is
not sufficient. We showed (figure 1) that the fit for
purpose to obtain RWE is depending on various
factors like regulatory, clinical, technology and
society perspectives as well as other considerations,
in particular data quality and the sufficiency of the
used and combined methods.
Thereafter, major challenges associated with the
use of RWD have been categorized by three areas -
compliance, registration and business model risks
(figure 2). Compliance risks are data-related and refer
to challenges along the assurance and governance of
RWD. Registration risks refer to product-related
challenges that might occur due to adapted approval
and development procedures once RWD become an
accepted means to prove drug effectiveness. Finally,
business model risks have been pointed out as RWD
have the potential to disrupt the life science and
healthcare industry.
Next, we addressed current trends to analyse
RWE with the help of AI techniques when trying to
demonstrate RWE. We concluded that, due to
shortcomings of current data-driven techniques, there
is a need for a multidimensional approach. We
selected and applied the concept of perceived HRQoL
to discuss our novel approach towards RWE.
Unlike pure hard factors (relatively easy to
analyse and measure with AI techniques), the
construct of perceived HRQoL is mostly determined
by soft factors - which are difficult to analyse and
quantify. Soft factors are not measurable via AI
exclusively; however, it is our claim that the soft
factors will be increasingly considered in future
decision-making related to RWE (see figure 3).
Thus, our developed and proposed approach has
the potential to contribute to one of the major
challenges of NLP - the ‘soft side’ of text and human
perceptions. Existing techniques mainly rely on text
fragments in which opinions/sentiments are explicitly
expressed (e.g., polarity terms and their co-
occurrence frequencies) (Cambria et al., 2016).
As an agenda for further research, a feasibility
study to test and apply the suggested procedures is
desired. As unit of analysis, patient interest group
data from a dedicated therapeutic area could be used.
In the scope of such a future study, the authors would
aim to limit the focus on assessing perceived HRQoL.
The design, development and validation of four
artefacts with relation to figure 4 is being considered:
1. Development of a quality assurance level (a
‘quality seal’) for the elementary evidence for
RWD resulting in a maturity-level-1-evidence.
2. AI-generated indicators to provide perceived
HRQoL algorithmic evidence (level 2) and as
input for human-centered evidence resulting in a
maturity-level-2-evidence.
3. ‘SD AI indicator set’ - a set of robust indicators
gathered via SD technique and passed back to the
AI-generated indicators resulting in a maturity-
level-3-evidence.
4. ‘RWE Framework’ - a prototypical frame-work,
which contains all relevant steps to achieve RWE
for a selective set of RWD perceived HRQoL −
associated with the ‘SD AI indicator set’.
These results could trigger a multiplier effect and
form the basis for future research. First, the concept
of using a three-layered multidimensional procedure
ICEIS 2019 - 21st International Conference on Enterprise Information Systems
388
and an evidence providing quality seal can be used in
various AI related contexts to improve AI based
results, not at least in providing RWD evidence.
Second, the created artefacts could be generalized for
a broader use. More specifically, the concept of a
combined methodically sound set of AI indicators
based on hard- and soft factors could become a
standard approach for the AI discipline. Third, the
‘RWE Framework’ could be applied on further RWD
sources to show data evidence in other areas.
REFERENCES
Apple, 2018. Heart Study in collaboration with Stanford
Medicine. Available at http://med.stanford.edu/
appleheartstudy.html.
Asadi-Lari, M., Tamburini, M., Gray, D., 2004. Patients’
needs, satisfaction, and health related quality of life:
Towards a comprehensive model. Health and Quality
of Life Outcomes, 2(32).
Bengtsson, M., Wall, G., Larsson, P., Becktor, J.P.,
Rasmusson, L., 2018. Treatment outcomes and patient-
reported quality of life after orthognathic surgery with
computer-assisted 2- or 3-dimensional planning: A
randomized double-blind active-controlled clinical
trial. American Journal of Orthodontics and
Dentofacial Orthopedics, 153(6), pp. 786-796.
Berger, M., Craig, L., Gutteridge, A., Axelsen, K., Subedi,
P., Madigan, D., 2015. Optimizing the Leveraging of
Real-World Data to Improve the Development and Use
of Medicines. Value in Health, 18(1), pp. 127-130.
Berger, M., Overhage, M., Daniel, G., Platt, R., Frank, K.,
Romine, M., Hernandez, A., Tunis, S., McClelland, M.,
Wilson, M., Okun, S., 2017a. A Framework for
regulatory use of real-world evidence, White paper,
Center for Health Policy at Duke University. Available
at https://healthpolicy.duke.edu/sites/default/files/atom
s/files/rwe_white_paper_2017.09.06.pdf.
Berger, M.L., Sox, H., Willke, R., Brixner, D., Eichler, H.-
G., Goettsch, W., Mullins, C.D., 2017b. Recomm-
endations for Good Procedural Practices for Real-
World Data Studies of Treatment Effectiveness and/or
Comparative Effectiveness Designed to Inform Health
Care Decisions: Report of the Joint ISPOR-ISPE
Special Task Force on Real-World Evidence in Health
Care Decision Making, Pharmacoepidemiology &
Drug Safety, 26, pp. 1033-1039.
Bipartisan Policy Center, 2016. Using real-world evidence
to accelerate safe and effective cures. Advancing
Medical Innovation for a Healthier America, White
Paper. Available at https://cdn.bipartisanpolicy.org/
wp-content/uploads/2016/06/BPC-Health-Innovation-
Safe-Effective-Cures.pdf.
Boell, S.K., Cecez-Kecmanovic, D., 2014. A hermeneutic
approach for conducting literature reviews and
literature searches. Communications of the Association
for Information Systems, 34 (12), pp. 257-286.
Cambria, E., Poria, S., Bajpai, R., Schuller, B., 2016.
SenticNet 4: A Semantic Resource for Sentiment
Analysis Based on Conceptual Primitives. Proceedings
of COLING 2016, the 26th International Conference on
Computational Linguistics: Technical Papers, Osaka,
Japan, pp. 26662677.
Davis, B., Morgan, J., Shah, S., 2018. Mission Critical.
Biopharma companies are accelerating real-world
evidence adoption, investment, and application. A
report by the Center for Health Solutions. Available at
https://www2.deloitte.com/content/dam/ insights/ us/ar
ticles/4354_Real-World-Evidence/DI_Real-World-Evi
dence.pdf.
Ekundayo, O., Edwards, N., Bansal, A., Ali, A., Tang, E.,
Xu, A., Li, V., Novak, M., Mucsi, I., 2018. Patient
Reported Outcome Measures Information System
(PROMIS) domains explain a large portion of variance
in quality of life in advanced chronic kidney disease,
Journal of Psychosomatic Research, 109, pp. 100-101.
European Union, 2016. European Union General Data
Protection Regulation, GDPR, EU-Regulation
2016/679. Available at https://gdpr-info.eu/.
FDA, 2016. Use of real-world evidence to support
regulatory decision-making for medical devices: Draft
guidance for industry and Food and Drug
Administration staff. Paper published by U.S.
Department of Health and Human Services Food and
Drug Administration. Available at www.fda. gov/down
loads/medicaldevices/deviceregulationandguidance/gu
idancedocuments/ucm513027.pdf.
FDA, 2017. Use of Real-World Evidence to Support
Regulatory Decision-Making for Medical Devices.
Guidance for Industry and Food and Drug
Administration Staff. Paper published by U.S.
Department of Health and Human Services Food and
Drug Administration. Available at www.fda.gov/
BiologicsBloodVaccines/GuidanceComplianceRegulat
oryInformation/Guidances/default.htm.
FDA, 2018. Framework for FDA’s REALWORLD
EVIDENCE PROGRAM. Available at www.fda.gov/do
wnloads/ScienceResearch/SpecialTopics/RealWorldE
vidence/UCM627769.pdf.
Forbes, 2018. Don't Trust Artificial Intelligence? Time To
Open The AI 'Black Box'. Available at www.
forbes.com/sites/jasonbloomberg/2018/09/16/dont-
trust
-artificial-intelligence-time-to-open-the-ai-black-box/#
3353c7533b4a.
Greenfield, S., 2017. Making Real-World Evidence More
Useful for Decision Making. Value in Health, 20(8), pp.
1023-1024.
Haenssle, H. A., Fink, C., Schneiderbauer, R., Toberer, F.,
Buhl, T., Blum, A., Kalloo, A., Ben Hadj Hassen, A.,
Thomas, L., Enk, A., Uhlmann, L., 2018. Man against
machine: diagnostic performance of a deep learning
convolutional neural network for dermoscopic
melanoma recognition in comparison to 58
dermatologists. Annals of Oncology, 29(8), 1836-1842.
Hirschberg, J., Manning, C.D., 2015. Advances in natural
language processing. Science, 349(6245), pp. 261-266.
Human-centered Artificial Intelligence: A Multidimensional Approach towards Real World Evidence
389
Hubbard, T.E., Paradis, R., 2015. Real World Evidence: A
New Era for Health Care Innovation. Available at
www.nehi.net/writable/publication_files/file/rwe_issue
_brief_final.pdf.
Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang,
Y., Dong, Q., Shen, H., Wang, Y., 2017. Artificial
intelligence in healthcare: Past, present and future.
Stroke and vascular neurology, 2(4), pp. 230243.
Jin, J., Sklar, G.E., Min Sen Oh, V., Chuen Li, S., 2008.
Factors affecting therapeutic compliance: A review
from the patient’s perspective. Therapeutics and
Clinical Risk Management, 4(1), pp. 269286.
Jones, L.D., Golan, D., Hanna, S.A., Ramachandran, M.,
2018. Artificial intelligence, machine learning and the
evolution of healthcare. Bone & Joint Research, 7(3),
pp. 223225.
Khan, K., Kunz, R., Kleijnen, J., Antes, G., 2011.
Systematic reviews to support evidence-based
medicine, 2nd edition. Medicine on the Move: Taylor
& Francis Group.
Lane, D.C., 1992. Modelling as learning: A consultancy
methodology for enhancing learning in management
teams. European Journal of Operational Research,
59(1), pp. 6484.
Lu, H., Li, Y., Chen, M., Kim, H., Serikawa, S., 2018. Brain
Intelligence: Go beyond Artificial Intelligence. Mobile
Networks and Applications, 23(2), pp. 368375.
Mahajan, R., 2015. Real world data: Additional source for
making clinical decisions. International journal of
applied & basic medical research, 5(2), p. 82.
Makady, A., de Boer, A., Hillege, H., Klungel, O.,
Goettsch. W., 2017a. What Is Real-World Data? A
Review of Definitions Based on Literature and
Stakeholder Interviews. Value in Health, 20(7), pp.
858865.
Makady, A., Ham, R.T., de Boer, A., Hillege, H., Klungel,
O., Goettsch, W., 2017b. Policies for Use of Real-
World Data in Health Technology Assessment (HTA):
A Comparative Study of Six HTA Agencies. Value in
Health, 20(4), pp. 520532.
Marwaha, S., Ruhl, M., Shorkey, P., 2018. Doubling
Pharma Value with Data Science. Boston Consulting
Group. Available at http://image-src.bcg.com /Images
/BCG-Doubling-Pharma-Value-with-Data-Science-Fe
b-2018_tcm9-184455.pdf.
McCarthy, J., Minsky, M.L., Rochester, N., Shannon, C.E.,
1955. A Proposal for the Dartmouth Summer Research
Project on Artificial Intelligence. Available at www-
formal.stanford.edu/jmc/history/Dartmouth/dartmouth.
html.
Moreno, A., Redondo, T., 2016. Text analytics: The
convergence of big data and artificial intelligence,
IJIMAI, 3(6), pp. 57-64.
Murff, H.J., FitzHenry, F., Matheny, M.E., Gentry, N.,
Kotter, K.L., Crimin, K., Dittus, R.S., Rosen, A.K.,
Elkin, P.L., Brown, S.H., Speroff, T., 2016. Automated
identification of postoperative complications within an
electronic medical record using natural language
processing. Available at www.semanticscholar.org/
paper/Automated-identification-of-postoperative-with
in-an-Murff-FitzHenry/7c299e9ffc6980283b119aff54
84834fcfe25aec.
Padilla, C.M., Kihal-Talantikit, W., Perez, S., Deguen, S.,
2016. Use of geographic indicators of healthcare,
environment and socioeconomic factors to characterize
environmental health disparities. Environmental
health: a global access science source, 15(1), p. 79.
Repenning, N.P., 2002. A simulation-based approach to
understanding the dynamics of innovation
implementation. Organization Science, 13(2), pp. 109
127.
Rudolph, W.J., Morrison, B., Carroll, J.S., 2009. The
dynamics of action-oriented problem solving: Linking
interpretation and choice. Academy of Management
Review, 34(4), pp. 733756.
Scott, R.J., Cavana, R.Y., Cameron, D., 2016. Recent
evidence on the effectiveness of group model building.
European Journal of Operational Research, 249(3), pp.
908-918.
Tranfield, D., Denyer, D., Smart, D., 2003. Towards a
methodology for developing evidence-informed
management knowledge by means of systematic
review. British Journal of Management, 14(3), pp. 207
222.
Van den Belt, M. (Ed.), 2004. Mediated modeling: A system
dynamics approach to environmental consensus
building. Island Press. Washington, D.C.
Vayena, E., Dzenowagis, J., Brownstein, J.S., Sheikh, A.,
2018. Policy implications of big data in the health
sector. Bulletin of the World Health Organization,
96(1), pp. 6668.
von Kutzschenbach, M., Schmid, A., Schoenenberger, L.,
2018. Using Feedback Systems Thinking to Explore
Theories of Digital Business for Medtech Companies.
In Business Information Systems and Technology 4.0.
Studies in Systems, Decision and Control, Springer.
Cham.
WHO, 2010. Environment and health risks: a review of the
influence and effects of social inequalities, World
Health Organization, Regional Office for Europe,
Geneva. Available at www.euro.who.int/data/assets/
pdf_file/0003/78069/E93670.pdf.
WHO, 2016. Global diffusion of eHealth: making universal
health coverage achievable. The third global survey on
eHealth [Internet], Geneva. Available at www.who.in
t/goe/publications.
ICEIS 2019 - 21st International Conference on Enterprise Information Systems
390