“Should I Trust or Should I Go?” or What Makes Health-Related
Websites Appear Trustworthy?
An Empirical Approach of Perceived Credibility of Digital Health Information and
the Impact of User Diversity
Luisa Vervier, André Calero Valdez and Martina Ziefle
Human-Computer Interaction Center, RWTH Aachen University, Campus Boulevard 57, Aachen, Germany
Keywords: Digital Health Information, Credibility Factors, User-Diversity, eHealth, Health Literacy.
Abstract: An abundance of information is available on the Internet. At the same time the quality of information
fluctuates largely. When people use such information for making important decisions, this becomes an issue.
“Health” is one of the most searched topics on the Web and search results might have the biggest consequences
for one’s life. However, trust in found information, or lack thereof, filters usage. To understand which
elements on a website convince people to trust the information or not, we conducted a study with two aims:
firstly, identify factors which trigger credibility; secondly, investigate to what extend both the medial
presentation and the severity of the related disease influence the assessment of credibility. Possible factors
were first collected in three focus groups and then operationalized in a questionnaire. We collected 184
responses where three different health websites differing in complexity and in the severity of disease (light
vs. life striking) were presented and assessed. Results show that for more severe diseases more complex
information is preferred. Further actually being ill influences the criteria significantly.
1 INTRODUCTION
In times of digitalization the Internet plays a
dominant role in people’s life. Besides of being used
for reasons of communication and entertainment, the
Internet represents a medium for the search of
information. The development of digital information
is increasing. Day by day, the amount and volume of
information is rising. Information about e.g., places,
persons, opening times, or news represent only some
search topics. One of the most often searched areas
embodies topics about health (Fischer & Dockweiler,
2016). Especially through a new awareness for health
and lifestyle (quantified self) as well as through the
development of the informed patient, information
becomes more relevant than ever before. Apart from
only informing about health topics, people also take
information as a basis for decision making regarding
treatment or the intake of medicine (Andreassen et al.,
2017). There are many advantages to digital health
information. Health information is available
whenever and wherever it is required. For many
people, it enables access to medical
information (Trepte et al., 2015). They can actively
participate in issues regarding their health and even
connect with other people who are dealing with
similar issues (Cline & Haynes, 2001). However,
disadvantages of so much information circulating the
Internet are also present. There is much unserious and
incorrect information available, which is not detected
as wrong or outdated (Trepte et. al., 2015) by the
reader. Thus, psychological or physical consequences
might occur due to delayed medical consultations or
wrong intake of medicine (Eysenbach, 2003).
Therefore, it is a big challenge to assess the
quality and credibility of a website for any user
looking for health information (Dierks et al., 2002).
People focus on different criteria for assessing
information as correct (Kim et al., 1999). There is a
growing need to understand how this type of
information is being accessed and used. What kind of
criteria are important for people’s decisions to trust
information. On the other hand, how diverse are
users? Which kind of user prefers which kind of
presentation? In this study, we investigate these kinds
of questions. The aim of the study is to find out and
understand in how far both the medial presentation of
a health website and the severity of a described illness
Vervier, L., Calero Valdez, A. and Ziefle, M.
“Should I Trust or Should I Go?” or What Makes Health-Related Websites Appear Trustworthy?.
DOI: 10.5220/0006734401690177
In Proceedings of the 4th Inter national Conference on Information and Communication Technologies for Ageing Well and e-Health (ICT4AWE 2018), pages 169-177
ISBN: 978-989-758-299-8
Copyright
c
 2019 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
169

play a role in the assessment of credibility. Moreover,
the study goal is to comprehend what kind of user
characteristics have an impact on the website
assessment.
2 STATE OF THE ART
To understand how medical information is available
on the Internet, we present an overview of digital
health information, credibility factors, health
websites and their complexity as well as the topic of
eHealth literacy.
2.1 Digital Health Information
Information about health are commonly searched for
online. More than 70% of people search this kind of
information on the Internet (Fischer & Dockweiler,
2016). Most of the people inform themselves about
their own health issues. Main search topics are
symptoms of diseases, prognoses, and treatment
possibilities (Medlock et al., 2015). In a study of
Stadtler et al. (2009) investigating on the impact of
patient’s online search on the patient-physician
interaction, 61% of users reported that the
information they have found influences their own
health. This result emphasizes the need to understand
people’s reasons for assessing information as
trustworthy whenever it has an impact on their health.
2.2 Quality of Health-related Websites
Health information is frequently accessed on the
Internet. Even though many tools and guidelines
already exist for site developers to keep their
information qualitatively high (Wilson, 2002),
health-related information and its quality still
fluctuate largely on the Internet (Fahy, 2014). Aspects
of lack in quality of information show a huge range
from information being not serious, not up-to-date, or
containing false information among others.
Furthermore, websites often serve rather as a platform
for commercials than as a platform for evidence-
based sources. One of the biggest challenges for
information seekers is therefore to evaluate the
information. However, not only the content but rather
the presentation of information including layout,
structure, pictures, etc. are aspects influencing the
evaluation by the user.
2.3 Credibility Factors of
Health-related Online Information
The amount of information which appears online,
whenever a search for information about health issues
on the Internet, is overwhelming. Still, people have
developed their own search behavior and coping
strategies. When it comes to assess e.g. websites with
health information they focus on specific aspects.
What are the so-called credibility factors that make
digital information appear useful and trustworthy?
Many studies have been conducted on this
phenomenon. Eysenbach and Köhler (2002) reported
that for instance a list of references, information about
the latest update, as well as information about the
authors and pictures are important credibility factors
of websites that are perceived as trustworthy.
Furthermore, information about alternative treatment
options as well as side effects on health-related
websites belong to further credibility factors (Bates at
al., 2006). Benigeri and Pluye (2003) made an attempt
of describing facilitating criteria for the quality
assessment of health-related digital information.
However, even though plenty of catalogues of
credibility factors already exist, the assessment of
quality still varies.
It seems that user-diversity determines the
different aspects strongly. Barnes et al. (2003) stated
that e.g. the extend of people’s personal involvement
has an impact on the information assessment. Less
involved people seem to focus more on layout than on
content and up-to-datedness as more involved people
do. Moreover, younger people focus more on website
layout (Fogg et al., 2004) compared to older users
who care more about references (Huntington, 2004).
This study aims at supporting these results as well as
identifying more specific aspects taking user-
diversity into account.
2.4 e-Health Literacy
Besides objective criteria such as above-mentioned
credibility factors, subjective ability factors play an
important role regarding the assessment of digital
health information. Literacy is one very important
aspect. People who possess the capability to read and
write and are literate, integrate and participate easier
in social life and are able to understand and carry out
a higher degree of control over everyday events
(Nutbeam, 2008). The term “eHealth literacy”
describes the ability to seek, find, read, understand,
and appraise health information from electronic
sources. It means that people own the skill to apply
the knowledge gained and address or solve a health
ICT4AWE 2018 - 4th International Conference on Information and Communication Technologies for Ageing Well and e-Health
170

problem (Norman, 2006). A higher literacy in health
correlates with better health outcomes. Health literacy
influences the utilization of health care, patient-
physician relationship and self-care (Paasche-Orlow
et al., 2007). It is apparent that due to the individually
trained competence, assessment of digital health
information differs strongly. To find out about the
phenomenon of health literacy it is integrated in our
investigation.
3 RESEARCH METHODOLOGY
This paper raises the question how digital healthcare
information is assessed by its recipient and in how far
it differs according to the disease severity. It focusses
on the perceived credibility of different medial
presentations of websites in conjunction with two
different disease severities. The aim of the paper is to
identify credibility factors which have an impact on
the recipients’ attitude to assess a website with health
information as being trustworthy. To identify,
evaluate, and quantify these factors, a two-fold multi-
methodological approach was chosen. In a first step
data were collected qualitatively by focus groups.
Based on the results a questionnaire was developed
and data were collected quantitatively. The main
research question guiding the investigation were:
(1) Which are the most important trust elements of a
website presenting health information in general? Do
age and gender have an impact on the assessment of
credibility factors? Does the importance of credibility
factors differ regarding the severity of disease?
(2) In how far do the medial presentation of a health
website and the severity of disease play a role
according to the user’s assessment?
(3) To which extend do user factors such as age and
gender affect the assessment of health information of
different disease severities, respectively?
3.1 Previous Focus Group Study
The focus groups aimed at identifying different
factors which have an impact on the assessment of the
credibility of a website. Therefore, three focus groups
with 17 participants were conducted with three
different age ranges (“digital natives”=14–19 years
(N=5), “digital immigrants”=30–54 years (N=6),
“silver surfers”=55–69 (N=6)) .In the beginning,
participants were encouraged to brainstorm about
sources they use when informing themselves about
health information. In a free discussion participants
started to share experiences. In a further step,
participants were asked to rate the mentioned factors.
Findings show that objectivity of health information
is rated as very important throughout all age groups.
Younger participants rated author information, a list
of sources, as well as the date of publication as very
important. Comprehensibility was rated very highly
by middle-aged participants as well as the website
layout. Older participants assessed the structure of the
page as very important.
3.2 Questionnaire Study
To quantify the findings from the focus group
discussions with a larger sample, a questionnaire was
developed. The survey consisted of four parts, dealing
with user factors, theoretical concepts, website
scenarios, and credibility factors.
Demographic Data and Further User Factors. The
first part of the questionnaire assessed age, gender,
highest education level, current activity and health
status. Moreover, general familiarity with the Internet
usage as well as general Internet activities were
collected. At last, usage frequency of information
sources regarding health topics, usability of online
sources such as search engines, platforms, forums,
chats, websites were measured.
Health Literacy Scale. Items regarding health
literacy were taken from Cameron D. Normen and
had to be answered on a 6-point-Likert scale from
1=is not true at all to 6=is totally true (e.g. “I know
how to find helpful health resources on the Internet.”
or “I feel confident in using information from the
Internet to make health decisions.”) The respective
items were added after having checked the scale
reliability (Cronbach’s α = .890).
Website Scenarios. To find out in how far
information about health topics are perceived and
assessed on websites, three existing electronic health
websites were chosen by authors and were arranged
to fictive collages without naming the website brand.
These developed websites could be distinguished
according to their complexity of content and
presentation. Regarding complexity, we consider a
website complex when the information is more
detailed and the layout contains more subunits.
At last, one website with low content and low
presentation was built (LowRep), one with a middle
degree of complexity (MidRep) and a third one with
a very high complexity (HighRep). Furthermore, two
diseases with different degrees of severity were
chosen. For a marginal but still serious disease hay-
“Should I Trust or Should I Go?” or What Makes Health-Related Websites Appear Trustworthy?
171

fever was described. Breast cancer was taken as an
example for a very severe and life striking disease.
Participants were asked to look at the website
leisurely and report their impression. Therefore, they
were asked to rate 5 items afterwards on a 6-point-
Likert scale (1=not at all to 6=yes, in any case) (1)
“Do you like the website?” (2) “Do you think you are
sufficiently informed about the disease?” (3) “Did
you perceive the website as trustworthy?” (4) “Would
you, after having seen the information, still continue
your search?” (5) “Would you, after having read the
information on that website, still want to see a
physician?”.
Items were analyzed regarding the different websites
types using factor analysis (principal component
analysis using varimax-rotation). Two factors could
be identified. The first one included items one, two
and three asking about the benefit, convenience of
information and credibility, showing high internal
reliability (Cronbach’s α = .840). Resulting, a scale
was built and called assessment scale. The second
factor consisted of the items four and five asking
whether more information is desired or a consultation
with a physician is still wanted. Cronbach’s alpha did
not allow to calculate a second scale (α = .571).
Overall, 6 websites were presented; three for each
disease (3 websites x 2 diseases). To avoid fatigue
and ordering effects, the order of websites was
randomized between participants.
Credibility Factors. After one “disease set”, namely
three website versions, participants were asked to rate
which kind of criteria a website must have to be rated
as being trustworthy.
Therefore, participants had to name five out of 18
criteria personally most important criteria (such as
e.g.: date of publication, source, seal of quality, etc.).
These criteria were based on results of the qualitative
pre-study.
3.3 Statistical Method
All subjective measures were rated on six-point
Likert scales. Data were analyzed quantitatively by
using Pearson correlations, ANOVA with repeated
measurement, and MANOVA. The level of
significance was set to p = .05. This means that
significant findings have a probability to appear in 1
out of 20 such studies, even if the effect is not
existent.
3.4 Sample Description
The questionnaire was completed by N=184
participants.
Demographic Data and Further User Factors. The
sample was divided into three age groups. It consisted
of a heterogeneous education level (see Table 1). The
current activity was wide spread (e.g. commercial
area (36%), technical area (19%), social field (16%),
medical field (7%)). In general, the sample
constituted a rather healthy group with M=4.2
(SD=0.85; 6 points max.).
When asked about familiarity with Internet usage, the
sample reported to be rather familiar (M=4.93,
SD=0.4). The highest duration of Internet activities
such as reading newspaper (M=2.53, SD=1.59),
posting in newsgroups (M=2.12, SD=1.28), getting
information about products (M=2.83, SD=1.17) or
buying products (M=2.25, SD=0.97) were limited to
an average duration of 0–60 minutes per week.
When asked about how often participants use
different sources when informing themselves about
health topics, participants reported that they used the
Internet on a monthly basis (M=4.44, SD=1.21),
followed by relatives (M=4.44, SD=1.15). Physician
(M=5.06, SD=0.66), medical journal (M=5.49, SD=
0.9), or self-help books (M=5.51, SD=0.89), were
considered or used 2–3 times a year.
Furthermore, when asked about how helpful different
sources were for the search of health information,
search engines (M=5.52, SD=1.17) were described as
most helpful followed by websites (M=4.22,
SD=0.98), platforms (M=3.41, SD=1.24), forums
(M=3.12, SD=1.21), and last chats (M=2.44,
SD=1.09).
It was also of interest what kind of information
participants search for on the web. As the top priority
information about a healthy lifestyle (M=4.13,
SD=1.22) was given, followed by information about
therapy of serious diseases (M=3.85, SD=1.17).
Information about medical treatments (
M=3.85,
SD=1.26) and physicians (M=3.75, SD=1.4) was
reported before reasons (M=3.79, SD=1.22) or
diagnosis of diseases (M=3.61, SD=1.26). The least
important search topics among health information
were given as causes for a cold (M=2.6, SD=1.27) and
diagnosis of a cold (M=2.49, SD=1.14).
Health Literacy Scale. The health literacy level was
averagely high with M=3.84 (SD=0.79). For a
detailed description of the demographic
characteristics, see Table 1.
ICT4AWE 2018 - 4th International Conference on Information and Communication Technologies for Ageing Well and e-Health
172

Table 1. Demographic characteristics of the aggregated
sample (N=184).
Demographic characteristics
Percentage of
respondents
Age
[years]
mean (SD) 43.5 (15.77)
17-32 digital natives 33.2%
33-53 digital immigrant 33.7%
54-79 silver surfer 33.2%
Gender women
59.8%
men 40.2%
Education
level
No college 61.9%
College or higher 38.1%
4 RESULTS
The presentation of the results is guided by the
research questions.
4.1 Factors Influencing Credibility
To identify credibility factors which have an impact
on the recipients’ attitude to assess a website with
health information as being trustworthy, participants
had to name five out of 18 criteria which are rated as
personally most important. As most important
comprehensibility was mentioned, followed by
objectivity of information, clear structure of website
reference as well as indication of negative side effects
or risks. Factors with low weighted credibility were
links to other websites, access to forums or chat
rooms or pictures of authors. Considering the severity
of diseases, a different picture occurs. Table 2 shows
the results.
Table 2: Five most important assessed credibility factors of
health-related Websites with different disease contexts in %
(N=184).
Light Disease In % Severe Disease In %
comprehensibility 64.7 comprehensibility 62
clarity 54.3 objectivity 48.4
objectivity 49.5 references 46.2
references 39.7 clarity 39.7
details about autho
r
29.9 date of publication 33.2
Comparing the five most mentioned credibility
factors of a light disease and a severe disease, it turns
out that four aspects are the same only with
differently attributed importance. Comprehensibility
e.g. is assessed as the most important aspect of health-
related information for both. Further mentioned
factors differ due to the severity of disease. Whereas
clarity of a Website plays the second most important
role for health-related information of a light disease,
objectivity is mentioned on the second stage. Stage
five contains details about authors as a credibility
factor for a light disease. In contrast to that, date of
publication is assessed as a further important
credibility factor regarding digital health information
of a severe disease.
Taking the three different age groups into
consideration, the named factors stay the same only
the order of mentioned factors differs slightly.
4.2 Assessment of Different Websites
To find out in how far the medial presentation of a
health website and the severity of disease play a role
according to the user’s assessment, a repeated
measure was calculated. Taking only the medial
presentation of all three website into account without
the content of diseases, no significant difference
could be detected (F(1.82, 300.57)=2.57, p=.084).
Still, the most complex version was rated best
(M=3.49, SD=0.77), followed by the second complex
version (M=3.44, SD=0.78) and finally the version
with the least complexity (M=3.34, SD=0.88).
Interestingly, including the different severity of
diseases significant differences were found (F(3.7,
581.5)=5.75, p<.01).
Since, Mauchly’s test indicated that the
assumption of sphericity had been violated

2
(14)=113.48, p<.01, the Greenhouse-Geisser
corrected tests are reported (=.74). For the less
severe disease, version two achieved the best rating
(M=3.64, SD=0.87). In contrast to these findings,
among the three different website versions, version
three was rated best for the more life striking disease
(M=3.44, SD=0.9).
Results indicate, that the assessment of
complexity of websites starts to play an important
role whenever a specific disease is described. More
severe diseases are more favored to be read on a
middle complex website in contrast to a light disease.
In our case participants liked to read information
about a light disease in a fancier way. When asked if
after having seen the website along with the kind of
disease, participants would want to search for more
information or see a physician, also significant results
were found (F(4.42, 680.15)=25.72, p<.01). Figure 2
portrays the different characteristics.
“Should I Trust or Should I Go?” or What Makes Health-Related Websites Appear Trustworthy?
173

Figure 1: Assessment of website regarding if more
information is wished for and if a physician wants to be seen
after having seen website. Error-bars denote standard error.
The desire, to receive more information and to see a
physician, increases with the severity of disease.
Regarding the assessment of website versions, a
different picture is shown. The most complex looking
website seems to inform better in case of a light
disease than in case of a serious disease in contrast to
the other versions. However, the less complex
website versions one and two seem to convey
satisfactory information regarding the more serious
disease than regarding a lighter disease. The same
results can be seen regarding the item if a physician
wants to be consulted afterwards. The less and most
complex websites seem to portray trustworthy
information. To sum up, at first glance the severity of
disease plays an important role however, small
deviations between disease and website version could
be detected.
4.3 Impact of User Diversity on
Assessment of Websites
To investigate to which extend user factors such as
age, gender, health status, or health literacy affect the
assessment of health information of different
severities of disease a MANOVA was run. The health
status and literacy variable were computed each into
three equal groups. Findings show, that three
significant interactions were found between gender
and assessment, age and health status. Female
participants rated the least complex presentation of
the light disease better than men (F(1, 113)=5.997,
p=.016 / M
female
=3.39, SD=0.93; M
male
=3.23,
SD=0.92). The least complex presentation of the
severe disease was rated better by digital natives
(M=3.81, SD=1.14) than by digital immigrants
(M=3.23, SD=0.88) and silver surfers (M=3.13,
SD=0.77) with F(2, 113)=4.915, p=.009. A further
significant result could be found referring to the
health status of participants. In general, participants
with a better health state (values for light disease
version: M
bestHealth
=3.57, SD=0.7; M
middleHealth
=3.25,
SD=0.86; M
badHealth
=2.75, SD=0.85) rated the least
complex presentation of both diseases better than not
so healthy people (F
lightdisease
(2, 113)=5.382, p=.006 /
F
severedisease
(2, 113)=4.443, p=.019). Two interactions
were found between gender and literacy as well as age
and literacy. The first interaction was detected on the
assessment of the least complex presentation of both
diseases (F
lightDisease
(2, 113)=5.579, p=.005 and
F
severeDisease
(2, 113)=3.854, p=.024). Women with a
significant higher health literacy rated the website
version better than men with a comparable low health
literacy. Findings regarding the lighter disease and
severe disease are very similar. Due to the spatial
limitations of this article only one finding according
to the light disease is shown in figure 2.
Figure 2: Interaction of gender and eHealth literacy (low,
middle, high) regarding assessment of least complex
presentation of light disease. Error bars denote standard
error.
A further interaction can be reported about age and
eHealth literacy referring to the most complex
website presentation and light disease
(F(4, 113)=2.655, p=.037). As can be seen in Figure
3 the youngest age group with a high eHealth literacy
value rates the most complex website version better
than the middle and older age-group.
5 DISCUSSION & GUIDELINES
In this article, we investigated factors of websites
with health information which trigger credibility. We
also examined to what extend both the medial
presentation of health websites and the severity of the
related disease play a role in the assessment of
credibility. To find out in how far information about
health topics are perceived and assessed on websites,
123456
more Info
see physician
more Info
see physician
light serious
version 3 version 2 version 1
level of agreement (min = 1, max=6)
2
2,5
3
3,5
4
4,5
low middle high
Website Assessment (1–6)
Health Literacy
men
women
... ...
ICT4AWE 2018 - 4th International Conference on Information and Communication Technologies for Ageing Well and e-Health
174

Figure 3: Interaction of three age-groups (digital native,
digital immigrant and silver surfer) and eHealth literacy
(low, middle, high) regarding assessment of least complex
presentation of light disease. Error bars denote standard
error.
we chose three existing electronic health websites and
arranged them to fictive collages without naming the
website brand. These developed websites could be
distinguished according to their complexity of
content and presentation. Regarding complexity, a
website was considered more complex when the
information was more detailed and the layout
contained more subunits. The results presented above
and illustrated in Table 2 show, that in general users
searching for health information pay attention to a
range of factors when deciding if the information is
trustworthy. Factors range from content-related
factors to layout-factors. The five most outstanding
factors and the meaning of them are described as
followed: Comprehensibility was one of the most
important factors triggering trustworthiness on
health-related websites. In other words, users attach
importance to information that is written and prepared
in an understandable way. A further mentioned aspect
refers to a clear structure of information. A clear
structure guides the users and leaves them confident
in their understanding of the information. Objectivity
of information counts to a further credibility aspect.
Information that conveys a neutral position is
accepted more than subjective descriptions of health
details. References to sources were also mentioned
among the five most important trustworthy elements.
Further, information about the sources are considered
to be important. More so, users want to learn about
the details about authors. In this regard, our findings
match the results of Eysenbach and Köhler (2002). In
our study, it was of additional interest in how far
aspects that trigger trustworthiness of information
differ, when searching for illnesses of different
severity. It turns out that the types of factors stay the
same, only the order of priority varies.
Comprehensibility remains the most important
aspect. Whereas for a light disease factors such as
clarity illustrate a factor of trustworthiness, for a more
severe disease, objectivity is demanded more
strongly. A further difference that we found was that
details about authors are of interest when informing
about a light disease (in this case hay fever) compared
to date of publication which triggers credibility when
searching for a more life striking disease. In the latter
case patients have a stronger urge to stay up-to-date
and not overlook the most recent advances in therapy.
Our findings referring to our three website scenarios,
differing in the grade of complexity and content
clarity, indicate that in general all three versions are
accepted averagely well. This also changes whenever
the search intention differs. When participants had to
imaging searching for information referring for a
specific grade of disease severity, versions were
considered more critically. When the participant was
searching with the intention to receive information
about a rather light disease, the most complex website
version was rated best. In contrast to that, users favor
to read information about more severe diseases in a
simple or middle complex website version. It can be
noted that the more severe a disease the less fancy the
presentation of information should be.
It is interesting to note that differences of the
perception of credibility do not solely stem from
differences in information complexity. There are no
differences in the assessment when a simple
comparison of means compares their ratings. Only
when user factors or the type of illness is incorporated
differences occur. This is important to know, as on
average all websites seem to do rather ok. When
looking at the details it becomes clear that for some
people some websites are very credible, while others
consider the same websites non-credible. Since there
are systematic variations in the judgements
depending on health literacy, gender, and age, these
factors should not be ignored when designing a
health-related website. It seems that no “one-size-fits-
all” solution exists when it comes to health
information on the Internet. Information providers
should be aware of how to design websites for
different target user groups and possibly consider
methods of participatory design to determine, who
needs what information when and how.
2,5
3
3,5
4
4,5
low middle high
Website Assessment (1–6)
Health Literacy
digital native
digital immigrant
silver surfer
... ...
“Should I Trust or Should I Go?” or What Makes Health-Related Websites Appear Trustworthy?
175

6 LIMITATIONS AND FUTURE
RESEARCH
As with any empirical study limitations apply to our
findings. The investigated interaction effects lend
themselves to relatively small subgroups of
participants. For example, the older male participants
with high health literacy are a rather small subgroup
of users. This can be seen in the size of the larger error
bars in the figures. Nevertheless, effects are present
even with higher error margins in mind. As our
findings are in line with previous research, further
confirmation and transferability of results would
require significantly larger samples or meta-
analytical methods to improve on the evidence. Since
the settings were generated from fictitious websites
constructed by the authors, we cannot be sure that our
perspective on complexity is shared equally among
all users. We have put effort into designing the
websites to increase in complexity in “equidistant”
steps. However, as the texts and images we used were
taken from actual websites, it is not easy to guarantee
this. The illnesses we selected (hay-fever, breast-
cancer) have very specific target groups. Men that
took part in the study noted that it was hard for them
to imagine themselves into a setting requiring therapy
for breast cancer. Although men might in reality
contract breast cancer, it was strongly considered to
be a women’s illness. As with all scenario based
questionnaires one must take all results with a grain
of salt, as the social desirability bias might distort
answers more strongly in settings more alien to the
participant.
Our findings indicate that websites trying to
inform the public about health issues must consider
their information and communication concept. It is
crucial to understand the intended audience and their
requirements for health information. Possibly,
adaptable websites that allow the user to seamlessly
increase the amount of complexity for a given illness
without impeding clear and simple information
access, could provide a solution to such challenges.
In this regard, the use of recommendation system that
is aware of the user’s health (Schäfer et al. 2017),
could be applied to identify information needs from
the user depending on the interaction on the website
(Calero Valdez et al. 2017). When other users interact
with information in forums or comments, additional
non-verified information enters the stage to. In such
health-related social media, certain users are more
active than others (Schaar et al. 2012). Information
and more importantly meta-information could “drift”
because of user interaction—especially when
algorithms determine the presentation of information
(e.g. by rating, liking). Integrating human-oversight
in doctor-in-the-loop approaches could be interesting
to investigate (Holzinger et al. 2016).
When such solutions are addressed, naturally
questions regarding ethics, privacy (Vervier et al.
2017), and trust play an important role. What
information are users willing to share to improve their
online experience on health-related websites? Here,
user diversity factors play an even bigger role
(Zeissig et al. 2017). Therefore, it is necessary to
understand the interplay of all user related factors and
the benefit users see in using such websites. By
modelling trade-offs of utility and privacy, better
services or mobile phone apps with better information
quality can be conceived. These services self-tailor
their presentation and complexity to the needs and
desires of the users even incorporating their current
usage context. Are they looking for help or just
browsing? Should they trust the information they find
or should the go see a doctor? Either way, the factors
that determine the credibility of health information
are crucial in helping patients, both online on the Web
or offline by a doctor.
ACKNOWLEDGEMENTS
The authors thank all participants for sharing their
thoughts and opinions. Furthermore, the research
support of Jenny Reinhard is highly acknowledged.
REFERENCES
Andreassen, H. K., Bujnowska-Fedak, M. M., Chronaki, C.
E., Dumitru, R. C., Pudule, I., Santana, S., ... & Wynn,
R. (2007). European citizens' use of E-health services:
a study of seven countries. BMC public health, 7(1), 53.
Barnes, M. D., Penrod, C., Neiger, B. L., Merrill, R. M.,
Thackeray, R., Eggett, D. L., & Thomas, E. (2003).
Measuring the relevance of evaluation criteria among
health information seekers on the Internet. Journal of
Health Psychology, 8(1), 71-82.
Bates, B. R., Romina, S., Ahmed, R., & Hopson, D.
(2006). The effect of source credibility on consumers'
perceptions of the quality of health information on the
Internet. Medical informatics and the Internet in
medicine, 31(1), 45-52.
Benigeri, M., & Pluye, P. (2003). Shortcomings of health
information on the Internet. Health promotion
international, 18(4), 381-386.
Calero Valdez, A., Ziefle, M., Verbert, K., Felfernig, A., &
Holzinger, A. (2016). Recommender systems for health
informatics: State-of-the-art and future perspectives.
In Machine Learning for Health Informatics (pp. 391-
414). Springer International Publishing.
ICT4AWE 2018 - 4th International Conference on Information and Communication Technologies for Ageing Well and e-Health
176

Cline, R. J., & Haynes, K. M. (2001). Consumer health
information seeking on the Internet: the state of the
art. Health education research, 16(6), 671-692.
Dierks, M. L., Lerch, M., Mieth, I., Schwarz, G., &
Schwartz, F. W. (2002). Wie können Patienten gute von
schlechten Informationen unterscheiden?. Der Urologe
B, 42(1), 30-34.
Eysenbach, G., & Kohler, C. (2003). What is the prevalence
of health-related searches on the World Wide Web?
Qualitative and quantitative analysis of search engine
queries on the internet. In AMIA annual symposium
proceedings (Vol. 2003, p. 225). American Medical
Informatics Association.
Eysenbach, G., & Köhler, C. (2002). How do consumers
search for and appraise health information on the world
wide web? Qualitative study using focus groups,
usability tests, and in-depth
interviews. Bmj, 324(7337), 573-577.
Fahy, E., Hardikar, R., Fox, A., & Mackay, S. (2014).
Quality of patient health information on the Internet:
reviewing a complex and evolving landscape. The
Australasian medical journal, 7(1), 24.
Fischer, F., & Dockweiler, C. (2016). Qualität von
onlinebasierter Gesundheitskommunikation. In eHealth
in Deutschland–Anforderungen und Potenziale
innovativer Versorgungsstrukturen.
Fogg, B. J., Soohoo, C., Danielson, D. R., Marable, L.,
Stanford, J., & Tauber, E. R. (2003, June). How do
users evaluate the credibility of Web sites?: a study with
over 2,500 participants. In Proceedings of the 2003
conference on Designing for user experiences (pp. 1-
15). ACM.
Holzinger, A., Calero Valdez, A. , & Ziefle, M. (2016).
Towards Interactive Recommender Systems with the
Doctor-in-the-Loop. Mensch und Computer 2016–
Workshopband.
Huntington, P., Nicholas, D., Gunter, B., Russell, C.,
Withey, R., & Polydoratou, P. (2004, December).
Consumer trust in health information on the web.
In Aslib Proceedings (Vol. 56, No. 6, pp. 373-382).
Emerald Group Publishing Limited.
Kim, P., Eng, T. R., Deering, M. J., & Maxfield, A. (1999).
Published criteria for evaluating health related web
sites. Bmj, 318(7184), 647-649.
Medlock, S., Eslami, S., Askari, M., Arts, D. L., Sent, D.,
de Rooij, S. E., & Abu-Hanna, A. (2015). Health
information–seeking behavior of seniors who use the
internet: a survey. Journal of medical Internet
research, 17(1).
Norman, C. D., & Skinner, H. A. (2006). eHealth literacy:
essential skills for consumer health in a networked
world. Journal of medical Internet research, 8(2).
Nutbeam, D. (2008). The evolving concept of health
literacy. Social science & medicine, 67(12), 2072-2078.
Paasche-Orlow, M. K., & Wolf, M. S. (2007). The causal
pathways linking health literacy to health outcomes.
American journal of health behavior, 31(1), S19-S26.
Schaar, A. K., Calero Valdez, A. , & Ziefle, M. (2012).
Social media for the ehealth context. A requirement
assessment. Advances in Human Aspects of Healthcare,
79.
Schäfer, H., Hors-Fraile, S., Karumur, R. P., Calero Valdez,
A., Said, A., Torkamaan, H., ... & Trattner, C. (2017,
July). Towards Health (Aware) Recommender
Systems. In Proceedings of the 2017 International
Conference on Digital Health (pp. 157-161). ACM.
Trepte, S., Baumann, E., Hautzinger, N., & Siegert, G.
(2005). Qualität gesundheitsbezogener Online-
Angebote aus sicht von usern und Experten. M&K
Medien & Kommunikationswissenschaft, 53(4), 486-
506.
Vervier, L.; Zeissig, E.-M.; Lidynia, C. & Ziefle,
M. (2017). Perceptions of Digital Footprints and the
Value of Privacy. Proceedings of the International
Conference on Internet of Things and Big Data (IoTBD
2017), pp. 80-91.
Wilson, P., & Risk, A. (2002). How to find the good and
avoid the bad or ugly: a short guide to tools for rating
quality of health information on the internet
Commentary: On the way to quality. Bmj, 324(7337),
598-602.
Zeissig, E. M., Lidynia, C., Vervier, L., Gadeib, A., &
Ziefle, M. (2017, July). Online privacy perceptions of
older adults. In International Conference on Human
Aspects of IT for the Aged Population (pp. 181-200).
Springer, Cham.
“Should I Trust or Should I Go?” or What Makes Health-Related Websites Appear Trustworthy?
177