EVALUATION OF A TELERADIOLOGY SYSTEM
Impact and User Satisfaction
Carla Pereira
Faculdade de Medicina da Universidade do Porto, Al. Prof. Hernâni Monteiro, 4200-319 Porto, Portugal
Ana Ferreira
Center of Informatics and CINTESIS, Faculdade de Medicina da Universidade do Porto
Al. Prof. Hernâni Monteiro, 4200-319 Porto, Portugal
Luis Antunes
Instituto de Telecomunicações, Faculdade de Ciências da Universidade do Porto, 4169-007 Porto, Portugal
Keywords: Teleradiology, Telemedicine, Applications of teleradiology, Focus groups.
Abstract: Teleradiology is the acquisition and transmission of radiologic images and respective diagnosis from a
different location where the patient performed the exam. A teleradiology system was implemented in the
ACES Alto Trás-os-Montes I Nordeste, in the Northeast of Portugal in 2008, aiming to overcome the
shortage of this healthcare resource and expertise in the region. The main objective of this paper is to assess
users’ satisfaction with the use of the teleradiology system presented above and investigate the main
changes and impact that this system had for the patients and healthcare professionals involved in the
process. Focus groups were applied and allowed to ascertaining the users’ perspectives regard the system.
The main characteristic of the participants was that they used the system to perform their work on a daily
basis or they benefited from its use. A focus group was performed for each of the three categories:
Radiology Technicians, Primary Care Physicians and Patients, and they lasted one hour on average. Both
healthcare professionals and patients have benefited with the introduction of the teleradiology system. The
methodology applied allowed to better explore why it is so for each different group and for the various
aspects of the system.
1 INTRODUCTION
With the increasing use of technology, the distance
has virtually ceased to be a problem; the
"approximation" by electronic means of health
resources is a reality today. Telemedicine is a proof
of this "approximation", because even without the
physical presence of certain health services,
populations have the opportunity to use and benefit
from them as well as get advice from experts in
various areas, even when they are miles away and
without needing to travel (Romero et al, 2008).
However in most cases, the impact of these new
health services provided by new technologies
together with users’ satisfaction is not properly
studied and evaluated.
In this perspective there are certain qualitative
research methods that have proved important in the
area of healthcare, in that they provide for the
knowledge perceptions and ideas of the people and
further explore the main reasons why it is so (Coelho
et al, 2009). One of these methods is for example the
Focus group. This method is an important method of
qualitative data collection and is characterized by
guided discussions among a small group of people
about a relevant issue or topic to be explored (Curry
et al, 2009).
Focus group is a method that has been
increasingly used by health professionals.
This method is, not only, very useful for
developing strategies, needs assessment, evaluation
of health promotion, but is also considered a great
295
Pereira C., Ferreira A. and Antunes L..
EVALUATION OF A TELERADIOLOGY SYSTEM - Impact and User Satisfaction.
DOI: 10.5220/0003165702950302
In Proceedings of the International Conference on Health Informatics (HEALTHINF-2011), pages 295-302
ISBN: 978-989-8425-34-8
Copyright
c
 2011 SCITEPRESS (Science and Technology Publications, Lda.)

challenge for the researcher (Rabiee, 2004).This
method can help going beyond the perception of
satisfaction. It is possible to understand the key
issues related to the main theme, the strengths of the
system, the concerns of each target group and gather
and explore views and experiences related to
specific themes (Ferreira et al, 2010).
In this context, Focus Group method is suitable
to explore and understand the degree of satisfaction
of the teleradiology system by different types of
users: those who use the system on a daily basis to
perform their work and the patients who benefit
from the services provided by that system. For this
reason this method helps to achieving the aim of this
article.
1.1 Telemedicine and Teleradiology
Telemedicine provides health care at distance
(Thrall, 2007) with resource to information
technology. The main objectives of telemedicine are
the equal access to specialized medical services,
increased quality of care, decreased costs, increased
cooperation between primary health care and
specialized medical resources and easy access to
health specialists’ care at a distance. There are a
number of areas where this practice can be applied
such as teleradiology, teledermatology,
telecardiology, teleophthalmology, among many
others (Romero et al, 2008).
The evolution of radiology to a digital level
opened several doors for medical imaging, new
opportunities for different clinical applications and
new models of practice (Thrall, 2005), including
teleradiology.
Historically, teleradiology has been explored for
nearly 50 years, within the area of telemedicine,
with the provision of health services at distance
(Thrall, 2007) and is today an active practice in the
world and has a enormous potential for the future
(White, 2002).
According to the American College of Radiology
(ACR), teleradiology is by definition the electronic
transmission of radiological images from one
location to another for the purpose of interpretation
and / or your query (White, 2002). With this system
the same image can be viewed simultaneously by a
large number of users in different locations. Properly
used, this valence can improve access to quality
radiological interpretation and consequently increase
the quality of patient care.
Over the years, teleradiology followed the
evolution of technological means, seizing
opportunities and new equipment that was emerging,
readjusting itself and becoming increasingly
effective. It searches through information
technology improvements to deliver health care in
regions that are most deprived of these resources.
The major contribution of technology to the
success of teleradiology focuses on the following
points: the possibility of rapid and reliable transfers
of radiological images and their diagnostic studies;
archiving and communication - PACS (Picture
Archiving and Communication System); high-speed
Internet access and secure virtual private networks
(Kenny and Lou, 2008).
Another important factor for the success of
teleradiology was the creation of DICOM standard
(Digital Imaging and Communications in Medicine).

This standard was developed by the ACR and the
Electrical Manufacturers Association (NEMA) in
1985 and enables digital communication between
diagnostic and therapeutic equipment and systems
from various manufacturers (RSNA, 2010). This
was very important for teleradiology because it
facilitated communication amongst different
systems.
The ACR stresses as the main benefits of
teleradiology: the interpretation of radiological
examinations by appropriate specialists; the
possibility of these being viewed simultaneously by
different users of the system; improvements in
access to quality interpretations with consequent
improvement of patient’s health care; and the fact
that teleradiology can provide new experiences for
education (White, 2002).
However, apart from the numerous advantages
for people who use this system there are some
problems, doubts and failures.
There is also a great concern about the transfer
and quality of images that are received and this may
bring problems to their interpretation. The failure to
use exactly the same standards increases the
possibility of error (Kenny and Lou, 2008). It is
important to stress also that the difference of
equipments, such as the monitors that display the
images, may be a determinant factor in the image
quality and its interpretation. A fault in the
equipment can also endanger the entire system’s
functionality.
Still on the technological aspects there are
sometimes some problems in integrating the various
systems needed for operating the network of
teleradiolgy, this is a major technological challenge
(Thrall, 2007).
On the other hand there is the question of
training in this area, it is important that whoever
works with these systems has the necessary know-
HEALTHINF 2011 - International Conference on Health Informatics
296

Table 1: Number of radiographies and examinations performed by the teleradiology system (each examination can have
more than one radiography). This data was obtained from the ACES Alto Trás-os-Montes I Nordeste report.
Year No Radiographies No Examinations Made No Examinations that
were reported
Number of Patients examined
in the 10 centers using
teleradiology
2008
36246 22416 19139 ----------
2009
44594 27317 22497 17573
knowledge and that there is cooperation between the
professionals involved in the process for the smooth
functioning and success of the system.
For the teleradiology system to work properly it
is essential that the sites involved are endowed with
the necessary conditions both of qualified personnel
and appropriate equipment so that, at the time of
implementation, all the necessary changes to the
organizational level and the whole environment are
taken into account.
1.2 Teleradiology in ACES Alto
Trás-os-Montes I Nordeste
The ACES (Group Health Centre) Alto Trás-os-
Montes I Nordeste, currently covers 14 health
centers that provide basic health care to
approximately 105,392 patients in the northeast of
Portugal. Figure 1 presents the northeast region of
Portugal with the health centers that are aggregated
in 10 groups and where teleradiology is available.
Figure 1: Portuguese northeast connections among health
group centers where teleradiology is available.
In an attempt to remedy the shortage of specific
health resources in this region, the ACES mustered
equipment and qualified professionals in various
areas. Radiology was one of the sponsored areas in
this project, hence the investment in the
implementation of a teleradiology system.
This system tried to fill the lack of expertise of
this medical speciality in this region and as such it is
important to quantify this shift, as well as the
benefits and/or problems brought to this
predominantly elderly population, with difficulties
in transportation.
Teleradiology currently exists in 10 of the 14
health centers in this region: Alfândega da Fé,
Bragança, Carrazeda de Ansiães, Freixo de Espada à
Cinta, Miranda do Douro, Mogadouro, Torre de
Moncorvo, Vila Flor, Vimioso e Vinhais.
As can be seen in Table 1 there is a considerable
number of examinations made every year. All
radiology services have digital equipment
acquisition, image processing and local servers for
teleradiology, linked by computer to the ACES in
the town of Bragança. The radiographic images are
archived on the PACS system that is also located at
the headquarters of ACES and the service provider
(Radiologists) has access via a private network,
where the reports are made. With the PACS
software, all the Primary Care Physicians can see
their examinations and reports conducted in ACES.
At this moment, when performing their
workflow, the radiology technicians take an average
of 3 days for a regular exam and 1h to emergency
ones.
It is very important to investigate the real change
and impact in the healthcare services provided for
this region by this teleradiology system, especially
the degree of satisfaction for its population and
healthcare professionals, which are the users of the
system. This is therefore the main objective of this
paper.
2 METHODS
2.1 Focus Groups
The focus group sessions consist of a discussion
with six to ten participants in the areas of
investigation and are led by one or two moderators.
All the discussion is recorded and transcribed later
for analysis. Throughout the conversation, the
pertinent points will be explored, giving participants
the opportunity to share their experiences and views
and discuss them among themselves (Ferreira et al,
2010).
As the success of the Focus Groups depends on
EVALUATION OF A TELERADIOLOGY SYSTEM - Impact and User Satisfaction
297

the interactions between participants, the discussion
must generate information that reflects the full
spectrum of opinions and experiences. The
moderators of the group should guide the dialogue in
order to allow a free exchange and ensure that
participants feel comfortable expressing their views
even in disagreement (Curry et al, 2009).
2.2 Population
The selection of participants was done according to
the area of interest for each group session and their
connection with the teleradiology system.
Thus, two of the focus group were selected by
occupational group, including Radiology
Technicians and Primary Care Physicians, since they
are the principal users of the teleradiology system on
a daily basis. In the focus group with the Patients,
the selection was specific for people who had used
the system to treat some healthcare problem. In
either case, the involvement was not confined to a
place where the system is in place.
Two moderators participated in the three focus
groups and were careful to avoid commenting as
much as possible and give opinions on the subject.
For each discussion group it was essential to use
some recording material to capture the sound and
image. To this end a laptop with a program for audio
recording and a video camera for recording audio
and image were used. In addition, it was important
to find a comfortable room, and organize the
furniture according to the objectives of the
discussion and so that to capture the image of all the
participants. The video record allows to capture
body language that can be important in the
construction and analysis of the data (Rabbie, 2004).
2.3 Line of Discussion
The group sessions were flexible and open to
potential issues raised by participants.
As in a Focus Group, there are no fixed questions
to ask the participants, there is the use of a line of
discussion to help in structuring and leading, in an
informal and non-structured way, the subsequent
discussion.
The initial part of the line of discussion, followed
by the moderators, was similar in the three groups:
a. A brief explanation of the issue to the
participants and other information about the
processes that would follow along the discussion
group;
b. Each participant was asked to give their consent
to participate;
c. Each participant was initially asked to give
details about the profession, age and experience with
teleradiology in general and with the system in
question specifically.
Then, the line of discussion was specific for each
group but with some similarities.
1) Radiology Technicians Focus Group (FG1):
a. User experience with teleradiology;
b. Training needed to use the system;
c. Usability (graphical interfaces, access);
d. Technology (system functionality, uptime,
workflow);
e. Trust in the system (information security);
f. Helpdesk - if there is an adequate support in
solving the problems;
g. Level of satisfaction;
h. Influences in the activity with other
professionals (teamwork);
i. Improvements to the region;
j. Difficulties with the system;
k. What can be improved.
Table 2: Description of each Focus Groups data collection.
Focus Group Date and time Recording Audio Video Moderators
Number of
partipants
FG1 24/03/2010 15h 1h15min Y Y 2 7
FG2
07/06/2010
14h45min
50min Y Y 2 8
FG3
07/06/2010
16h45min
1h Y Y 2 5
HEALTHINF 2011 - International Conference on Health Informatics
298

2) Patients Focus Group (FG2):
a. Experience as a patient using the
teleradiology system;
b. Perception of increased speed in medical
consultation after the introduction of this new
system teleradiology (waiting time for
implementation and for obtaining the results of
radiological exams);
c. Greater or lesser facility in performing
radiological examinations;
d. Decrease or increase the distance needed to
perform radiological examinations;
e. More or less comfort in using this service
compared with the previous alternatives
available;
f. Level of satisfaction with the system
teleradiology;
g. Improvements to the region;
h. What can be improved.
3) Primary Care Physicians Focus Group (FG3):
a. User experience with teleradiology;
b. Training needed to use the system;
c. Technology (system functionality, uptime,
workflow);
d. Trust in the system (integrity of information /
security);
e. Usability (efficiency obstacle / enhancer,
graphical interfaces, access);
f. Helpdesk - if there is adequate support to
solve problems;
g. Perception of increased speed in medical
consultation after the introduction of this new
system teleradiology (waiting time for
implementation and for obtaining the results of
radiological);
h. Level of satisfaction;
i. Influences in the activity with other
professionals (teamwork);
j. Contribution in service to users;
k. Improvements to the region;
l. Difficulties with the system;
m. What can be improved.
3 RESULTS
3.1 Data Collection and Analysis
Data were collected using the audio and video recor-
ding made in each session. Data analysis was done
with the transcription of the whole discussions to
Word documents. Selected main issues and opinions
were placed on a table in another document, along
with the profile of each person. Those issues that
were most discussed during the course of the
discussions were selected and ordered.
3.2 Results from FG1
In this group discussion participated seven radiology
technicians: two women and five men
whose work is
distributed by the 10 health centers that provide
teleradiology. They work with the teleradiology
system on a daily basis and are aged between 24 and
30 years. This was a very dynamic and participatory
group. Two participants started to use this system
from the beginning and went through all the
intermediate stages until the actual state of the
system. Other participants just know the system
exactly as it is now, so the contribution of this group
for this analysis was indeed important. The results of
a preliminary analysis of this discussion are
presented in Table 3.
The main issues that dominated the discussion
were:
a. Workflow;
b. Evolution of the System / Turning Points /
Equipment;
c. Limitations / Main problems;
d. Training;
Many different experiences were described: the
experience of a mother with her baby, the routine
tests, emergency examinations, and examinations
made in a local to be seen by Primary Care
Physicians in a different location. This diversity of
ages and professions was very important to study
because it was possible to obtain opinions from a
wider range of people who use these health services.
This discussion came to the following results that
are presented in Table 4.
The main issues that dominated the discussion
were:
a. Teleradiology - Before / After;
b. Influences on Attendance;
c. Security in data transmission.
EVALUATION OF A TELERADIOLOGY SYSTEM - Impact and User Satisfaction
299

Table 3: Description of results from FG1.
Issue Related issues
Teleradiology
The system is an essential tool
It has several platforms
Interoperability needs are still required
The workflow is equal for all the participants
System evolution
/Turning
points/Equipments
The equipment to treat radiological images is the same everywhere
The acquisition of a PACS service was essential to implement the teleradiology service
Acquisition of a dedicated line
Usability
The platform Siima (Radiology Information System (RIS), for making the patients
report request) could be improved, is functional but not very intuitive
Education
All the participants had education but they agree that it was not a very good one
The first professionals to use the system had more difficulties, the others had more help
from them
Security and trust in
data transmission
All participants Access the system with a login and a password
There is a Great trust in the integrity of the information that is stored within the system
If the process is done with the correct steps and if there is no human error, the reliability
is quite big
Helpdesk
They all know who to report the errors
Problems’ resolution does not take too long
There is rarely feedback regarding the reported problem
Satisfaction degree
4 in a scale from 1 to 5
Teamwork
It has improved at several levels and with the different healthcare professionals that they
interact on a daily basis
Service providing
It has lessened patients’ waiting time
Improvements to the
region
It has shortened the distances
It has improved accessibility
It has reduced the need for transportation
Improvement in healthcare
Limitations/main
problems
Interoperability with other systems
Bandwith is limited and not enought to satisfy all the needs
To improve
There is the need for some applications to be developed in order to facility the
radiologist activities
Resolution of the main problems
Direct connection with the hospitals in that region
Table 4: Description of results from FG2.
Issue Related issues
Teleradiology
Before/After
With this new system everything is faster
Trust in the system
They trust the system
Service providing
The human factor (healthcare professionals assistance) is essential for them to
connect with the system
Security and trust in data
transmission
There is some concern that the data may not be transmitted in a secure way
Satisfaction degree
5 in a scale from 1 to 5
Improvements to the region
5 in a scale from 1 to 5
To improve
It can always be improved (they did not specify what)
Suggestions
Possibility to access their own medical records with an access code, similar to
other areas, for instance banking
HEALTHINF 2011 - International Conference on Health Informatics
300

Table 5: Description of results from FG3.
Issue Related issues
System evolution
• Is better now than before
• The access to the radiological exams is faster
• Saves time and costs
• Is possible to manipulate the image
Usability
• Access to the image should be even more straightforward
Education
• All the participants had education to use the system
Security and trust in data
transmission
• Everybody access the system with their own login and password
• There is a great trust on the information integrity that is stored within the
system
Helpdesk
• They know to whom report the problems
Satisfaction degree
• 4 in a scale from 1 to 5
Teamwork
• 4 in a scale from 1 to 5
Service providing
• 4 in a scale from 1 to 5
• Faster consultations
Improvements to the region
• 4 in a scale from 1 to 5
• It has shortened the distances
Limitations/main problems
• General difficulties in using informatics equipment
• Sometimes it is slow to access the radiological images (bandwith problems)
To improve
• Resolution of the main problems
• Faster and more direct access to the images
3.3 Results from FG3
Five Primary Care Physicians (three women and two
men) attended this discussion group. Four of them
work in the same location and one works in a
different one. This group was the hardest to get
together to make the group session, due to limited
availability and scheduling conflict. Nevertheless,
those that were present were very participatory thus
giving a positive contribution to the study. This
discussion came to the results that are presented in
Table 5.
The main issues that dominated the discussion
were:
a. System’s evolution;
b. Limitations / Main problems;
c. Security and trust in data transmission.
4 DISCUSSION
In the analysis of the obtained results we can
observe that there are several issues related to this
system of teleradiology common to all the Focus
Group: great satisfaction, increased ease of access
to this feature of health and the major improvements
that brought to the region. These common aspects
show the importance that the implementation of this
system represents for this region.
Apart from all the positive aspects mentioned by
participants of the Focus Groups, all groups showed
improvement suggestions. The group of Radiology
Technicians (FG1) was even interested and willing
to contribute to the progress and further evolution of
the system. They also added that the participation of
a Radiology Technician is very important in the
process of developing and implementing a system
like this. Results show that in addition to the
dedication that this group has for the success of this
system, which claiming to be in a large part in their
daily work, another key factor for this success is the
age of the elements of this team. Being a young
group that lives surrounded by encouraging new
technologies helps them achieve more effective
results when using a teleradiology system with
technological characteristics.
By contrast, some physicians from FG3 reported
that they had some problems with the use of the
system such as the difficulty of working with a
computer.
EVALUATION OF A TELERADIOLOGY SYSTEM - Impact and User Satisfaction
301

An important issue that emerged within FG2 was
the fact that they would be very much interested in
accessing their electronic medical record using a
computer or other emerging technologies.
The major difficulty of this study was to bring
together health professionals on the same day, same
time and keep them in a room for at least one hour,
taking into account that there is a big mismatch in
schedules and, in this case, with the aggravation of
having to move to a different place.
With the patients it was the difficulty to find
people willing to participate with time available, and
who could move to the place of the meeting.
This difficulty was increased by the fact that it
was important the presence of participants from at
least two different locations, in any of the groups, as
it was a Teleradiology system that was being
evaluated.
Another difficulty was that there is not published
material available on this specific research that aims
to evaluate the satisfaction of the System
teleradiology using focus groups.
5 CONCLUSIONS
Although the study has not yet been completed and
these results are only an initial analysis, which still
need further examination, the authors found that all
groups were mainly satisfied with the teleradiology
system and they felt that it was an important asset
for the region.
Moreover, we can infer that it is extremely
important not only to know the degree of satisfaction
but what leads to this satisfaction and what are the
main advantages and concerns regarding the system.
In short, it is important to hear those who work and
use the system so that, with their collaboration, can
be possible to improve further the existing
conditions. In this point, the focus group method has
proved to be effective and very important for this
study because it allowed knowing the opinions and
various relevant aspects experienced by all the
intervenient such as radiology technicians, patients
and primary care physicians.
The authors intend to conduct a second phase of
the study, where they will apply structured
questionnaires, preferably to the same participants or
the same category of participants that took part in
the discussion groups. The questionnaires will be
used to further explore the issues that came up and
were summarized with this preliminary analysis of
the discussion groups’ data.
One can also conclude that Focus Groups can be
interesting method to study other health information
systems, where it is important understand the impact
of its implementation, use and the satisfaction and
suggestions for improvement of those who use the
system.
REFERENCES
Coelho, P., et al., 2009. Perceptions, attitudes and
knowledge of evidence-based medicine in primary
care in Spain: a study protocol. BMC Health Services
Research, 9:80.
Curry, L., et al., 2009. Qualitative and Mixed Methods
Provide Unique Contributions to Outcomes Research.
Circulation; 119;1442-1452.
Ferreira, A., et al., 2010. Grounding information security
in healthcare. Int. J. Med. Inform. Apr, 79(4): 268-283.
Keny, Lm, Lau, Ls., 2008. Clinical teleradiology-the
purpose of principles. MJA. 188 (4): 197-198.
Rabiee, F., 2004. Focus-group interview and data analysis,
Proceedings of the Nutrition Society, 63, 655–660.
Romero G., et al. 2008. Telemedicine and
Teledermatology (I): concepts and applications. Actas
Dermosifiliogr. 99(7):506-22.
RSNA - Radiological Society of North America. 2010.
Available at: http://www.rsna.org/Technology/DI
COM/index.cfm.
Thrall, Jh., 2005. Reinventing radiology in the digital age.
Part II. New directions and new stakeholder value.
Radiology. 237(1):15-8.
Thrall, Jh., 2007. Teleradiology. Part I. History and
clinical applications. Radiology.243:613-617.
Thrall, Jh., 2007. Teleradiology. Part II. Limitations, risks,
and opportunities. Radiology. 244(2):325-8.
White, P., 2002. Legal issues in teleradiology-distant
thoughts! Br J Radiol. 75(891):201-6.
HEALTHINF 2011 - International Conference on Health Informatics
302