DESIGN OF A PROTOTYPE FOR PERFORMING HOSPITAL
BENCHMARKING
Production and Management of Hospital Quality Indicators
Juliano Gaspar, Nuno Rocha and Alberto Freitas
CIDES – Department of Health Information and Decision Sciences, Porto, Portugal
CINTESIS – Center for Research in Health Technologies and Information Systems, Porto, Portugal
Faculty of Medicine, University of Porto, Porto, Portugal
Keywords: Hospital benchmarking, Dashboards, Data quality problems, Hospital databases.
Abstract: Introduction: Hospitals are complex organizations and the quality and efficiency of care and the hospital
assessment performance are complex features to measure and estimate.
Aim: To extract an useful knowledge from hospital databases to develop a hospital dashboard for quality
and management indicators and to generate sufficient information, relevant and timely, to assist in decision-
making processes.
Methods: This study was structured with four distinct phases: preliminary study, literature review, definition
and development, and evaluation results.
Results: The preliminary studies were grouped in production and quality hospital indicators. In 2010 there
was a reduction (3.5%) in the total number of episodes, a reduction (30%) on exceptional short LOS
episodes, a significant reduction of obstetric complications, and an increase the problems related to clinical
coding.
Discussion: In a preliminary way, it can be observed the importance in a hospital management of such
results synthesized and summarized in groups of production and quality hospital indicators.
Conclusion: This work comprises the study of solutions can contribute to the improvement of healthcare
delivery, aiming towards management support and the desired hospital operating costs reduction.
1 INTRODUCTION
Medical databases usually involve several data types
and store large amounts of information. Among the
various health services, the hospital stands out for
gathering/organizing the procedures of greater
complexity and cost. The availability of reliable
information, from solid data, is vital to support
health professionals and technicians in their
decision-making and also to support hospital
managers.
Over the last 20 years, one of the major
challenges of most western countries health systems
has been dealing with rising health care costs,
which, in many cases, obliged systems to cut
spending in other sectors in order to provide the
public with a reasonable healthcare system (Ronen
and Pliskin, 2006).
Hospitals are often acknowledged as
organizations with the greater complexity in its
structure and administration (Freitas et al., 2010
);
(Freitas et al., 2011). The multiproduct nature in this
activity, due to an enormous variety of diagnoses
and procedures performed in a hospital and matters
related to hospital management, out of which
emerges the quality and efficiency care’s, delivery
and financing of health organizations, contribute to
the complexity of management, administration, cost
reduction, setting and measurement of production, as
well as the evaluation of hospital performance
(Gaspar et al., 2011
); (Costa et al., 2010).
The problems with the recent budgetary control
and the difficulties in the Portugal economy,
specially the urgent necessity to control the national
growth expenditure and the immediate cost
reduction, revealed a need to develop new solutions
312
Gaspar J., Rocha N. and Freitas A..
DESIGN OF A PROTOTYPE FOR PERFORMING HOSPITAL BENCHMARKING - Production and Management of Hospital Quality Indicators.
DOI: 10.5220/0003875603120317
In Proceedings of the International Conference on Health Informatics (HEALTHINF-2012), pages 312-317
ISBN: 978-989-8425-88-1
Copyright
c
 2012 SCITEPRESS (Science and Technology Publications, Lda.)

and strategies in the health public institutions
management (Largartinho and Anunciação, 2011).
In 2011 May, through the “Portugal
Memorandum of Understanding on Specific
Economic Policy Conditionality”, Portugal is
committed to fulfill measures aimed at the economy
stabilization in the country. Among health matters,
goals standout such as “Improve efficiency and
effectiveness in the health care system, inducing a
more rational use of services and control of
expenditures; […] generate additional savings in
hospital operating costs” (EFSM et al., 2011).
Specifically for this kind of services, two items stand
out (EFSM et al., 2011):
 3.75 - “Set up a system for comparing hospital
performance (benchmarking) on the basis of a
comprehensive set of indicators and produce regular
annual reports”;
 3.76 - “Ensure full interoperability of IT systems
in hospital, in order for the ACSS to gather real time
information on hospital activities and to produce
monthly reports to the Ministry of Health and the
Ministry of Finance”.
1.1 Dashboards
Dashboards can have an important contribution,
while providing a visual representation of main
performance indicators (Truttero, 2008), as they use
carefully selected indicators that help managers to
continuously monitor, measure and manage
performance of an institution (Korst et al., 2011). It
provides a pictorial representation of organizational
performance, while generating an overview of the
current institution situation, providing an intuitive
and timely strategic, financial and operational data
visualization.
The dashboards are characterized by an interface
consisting of one or more virtual instruments such as
dials or bar graphs, in which variables are associated
in order to be monitored as well as graphs showing
the evolution of variables (Wolpin, 2005).
A dashboard tool usage can help understand and
optimize hospital processes, as well as extend the
institutions self-knowledge (Hagland, 2011). Due to
specific characteristics and complex needs of
hospitals and management, there is not a default
dashboard (Norton, 2009). One of its greatest
qualities is the fact that it is structured, developed
specifically and designed to meet the real needs of
an institution (Riedel, 2007).
2 AIM
The main aim of this study is to extract knowledge
in hospital databases that might be useful to improve
the delivery of health care.
Specifically, it is intended to develop a
dashboard, a summary of quality and management
indicators for inpatient and outpatient episodes,
which may represent the production and the quality
of the hospital.
This study should provide relevant information
for the managers, to assist them in their decision
making processes, either at a local level (hospital), a
regional level (ARS
1
), or even at a national level
(ACSS
2
).
3 METHODS
The structure (acquisition, validation and
understanding) of existing data in hospital databases
will be a major focus on this work.
It is intended to evaluate the quality and
management indicators currently being used by
hospitals, analyze the results, propose new indicators
and, as well, assess the need to collect data from
other data sources of the institution (data
integration), e.g. administrative and organizational
databases.
It is expect to analyze the possible contributions
that techniques, such as Data Mining, Balanced
Scorecard, Business Intelligence and maturity
models for software development (CMMI
3
and
TDQM
4
), can bring to the objective proposed.
Finally, the data quality is an important factor to
be considered in the development of quality
indicators and dashboard, since the level of accuracy
and data quality is proportionally connected to the
significance and relevance of results.
In this prototype implementation, it is intended to
use technologies such as PHP, HTML, CSS and
JavaScript, as well as open source packages like
JQuery User Interface, Google Graphs API, and
others.
This study was divided into four distinct phases:
 Preliminary study;

1
ARS: Administração Regional de Saúde (in portuguese)
2
ACSS: Administração Central do Sistema de Saúde (in
portuguese)
3
CMMI: Capability Maturity Model Integration
4
TDQM: Total Data Quality Management
DESIGN OF A PROTOTYPE FOR PERFORMING HOSPITAL BENCHMARKING - Production and Management of
Hospital Quality Indicators
313

 Review of the literature;
 Definition and development of the prototype;
 Assessment of the results.
3.1 Preliminary Study
Accomplish a preliminary study with a sample of
data from a Portuguese hospital, in order to fit the
existing data previously in summarized tables and
getting a vision of the possible outcomes that the
dashboard can produce.
This study proposes to perform a comparative
study between the first semester of 2009 and first
semester of 2010 of the Portuguese hospital from
NHS
5
, involving approximately 29 thousand records.
The database used is composed by hospital episodes,
hospitalizations and ambulatory data from medical
or surgical specialties.
Access to this data was possible due to the ACSS
collaboration and the HR-QoD
6
project developed
by the Department of Health Information and
Decision Sciences (CIDES
7
) of Faculty of Medicine
of University of Porto. All data used was anonymous
and clinical data was coded in ICD-9-CM
8
and
DRG
9
.
3.2 Literature Review
The initial knowledge on the proposed topic should
be updated, detailed and systematic, consolidating
the respective knowledge base in this area. Thus it is
expected an initial literature review, with the
purpose of identifying quality indicators used in
other countries.
The purpose is to study quality and management
indicators used by hospitals, and analyze the results
and data acquisition processes. This step should
result in a consolidated set of indicators that are
common to hospitals and based on data currently
available in hospital databases.
After this stage, a study on techniques for
developing dashboards, balance scorecards, business
intelligence and data mining that can be applied in
healthcare data should be performed.

5
National Health Service
6
HR-QoD: Quality of data (outliers, inconsistencies and errors) in
hospital inpatient databases: methods and implications for data
modeling, cleansing and analysis
7
CIDES: Ciências da Informação e da Decisão em Saúde (in
portuguese)
8
ICD-9-CM: International Classification of Diseases, Ninth
revision, Clinical Modification
9
DRG: Diagnosis Related Group
And finally, the necessity for indicators related to
hospital, at the hospital level, regional, national and
international, should be defined. It is intended to
identify the indicators currently used and to identify
other needs for the aid on decision making by
managers.
3.3 Definition and Development
This stage should be used to:
- consolidate the results obtained from the
literature review, systematize and compare the
existing indicators, as well as the data acquisition
process.
- Identify possible measures of quality and health
quality indicators used in other countries, in order to
obtain a pool of potential candidates for
implementation.
- Evaluate the possibility of using data from
administrative and organizational databases
hospitals in developing the new indicators.
- Implement quality and management indicators
hospital defined. Implement the techniques and
selected parameters to create a dashboard prototype.
- Define and implement a set of user definable
alerts. Identify and present the possible
inconsistencies or errors data.
3.4 Results Evaluations
With the implemented indicators, results can be
evaluate their fidelity, reliability and compare the
results to local, regional, national and international,
with the assessment of experts in the field of hospital
management. It’s possible to determine the
relevance of the results and identify possible areas of
improvement.
Based on the results presented by the dashboard
is intended to evaluate and compare the quality and
management of hospital care at different levels of
the health system (local, regional and national). At
this stage, based on summaries and automatic
reports, there will be an evaluation of the provision
of health care, especially through the performance
and quality indicators pointing to possible problems
by comparing with predefined targets, in order to
help managers to develop strategies to overcome
problems and improve the delivery of health care as
well as to prevent problems and reduce costs by
optimizing existing resources.
Accomplish a comparative study sorted by years,
as well as a benchmarking a set of Portuguese
hospitals.
HEALTHINF 2012 - International Conference on Health Informatics
314

To analyze, evaluate and quantify data quality
problems existent in databases, it’s intended that the
figures presented in the dashboard, have a value
associated with the degree of reliability values
obtained when considering the amount of data
quality problems detected in the used data.
It is intended to correct any problems where
feasible, propose solutions and recommendations to
the authorities so they can avoid or resolve possible
problems, both in terms of data quality in general
and quality in the health context.
4 RESULTS
The preliminary study results are presented by
period (first half of 2010 and first half of 2009). The
results were grouped into production and quality
hospital indicators.
4.1 Hospital Production Indicators
Table 1: Number of Patients Discharged.
Description Year N %*
Global episodes
2010
14278 -
2009 14782 -
Medical DRG
2010
9966 69.8
2009 10282 69.6
Surgical DRG
2010
4312 30.2
2009 4500 30.4
*Percentage of total episodes
The table 1 shows de number of patients
discharged global and by type of DRG classification
(Medical DRG and Surgical DRG). It is observed a
reduction of 3.5% in the total volume of episodes in
2010 face the same period last year.
Table 2: Length-of-stay (LOS).
Description Year N
Global episodes
2010
7.7
2009 7.2
Medical DRG
2010
8.1
2009 7.8
Surgical DRG
2010
6.7
2009 5.9
The table 2 presents the global Length-of-Stay
(LOS) and Medical and Surgical LOS. It is verified
an increase of 0.5% in the Global LOS, a increase of
0.3% in the Medical DRG LOS and a increase of
0.8% in the Surgical DRG LOS.
Table 3: Episodes with Short LOS.
Description Year N %*
Global episodes
2010
179 1.25*
2009 254 1.72*
Medical DRG
2010
170 1.70**
2009 210 2.04**
Surgical DRG
2010
9 0.20***
2009 44 0.98***
* Percentage of total episodes
** Percentage of total Medical DRG episodes
*** Percentage of total Surgical DRG episodes
The table 3 shows the episodes with a
exceptional LOS, in this case, episodes with short
stay. It is observed a reduction of 30% in 2010 over
the same period in 2009, and approximately 86% are
Medical DRG episodes.
Table 4: Case-mix Index.
Description Year Value
Global episodes
2010
0.963
2009 0.976
Medical DRG
2010
0.748
2009 0.805
Surgical DRG
2010
1.422
2009 1.314
It is noted in Table 4 a reduction in the global
case-mix index in 2010. This decrease is related with
Medical DRG, since Surgical DRG case-mix index
increase.
4.2 Hospital Quality Indicators
Table 5: Readmission and Mortality.
Description Year N %*
Deaths
2010
779 5.5
2009 806 5.5
Readmission
2010
403 2.8
2009 398 2.7
* Percentage of total episodes
Table 5 shows a 3.4% reduction in hospital
mortality rate, however in the same period
prescribed, an increase of 2.2% in readmission rates.
Table 6: Complications in Medical DRG.
Description Year N %*
Pressure ulcers
2010
80 1.86
2009 103 2.29
Urinary infections
2010
406 9.41
2009 327 7.27
* Percentage of total Medical DRG episodes
The table 6 shows the complications in Medical
DRG. The hospital reduces the complications in
DESIGN OF A PROTOTYPE FOR PERFORMING HOSPITAL BENCHMARKING - Production and Management of
Hospital Quality Indicators
315

22.4% with pressure ulcers, although increases
19.5% the complications related with urinary
infections.
Table 7: Complications in Surgical DRG.
Description Year N %*
Hemorrhage or hematoma
2010
14 0.14
2009 27 0.26
Accidental puncture or laceration
2010
10 0.10
2009 0 0.00
Postoperative infections
2010
20 0.20
2009 15 0.15
Dehiscences
2010
29 0.29
2009 20 0.19
* Percentage of total Surgical DRG episodes
The table 7 shows the complications in specific
Surgical DRG like hemorrhages or hematoma,
accidental puncture or lacerations, dehiscenses and
postoperative infections.
Table 8: Obstetric Quality Indicator.
Description Year N %*
Deliveries
2010
843 -
2009 850 -
Cesarean
2010
360 42.7
2009 358 42.1
Birth trauma
2010
2 0.2
2009 19 2.2
Obstetrical trauma
2010
2 0.2
2009 43 5.0
Puerperium readmission
2010
1 0.1
2009 4 0.5
* Percentage of total deliveries
It is observed in table 8, that the hospital has a
high percentage of cesareans, 42.7% in 2010,
considering the national benchmark 32.9%. It is also
observed a significant reduction in 2010 of the
obstetric complications, such as birth trauma and
obstetrical trauma.
Table 9: Clinical Coding Quality Indicator.
Description Year N %
Non-specific principal diagnosis
2010
275 1.92*
2009 128 0.87*
Questionable principal diagnosis
2010
35 0.25*
2009 0 0.00*
Unacceptable principal diagnosis
2010
14 0.10*
2009 9 0.06*
Non-specific surgical procedures
2010
116 2.69**
2009 55 1.22**
* Percentage of total episodes
** Percentage of total Surgical DRG episodes
The table 9 presents the clinical coding quality
indicators. In general all indicators show in this table
show the increase of the problems related with the
clinical coding in 2010.
These indicators were chosen to exemplify some
of the possible outcomes that can display the
dashboard. A specific study should be conducted to
define what the best indicators to delineate the
hospital health.
5 DISCUSSION
It can be observed that the preliminary study results
on the hospital production have reduced in the total
volume of patients in 2010 over the same period in
2009, however there was a small increase of the
global LOS. There is a positive feature which 30%
the reduction of the episodes with exceptional short
LOS, considering that the hospital is not repaid if the
episode with LOS under the preset limit. On the
other hand, the case-mix index for the hospital was
reduced which could lead to a lower repayment by
the hospital with funding sources.
Related with the hospital quality, was observed
an improvement in mortality rates and complications
such as pressure ulcers in Medical DRG, birth
trauma, obstetrical trauma and puerperium
readmission. However, on the other side was also
observed a worse performance in hospital quality
when analyzed the rates of readmission,
complications in Surgical DRG, complications in
Medical DRG (urinary infections) and increase of
the volume of cesareans. They also observed that the
clinical coding quality is much worse in 2010
compared to 2009.
Preliminarily can be observed the importance for
the hospital management of these results,
synthesized and summarized in two groups, hospital
production indicators and hospital quality indicators.
This demonstrates the relevance and importance of
these kinds of studies and developments with the
intention of improve the quality of hospital health
care.
6 CONCLUSIONS
The scope of work includes the study of solutions
that can contribute to improve the supply of health
care, in order to help manage and reduce in
operating costs of NHS hospitals.
This work intends to understand an analysis of
the quality indicators and hospital management
already used by Portuguese hospitals, the
HEALTHINF 2012 - International Conference on Health Informatics
316

development of new indicators and the
implementation of a dashboard with the most
relevant information and summarized, that includes
information on hospital performance through an
intuitive visualization and timely data on the quality
of care, financial and operational costs of hospitals.
Evaluate the quality of data used is crucial to
obtain reliable results and accepted by the health
care providers and by the various kinds of managers
in health. From this assessment, we intend to
identify and propose improvements to data quality
problems detected.
6.1 Future Work
In a future work we intend to realize a literature
review, the definition of which indicators can be use
and a prototype development and evaluation of
results.
ACKNOWLEDGEMENTS
The authors would like to thank the support given by
the research project HR-QoD - Quality of data
(outliers, inconsistencies and errors) in hospital
inpatient databases: methods and implications for
data modeling, cleansing and analysis (project
PTDC/SAU-ESA/75660/2006).
REFERENCES
Costa, C., Costa, F. L. D., and Lopes, S., (2010).
Avaliação do Desempenho dos Hospitais Públicos
(Internamento) em Portugal Continental: Resultados
2009 (Escola Nacional de Saúde Pública da
Universidade Nova de Lisboa), pp. 113.
EFSM, EU, BCE, and FMI, (2011). Memorandum of
Understanding on Specific Economic Policy
Conditionality (17 May 2011), EFSM, EU, BCE, and
FMI, eds., pp. 36.
Freitas, A., Costa-Silva, T., Marques, B., Gaspar, J.,
Gomes, J., Lopes, F., and Lema, I., (2011). A
framework for the production and analysis of hospital
quality indicators. Paper presented at: ITBAM 2011-
2nd International Conference on Information
Technology in Bio- and Medical Informatics
(Toulouse, France).
Freitas, A., Silva-Costa, T., Marques, B., and Costa-
Pereira, A., (2010). Implications of data quality
problems within hospital administrative databases.
Paper presented at: 12Th Mediterranean Conference
on Medical and Biological Engineering and
Computing - MEDICON 2010 (Greece), pp. 823-826.
Gaspar, J., Lopes, F., and Freitas, A., (2011). Detecção de
Inconsistências em Dados Hospitalares:
Desenvolvimento de um Protótipo. Paper presented at:
CISTI 2011 - 6ª Conferência Ibérica de Sistemas e
Tecnologias de Informação (Chaves, Portugal), pp.
278-281.
Hagland, M., (2011). First place organization: Children's
Hospital & Medical Center. Data-driven. How
children's of Omaha is bending the curve on quality
and efficiency through dashboard use. In: Healthc
Inform 28, 11-12, 14, 24.
Korst, L. M., Aydin, C. E., Signer, J. M., and Fink, A.,
(2011). Hospital readiness for health information
exchange: Development of metrics associated with
successful collaboration for quality improvement. In:
Int J Med Inform.
Largartinho, L. M. G., and Anunciação, P. F. D., (2011).
Balance Scorecard ou Business Intelligence: O Centro
Hospitalar de Setúbal, EPE. Paper presented at: CISTI
2011 - 6ª Conferência Ibérica de Sistemas e
Tecnologias de Informação (Chaves, Portugal), pp.
515-518.
Norton, S. A., (2009). New Hampshire’s Healthcare
Dashboard 2009 (New Hampshire, New Hampshire
Center for Public Policy Studies), pp. 22.
Riedel, J. E., (2007). Using a health and productivity
dashboard: a case example. In: Am J Health Promot
22, 1-10, iii.
Ronen, B., and Pliskin, J. S., (2006). Focused Operations
Management for Health Services Organizations, Vol 1
(Jossey-Bass), 354.
Truttero, S., (2008). Keeping an eye on the dashboard: key
indicators of quality care. In: Pract Midwife 11, 4-5.
Wolpin, S.E. (2005). Design and implementation of an
intranet dashboard. In: AMIA Annu Symp Proc, 1154.
DESIGN OF A PROTOTYPE FOR PERFORMING HOSPITAL BENCHMARKING - Production and Management of
Hospital Quality Indicators
317