MDFLUXO: OPHTALMOLOGY EDUCATION WITH A PDA

EFFICACY AND USABILITY EVALUATION

Vladimir Camelo Pinto, Thiago Martini da Costa, Marcelo Contardo Moscoso Naveira

Daniel Sigulem, Paulo Schor and Ivan Torres Pisa

Medical Informatics Department, Federal University of São Paulo (UNIFESP), Rua Botucatu 862, São Paulo, SP, Brazil

Keywords: PDA, Students, Health professionals, Usability.

Abstract: In the last decade, handheld computers, also known as Personal Digital Assistants (PDAs), have become

popular among physicians, residents and medical students. We have developed a PDA-software, called

MDFluxo, to assist ophthalmologic teaching as a guide. We used a user-centered design to try to diminish

usability problems. It’s interesting that the literature concerning PDA use on health area doesn’t emphasize

usability attributes and it’s evaluation, which is important to widespread PDA use among health profession-

als. MDFluxo efficacy was evaluated comparing it to traditional book guide and a non-guide self-learning.

Usability inspection methods, such as Think Aloud Method, Nielsen Heuristic Evaluation, QUIS 5.0 adapta-

tion, and Cognitive Walkthrough are being used to identify usability problems. The usability inspection

methods help us to identify usability problems and correct them. We evidenced that the students who used

MDFluxo improved the resolution of the clinical case when compared to participants which did not use any

material as support, however there was no significant difference when comparing the MDFluxo students to

students using a book, thus, as a support to ophthalmology learning, MDFluxo is an efficacy tool.

1 INTRODUCTION

In the last decade, handheld computers, also known

as Personal Digital Assistants (PDAs), have become

popular among physicians, residents and medical

students. It is estimated that near 60-70% of the

medical students and residents on the USA use PDA

for some purpose related to health (Kho et al.,

2006). Some equipment characteristics such as: size,

mobility, objectivity at offering relevant clinical in-

formation can be related to the observed spread

(Kho et al., 2006). The main applications used

among medical students and residents are: research

tools (Sutton, 2004), medical calculators (Honey-

bourn et al., 2006), bibliographical references (Rud-

kin et al., 2006), and drugs databases (Stroud et al.,

2005). We didn’t find papers related to health pro-

fessionals and medical students PDA usage on Bra-

zil, although we are managing an electronic research

to get this data (http://telemedicina.unifesp.br/palm).

Widespread use and growth in the use of PDA

can be due to it usability. On a recent review

(http://telemedicina.unifesp.br/mdfluxo) that we

have performed, we have evidenced the increasing

use of PDA in the health environment, as well as

PDA-software efficiency and effectiveness in the aid

of teaching and medical practice. There were few

studies regarding PDA-software development cen-

tering on health professional opinions. This could

generate problem in understanding results, since

Nielsen (1993) asserts “is virtually impossible to de-

sign a user interface that has no usability problems

form the start.

The aim of this paper is to present MDFluxo, for

learning ophthalmology based on handheld com-

puters, and its evaluation among medical students.

There will also be introduced preliminaries results

concerning usability inspection methods we are ap-

plying.

2 MATERIALS AND METHODS

The MDFluxo software is based on the book “Fed-

eral University of São Paulo (UNIFESP) Guide for

Ambulatories Medicine - Ophthalmology” written

by Schor et al. (2004). In the opening pages Schor et

al. (2004) mention that the book is both a clinical

and surgery manual and that it was created to help

medical practice, describing the most common le-

227

Camelo Pinto V., Martini da Costa T., Contardo Moscoso Naveira M., Sigulem D., Schor P. and Torres Pisa I. (2008).

MDFLUXO: OPHTALMOLOGY EDUCATION WITH A PDA EFFICACY AND USABILITY EVALUATION.

In Proceedings of the First International Conference on Health Informatics, pages 227-230

 SciTePress

sions in each human eye segment, followed by ana-

tomical location. Each chapter possesses a flowchart

that allows grouping manifestations by their com-

mon characteristics. The book offers a total of 55

flowcharts regarding each disease. Each flowchart

summarizes the medical knowledge discussed in the

current chapter and offers mnemonics to compact

the text with objectiveness.

The flowcharts extracted from the book were,

first revised by specialists, and then transcribed to

eXtensible Markup Language (XML) due to its

flexibility, enhanced integration and SOA-oriented

services for different environments (Pinto et al.,

2006).

On this study, we used Palm OS Simulator

ver-

sion 5.2.1 to simulate a Palm environment on desk-

top PC. We also used palmOne

Tungsten E hand-

held devices with 320x320 screens. On the desktop

PC, we had installed Microsoft Visual Basic

6.0

with the plug-in MobileVB

4.0.0 of the

AppForge

, and database Microsoft Access

2003.

According to general Human-Computer Interface

(HCI) literature and our expertise on developing

health-applications, a close relationship between

health professionals is vital to avoid failures. Thus,

we had chosen a user-centered design process (Nor-

man, 1983) in which we included ophthalmologists,

physicians and residents on prototyped development

process.

2.1 Usability Inspection

Trying to evaluate the five usability quality compo-

nents: learnability, efficiency, memorability and sat-

isfaction (Nielsen, 2003), we planned to use the fol-

lowing methods: Think Aloud Method, Nielsen’s

Heuristic Evaluation, Questionnaire for User Inter-

face Satisfaction (QUIS 5.0) and Cognitive Walk-

through.

A Think Aloud Method (Someren et al., 1994)

was applied to ophthalmology specialists, on a la-

boratorial environment, to evaluate learnability and

efficiency in a simulated case-solving.

Nielsen’s heuristic evaluation (Nielsen, 1994)

was applied by 5 biomedical informatics students to

identify problems on the five usability components.

There are two other usability inspection methods

that will be applied. We translated to Brazilian Por-

tuguese language and adapted the QUIS 5.0 (Chin et

al., 1988). We planed to apply it on several voluntry

physicians to assess their satisfaction using

MDFluxo. A Cognitive Walkthrough Evaluation

Process (Lewis et al., 1990) is being planned to

quantify the efficiency, the learnability, and memo-

rability with medical student in a PBL evaluation.

2.2 MDFluxo Comparing

We didn’t intend to evaluate MDFluxo as a substi-

tute to the traditional learning. On this paper we

compared if MDFluxo is a valid tool to complement

the ophthalmologic learning.

One of the authors (Paulo Schor) developed three

different, equivalent clinical cases to assess

MDFluxo efficacy. Patient's main and secondary

complains, personal info, clinical history, and related

issues were included on the clinical cases.

From UNIFESP and Faculdade de Ciências

Médicas de Santos – Centro Universitário Lusíada

(FCMS-UNILUS), fourth and fifth year medical stu-

dents were divided in three groups. The first group

used MDFluxo as a didactic support, the second

group used the Guide of Ophthalmology book

(Schor et al., 2004) and the last one group didn’t use

any didactic material as support to solve the clinical

case.

Each participant was required to answer three

questions: the diagnostic hypothesis, etiology for

them, and conduct. To each right answer, half right

and wrong it will be attributed values 1, 0.5 and 0

respectively and the accomplishment of the trial was

timed.

The students’ response was corrected and the

data tabulated in an electronic spreadsheet. For each

question of the test (diagnostic hypothesis, etiology,

conduct) the grades and time spent to answer were

evaluated. We used the Friedman’s test (Siegel,

1988) to verify the variance among the groups.

3 RESULTS

The MDFluxo project resulted in 22 prototype ver-

sions, each one built considering the physicians

opinion, regarding flowcharts content, usability at-

tributes, including layout design. We evidenced

great differences between initial and final proto-

types, as shown in Figure 1 and Figure 2 respec-

tively.

Figure 1: Two screenshots from 5

prototype, with several

buttons and few information.

HEALTHINF 2008 - International Conference on Health Informatics

228

Figure 2: Two screenshots from 22

prototype, with more

information and improved navigability.

3.1 Usability Inspection

Some preliminaries results of usability inspection

that we are conducting are listed below:

1. We diminished the amount of options present

on preliminary releases to clear the screen.

2. There were no icons on the system. Icons in-

creases the mapping with the real world.

3. The exclamation point (“!”) used for “View info

on the Author” and the button “Close” have not

a consistent use on the system.

4. The red color is often used in the system for

normal instruction of pure text, however, this

color should be used only for critical alerts.

5. It is not obvious that the “Eye Segment” is se-

lectable.

6. There were grammar and writing mistakes on ti-

tles and menus and generated texts.

7. For the “gender” and “age” we used Select Box,

which does not follow the WIMP standard.

8. The “Instructions” option doesn’t offer a step-

by-step manual, but a short explanation on each

button function. The names “Instructions” and

“Project Information” are not appropriated to

their meaning.

9. In some screens the user could not click on the

text and change it, however, this feature was

wrongly enabled.

10. Error reports were shown several times, such as

“Form was unable to load” and “subscript out

of range” during the system loading.

11. The “Help” function is only presented on the

starting screen. During navigation there is no

other “Help” button or “Return to Home/Main”.

3.2 MDFluxo Comparing

The descriptive statistic, which describes the study

results, is present on Table 1. On this table we pre-

sent average results for the groups that used

MDFluxo on PDA, book and the group that didn’t

use anything for each question and time.

Table 1: Descriptive statistics containing average values

for each studied feature.

Applying Friedman’s test to verify statistical dif-

ferences in the values presented on Table 1, we evi-

denced that using PDA leads to a better etiology

resolution and conducts when compared to group

that doesn’t use any material.

Beyond what has been stated, there was no sig-

nificant result for resolution of diagnostic hypothesis

in comparison to the other two groups. In the analy-

sis among the medium time required for the accom-

plishment of the clinical case among three groups,

the use PDA as reference material was better than

the book and without the aid of any material. How-

ever it should be taken into consideration that the

students did not receive previous training for the use

of the software.

On Table 2 we present the statistical difference

in diagnostic hypothesis, etiology, conduct, time and

general result for MDFluxo comparing to book and

students with nothing.

Table 2: Analysis of variance according to Friedman’s

test. (*) p-value > 0.050 (without significance).

Diagnostic

hypothesis

Etiology Conduct Time

General

Result

calc =

1.720

(p < 0.423)

calc =

9.140

(p < 0.010)

calc =

6.140

(p < 0.050)

calc =

8.400

(p < 0.015)

calc =

4.630

(p <

0.099)

PDA & Book

> Without

nothing

PDA > With-

out nothing

Book > PDA

& Without

nothing *

4 CONCLUSIONS

There are many boundaries on PDA-software devel-

opment, especially when we consider the health en-

vironment and its applications. For MDFluxo pro-

ject, these restraints only reinforced the role of the

participating user during the development process. A

weak mnemonic gap could have been created be-

cause, due to the PDA screening limitation, flow-

charts are showed in parts, instead of the whole, as

in the book. However the group that used PDA-

software MDfluxo achieved better result than the

group that did not use any material for the resolution

clinical case.

Diagnostic

hypothesis

Etiology Conduct Time

General

Result

PDA

0.600 0.350 0.450 16.980 1.400

Book 0.650 0.325 0.425 14.420 1.400

Without

nothing 0.475 0.050 0.125 12.220 0.650

MDFLUXO: OPHTALMOLOGY EDUCATION WITH A PDA EFFICACY AND USABILITY EVALUATION

229

The participants that used PDA-software

MDFluxo did not have any difficulties to navigate

through the different. However the amount of flow-

charts presented by the software did not correspond

to the users' expectations.

The ophthalmologic clinicians and students

found MDFluxo as a portable, quick, and intuitive

guideline and a method of learning on managing

clinical cases. Improvements, such as considering

the usability on the development process, could turn

the PDA into a more useful resource to physicians,

enhancing the assistance of patients.

A research was acconmplished in medical data-

base with objective of finding in the medical litera-

ture representations of ophthalmologic information

similar to the representations of the PDA-software

MDFluxo that were extracted of the book of Schor et

al., (2004). However, it was not found similar stud-

ies it to allow accomplishing comparisons with the

results of this study.

ACKNOWLEDGEMENTS

We thank the support provided by the Ophthalmol-

ogy Department, Medical Informatics Department,

and Biostatistics and Technology Information Divi-

sion of UNIFESP, and the volunteers. This project

was accomplished with CAPES master’s scholar-

ship.

REFERENCES

Chin, J.P., Diehl, V.A., Norman, K.L., 1988. Development

of an instrument measuring user satisfaction of the

human-computer interface. In Proceedings of the

SIGCHI Conference on Human Factors in Computing

Systems. Washington, D.C., United States. ACM

Press, New York, pp.213-218.

Honeybourne, C., Sutton, S., Ward, L., 2006. Knowledge

in the Palm of your hands: PDAs in the clinical set-

ting. Health info libr J. 23(1), pp.51-9.

Kho, A., Henderson, L.E., Dressler, D.D., Kripalani, S.,

2006. Use of handheld computers in medical educa-

tion. J Gen Intern Med. 21(5), pp.531-537.

Lewis, C., Polson, P.G., Wharton, C., Rieman, J., 1990.

Testing a walkthrough methodology for theory-based

design of walk-up-and-use interfaces. In Proceedings

of the SIGCHI. Seattle, Washington, D.C., J. C. Chew

and J. Whiteside, Eds. CHI '90. ACM Press, New

York, pp.235-242.

Nielsen, J., 1993. Iterative user-interfaces design. IEEE

Computer. 26(11), pp. 32-41.

Nielsen, J., 1994. Heuristic evaluation. In: Nielsen J, Mack

RL (Eds.). Usability Inspection Methods, John Wiley

& Sons, New York, NY.

Nielsen, J., 2003. Usability 101: Introduction to Usabilty.

(online). Available from: http://www.useit.com/

alertbox/20030825.html. (cited 11 July 2007)

Norman, D.A., 1983. Design principles for human-

computer interfaces. In Proceedings of the SIGCHI.

Boston, Massachusetts, United States, A. Janda, Ed.

CHI '83. ACM Press, New York, NY, pp.1-10.

Pinto, V.C., Neira, R.A.Q., Bottós, J.M., Bottós, K.M.,

Macedo, A.A., Schor, P., Pisa, I.T., 2006. MDFluxo:

Ferramenta de auxílio ao ensino em oftalmologia

baseada em computador de mão. XCBIS’2006.

Florianópolis, SC.

Rudkin, S.E., Langdorf, M.I., Macias, D., Oman, J.A.,

Kazzi, A.A., 2006. Personal digital assistants change

management more often than paper texts and foster pa-

tient confidence. Eur J Emerg Med. 13(2), pp.92-6.

Schor, P., Chamon, W., Belfort, R., 2004. Guia de

Oftalmologia - Série Guia de Medicina Ambulatorial e

Hospitalar da UNIFESP/EPM. 1 ed. Manole, São

Paulo.

Siegel, S., Castellan, Jr. N.J., 1988. Nonparametric

Statistics. 2 ed. McGraw-Hill Int. Ed, New York, pp-

399.

Sutton, J., Stockton, L., McCord, G., Gilchrist, V.J., Fe-

dyna, D., 2004. Handheld computer use in a family

medicine clerkship. Acad Med. 79(11), pp.1114-9.

Stroud, S.D., Erkel, E.A., Smith, C.A., 2005. The use of

personal digital assistants by nurse practitioner stu-

dents and faculty. J Am Acad Nurse Pract. 17(2),

pp.67-75.

van Someren, M.W., Barnard, Y.F., Barnard, J.A.C., 1994.

The Think Aloud Method: A Practical Guide to Mod-

elling Cognitive Processes. ISBN-0-12-714270-3.

Academic Press, Inc., 525 B Street, Suite 1900, San

Diego.

HEALTHINF 2008 - International Conference on Health Informatics

230