Measurement of Problem Difficulty Level in User Interface Medical

Management System (MMS) Application with Heuristic Evaluation

Approach at Surabaya Medical Services Hospital

Amir Ali

, Muhadi

, M. Udin Harun Al Rasyid

and Iwan Syarif

Departement of Medical Record and Information Technology, Stikes Yayasan Rumah Sakit Dr.Soetomo, Kalidami,

Surabaya, Indonesia

Department of Informatics and Computer Engineering, Politeknik Elektronika Negeri Surabaya (PENS),

Raya ITS Keputih Sukolilo, Surabaya, Indonesia

Keywords: SIMRS, Heuristic Evaluation, Severity Rating, MMS, SIRS.

Abstract: Surabaya medical service hospital has a Hospital Management Information System (SIMRS) application

called the Medical Management System (MMS). However, the level of difficulty of the problem on the user

interface has not been measured from SIMRS application. The user interface can be examined by using an

inspection approach, one of which is heuristic evaluation to determine the system's level of usability. The

problem can be rated according to its difficulty level by using severity ratings. This research is to measure the

level of difficulty of the user interface SIMRS MMS Surabaya Medical Service Hospital uses a heuristic

evaluation approach. The method used in this study was in form of distribute questionnaires to the research

object under investigation and apply the severity ratings methodology to calculate the findings of the heuristic

evaluation. According to the usability test with heuristic evaluation, the most serious usability issues are in

the areas of recognition rather than recall; help users recognize, diagnose, and solve problems with a severity

rating of 6.75, while the lowest score is in the areas of help users recognize, diagnose, and recover from errors

with a severity rating of 1. According to the results of the heuristic evaluation of the SIMRS MMS application,

it has usability issues with an average value of 2.42 on a scale of 2 which means the category of minor usability

problems where fixing this problem is given low priority.

1 INTRODUCTION

Sistem Informasi Rumah Sakit (SIRS) has become

necessity for hospital facilities. In line with the

Regulation of the Minister of Health (PERMENKES)

RI Number 1171/MENKES/PER/VI/2011

(Permenkes, 2011) concerning Hospital Information

System or (SIRS) stated that: "Every hospital is

obliged to carry out SIRS as a process of collecting,

processing, and presenting hospital data.

In relation to government’s regulation, Surabaya

Medical Service Hospital has been running an

Informasion Management Rumah Sakit (SIMRS)

application called Medical Management System

(MMS). This SIMRS application, however, has not

been measured against the difficulty of problems in

the user interface to determine how easy users of the

SIMRS application find it to use.

The user interface an application can be evaluated

using an inspection method, one of which employs

heuristic evaluation to determine the level of usability

of the system (Sulistiyono, 2017). Moreover,

usability concerns in SIMRS MMS applications can

be identified via heuristic evaluation. By using

severity ratings, the problem can be assessed based on

its level of difficulty (Ito, Yoshihiro. Nomura, 2013).

2 METHOD

In this study, the research population was users of

SIMRS MMS application with 10 users from the

registration counter, medical records, outpatient

service registration place or Tempat Pendaftaran

Pelayanan Rawat Jalan (TPPRJ), inpatient service

registration place or Tempat Pendaftaran Pelayanan

Rawat Inap (TPPRI), pharmacy, laboratory, billing

system.

The research variables used in this study are:

Ali, A., Muhadi, ., Rasyid, M. and Syarif, I.

Measurement of Problem Difﬁculty Level in User Interface Medical Management System (MMS) Application with Heuristic Evaluation Approach at Surabaya Medical Services Hospital.

DOI: 10.5220/0010949500003260

In Proceedings of the 4th International Conference on Applied Science and Technology on Engineering Science (iCAST-ES 2021), pages 587-594

ISBN: 978-989-758-615-6; ISSN: 2975-8246

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

587

a. Visibility of System Status

b. Match between system and the real word

c. User Control and Freedom

d. Consistency and Standard

e. Error Prevention

f. Recognition rather than recall

g. Flexibility and Efficient of Use

h. Aesthetic and Minimalist Design

i. Help Users Recognize, dialogue and recovers

from errors

j. Help and documentation

By using heuristic evaluation, usability issues in

SIMRS MMS applications can be identified. The

problem can be assessed according to the level of

difficulty of the problem (Severity Ratings).

2.1 Research Variable and

Measurement

The research variables, operational definitions, and

methods for measuring them are listed below

Table 1: Research Variables and Measurement.

Variable

Inspection

Variable

Definition

How to

Measure

(Visibility of

System Status)

The system should

give information to

the user about

everything happen

through the right

feedback in the right

time.

The severity

rating on usability

problems is

determined by a

scale 0 to 4.

1. Scale 0 : No

usability problem

2. Scale 1:

Cosmic problem

category

3. Scale 2: Minor

usability

category.

4.Scale 3 Major

usability category

5. Scale 4:

catastrophe

usability category

Match

between

system and the

real word

The system must use

the user language

with words, phrases,

and concepts that the

user understands

rather than the

language that the

system understands.

Furthermore, always

adhere to current

trends and arrange

Severity rating

level on usability

issue is

determined on a

scale of 0 to 4.

1. Scale 0: No

usability problem

2. Scale 1:

Cosmic problem

category

information in a

natural and orderly

manner.

3. Scale 2: Minor

usability

category.

4.Scale 3 Major

usability category

5. Scale 4:

catastrophe

usability category

User Control

and Freedom

Users frequently

make mistakes when

selecting functions or

buttons in the system,

whether on purpose

or unintentionally. In

this state, they

require a clear exit

sign to exit the

unwanted screen

without having to

begin or read a series

of instructions. As a

result, undo and redo

functions are

required.

Consistency

and Standard

The user should not

be confused;

otherwise, different

words, situations, and

actions will have the

same meaning. It

should be noted that

in the system, all

navigation must be

consistent.

Severity rating

level on usability

issue is

determined on a

scale of 0 to 4.

1. Scale 0: No

usability problem

2. Scale 1:

Cosmic problem

category

3. Scale 2: Minor

usability

category.

4.Scale 3 Major

usability category

5. Scale 4:

catastrophe

usability category

Error

Prevention

What is preferable to

displaying an error

message is to create a

good system design

that can anticipate the

appearance of

problems in the

system's early stages.

Also, before the user

proceeds with the

action, provide

confirmation options.

Severity rating

level on usability

issue is

determined on a

scale of 0 to 4.

1. Scale 0: No

usability problem

2. Scale 1:

Cosmic problem

category

3. Scale 2: Minor

usability

category.

4.Scale 3 Major

usability category

5. Scale 4:

catastrophe

usability category

iCAST-ES 2021 - International Conference on Applied Science and Technology on Engineering Science

588

Table 1: Research Variables and Measurement (cont.).

Variable

Inspection

Variable

Definition

How to Measure

Recognition

rather than

recall

Make objects,

actions, and

choices clear so

that users don't

have to remember

the same

information in

different sections.

Instructions for

using the system

should also be

easily accessible

whenever the user

requires them.

Severity rating

level on usability

issue is determined

on a scale of 0 to 4.

1. Scale 0: No

usability problem

2. Scale 1: Cosmic

problem category

3. Scale 2: Minor

usability category.

4.Scale 3 Major

usability category

5. Scale 4:

catastrophe usability

category

Flexibility and

Efficient of

Use

Accelerators,

which are often

overlooked by new

users, can speed up

interaction for

existing users.

That is what the

system should be

able to accomplish.

What should be

considered is that

the system must

provide functions

that both

experienced and

novice users can

understand.

Severity rating

level on usability

issue is determined

on a scale of 0 to 4.

1. Scale 0: No

usability problem

2. Scale 1: Cosmic

problem category

3. Scale 2: Minor

usability category.

4.Scale 3 Major

usability category

5. Scale 4:

catastrophe usability

category

Aesthetic and

Minimalist

Design

Dialogues should

not include

information that is

irrelevant or only

occasionally

required. More

specific

information in a

dialog should be

adapted to the unit.

Severity rating

level on usability

issue is determined

on a scale of 0 to 4.

1. Scale 0: No

usability problem

2. Scale 1: Cosmic

problem category

3. Scale 2: Minor

usability category.

4.Scale 3 Major

usability category

5. Scale 4:

catastrophe usability

category

Help users

recognize,

dialogue and

recovers from

errors

Error messages

should be

explained in clear

language (not

coded) that clearly

Severity rating

level on usability

issue is determined

on a scale of 0 to 4.

indicate the error

and suggest

solutions.

1. Scale 0: No

usability problem

2. Scale 1: Cosmic

problem category

3. Scale 2: Minor

usability category.

4.Scale 3 Major

usability category

5. Scale 4:

catastrophe usability

category

Help and

Documentation

Although it is

preferable if the

system can be used

without

documentation,

help and

documentation

may be required.

Each piece of

information should

be easily

accessible, relevant

to the user's task,

provide concrete

steps of

workmanship, and

not be overly

lengthy.

Severity rating

level on usability

issue is determined

on a scale of 0 to 4.

1. Scale 0: No

usability problem

2. Scale 1: Cosmic

problem category

3. Scale 2: Minor

usability category.

4.Scale 3 Major

usability category

5. Scale 4:

catastrophe usability

category

2.2 Heuristic Evaluation Aspect and

sub aspect

Heuristic evaluation value was obtained by doing

calculations based on the following heuristic

evaluation aspect table (Farida, Dwi, 2016)

Table 2: Heuristic Evaluation Aspect.

Usability Aspect Kode

Visibility of the system status H1

Compatibility between the system and

the reality

User control and freedom H3

Consistency and standardize H4

Error prevention H5

Help users identify, diagnose and

resolve problems

Flexibility and Efficiency H7

Minimalist and aesthetic design H8

Assist users in recognizing, conversing

about, and correcting errors.

Documentation and help feature H10

Measurement of Problem Difﬁculty Level in User Interface Medical Management System (MMS) Application with Heuristic Evaluation

Approach at Surabaya Medical Services Hospital

589

The average results of each attribute in each

aspect / principle of heuristic evaluation above were

used to calculate questionnaire results.

Calculations on heuristic evaluation using equations

(1):

∑=0∗+1∗+2∗+3∗+4∗… (1)

∑ = Number of rating scores from sub aspects of

usability in each aspect of usability (H1, H2, …, H10)

= usability points (valued at 1/0)

Then, using equation (2), generate a severity rating

value for each aspect of usability:

(2)

 = Severity rating has an effect on one aspect of

usability

 = the number of usability sub-aspects in each

usability aspect

The severity level of a usability problem can be

determined using the following scale of 0 to 4

a. Scale 0: There are no usability issues.

b. Scale 1: Cosmetic problem; problem does not

need to be fixed unless there is time remaining in

the project.

c. Scale 2: Category minor usability issue, resolve

this issue given a low ranking

d. Scale 3: Category major usability issue, resolve

this issue given top priority.

e. Scale 4: Category usability disaster; this issue

must be resolved prior to the product's release

Here is a list of heuristic evaluation aspects and

sub-aspects of heuristic evaluation

Table 3: Usability Aspect and sub aspects.

Aspect Sub Aspect Usability

Visibility of the

system status

1. Each page has a title that describes

the page's content.

2. Each symbol or icon, as well as the

design scheme, on each page is

consistent.

3. There is a visually distinguishing

response when an object (button, option

button) is pressed or selected.

4. The menu and page names

correspond to the content.

5. The display menu can already

distinguish between the currently

selected menu and those that are not.

Compatibility

between the

system and the

reality

1. Icons that can be used by anyone

2. The menu name is written logically

and is understood by the user.

3. The shape or image used is

appropriate for the user's culture.

4. For users who are actively using the

app, there is a language option.

User control

and freedom

1. There is a help button if the system

does not display the results of any

process, such as if an error occurs.

2. Users have the ability to search for

data in a variety of ways.

3. If the system has a nested menu, the

user should be able to easily navigate to

the previous page.

Consistency

and standardize

1. Each page is labeled with a title.

2. Each page has consistently written

standard writing.

3. Each page's title is consistent in terms

of font shape, size, and paragraph

length.

4. The appearance of the form on the

web for each page is consistent and the

same.

5. Other commonly spoken languages

are available as language options.

6. Not only images can be displayed,

but there are also accessibility standards

for users on each page, which is

especially important for those with

special needs (blind people, deaf

people)

Error

prevention

1. The text provided is in clear

instructions and avoids ambiguity

2. All information has been properly

organized.

3. Each page contains navigation

instructions for the user.

Help users

identify,

diagnose and

resolve

problems.

1. When it is unable to access the page,

an error message appears.

2. When a user makes a mistake when

making changes, a warning sign

appears.

Flexibility and

Efficiency

1. The application page displays content

in the language selected by the user.

2. Menus and other information are

presented in an attractive manner.

3. Group menus and other information

are simple to recall.

4. On each page, there is a navigation

menu that can assist us.

5. The Navigation Menu is in the proper

location.

6. The search menu is simple to find and

use.

iCAST-ES 2021 - International Conference on Applied Science and Technology on Engineering Science

590

Table 3: Usability Aspect and sub aspects (cont.).

Aspect Sub Aspect Usability

Minimalist and

aesthetic design

1. There is a variety of foreign

languages available to accommodate

users from other countries.

2. The search menu is simple to

remember and use, especially for

inexperienced users.

3. The menu's layout is very familiar

and easily accessible to the user.

4. The system allows you to change the

size of the letters.

5. There is no color selection as a

system action code.

Assist users in

recognizing,

conversing

about, and

correcting

errors

1. The information displayed on each

page is clear, allowing the user to make

subsequent decisions.

2. The use of appropriate font sizes and

types on each page makes visitors feel

at ease.

3. Each page's structure is consistent

and uniform.

4. Each page's title is clear and

informative.

5. There are no irrelevant

characteristics.

Documentation

and help feature

1. There is a menu map, so users can

easily see what is available.

2. There is a help menu that can assist

users in navigating the site more

effectively.

3. There is contact information/

correspondent information from the

page's owner.

3 RESULT AND DISCUSSION

3.1 Result

The display page of SIMRS MMS application was

evaluated as shown below:

Figure 1: Interface Page Display Cashier Registration

Menu.

Figure 2: Pharmaceutical Interface Page Views.

Figure 3: Laboratory Interface Page View.

Figure 4: Master Menu Interface Page View.

The following are the results of the usability

aspects calculation using the heuristic evaluation

method:

Table 4: Calculation of Usability Aspects and Sub Aspects

of Usability 1.

Aspect

Usability

Sub

Aspect

Usability

Total

Score

0 1 2 3 4

A B C D E F G I J

1 1 7 1 0 0 2 9 1,8

2 7 2 0 0 1 6 1,2

3 8 0 1 1 0 5 1

4 7 2 0 1 0 5 1

5 8 1 0 1 0 4 0,8

37 6 1 3 3

5,8

0 6 2 9 12 1 1,16

Description: SR (Severity Rating)

The calculation for heuristic evaluation is shown

in Table 4. The value of the heuristic evaluation is

Measurement of Problem Difﬁculty Level in User Interface Medical Management System (MMS) Application with Heuristic Evaluation

Approach at Surabaya Medical Services Hospital

591

represented by the list of severity rating values in the

C,D,E,F,G column. Column I is a column that

contains the number of severity ratings obtained by

adding the severity rating values together. According

to the equation (1) that

I = (0*C1)+(1*D1)+(2*E1)+(3*F1)+(4*G1)

Furthermore, in column J, the severity rating value

obtained is J = I/5, where 5 is the number of usability

sub-aspects in table 3.

Table 5: Calculation of Usability Aspects and Sub Aspects

of Usability 2.

Aspect

Usability

Sub

Aspect

Usability

Total

Score

0 1 2 3 4

A B C D E F G I J

2 1 5 3 0 0 2 11 2,75

2 9 0 0 0 1 4 1

3 7 2 1 0 0 4 1

4 6 2 1 0 1 8 2

27 7 2 0 4

6,75

0 7 4 0 16 2 1,69

Description: SR (Severity Rating)

Table 6: Calculation of Usability Aspects and Sub Aspects

of Usability 3.

Aspect

Usability

Sub

Aspect

Usability

Total

Score

0 1 2 3 4

A B C D E F G I J

3 1 3 1 2 0 4 21 7

2 6 1 1 1 1 10 3,33

3 7 1 0 1 1 8 2,67

16 3 3 2 6

13,00

0 3 6 6 24 4 4,33

Description: SR (Severity Rating)

Table 7: Calculation of Usability Aspects and Sub Aspects

of Usability 4.

Aspect

Usability

Sub

Aspect

Usability

Total

Score

0 1 2 3 4

A B C D E F G I J

4 1 7 0 2 0 1 8 1,33

2 8 0 1 0 1 6 1

3 8 0 2 0 0 4 0,67

4 8 0 1 0 1 6 1,00

5 5 1 3 0 1 11 1,83

6 6 2 1 0 1 8 1,33

42 3 10 0 5 7,17

0 3 20 0 20 1 1,19

Description: SR (Severity Rating)

Table 8: Calculation of Usability Aspects and Sub Aspects

of Usability 5.

Aspect

Usability

Sub

Aspect

Usability

Total

Score

0 1 2 3 4

A B C D E F G I J

5 1 6 1 0 0 3 13 4,33

2 7 2 1 0 0 4 1,33

3 6 2 1 1 0 7 2,33

19 5 2 1 3

8,00

0 5 4 3 12 3 2,67

Description: SR (Severity Rating)

Table 9: Calculation of Usability Aspects and Sub Aspects

of Usability 6.

Aspect

Usability

Sub

Aspect

Usability

Total

Score

0 1 2 3 4

A B C D E F G I J

6 1 5 1 0 1 3 16 8

2 5 2 1 1 1 11 5,5

10 3 1 2 4

13,5

0 3 2 6 16 7 6,75

Description: SR (Severity Rating)

Table 10: Calculation of Usability Aspects and Sub Aspects

of Usability 7.

Aspect

Usability

Sub

Aspect

Usability

Total

Score

0 1 2 3 4

A B C D E F G I J

7 1 7 2 0 0 1 6 1,00

2 6 1 1 2 0 9 1,50

3 8 0 1 1 0 5 0,83

4 6 3 0 0 1 7 1,17

5 8 0 1 0 1 6 1,00

6 6 3 0 0 1 7 1,17

41 9 3 3 4

6,67

0 9 6 9 16 1 1,11

Description: SR (Severity Rating)

Table 11: Calculation of Usability Aspects and Sub Aspects

of Usability 8.

Aspect

Usability

Sub

Aspect

Usability

Total

Score

0 1 2 3 4

A B C D E F G I J

8 1 4 3 2 0 1 11 2,2

2 8 0 1 0 1 6 1,2

3 7 1 1 0 1 7 1,4

4 7 1 2 0 0 5 1

5 3 2 3 1 1 15 3

29 7 9 1 4

8,8

0 7 18 3 16 2 1,76

Description: SR (Severity Rating)

iCAST-ES 2021 - International Conference on Applied Science and Technology on Engineering Science

592

Table 12: Calculation of Usability Aspects and Sub Aspects

of Usability 9.

Aspect

Usability

Sub

aspect

Usability

Total

Score

0 1 2 3 4

A B C D E F G I J

9 1 7 1 1 0 1 7 1,4

2 7 1 2 0 0 5 1

3 7 2 1 0 0 4 0,8

4 8 1 0 1 0 4 0,8

5 7 1 2 0 0 5 1

36 6 6 1 1

0 6 12 3 4 1 1

Description: SR (Severity Rating)

Table 13: Calculation of Usability Aspects and Sub Aspects

of Usability 10.

Aspect

Usability

Sub

Aspect

Usability

Total

Score

0 1 2 3 4

A B C D E F G I J

10 1 5 2 1 1 1 11 3,67

2 6 2 1 1 0 7 2,33

3 7 1 2 0 0 5 1,67

18 5 4 2 1

7,67

0 5 8 6 4 3 2,56

Description: SR (Severity Rating)

Table 14: Severity Rating Recapitulation In SIMRS MMS

Application.

Usability

Aspect

Average Value

Of Severity Rating

Rounding

Value

Scale 0-4

1 1,16 1

2 1,69 2

3 4,33 4

4 1,19 1

5 2,67 3

6 6,75 7

7 1,11 1

8 1,76 2

9 1 1

10 2,56 3

Rounding

Value

Scale 0-4

2,42 2

3.2 Discussion

The distribution of questionnaires against the

research objects investigated, as well as the

calculation of heuristic evaluation findings using

severity ratings approaches, were the methods used in

this study.

The following are the steps of the methods used

in this study:

1. Problem Identification

2. Literature of the study

3. Arrangement and Distribution of Questionnaire

4. Data collection

5. Usability measurement using heuristic evaluation

of SIMRS MMS

6. The result of heuristic evaluation Analysis

To calculate the difficulty level of the problem in

the SIMRS SMS application user interface using the

inspection method with a heuristic evaluation

approach, the following equation is used in the

calculation of the value of the questionnaire

evaluation results:

Heuristic evaluation calculations using equation

(1):

∑=0∗+1∗+2∗+3∗+4∗… (3)

∑ = Number of rating scores from sub aspects of

usability in each aspect of usability (H1, H2, …, H10)

= usability points (valued at 1/0)

Furthermore, using equation (2), calculate the

severity rating value for each usability aspect:

(4)

 = Severity rating has an effect on one aspect of

usability

 = the number of usability sub-aspects in each

usability

4 CONCLUSIONS

The following conclusions are drawn from the results

of the SIMRS MMS application usability test:

1. Based on usability test with heuristic evaluation,

the biggest usability problem is in the aspect of

recognition rather than recall; help users to

recognize, diagnose and solve problems with a

severity rating value of 6.75 while the lowest

value is in the Help users recognize, diagnose

and recover from error aspect; Help users

identify, diagnose, and recover from errors with

a severity rating of 1.

Measurement of Problem Difﬁculty Level in User Interface Medical Management System (MMS) Application with Heuristic Evaluation

Approach at Surabaya Medical Services Hospital

593

2. It received a value of 2 in the 7th aspect of the

heuristic evaluation technique, namely flexibility

and efficiency where the menu and other

information sub-aspects are well packaged,

indicating that it is necessary to prioritize

improvements from that side for SIMRS MMS

applications.

3. The overall average value of all usability aspects

is 2.42 or a scale of 2, indicating that this problem

categorized into minor usability problems, with

repair given low priority

REFERENCES

Farida, Dwi, L. (2016). Pengukuran user experience dengan

pendekatan usability (studi kasus: website pariwisata di

asia tenggara). Seminar Nasional Teknologi Informasi

Dan Multimedia 2016.

Ito, Yoshihiro. Nomura, Y. (2013). Evaluation of Influence

of IP QoS Degradation on Web-QoE by Severity Rating

Method. Asia-Pacific Conference on Communications

(APCC).

Permenkes, M. (2011). Permenkes RI Tentang SIMRS.

Sulistiyono, M. (2017). Evaluasi heuristic sistem informasi

pelaporan kerusakan laboratorium universitas amikom

yogyakarta. Jurnal Ilmiah DASI, 18.

iCAST-ES 2021 - International Conference on Applied Science and Technology on Engineering Science

594