EVALUATION OF THE QUALITY OF THE INTERACTION

BETWEEN USERS AND CUSTOM-MADE SOFTWARE

PRODUCTS

Simone Vasconçelos Silva

North Fluminense State University ,Av. Alberto Lamego, 2000, Horto. Campos-RJ, Brazil

Daniel I. De Souza Jr.; Ph.D.

Fluminense Federal and North Fluminense State Universities, Brazil

University of Florida, ISE Department, Gainesville, FL, USA

Keywords: Quality, usability of software products, custom-made software products

Abstract: In this work, we will develop a methodology based on the quality of the interaction between users and

custom-made software products, aiming at the determination of the degree of users’ satisfaction, that is, the

usability. This approach is divided into four parts: First we will define the criterions to be used in the

evaluation of the quality of software products. After these criterions have been chosen, an approach will be

developed to classify them by their level of importance related to a selected area of application. The level of

importance of a usability criterion will vary depending on the area of application. As a next step, a second

approach will be introduced to classify usability criterions by the quality level presented by a certain

software product utilized in a selected area. This classification is based on the users’ opinion about the

quality level a certain usability criterion related to the above software product would have. Finally a third

and final approach will be set up to evaluate the usability of software products according to users’ judgment.

It combines the first method with the second method in such a way as to define a usability factor that will be

used to evaluate the quality of the selected software product. This proposed methodology can be applied to a

finished software product to evaluate its usability according to users’ judgment, and can be also applied

during the development of the product to ensure it will have the desired usability attributes. In a future

work, we will show an application of this methodology in the evaluation of the usability of software

products in Brazil.

1 INTRODUCTION

In the last years the approach of software

development has been concentrated in the structural

and functional features of the product. The man-

machine interface is mainly centered in the

performance of the product during its useful life.

Quite often, in the development of a software

product, the technological level presented by the

product is given higher importance than the facility

of its use. The immediate response to this situation is

the complete dissatisfaction by the users with the

software. In some cases, the software is simply

abandoned by the customers. The man-machine

interface, in the past neglected due to a higher

emphasis in software development techniques,

achieved major importance after the realization of

the necessity to focus the information system in the

user.

The software industry has recently shown an

increased concern with the development of software

methodologies that allow at the same time effective

quality control of the product and the customers’

satisfaction in using the product. Due to this fact,

one of the most important characteristics associated

with the quality presented by the software is related

to the interaction between the user and the computer,

the usability.

The usability can be comprehended as a major idea

in the conception of the project, placing the user’s

needs as a main aspect of the project (Caldeira,

2000). The communication between the users and

Vasconçelos Silva S. and I. De Souza D. (2004).

EVALUATION OF THE QUALITY OF THE INTERACTION BETWEEN USERS AND CUSTOM-MADE SOFTWARE PRODUCTS.

In Proceedings of the First International Conference on E-Business and Telecommunication Networks, pages 96-102

DOI: 10.5220/0001381000960102

 SciTePress

the system, the usability, is today regarded as

important as the whole computation performed by

the system. Evaluation techniques as well as

usability tests should become important tools in

helping the software development area to achieve a

high degree of satisfaction among its users; thereby,

coordinating the characteristics of the software with

the users’ requirement.

In this work, we will develop a methodology based

on the quality of interaction between the users and

the software products, developed under customer

specifications, aiming at the determination of the

degree of users’ satisfaction, that is, the usability.

2 SOFTWARE QUALITY

The continuous improvement of the quality of

products and services is at the moment the main

driving focus in all the areas related to the human

environment. It is highly desirable to receive and to

offer products and services with high quality levels.

In today’s very competitive world, quality is the true

differential, most of the time responsible for the long

term success of a product or service. The

development of high quality software is paramount

for the majority of companies. The main issue to be

achieved is to satisfy customers’ requirements,

which in some cases is not necessarily the same as

the defined specifications; for that reason, in order to

achieve a system with a high quality level, the first

step to take is to assure that the specifications be

defined according to the customer’s needs (Da Silva,

2001).

During the 80’s, the main objective of the

software industry was to focus on the productivity

increase of its products; the software quality was

solely concerned with the observance of the

specifications related to the product and the delivery

of products on time with low costs; starting in the

90’s, the emphasis related to the quality of the

software was directed to characteristics such as

reliability, efficiency, interaction with the users, the

lack of defects, usability, etc. With the development

of information technology, which is significantly

helping in fulfill customer’s requirements and

allowing the development of a marketing-

competitive information system, the importance of

software products with high quality is essential to

achievement of the customers’ goals (Da Silva,

2001).

According to (Pressman, 1995), software quality

means conformance of specified functional and

performance characteristics with documented

development standards and with implicit

characteristics presented in all high level software.

The evaluation of the quality of software can be

performed on two occasions: during the

development of the software, the process phase, and

after the finishing of the software, the product phase.

At the process phase, the objective is to evaluate the

development of the software, identifying features

that could lead to problems related to the quality of

the product and developing and utilizing

mechanisms that could prevent such problems from

occurring. At the product phase, the purpose is to

evaluate the quality of the product with the intention

of identifying its deficiencies and limitations related

to its applicability as a final product.

According to (Fernandes , 2001), quality control

is an important requirement, highly useful in the

evaluation of the quality of software. This software

quality evaluation is very important for the:

• Software producer: during the process phase, to

ensure the high quality of the final product and to

correct any possible problem before the product is

released; during the product phase, to employ

possible corrective and developing actions.

• Purchaser: to help in the selection of a product

suitable to his requirements.

• User: to increase his confidence in the product

he is utilizing.

• Seller: uses the quality of the product as a

reason to sell.

2.1 Standard ISO/IEC 9126 and

ISO/IEC 9126-1

The Standard ISO/IEC 9126 defines a group of

characteristics that allow evaluating the quality of a

software product. These characteristics were chosen

in such a way as to insure, as much as possible, the

lack of correlation among them. This standard was

published in 1991 and in its section 1 is proposed a

quality model for software.

The standard ISO/IEC 9126-1 allows the

evaluation of the quality of a developed software

product through the evaluation of a group of

characteristics and sub-characteristics. This standard

presents a group of six characteristics that should be

present in any software product with high quality: It

has to be functional, reliable, usable, efficient,

moveable, and of easy maintenance.

2.2 Interaction Man – Computer

According to (Conçalves, 2001), “the interaction

man-computer is the determining factor related to

the strategy and accomplishment of a user doing his

work. The use of products or information systems

with poor quality related to the usability can be

EVALUATION OF THE QUALITY OF THE INTERACTION BETWEEN USERS AND CUSTOM-MADE SOFTWARE

PRODUCTS

associated with reasons for low productivity levels

and with financial failures linked to investments in

the information area”.

Users of computer programs evaluate the quality

of a software product through the existing

interaction between man and computer. It is not

sufficient for a program to only display several

advance functions, it is fundamental that the user be

able to utilize them. To achieve that, a high degree

of usability is mandatory.

It is important to realize that at the moment only

a small number (around 30%) of Brazilian software

corporations perform marketing research focusing

on customer satisfaction (around 30%). These

numbers are presented in Table 1.

Table 1: Frequency of marketing research focusing on

customer satisfaction performed by Brazilian companies.

Source: Quality and Productivity of Brazilian Software -

2001. (MCT/SEPIN, 2002)

Companies

Categories

Nº %

All the time 130 29.5

Occasionally 173 39.2

Thinking about it 51 11.6

Utilize published data 11 2.5

Never 76 17.2

Total 441 100

3 USABILITY

The term “user friendly” was very popular in the

60’s and 70’s when computer manufacturers first

started viewing users as more than a nuisance. This

term was lately replaced by a most appropriate one,

“usability”. Efforts to come up with a clear and

concise definition of usability have shown elusive

results. The following definitions of usability show a

noticeable similarity:

According to the International Organization for

Standardization ISO – 9241-11, “Usability” is the

extent to which a product can be used by specified

users to achieve specified goals in a specified

context of use with effectiveness, efficiency, and

satisfaction. According to (Mayhew, 1999), usability

is a measurable characteristic of a product-user

interface that is present to a greater or lesser degree.

According with (Nielsen, 1998), usability is the

measure of the quality of the user experience when

interacting with something – whether a Web site, a

traditional software application, or many other

devices the user can operate in some way or another.

There are several attributes which are associated

with usability. The five of them traditionally

associated with usability are:

Learnability: Probably the most fundamental

usability attribute, indicating that the system must be

easy to learn;

Efficiency of Use: Once it is learned by the user,

a high level of productivity is possible to be

achieved by the user.

Errors: The system should have a low user error

rate. Users should make as few errors as possible

and be able to easily recover from them.

Catastrophic errors must not occur.

Memorability: The system should be easy to

remember. Occasional users should be able to return

to the system after a time and not have to learn it all

over again.

Satisfaction: This attribute refers to how pleasant

it is to use the system. The users should be satisfied

when using the system.

As we can see in all these definitions of usability,

the focus is on the user, not on the product.

Usability evaluation can be performed on any

occasion in the development of a system: at the

initial phase, it is useful in identifying parameters to

be improved in the system; at the intermediate

phase, it is useful to validate or improve the project;

and at the final phase, it assures that the needs and

goals of users are fulfilled by the system.

4 METHODOLOGY

The methodology proposed in this paper is related to

the evaluation of the quality presented by custom-

made software, based on the customers’ satisfaction

with the final product. This approach is divided into

four parts:

a) Definition of the criterions to be used in the

evaluation of the quality of software products. A

total of 16 criterions will be employed by the user to

evaluate the software product. These standards are

the following: Online Help; Navigation, Easy to be

Installed; Operation Errors Prevention; Auditability;

Reuse of Input Data; Standardization; Messages;

Documentation; Self-instruction; Glossary;

Accuracy; Processing Time; Security; Data

Recovery; Resistance to Errors.

ICETE 2004 - GLOBAL COMMUNICATION INFORMATION SYSTEMS AND SERVICES

b) Method used to classify the usability criterions

by their level of importance. The purpose of this first

method is to establish an approach to classify the

usability criterions according to their level of

importance related to a selected area of application.

The level of importance of a usability criterion will

vary depending on the area of application.

c) The second method has as an objective the

development of an approach to classify usability

criterions by the quality level presented by a certain

software product utilized in a selected area. This

classification is based on the users’ opinion about

the quality level a certain usability criterion related

to the above software product would have.

d) The purpose of this third and final method is

to establish an approach to evaluate the usability of

software products according to users’ judgment. It

combines the first method with the second method in

such a way as to define a usability factor that will be

used to evaluate the quality of the selected software

product.

4.1 Definition of the Usability

Criterions

1. Online Help

– Online information has the potential for getting

users the precise information they need faster than

manuals or any other feature.

2. Navigation

– Is the facility to move from one screen to

another screen.

3. Easy to be Installed

– Is the capacity of the software in interacting

with the user during installation in a simple and easy

way.

4. Operation Errors Prevention

– Is the capacity presented by the software in

continuing working through errors made by the

users. In addition to having a good error messages,

the system should also provide good error recovery.

5. Auditability

– Is the capacity shown by the software in

following the logic associated with the design of a

system during its development and the reasons

underlying any possible modification in the original

design.

6. Reuse of Input Data

– Is the ability shown by the software in

requiring the inputting of data only once during the

software operation by the user.

7 Standardization

– This specifies how the exchange of information

from the software to the users should appear.

According to Nielsen, J. (1993a), one of the major

objectives of standards is to have interface

consistency.

8. Messages

– The messages should be simple and as

objective as possible and easy to understand. Error

messages should be linked to online help with

further explanation of the error and possible

solutions.

9. Documentation

– The ideal scenario would be to have a system

so easy to use that no further documentation is

needed to complement the user needs in operating

the software. Since this goal cannot always be met, a

model of documentation should be provided. This

model should allow the user to easily locate the

information related to a specific need, to understand

the information and to carry out, without further

trouble, a procedure as described in the

documentation.

10. Self-instruction

– The system should allow for the user to easily

learn with demonstrative lessons.

11. Glossary

– Provision by the software of a glossary with an

explanation of each of the important terms relating

to a user’s application and translation of those terms

into whatever language the user specifies.

12. Accuracy

– Is the precision presented by software products

in developing their tasks.

13. Processing Time

–Is the time a software product takes to complete

a specified task. It is also the number of tasks of

various kinds that can be completed by a software

program within a give time limit.

14. Security

– Is the capacity presented by the software in

avoiding non-authorized users access to the system

and in displaying error messages in cases where the

use of certain options is restricted.

15. Data Recovery

– Is the capacity presented by the software in not

loosing all or part of the existing data in the

occurrence of errors.

16. Resistance to Non-catastrophic Errors

– The system should provide enough information

to allow non-catastrophic-errors to be corrected

easily by the user and have no effect other than to

slow down to some extent the user’s transaction rate.

EVALUATION OF THE QUALITY OF THE INTERACTION BETWEEN USERS AND CUSTOM-MADE SOFTWARE

PRODUCTS

4.2 An Approach for Classifying

Usability Criterions by their Priority

Level

Generally, not all usability criterions can be given

equal weight in a given design project. So, it is

necessary to formulate the priorities on the basis of

the analysis of the users and their everyday jobs. For

example, messages would be especially important

for engineering’s software, and standardization

would be of great importance for software to be used

in library work. This approach has two objectives:

• To classify usability criterions according to

their priorities in relation to selected areas;

• To show that the priority of the criterions vary

according to the selected area.

In order to verify the existence of priority

variation shown by usability criterions in relation to

different areas of application, a questionnaire or

survey form is used. This questionnaire is based on

the users’ opinion about the priority a certain

criterion will have in their areas of interest.

The employed questionnaire contains all the 16

usability criterions defined in Section 4.1, evaluated

by selected users according to the following options:

(1) High Importance (HI); (2) Average Importance

(AI), (3) Low Importance (LI); (4) Without

Importance (WI). Weights were initially associated

with each level of importance, as presented in Table

2 below:

Table 2: Weights Associated with Each Level of

Importance

Option Number Weigh

option i

(Wh

)

% of the

answers

obtained

option i

(1) High Importance (HI) 6

(2) Average Importance (AI) 4

(3) Low Importance (LI) 2

(4) Without Importance (WI) 0

Total  100%

After the evaluation of the 16 criterions by the

users, the level of importance that each one of these

16 criterions has, according to each one of the

selected areas, is calculated. Equation (1) shows the

calculations for each one of the 16 criterions:

k,j

∑

Wh ×

k,j,i

(1)

j,k

– Level of Importance of criterion number j

in relation to selected area k.

– Weight of option number i.

i,j,k

– Percentage of the answers obtained by

option number i in the evaluation of criterion

number j in relation to selected area k.

For each one of the selected areas, the final level

of importance of criterion j will be classified

according to the interval levels (weight) presented in

Table 3.

Table 3: Interval Levels (weight) used in the classification

of each one of the 16 usability criterions by their priority

levels

Classification Interval Levels

(weight)

High Importance (HI) 6.0  4.5

Average Importance (AI) 4.5  3.0

Low Importance (LI) 3.0  1.5

Without Importance (WI) 1.5   0.0

For example, suppose that criterion number 2,

navigation, evaluated in relation to the civil

engineering area, presented a final weight of 4.2;

then, according to Table 3, it will be classified as

having average importance to the civil engineering

area.

4.3 An Approach for Classifying

Usability Criterions by the

Quality Level Presented by a

Chosen Software Product

This approach has the objective of classifying

usability criterions by the quality level presented by

a certain software product utilized in a selected area.

This classification is based on the users’ opinion

about the quality level a certain usability criterion

related to the above software product would have.

The employed questionnaire for this evaluation

also contains all the 16 usability criterions defined in

Section 4.1, evaluated by selected users according to

the following evaluation options: (1) Very Good

(VG); (2) Good (G); (3) Average (AV); (4) Poor (P);

(5) Very Poor (VP); and (6) Non-existent (NE).

Weights were associated with each one of the

above listed evaluation options, as presented in

Table 4 below.

ICETE 2004 - GLOBAL COMMUNICATION INFORMATION SYSTEMS AND SERVICES

100

Table 4: Weights Associated with Each Evaluation

Options

Option Number Weight of

Option i

(Wh

)

% of the

answers

obtained

by option i

(1) Very Good (VG) 10

(2) Good (G) 8

(3) Average (AV) 6

(4) Poor (P) 4

(5) Very Poor (VP) 2

(6) Non-existent (NE) 0

Total



100%

After the evaluation of the 16 criterions by the

users, the quality level that each one of these 16

criterions has, according to the analyzed software

product related to a selected area is calculated.

Equation (2) shows the calculations for each one of

the 16 criterions.

k,j

∑

Wh ×

j,i

(2)

j,k

– Level of Importance of criterion number j

based on the quality level presented by the software

product under analysis. The software product is

related to a selected area k.

– Weight of option number i.

i,j

– Percentage of the answers obtained by

option number i in the evaluation of criterion

number j.

For the software product being analyzed from

selected area k, the final level of importance of

criterion j will be classified according to the interval

levels (weight) presented in Table 5.

Table 5: Interval Levels (weight) used in the classification

of each one of the 16 usability criterions based on the

quality level presented by the software product under

analysis

Classification Interval Levels

(weight)

Very Good (VG) 10.0  8.0

Good (G) 8.0  6.0

Average (AV) 6.0  4.0

Poor (P) 4.0  2.0

Very Poor (VP) 2.0  0.0

Non-existent (NE) 0

4.4 An Approach for Classifying the

Quality of Software Products by

the Users’ Satisfaction

This approach has the objective of classifying the

usability level of a software product according to

users’ satisfaction. It combines both approaches

presented in sections 4.2 and 4.3 in the

determination of a usability level, which will be used

to classify a software product by its interaction with

its users.

Equation (3) shows the calculations utilized in

determining the usability level of software products.

U =

()

k,jk,j

SLIQLLI

∑

(3)

Here,

SLI =

∑

k,j

U – Usability level of the analyzed product.

j,k

– Level of Importance of criterion number j

in relation to selected area k.

j,k

– Level of Importance of criterion number j

based on the quality level presented by the software

product under analysis. The software product is

related to a selected area k.

SLI

= Sum of the Levels of Importance of all 16

criterions in relation to selected area k.

The analyzed software product from a selected

area k will be finally classified according to its

usability value obtained from equation (3). Table 6

gives the final classification obtained by the

analyzed software product based on its usability

level.

Table 6 – Final Product Classification based on its

Usability Levels

Usability Levels Products Classification

0.0 ≤ U ≤ 2.5 Poor Usability

2.5 < U ≤ 5.0 Average Usability

5.0 < U ≤ 7.5 Good Usability

7.5 < U ≤ 10.0 Excellent Usability

EVALUATION OF THE QUALITY OF THE INTERACTION BETWEEN USERS AND CUSTOM-MADE SOFTWARE

PRODUCTS

101

5 CONCLUSION

The methodology proposed in this paper is related to

the evaluation of the quality presented by custom-

made software, based on the customers’ satisfaction

with the final product. This method was divided into

four parts:

First we defined the 16 criterions to be used in

the evaluation of the quality of software products.

After these criterions had been chosen, an approach

was used to classify them by their levels of

importance related to a selected area of application.

The level of importance of a usability criterion will

vary depending on the area of application. As a next

step, an approach was developed to classify usability

criterions by the quality level presented by a certain

software product utilized in a selected area. This

classification is based on the users’ opinion about

the quality level a certain usability criterion related

to the above software product would have. Finally

an approach was established to evaluate the usability

of software products according to users’ judgment. It

combines the first method with the second method in

such a way as to define a usability factor that will be

used to evaluate the quality of the selected software

product. This proposed methodology can be applied

to a finished software product to evaluate its

usability according to users’ judgment, and can be

also applied during the development of the product

to ensure it will have the desired usability attributes.

In a future work, we will show an application of

this methodology in the evaluation of the usability of

software products in Brazil.

ACKNOWLEDGEMENT

The authors wish to acknowledge CNPq (Brazilian

National Scientific and Technological Development

Council) for their continued support and

encouragement.

REFERENCES

Associação Brasileira de Normas Técnicas, 1996.

Tecnologia de Informação – Avaliação do Produto de

Software – Características de Qualidade e Diretrizes

para seu uso: NBR 13596. Rio de Janeiro, RJ,Brazil.

Caldeira, P., 2000. A Tecnologia a Serviço do Homem?

Universidade de Algarve, Portugal.

Da Silva, M., 2001. O Comprometimento com a

Qualidade dos Sistemas de Informação: Um Enfoque

nas Competências das Pessoas. Dissertação (Mestrado

em Engenharia de Produção) – Universidade Federal

de Santa Catarina –UFSC, 110p. Santa Catarina-RS.

Brazil http://teses.eps.ufsc.br/ em 23/03/02.

Fernandes, C., 2001. Aplicação de Normas Internacionais

para Avaliação de Qualidade de Produto de Software.

– Centro Universitário Franciscano – UNIFRA, 95p.

Santa Maria-RS, Brazil; http://cpd-

srv03.unifra.br/~proj-walkiria/claudio em 18/05/002.

Gonçalves, E., 2001. A Interação com o Usuário na

Validação do Software Oficina de Relatório.

Dissertação (Mestrado em Engenharia de Produção) –

Universidade Federal de Santa Catarina –UFSC, 104p.

Santa Catarina-RS, Brazil; http://teses.eps.ufsc.br/ em

23/03/02.

MCT/SEPIN, 2002 Qualidade e Produtividade no Setor

de Software Brasileiro – 2001. 258p. Brasília, Brazil.

Mayhew, Deborah J. (1999). The Usability Engineering

Lifecycle. Morgan Kaufmann Publishers, Inc., San

Francisco, California, USA.

MCT/SEPIN, 2002. Qualidade e Produtividade no Setor

de Software Brasileiro. 258p. Brasilia, DF, Brazil

Nielsen, J., What is Usability?. 29 September, 1998

<http://www.zdnet.com> USA.

Nielsen, J., 1993a Usability Engineering. Academic Press,

Inc. New York, USA.

Pressman, R., 1995. Engenharia de Software. 3. ed.

Makron Books, 1057p. São Paulo, Brazil.

ICETE 2004 - GLOBAL COMMUNICATION INFORMATION SYSTEMS AND SERVICES

102