Towards Understanding Industry’s Perspectives on the Software Quality

Characteristics: A Survey

Mert Ozkaya and Nurdan Canbaz

Department of Computer Engineering, Yeditepe University, Istanbul, Turkey

Keywords:

Software Quality, Survey, ISO SQuaRE Quality Standard, Software Modeling Languages, Analysis Tools.

Abstract:

The ISO SQuaRE software quality standard categorises software quality into eight different characteristics:

functional suitability, performance efﬁciency, compatibility, usability, reliability, security, maintainability and

portability. Each quality characteristic is further considered in terms of a cohesive set of sub-characteristics.

In this paper, a survey has been conducted with the goal of understanding which software quality character-

istics are popular in industries and any modeling languages and tools used for analysing the software quality.

The survey has been performed with 16 experienced practitioners, who work for a Turkish software com-

pany and have been requested to answer a pre-determined set of questions. The survey results lead to many

interesting outcomes, some of which are as follows: (i) maintainability and performance efﬁciency are the

top-popular characteristics, while portability is the least, (ii) time behaviour is the top-considered performance

sub-characteristic, (iii) interoperability is the top-considered compatibility sub-characteristic, (iv) learnability

and appropriateness recognisability are the top-considered usability sub-characteristics, (v) availability is the

top-considered reliability sub-characteristic, (vi) conﬁdentiality and integrity are the top-considered security

sub-characteristics, (vii) modularity and reusability are the top-considered maintainability sub-characteristics,

(viii) none of the participants use any modeling languages for analysing the software quality early on, and (ix)

the participants use COTS tools for analysing the software implementation.

1 INTRODUCTION

Today, software systems have penetrated in almost all

areas of human life and been actively used in various

industries such as automative (Haghighatkhah et al.,

2017), avionics (Marques and Cunha, 2013), military

and defense (C¸ iﬂikli et al., 2012), telecommunica-

tions (Nitze and Schmietendorf, 2015), ﬁnance (Saji

et al., 2017), healthcare (Richardson et al., 2016), etc.

Given also the ever-increasing advancements of tech-

nology and the customer demands, the expectations

from software systems have been increasing rapidly.

This essentially raised the issue of software quality,

which is concerned with how well the software sys-

tem meets its requirements. While many attempts

have been made on explaining how the software qual-

ity should be understood (Kan, 2014; Kitchenham

and Pﬂeeger, 1996; Tian, 2005; Cavano and McCall,

1978), one can basically consider it as the software

system’s compliance with the user requirements that

can either be functional or non-functional. In the

mid eighties, Garvin (Garvin, 1984) proposed a semi-

nal approach that categorises the software quality into

different views, which are transcendental, user, manu-

facturing, product, and value-based. The transcenden-

tal view considers the software quality in terms of its

elegance and promotes the idea that one cannot mea-

sure the software quality but simply try their best to

maximise the level of elegance. The user view is con-

cerned with the user expectations and ensuring that

the software system meets all the user requirements.

The manufacturing view is concerned with the pro-

cess followed in developing and deploying the soft-

ware system and ensuring that the software system

is developed right that conforms to the requirements

and speciﬁcations of the system and thus any poten-

tial software faults are minimised. The product view

is concerned with the internal and external qualities

of a software system and how internal quality impacts

on the external quality. The value-based view is con-

cerned with the considerations of the software quality

from multiple views discussed above and their consis-

tencies with each other.

As indicated in (Ozkaya, 2018), many software

modeling languages have been proposed for the mod-

eling of software systems and their analysis for some

quality properties. So, the quality issues may be de-

tected early in the design phase before implement-

Ozkaya, M. and Canbaz, N.

Towards Understanding Industry’s Perspectives on the Software Quality Characteristics: A Survey.

DOI: 10.5220/0007742004170426

In Proceedings of the 14th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2019), pages 417-426

ISBN: 978-989-758-375-9

417

ing the systems. While some languages support the

exhaustive model checking or theorem proving tech-

niques for proving the correctness software systems,

some languages are supported with the analysis tools

that analyse the software models for an quality prop-

erties of interest. AADL (Feiler et al., 2006) is, for

instance, one of the most well-known software mod-

eling languages for the embedded systems domain.

AADL is supported by various analysis tools that al-

low for the analysis of the embedded software models

for many properties, including latency, schedulability,

timing, and resource utilisation. AADL’s tools also

generate C code from the embedded software mod-

els. Java Modeling Language (Chalin et al., 2005) is

another highly popular software modeling language,

which is tailored to Java and allows for combining

the software modeling with implementation. Indeed,

Java methods can be annotated with contract speciﬁ-

cations and, using the exhaustive veriﬁer tools avail-

able, the Java programs can be proved for the con-

tractual speciﬁcations. Another well-known tool is

UPPAAL (Larsen et al., 1997), which supports the

modeling, simulation and analysis of the real-time

systems. Using UPPAAL, one can specify their real-

time systems model visually using timed automata,

simulate its behaviour, and perform exhaustive model

checking for any safety and liveness properties.

While software quality is one of the most crucial

topics in software engineering and many languages

(or tools) have been existing for promoting the qual-

ity software systems, it is not yet clearly known how

practitioners in industries approach towards the soft-

ware quality in their software development projects.

Indeed, it is ambiguous as to which quality charac-

teristics are more crucial for practitioners and how

practitioners deal with those quality characteristics in

their software development. So, in this study, a survey

has been conducted on a group of practitioners who

are involved in the software development for various

industries. The goal of this survey is to understand

which software quality characteristics are popular in

industries and any modeling languages and tools used

for dealing with the software quality.

2 RESEARCH METHODOLOGY

In our survey, we focus on the software quality char-

acteristics determined by the ISO/IEC 25010:2011

SQuaRE software quality standard (ISO Central Sec-

retary, 2011). As shown in Figure 1, ISO SQuaRE

categorises the software quality into 8 different char-

acteristics, which are functional suitability, perfor-

mance efﬁciency, compatibility, usability, reliability,

security, maintainability, and portability. Each soft-

ware quality characteristic is also considered in terms

of a cohesive set of quality sub-characteristics.

Our survey consists of 38 different questions,

which are divided into four sections (i.e., one for each

research question). Table 1 shows the list of ques-

tions. The questions 7-10 repeat for each software

quality characteristic given in Figure 1. Also, some

questions offer multiple-choice answers that allow the

participants for choosing one or more answers. The

multiple-choice questions with the free-text option es-

sentially allow the participants to write their own an-

swers that are not in the answer list. We evaluate

the free-text answers and eliminate any of them that

does not make sense for the question. Some ques-

tions require yes/no answers that are represented with

ﬁve alternative options to maximise precision: always

(100%), much of the time (>=75%), often (>=50%),

sometimes (<50%), and never (0%).

The survey has been targeted for 16 different prac-

titioners who have considerable experiences on soft-

ware development and work for the Logo company

which is one of the largest software development

companies in Turkey and has been offering various

software solutions and services to different industries

in Turkey for more than 30 years. To enhance the pre-

cision of the data gathered from the participants, each

participant has been invited to a 30-60 minutes long

session, in which one of us has been ready to interact

with the participant. So, the participant has been re-

quested to ﬁll in the survey during the session and ask

us any question that he/she may have about the survey

questions. Also, the participants have been supplied

with the ISO SQuaRE speciﬁcation document so as to

ﬁnd out the precise deﬁnitions of the quality charac-

teristics and their sub-characteristics if they need so.

Before executing the survey, we performed a plot

study to get some initial feedback about the survey

structure and questions. To this end, four different

practitioners who have got 10+ years of experiences

on the software development in IT industry shared

their thoughts. So, this let us determine the missing

or ambiguous answers, ambiguous questions, the du-

ration of the survey sessions, and any supplementary

materials for the participants taking the sessions.

3 RESEARCH QUESTIONS

To meet the goal stated in Section 1, a set of research

questions has been determined. To understand how

each research question is targeted, the survey ques-

Logo web-site: https://www.logo.com.tr/en/

ENASE 2019 - 14th International Conference on Evaluation of Novel Approaches to Software Engineering

418

Figure 1: The software quality categorisation of the ISO SQuaRE software quality standard (ISO Central Secretary, 2011).

Table 1: The survey questions.

Res.

Ques.

Survey

Questions

Multiple

Answers

Free Text

Yes/No

Question

Integer Answer

RQ1

1- What is (are) your current job position(s)? Yes Yes No No

2- What is(are) the type(s) of the software project(s)? Yes Yes No No

3- How many years of experience do you have in software development? No No No Yes

4- Which industries do your customers work in ? Yes Yes No No

RQ2

5- Which of the following software quality characteristics do you

consider in software development projects ?

Yes No No No

6- If you do not consider any of those quality characteristics, please tell

us the reason(s).

No Yes No No

The questions 7-10 repeat for each software quality characteristic considered.

RQ3

7- Can you rate for each software quality property how frequently you consider

that property in your software projects?

No No Yes No

RQ4

8- If you use model-driven approaches for modeling, analysing, and implementing the software

systems for the quality properties of the current quality characteristics, please tell us about them.

No Yes No No

9- If you use any software tools for analysing the software systems for the quality

properties of the current quality characteristic, please tell us more about them.

No Yes No No

10- If you manually analyse the quality properties of the current quality characteristic,

please tell us more about how you perform the manual analysis

No Yes No No

tions that are used for answering the research ques-

tions are shown in Table 1.

RQ1 - What Are the Proﬁles of the Participants?

This research question aims to understand the partici-

pants’ proﬁles including their job positions, the types

of software projects involved, years of experiences,

and work industries.

RQ2 - Which Software Quality Characteristics Do

Practitioners Consider in Their Software Develop-

ment? This research question aims to understand

which of the software quality characteristics proposed

by ISO SQuaRE are considered by practitioners in

their software development projects.

RQ3 - Which Sub-characteristics of the Software

Quality Characteristics Do Practitioners Consider

in Their Software Development? The aim herein is

to understand for each software quality characteristic

of ISO SQuaRE which of its sub-characteristics are

considered by practitioners for the quality analysis of

their software systems.

RQ4 - Which Languages and Tools Do Practition-

ers Use for Analysing Their Software Quality? The

aim here is to learn the software modeling languages

that practitioners may be using for modeling their

software systems and checking their software models

for the quality properties of interest via the language

toolset. Also, any commercial-off-the-shelf (COTS)

tools that practitioners may be using for any quality

characteristics are aimed to be learned.

4 THE SURVEY RESULTS

4.1 Proﬁle Questions

Figure 2 shows the job positions of the the partici-

pants. So apparently, the design team lead and analyst

Figure 2: The participants’ job positions.

Towards Understanding Industry’s Perspectives on the Software Quality Characteristics: A Survey

419

positions have been the top selected ones, which are

followed by the software developer and product man-

ager positions. The rest of the job positions shown in

Figure 2 are not that popular among the participants.

Figure 3: The types of the software projects that the partic-

ipants are involved in.

Figure 3 shows the different types of software

projects that the participants are involved in. So, most

participants develop business applications software,

which is followed by those developing web applica-

tions. A few participants stated that they develop

mobile applications. The other types of software

projects, such as safety-critical and mission-critical

software systems and systems software are not in the

scope of the participants.

Figure 4: The participants’ years of software development

experiences.

Figure 4 shows the participants’ years of experi-

ences on the software development. So apparently,

almost half of the participants have got 10+ years of

experiences, 31% of them have 6-10 years of experi-

ences, and the rest have 2-5 years of experiences.

Nearly all the participants develop software sys-

tems for various industries, which include automa-

tive and transportation, consumer electronics, de-

fense/military & aviation, ﬁnance and accounting,

government, healthcare and biomedical, IT and

telecommunications, and software outsourcing. One

of the participants stated that they work for the

government only and another one work for the de-

fense/military & aviation only.

Figure 5: Participants’ frequency of consideration for each

software quality characteristic.

4.2 Software Quality Characteristics

Figure 5 shows the participants’ consideration for the

ISO SQuaRE quality characteristics in their software

projects. So, maintainability is the top-selected char-

acteristic, followed by the performance efﬁciency,

reliability, and functional suitability characteristics.

The compatibility and security characteristics are not

so popular among the participants, and portability has

not been considered by any of the participants at all.

Figure 6: Participants’ considerations for the functional

suitability sub-characteristics.

4.2.1 Functional Suitability

The functional suitability quality characteristic is con-

cerned with how well the provided functionalities

of a software system meet the user expectations.

The functional suitability is considered in terms of

the functional correctness and functional complete-

ness sub-characteristics. The functional correctness

is to do with executing the system functionalities and

checking that the obtained results are correct with re-

gard to the system speciﬁcations. The functional com-

pleteness is to do with checking that the system func-

tionalities cover all the required tasks.

ENASE 2019 - 14th International Conference on Evaluation of Novel Approaches to Software Engineering

420

Figure 6 shows the participants’ frequency of con-

sideration for the functional correctness and com-

pleteness. So, most of the participants always (100%)

consider the functional correctness and completeness

in their software projects.

As the survey results reveal, the participants use

the Behaviour-Driven Development (BDD) method-

ology (Solis and Wang, 2011) for developing and

testing software systems for the functional suitability.

With BDD, different stakeholders (e.g., customers,

users, and developers) interact to have the same un-

derstanding of the system functionalities and how

each functionality is expected to behave. Then, the

test scenarios are deﬁned together with the users and

customers in a natural language. The informal test

scenarios can be translated into the executable test

cases and the systems can be run for those test cases.

To apply BDD in their software projects, the partici-

pants use the Cucumber framework

The survey results also indicate that the partic-

ipants have never used any software modeling lan-

guages for the modeling and analysis of the functional

suitability properties.

4.2.2 Performance Efﬁciency

The performance efﬁciency quality characteristic is

concerned with how much resources are consumed

by a software system. The performance efﬁciency is

considered in terms of the time behaviour, resource

utilisation, and capacity sub-characteristics. The time

behaviour is to do with ensuring that the time spent by

the software system (e.g., sending a response or per-

forming a particular operation) or the system through-

put time meet the system requirements. The resource

utilisation is to do with ensuring that the resources

which the software system consumes meet the system

requirements. The capacity is to do with ensuring that

the upper limits that system functionalities can reach

meet the system requirements.

Figure 7: Participants’ considerations for the performance

efﬁciency sub-characteristics.

https://cucumber.io/

Figure 7 shows the participants’ considerations of

the performance sub-characteristics in their software

projects. While all the time behaviour, resource con-

sumption, and capacity are frequently (i.e., always or

much of the time) considered by most of the partici-

pants, the time behaviour attracted the greatest inter-

est. Capacity is relatively less popular than the other

two sub-characteristics – just 25% of the participants

always (100%) consider the capacity analysis in their

software project.

The participants do not use any software modeling

languages for modeling and analysing the software

systems against the performance properties. While

some participants use their own software solutions

for analysing performance, some use the COTS tools.

These tools include dotMemory

for analysing the

memory usage, dotTrace

for analysing the time be-

haviour, and Apache JMeter

for analysing capacity.

4.2.3 Compatibility

The compatibility quality characteristic is concerned

with to what extent a software system can share re-

source(s) with another software system or work to-

gether with different systems without the need for

any interventions. The compatibility is considered

in terms of the co-existence and interoperability sub-

characteristics. The co-existence property is to do

with the ability of multiple software systems to share

the same resources or execution platforms success-

fully. The interoperability property is to do with the

ability of multiple software systems to interact with

each other and exchange data successfully.

Figure 8: Participants’ considerations for the compatibility

sub-characteristics.

Figure 8 shows the participants’ considerations

of the co-existence and interoperability sub-

characteristics in their software projects. So, while

interoperability is always (100%) considered by the

https://www.jetbrains.com/dotmemory/

https://www.jetbrains.com/proﬁler/

https://jmeter.apache.org/

Towards Understanding Industry’s Perspectives on the Software Quality Characteristics: A Survey

421

participants, co-existence is not that popular es-

sentially.

The participants do not use any software mod-

eling languages for the modeling and analysis of

their software systems and the compatibility sub-

characteristics. However, the participants use their

in-house software solutions for the analysis of the ex-

ecutable software systems for compatibility.

4.2.4 Usability

The usability quality characteristic is concerned with

to what extent the potential users of a software system

are capable of using the system in a way that meets

the system requirements. Thu usability is considered

in terms of the appropriateness recognisability, learn-

ability, operability, user error protection, user inter-

face aesthetics, and accessibility sub-characteristics.

The appropriateness recognisability is to do with the

users’ acceptance of the software system for their

needs. The learnability is to do with how easy it is

to learn and use the software system. The operability

is to do with how easy it is to operate the software

system while the system is in use. The user error pro-

tection is to do with to what extent the software sys-

tem prevents the users from making errors while using

the system. The user interface aesthetics is to do with

how attractive the user interfaces of the software sys-

tem are to the users. The accessibility is to do with

to what extent the software system can be used by the

users with different types of disabilities.

Figure 9: Participants’ considerations for the usability sub-

characteristics.

Figure 9 shows the participants’ considerations

of the different usability sub-characteristics in their

software projects. So, all the participants frequently

(i.e., much of the time or always) consider analysing

the appropriateness recognisability and learnability.

While operability and user error protection are also

frequently considered by many participants, a few

of the participants show less interest to them. Con-

cerning the user interface aesthetics and accessibil-

ity, some participants do not frequently consider those

two usability sub-characteristics in their software de-

velopment. Indeed, the number of the participants

who chose often or sometimes are the highest for the

user interface aesthetics and accessibility.

None of the participants use any software model-

ing languages for the modeling and analysis of soft-

ware systems and their usability characteristics. The

participants’ companies have user-experience (UX)

teams who are responsible for testing the usability

of software systems. Moreover, one of the partici-

pants stated that they use the A/B testing (Dixon et al.,

2011) to analyse their systems’ usability.

4.2.5 Reliability

The reliability quality characteristic is concerned with

how well a software system performs its functionali-

ties in its environment within the expected amount of

time. The reliability is considered in terms of the ma-

turity, availability, fault tolerance, recoverability sub-

characteristics. The maturity is to do with how long

the software system could be used in a way that satis-

ﬁes the reliability requirements. The availability is to

do with to what extent the software system is available

for use when requested by the users. The fault toler-

ance is to do with to what extent the software system

carries on operating in the case of any faults occur-

ing. The recoverability is to do with to what extent

the software system maintains its state without losing

data in the case of any crashes due to some failures.

Figure 10: Participants’ considerations for the reliability

sub-characteristics.

Figure 10 shows how frequently the participants

consider the reliability sub-characteristics in their

software projects. So, all the participants frequently

(i.e., either always or much of the time) analyse the

availability of their software systems. While the ma-

turity and fault tolerance are also frequently analysed

by many participants, some of the participants often

(>=50%) consider them. The recoverability property

is shown quite less interest by the participants.

Concerning the languages and tools used, the par-

ticipants do not use any languages or tools for the

modeling and analysis of software systems and their

reliability sub-characteristics.

ENASE 2019 - 14th International Conference on Evaluation of Novel Approaches to Software Engineering

422

4.2.6 Security

The security quality characteristic is concerned with

to what extent a software system protects its data from

malicious attacks so that the system data can only be

accessed by the speciﬁed systems and people at the

speciﬁed levels. The security quality characteristic is

considered in terms of the conﬁdentiality, integrity,

non-repudiation, accountability, and authenticity sub-

characteristics. The conﬁdentiality is to do with en-

suring that the system data is accessible by the autho-

rised users only. The integrity is to do with to what

extent the software system protects its data against the

modiﬁcations performed by unauthorised users. The

non-repudiation is to do with proving for the commu-

nication of any two parties that they cannot deny any

event/action occuring between each other such as the

message/data exchange. The accountability is to do

with to what extent the actions taken by the software

component or a person using the software system can

be monitored to avoid any unwanted access to the sys-

tem data or service. The authenticity is to do with

whether it is possible to prove the identity of the users

and any other entities or not.

Figure 11: Participants’ considerations for the security sub-

characteristics.

Figure 11 shows the participants’ considerations

of the security sub-characteristics in analysing the

software security. So, integrity is the most frequently

considered sub-characteristic, which is followed by

conﬁdentiality. While non-repudiation, accountabil-

ity, and authenticity properties are also frequently

considered, some participants consider them infre-

quently (i.e., often or sometimes). Authenticity is es-

sentially the least popular security sub-characteristic,

as some participants stated to sometimes (<50%)

analyse the authenticity of their software systems.

The participants do not use any software model-

ing languages for modeling and analysing the security

sub-characteristics. However, the participants use the

CheckMarx tool

for analysing their software system

implementation against some security issues.

https://www.checkmarx.com/

Figure 12: Participants’ considerations for the maintainabil-

ity sub-characteristics.

4.2.7 Maintainability

The maintainability quality characteristic is con-

cerned with to what extent a software system can be

modiﬁed, repaired, and improved effectively. The

maintainability is considered in terms of the modular-

ity, reusability, analysability, modiﬁability, and testa-

bility sub-characteristics. The modularity is to do

with how well the software system is decomposed

into modules that are independent from each other

and each represent the cohesive set of functionalities

of the system. The reusability is to do with to what ex-

tent the modules composing the software system can

be re-used within the same system or across differ-

ent systems. The analysability is to do with whether

the software system can be analysed at any levels of

abstractions for detecting any issues about the sys-

tem. The modiﬁability is to do with to what extent

the software system can be modiﬁed effectively with-

out causing new errors in the system that require an-

other modiﬁcation(s). The testability is to do with at

what level of coverage the test scenarios for the soft-

ware system can be determined and the effective use

of the test scenarios for testing the software system.

Figure 12 shows the participants’ considerations

of the maintainability sub-characteristics in their soft-

ware projects. Modularity and reusability are the top-

considered maintainability sub-characteristics by the

participants, which are followed by the analysability.

The modiﬁability and testability sub-characteristics

are relatively less popular among the participants.

The participants do not use any software model-

ing languages for the maintainable software develop-

ment. The participants use the SonarQube analysis

tool

for analysing the software implementation for

several maintainability issues.

4.2.8 Portability

The portability quality characteristic is concerned

with to what extent a software system can be used on

https://www.sonarqube.org/

Towards Understanding Industry’s Perspectives on the Software Quality Characteristics: A Survey

423

another operating system or hardware. The portabil-

ity is considered in terms of the adaptability, instal-

lability, and replaceability sub-characteristics. The

adaptability is to do with to what extent the soft-

ware system can be adapted to work in a different

hardware or software environment. The installabil-

ity is do with to what extent the software system can

be installed/uninstalled successfully in/from any de-

sired environments. The replaceability is to do with

to what extent the software system or its components

can be replaced with another system that can exhibit

the same functionality.

Portability is just considered by one of the partic-

ipants who stated to use the Jerkins tool for dealing

with the portability issues.

5 SUMMARY OF THE RESULTS

In this section, the key outcomes obtained from the

analysis of the survey results are summarised.

The Design Team Lead and Analyst Positions Are

Highly Popular. While the participants represent

many different job positions (e.g., software developer,

consultant, manager positions, and tester), the design

team lead and analyst are the top-selected job posi-

tions by the participants.

Business Applications Software Is the Top Popular

Software Project Type. Most of the participants are

involved in the business applications software devel-

opment (75%). Concerning the rest, 19% are involved

in the web applications development and 6% involved

in the mobile applications.

Most Practitioners Have at Least 6 Years of Expe-

riences on Software Development. While 44% of

the participants have 10+ years of experiences, 31%

have 6-10 years of experiences, and the rest have 2-5

years of experiences.

Maintainability and Performance Efﬁciency Are

the Top Popular Quality Characteristics. Among

the software quality characteristics proposed by ISO

SQuaRE, most of the participants (>70%) consider

the maintainability and performance efﬁciency char-

acteristics for the analysis of their software systems.

None of the participants (except one) are interested in

analysing the software portability.

The Functional Suitability Sub-characteristics Are

Equally Important. Almost all the participants

who are interested in functional suitability always

(100%) analyse the functional correctness and func-

tional completeness of their software systems.

The Time Behaviour Is the Top Considered Perfor-

mance Sub-characteristic. While most of the par-

ticipants always (100%) analyse their software sys-

tems for the time behaviours, the resource consump-

tion and capacity performance sub-characteristics did

not attract that level of interest.

The Interoperability Is the Top Considered Com-

patibility Sub-characteristic. While the participants

who are interested in compatibility always (100%)

analyse their software systems for interoperability,

those participants do not consider the co-existence

sub-characteristic of compatibility so frequently.

The Learnability and Appropriateness Recognis-

ability Are the Top Considered Usability Sub-

characteristics. The participants frequently (i.e., al-

ways (100%) or much of the time(>=75%)) anal-

yse the learnability and appropriateness recognisabil-

ity of their software systems. While operability and

user error protection are also quite frequently con-

sidered, the user interface aesthetics and accessibility

sub-characteristics of usability are not so popular.

The Availability Is the Top Considered Reliabil-

ity Sub-characteristic. The participants who are in-

terested in reliability frequently analyse their soft-

ware systems for availability, and that is followed by

the analysis of the maturity and fault tolerance sub-

characteristics. Recoverability is shown relatively

less interest by the participants.

Conﬁdentiality and Integrity Are the Top Con-

sidered Security Sub-characteristics. The partici-

pants who are interested in security frequently anal-

yse their software systems for conﬁdentiality and in-

tegrity. While non-repudiation and accountability are

also quite popular, authenticity is shown relatively

less interest by the participants.

Modularity and Reusability Are the Top Consid-

ered Maintainability Sub-characteristics. The par-

ticipants who are interested in maintainability always

consider the modularity and reusability of their soft-

ware systems. The analysability of software systems

is also quite popular. However, modiﬁability and

testability are shown less interest by the participants.

None of the Participants Use Any Modeling Lan-

guages for Analysing The Software Quality Early

On. While many software modeling languages have

been existing today for the modeling and analysis of

software systems for different quality characteristics,

the participants are not familiar with any of them.

The Participants Use Cots Tools for the Software

Analysis. The participants have been observed to use

several COTS tools available for analysing the soft-

ware system implementations for the quality charac-

teristics of interest. While some of these tools are free

and open-source, some are commercial tools.

ENASE 2019 - 14th International Conference on Evaluation of Novel Approaches to Software Engineering

424

6 RELATED WORK

While the literature includes many survey studies on

the software quality, it is really hard to ﬁnd any sur-

veys that focus on various types of quality character-

istics in general and give any results regarding the per-

spectives of the practitioners who are involved in the

software development.

In (P

erez et al., 2013), the authors targeted the

Belgian companies and surveyed 44 different partic-

ipants working in different companies to understand

the processes, tools, and techniques that they employ

for improving the software quality in general. How-

ever, P

erez et al. do not focus on any software qual-

ity characteristics and practitioners’ perspectives on

them, as is the case in our survey. In (Garousi and Zhi,

2013), the authors surveyed 245 practitioners who

work in Canada and aimed at understanding the tech-

niques, tools, and metrics used for testing software

systems. In (Bygstad et al., 2008), the authors sur-

veyed 78 different IT companies in Norway to under-

stand their perspectives towards the software usabil-

ity. In (Gulliksen et al., 2004), the authors surveyed

194 different practitioners who work in Sweden to

understand the techniques and tools that they use for

promoting the usable software systems and their per-

spectives towards maximising the software usability.

In (Geer, 2010), the author surveyed among 46 differ-

ent practitioners who hold either the consultant, man-

ager, or developer positions, and intended to under-

stand their perspectives towards the secure software

development life-cycles. In (Moores and Edwards,

1992), the author surveyed 54 different large IT com-

panies that are located in United Kingdom and aimed

to understand to what extent the practitioners in those

companies use the cost estimation tools for the pur-

pose of estimating the size and cost of their software

development projects. In (Sousa and Moreira, 1998),

the authors conducted a survey among the 37 different

organisations located in Portugal. Sousa et al. aimed

at understanding how the software maintenance pro-

cess is carried out and the difﬁculties encountered. In

(Xie et al., 2011), the authors interviewed 15 differ-

ent software developers to basically understand the

software security issues that they encountered in their

software projects, the methods, techniques, and tools

that they use for handling the software security issues,

and their opinions about how the software security re-

lates to the software development life-cycle.

7 CONCLUSION

In this paper, a survey has been conducted on under-

standing the practitioners’ perspectives on different

software quality characteristics and the modeling lan-

guages/tools used for analysing the software quality.

To this end, the survey focuses on the ISO SQuaRE

software quality standard, which categorises the soft-

ware quality as the functional suitability, performance

efﬁciency, compatibility, usability, reliability, secu-

rity, maintainability and portability characteristics.

Each software quality characteristic is deﬁned by ISO

in terms of a relevant set of sub-characteristics. The

survey consists of 38 questions that have been an-

swered by 16 participants who work for a Turkish

software company called Logo. Each participant has

been allocated a 30-60 minutes session during which

the participant has been requested to ﬁll in the sur-

vey and allowed to ask any questions that he/she has

about the survey questions. The analysis of the sur-

vey results lead to many interesting lessons. Practi-

tioners give the highest importance to (i) the main-

tainable development of software systems that can be

easy to change and improve after deployment and (ii)

the efﬁcient use of the resources. The portable soft-

ware development for facilitating the use of the soft-

ware system in different environments is not shown

the least interest by the practitioners. Practitioners

also showed varying levels of interests to the differ-

ent sub-characteristics of the software quality char-

acteristics proposed by ISO SQuaRE, and that led to

determine practitioners’ priorities regarding the anal-

ysis of software systems for each quality character-

istic. Moreover, practitioners do not use any soft-

ware modeling languages for the high-level model-

ing and analysis of software systems. Note that many

modeling languages have been actually existing (e.g.,

AADL, UPPAAL, and JML), which enable the ex-

haustive model checking of the software models and

prove the models for different quality properties of

interest before proceeding with the software imple-

mentation. Practitioners prefer to use the COTS tools

available on the market that allow for checking the

software implementation for some quality character-

istics. However, any quality issues detected may not

easily be ﬁxed at this stage, as the software systems

analysed have already been design and implemented.

Concerning the threats to the validity of the sur-

vey results, the participants have been selected from

a single company in Turkey. While this may cause

biases regarding the selection of the participants, the

Logo company essentially develops software systems

for various customers of different industries in Turkey

and that helps addressing the software quality require-

Towards Understanding Industry’s Perspectives on the Software Quality Characteristics: A Survey

425

ments of customers with different needs. Also, some

software quality characteristics addressed in the sur-

vey may not be understood precisely by all the par-

ticipants, and therefore, the survey has been decided

to be conducted via sessions in which the participants

are requested to ﬁll in the survey and allowed to ask

for any clariﬁcations. Lastly, the survey has been con-

ducted with 16 different participants only. While this

essentially helped us to get an initial idea about the re-

search questions targeted, we plan to conduct a larger

survey with similar research questions in the future.

REFERENCES

Bygstad, B., Ghinea, G., and Brevik, E. (2008). Soft-

ware development methods and usability: Perspec-

tives from a survey in the software industry in norway.

Interacting with Computers, 20(3):375–385.

Cavano, J. P. and McCall, J. A. (1978). A framework for

the measurement of software quality. SIGSOFT Softw.

Eng. Notes, 3(5):133–139.

Chalin, P., Kiniry, J. R., Leavens, G. T., and Poll, E. (2005).

Beyond assertions: Advanced speciﬁcation and veri-

ﬁcation with JML and ESC/Java2. In de Boer, F. S.,

Bonsangue, M. M., Graf, S., and de Roever, W. P., ed-

itors, FMCO, volume 4111 of Lecture Notes in Com-

puter Science, pages 342–363. Springer.

C¸ iﬂikli, B.,

Ozcan, O., and Uysal, A. B. (2012). Model

driven software development for military command

and control systems. In 2012 IEEE 36th Annual Com-

puter Software and Applications Conference, pages

494–497.

Dixon, E., Enos, E., and Brodmerkle, S. (2011). A/b testing

of a webpage. US Patent 7,975,000.

Feiler, P. H., Lewis, B. A., and Vestal, S. (2006). The

SAE architecture analysis & design language. In

IEEE Intl Symp. on Intell. Control, pages 1206–1211.

www.aadl.info.

Garousi, V. and Zhi, J. (2013). A survey of software testing

practices in canada. Journal of Systems and Software,

86(5):1354–1376.

Garvin, D. (1984). What does product quality really mean?

Sloan Management Review, 26:25–45.

Geer, D. (2010). Are companies actually using secure de-

velopment life cycles? Computer, 43(6):12–16.

Gulliksen, J., Boivie, I., Persson, J., Hektor, A., and Herulf,

L. (2004). Making a difference: a survey of the usabil-

ity profession in sweden. In Raisamo, R., editor, Pro-

ceedings of the Third Nordic Conference on Human-

Computer Interaction 2004, Tampere, Finland, Octo-

ber 23-27, 2004, pages 207–215. ACM.

Haghighatkhah, A., Oivo, M., Banijamali, A., and Kuvaja,

P. (2017). Improving the state of automotive software

engineering. IEEE Software, 34(5):82–86.

ISO Central Secretary (2011). Systems and software engi-

neering – systems and software quality requirements

and evaluation (square) – system and software quality

models. Standard ISO-IEC 25010:2011, International

Organization for Standardization, Geneva, CH.

Kan, S. H. (2014). Metrics and Models in Software Qual-

ity Engineering - Paperback. Addison-Wesley Profes-

sional, 2nd edition.

Kitchenham, B. A. and Pﬂeeger, S. L. (1996). Software

quality: The elusive target. IEEE Software, 13(1):12–

21.

Larsen, K. G., Pettersson, P., and Yi, W. (1997). UPPAAL

in a nutshell. STTT, 1(1–2):134–152.

Marques, J. and Cunha, A. (2013). A reference method for

airborne software requirements. In 2013 IEEE/AIAA

32nd Digital Avionics Systems Conference (DASC),

pages 1–29.

Moores, T. T. and Edwards, J. S. (1992). Could large uk

corporations and computing companies use software

cost estimating tools? – a survey. European Journal

of Information Systems, 1(5):311–320.

Nitze, A. and Schmietendorf, A. (2015). A survey on mo-

bile users’ software quality perceptions and expecta-

tions. In 2015 IEEE Eighth International Conference

on Software Testing, Veriﬁcation and Validation Work-

shops (ICSTW), pages 1–2.

Ozkaya, M. (2018). The analysis of architectural languages

for the needs of practitioners. Softw., Pract. Exper.,

48(5):985–1018.

erez, J., Mens, T., and Kamseu, F. (2013). A pilot study on

software quality practices in belgian industry. In 2013

17th European Conference on Software Maintenance

and Reengineering, pages 395–398.

Richardson, I., Reid, L., and O‘Leary, P. (2016). Health-

care systems quality: Development and use. In 2016

IEEE/ACM International Workshop on Software Engi-

neering in Healthcare Systems (SEHS), pages 50–53.

Saji

c, M., Bundalo, D., Bundalo, Z., and Pa

sali

c, D. (2017).

Digital technologies in transformation of classical re-

tail bank into digital bank. In 2017 25th Telecommu-

nication Forum (TELFOR), pages 1–4.

Solis, C. and Wang, X. (2011). A study of the characteristics

of behaviour driven development. In 2011 37th EU-

ROMICRO Conference on Software Engineering and

Advanced Applications, pages 383–387.

Sousa, M. J. C. and Moreira, H. M. (1998). A survey on the

software maintenance process. In Proceedings. Inter-

national Conference on Software Maintenance (Cat.

No. 98CB36272), pages 265–274.

Tian, J. (2005). Software Quality Engineering: Testing,

Quality Assurance, and Quantiﬁable Improvement.

Wiley-Interscience, New York, NY, USA.

Xie, J., Lipford, H. R., and Chu, B. (2011). Why do

programmers make security errors? In 2011 IEEE

Symposium on Visual Languages and Human-Centric

Computing (VL/HCC), pages 161–164.

ENASE 2019 - 14th International Conference on Evaluation of Novel Approaches to Software Engineering

426