The Inﬂuence of Software Product Quality Attributes

on Open Source Projects: A Characterization Study

Antonio Cesar Brandao Silva

, Kattiana Constantino

, Glauco de Figueiredo Carneiro

Antonio Carlos M. de Paula

, Eduardo Figueiredo

, Miguel P. Monteiro

and Fernando Brito e Abreu

Universidade Salvador (UNIFACS), Salvador, Bahia, Brasil

Universidade Federal de Minas Gerais (UFMG), Belo Horizonte, Minas Gerais, Brasil

Universidade Nova de Lisboa (UNL/NOVA LINKS), Lisboa, Portugal

Instituto Universitario de Lisboa (ISCTE-IUL), Lisboa, Portugal

Keywords:

Software Releases, Open Source Software (OSS) Projects, Software Product Quality Attributes.

Abstract:

Several Open Source Software (OSS) projects have adopted frequent releases as a strategy to deliver both new

features and ﬁx bugs on time. These entails express requests from the project’s community, registered as issues

in bug repositories by active users and developers. Each OSS project has its own priorities established by their

respective communities. A still open question is to what extent these priorities inﬂuence selection of the issues

that should be tackled ﬁrst, implemented/solved and delivered in subsequent releases. In this paper, we present

an exploratory study on the inﬂuence of target product quality attributes in software release practices of OSS

projects. The goal is to search for evidence that clarify the relationships between target attributes, priorities

assigned to the registered issues and the ways they are delivered by product releases. To this end, we asked

a set of participants to identify these attributes through the data analysis of repositories of three well-known

OSS projects: Libre Ofﬁce, Eclipse and Mozilla Firefox. Evidence provided by the participants suggest that

OSS community developers use criteria/priorities driven by speciﬁc software product quality attributes to plan

and perform software releases.

1 INTRODUCTION

The attractiveness of Open Source Software (OSS)

projects for users and developers communities has

aroused the interest of Software Engineering re-

searchers (Michlmayr et al., 2015) (Fitzgerald,

2006) (Gonzalez-Barahona et al., 2013) (Gonzalez-

Barahona and Robles, 2013). OSS development dif-

fers from proprietary development in various aspects,

such as development processes, team structure, and

developer incentives (Jonsson et al., 2015). Under-

standing the rationale behind the success of OSS

projects can help the staff and developers of non-OSS

projects to draw lessons from reported best practices

and apply them to their projects (Stol and Fitzgerald,

2015) (Rigby et al., 2012).

According to Adams and colleagues (Adams

et al., 2015), there are more than 400 active OSS

distributions, and each year an always greater num-

ber of projects are created. Due to competition and

pressure of users and developers, OSS projects need

to release new features and bug ﬁxes within increas-

ingly shorter time spans. Releasing software every

few weeks is typically referred to as a frequent re-

lease cycle, while releasing quarterly or yearly is typ-

ically referred to as a traditional release (TR) cycle

(Mantyla et al., 2013). The adoption of frequent re-

leases in OSS projects takes into account that users

of these projects are scattered all over the world and

eagerly download each new version as soon as it is

released and test it as thoroughly as they can. The

global dispersion of users means that the code can

be tested 24 hours a day (Thomas et al., 2009). To

achieve this, OSS projects rely on volunteers to an-

alyze, implement and test demands registered as bug

issues to integrate them to the latest version (Adams

et al., 2015). The process of reporting and resolv-

ing issues for a system during its development and/or

maintenance is often handled through the use of an

Issue Tracking System (ITS). For each issue, an ITS

can typically record its type (e.g., defect, enhance-

ment, patch, task), its state (e.g., new, assigned, re-

solved, closed) and date of submission. For each state

change, an ITS usually allows recording the submit-

Silva, A., Constantino, K., Carneiro, G., Paula, A., Figueiredo, E., Monteiro, M. and Abreu, F.

The Inﬂuence of Software Product Quality Attributes on Open Source Projects: A Characterization Study.

DOI: 10.5220/0006265400290039

In Proceedings of the 19th International Conference on Enterprise Information Systems (ICEIS 2017) - Volume 2, pages 29-39

ISBN: 978-989-758-248-6

ter, any comments by others, and indications of sever-

ity and/or priority (Bijlsma et al., 2012).

Despite being supported by volunteers, relevant

OSS projects have experts in their communities that

have in-depth knowledge of the source code and com-

ponents (Ye and Kishida, 2003). They are usually

ranked according to their reputation in the project

community as a result of their contributions through-

out the releases (Oreg and Nov, 2008).

Several aspects may inﬂuence issue solving prior-

ity. Recurring tickets on ITS imply bug ﬁxing (im-

proving reliability) or adding new features (improv-

ing functionality) and those are the ones that commu-

nity members usually vote for assigning a scheduling

order (Crowston et al., 2003). However, other soft-

ware product quality attributes may be critical for the

project’s success. It is therefore important to recall

the ISO/IEC 25010 international standard for soft-

ware product quality (ISO, 2011), that replaced the

"old" ISO/IEC 9126 standard (ISO/IEC, 2001), by

providing a quality model composed of several soft-

ware quality characteristics, that are further broken

down into many sub-characteristics, as represented in

Figure 1.

In this paper, we describe an exploratory study

to investigate the inﬂuence of target product quality

attributes in software release practices of three OSS

Projects (Libre Ofﬁce, Eclipse and Mozilla Firefox).

That is, we aim at obtaining evidence on how those

quality attributes are related to the priorities assigned

to registered issues and how they are delivered by

product releases.

The paper is organized as follows. Next section

presents the related work. Section 3 outlines the ex-

ploratory study. Section 4 presents the data collec-

tion and analysis. Section 5 presents the conclusion,

threats to validity and scope for future research.

2 RELATED WORK

Some studies reported the importance and inﬂuence

that software quality attributes have in the software

development cycle of a product (Perepletchikov et al.,

2005)(Offutt, 2002). This inﬂuence is also true for

OSS projects. Two surveys in this direction were

conducted by Henningsson and Wohlin (Henningsson

and Wohlin, 2002). The ﬁrst survey focused on the

literature to capture the understanding of the quality

attributes in the research community. The second one

is an interview survey focused on the perception of

the industry regarding the practice and understanding

of the quality attributes in an industrial context. The

authors concluded that it is clear, from both the litera-

ture and the industrial surveys, that relations do exist

among many of the software quality attributes.

Quality is about perception and depending on the

needs and how well the ﬁnal system or construction

can fulﬁll these needs, the grading of a particular

quality attribute is inﬂuenced. "Grading" herein refers

to the perceived importance of that attribute (Hen-

ningsson and Wohlin, 2002).

The Eclipse Foundation has also undertaken an-

nual surveys to the Eclipse community to better un-

derstand things such as user perception regarding

Eclipse functionalities or how users and developers

get engaged into the OSS community.

1 2 3

3 EXPLORATORY STUDY

In this section, we present an exploratory study to

analyze how software quality attributes inﬂuence the

prioritization of issues to be implemented/solved and

delivered in future releases. Exploratory studies are

intended to lay the groundwork for further empirical

work (Wohlin et al., 2012). For this reason, there is

no control group to compare to. The adopted strategy

consisted in asking developers, outside the commu-

nity of the selected projects, to search for evidence

in data provided by the repositories of the selected

OSS projects. This paper aims to address the follow-

ing four Research Questions.

RQ1- Considering data available in public repos-

itories, which software product quality attributes

are prioritized in software release practices by OSS

Projects? The identiﬁcation of software quality at-

tributes in OSS projects is a key to understand their

priorities throughout their evolution. In practice, de-

pending on the software quality attributes that are pri-

oritized, speciﬁc issues are targeted and implemented

in the next releases while others will be postponed or

even discarded.

RQ2- Which strategies can be used to identify the

prioritized software quality attributes of OSS project

repositories? Unveiling strategies to ﬁnd prioritized

software quality attributes are useful to understand the

rationale used by OSS projects. Successful identiﬁed

practices can be referenced and followed by industry

practitioners, as well as by other OSS projects.

RQ3- What are the values assigned by developers to

the ﬁelds "priority" and "severity" of issues registered

in the projects’ bug repositories related to both pri-

ority and non priority software product quality at-

https://goo.gl/LtzK5E

https://goo.gl/2Fpm7c

https://goo.gl/R3NHpw

ICEIS 2017 - 19th International Conference on Enterprise Information Systems

Figure 1: The ISO/IEC FCD 25010 Product Quality Standard (ISO, 2011).

tributes? This evidence can show how OSS soft-

ware projects address other issues that are not directly

related to the likely prioritized software quality at-

tributes.

RQ4- Are issues related to priority software product

quality attributes delivered in releases just after they

were registered? The time elapsed to deliver an issue

can be an evidence of its priority, according to the

software quality attributes related to it.

To answer these questions, we deﬁned a protocol

(Subsection 3.3) and instantiated it in three phases.

These phases are conveyed in Figure 2, where Steps

1 and 2 were related to the planning of the study and

Steps 3 to 6 were repeated in Phases 1-3 of this ex-

ploratory study. This protocol focused on the analy-

sis of data that have relationships with the bug issues

of three OSS projects (Subsection 3.2). One of the

authors conducted a tutorial session to present partic-

ipants the tools and resources that could be used to

perform the tasks.

3.1 Data Collection

Data Collection. Data collection was carried out

directly from answers provided by the participants.

To answer RQ1, we collected information related to

quality attributes. To answer RQ2, we collected ev-

idence based on references provided by participants,

e.g., print screens and URLs from which relevant in-

formation is accessible. To answer RQ3 and RQ4, we

analyzed the issues indicated by the participants and

the authors.

Passing Score. After the tutorial, we analyzed data

provided by the participants in accordance with the

following two criteria. Criterion 1 was indication of

at least one quality attribute consistent with data from

the analyzed repository. Therefore, inappropriate in-

dications of quality attributes do not meet this cri-

terion. Criterion 2 was indication of evidence from

the repositories supporting the choice of the selected

quality attribute. Therefore, inappropriate indications

of evidence do not meet this criterion.

3.2 Target OSS Projects

This study relies on three highly regarded open source

projects: Mozilla Firefox, Eclipse, and Libre Ofﬁce.

We selected those projects mainly due to the three fol-

lowing reasons. First, they adopted frequent releases

implementation (da Costa et al., 2014) (Gamalielsson

and Lundell, 2014). Second, they have a very active

developer community comprised of 290 (Libre Of-

ﬁce), 1087 (Mozilla Firefox), and 113 (Eclipse) mem-

bers, respectively. Third, all projects provide a vast

repository of documentation, publicly available and

readily accessible. Finally, these projects have been

cited in the literature in different software engineering

studies, which enhances their value as the basis for the

present study (Gamalielsson and Lundell, 2014).

3.3 The Study Protocol

Seven participants took part in this study to answer re-

search questions from RQ1 to RQ4. They were asked

to register the collected evidence related to the analy-

sis they performed in the questionnaire form. More-

over, the participants were asked to describe their

strategies, as well as their experience while using the

OSS project repositories to accomplish the tasks. The

participants had available one week to performed the

asked tasks.

The study involved seven participants recruited

from a MSc and PhD programs in Computer Sci-

ence from two different universities. This number

of participants offered a reasonable trade-off between

the effort to plan and execute the study and detailed

qualitative analysis and the generalizability of results

(Pﬂeeger, 1995) (Yin and Campbell, 2003). They

were all volunteers and no compensation was pro-

vided for their participation in this study.

We were interested in making observations based

on a qualitative analysis of OSS projects repositories,

regarding the possible inﬂuence of software product

quality attributes on the scheduling of pending is-

sues that may have affected the evolution of the an-

The Inﬂuence of Software Product Quality Attributes on Open Source Projects: A Characterization Study

Figure 2: Overview of the exploratory study.

alyzed software product, rather than testing causality

hypotheses using statistical inference. To be eligible

for inclusion, participants ﬁlled out a pre-study ques-

tionnaire (ﬁgure 3) to describe their proﬁle and ex-

perience in software release practices, OSS projects

and software quality product attributes. In Table 1

the corresponding answers using the following scale:

none(N)/low(L)/medium(M)/high(H).

Figure 3: Pre-study questionnaire.

The Tutorial Session. Prior to the tasks, the partici-

pants attended a tutorial session focusing on how to

search for evidence in the Open Stack IAAS Cloud

Platform project using data provided by tools such

as: Bugzilla (issue tracking system), Git (conﬁgura-

tion management system), Gerrit (code review tool

for Git) and a Wiki. These tools are also used by the

selected OSS projects for this study. We selected a

different project in the tutorial session to avoid bias-

Table 1: Results of the Pre-activity Questionnaire.

ID P1 P2 P3 P4 P5 P6 P7

OSS-1 N M L H M L H

OSS-2 N N N N N N M

SRP-1 M L L H M L L

SRP-2 M L L H M M H

SRP-3 M M L H M N M

QA-1 L M M M L L L

QA-2 L M M L N M M

QA-3 L M L N N N L

ing in the study results. The ﬁrst part of the tutorial

focused on how to understand and search for data us-

ing the tool repositories for the Open Stack project.

The second part focused on using the selected repos-

itories to identify evidence that could establish rela-

tionships with the corresponding software product at-

tributes prioritized by Open Stack IAAS Cloud Plat-

form.

4 DATA ANALYSIS

This section presents the analysis of the answers pro-

vided by the participants to the aforementioned re-

search questions.

4.1 First Phase

In the ﬁrst phase of this experiment, we asked partic-

ipants to indicate software product quality attributes

that are best related to each project along with respec-

tive evidence that justify the indication. The goal was

to answer research question RQ1: Considering data

available in public repositories, which software prod-

uct quality attributes are prioritized in software re-

lease practices by the OSS Projects Mozilla Firefox,

Libre Ofﬁce and Eclipse? and RQ2: Which strategies

can be used to identify the prioritized software quality

attributes of the selected OSS project repositories?

There was no predeﬁned quantity of attributes

to be indicated. Table 2 presents the quality at-

tributes indicated by the participants for the three tar-

get projects. The following attributes, already men-

ICEIS 2017 - 19th International Conference on Enterprise Information Systems

tioned in Section 2, were considered in the table:

Functional Suitability (FS), Performance Efﬁciency

(PE), Compatibility (C), Usability (U), Reliability

(R), Security(S), Maintainability (M), and Portability

(P).

According to the data shown in Table 2, the qual-

ity attributes indicated by the participants are pre-

sented as follows to answer RQ1. Figure 4 conveys

the quantity of attributes indicated by each partici-

pant of this study. 5 (71%) participants indicated for

Eclipse the attributes Compatibility and Maintainabil-

ity; Portability was indicated by 4(57%) participants.

In the case of Mozilla FireFox, Maintainability and

Portability were both indicated by 4(57%) of the par-

ticipants, whereas Performance Efﬁciency, Reliability

and Security were also indicated for this project for

3(42%) participants. FireFox had indications of all

quality attributes conveyed in Figure 1. Finally, Li-

bre Ofﬁce had Usability as the quality attribute with 6

indications. The possible reason for this result is the

concern of the developer’s community with a friendly

and intuitive product, as well as the competition with

other similar products. Functional Suitability and

Maintainability were mentioned by 4 participants. Li-

bre Ofﬁce had less uniformity among the indicated

attributes. For example, Reliability and Security were

not mentioned and Performance Efﬁciency and Porta-

bility received just 1 vote each

After the indication of the attributes, the partici-

pants looked for evidence to answer RQ2. The study

protocol did not establish a minimum number of is-

sues, so participants were free to present as many is-

sues as they saw ﬁt

Table 3 conveys the distribution of issues indi-

cated by participants with respective associated soft-

ware product quality attributes for the target projects.

To point out the selected quality attributes, partic-

ipants 1, 3, 5, 6 and 7 adopted the strategy to ﬁlter

issues by their severity level together with their re-

spective bug priority. These participants justiﬁed the

use of this strategy considering that projects adopting

the frequent release approach need to foster the selec-

tion of failures and new features that may have a sig-

niﬁcant impact on the software product. Participant

4 adopted another strategy. He searched for relevant

quality attributes by analyzing each software project

proﬁle and scrutinizing data in the software project

portals and wikis. However, this strategy revealed as

not effective, considering that only selecting quality

attributes through portals and wikis, does not neces-

sarily reﬂect the relationship of these attributes to the

release schedules of those projects. Finally, partici-

pant 2 performed a quantitative analysis of the issues

These issues are available at https://goo.gl/LdkXwJ

found in the projects’ bug tracking systems, consid-

ering the response/solution time for the main compo-

nents of the projects based on the quality attributes

chosen by the participants.

4.2 Second Phase

The second phase aimed at identifying the attributes

of issues that have severity assigned as blocker or crit-

ical to compare these attributes with those indicated

by the participants in the ﬁrst phase. The reason to

search for issues with these types of severity is to fo-

cus on issues considered relevant in the context of the

analyzed projects. During this search, we targeted

the same versions/releases of the projects indicated

by the participants. This second phase comprised the

following researching question: RQ3: What are the

values assigned by developers to the ﬁelds "priority"

and "severity" of issues registered in the projects’ bug

repositories related to both priority and non priority

software product quality attributes?

4.2.1 Eclipse

The search for the issues of Eclipse considered ver-

sions 3.3.2, 3.4, 4.4, 4.5 and 4.6 as well as severity

blocker. As a whole, it returned 86 issues. The is-

sues were analyzed one by one by the researcher to be

characterized one or more assigned quality to which

it is deﬁned.

Figure 5 presents the results of this characteriza-

tion of issues for the Eclipse project. Of the 44 issues

returned in the searches, 66% were about Compati-

bility issues, either with one of the previous versions,

or with some of the coexisting packages. Eclipse is

a large project with several coexisting packages and

many problems have been reported related to this is-

sue. 21% of the issues raised relate to Functional Suit-

ability and 9% to Portability , mainly on installation.

Just 4% relate to Performance Efﬁciency. Other at-

tributes were not identiﬁed for any issue. By coinci-

dence, all issues had the same level of priority.

4.2.2 Mozilla Firefox

For Mozilla FireFox, the following versions were

considered: 23 Branch, 42 Branch, 43 Branch, 45

Branch, 48 Branch, and 49 Branch. In addition, the

severities considered were blocker and critical. In

these conﬁgurations, in total, 28 issues were returned

and an individual analysis to identify which attribute

of quality is more closely represented was carried out.

Figure 6 presents the results. The ﬁrst perception is

that several attributes were identiﬁed in this sample,

as presented in table 3 in the participants’ perception

The Inﬂuence of Software Product Quality Attributes on Open Source Projects: A Characterization Study

Table 2: Product Quality Attributes Indicated by the Participants.

Eclipse Mozilla Firefox Libre Ofﬁce

FS PE C U R S M P FS PE C U R S M P FS PE C U R S M P

P1 1 1 1 1 1 1 1

P2 1 1 1 1 1 1 1 1 1 1 1 1 1 1

P3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

P4 1 1 1 1 1 1

P5 1 1 1 1 1 1

P6 1 1 1

P7 1 1 1 1 1 1 1 1 1 1 1

Total 2 2 5 3 2 0 5 4 1 3 2 2 3 3 4 4 4 1 2 6 0 0 4 1

Figure 4: Software product quality attributes prioritized on releases according to participants.

Table 3: Issues indicated by Participants in Phase 1.

PQA Eclipse Mozilla FireFox Libre Ofﬁce

Functional Suitability 1 0 5

Performance Efﬁciency 3 2 1

Compatibility 2 1 1

Usability 0 0 4

Reliability 1 2 0

Security 0 0 0

Maintainability 2 2 0

Portability 6 3 0

Figure 5: Product Quality Attributes of the Eclipse Issues.

when they also mention more uniformly all the qual-

ity attributes for FireFox. The attributes mentioned

were Compatibility 29%, Functional Suitability and

Reliability with 21%, Performance Efﬁciency with

14%, Security with 11% and Usability with 4%.

4.2.3 Libre Ofﬁce

Finally, for Libre Ofﬁce, the following versions were

considered: 3.4.3, 4.0.0.3, 4.1.0.2, 4.2.0.3, 4.2.4.2,

Figure 6: Product Quality Attributes of the FireFox Issues.

5.0.0.5 and 5.0.3.2. The levels of severity considered

were blocker and critical, from which 11 issues were

derived. The issues were analyzed individually. Fig-

ure 7 presents the results of this characterization for

Libre Ofﬁce. Of the 11 issues returned, 42% were

about Functionality , 33% relate to Performance Efﬁ-

ciency, 17% to Compatibility and 8% relate to Porta-

bility . Other attributes were not identiﬁed for any

issue.

ICEIS 2017 - 19th International Conference on Enterprise Information Systems

Figure 7: Product Quality Attributes of the Libre Ofﬁce Is-

sues.

4.3 Third Phase

The third Phase was to search for the issues reposi-

tory, to check those that were not indicated by par-

ticipants, but had a high level of severity in the three

projects and related to the quality attributes selected

by the participants pointed in Figure 4. We also

checked issues implemented/resolved in releases near

or far from their date of registration. This way, it is

possible to verify the consistency of the selected at-

tributes and allows to verify how they were prioritized

in the release planning.

The adopted strategy was to initially deﬁne a ﬁl-

ter that could select the issues of higher priority and

severity in the Bugzilla and that were already ﬁnalized

or resolved.

The following step was to read the description of

the issues. If any evidence was found, which pointed

to a given quality attribute, that issue would be se-

lected and all its ﬂow would be analyzed so as to con-

ﬁrm it. As in step 1, it was taken into account whether

the issues were implemented/ resolved in releases that

were close to their registration date. We wanted to de-

tect the priority relationships in the release planning

depending on the quality attribute indicated.

Finally, with this phase we derived research ques-

tion RQ4, as stated in Section 3.

4.3.1 Eclipse

For Eclipse, the following ﬁlter was deﬁned: (Status:

Resolved, Closed; Resolution: Fixed; Priority: P1;

Severity: Blocker, Critical; Classiﬁcation: Eclipse).

The focus was on the following quality attributes:

(Functional Suitability (FS), Performance Efﬁciency,

Compatibility, Usability, Reliability, Maintainability

and Portability).

This query returned 492 issues, which exposed 24

issues that had some connections to the selected qual-

ity attributes. One example that stood out was issue

1310554

, which includes the following description

"[GTK3] Problem with table/tree editing". It was reg-

istered on the 29th of June 2014.

This description enabled us to make a connec-

tion with the product’s functionality. However when

analysing the bug report for more detailed informa-

tion, we established a connection with Compatibility,

because the user pointed out that when using a version

of GTK (the multi-platform toolkit for creating graph-

ical user interfaces) some commands would behave

abnormally and that with a newer version of GTK no

command would be working "The editing works cor-

rectly with GTK2. GTK3 <= 3.8 - it works, but a

context menu doesn’t work. Right-click in an editing

ﬁeld, leave edit mode and actions (Cut, Copy, Paste

...) don’t work. GTK3 >= 3.10 - you can’t enter edit-

ing mode.".

Finally, Figure 8 shows the distribution qual-

ity attributes. Of the 24 issues returned, 20.8%

were about problems relating to Functional Suitabil-

ity (FS), 12.5% were about Performance Efﬁciency,

25% about Compatibility, 16.70% about Reliabil-

ity, 8.30% about Maintainability and 16.70% about

Portability.

Figure 8: Findings from Eclipse Issues.

4.3.2 Mozilla Firefox

For Mozilla Firefox, the following ﬁlter was deﬁned

to query the bugs: (Status: Resolved, Closed; Resolu-

tion: Fixed; Priority: P1; Severity: Blocker, Critical;

Product: Firefox). The focus was on the following

quality attributes: (Functional Suitability (FS), Per-

formance Efﬁciency, Compatibility, Usability, Relia-

bility, Security, Maintainability and Portability )

This query returned 83 issues. Analysis of the list

led to 9 issues that had a relation with the selected

quality attributes. One example is issue 1310554

which has the following description "updateChildIn-

dex in PlacesSyncUtils is extremely inefﬁcient" and

https://goo.gl/47eHGz

https://goo.gl/wnca9g

The Inﬂuence of Software Product Quality Attributes on Open Source Projects: A Characterization Study

was registered on the 16th of October 2016, allow-

ing the identiﬁcation of issues related to the product’s

efﬁciency.

When analyzing the bug report to obtain more de-

tails, we found a connection with Efﬁciency, since

the user pointed out that having a few markers in-

side a folder causes Firefox to make a call for each

marker, each call lasting between 5 and 10 seconds

"If you have 10000 bookmarks in a single folder, ap-

plying those records takes many hours, during which

Firefox is completely unresponsive. Instrumenting

PlacesSyncUtils shows this is called 10000 times,

with each call taking around 5-10 seconds.".

The dialog between users reveals that on the 24th

of October 2016, a modiﬁcation was carried out so as

to simplify the way bookmarks are synchronized and

that this update would be available in the current and

subsequent versions.

Figure 9 shows the distribution between attributes.

Of the 9 issues returned, 22.20% were about Perfor-

mance Efﬁciency (PE) problems, 22.20% Usability,

22.20% Security and 33.40% relate to Maintainabil-

ity.

Figure 9: Findings from Mozilla Firefox Issues.

4.3.3 Libre Ofﬁce

For Libre Ofﬁce, the following bug ﬁlter was deﬁned

(Status:Resolved, Closed; Resolution:Fixed; Prior-

ity: Highest; Severity:Blocker, Critical). Focusing

on the following quality attributes: (Functional Suit-

ability (FS), Compatibility, Portability, Usability and

Maintainability)

As a result of this query, we obtained 207 bugs.

The corresponding descriptions were read with a fo-

cus on the quality attributes suggested by the partic-

ipants. If the text had a section that could be asso-

ciated with the qualities attributes indicated, this is-

sue would be analyzed more closely. For example in

issue 84752

says: "Document With Form Controls

Unusable Speed". This section calls into attention the

terms "unusable" and "speed", which suggest a rela-

tion among the attribute Usability. When we accessed

https://goo.gl/4Eks9m

Table 4: Summary of the delivered issues in phase 1.

Quality

Atribute

Release

Interval

Phase 1

Eclipse

Portability 290182 1(4.6.1 - 4.6.2)

Portability 187062 6(4.4.0 - 4.6.0)

Portability 498196 1(4.6 - 4.6.1)

Compatibility 228109 2(3.3.2 - 3.4)

Compatibility 222885 5(3.4 - 3.5M2)

Portability 234307 2(3.4- 3.4RC2)

Portability 232641 2(3.4 - 3.4RC2)

Functional

Suitability

250946 5(3.4 - 3.5M2)

Performance

Efﬁciency

234718 2(3.4 - 3.4RC2)

Firefox

Portability 1242901 9(43 - 52)

Performance

Efﬁciency

1260850 6(45 - 51)

Portability 1287823 4(48 - 52)

Maintanability 1308840 3(49 - 52)

Maintanability 959567 1(42 - 43)

Libre Ofﬁce

Functional

Suitability

74104 1(4.2.0 - 4.2.1)

Functional

Suitability

78801 2(4.2.4 - 4.2.6)

Usability 95709 13(5.0.3 - 5.2.0)

Usability 41169 34(3.4.3 - 4.2.0)

Functional

Suitability

62038 19(4.0.0 - 4.2.5)

Usability 66924 1(4.1.0 - 4.1.1)

Usability 96742 0(5.2.0 - 5.2.0)

the issue to obtain more information, we learned that

when the user made the upgrade, the execution of

some controls became so much slower as to render

the system almost unusable. Another example is the

following issue description: “We have upgraded from

4.2 to 4.3 and found that documents with form con-

trols are too slow to use.”. This issue exposes a con-

nection with the Usability attribute.

Finally, when analyzing the issue list we detected

18 issues that had a relation with the selected qual-

ity attributes. Figure 10 depicts how the distribution

between attributes. From the selected issues 41.17%

were about Usability, 29.41% Functional Suitability

(FS) problems, 11.76% Compatibility, 11.76% Porta-

bility and 5.88% relate to Maintainability.

Figure 10: Findings from Libre Ofﬁce Issues.

4.3.4 Delivered Issues in Releases

Finally, the analysis based on the issues indicated in

phase 1 and 3 was carried out. In this analysis, it was

ICEIS 2017 - 19th International Conference on Enterprise Information Systems

Table 5: Summary of the delivered issues in phase 3.

Quality

Atribute

Release

Interval

Phase 3

Eclipse

Compatibility 308364 5(1.0-1.0M5)

Compatibility 363334 6(4.2-4.2M6)

Compatibility 74341 2(3.0-3.1)

Compatibility 438505 5(4.4-4.5RC2)

Compatibility 28727 4(2.1-2.1M4)

Functional

Suitability

479435 12(4.5-4.6M4)

Functional

Suitability

294650 20(3.5.1-3.7M7)

Functional

Suitability

64540 1(3.0-3.0RC1)

Functional

Suitability

63296 1(3.0-3.0RC1)

Functional

Suitability

14291 5(2.0-2.0M5)

Reability 335263 1(3.6.1-3.6.2)

Reability 97148 2(3.1-3.1RC2)

Reability 16208 6(2.0-2.0M6)

Portability 184433 1(3.3-3.3RC1)

Performance

Efﬁciency

129979 6(3.2-3.2M6)

Performance

Efﬁciency

52982 7(3.0-3.0M7)

Performance

Efﬁciency

27264 4(2.1-2.1M4)

Maintainability 270744 7(3.5-3.5M7)

Maintainability 179210 2(3.3-3.3RC2)

Firefox

Maintainability 1261651 0(48-48)

Maintainability 1211016 0(42 - 42)

Performance

Efﬁciency

1310554 0(51-51)

Performance

Efﬁciency

1303611 1(51-52)

Security 1215885 0(44-44)

Libre Ofﬁce

Usability 84752 3(4.3.0-4.3.3)

Usability 72647 0(4.2.0-4.2.0)

Usability 73464 1(4.2.0-4.2.1)

Usability 89873 0(4.5.0-4.5.0)

Usability 47368 6(3.5.1-3.6.0)

Usability 73243 7(4.2.0-4.3.0)

Usability 72741 13(4.1.0-4.2.5)

Functional

Suitability

43868 0(3.4.5-3.4.5)

Functional

Suitability

55493 4(3.6.0-3.6.4)

Functional

Suitability

76949 1(4.2.3-4.2.4)

Functional

Suitability

36982 6(3.4.0-3.4.6)

Compatibility 38452 7(3.4.0-3.5.0)

Compatibility 64490 30(3.5.4-4.4)

Portability 69517 0(4.2.0-4.2.0)

Portability 92041 0(5.0.0-5.0.0)

taken into account if the issues were implemented/ re-

solved in the following releases relative to their regis-

tration date. This way you could ﬁnd some statement

of priority in the release planning according to the in-

dicated quality attribute. The strategy adopted at this

stage was to ﬁnd through the information provided in

the issues the version that was found the error / ﬁt and

also in which version this was made available. With

this information we access the project portal to ﬁnd

information about project releases and identify how

many versions have been deployed until the issue cor-

rection is made available. This factor we are calling

the interval between releases.

Table 4 presents a summary of the ﬁndings in

phase 1 issues. Table 5 presents the ﬁndings of the

issues in phase 3, which show the attributes of qual-

ity, issues id, and interval between the version that

was made available and the corresponding ﬁx made

4.4 Comparison of the Results

Figures 11, 12 and 13 presents the issue counts found

in the three phases, for the three projects under study.

Figure 11: Software product quality attributes prioritized in

Phase 1, 2 and 3 in Eclipse.

Figure 12: Software product quality attributes prioritized in

Phase 1, 2 and 3 in Mozilla Firefox.

Figure 13: Software product quality attributes prioritized in

Phase 1, 2 and 3 in Libre Ofﬁce.

For Eclipse, Compatibility was the quality at-

tribute that came most to the fore. Especially in

phase 2 of the study, a possible justiﬁcation is due

These issues are available at https://goo.gl/LdkXwJ

The Inﬂuence of Software Product Quality Attributes on Open Source Projects: A Characterization Study

to being a tool that integrates several other tools, and

runs in multiple versions of several operating systems.

Eclipse and the projects created around it must be

compatible and able to adapt to heterogeneous infras-

tructure (hardware, operating systems, system ver-

sions, etc). In addition, Performance Efﬁciency (PE)

presented similar results in the 3 phases of the study.

In Mozilla Firefox, the attributes with most is-

sues raised were Compatibility and Performance Ef-

ﬁciency. However, while Compatibility had a signif-

icant number of issues in phase 2, Performance Ef-

ﬁciency presented more homogeneous results in the

three phases. This may be justiﬁed by the fact that

Firefox is a Web Browser and must be compatible

with different technologies used in portals and be able

to be installed on various operating systems (Compat-

ibility). All this must take place as transparently to the

user as possible, without problems of slowness or in-

operability arising. The performance of the program

is fundamental, because the competition in the mar-

ket is ﬁerce as regards performance (Performance Ef-

ﬁciency).

For Libre Ofﬁce, Functionality was the attribute

indicated most often. It is present in all three phases.

Usability also comes to the fore, although no issues

were found in the second phase. One thing that may

explain the relative prominence of these attributes is

that it is strongly user-oriented, as well as being an

ofﬁce suite, which demands characteristics intrinsic to

its application domain. Therefore, it needs to provide

a signiﬁcant number of functionalities, in ways meet

the (demanding) expectations of its users. In addition,

it must be easily approached by non-technical users

and have an attractive and modern graphical interface,

so that its use becomes intuitive.

5 CONCLUSION

This paper presented an exploratory study to investi-

gate the inﬂuence of target product quality attributes

in software release practices of OSS Projects. We

targeted on four research questions to search for ev-

idence that show the relationships among the target

attributes, the priorities assigned to the registered is-

sues and how they are delivered by product releases.

With this initiative we aim at understand how OSS

projects perform to align their goals with their users

needs.

In this way, it was possible to verify that this prior-

itization is strongly inﬂuenced by the proﬁle and ob-

jective of the project, however it was not possible to

conclude if a certain quality attribute has priority in

the planning of releases of these projects.

5.1 Threats to Validity

In this study three limitations were founded. The ﬁrst

can be assigned to the reduced number of users that

participate in the study. The results of the sample

may have been inﬂuenced because only seven users

participated. A tutorial with a test project was pre-

sented, with a view to reduce this risk, in the hope

of ﬁlling possible gaps that could occur in the data

taken from the public repositories. The second threat

was a possible misinterpretation of the requested ac-

tivity. In order to minimize this threat, the authors

were available to answer any doubts that may be nec-

essary during the execution of this activity. The last

threat concerns to the identiﬁcation of product qual-

ity attributes and issues from Eclipse, Mozilla Fire-

fox and, Libre Ofﬁce projects. To mitigate this risk,

all issues found were discussed to make more reli-

able quality attributes characterized for the selected

projects.

5.2 Work in Progress

As ongoing work, we are know replicating this study

with the support of machine learning techniques. The

goal is to collect and analyze a larger amount of data

to conﬁrm the software quality attributes considered

as priority for each project.

REFERENCES

Adams, B., Kavanagh, R., Hassan, A. E., and German,

D. M. (2015). An empirical study of integration ac-

tivities in distributions of open source software. Em-

pirical Software Engineering, pages 1–42.

Bijlsma, D., Ferreira, M. A., Luijten, B., and Visser, J.

(2012). Faster issue resolution with higher techni-

cal quality of software. Software quality journal,

20(2):265–285.

Crowston, K., Annabi, H., and Howison, J. (2003). Deﬁn-

ing open source software project success. ICIS 2003

Proceedings, page 28.

da Costa, D. A., Abebe, S. L., McIntosh, S., Kulesza, U.,

and Hassan, A. E. (2014). An empirical study of de-

lays in the integration of addressed issues. In ICSME,

pages 281–290.

Fitzgerald, B. (2006). The transformation of open source

software. Mis Quarterly, pages 587–598.

Gamalielsson, J. and Lundell, B. (2014). Sustainability

of open source software communities beyond a fork:

How and why has the libreofﬁce project evolved?

Journal of Systems and Software, 89:128–145.

Gonzalez-Barahona, J. M., Izquierdo-Cortazar, D., Maf-

fulli, S., and Robles, G. (2013). Understanding how

companies interact with free software communities.

IEEE software, (5):38–45.

ICEIS 2017 - 19th International Conference on Enterprise Information Systems

Gonzalez-Barahona, J. M. and Robles, G. (2013). Trends in

free, libre, open source software communities: From

volunteers to companies. it–Information Technology

it–Information Technology, 55(5):173–180.

Henningsson, K. and Wohlin, C. (2002). Understanding the

relations between software quality attributes-a survey

approach. In Proceedings 12th International Confer-

ence for Software Quality. Citeseer.

ISO, I. (2011). Iec 25010: 2011,“. Systems and Software

Engineering—Systems and Software Quality Require-

ments and Evaluation (SQuaRE)—System and Soft-

ware Quality Models.

ISO/IEC (2001). ISO/IEC 9126. Software engineering –

Product quality. ISO/IEC.

Jonsson, L., Borg, M., Broman, D., Sandahl, K., Eldh, S.,

and Runeson, P. (2015). Automated bug assignment:

Ensemble-based machine learning in large scale in-

dustrial contexts. Empirical Software Engineering,

pages 1–46.

Mantyla, M. V., Khomh, F., Adams, B., Engstrom, E.,

and Petersen, K. (2013). On rapid releases and soft-

ware testing. In Software Maintenance (ICSM), 2013

29th IEEE International Conference on, pages 20–29.

IEEE.

Michlmayr, M., Fitzgerald, B., and Stol, K.-J. (2015). Why

and how should open source projects adopt time-based

releases? Software, IEEE, 32(2):55–63.

Offutt, J. (2002). Quality attributes of web software appli-

cations. IEEE software, 19(2):25.

Oreg, S. and Nov, O. (2008). Exploring motivations for

contributing to open source initiatives: The roles of

contribution context and personal values. Computers

in human behavior, 24(5):2055–2073.

Perepletchikov, M., Ryan, C., and Tari, Z. (2005). The im-

pact of software development strategies on project and

structural software attributes in soa. In OTM Con-

federated International Conferences" On the Move

to Meaningful Internet Systems", pages 442–451.

Springer.

Pﬂeeger, S. L. (1995). Experimental design and analysis in

software engineering. Annals of Software Engineer-

ing, 1(1):219–253.

Rigby, P. C., Cleary, B., Painchaud, F., Storey, M.-A., and

German, D. M. (2012). Contemporary peer review in

action: Lessons from open source development. Soft-

ware, IEEE, 29(6):56–61.

Stol, K.-J. and Fitzgerald, B. (2015). Inner source–adopting

open source development practices in organizations:

A tutorial. IEEE Software, (4):60–67.

Thomas, L., Schach, S. R., Heller, G. Z., and Offutt, J.

(2009). Impact of release intervals on empirical re-

search into software evolution, with application to the

maintainability of linux. Software, IET, 3(1):58–66.

Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Reg-

nell, B., and Wesslén, A. (2012). Experimentation in

software engineering. Springer Science & Business

Media.

Ye, Y. and Kishida, K. (2003). Toward an understanding of

the motivation of open source software developers. In

Software Engineering, 2003. Proceedings. 25th Inter-

national Conference on, pages 419–429. IEEE.

Yin, R. K. and Campbell, D. (2003). Case study research:

Design and methods. applied social science research

methods series, vol. 5.

The Inﬂuence of Software Product Quality Attributes on Open Source Projects: A Characterization Study