Evaluation of Non-Functional Requirements for IoT Applications

Joseane O. V. Paiva

, Rossana M. C. Andrade

and Rainara Maia Carvalho

Group of Computer Network, Software Engineering and Systems, Federal University of Cear

a, Fortaleza, Brazil

Keywords:

Internet of Things, Software Quality, Non-Functional Requirements, Systematic Mapping.

Abstract:

Internet of Things (IoT) is a paradigm that enables physical objects to interact and to work together. IoT

applications have particular characteristics, such as context-awareness, interconnectivity, and heterogeneity,

and particular types of interaction, user interaction with devices (called human-thing interaction), and the

interaction between devices (called thing-thing interaction). These characteristics represent the expectation

around the system and are also known as Non-Functional Requirements (NFRs). So, during the requirements

elicitation of such systems, they can appear as NFRs, and their combination often increases the complexity

of the IoT application development and evaluation. Thus, this work aims to identify which approaches and

NFRs have been considered in the literature to evaluate IoT applications and the main challenges faced by

the evaluators. We use the systematic mapping methodology to provide a comprehensive view of approaches,

methods, tools, and processes. As a result, we identiﬁed two tools, six approaches, one method, and one

process that can be used to evaluate NFRs, a set of 42 NFRs that can be considered for IoT applications, and

the main challenges related to the NFRs evaluation for the IoT applications.

1 INTRODUCTION

Internet of Things (IoT) can be deﬁned as a paradigm

where intelligent objects interact in an environment

through a wireless connection, being able to cooper-

ate to provide services and to achieve common goals

(Atzori et al., 2010).

IoT applications are increasingly present in our

daily lives, on the streets, in the malls, at work, or

in our homes (Rowland et al., 2015). Users have

been adapted themselves to the presence of devices

that, through sensors, capture data about their envi-

ronment, their health, their behavior, and learned to

use the facilities provided by these solutions.

The growing presence around us of intelligent ob-

jects or things like smartphones, smartwatches, intel-

ligent lamps, among others, shows that IoT has be-

come a reality in our lives, being it in the industrial or

personal context.

On the other hand, the way users interact with

these solutions may differ from how s/he interacts

with traditional applications.

IoT applications deal with countless other forms

of interactions such as gestures, vocal commands,

https://orcid.org/0000-0003-1700-8765

https://orcid.org/0000-0002-0186-2994

https://orcid.org/0000-0002-4806-0542

or the total absence of user actions (Rowland et al.,

2015).

Besides the user interaction with devices (i.e.,

Human-Thing Interaction), IoT applications present

a new type of interaction, the Thing-Thing Interac-

tion, which is the interaction between intelligent ob-

jects. The connected devices relate without human

intervention to perform a task (Andrade et al., 2017).

Moreover, IoT applications have particular char-

acteristics, such as context-awareness, interconnectiv-

ity, and heterogeneity, that should be considered in the

requirements elicitation and evaluation.

In this scenario, due to the inherent complexity

of IoT applications, the traditional evaluation meth-

ods available in the literature may not cover all the

quality properties that need to be considered. These

quality properties or attributes are also known as non-

functional requirements (NFRs) (Uckelmann et al.,

2011).

NFRs can be deﬁned as a description of a prop-

erty or characteristic that a system must exhibit or a

constraint that it must respect (Wiegers and Beatty,

2013).

Meeting the chosen NFRs is vital to the success

of the software (Chung and Prado Leite, 2009). How-

ever, ensuring that the system meets NFRs is not a

trivial task and there is a need to ﬁnd out how NFRs

Paiva, J., Andrade, R. and Carvalho, R.

Evaluation of Non-Functional Requirements for IoT Applications.

DOI: 10.5220/0010461901110119

In Proceedings of the 23rd International Conference on Enterprise Information Systems (ICEIS 2021) - Volume 2, pages 111-119

ISBN: 978-989-758-509-8

111

of IoT applications have been evaluated.

Quality evaluation for IoT applications is a topic

that has received attention from the scientiﬁc commu-

nity. Kim (2016), for example, proposed a quality

model for IoT applications and speciﬁc measures for

this type of application.

Andrade et al. (2017) discuss how we can beneﬁt

from the ubiquitous ﬁeld of systems interaction eval-

uation to evaluate interaction with IoT applications,

focusing on both systems’ main differences and simi-

larities and, Bures et al. (2020) present a set of quality

characteristics for IoT applications which speciﬁcally

emphasizes the aspects of security, privacy, reliability

and usability.

However, we did not ﬁnd out a work that summa-

rizes how the evaluation of NFRs for IoT applications

has been conducted. Thus, in order to contribute with

the NFR evaluation ﬁeld, this work aims to provide

a comprehensive view on approaches, methods, tools,

and processes used for the evaluation of NFRs for IoT

applications as well as the main challenges faced by

the evaluators.

For this, we apply the systematic mapping

methodology, which is designed to give an overview

of a research topic by classifying and counting contri-

butions in relation to the categories of that classiﬁca-

tion (Petersen et al., 2015).

The remainder of this paper is organized as fol-

lows: Section 2 provides a theoretical background,

discussing IoT challenges, and the non-functional re-

quirements. Section 3 describes our methodology;

Section 4 presents the results; in Section 5, we dis-

cuss the results; and in Section 6 we expose our ﬁnal

considerations and future work.

2 BACKGROUND

This study aims to provide an overview of how NFRs‘

assessments have been conducted for the IoT applica-

tions. Therefore, we consider it appropriate to provide

a theoretical foundation regarding the IoT and NFRs

as follows.

2.1 Internet of Things

The term Internet of Things was introduced by Kevin

Ashton in 1999 and was associated with RFID tech-

nology (Ashton, 2009).

Since then, the purpose and use of this technol-

ogy have evolved, and today IoT can be deﬁned as a

paradigm where smart objects interact in an environ-

ment through a wireless connection, being able to co-

operate in providing services and achieving common

goals (Atzori et al., 2010).

IoT can be considered an extension of Ubiquitous

Computing (Carvalho et al., 2020), which refers to

devices connected everywhere in such a transparent

way that we will not realize they are there (Weiser,

1991).

Due to their similarities, such as mobility and

context-awareness, the artifacts that come from Ubiq-

uitous Computing are suitable for IoT applications

(Andrade et al., 2017). Therefore, in this work, we

consider the NRFs focused on Ubiquitous Computing

as suitable for IoT applications.

On the other hand, IoT applications have also

some singularities, such as interconnectivity, hetero-

geneity, and services related to objects.

We present then the following list of IoT features

and their deﬁnitions (Patel et al., 2016):

• Interconnectivity: In IoT applications, differ-

ent objects can be connected, sharing information

within a communication infrastructure.

• Object-related Services: IoT applications can

provide services related to the objects that inte-

grate the solution, such as privacy protection and

semantic consistency between objects.

• Heterogeneity: An IoT application is composed

of heterogeneous devices, both in aspects related

to hardware and aspects related to how they were

implemented. However, these devices can interact

with other devices or service platforms through

different networks.

• Dynamicity: The state of devices and the context

changes dynamically. Devices can, for example,

be connected or disconnected, vary in location,

have their settings changed, among others. The

number of devices that makes up an IoT applica-

tion can also vary dynamically, depending on the

environment and other entities’ presence (for ex-

ample, objects and people).

• Scalability: IoT applications provide the recog-

nition of each object that makes up a solution and

ensure communication between them. Since the

number of objects that makes up an IoT solution

can vary, the system must remain operational re-

gardless of the number of connected objects.

• Security: IoT applications must be designed with

the security of their use in mind; this includes per-

sonal data and their user’s well-being.

• Interconnectivity: Allows network access and

devices’ ability to consume and produce data

within an IoT solution.

ICEIS 2021 - 23rd International Conference on Enterprise Information Systems

112

Limitations related to these features diminish their

potential and impact usability (Rowland et al., 2015).

Therefore, it is critical to ensure that these properties

are performing as expected in IoT applications.

2.2 Non-Functional Requirements

Non-Functional Requirements (NFRs) can be deﬁned

as requirements that are not directly related with spe-

ciﬁc functionalities offered by the software. They can

be related to the emerging properties of the system,

such as Reliability and Security (Sommerville, 2011).

According to (Wiegers and Beatty, 2013), quality

characteristics are also known as a type of NFRs and

they describe the product’s characteristics in various

dimensions considered important by the stakeholders,

such as security and usability.

Thus, the software quality assurance demands that

quality characteristics (or NFRs) are speciﬁed, mea-

sured, and evaluated, whenever possible, using vali-

dated or widely accepted measures and measurement

methods (ISO/IEC 25000, 2011).

Another important point to stress in this paper

concerning the evaluation of NFRs is that they can

be correlated, which means that one NFRs can has

a impact in another one. This impact could be pos-

itive (helps) or negative (hurts) (Wiegers and Beatty,

2013).

3 METHODOLOGY

As we mentioned in the introduction section, this re-

search aims to identify tools, methods, approaches,

and processes for evaluating NFRs in IoT applica-

tions.

To this end, we conducted a systematic mapping

of the literature (SML), following the guidelines pro-

posed by (Kitchenham and Charters, 2007), which di-

vides the research conduction into three stages: Plan-

ning (Section 3.1), Conduction (Section 3.2), and

Analysis and Reporting (Section 4).

3.1 Planning

In the ﬁrst stage of the SML methodology, we set

up all the research parameters for conducting the re-

search in a document called the research protocol. In

this protocol, we deﬁned the research questions, the

search strategy, the selection criteria, and the data to

be extracted.

3.1.1 Research Questions

According to the aim of this study, the following re-

search question was deﬁned:

RQ1 - How Are the Non-Functional Requirements

Evaluations Conducted in IoT Applications?

To better answer this question, we had also de-

ﬁned a set of secondary questions as follows:

• SQ1 - Which are the approaches, tools, methods,

and processes used to evaluate non-functional re-

quirements in IoT applications?

• SQ2 - Which are the non-functional requirements

observed when evaluating IoT applications??

• SQ3 - Which are the challenges faced when eval-

uating non-functional requirements?

3.1.2 Search Strategy

The ﬁrst step in selecting the papers that answered

our search questions was to conduct an automated

database search.

For this, we used PICO (Population, Intervention,

Comparison, and Outcomes), which is a strategy sug-

gested by Kitchenham and Charters (2007) to identify

keywords related to the deﬁned search questions for

the search string.

• Population: In software engineering, the popu-

lation can represent a speciﬁc software engineer-

ing function, an application area, or an industry

group. In our context, the population was deﬁned

as IoT applications, and we also considered ubiq-

uitous and pervasive systems due to their similar-

ities;

• Intervention: Intervention is deﬁned with the

methodology, tool, technology, or software proce-

dure used to address a speciﬁc issue. In our case,

the intervention was deﬁned as the non-functional

requirements;

• Comparison: Comparison can be deﬁned as the

methodology, tool, technology, or procedure with

which the intervention is being compared. For this

research, we do not deﬁne elements of compari-

son;

• Outcome: Outcomes should relate to factors of

importance to practitioners in the study area. In

our case, we selected terms that represented eval-

uation approaches as outcomes.

For each PICO element, we also selected syn-

onyms to minimize the risk of restricting the search

results. Table 1 shows the terms chosen by each cate-

gory.

Evaluation of Non-Functional Requirements for IoT Applications

113

Table 1: Selecting search terms.

PICO Strategy

Population

(“Internet of Things” OR IoT OR

“Pervasive*” OR “Ubiquitous”)

Intervention

“non-functional requirement” OR

nfr OR “non-functional property”

OR “quality characteristic” OR

“quality attribute” OR

“quality requirement” OR

“extra-functional requirement” OR

“non-behavioural requirement” OR

“quality factor”)

Comparison No terms selected

Outcome

(evaluation OR assessment OR

veriﬁcation OR validation)

AND (method OR tool OR

technique OR approach OR

process)

When connecting the search terms with logical

operators, we elaborated the following search string:

(“Internet of Things” OR IoT OR “per-

vasive” OR “ ubiquitous”) AND ( “non-

functional requirement*” OR nfr OR “non-

functional propert*” OR “quality character-

isti*” OR “quality attribute*” OR “qual-

ity requirement*” OR “extra-functional re-

quirement*” OR “non-behavioural require-

ment*” OR “non-behavioral requirement*”

OR “quality factor*” ) AND ( evaluation OR

assessment OR veriﬁcation OR validation)

AND ( method* OR tool* OR technique* OR

approach* OR process*)

3.1.3 Research Sources Selection

We applied the search string in two databases that

were chosen due to their good coverage and stabil-

ity: Scopus

and Web of Science

. Therefore, Scopus

incorporates other important bases for the Software

Engineering ﬁeld, such as ACM and IEEE.

In addition, we also performed a snowballing for-

ward procedure, which means identifying new papers

http://www.scopus.com/search

http://www.webofknowledge.com

based on a set of papers that cite the paper being ex-

amined (Wohlin, 2014).

3.1.4 Study Selection Criteria

We selected for this research, papers that present

tools, approaches, methods, or processes for evaluat-

ing non-functional requirements for IoT applications.

As for time range, once IoT was ﬁrst cited in 1999

(Ashton, 2009), articles published between 2000 and

2019 were included in the current research. To ensure

the correct interpretation of our research results and

the replicability of the study, we limited our search to

papers written in English.

Regarding the exclusion criteria, we did not con-

sider eligible for this research papers that:

• are secondary or tertiary works;

• are books, technical reports, white papers, short

papers (less than 5 pages);

• are not available for download; and

• do not present tools, approaches, methods, or pro-

cesses of non-functional requirements evaluation

for IoT applications.

3.1.5 Data Extraction

After we select the studies, they were submitted to

the data extraction step that consists of exploring the

work selected to extract information that will help us

to answer our research questions.

Table 2 presents the data to be extracted in the se-

lected papers.

3.2 Conducting

This conducting stage was divided into two steps.

First, we applied the search string in the Web of Sci-

ence and Scopus databases. We selected the papers

through the study selection criteria. At the end of this

process, we obtained a set of approved papers. Then,

we used the selected papers to apply the snowballing

forward procedure (Wohlin, 2014).

3.2.1 Fist Step: Search on Databases Sources

We started by applying the string search to the se-

lected databases.

The search string returned 174 papers, 121 by

Scopus and 51 by Web of Science. After removing

49 duplicated studies, we obtained 125 papers for the

title/abstract screening. At this step, we excluded 88

papers based on our study selection criteria. Thus, we

got a set of 37 papers eligible for the entire reading.

ICEIS 2021 - 23rd International Conference on Enterprise Information Systems

114

Table 2: List of data to be extracted.

Data extraction form

Study metadata

Title, authors,

and year of publication.

Approaches,

methods, tools,

or process

Strategy adopted

to evaluate the NFRS.

Type of system

Ubiquitous System

or IoT application.

NFRs

The NFRs evaluated.

Evaluation focus

System in development,

system implemented

or both.

Challenges

The challenges on

evaluating NFRs

reported within the paper.

After downloading and reading the full text of the

37 papers, we excluded 27 papers that had not an-

swered our research questions. Thus, we got a set of

10 papers for data extraction.

3.2.2 Second Step: Snowballing Forward

To complement the search results through the

databases, we decided to perform the Forward Snow-

balling procedure, which consists of choosing key pa-

pers from the study area and analyzing the papers that

have cited them (Wohlin, 2014).

For this research, we analyzed the citations refer-

ring to the 10 papers selected during the ﬁrst phase.

We performed these search through Google Scholar

, by searching for each key papers, and then for all

the papers that cited them.

Thus, 80 papers were identiﬁed and submitted to

the same steps and study selection criteria used in the

ﬁrst stage. At the end, we excluded 78 papers that

did not answer our research questions and selected 2

papers for data extraction.

4 RESULTS

This section describes the third step of our SML

(i.e., Analysis and Reporting), where we analyzed

http://scholar.google.com

and summarized the research results to answer the re-

search question: RQ1 - How are the non-functional

requirements evaluations conducted in IoT applica-

tions?

Since we highlight the singularities of IoT appli-

cations, it is necessary to understand how these solu-

tions have been evaluated.

Finally, we selected 12 documents for data extrac-

tion, following the data extraction form. After that,

we analyzed the results and answered our research

questions. Table 3 shows the ﬁnal set of studies and

their respective ID.

Table 3: List of ﬁnal set of studies.

ID Reference

S1 (Ruiz-L

opez et al., 2012)

S2 (Ruiz-L

opez et al., 2013)

S3 (Kim et al., 2017)

S4 (Rocha et al., 2017)

S5 (Munnelly and Clarke, 2008)

S6 (Jazdi et al., 2016)

S7 (Ruiz-L

opez et al., 2013)

S8 (Badii et al., 2018)

S9 (Ghasemi et al., 2019)

S10 (Chauhan and Babar, 2017)

S11 (Filho et al., 2020)

S12 (Alegre-Ibarra et al., 2018)

Figure 1 represents the PRISMA Flow Diagram

(Moher et al., 2009) that describes the process

adopted to conduct this research and its results.

Figure 2 and Figure 3 show, respectively, the dis-

tributions of studies by the type of systems that they

were focused on and the type of NFR evaluation, if

they were focused on systems in development or sys-

tems already implemented.

SQ1 - Which Are the Approaches, Tools, Methods,

and Processes Used to Evaluate Non-Functional

Requirements in IoT Applications?

We identiﬁed 2 tools, 6 approaches, 1 method, and 1

process that can be used for the evaluation of NFRs.

Table 4 presents the strategies identiﬁed for evaluat-

ing NFRs in IoT systems.

SQ2 - Which Are the Non-Functional Require-

ments Observed When Evaluating IoT Applica-

tions?

Once we understood what had been used to conduct

the evaluations, it is important to identify if the eval-

Evaluation of Non-Functional Requirements for IoT Applications

115

Figure 1: PRISMA ﬂow diagram.

uators had considered speciﬁc NFRs to evaluate IoT

applications.

We found 7 studies that described one or more

NFRs as suitable for evaluating ubiquitous systems or

IoT applications.

A total of 42 NFRs were identiﬁed

. Table 5

shows the list of NFRs considered during quality eval-

uations in IoT applications.

SQ3- Which Are the Challenges Faced When Eval-

uating Non-Functional Requirements?

By analyzing the ﬁnal set of studies, we also identiﬁed

some challenges concerning the evaluation of NFRs

in IoT applications.

The most recurrent challenge is the necessity to

deal with context-awareness. Other problems are the

The list of 42 NFR with its descriptions is available on

https://github.com/great-ufc/NFRs4IoT

Figure 2: Type of system prioritized by the studies.

Figure 3: Focus of NFR Evaluation prioritized by the stud-

ies.

lack of a systematic approach and appropriate meth-

ods for evaluating non-functional requirements in IoT

applications.

5 DISCUSSION

We performed a systematic mapping to answer the

following primary research question: How are the

non-functional requirements evaluations conducted in

IoT applications?

Moreover, to better address the aspects involved in

this question, we decided to divide this macro ques-

tion into three secondary questions (SQs). The an-

swers came from a detailed analysis of 12 out of 174

papers returned from the search in databases and the

snowballing forward application.

Regarding SQ1, we noticed that most of the ap-

proaches, methods, and tools are focused on the re-

quirements elicitation stage and not on evaluating if

the IoT systems meet the NFRs previously estab-

lished.

ICEIS 2021 - 23rd International Conference on Enterprise Information Systems

116

Table 4: Strategies identiﬁed by each study.

Study Proposal Type

S1 MD-UBI Process

S2 MD-UBI Process

Architecture-based

energy evaluation

Approach

S4 HUbis Tool

S5 GQM Method

Determination of the

software reliability

approach

Approach

S7 REUBI Method

S8 Performance evaluation Approach

ATAM scenario-based

approach

Approach

S10

Reference Architectures

for Design and Evaluation

of Web of Things Systems

Approach

S11 MALTU Approach

S12 RC-ASEF Tool

Also, we only identiﬁed one process by this re-

search, MD-UBI (Ruiz-L

opez et al., 2013). This pro-

cess was deﬁned to support developers and evalua-

tors in elicitation, representation, and evaluation of re-

quirements, focusing on NFRs. This process is model

oriented and aims to develop high-quality Ubiquitous

Systems, covering the entire software development

lifecycle.

Regarding to SQ2, we identiﬁed 42 non-

functional requirements.

Among the 42 NFRs identiﬁed, 14 of these

are present as characteristics and sub-characteristics

in the quality models proposed by ISO/IEC 25000

(2011).

• System/software product quality: adaptation,

availability, ease of use, interoperability, main-

tenability, modiﬁability, performance, reliability,

reusability, security, testability and usability.

• Quality in use: efﬁciency and effectiveness.

Two of the selected papers have also cited 27 qual-

ity characteristics for ubiquitous systems. This set

is proposed by Carvalho et al. (2017) and has some

characteristics in common with the two quality mod-

els proposed by ISO/IEC 25000 (2011), but it con-

tains speciﬁc characteristics for ubiquitous systems:

Table 5: NFRs distribution per study.

NFRs Cited by

Acceptability S4, S11

Adaptation S4, S11

Attention S4, S11

Availability S4,S9, S10, S11

Calmness S4, S11

Context-awareness S4,S11

Comprehensibility S5

Data Input S4,S11

Device Capability S4, S11

Ease of use S4,S11

Effectiveness S4, S11

Efﬁciency S4,S11

Elasticity S10

Energy efﬁciency S3

Familiarity S4,S11

Flexibility S4,S11

Information display S4, S11

Interconnectivity S4, S11

Interoperability S9, S10

Maintenability S5

Maneageability S5

Mobility S4, S11

Modiﬁability S9

Multi-tenancy S10

Network Capability S4,S11

Performance S8, S9

Positioning of

components S4,S11

Predictability S4, S11

Privacy S4,S11

Reliability S4,S6, S10, S11

Reusability S5

Robustness S4,S11

Safety S4,S11

Scalability S4,S5, S8, S10, S11

Security S4,S9, S10, S11

Simplicity S4,S11

Testability S5

Transparency S4,S11

Trust S4, S11

Usability S4,S5, S9, S11

User Satisfaction S4,S11

Utility S4,S11

• Acceptability, attention, calmness, context-

awareness, device capability, familiarity,

interconnectivity, mobility, network capabil-

ity, predictability, privacy, robustness, safety,

scalability, simplicity, transparency, trust, user

satisfaction, and utility.

Evaluation of Non-Functional Requirements for IoT Applications

117

Given the similarities between IoT applications

and ubiquitous systems, the use of this set of quality

characteristics is also applicable for conducting qual-

ity evaluations on IoT systems.

The most evaluated NFR was Scalability, which is

the ability to provide services to a few or a large num-

ber of users (Carvalho et al., 2017). Scalability was

considered an important NFR to evaluate by 41,6%

of the studies (5 papers). Another NFR identiﬁed is

related with scalability:

• Elasticity: The ability of the system to provide

particular service on demand during a time inter-

val (Chauhan and Babar, 2017).

Availability, reliability, security and usability were

also highlighted. They were both cited by 33% of the

studies (4 papers).

For SQ3, we identiﬁed that the most signiﬁcant

challenge when evaluating IoT systems is the context-

awareness, which is the system’s ability to discover

and take advantage of contextual information such as

user location, time of day, and user activity (Musumba

and Nyongesa, 2013). Thus, it is necessary to ensure

that the system correctly identiﬁes the contextual in-

formation, so the users will be beneﬁted instead of

affected by this non-functional requirement.

Another challenge is the lack of speciﬁc ap-

proaches for the evaluation of NFRs in IoT applica-

tions. This result is in line with the discussion held at

SQ1, where we identiﬁed only 3 studies that focused

on the evaluation of NFRs satisfaction and only one

process that has a step to evaluate NFRs in the ﬁnal

product.

6 CONCLUSION

IoT applications are gaining more and more space in

our daily lives. Therefore, it is necessary to look at

how these solutions have been evaluated, mainly con-

cerning non-functional requirements, because they

are directly related to the software quality evaluation

and the user expectations through the system.

In this work, we perform a systematic mapping on

the evaluation of non-functional requirements in IoT

systems to provide a comprehensive view about this

ﬁeld.

As the main contributions, we identify artifacts

(tools, methods, approaches, and processes) to eval-

uate NFRs and we perceived the lack of artifacts

and systematic approaches to evaluate whether IoT

applications satisfy the previously established non-

functional requirements or not.

We also identiﬁed 42 NFRs that can be considered

to design and evaluate of IoT applications. Most ex-

isting artifacts are focused on the usability and relia-

bility of the solutions, which may represent a concern

about how the users interact and how they relate to

IoT applications.

As future work, there is a need to develop and

validate a process to systematize the NFR evaluation

steps for IoT applications with a particular eye to the

human-thing and thing-thing interactions.

ACKNOWLEDGEMENTS

We would like to thank CNPq for the Productiv-

ity Scholarship of Rossana M. C. Andrade DT-2

◦

315543 / 2018-3) and UFC, FASTEF, and Dell

cooperation using the Brazilian Informatics Law (No

10.176of 1/11/2001) incentives that help us to support

our research laboratory.

REFERENCES

Alegre-Ibarra, U., Augusto, J. C., and Evans, C. (2018).

Rc-asef: An open-source tool-supported requirements

elicitation framework for context-aware systems de-

velopment. In 2018 Federated Conference on Com-

puter Science and Information Systems (FedCSIS),

pages 829–838.

Andrade, R. M., Carvalho, R. M., de Ara

ujo, I. L., Oliveira,

K. M., and Maia, M. E. (2017). What changes from

ubiquitous computing to internet of things in inter-

action evaluation? In Lecture Notes in Computer

Science (including subseries Lecture Notes in Artiﬁ-

cial Intelligence and Lecture Notes in Bioinformatics),

volume 10291 LNCS, pages 3–21. Springer Verlag.

Ashton, K. J. (2009). That ‘internet of things’ thing.

Atzori, L., Iera, A., and Morabito, G. (2010). The In-

ternet of Things: A survey. Computer Networks,

54(15):2787–2805.

Badii, C., Belay, E. G., Bellini, P., d. Cenni, Marazzini, M.,

Mesiti, M., Nesi, P., Pantaleo, G., Paolucci, M., Val-

tolina, S., Soderi, M., and Zaza, I. (2018). Snap4city:

A scalable iot/ioe platform for developing smart city

applications. In 2018 IEEE SmartWorld, Ubiquitous

Intelligence Computing, Advanced Trusted Comput-

ing, Scalable Computing Communications, Cloud Big

Data Computing, Internet of People and Smart City

Innovation (SmartWorld/ SCALCOM/ UIC/ ATC/ CB-

DCom/ IOP/ SCI), pages 2109–2116.

Bures, M., Bellekens, X., Frajtak, K., and Ahmed, B. S.

(2020). A comprehensive view on quality charac-

teristics of the iot solutions. In Jos

e, R., Van Laer-

hoven, K., and Rodrigues, H., editors, 3rd EAI In-

ternational Conference on IoT in Urban Space, pages

59–69, Cham. Springer International Publishing.

Carvalho, R. M., Andrade, R. M., and Oliveira, K. M.

(2020). How developers believe invisibility impacts

ICEIS 2021 - 23rd International Conference on Enterprise Information Systems

118

nfrs related to user interaction. In 2020 IEEE 28th

International Requirements Engineering Conference

(RE), pages 102–112. IEEE.

Carvalho, R. M., Castro Andrade, R. M., Oliveira, K. M.,

Sousa Santos, I., and Bezerra, C. I. (2017). Quality

characteristics and measures for human—computer

interaction evaluation in ubiquitous systems. Software

Quality Journal, 25(3):743–795.

Chauhan, M. A. and Babar, M. A. (2017). Chapter 7

- using reference architectures for design and eval-

uation of web of things systems**this chapter de-

scribes a method for software architecture design,

analysis, evaluation and evolution of individual inter-

net of things (iot) subsystems and how individual iot

subsystems can be used for design of web of things

(wot) systems.: A case of smart homes domain. In

Sheng, Q. Z., Qin, Y., Yao, L., and Benatallah, B., ed-

itors, Managing the Web of Things, pages 205 – 228.

Morgan Kaufmann, Boston.

Chung, L. and Prado Leite, J. C. (2009). On Non-

Functional Requirements in Software Engineering,

page 363–379. Springer-Verlag, Berlin, Heidelberg.

Filho, J. C. S., Brito, M. R. F., and Sampaio, A. L.

(2020). Comparing heuristic evaluation and maltu

model in interaction evaluation of ubiquitous systems.

In Anais Estendidos do XIX Simp

osio Brasileiro so-

bre Fatores Humanos em Sistemas Computacionais

(IHC’20), Porto Alegre, RS, Brasil. SBC.

Ghasemi, F., Rezaee, A., and Rahmani, A. M. (2019).

Structural and behavioral reference model for IoT-

based elderly health-care systems in smart home.

International Journal of Communication Systems,

32(12):e4002.

ISO/IEC 25000 (2011). Iso/iec 25000 - systems and soft-

ware engineering - systems and software quality re-

quirements and evaluation (square). Technical report.

Jazdi, N., Oppenlaender, N., and Weyrich, M. (2016).

Quantiﬁcation of the quality characteristics for

the calculation of software reliability. IFAC-

PapersOnLine, 49(30):1 – 5. 4th IFAC Symposium

on Telematics Applications TA 2016.

Kim, D., Choi, J.-Y., and Hong, J.-E. (2017). Evaluating

energy efﬁciency of internet of things software archi-

tecture based on reusable software components. In-

ternational Journal of Distributed Sensor Networks,

13(1):1550147716682738.

Kim, M. (2016). A quality model for evaluating IoT appli-

cations. International Journal of Computer and Elec-

trical Engineering, 8(1):66–76.

Kitchenham, B. and Charters, S. (2007). Guidelines for per-

forming systematic literature reviews in software en-

gineering. Technical Report EBSE 2007-001, Keele

University and Durham University Joint Report.

Moher, D., Liberati, A., Tetzlaff, J., and Altman, D. G.

(2009). Preferred reporting items for systematic re-

views and meta-analyses: the prisma statement. An-

nals of Internal Medicine, 151(4):264–269.

Munnelly, J. and Clarke, S. (2008). Infrastructure for ubiq-

uitous computing: Improving quality with modulari-

sation. In Proceedings of the 2008 AOSD Workshop

on Aspects, Components, and Patterns for Infrastruc-

ture Software, ACP4IS ’08, New York, NY, USA. As-

sociation for Computing Machinery.

Musumba, G. and Nyongesa, H. (2013). Context awareness

in mobile computing: A review. International Journal

of Machine Learning and Applications, 2(1):5.

Patel, K. K., Patel, S. M., and Scholar, P. G. (2016). Inter-

net of Things-IOT: Deﬁnition, Characteristics, Archi-

tecture, Enabling Technologies, Application & Future

Challenges. International Journal of Engineering Sci-

ence and Computing.

Petersen, K., Vakkalanka, S., and Kuzniarz, L. (2015).

Guidelines for conducting systematic mapping stud-

ies in software engineering: An update. Information

and Software Technology, 64:1 – 18.

Rocha, L. C., Andrade, R. M. C., Sampaio, A. L., and Lelli,

V. (2017). Heuristics to evaluate the usability of ubiq-

uitous systems. In Streitz, N. and Markopoulos, P.,

editors, Distributed, Ambient and Pervasive Interac-

tions, pages 120–141, Cham. Springer International

Publishing.

Rowland, C., Goodman, E., Charlier, M., Light, A., and Lui,

A. (2015). Designing connected products: UX for the

consumer Internet of Things. ” O’Reilly Media, Inc.”.

Ruiz-L

opez, T., Noguera, M., Rodr

ıguez F

ortiz, M. J., and

Garrido, J. L. (2013). Requirements systematization

through pattern application in ubiquitous systems. In

van Berlo, A., Hallenborg, K., Rodr

ıguez, J. M. C.,

Tapia, D. I., and Novais, P., editors, Ambient Intelli-

gence - Software and Applications, pages 17–24, Hei-

delberg. Springer International Publishing.

Ruiz-L

opez, T., Rodr

ıguez-Dom

ınguez, C., Noguera, M.,

and Rodr

ıguez, M. J. (2012). A model-driven ap-

proach to requirements engineering in ubiquitous sys-

tems. In Novais, P., Hallenborg, K., Tapia, D. I., and

Rodr

ıguez, J. M. C., editors, Ambient Intelligence -

Software and Applications, pages 85–92, Berlin, Hei-

delberg. Springer Berlin Heidelberg.

Ruiz-L

opez, T., Rodr

ıguez-Dom

ınguez, C., Noguera, M.,

Rodr

ıguez, M. J., Benghazi, K., and Garrido, J. L.

(2013). Applying model-driven engineering to a

method for systematic treatment of nfrs in ami sys-

tems. J. Ambient Intell. Smart Environ., 5(3):287–310.

Sommerville, I. (2011). Software engineering 9th edition.

ISBN-10, 137035152:18.

Uckelmann, D., Harrison, M., and Michahelles, F. (2011).

An Architectural Approach Towards the Future Inter-

net of Things, pages 1–24. Springer Berlin Heidel-

berg, Berlin, Heidelberg.

Weiser, M. (1991). The computer for the 21 st century. Sci-

entiﬁc american, 265(3):94–105.

Wiegers, K. and Beatty, J. (2013). Software Requirements,

Third Edition. Microsoft Press.

Wohlin, C. (2014). Guidelines for snowballing in system-

atic literature studies and a replication in software en-

gineering. In Proceedings of the 18th International

Conference on Evaluation and Assessment in Software

Engineering, EASE ’14, New York, NY, USA. Asso-

ciation for Computing Machinery.

Evaluation of Non-Functional Requirements for IoT Applications

119