Specifying Industrial System Requirements using Specification Patterns:
A Case Study of Evaluation with Practitioners
Predrag Filipovikj and Cristina Seceleanu
School of Innovation, Design and Engineering, M
¨
alardalen University, V
¨
aster
˚
as, Sweden
Keywords:
Specification Patterns, Pattern-Based Requirements Specification, Industrial Case Study.
Abstract:
With the ever-increasing size and complexity of the industrial software systems there is an imperative need for
an automated, systematic and exhaustive verification of various software artifacts, such as system specifica-
tions, models, code, etc. A potential remedy for this need might lie in a pool of techniques for computer-aided
verification of software related artifacts, including system specifications. The Achilles’ heel of these tech-
niques, and the main hinder for their wider adoption in the industrial development process are the complexity
and the specialized skill-set required for the formal encoding of specifications. To alleviate this problem,
Specification Patterns that are based on the observation that the system specifications are framed within reoc-
curring solutions have been proposed. The approach has been shown to be expressive enough for capturing
requirements in the automotive domain, however, there is a lack of empirical data that can be used to judge its
practical usefulness. In this paper, we involve an existing specification-patterns-based tool, called PROPAS,
and propose a small-size evaluation of the approach with practitioners, on a case study conducted in coopera-
tion with Scania, Sweden, one of the world’s leading manufacturers of heavy-load vehicles. Our results show
that the specification patterns that are supported by an adequate tooling have the potential to be practically
usefull for the non-experts in formal methods.
1 INTRODUCTION
Developing and maintaining system requirements
specifications (system specifications from now on) for
complex software systems is a daunting task. Ac-
cording to the current state-of-the-practice, the system
specifications are written using free text, predomi-
nantly in English. While the free text natural language
specifications are easy to capture and read, such way
of specifying systems suffers from a number of limita-
tions. First, the unrestricted natural language specifi-
cations lack structure, thus leading to situations where
the same requirement can be interpreted in multiple
ways, usually depending on the domain knowledge of
the person who is reading it. Second, using the un-
restricted natural language can potentially lead to an
inconsistent encoding. The most prominent example
of this kind is when a same concept is referred to by
different names.
Such reasons, combined with the ever-increasing
safety-critical importance of the functions, call for
more precise and unambiguous ways of encoding re-
quirements that will also enable rigorous verification
*corresponding author
of various models. Existing techniques for computer-
aided analysis of software-related artifacts, including
the system specifications (Barnat et al., 2016; Post
et al., 2011; Filipovikj et al., 2018) are mostly based
on formal methods, which have the advantage of be-
ing completely automated, mathematically rigorous
and possibly exhaustive. Despite their potential added
value, the formal analysis techniques are still just
seldomly applied in industrial practice. One of the
main obstacles towards a broader industrial adoption
of such techniques seems to be the formal encoding
of the textual specifications in some form of tempo-
ral logic, which is a challenging task for practitioners,
without appropriate tool support.
Specification Patterns (SPS) (Dwyer et al., 1998)
is an approach intended to help practitioners to for-
mally encode system specifications. The approach is
based on reoccurring solutions for specifying require-
ments, which have been collected in a catalogue of
patterns. The initial set of patterns has been extracted
by analyzing more than 500 requirements from vari-
ous domains. The advantage of the specification pat-
terns is that they have a well defined structure and se-
mantics, and as such can be encoded in many differ-
92
Filipovikj, P. and Seceleanu, C.
Specifying Industrial System Requirements using Specification Patterns: A Case Study of Evaluation with Practitioners.
DOI: 10.5220/0007726600920103
In Proceedings of the 14th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2019), pages 92-103
ISBN: 978-989-758-375-9
Copyright
c
2019 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
ent ways, including the constrained natural language
and various formal notations that accurately capture
the semantics of the system behavior described by the
particular requirement.
Besides the body of work focusing on enrich-
ing the specification pattern catalogue (Dwyer et al.,
1998; Konrad and Cheng, 2005; Grunske, 2008; Au-
tili et al., 2015) and providing specialized tool sup-
port (Smith et al., 2002; Mondragon and Gates, 2004;
Remenska et al., 2014), there exist several endeav-
ors targeting the expressiveness of the specification
patterns in the automotive domain, showing that the
specification patterns are expressive enough for for-
malizing the requirements in the domain (Post et al.,
2012; Filipovikj et al., 2014). However, in all stud-
ies the generation of the formal system specifications
based on the specification patterns is performed by
formal methods experts. Consequently, there is a lack
of empirical data when it comes to applying the ap-
proach in industrial settings, by the intended users,
that is, the engineers.
In this paper, we report the results from an ex-
ploratory case study, which is intended to collect data
about the practical usefulness of the specification pat-
terns in industry. The exploratory nature of the case
study means that the generated results are to be used
for proposing hypotheses that can then be tested in
future studies. The case study is performed in coop-
eration with Scania, one of the world’s leading manu-
facturers of heavy-load vehicles (trucks and buses),
as part of an ongoing academic-industrial coopera-
tion. In this case study, we involve five different stake-
holders of the system development process, as case-
study subjects (CSS), who use the specification pat-
terns and the existing tool called PROPAS (Filipovikj
et al., 2018) to specify a predefined set of require-
ments of an operational automotive system. For the
purpose of our study, we collected both quantitative
and qualitative data. The quantitative data consists
of: i) specification correctness, which is measured
in terms of how many of the requirements are cor-
rectly specified using the specification patterns and
the PROPAS tool, and ii) the required time for spec-
ification. The qualitative data on the other hand, is
collected via a survey of ten statements, which can be
answered using five predefined options.
The paper continues as follows. In Section 2 we
present an overview of the specification patterns (Sec-
tion 2.1) and the PROPAS tool (Section 2.2). Next,
in Section 3 we provide details on the case study re-
search method, followed by Section 4 in which we
show the details about the setup of our case study. In
Section 5 we present the results of the evaluation and
the threats to validity, followed by the discussion of
the results in Section 6. We compare to related work
in Section 7, and in Section 8 we outline the conclu-
sions and directions for future work.
2 SPECIFICATION PATTERNS
AND PROPAS TOOL
In this section, we give an overview of the specifica-
tion patterns (Section 2.1) and the PROPAS tool (Sec-
tion 2.2).
2.1 Specification Patterns
The specification Patterns System (Dwyer et al.,
1998) is an approach proposed to facilitate the for-
mal specification of system properties for practition-
ers who are not experts in formal methods. It is based
on the assumption that the systems’ specifications are
framed within reoccurring solutions, from which a set
of patterns can be extracted and saved for future reuse.
Each pattern is characterized by a behavior that it cap-
tures, and an extent of the program execution, called
scope, within which the behavior must hold. The pat-
terns are expressed as a combination of literal and
non-literal terminals. The non-literal terminals can be
either Boolean expressions that describe system prop-
erties, or integer values that capture timing aspects.
The rest of the pattern is made of literal terminals,
which represent the fixed part of the pattern and can-
not be changed.
The original SPS catalog proposed by Dwyer
et al. (Dwyer et al., 1998) is compiled by analyz-
ing more than 500 examples of property specifica-
tions from various domains. It contains 13 quali-
tative patterns, which for easier navigation are di-
vided into two categories: order and occurrence, ex-
pressed in various types of temporal logic, including
but not limited to (Timed) Computation Tree Logic
((T)CTL) (Alur et al., 1993), Real-time Interval Logic
(RTGIL) (Moser et al., 1997), Metrics Temporal Log-
ics (MTL) (Alur and Henzinger, 1993), etc. The pat-
terns in the later category describe the occurrence of
a given event in the system, while the patterns from
the ordering category are used to capture the rela-
tive ordering of the occurrence of multiple events dur-
ing system execution. In order to be able to specify
real-time systems properties, Konrad and Chen (Kon-
rad and Cheng, 2005) have subsequently extended the
specification patterns catalog with a new category of
patterns called real-time. Consequently, the new and
extended catalog was named Real-time Specification
Pattern System (RTSPS). Besides the introduction of
the new set of patterns, in the same work, Konrad and
Specifying Industrial System Requirements using Specification Patterns: A Case Study of Evaluation with Practitioners
93
Cheng proposed an additional encoding of the specifi-
cation patterns in controlled natural language (CNL).
The intended purpose of the CNL representation is
to enable various stakeholders who are not trained
in formal methods to read and interpret the system
specification expressed via patterns. For instance, a
bounded-response pattern over events P and S with
global scope expressed in CNL, reads as follows:
Globally, it is always the case P is followed by S
within T seconds.
Based on the formal definitions for both the
scope (Globally) and the behavior (bounded-
response) (Dwyer et al., 1998; Konrad and Cheng,
2005), one can automatically generate the following
TCTL property specified using the CNL pattern
above:
AG(P AF
T
S)
The introduction of the real-time patterns did not
yield the catalog of patterns complete. In order to sat-
isfy the different purposes, the catalog has since been
extended in many directions. The most notable ones
include the introduction of probabilistic patterns by
Grunske (Grunske, 2008), which provide means for
one to specify probabilistic system properties. The
most comprehensive catalog compiled to date is by
Autili et al. (Autili et al., 2015). In our work, we
use the following subset of real-time specification pat-
terns:
P1: Globally, Universally: Globally, it is always the case
that P holds.
P2: Timed Globally, Universally: Globally, it is always the
case that if P held for T time units, then S holds.
P3: Globally, Real-time Response: Globally, it is always
the case that if P holds, the S eventually holds within T
time units.
P4: After Q Until R Universally P: After Q, it is always the
case case that P holds until R holds.
P5: Timed After Q Timed Until R Universally P: After Q
holds for T time units, it is always the case that P holds
until R holds or at most T
1
time units.
2.2 ProPaS Tool
The PROPAS tool (Filipovikj et al., 2018) is an open-
source tool that is used for the formal system specifi-
cation using specification patterns. In the following,
we overview in details the user interface (UI), as well
as other non-UI-related features that have been intro-
duced to support our experiment.
The UI of the PROPAS tool is given in Figure 1,
and it represents an improvement of an earlier tool
called SeSAMM Specifier (Filipovikj et al., 2016). In
the top left part of the UI we have the list of all the
Figure 1: ProPaS Tool User Interface.
system specifications (denoted with 1). The list can
be manipulated via commands for deleting an exist-
ing specification or creating a new one, which are pro-
vided by a context menu that is available on right click
of the list. Next to it, the list of properties in the cur-
rently active system specification are displayed. Sim-
ilarly as for the list of system specifications, one can
manipulate the list of properties (2) using the context
menu that is displayed upon right-clicking the list.
The currently available actions over the list of proper-
ties include removing an existing property or adding
a new one.
The middle part of the UI (3) contains details
about the currently active system property. The
details are composed of editable and read-only at-
tributes. The first editable attribute for the system
property is its name, which is also used as an iden-
tifier. In the particular example illustrated in Figure 1,
the name of the currently active property is “req1”.
Next comes the first read-only attribute, called de-
scription, which is defined by the selected pattern.
The specification pattern for the property is applied
from the drop-down list, which contains all the avail-
able patterns that the tool has access to. Below the list
of available specification patterns, a list of proposi-
tions (4) characteristic for the specific pattern is given.
In the bottom part of the UI (5) there are two elements
that serve the purpose of giving feedback to the user
on the specified behavior. First, there is a text field
in which the combination of the property and the pat-
tern specific attributes are combined to automatically
ENASE 2019 - 14th International Conference on Evaluation of Novel Approaches to Software Engineering
94
generate the different views for the given property. In
this case, the specification pattern is encoded in con-
strained natural language and TCTL. Lastly, the UI
shows a visual representation (6) of the ordering of
the events as specified by the selected pattern, which
in the example given in Figure 1 is the ordering of the
events according to the bounded-response pattern.
Beside the UI, the PROPAS tool provides other
features, out of which some are relevant and some
are not for the case study. Out of the study-relevant
features, the most important one is the timer feature,
which is used to keep track of the amount of time
that is spent on specifying each property of the set.
Out of the non-relevant features worth mentioning is
the consistency analysis feature, which has been pro-
posed and implemented in the previous version of the
tool. For more details on the consistency analysis,
we refer the reader to earlier work (Filipovikj et al.,
2018).
3 RESEARCH METHOD
In this section, we present the case-study research
method, which is used in the evaluation of the speci-
fication patterns and the PROPAS tool.
A case study is a research method in software en-
gineering research that aims at studying a contempo-
rary phenomena in their natural context (Wohlin et al.,
2012). Initially the term case study was used to de-
note a demonstration case, during which an imple-
mentation of some tool or other type of concept is pre-
sented. Subsequently, it got a broader meaning and is
also used to refer to self-experienced or self-reported
investigation.
The case-study research method is mostly used to
provide deeper understanding of the phenomena un-
der study. According to Runeson and H
¨
ost (Rune-
son and H
¨
ost, 2009), a case-study research method
is “well suited for many kinds of software engineer-
ing research, as the objects of study are contempo-
rary phenomena, which are hard to study in isola-
tion”. In general, the case-study research method
represents a more liberal way of studying some phe-
nomena related to a single project. It is easy to plan
and execute, and as such, it is useful for evaluating
new methodologies and tools in industrial settings.
On the other side, compared to other empirical re-
search methods, such as for instance the controlled
experiment (Wohlin et al., 2012), the results obtained
from a case study are generally more difficult to in-
terpret and generalize. Despite the limitations, if a
well-established case-study research methodology is
followed, then the obtained results are valuable.
Relevant literature (Runeson and H
¨
ost, 2009) sug-
gests that any well-designed case study should fol-
low at least the following ve process steps: i) case
study panning and design, ii) data collection prepara-
tion, iii) data collection, iv) data analysis v) report-
ing. In order to better plan and execute the listed
steps of a case study, there are several papers that
provide comprehensive check lists (Kitchenham et al.,
1995; Runeson and H
¨
ost, 2009), which contain series
of questions intended to guide the researchers to cor-
rectly plan and execute the research process.
During the planning and design phase, the most
important thing is to correctly determine the objec-
tive of the study. In software engineering, in gen-
eral there are four predominant types of case-study re-
search: exploratory, descriptive, explanatory and im-
proving (Robson and McCartan, 2016). Basically, the
type of the case-study research determines the charac-
teristics of the subsequent steps. For instance, if there
is a lack of empirical data on the subject, one resorts
to an exploratory case study where the main objec-
tive is to provide information on what is happening,
and generate ideas and hypothesis for new research.
Another important aspect to consider during the plan-
ning and design phase is to account for the trade-off
between the level of control over the experiment and
the degree of realism. Given that the primary focus
of a case-study research process is to study some phe-
nomena in their natural context, it is expected to lose
some control in favour of a high degree of realism.
In a case-study research, there are two types of
variables: independent and dependent. Independent
variables are those variables that can be controlled
and changed during the experiment. They must have
some effect on the dependent variables. The depen-
dent variable(s) a.k.a. response variables are the vari-
ables that cannot be controlled in the experiment, and
they are usually measured based on the independent
variables.
The data collected in a case study is either quali-
tative or quantitative. Quantitative data is represented
with numbers, whereas the qualitative data is usually
expressed in words, descriptions, diagrams, etc. Due
to its descriptive nature, the qualitative data is usu-
ally preferred in case-study research, however, a com-
bination of qualitative and quantitative data provides
better understanding of the phenomena under study.
There are three well-established methods for data col-
lection in case study, namely: direct, indirect and in-
dependent (Runeson and H
¨
ost, 2009). Regardless of
which method is collected for the study, it is recom-
mended that multiple data sources are considered in
order to reduce the effects of the interpretation bias.
When it comes to the data analysis, different tech-
Specifying Industrial System Requirements using Specification Patterns: A Case Study of Evaluation with Practitioners
95
Table 1: Case study subjects and their roles.
CSS Role
CSS
1
Software architect for autonomous systems
CSS
2
Development engineer (former quality assurance engineer)
CSS
3
Development engineer/researcher
CSS
4
Requirements Coach
CSS
5
Information/safety architect
niques apply for qualitative and quantitative data. The
accepted analysis methods for quantitative data in-
clude: descriptive statistics, correlation analysis, de-
velopment of predictive models and hypothesis test-
ing, whereas the qualitative data are usually analyzed
using categorization or sorting.
Finally, the report of the case study serves the pur-
pose of communicating the case-study results. It is
often the case that the report cannot be distinguished
from the case study itself, and as such it is the main
source for judging the overall quality of the study. For
communicating results from a case study to a research
community the preferred forms of reporting include
conference or journal articles or in some cases even
technical reports.
4 CASE STUDY PLANNING AND
EXECUTION
In this section, we present the details about the plan-
ning and execution of our case study, in which we
evaluate the PROPAS tool and the specification pat-
terns based on their application by the industrial prac-
titioners.
4.1 Case Study Design
There is a lack of empirical data on whether the spec-
ification patterns can be effectively and efficiently
used by the engineers and other stakeholders for spec-
ifying their systems. Given this, the objective of our
case study is to be exploratory, that is, to provide an
initial set of empirical data that can be used later to
form hypotheses and execute larger case studies in
various companies and domains. For planning and
executing the case study, we are mainly guided by the
checklists for case-study planning and execution by
Kitchenham et al. (Kitchenham et al., 1995) and Run-
neson and H
¨
ost (Runeson and H
¨
ost, 2009).
For the case study, we consider an operational sys-
tem, namely the Fuel Level Display (FLD), which is
installed in all heavy load-vehicles produced by Sca-
nia, Sweden. The core of the system is a software
module, which is responsible for: i) reading the value
of the sensor installed inside the fuel tank, ii) calculat-
ing the estimated total available fuel left in the tank,
and iii) showing the estimated value on the dashboard
to the driver. The specification of the FLD system
consists of twenty four requirements. As the intended
functionality of the system is not of a primary inter-
est for this case study, we omit further details. For
a detailed system description, we refer the reader to
the paper by Westman and Nyberg (Westman and Ny-
berg, 2014).
The main objective of this study is to collect data
that can be used to form a future hypothesis regard-
ing the practical usefulness of the specification pat-
terns and the PROPAS tool for specifying require-
ments in the automotive domain. The data gathering
phase of the case study has been executed at the com-
pany premises, involving five engineering profession-
als from Scania as case-study subjects, each having
a different role in the systems development process.
The roles for each of the subjects are given in Table 1.
For the purpose our our study, the subjects have been
asked to specify the following subset of FLD require-
ments using a predefined set of specification patterns
(see Section 2.1) and the PROPAS tool:
R
1
If actualParkingBrake is false, then the indicated-
FuelVolume showed by the fuel gauge is less or
equal to actualFuelVolume.
R
2
If CAN1 is equal to a CAN message rmsg, then
rmsg is in HWreceiveBuffer and Rcan decodeCan
is set to true.
R
3
The position of the floater sensedFuelLevel,
sensed by the fuel sensor, does not deviate more
than 10% from actualFuelVolume in the fuel tank.
R
4
If CAN1 is equal to a CAN message rmsg, then
rmsg is in FiFoBuffer within 20ms.
R
5
If the DMA channels timerCh and rfifoCh are en-
abled for approximately 20ms, then a RAW value
of VFuel is available in ADC RFIFO.
R
6
If actualParkingBrake is false and CAN1 is equal
to CAN message FuelEconomy and it has not
passed more than 300ms since the last time CAN1
was equal to FuelEconomy, then the CAN signal
FuelRate in FuelEconomy does not deviate more
than 1% from the derivative of actualFuelVolume.
The given requirements represent a diverse subset
from the FLD functional requirements. The require-
ments capture both the untimed (R
1
, R
2
and R
3
) and
timed (R
4
, R
5
, R
6
) system behavior.
The case study is executed in three major phases.
In the first phase, we compiled a list of potential case-
study subjects and contacted them personally in order
to obtain their consent for participation in the study.
ENASE 2019 - 14th International Conference on Evaluation of Novel Approaches to Software Engineering
96
Upon our request all of the potential case-study sub-
jects agreed to participate. Since none of the sub-
jects has previous knowledge or experience in using
the specification patterns and the PROPAS tool, we
prepared and distributed to each of them a short tu-
torial. The tutorial, in a form of a written document,
contains a compact description about the general idea
of specification patterns, the set of requirements in-
tended for the case study and a set of step-by-step
instructions (complemented by screen shots of each
step) on how to use the PROPAS tool. During the sec-
ond phase of the case study, which is the data collec-
tion phase, one of the researchers met with each of the
subject in person in order to carry out the given task
of specifying the predefined set of requirements us-
ing the PROPAS tool. The last phase of the case study
included data analysis, drawing conclusions and pro-
ducing the case-study report.
As independent variables in our case study we
identify the following: i) the system (FLD) and the
subset of the functional requirements used for speci-
fication, ii) the PROPAS tool and the set of patterns,
and iii) the case-study subjects. The set of dependent
variables consists of the following: i) correctness of
the specification, ii) required time for specification,
and iii) level of domain expertise of the case-study
subjects.
4.2 Data Collection Preparation
The quantitative and qualitative data in this case study
is collected from more than one source. For collecting
the qualitative data, we use a direct method facilitated
by the “save to disk” feature of the PROPAS tool,
which enables the subjects to save their system speci-
fications on a computer hard drive. The data produced
by the PROPAS tool includes a pattern-based spec-
ification of the predefined set of requirements aug-
mented with information on how much time was spent
for specifying each individual requirement.
For qualitative data, we use a survey composed of
the following ten statements:
S
1
I can specify the given set of requirements using
the specification patterns (SPS).
S
2
The SPS formalized patterns expressed in con-
strained natural language are comprehensible.
S
3
The visual representation of the pattern is helpful
in understanding the behavior represented by the
selected pattern.
S
4
The PROPAS tool is intuitive and easy to use.
S
5
The PROPAS tool is practically useful for specify-
ing requirements using SPS.
S
6
The quality of generated specifications is high.
S
7
I feel confident using the PROPAS tool.
S
8
I am able to use the tool without any additional
support.
S
9
The tool is helping in decreasing the specification
effort.
S
10
I can easily select the appropriate pattern for a
given requirement.
Each of the above statements is intended to cap-
ture the subjects’ opinion in terms of degree of satis-
faction, via the following set of predefined options: i)
“I strongly agree”, ii) “I agree”, iii) “I neither agree
or disagree”, iv) “I disagree” and v) “I strongly dis-
agree”. For collecting the qualitative data, we use
the online Google Survey. Some of the main advan-
tages of using this tool for collecting the qualitative
data are: fast and straightforward survey generation,
distributed access to the questions, real-time access to
the data once entered by the case-study subjects and
the convenient visual representations of the results of-
fered by the tool.
4.3 Data Collection
The data collection was performed according to the
data collection plan described in Section 4.2. The
qualitative data was collected in real-time during the
study execution through the PROPAS tool, at Scania
premises. To avoid technical problems, such as soft-
ware compatibility that includes the operating sys-
tem and the run-time environment required by the
PROPAS tool, the tool was pre-installed on a laptop
provided by the researchers. Then, the machine with
the PROPAS tool was used by each of the case-study
subjects to perform the system specification and to
fill in the survey. During the data collection phase,
only the current case-study subject and one of the re-
searchers involved in the case study were present in
the room. The subjects were expected to use the tool
on their own, based on the information from the pre-
viously provided tutorial. However, during the spec-
ification process, the subjects were allowed to inter-
act with the researcher/expert in order to improve the
accuracy of their results and increase the confidence
in their work. Each interaction initiated by the case-
study subject was recorded by the researcher, as part
of the qualitative data that characterizes the practical
usefulness of the method and the tool. In general, the
less the interaction between the subject and the expert,
the higher the understanding of the subject regarding
the method and the tool.
After the subjects declared that they were done
with the specification of the predefined set of require-
Specifying Industrial System Requirements using Specification Patterns: A Case Study of Evaluation with Practitioners
97
ments, they were asked to fill in the survey with
qualitative data. In addition to the survey, we col-
lected the reflections and the comments of the sub-
jects expressed while performing the system specifi-
cation task using PROPAS.
After both the system specification and the sur-
vey were completed, the data collection phase was
concluded for that particular subject. At the end of
the data collection phase, the following data was col-
lected: ve system requirements specifications con-
taining the predefined set of requirements from the
FLD system complemented with the amount of time
spent per requirement, and five survey responses to
our survey for the qualitative data.
4.4 Data Interpretation and Analysis
The third, and the last phase of the case study is the
data interpretation and analysis. The gathered quanti-
tative data is analyzed using statistical methods. In
order to determine the correctness of the specifica-
tion, that is, to determine whether a correct specifi-
cation pattern is used for a given requirement, we use
a baseline system specification of the same set of re-
quirements, which was created by a formal methods
expert.
The gathered qualitative data on the other hand,
is interpreted in a less formal way, mainly through a
discussion of the obtained survey data. Given that the
primary purpose of the qualitative data is to provide
us with a subjective reflection of the case-study sub-
jects on the whole process, the discussion is mainly
intended to draw conclusions and extract suggestions
that can be useful for setting up future similar case
studies, but also for the researchers in order to further
improve the PROPAS tool or any other tool designed
for the purpose of specifying (formal) requirements.
5 RESULTS
In this section we show the results obtained from the
case study. In Section 5.1 we present the quantitative
data, followed by the qualitative data results in Sec-
tion 5.2, after which we discuss the potential threats
to validity of our results in Section 5.3.
5.1 Quantitative Data Analysis
The collected quantitative data in the case study, rep-
resented through the correctness of the specifications
produced using the specification patterns and the time
for specification are given in Tables 2 and 3, respec-
tively.
Table 2: Pattern-based requirements specification correct-
ness.
CSS R
1
R
2
R
3
R
4
R
5
R
6
total
CSS
1
X X X X X X 100%
CSS
2
× × X X X X 66.67%
CSS
3
X X X X X X 100%
CSS
4
X X X X X X 100%
CSS
5
X X X X X × 83.33%
total 80% 80% 100% 100% 100% 80% 90%
Table 3: Requirements specification time in minutes.
CSS R
1
R
2
R
3
R
4
R
5
R
6
total
CSS
1
3 2 2 2 4 13 26
CSS
2
3 6 2 2 1 16 30
CSS
3
2 2 2 2 2 16 26
CSS
4
3 2 2 2 2 18 29
CSS
5
2 3 2 2 2 10 21
avg. 2.16 2.50 1.66 1.66 1.83 12.16 26.4
The correctness of the produced system specifica-
tions given in Table 2, shows the data both per case-
study subject and the individual requirements, respec-
tively. In the same table, we denote by the symbol (X)
that a given subject has specified a particular require-
ment correctly, meaning that the appropriate patterns
for specification has been selected, and the result is
equivalent with the baseline specification. The oppo-
site case, that is, when an incorrect pattern is used is
denoted by the symbol (×). By analyzing the specifi-
cation correctness per subject, we observe the follow-
ing: three of the subjects achieve 100% correctness,
that is, for each of the six requirements a correct spec-
ification pattern is used. One of the subjects selected
an incorrect pattern for requirements R
1
and R
2
, thus
achieving (66.67%) correctness of the specification.
Lastly, one subject used an incorrect pattern for one
requirement, namely R
6
, resulting in 83.33% correct-
ness of the produced specification.
If we analyze the collected data per requirement,
we observe a single mistake (80% correctness) in the
specification of the requirements R
1
, R
2
and R
6
. The
rest of the requirements are specified correctly (100%
correctness) by all of the case-study subjects. If we
consider the total number of instances of the speci-
fied requirements, which is thirty and compare them
to only three mistakes, the overall conclusion is that
we have 90% correctness for the specification of all
requirements by all subjects.
The second aspect of the collected quantitative
data is the specification time, which is given in Ta-
ble 3. The table shows the specification time in min-
utes. Similarly as for the correctness of the specifica-
tion, the results for measured specification time can be
analyzed both per subject and per requirement. Ana-
lyzing the specification time per subject, shows that
ENASE 2019 - 14th International Conference on Evaluation of Novel Approaches to Software Engineering
98
the shortest time for specifying the complete set of
requirements is 21 minutes, whereas the slowest one
is 30. The average specification time for the set of
the requirements per subject is 26.4 minutes. The av-
erage time per requirement is 4.4 minutes, or more
accurately around 2 minutes for R
1
to R
5
, whereas for
R
6
is around 12 minutes, which accounts for almost
half of the time spent on the entire specification.
5.2 Qualitative Data Analysis
The qualitative data in the case study is collected in
two ways: through a survey and by taking notes of
comments by the case-study subjects during the data
collection process. While the data collected from the
survey is based on answers to a predefined set of state-
ments, the notes taken during the data collection are
based on comments by the subjects and as such are in-
tended to replace the free-text question in the survey.
The intended purpose of the qualitative data collec-
tion in this case study is to give us a more clear picture
on how the subjects perceive the system specification
using the specification patterns and the PROPAS tool,
and to point to some directions for future work with
respect to both the method and the tool, such that their
practical usefulness is improved.
Based on the responses of the subjects, we have
the following quantitative data:
S
1
20% strongly agree, 60% agree, 20% neither
agree or disagree
S
2
60% strongly agree, 40% agree
S
3
20% strongly agree, 20% agree, 60% neither
agree or disagree
S
4
20% strongly agree, 80% agree
S
5
40% strongly agree, 40% agree, 20% neither
agree or disagree
S
6
60% strongly agree, 40% neither agree or disagree
S
7
20% strongly agree, 40% agree, 20% neither
agree or disagree, 20% disagree
S
8
40% strongly agree, 40% agree, 20% neither
agree or disagree
S
9
40% strongly agree, 60% agree
S
10
20% strongly agree, 20% agree, 60% neither
agree or disagree
The obtained qualitative data can be interpreted in
various ways, however, it is clear that the case-study
subjects are positively inclined towards the specifica-
tion patterns and the PROPAS tool, which can be con-
cluded based on the fact that majority of the state-
ments are described using positive to neutral phrases.
The only negative description can be observed for the
statement S
7
, where 20% of the subjects express that
they do not feel confident in using the PROPAS tool.
The qualitative data obtained through the survey
is additionally complemented with unrestricted feed-
back provided by the case study subjects while they
are performing the specification using the PROPAS
tool. Since the case-study subjects were not explic-
itly asked to provide this kind of feedback, CSS
3
and
CSS
5
completed the specification task without giv-
ing any feedback. The rest of the subjects provided
their opinion either during the specification process
or immediately after the specification phase was con-
cluded. The provided feedback is not extensive, but it
is very important, as the case-study subjects were free
to elaborate on the aspects of both the method and the
tool that were not reflected by the survey. From the
obtained feedback, we point out the following: CSS
1
was positive during the specification of the require-
ments. The subject pointed out that the method is
useful, however, the subject also added some sugges-
tions for improving the tool. Concretely, it has been
pointed out that in order for a tool such as PROPAS
to be considered for adoption in in the industrial de-
velopment process of automotive systems, it should
also provide requirements management features such
as versioning, version comparison, etc. The reflection
of CSS
2
is very interesting as the subject has phrased
his experience with the specification patterns and the
PROPAS tool as follows: “I can write requirements
in natural language as free-text much faster, but the
pattern-based requirements specified via the PROPAS
tool are much more concise”. Lastly, similar to CSS
1
and CSS
2
, CSS
4
also had a positive incline towards
the method and the tool, however, the subject pointed
out that this version of the tool must be improved in
order to be considered for industrial adoption. Espe-
cially, it has been pointed out that the visual feedback
mechanism should be coupled with the specified re-
quirement and not with the pattern as it is now in the
tool. Similar comment has been given by CSS
1
.
5.3 Threats to Validity
In this section we discuss the threats to validity of the
results obtained in this case study. First, we discuss
the internal threats to validity and then the external
ones.
Internal Threats to Validity are often defined as
influences that can affect the independent variables
with respect to causality without the knowledge of
the researchers. Based on this definition, we iden-
tify the following internal threats to validity for our
case study: i) case-study subjects selection, and ii)
the number of requirements used in the case study.
Specifying Industrial System Requirements using Specification Patterns: A Case Study of Evaluation with Practitioners
99
The selection of the case study subjects is done in
such a way as to satisfy a number of criteria, includ-
ing the following: i) have at least one representative
of each category of stakeholders involved in the sys-
tem development process, and ii) have subjects with
various levels of experience and expertise. The case
study subjects and their roles are given in Table 1.
Given that there are number of Scania employees
satisfy the above criteria, we additionally took into
consideration the following factors, which are not rel-
evant for the validity of the results: the availability of
the subjects at a specific time and their willingness to
participate in the case study.
Due to the limited time that the case-study sub-
jects were assigned for the participation in the case
study, we were required to isolate a small subset of all
the possible requirements. For that purpose, we com-
piled a set of requirements that are feasible to be spec-
ified within the given time frame of around one hour.
Regarding the selection of the system and the require-
ments to be used in the case study, we had to take into
consideration several factors such as: complexity of
the system and its specification, and the possibility to
make the requirements public.
External Threats to Validity represent the condi-
tions that limit the ability to generalize the results
from the case study over the whole domain. In our
case study, we consider the following two factors as
external threats to validity: i) the selection of the par-
ticular system and the particular set of requirements
used in the study, and ii) the relative domain expertise
of the case study subject compared to their peers from
the same company or from the companies within the
automotive domain.
The system used in the case study and the rather
small set of requirements hinder us from the general-
izing the obtained results, however, the obtained re-
sults serve well the purpose of an exploratory case
study aimed at gathering empirical data. There are
several concerns that arise from the system and the
mentioned selections, such as: will the results be sim-
ilar if a different system would be used? Will results
be the same if more requirements of the same sys-
tem were used or more requirements but from differ-
ent system(s)? In order to answer these questions, a
larger case study that includes more systems and case
study subjects is required.
The second external threat to the validity of the re-
sults is the relative expertise of the case study subjects
compared to their peers, be they from the same com-
pany or from a different one. We have no means nor a
set of well-defined parameters on how to measure the
expertise of the case-study subjects. Consequently, in
order to be able to pose and test hypotheses for the
whole domain, one needs to include more case study
subjects preferably from different companies in the
same domain, or at least to find an objective metrics
to quantify the expertise of the case study subjects.
6 DISCUSSION
In this section, we discuss both the quantitative and
the qualitative data obtained from the case study.
Based on the discussion, we draw some conclusions
which can be useful in defining future hypothesis.
The collected quantitative data shows that on av-
erage the industrial practitioners achieve 90% correct-
ness of the specification, while spending 4.4 minutes
on average per requirement. However, a more in-
depth analysis of the results shows some surprising
trends. The selected group of requirements are not
of equal complexity. For instance, the simplest re-
quirements specify instant causality (such as in R
3
),
whereas the most complex one (R
6
) specifies com-
plex behavior captured by occurrence of two events.
Intuitively, one would expect that most of the mis-
takes in the specification to occur in specifying the
most complex requirements, nevertheless, our data
does not provide evidence to support this expectation.
If we refer back to Table 2, we can see that three mis-
takes made in the specification have been connected
to different requirements, with only one being for R
6
.
When it comes to required time for specification, the
data given in Table 3 shows that the time required for
specifying the requirements is proportional to their
complexity. The average time required for specifying
R
6
is 12.1 minutes, which is almost half of the total
average time for the whole specification.
Some of the subjects employed interesting strate-
gies/heuristics when it comes to selecting the appro-
priate pattern and identifying the propositions for a
given requirement. For instance, CSS
2
selected the
wrong pattern, that is, P3 instead of P1 for both R
1
and
R
2
. Given the fact that P3 by definition incorporates
three propositions, compared to only one in P1, the
subject introduced new data that does not exist in the
original requirement (the maximum response time) in
order to fill in the missing propositions. CSS
4
also
employed an interesting heuristic for selecting a pat-
tern for a specific requirement, as follows: first, when
reading the requirement, the subject tried to determine
whether the requirement incorporates timing informa-
tion or not. Given that in the case study only one of
the pattern is untimed (does not encode time explic-
itly), for requirements that do not incorporate time the
subject selected P1 without much hesitation. For the
rest of the requirements, which are timed, the subject
ENASE 2019 - 14th International Conference on Evaluation of Novel Approaches to Software Engineering
100
analyzed the structure of the pattern by propositions
that model events and time and tried to match it with
the information encoded in the given requirement.
The analysis of the qualitative data shows that
the industrial practitioners are generally favourable
for the specification patterns. This is supported by
the fact that they have expressed positive opinions
about the method and the quality of the generated sys-
tem specifications. However, based on the qualitative
feedback, some issues, primarily related to the tool
support must be address in order for the pattern-based
specification to have more significant penetration in
industrial settings.
Overall, the specification patterns and the
PROPAS tool seem to be a promising combination for
a pattern-based specification of requirements in in-
dustrial settings. The method and the tool in general
are easy to use and the quality of the specifications is
perceived as good by the industrial practitioners. The
qualitative data also suggests that the combination of
specification patterns and PROPAS tool seems to re-
duce the specification effort for the engineers. On the
other hand, the main challenge for the pattern-based
specification of requirements for engineers who are
not skilled in formal methods is the validation of the
expressed behavior. This is observed by the fact that
the subjects express neutral and in some cases nega-
tive opinion (S
6
and S
7
) on the quality of the generated
specifications and the confidence in what they are do-
ing.
7 RELATED WORK
In the literature, there exists a body of works that used
the specification patterns for capturing industrial re-
quirements from various domains. In the following,
we list some of the recent attempts.
To the best of the authors’ knowledge, there are
only two reported cases in the literature that are fo-
cused on accessing the suitability of the specification
patterns for specifying requirements in the automo-
tive domain. The first case study was reported by
Post et al. (Post et al., 2012). In their work, the au-
thors conduct a case study at BOSCH, one of the lead-
ing suppliers in automotive domain, in which they
aim to study whether the specification patterns are
expressive enough to formalize requirements in the
automotive domain. The results from the case study
suggest that the specification patterns are expressive
enough for capturing most of the requirements used
in the study. Moreover, they also observed that the
small subset of patterns are enough to formalize most
of the requirements. Since the results were obtained
based on a rather limited set of requirements and in-
volved only one company, it is hard to generalize
the results to the entire domain. In order to further
strengthen or disprove the claims of Post et al., a fol-
low up case study was carried out at Scania by Fil-
ipovikj et al. (Filipovikj et al., 2014). The case study
involved one hundred requirements from four differ-
ent systems. The results of the study are aligned with
the ones of Post et al., and as such represent one step
close towards being able to generalize for the whole
domain. Compared to our current study, both of the
previous attempts were completely oriented towards
accessing the expressiveness of the specification pat-
terns, and as such were performed by formal meth-
ods experts and not actual engineers. Also, neither of
the studies involved a specialized tool support since
the experts where comfortable with using general pur-
pose text editors or pen and paper.
In order to support potential users of the spec-
ification patterns, be they from academia or indus-
try, number of research tools of various maturity
have been proposed. The Property Specifier tool
(Prospec) (Mondragon and Gates, 2004) is one of the
first tools proposed to facilitate the usage of specifica-
tion patterns for users not trained in formal methods.
It provides a complex interface that supports many
degrees of freedom when specifying complex prop-
erties. Another similar tool is the Property Elucidator
(PROPEL) tool (Smith et al., 2002). Both tools rep-
resent the pattern-expressed requirements as a finite
state machines, however, none of them has been eval-
uated by industrial practitioners, thus no data on their
practical usefulness is available. Another tool for as-
sisting non-experts in formal methods is the Property
Assistant (PASS) proposed by Remenska et al. (Re-
menska et al., 2014). In comparison to the afore-
mentioned tools, PASS uses UML constructs to show
the specified behavior captured by the patterns. The
tool also provides a multi-feature interface that can
be used to specify complex system properties, how-
ever, similar to the previous two tools, it has not been
evaluated by industrial practitioners. Lastly, the PSP-
Wizard tool (Autili et al., 2015) represents the latest
attempt to produce a specification-pattern-based tool.
Compared to the previous tools, the PSPWizard in-
corporates the most comprehensive catalog of specifi-
cation patterns and as such can potentially be used to
specify the widest pallet of different properties. The
PSPWizard tool and the pattern-based specification
framework proposed in (Autili et al., 2015) have suc-
cessfully been applied on an industrial use case, how-
ever, the assessment was performed by researchers
(presumably experts in formal methods) and not the
actual practitioners (engineers).
Specifying Industrial System Requirements using Specification Patterns: A Case Study of Evaluation with Practitioners
101
Compared to all of the aforementioned reports, in
this case study we specifically aim at accessing the
practical usefulness of the specification patterns via
the PROPAS tool in industrial settings. As such, the
purpose of our exploratory case study is to generate
an initial set of empirical data about the potential us-
ability of the specification patterns in industrial set-
tings, which should serve a good starting point for
conducting bigger case studies of similar type in order
to empirically access the usefuleness of the specifica-
tion patterns.
8 CONCLUSIONS
In this paper, we presented an exploratory case study
through which we gather empirical data for the prac-
tical usefulness of the specification patterns via the
PROPAS tool for specifying system requirements in
industrial settings, more precisely in the automotive
domain.
The reported case study was performed in collab-
oration with Scania, one of the world’s leading manu-
facturer of heavy-load vehicles, including trucks and
buses. As case-study subjects, we used ve stakehold-
ers from the embedded software development pro-
cess at Scania. The case-study subjects belong to the
following roles: two development engineers, one re-
quirements coach, one safety architect and one sys-
tems architect. The case study was executed in three
major phases: preparation, data gathering and analy-
sis, and reporting. In the first phase, the subjects were
provided with a short tutorial on the specification pat-
terns and the PROPAS tool. In the second phase, we
visited the subjects at the company premises where
they performed the specification of the requirements
using the PROPAS tool and answered a survey, such
that both quantitative and qualitative data is collected.
For the quantitative data we have considered the fol-
lowing: correctness of the specification and time for
specification, whereas for collecting the qualitative
data we used a survey of ten statements, which was
completed by the case-study subjects immediately af-
ter the specification of the requirements.
The analysis of the quantitative data shows the fol-
lowing: the overall specification correctness is 90%,
meaning that only three out of thirty requirements
specification instances were incorrectly specified by
the subjects. The correctness results do not show a
direct causal relationship between the complexity of
the requirements from the system specification and
the correctness of the same. The average time for
formalizing the predefined set of requirements is 26.4
minutes, or 4.4 minutes per requirement. If we look at
the specification time per individual requirement (see
Table 3), it is clear that the complexity of the require-
ment has a direct effect on the specification time. For
instance, the average specification time for the most
complex requirement (R
6
) is 12.16 minutes, which
is very close to the time spend for the specification
of the rest of the five requirements combined. The
qualitative data, which was obtained via survey (see
Section 4.1) shows that the engineers are positively
inclined towards system specification via specifica-
tion patterns supported by an adequate tool support,
such as the PROPAS tool. However, the same data re-
veals that in order to have more penetration and real
industrial impact, the specialized tool support must
be significantly improved, especially in the following
aspects: provide a validation mechanism that should
help non-experts in formal methods to validate the
specified behavior, as well as requirements manage-
ment features such as traceability, versioning, etc. De-
spite the identified drawbacks, our results show that
the specification patterns that are supported by an ad-
equate tooling can be practically useful for formaliz-
ing requirements by engineers and other stakeholders
who are not experts in formal methods.
Based on the results from the case study, we en-
vision the following directions for future research.
First, we aim to define a set of hypotheses to be tested
in future case studies, which should involve more sys-
tems, subjects and possibly include other companies,
either from the same or different domain. The second
direction for future work is to further improve the spe-
cialized tool support, meaning to enrich the PROPAS
tool with new features that address the recommenda-
tions and suggestions extracted from the qualitative
data. This direction involves both engineering (tool
implementation) and research efforts (finding a suit-
able method for validation of specified behavior).
ACKNOWLEDGEMENTS
This work has been funded by the Swedish Govern-
mental Agency for Innovation Systems (VINNOVA)
under the VeriSpec project 2013-01299.
REFERENCES
Alur, R., Courcoubetis, C., and Dill, D. (1993). Model-
checking in dense real-time. Information and compu-
tation, 104(1):2–34.
Alur, R. and Henzinger, T. A. (1993). Real-time log-
ics: Complexity and expressiveness. Information and
Computation, 104(1):35–77.
ENASE 2019 - 14th International Conference on Evaluation of Novel Approaches to Software Engineering
102
Autili, M., Grunske, L., Lumpe, M., Pelliccione, P., and
Tang, A. (2015). Aligning qualitative, real-time, and
probabilistic property specification patterns using a
structured english grammar. IEEE Transactions on
Software Engineering, 41(7):620–638.
Barnat, J., Bauch, P., Bene
ˇ
s, N., Brim, L., Beran, J., and
Kratochv
´
ıla, T. (2016). Analysing sanity of require-
ments for avionics systems. Formal Aspects of Com-
puting, 28(1):45–63.
Dwyer, M. B., Avrunin, G. S., and Corbett, J. C. (1998).
Property specification patterns for finite-state verifica-
tion. In Proceedings of the second workshop on For-
mal methods in software practice, pages 7–15. ACM.
Filipovikj, P., Jagerfield, T., Nyberg, M., Rodriguez-Navas,
G., and Seceleanu, C. (2016). Integrating pattern-
based formal requirements specification in an indus-
trial tool-chain. In Computer Software and Applica-
tions Conference (COMPSAC), 2016 IEEE 40th An-
nual, volume 2, pages 167–173. IEEE.
Filipovikj, P., Nyberg, M., and Rodriguez-Navas, G. (2014).
Reassessing the pattern-based approach for formal-
izing requirements in the automotive domain. In
Proceedings of the 22nd IEEE International Re-
quirements Engineering Conference (RE), volume 00,
pages 444–450, Los Alamitos, CA, USA. IEEE Com-
puter Society.
Filipovikj, P., Rodriguez-Navas, G., Nyberg, M., and Sece-
leanu, C. (2018). Automated SMT-based consistency
checking of industrial critical requirements. ACM
SIGAPP Applied Computing Review, 17(4):15–28.
Grunske, L. (2008). Specification patterns for probabilis-
tic quality properties. In 2008 ACM/IEEE 30th Inter-
national Conference on Software Engineering, pages
31–40. IEEE.
Kitchenham, B., Pickard, L., and Pfleeger, S. L. (1995).
Case studies for method and tool evaluation. IEEE
software, 12(4):52–62.
Konrad, S. and Cheng, B. H. C. (2005). Real-time specifica-
tion patterns. In Proceedings of the 27th International
Conference on Software Engineering, ICSE ’05, pages
372–381, New York, NY, USA. ACM.
Mondragon, O. A. and Gates, A. Q. (2004). Support-
ing elicitation and specification of software properties
through patterns and composite propositions. Inter-
national Journal of Software Engineering and Knowl-
edge Engineering, 14(01):21–41.
Moser, L. E., Ramakrishna, Y., Kutty, G., Melliar-Smith,
P. M., and Dillon, L. K. (1997). A graphical envi-
ronment for the design of concurrent real-time sys-
tems. ACM Transactions on Software Engineering
and Methodology (TOSEM), 6(1):31–79.
Post, A., Hoenicke, J., and Podelski, A. (2011). rt-
inconsistency: a new property for real-time require-
ments. In International Conference on Fundamen-
tal Approaches to Software Engineering, pages 34–49.
Springer.
Post, A., Menzel, I., Hoenicke, J., and Podelski, A. (2012).
Automotive behavioral requirements expressed in a
specification pattern system: a case study at bosch.
Requirements Engineering, 17(1):19–33.
Remenska, D., Willemse, T. A., Templon, J., Verstoep, K.,
and Bal, H. (2014). Property specification made easy:
Harnessing the power of model checking in uml de-
signs. In International Conference on Formal Tech-
niques for Distributed Objects, Components, and Sys-
tems, pages 17–32. Springer.
Robson, C. and McCartan, K. (2016). Real world research.
John Wiley & Sons.
Runeson, P. and H
¨
ost, M. (2009). Guidelines for conduct-
ing and reporting case study research in software en-
gineering. Empirical Softw. Engg., 14(2):131–164.
Smith, R. L., Avrunin, G. S., Clarke, L. A., and Osterweil,
L. J. (2002). Propel: an approach supporting property
elucidation. In Proceedings of the 24th International
Conference on Software Engineering, pages 11–21.
ACM.
Westman, J. and Nyberg, M. (2014). Environment-Centric
Contracts for Design of Cyber-Physical Systems. In
MODELS, pages 218–234.
Wohlin, C., Runeson, P., H
¨
ost, M., Ohlsson, M. C., Reg-
nell, B., and Wessl
´
en, A. (2012). Experimentation in
software engineering. Springer Science & Business
Media.
Specifying Industrial System Requirements using Specification Patterns: A Case Study of Evaluation with Practitioners
103