Validating Sociotechnical Systems’ Requirements through Immersion

Andreas Gregoriades

and Maria Pampaka

Cyprus University of Technology, Limassol, Cyprus

The University of Manchester, Manchester, U.K.

Keywords: Requirements Validation, Simulation, Human Factors, Virtual Reality.

Abstract: One of the most critical phases in complex socio-technical system development is the validation of non-

functional requirements (NFR). During this phase, system designers need to verify that the proposed system’s

NFRs will be satisfied. A special type of NFRs which is often ignored regards the Human Factors (HF) NFRs.

These requirements are of vital importance to socio-technical systems since they affect the safety and

reliability of human agents within such systems. This paper presents a scenario-based approach for validating

HF NFRs using VR CAVE simulation. A case study is used to demonstrate the application of the method in

the validation of the situation awareness NFR of an in-vehicle Smart driver assistive technology (SDAT).

Such systems aim to alleviate accident risks by improving the driver’s situation awareness by drawing their

attention on critical information cues that improve decision making. The assessment of the HF NFR is

achieved through an experiment with users in a virtual environment. This work describes and demonstrates a

method that utilizes a custom-made, modular 3D simulator that uses a number of hazardous scenarios, for the

validation of the HF NFRs of prospective systems.

1 INTRODUCTION

Requirements validation constitutes an important

facet of a successful system development. Unlike

functional requirements, which can be

deterministically validated, non-functional

requirements (NFRs) are considered as soft/latent

variables not directly observed or implemented;

instead, they are satisfied (Zhu et al., 2012) by

functional requirements. Despite their importance,

NFRs are usually addressed at a late stage of system

development, whilst functional requirements are

considered at the early phase of software

development (Marew, 2009). Therefore, the initial

stages of a system’s specification may not address the

NFRs adequately, which could lead to system failure

once the system has been commissioned (Adams et

al., 2015). NFR analysis approaches range from

unstructured and informal, to highly formal and

mathematically-driven. The former include

approaches such as KAOS (Nwokeji et al., 2014), a

goal-oriented software requirements capturing

approach. In the same vein, i* approach (Chung et al.,

2000) uses goals and enables the quantification of

requirements from goal diagrams. The latter category

includes formal methods based on model checking such

as Z, Markov, and queuing models (Matoussi, 2008).

This paper introduces a Human Factors (HF)

requirements validation method that exploits

scenario-based testing through immersion. The

application of the methodology is demonstrated

through a case study on the analysis of the situation

awareness NFR of a future smart driver assistive

technology (SDAT). The uses a custom made virtual

reality (VR) simulator that mimics the environment

and models prototype SDATs using 3D visualizations

that simulate the candidate designs.

The paper next reviews the literature in NFR

assessment, HF and situation awareness (SA). This is

followed by the NFR validation methodology. Next a

case study demonstrates the application of the HF

NFR validation method, followed by analysis of the

data from the experiment and presentation of the

emerging results. The paper concludes with a brief

discussion of methodological and substantial

implications.

2 LITERATURE REVIEW

The majority of NFRs in complex socio-technical

systems address system properties such as

performance, reliability and security. However, there

is an additional dimension that needs to be analysed,

456

Gregoriades, A. and Pampaka, M.

Validating Sociotechnical Systems’ Requirements through Immersion.

In Proceedings of the 18th International Conference on Enterprise Information Systems (ICEIS 2016) - Volume 1, pages 456-463

ISBN: 978-989-758-187-8

which is the human dimension. By definition a socio-

technical system exhibits both technical and social

complexity. These systems are composed of human

and machine entities that work together to accomplish

a common goal. Transportation systems belong to this

category of complex systems since they incorporate

vehicles, drivers, road infrastructure and intelligent

systems in vehicles. The technical aspects of these in-

vehicle systems refer to the functional requirements

of machine agents and the human-machine

interaction metaphors. The social facet of the system

relates to human factors and the associated human

performance constraints. Thus, designing such

complex systems requires the investigation of all

facets. The technical dimension is addressed by the

functional requirements and the system NFRs, while

the human dimension is influenced by HF NFRs.

These are defined by human agent limitations

affected by the diverse nature of human

characteristics, such as ability, stress, concentration,

SA etc. However, despite their importance as a

critical cause of systems failure, human factors have

not been adequately considered by practitioners

during the design, development, and testing of

systems (Gregoriades, 2004).

Moreover, even though human factors and

requirements have a lot to share, only a few studies

apply human factors knowledge to requirements

engineering. While NFR such as performance,

security and maintainability are considered for

software functions, NFRs for people, such as SA and

workload, have received less attention. Such

requirements have been proven very significant in

preventing system failure, articulated in the form of

accidents in complex systems such as transportation

(Gregoriades, 2010). Therefore, the systematic

analysis of this type of NFRs prior to any system

implementation is considered vital. The main

problem in validating these requirements is the need

for a detailed specification of the envisioned system

or the implementation of a prototype system. Both of

these activities are time consuming and expensive.

The former requires formal methods which are hard

to comprehend by stakeholders and the latter requires

time effort and cost. Once either of the two is realised

it is possible to perform an analysis of system

behaviour under a number of test scenarios. Formal

methods, though, suffer from being too specific,

hence their application in validating NFRs is

constrained. Prototyping, on the other hand, provides

a more generic model based on which different facets

of the system can be tested such as people, technology

and tasks. This, however, is expensive and risky.

Therefore, the use of a simulated environment for

requirements analysis saves the costs of prototypes,

especially for complex systems (Sutcliffe et al., 2004)

and makes the process safe. This approach, employed

in this study, enables testing technological solutions

and the evaluation of their effect prior to

implementation.

Designing complex systems such as the smart in-

vehicle information systems requires the effective

and efficient management of requirements. The

inappropriate specification of functional and non-

functional requirements increase dramatically the risk

of failing to meet customer needs (Peng, 2012).

Functional requirements have received much

attention in this process, while, NFRs have been more

or less deliberately ignored (Illa, 2000). This led to a

lot of systems failing due to improper management of

NFRs. Past research addressed extensively different

sets of NFRs along with frameworks of NFR such as

Softgoal Interdependency Graphs (SIGs) (Zhu, 2012).

NFR validation has attracted significant attention

in recent years due to the importance of NFRs in

overall system acceptance. Traditional approaches to

NFR validation include prototyping and inspection.

Recent approaches focus on the quantitative analysis

of NFRs. In our previous work we used a Bayesian

Networks (BN) approach to model NFRs using

knowledge elicited from the domain (Gregoriades,

2005). NFRs are assessed based on a scenario

generation and evaluation algorithm that runs the BN

with different input. The output is a quantitative

estimation of the satisfiability of the NFR. Other

groups (Zayaraz et al., 2005; Sadana et al., 2007) also

used a quantitative model to analyze conflicts among

NFRs. This approach, however, is limited to high

level architectural requirements. In the same vein,

Marew and colleagues (Marew et al., 2009) used

Quantified Softgoal Interdependency Graphs

(QSIGs) to assess the degree of softgoal satisfaction.

However, the assessment of QSIGs is based on

subjective estimates of the degree of interdependencies

among softgoals. Similarly, Zhu et al. (2012) apply

fuzzy qualitative and quantitative softgoal

interdependency graphs for NFRs tradeoff analysis.

Based on the above, it is evident that NFRs assessment

is an ongoing research issue. The growing ubiquity of

complex sociotechnical systems led to more NFRs to

be analysed during systems’ design phase. One

example of such NFRs is safety which is addressed in

this study and expressed in terms of accidents.

2.1 Human Factors & Requirements

NFRs such as performance and maintainability are

specified for software or hardware systems. NFRs for

Validating Sociotechnical Systems’ Requirements through Immersion

457

people, such as SA and workload, have received less

attention. These requirements, however, have been

proven very crucial in preventing system failure.

Specifically, in transportation, road accidents are

usually attributed to human error (Fuller, 2002;

Theeuwes et al., 2012) that is induced from low SA

caused by increased workload. Humans, as

information processing systems, have a number of

information flow channels (visual, auditory, tactile)

processing various information sources (e.g. a

navigation system display, the forward view through

the windscreen) of varied bandwidths (e.g. high-

density traffic will require a higher sampling rate than

low-density traffic). Our cognitive capacity is limited,

and consequently there is an upper threshold to the

amount of information we can process per second and

channel (Endlsey, 2000; Fuller, 2002; Holohan et al.,

1978). Therefore, we tend to share our attention

among a few information sources. An overloaded

driver is less likely to deal effectively with an

unexpected event (Konstantopoulos et al., 2010).

Fuller (2012) also expresses accident risk as a

function of the driver’s cognitive resources and task-

demand in the driver-road system.

Therefore, the systematic analysis of these HF

NFRs prior to any system implementation is

considered vital. The main problem in evaluating

these requirements is the need to implement a

prototype design of a hardware-software system,

which is expensive (Stone et al., 2001). Hence, the

use of virtual reality (VR) settings is becoming very

popular. One of the most important applications of

VR technology has been the use of virtual prototypes

for functional requirements analysis (Sutcliffe et al.,

2004). However, the use of VR for HF requirements

analysis has not been addressed. Essentially, HF

requirements can be expressed in terms of a threshold

value that defines their minimum quantification or

satisfaction level. These define the cognitive and

physical capabilities of humans. These capabilities

are put to the test when processing dynamically

changing information during driving. If these

capabilities are reached then this in effect increases

the likelihood of committing an error due to high

workload. Workload, however, is directly related to

SA; the link between the two has been previously

established (Gregoriades et al., 2007). When the

perceived information increases people tend to

prioritise which increases the risk of an incorrect

comprehension. In traffic safety, SA constitutes a

major critical factor, since it provides the driver with

the ability to anticipate events given perceived

driving and environmental conditions.

Validating HF requirements for such systems

makes the use of VR simulators inevitable due to the

complexity, effort and cost associated with the

development of prototypes. In the same vein,

controlling infrastructural parameters in the real

world is unethical. Moreover, ruling out confounding

effects to examine the influence of control measures

on HF is very difficult in field experiments. Driving

simulators provide the researcher with a powerful tool

to test driving behaviour under controlled settings.

Apart from the usually high cost of the simulator,

outsourcing of experiments to analyse driving

behaviour using native users is difficult, if not

impossible in some cases, due to the large number of

subjects needed for reliable results. On the other

hand, low cost driving simulators do not provide a

sufficient level of realism to analyse human factors.

Unrealistic conditions may affect the driving

behaviour which effectively could influence the

validity of the experimental study. The method

proposed herein demonstrates the design of a driving

simulator that exploits 3D modelling tools in a

scenario-based approach to promote realism and

interactive representation of road networks. The

approach simplifies the process of implementing 3D

road infrastructure models through the utilization of

reusable modules that represent different in-vehicle

technologies or infrastructural components. This

simplifies the process of designing/modifying the

simulation model by reusing model constructs in a

plug and play fashion, which enables the analyst to

easily design a range of experimental conditions (i.e.

scenarios), to evaluate assumptions and hypotheses

from different perspectives.

2.2 Situation Awareness

SA constitutes a major critical factor in complex

socio-technical systems. In transportation, it provides

the driver with the ability to anticipate events given

perceived driving and environmental conditions. SA

defines the process of perceiving information from

the environment (level 1), comprehending its

meaning (level 2) and projecting it into the future

(level 3). SDAT have been developed to alleviate

accident risk by either reducing driver workload or

assessing driver attentiveness. Examples include

adaptive cruise control, collision notification, driver

monitoring, traffic signal recognition, night vision,

lane departure warning systems and blind spot

monitoring. Such systems aim to draw drivers’

attention on critical cues that improve their decision

making. However, they only provide limited support

to SA since they address isolated factors and in some

cases with negative effect due to the extra information

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

458

load they incur to the driver. The first step in

improving drivers’ SA is to enhance their capability

of perceiving and interpreting traffic and

environmental conditions (i.e. level 1 and 2 of the

aforementioned SA model). However, such smart

systems facilitate level 3 SA for navigation, which

might decrease drivers’ attention, due to secondary

task execution, that could lead to reduced level 1 SA.

This could undermine attention to operational or

tactical driving activities (e.g. braking, lane changing,

gap acceptance etc.). To that end, three important

issues need to be addressed prior to any SDAT

development: (i) identification of drivers’

information needs that could enhance SA, (ii) the

specification of a SDAT feedback metaphor

(feedback type and appropriate time for issuing

warnings) to support those needs without impairing

driver attention, and (iii) the evaluation of the effect

of a prospective SDAT on traffic safety. This is a

complex process and in most cases is only feasible

once a prototype of the system is available.

Endlsey et al. (2012) warn socio-technical system

designers of the importance of maintaining SA in

complex systems and draw attention on the issues that

could inhibit SA. One of the most important strains of

SA is information overload. Too much information at

any point in time hinders human operators’ adequate

SA. Overloading divides the decision maker’s

attention among numerous stimuli resulting in

increased demand for cognitive resources. This is

known as attentional tunnelling (Endlsey, 2012) and

results in reduced information scanning capability.

3 NFR VALIDATION METHOD

The proposed NFR validation method is based on the

design science (Hevner et al., 2010) paradigm, and in

particular its evaluation phase which investigates the

effectiveness of an artefact and guides its re-design

through changes in specification. Design science

synthesises the sciences of the artificial, engineering

design, information systems development, system

development as a research methodology, and

executive information system design theory for the

building and evaluating of IT artefacts for specific

problems (Hevner et al., 2010). The design and

development of new artefacts such as the SDAT,

described herein, requires a systematic approach

towards artefact design, development and evaluation.

This aims to assure that the artefact contributes

towards resolving a particular problem.

The method is composed of a number of steps that

are executed both in sequence and in parallel at

certain stages. Initially, the problem needs to be

expressed in terms of human factors specification.

This could be articulated in terms of human

performance and human reliability, and in particular,

as the acceptable SA and workload levels of human

agents in a system. These are conditions that could

incur high likelihood of human error (Gregoriades,

2010). Once the problem to be analysed is clearly

stated and the critical HF NFRs are identified, then

the minimum level NFR satisfiability needs to be set.

The refinement of HF NFRs into functional

specifications which when realised will guarantee the

satisfaction of the NFR comes next. This is achieved

using a combination of domain knowledge and input

from subject matter experts. For instance, guidelines

for enhanced SA, as specified by Endlsey (2012), are

expressed in terms of information requirements,

visualisation metaphors and interaction styles which

are functional requirements that the SDAT should

have. The next step in the process is the specification

of the test scenarios, based on which the artefact is

going to be evaluated. Grounded within the problem

to be analysed, the goals of the desired virtual

environment are set. Accordingly, specifications of

the virtual environment to be used for the evaluation

of the artefact are also set. During this stage a generic

VR simulator is customized based on the above goals,

to model the problems in question. The

customization of the simulator is composed of three

steps: 1) the development of the test environment in

terms of buildings, infrastructure and traffic

conditions. 2) The modelling of the scenarios, as

described by the domain experts; these include

atypical events in the simulation that would stress test

the subjects in the experiment. 3) The modelling of

the virtual version of the artefacts under scrutiny.

Prior to its use, the VR simulator needs to be validated

against a number of factors such as realism, to

guarantee the correctness of the NFR assessment.

NFRs quantification is achieved through an

experiment with users in the VR environment. The

specification of the experiment is defined by an HF

expert. The assessment of NFR is then refined into

phenotype behaviours that can be monitored in a

driving simulator. Phenotype driving behaviours are

monitored and logged into the systems database. The

logged observations from the simulation are pre-

processed, analysed and subsequently collated into a

single metric that corresponds to the assessed NFR.

The NFR assessment is compared against the desired

NFR level. If the minimum level of NFR is not

satisfied then the virtual artefact under scrutiny needs

to be redesigned. The process is repeated until the

NFR is satisfied.

Validating Sociotechnical Systems’ Requirements through Immersion

459

4 CASE STUDY

To demonstrate the application of the method, a case

study was conducted for the validation of the SA

NFRs of a future SDAT. The NFR evaluation method

is based on the paradigm of scenario-based testing. In

each scenario, participants were required to drive

through a pre-specified path on a road network.

Throughout the driving task, participants had to

respond to emerging hazardous situations. Situational

cues were visualised through the SDAT in the form

of a virtual augmented reality head-up display (HUD)

interface within the virtual vehicle. The SDAT

interface was designed based on identified driver

information requirements and domain knowledge

(Endlsey, 2012). SDAT designs aimed to address

drivers' information needs for better SA. Specifically,

vehicle's peripheral traffic, road works, road signs

and approaching traffic jam were projected through

the virtual SDAT. The goal was to assess the effect of

each SDAT design on drivers’ SA. Satisfiability of

SA NFR is specified as an improvement in drivers'

SA using SDAT compared to no SDAT use, and is

specified as a threshold value. Two SDAT designs

were developed using Endlsey’s (2012) design

principles for SA support. The functional

requirements of the SDAT systems have been

implemented using the guidelines of: information

prioritization, timeliness and relevance of

information, information filtering, familiarity of the

visual metaphors, and presentation of information in

the right context. These aim to alleviate information

overload, reduce display density, enhance driver’s

ability to comprehend the meaning of information and

finally assist in developing projections of the situation

into the future. The SDATs utilise the above through

fusion of vast amount of information from the

environment into meaningful attentional directives/cues

that describe the driving situation in real-time.

Figure 1: The driving simulator in the VR CAVE. A

participant doing the experiment while being observed by

researchers.

As part of the NFR validation method, the first

step is the design and implementation of the driving

simulator. Figure 1 illustrates the developed simulator

in VR CAVE that enables the stereoscopic interaction

of participants with the experimental conditions.

Participants are immersed with the experimental

scenarios through a combination of augmented reality

and tangible interaction styles, for a more realistic

experience. The second step in the method is the

design of the virtual prototype SDAT systems in the

virtual environment. The development of the virtual

SDATs is realized using a scripting language. The

virtual SDAT had to abide to the functional

requirements specified in previous steps. The third

step is the specification of the hazardous scenarios.

Figure 2: The radar design (right) and information

prioritisation –arrow design (left).

The user interface of SDAT systems is of

paramount importance in improving SA. Hence, it

was designed to provide blind-spot information and

to alert drivers of unseen imminent threats. The

system uses a combination of HUD with augmented

reality capabilities, so that the direction of the threat

is clearly comprehended by drivers. The information

architecture of the UI aimed to provide the driver with

enhanced peripheral vision with a dynamic

assessment of the most critical entities within the

immediate periphery of the vehicle. The blue print

designs of the candidate systems are depicted in

Figure 2. In the first design (radar), the host vehicle is

shown in a circle (in blue) surrounded by red and

green vehicles of different sizes. The size and colour

of surrounding vehicles denotes the level of risk.

Hence, vehicles that are in the driver’s blind spot are

considered high risk and are represented by big red

icons. Low risk cars are depicted with small green

icons. High proximity or hidden vehicles at

intersections are also high risk and hence are shown

as big and red. Surrounding vehicles’ positions and

speeds can be obtained from on-board vehicle

sensors. Vehicles at intersections can be obtained

through vehicle-to-vehicle communication protocol.

The prototype visualization metaphor presented in

Figure 2 is depicted on the vehicles windshield. The

second design (arrows) of the system is based on the

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

460

need to prioritize information based on risk level and

aims to warn drivers of vehicles that are expected to

emerge from side roads and are not yet visible or

vehicles that are in driver’s blind spot. This, as

illustrated in Figure 2, is expressed using arrows, on

the augmented reality windscreen, pointing to the

direction of the imminent threat, and is depicted on

the vehicle’s smart windshield. The most critical

threat is depicted on the screen so as not to split the

attention of the drivers among competing risks. This

gives extra time to drivers to react to critical situations

The assessment of SA is achieved through an

experiment with subjects using the developed driving

simulator and virtual SDAT in a 3D CAVE facility.

During this stage, 17 participants were involved, each

spending on average of 90 min to complete the

experiment in the VR CAVE lab. The analysis of the

data collected from the experiments aimed to assess

the SA NFR for the two candidate SDAT designs.

Data was collected in three phases: before, during

and after the experiment. During the pre-experimental

phase, the Manchester Driving Style questionnaire

(Reason et al., 1990) was used to elicit the driving

style of participants along with their demographic

information. At the post experimental phase, data

collection focused on the evaluation of the two

candidate designs using a series of questions on four

constructs: functionality, information visualization,

usability and usefulness. During the experiment,

participants’ SA was measured while they were

driving in a pre-specified route in the artificial road

network within the 3D driving simulator (Figure 1),

both with and without the SDATs. In particular,

participants were asked to consult the HUD SDAT as

during the driving simulation surrounding vehicles

engaged the host vehicle by either pulling in or

stopping in front of the driver. During the drivers’

engagement with the experimental conditions,

phenotype behavioural data related to driver

workload and SA was recorded. Driver related data

was recorded in a log-file on a simulation time-step

basis. Specifically, manifestations of workload, such

as lateral deviations (Montella et al., 2011), attention

level through an electroencephalography (EEG)

measurement, lane change, headway, speed,

acceleration, deceleration, breaking patterns and

steering wheel angle, were recorded on a time-

location log-file. Collected data was automatically

assigned to road sections that were specified in

advance by the analysts, based on infrastructural

properties. The assessment of the drivers’ SA was

achieved using the SAGAT (Situation awareness

global assessment technique) method, which uses

objective measures of SA gathered during an

interruption in task performance. Hence, during each

scenario with the participant, the simulation was

stopped (freeze) three times, at points on the road

network where the three dangerous scenarios were

unfolding (car pulling in from the left, car stopping in

front, car pulling in from the right). At each

simulation freeze, participants were asked to

complete a questionnaire that inquired their

understanding of the situation. During the freeze, the

simulators screens were blank. The simulator saved

several screenshot of the situation just before the

freeze to enable the comparison between the ‘actual’

event and the subjects’ perceived situation.

5 RESULTS

Data collected from the simulations were pre-

processed and analysed to identify differences

between the actual situation and the participants’

perceptions of the situation under the three conditions

and the three interventions (phases). Analysis was

conducted on both the post-experiment and the

experiment data. Results from the post experiment

data revealed that both SA enhancement systems

were perceived by the users as improvements over the

control condition (i.e. without any enhancement).

Specifically, the post-experiment questionnaire

addressed the following dimensions of each candidate

design: features, user interface, ease of learning,

system capabilities, usefulness, ease of use, and SA.

Each dimension was supported on average by 5

questions, on a 7 point response scale from 1

(negative effect) to 7 (positive effect). To increase the

discrimination in the evaluators’ judgment,

participants’ were asked to report the reasons for their

choices and any interaction problems they had

experienced under the relevant heuristic. Figure 3

shows the percentages of positive responses (i.e. >4, or

<4 for negatively worded statements) for each of the

measured dimension on which the two designs were

evaluated. Based on this analysis, there do not seem to

be noticeable differences in regards to user interface

and ease of use. However, overall the radar design

seems to have been perceived more positively than the

arrows, especially in relation to learning, system

capabilities, and usefulness. This might be attributed to

the small size of the arrows that were popping up on

the smart windshield. Among the two designs the radar

design was also considered more appropriate to

support driver SA. Moreover, based on open responses

from participants, in certain occasions, the number of

arrows that were present on the windshield were more

than two. Hence, the cues were becoming destructing

Validating Sociotechnical Systems’ Requirements through Immersion

461

rather than informative. On the other hand, the Radar

design also had its shortcoming in terms of

visualization of the threats. Specifically, the colouring

and size of threats were considered insufficient.

Figure 3: Percentages of positive responses in each of the

measured dimensions, by design.

Results from data during the experiment aimed to

assess the SA NFR using a combination of the

SAGAT data and the driver behaviour data from the

simulation log files. Initially the SAGAT and the

driver behaviour data were integrated into one dataset

for each participant. Subsequently the data that

represented the actual situation was compared with

the data that represented the perceived situation for

each participant at each phase of the experiment. The

similarity assessment between actual and perceived

was estimated using the Euclidian distance metric.

Analysis of the SA data was then performed using

ANOVA in a within-subjects model. Based on the

results, the use of both SDAT designs in an

augmented reality overhead display demonstrated a

superior performance to no-design. Results from the

SAGAT analysis also revealed that design 1 (radar)

was superior to design 2(arrows) and no design. This

was identified as significant based on figure 4. In the

same vein, the phase of the simulation freeze,

denoting the sequence of the freeze, was also

identified as a significant factor with phase 3 in the

radar design having on average a SA metric of 85%

compared to 63% in the control condition (no design).

Figure 4: Estimated marginal mean for the 3 designs (radar

-1, arrows-2, no-design-3) and the three phases of the

simulation (freeze1-3).

6 CONCLUSIONS

The HF NFR validation method presented herein

provides a novel cost effective solution to validating

HF NFRs of prospective complex sociotechnical

systems. It enables the evaluation of NFRs through

experimentation in VR settings under an envelope of

test scenarios. The developed driving simulator is

component-based and hence enables the requirements

engineer to easily customize it to the problem in hand.

Requirements are realised in virtual settings and this

provides designers with the flexibility of customizing

the functionality of the SDAT in an attempt to satisfy

the HF NFR under consideration. Results from the

application of this method in the validation of the SA

NFR of an in-vehicle SDAT revealed the method’s

practicality. The method is based on design science

and encourages the redesign of the artefact until it

satisfies the NFR. Results indicate that what the users

experience during their interaction with the artefact

and what they perceived of this experience as

reported in the post-test questionnaire point to the

same conclusion. Specifically, statistical analysis of

the data collected indicated that the radar design is

superior to arrows and no design. Similarly,

subjective evaluation of the candidate designs also

revealed the same results. Hence, this agreement is a

good indication that the NFR validation method is

producing accurate estimations. Limitations of this

work concentrate on the simulator’s level of realism

and immersion factors that laboratory methods suffer

from. Simulated settings do not currently offer the

resolution of the real world, and so they may affect

driving behavior. Future work will include the

improvement of the realism factor which in turn will

improve observational accuracy. Moreover, the

experimental design for the evaluation of the SA was

very time consuming. This could be optimized though

the use of a cut down version of the SAGAT

questionnaire.

ACKNOWLEDGEMENTS

This work was funded by the Cyprus Research

Promotion Foundation, grand NEKYP/0311/02

REFERENCES

Adams K., et al, 2015, Non-functional Requirements in

Systems Analysis and Design.in Topics in Safety, Risk,

reliability and Quality, Springer.

10%

20%

30%

40%

50%

60%

70%

80%

90%

General

Features

User

Interface

Learning Capabilities Usefulness Ease of Use Situation

Awareness

Percentage of Users scoring positively (>4)

Radar Arrows

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

462

Chung, L., Nixon, B. A., Yu, E. and J. Mylopoulos, 2000.

Non-functional requirement in Software Engineering,

Kluwer Academic Publishing.

Davenne D., Lericollais R., Sagaspe P., Taillard J., Gauthier

A., Espié S., Philip P. 2012. Reliability of simulator

driving tool for evaluation of sleepiness, fatigue and

driving performance, Accident Analysis and

Prevention, 45, pp.677-682.

Dingus, T. A., Klauer, S. G., Neale, V. L., Petersen, A., Lee,

S. E., Sudweeks, J., et al. 2006. The 100-car naturalistic

driving study: Phase II – Results of the 100-car field

experiment. Washington, DC: National Highway

Traffic Safety Administration.

Endlsey M. R 2012. Designing for Situation Awareness: An

Approach to User-Centered Design, Second Edition,

CRS press.

Fuller R. and Santos J. 2002. Human Factors for Highway

Engineers. New York: Pergamon.

Gregoriades. A and Sutcliffe. A. 2007. Workload prediction

for improved design and reliability of complex systems.

Reliability. Eng. System Safety, 39, n.4, pp.530–549.

Gregoriades A, Sutcliffe A, Papageorgiou G, Louvieris P.

2010 Human-Centred Safety Analysis of Prospective

Road Designs, IEEE Transactions on Systems, Man and

Cybernetics, Part A, Vol 40, 2, pp 236-250.

Gregoriades A., Sutcliffe A. S., 2005. Scenario-based

assessment of non-functional requirements, IEEE

Transactions on Software Engineering, , Vol 31, 5, pp

392-409.

Hevner, A. & Chatterjee, S 2010. Design Research in

Information Systems, Integrated Series in Information

Systems, vol 22, Springer.

Holohan, C., Culler, R., & Wilcox, B. 1978. Effects of

visual distraction on reaction time in a simulated traffic

environment, Human Factors, 20, pp.409–413.

Illa, X. B., Franch, X, & Pastor, J.A. 2000. Formalising

ERP selection criteria. In Proceedings of the 10th

international workshop on software specification and

design (IWSSD’00), California, pp. 115–123.

Konstantopoulos P., Chapman P., Crundall D. 2010.

Driver’s visual attention as a function of driving

experience and visibility. Using a driving simulator to

explore drivers’ eye movements in day, night and rain

driving, Accident Analysis and Prevention, 42, pp.827-

834.

Marew, T. et al. 2009. Tactics based approach for

integrating non-functional requirements in object-

oriented analysis and design Syst. Software, 82, pp.

1642–1656.

Matoussi, A., and Laleau, R. 2008. A Survey of Non-

Functional Requirements in Software Development

Process, Report No. TR-LACL-2008-7, Departement

d’Informatique Universite Paris 12, 2008.

Montella A., Ariab M., D’Ambrosiob A., Galantea F.,

Maurielloa F., Pernettic M. 2011. Simulator evaluation

of drivers’ speed, deceleration and lateral position at

rural intersections in relation to different perceptual

cues, Accident Analysis and Prevention, 43, pp.2072-

2084.

Nwokeji J., et al, 2014. ER 2014- International Workshp on

Conceptual Modeling in Requirements and Business

Analysis (MREBA), USA.

Peng, Y. G. Wang, H. Wang, 2012. User preferences based

software defect detection algorithms selection using

MCDM, Information Sciences 191. 3–13.

Reason, J., A. Manstead, S. Stradling, J. Baxter, K.

Campbell. 2005. Errors and violations on the roads: a

real distinction, Ergonomics, 33 (10–11) (1990), pp.

1315–1332.

Stone, R. J. 2001. Virtual Reality for Interactive Training:

An Industrial Practitioners Viewpoint, International

Journal of Human-Computer Studies, vol. 55, pp. 699-

711.

Sadana V., Liu X., 2007. Analysis of conflicts among non-

functional requirements using integrated analysis of

functional and non-functional requirements, Computer

Software and Applications Conference, COMPSAC.

Zayaraz G, Thambidurai P., Srinivasan M., Rodrigues P..

2005. Software quality assurance through COSMIC

FFP ACM SIGSOFT Software Engineering Notes, 30

(5).

Marew, T. et al.2009. Tactics based approach for

integrating on-functional requirements in object-

oriented analysis and design Syst. Software, 82, pp.

1642–1656.

Zhu M., Luo X., Chen X., Dash W.,2012. A non-functional

requirements tradeoff model in Trustworthy Software,

Information Sciences, Volume 191, 15, pp 61-75.

Sutcliffe, A. and Gault, B. 2004, The ISRE method for

analyzing system requirements with virtual prototypes.

Syst. Engin., 7: 123–143.

Theeuwes J. et al, 2012, Designing Safe Road Systems: A

Human Factors Perspective, Ashgate.

Validating Sociotechnical Systems’ Requirements through Immersion

463