Towards Common Information Systems Maturity Validation

Resilience Readiness Levels (ResRL)

Rauno Pirinen

The National Defence University, Helsinki, Finland

Laurea University of Applied Sciences, Espoo, Finland

Keywords: Adaptive Information Systems, Common Information Sharing, Resilience, Resilience Readiness Levels,

Resilient Systems, Maturity, Operational Validation.

Abstract: The intent of this study is on a proposal of resilience readiness level (ResRL) metrics towards their aspects,

factors, definition, criteria, references and further questionnaires for the contribution of combined-total

maturity measures and pre-operational validation of shared and adaptive information services and systems.

The study attempts to answer the following research question: how can ResRL metrics be understood in the

domain of shared information systems and services. It aims to improve ways of the acceptance, operational

validation, pre-order validation, risk assessment and development of adaptive mechanisms as well as the

integration of information systems and services by actors and authorities across national borders.

1 INTRODUCTION

In the operative environment of this study,

knowledge management is understood as a

discipline concerned with the analysis and technical

support of practices used in an authority-related

organization and decision-making to identify, create,

represent, distribute and enable the adoption and

leveraging of real-world practices, which were used

in collaborative authority settings and, in particular,

public authority organizational processes.

This study of resilience is based on the ongoing

and cumulative data collection of three (n=3)

preliminary research and development (R&D)

projects: 1) European Union’s Common Information

Sharing Environment (EU_CISE_2020), including

R&D-related research on work packages (n=8) of

the EU_CISE research consortium and research

agenda targets related to the public authority in

Finland; 2) Maritime Integrated Surveillance

Awareness (MARISA) including eight work

packages (n=8) as current H2020 project and

EU_CISE continuum; and 3) From Failand to

Winland, the Academy of Finland Strategic

Research Council project as ongoing National

Critical Research Project (#WINLandFI) covering

five (n=5) work packages. The perspective of study

is in contribution of information systems combined-

total maturity validation and new resilience metrics.

The study addresses to information sharing

environments that foster cross-sectorial and cross-

border collaboration among public authorities, the

dissemination of the EU_CISE initiative and steps

along the Maritime EU_CISE roadmap. EU_CISE

work entails the widest possible experimental

environments encompassing innovative and

collaborative services and processes between

European institutions and takes as reference, a broad

spectrum of factors in the field of European

integrated services arising from the European legal

framework as well as collaborative studies and

related pilot projects: [EU_CISE_2020; Project ID

608385; Funded under FP7-SECURITY].

The overarching goal of MARISA project is to

provide the security communities operating at sea

with a data fusion toolkit, which provides a suite of

methods, techniques and software modules to

correlate and fuse various heterogeneous and

homogeneous data and information from different

sources, including Internet and social networks, with

the aim to improve information exchange, situational

awareness, decision-making, reaction capabilities

and resilience. The expected new solutions will

provide mechanisms to get insights from big data

sources, perform analysis of a variety of data based

on geographical and spatial representation, use

techniques to search for typical and new patterns

that identify possible connections between events,

Pirinen R.

Towards Common Information Systems Maturity Validation - Resilience Readiness Levels (ResRL).

DOI: 10.5220/0006450802590266

In Proceedings of the 9th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (KMIS 2017), pages 259-266

ISBN: 978-989-758-273-8

discover predictive analysis models to represent the

effect of relationships of observed objects and

phenomena: [MARISA: Project ID 740698; Funded

under H2020].

The #WINLandFI research project will take you

from Failand (failed future Finland) to Winland, in

such as Finland where key security threats have been

responded to with resilient policy-making. What

kinds of security risks and threats could paralyse

Finland so fundamentally that our country becomes

Failand? The project data includes arguments that

Failand becomes reality if two of the most

fundamental elements of a functioning society fail

food security and energy security, which both are

closely linked to water security. In addition, this

research data comprises reasoning for a setting of

resilience that such failure is likely to result from the

sum of four key components: long-term pressures,

shocks and surprises, decision-making, and policy

responses: [#WINLandFI; Funding ID 303623;

Funded under the Strategic Research Council (SRC)

at the Academy of Finland].

This “study of resilience” is challenged by

adaptive nature of networked systems, they become

increasingly difficult to understand, predict and

control. However, no single agreed upon definition

of the term “resilience” exists; there are numerous

theories and literature to explain resilience and its

sources, paths and impacts. In this study, the

rationality and motivation to the proposal

description of the resilience metrics is in usefulness

of these themes and categories in data collections,

data fusions, knowledge fusions, analysis and

especially triangulation fashion in real R&D cases,

research consortiums, and externally funded R&D,

for implementation and design of thematic studies,

domain configuration and its integration strategy.

In this specific operative environments, the term

“resilience in information systems or services” is

understood as a complex process involving multiple

overlapping and iterative tasks that address to design

theory and system theory as well as a multi-

methodological approach that involves thinking,

building, improving and evaluating a successful

information system and its communication, which

fits the needs of the applied domain, information

sharing and resilience readiness viewpoints.

An expected contribution of information sharing

related resilience is related to the alignment of

ontology of information technology, data additivity

capabilities, parallel communication protocols,

nexus management and adaptive dynamic factors of

high-value impacting technological artifacts, digital

infrastructures and critical systems, e.g., ontology

and semantic fusion capabilities taking advantage of

Web Ontology Language (OWL) and Resource

Description Framework (RDF) languages.

Although standardization is indeed an essential

element in sharing information, information systems

resilience and effectiveness requires going beyond

the syntactic nature of information technology and

delving into the human functions at the semantic,

pragmatic, critical realist and social levels of

institutional-organizational functions.

The research domain prioritizes improvements in

resilience settings of a complex service or system.

The term “external validity”, in resilience

viewpoints, refers to establishing the expanded

cross-domain in which the study’s findings and

conclusions can be generalized. This study adopts

the method of increasing understanding through

information systems research and maturity-

integration facilities, such as utility and

communication, resilience readiness and networked

realization capability.

The expected contribution of study addresses to

the operational and pre-operational validation (POV)

and utility of ISO standardization and

interconnection followed: 1) improvement in metrics

for information system and service integration; 2)

advances in global procurement management and

pre-order validation; 3) pre-operational validation in

information system investigations; 4) progress in

operational validation in information system

implementation; 5) findings of methodological

implications for the implementation of ResRL

metrics and improved resilience; 6) usefulness of

information system sharing and interconnection; 7)

expansion of large and networked information-

intensive services that can extend shared solutions

and routes of shared information utilization and

common global information and information system

sharing; and 8) educational advances in R&D-

related functions in higher education institutions,

which in this case, can be shared across national

borders.

The macro-level target of this research is to

further the examination of how existing TRL, IRL

and new ResRL metrics and their definition, criteria,

references, questionnaires and guidelines can be

useful and employed to realize and validate

integration, communication and dynamic

functionalities in information systems and

information sharing.

At the micro level, this study was performed on

shared information systems in the case of shared

maritime systems and focuses on readiness targets as

realizations and validation. Realization such as the

usefulness, sharing and dissemination of an

information system as a common digital service,

product or solution involving shared information

across appropriate borders of applied domains.

Validation, that is, pre-operational validation, pre-

order validation for procurements, internal validity

and external validity, which can, for example, be

useful in the national and global deployment-

dissemination processes, operational validation of

information systems, improving integration success,

outsourcing, achieving common ontological

understanding and improving methods.

The overall motivation of this research

continuum is to address increasing trustworthiness

such that related studies make sense and are credible

for such as HORIZON audiences. The study design

is based on a combination of a thorough

understanding of the theoretical framework, studies

in the related literature and experimental knowledge

of the collaborative integration used to explain the

research question as well as learning processes and

their meaning. Validity in this analysis refers to the

establishment of casual relationships such as nexus-

mutual-impacts. Causal relationships appears in such

as interactions and relationships among shared

readiness measures and information systems

realizations from the perspective of readiness levels,

information sharing across borders of various

domains and the use of shared information systems.

2 LITERATURE

The key knowledge aspects for development of

ResRL metrics proposal included path-dependencies

with the related literature, for example, system

engineering (Eisner, 2011), systems readiness levels

(Sauser, Verma, Ramirez-Marquez and Gove, 2006)

and the development of an integration readiness

level (IRL) metrics (Sauser, Gove, Forbes and

Ramirez-Marquez, 2010).

Following these works, as continuum, the overall

research question was that how can ResRL metrics

be understood in the domain of shared information

systems and services. First, here, how ResRL

metrics can be understood and realized in the

context of the Common Information Sharing in such

as EU CISE, MARISA and #WINLandFI

environments using generally understood and related

metrics and models for the realization and reasoning

of furthered common maturity development.

This study, thus far, showed that a technological

readiness and integration readiness metric are two

basic elements of the thinking, building, improving

and testing of information systems, networked or

distributed integration and ontology. This view is

furthered by combined system readiness level (SRL)

metrics, which have been described as a

combination of TRLs function of technologies and

IRLs of integrations, as introduced by (Sauser et al.,

2006) and continued by (Luna, Lopes, Tao, Zapata

and Pineda, 2013). Figure 1 describes an approach

towards the combined-total maturity in Information

Systems Maturity Validation in the context of this

continuum of studies.

Figure 1: An Approach towards Common Information

Systems Maturity Validation.

In Figure 1, described Common Information

Systems Maturity Concept and SRL metrics are

understood here as the collector of metrics

represented by a single SRL metric defined on the

basis of the amalgamation of other existing readiness

levels, thus providing a method to chain and utilize

different readiness level metrics.

An aspect of SRL’s significance is that it gives

credibility to the quantitative collection of readiness

levels and opens possibilities to expand SRLs by

incorporating other readiness-level and validity

metrics, such as the manufacturing readiness level,

software readiness level, SRLs, and information

systems maturity as well as validity on an overall

scale [see also (Tan et al., 2011)].

In the context of EU_CISE information sharing,

MARISA data fusion and #WINLandFI resilience

and learning, it is noteworthy, that the related

literature on readiness metrics has similarities to a

combination of decision-making items, such as a

component of pre-operational or pre-order validation

and procurement management viewpoints.

The first widely understood and well-known

model regarding of our ResRLs proposal

development was open system interconnection (OSI)

(Zimmermann, 1980), described in Figure 2. For

related IRLs development, Sauser et al. (2006)

described this development path as follows: ‘it was

necessary to develop an index that could indicate

how integration occurs’ (p. 6). This index

‘considered not only physical properties of

integration, such as interfaces or standards, but also

interaction, compatibility, reliability, quality,

performance and consistent ontology when two

pieces are being integrated’.

Figure 2: Interpretations of OSI 7 layer model

(Zimmermann, 1980; revised form Pirinen et al., 2014).

Figure 2 describes the compacted structure of the

OSI model as the first approach to our ResRLs

proposal development. As well, Sauser et al. (2006)

selected the OSI model, its layers and targets, Figure

2 as the starting point of overall maturity readiness

levels development. The OSI model has been widely

referenced in computer networking to structure data

transmitted on a network and allows for the

integration of various technologies on the same

network, networking theme (Beasley, 2009) and

system approach to computer networks (Peterson

and Davie, 2012).

Much of the early works in this field involved

defining the risks and costs associated with various

TRLs. The related literature indicates that TRLs

addresses the evaluation of the readiness and

maturity of an individual technology. Hence, TRL

metrics adopt a given technology from the basic

principles as well as concept evaluation, validation,

prototype demonstration, and finally, completion

and successful operations.

These TRL characterizations are useful in

technology development, they address, to an extent,

how this technology is integrated and on needed

changes adapted within complete information-

intensive systems and applied services. In addition,

we recognised that, currently, many complex

systems fail in the integration phase and especially

in case of “adaptive change needed on demand” and

then, these readiness functionalities of resilience are

proposed for further development and discussions.

The Horizon Work Programmes includes TRL

guidelines, which are widely referenced and used in

H2020 proposals and evaluation. Figure 3 describes

the TRL metrics with methodologies used the R&D

context of study.

Figure 3: Description of TRL metrics and used R&D

methodology in the study.

In addition, integration, nexus as mutual causalities

and impacts of integration processes which are

owing increasing speed of technological

development, effects of new updates and needs of

more resilient systems for relevant adaptive needs

were implicated (Tan, Ramirez-Marquez and Sauser,

2011).

The IRL metrics were introduced by the Systems

Development and Maturity Laboratory at the

Stevens Institute of Technology and developed to

assess the progress of information system integration

and communication in the engineering field. The

study aimed at realizing and validating IRL metrics

in the extended context of the ISO DIS 16290

standard development framework by the

International Standards Organization.

A reason underpinning the present IRLs research

is that the TRLs do not accurately capture the risk

involved in adopting a new technology and that

technology can have an architectural difference

related to integration readiness and system

integration. In this environment, because the

complexity of a system or information increases, and

a practical situation often involves a service-oriented

network and shared systems, it is reasonable to

employ a reliable method and ontology for

integration-resilience readiness. This also allows

other readiness levels to be collectively combined

for the development of complex information-

intensive systems in information sharing and the

integration of systems as a common shared system.

The IRL metrics are defined as a ‘systematic

measurement of the interfacing of compatible

interactions for various technologies and the

consistent comparison of the maturity between

integration points’ (Sauser et al., 2006) (p. 5). IRL

metrics are used to describe of the integration

maturity of a developing technology using another

technology or mature information systems.

IRLs contribute to TRLs by checking where the

technology is on an integration readiness scale and

offering direction to improve integration with other

technologies. In general, just as TRLs has been used

to assess risks associated with developing

technologies, IRLs was designed to assess the risk

and development needs of information systems

integration.

Sauser et al. (2006) described IRLs development

path dependency that is based on the OSI model as

follows: ‘to build a generic integration index

required first examining what each layer really

meant in the context of networking and then

extrapolating that to general integration terms’ (p.

6). With this description, as shown on the left-hand

side of Figure 4, IRLs were defined to describe the

increasing maturity of the integration between any

two technologies between 2006 and 2010 through

the development of an integration readiness level

(Sauser et al., 2010) and using a system maturity

assessment approach (Tan et al., 2011). On the right-

hand side of Figure 4, the IRL metrics are described

in the context of the continuum of study.

The integration and data fusion standpoints can

also be related to a modular implementation strategy

as an approach that addresses challenges related to

the mobilization, steering and organization of

multiple stakeholders in wide-scale R&D

collaboration. Here, the focus is on the challenges of

realizing large-scale technological and information-

intensive systems, which are understood not as

standalone entities, but as those integrated with other

information systems, communication technologies

and technical and non-technical elements as well as

in a data and information fusion functions.

It is also included with the fact that an integrated

system can be a shared system in a network of

shared information [cf. building nationwide

information infrastructures (Aanestad and Jensen,

2011) and the case of building the Internet (Hanseth

and Lyytinen, 2010)].

In Figure 4, the description of IRL metric

includes nine levels (Sauser et al., 2006). The IRL

and TRL metrics are developed to assess technology

and integration by research interventions included in

numerous of National Aeronautics and Space

Administration and United States Department of

Defence efforts.

As shown in Figure 4, IRL layer 1 represents an

interface level: it is not possible to have integration

without defining a medium. In turn, selecting a

medium can affect the properties and performance of

a system. Layer 2 represents interaction, the ability

of two technologies to influence each other over a

given medium; this can be understood as an

integration proof of the concept, such as facilitating

bandwidth, error correction and data flow control.

Layer 3 represents compatibility. If two integrating

technologies do not use the same interpretable data

constructs or a common language, then they cannot

exchange information and data fusion if difficult.

Layer 4 represents a data integrity check. There is

sufficient detail in the quality and assurance of the

integration between technologies, which means that

the data sent are those received and there exists a

checking mechanism. In addition, the data could be

changed if part of its route is on an unsecured

medium [cf. realizations (Beasley, 2009) and

understanding of layers (Sauser et al., 2010)].

Figure 4: Integration readiness levels

(Sauser et al., 2010; Pirinen et al., 2014).

In Figure 4, IRL layer 5 represents integration

control: establishing, maintaining and terminating

integration, for example, possibilities to establish

integration with other nodes for high availability or

performance pressures. Layer 6 represents the

interpretation and translation of data, specifying the

information to be exchanged and the information

itself as well as the ability to translate from a foreign

data structure to a used one. Layer 7 represents the

verified and validated integration of two

technologies, such as the integration achieving

performance, throughput and reliability

requirements. Layers 8 and 9 describe operational

support and proven integration with a system

environment, corresponding to levels 8 and 9 of the

TRLs (Sauser et al., 2010). In IRL, level 8, a system-

level demonstration in the relevant environment can

be performed (the system is laboratory-test proven).

Level 9 denotes that the integrated technologies are

being successfully used both in the system

environment and operations [see Tan et al., (2011)].

In this study, the term “resilience” following

with Latin world “resilier” was extended as the

study of proactive-response ability and learning to

rebound, recover or jump back in the addressed

critical fields of national and cross-border decision

process systems and models. Here, the term

“resilience” can be address foremost to an ability of

critical, institutional, organizational, hardware,

software or operative service-systems to mitigate the

severity and likelihood of failures or losses, to adapt

to changing condition, and respond appropriately

after the evidence of failure, fact-finding, proactive

preparedness, consideration of response, and

scenario-based alignment and progress of action

competencies. Note literature: Resilience

Engineering (Atooh-Okine, 2016) and viewpoints of

robustness, persistence and resilience (Kott and

Abdelzaher, 2014).

3 METHODOLOGY

First, we decide whether to continue with a case

analysis (Pirinen, 2014) or cross-case analysis

according to (Patton, 1990). The first two pilot

studies (Pirinen et al., 2014) were conducted on

integration projects in the context of industrial

solutions and operative systems: Industrial System

Projects (Sivlén and Pirinen, 2014) and Operative

System Projects (Mantere and Pirinen, 2014).

We begin with a case analysis, which involved

writing a case study for each integrated unit. These

results are documented and comprise a research data

continuum [according to the Art of Case Study

Research (Stake, 1995) and the description of

multiple cases in (Yin, 2009)]. A description of our

overall continuum of R&D based environments and

data collection of externally funded projects between

2007 and 2017 is briefly introduced in the Table 1.

As a research continuum, this study employs a

complementary multiple-case analysis, which means

grouping together answers to various common

questions and analysing different perspectives on

central issues as resilience themes in (n=3) projects

EU_CISE, MARISA and #WINLandFI.

A summary list of research attributes was made

to validate and describe the methodological rigor in

the performed case study analysis (Dubé and Paré,

2003). While the level of achieved methodological

rigor has been used in different cases with respect to

specific attributes, the overall assessed rigor can be

still extended and furthered [cf. (Davison et al.,

2004)].

In this study, the multiple-case study approach

was used; the method is well explained in many

references, e.g., the case research strategy in studies

of information systems; building theories from case

study research (Eisenhardt, 1989); case studies and

theory development in the social sciences;

qualitative data analysis; the real world research

(Robson, 2001); and “case study research design and

methods” (Yin, 2009).

Table 1: A continuum of externally funded R&D.

R&D Project

Funding

RIESCA

SF-TEKES-SEC 2007-2013

MOBI

SF-TEKES-SEC 2007-2013

PERSEUS

EC-FP7-SECURITY-261748

AIRBEAM

EC-FP7-SECURITY-261769

ABC4EU

EC-FP7-SECURITY-312797

EU_CISE_2020

EC-FP7-SECURITY-608385

MARISA

EC-H2020-740698

#WINLandFI

SF-ACADEMY-SRC-303623

According mainly to Dubé and Paré (2003), the

main research attributes of this study are as follows:

1) title of the study: Towards Common Information

Systems Maturity Model: Resilience Readiness

Levels (ResRL); 2) research questions: ‘How can

ResRL metrics be understood in the domains of EU

CISE 2020 (information sharing), MARISA (data

fusion) and #WINLandFI (resilience and learning)”;

3) unit of analysis (UoA): an experience of samples

of resilience aspects of information systems

integration and data fusion cases which are

implemented, well documented and experienced; 4)

importance of the study: contributes to research on

information systems maturity, ResRL metrics and

related development of the ISO/DIS 16290 standard

series in EU CISE 2020, MARISA and

#WINLandFI projects; 5) methodological focus:

discovery of a continuums of case study analysis,

including triangulation (Campbell and Fiske, 1959)

and final cross-analysis; 6) analysis form: mainly a

qualitative analysis, saturation and triangulation

(Patton, 1990); 7) research target: information

service-system standardization and dissemination; 8)

data collection extensions and methods: MARISA

strategy canvas (n=38 participators and n=4 parallel

sessions) graphical canvas representations produced

(n=4) of high-value elements of authorities and

stakeholders that connects determination of

development targets, purchase choices and

continuums for utilization of innovative data fusion

functionalities, product and service; and 9) the

Academy of Finland Strategic Research Security

Programme namely From Failand to Winland

(#WINLandFI) data collection of co-creative work

including n=62 stakeholders and n=82 documents.

4 RESEARCH FINDINGS

In this operative environment, described in Figure 1,

information systems maturity validation is

understood as an approach that an individual

institution with respect to a specific validation

depends on, for example, the rules, guidance,

regulation, legislation, standards, agreements,

adoption model, best practices, ethical-legislation

codex, and characteristics of the system, which

aspects are then validated as an obligatory

prerequisite for activation.

Before activation, the validation processes are

used to determine whether the improved or

developed service or product meets the requirements

of the activity and whether the service or product

satisfies its intended trust-based use, collectively

agreed and with understood needs. The validation

processes have similarities with methodological

validation in a grounded approach and especially in

a triangulation.

Study revealed that there are certain similarities

between the activities performed in practical

validation and the type of documented information

produced for the validity of integrated information

systems. Obtained understanding of these practices

case analysis addressed to the canonical documents

and standards accumulated in the practices of the

actors in question and their operative experiences

continued with CANVAS settings. As examples of

such experienced documents included following:

users and stakeholders needs; requirements and

specifications; field regulations; validation plans;

project plans; supplier audit reports; functional

specifications; design specifications; task reports;

risk assessments; infrastructure and architecture

experiences; operational qualifications; standard

operating procedures; performance qualification;

security qualification; and validation descriptions,

plans and reports.

Study discovered that the term “resilience”,

“functionalities of resilience” and “resilient

learning” are depending on case, evolution path,

institution, cultural and development paths, event

mechanisms, integrations, and applied technology.

Here, the term “resilience” concentrated to a

proactive view of response design and achievements

of surviving capabilities for unexpected changes and

manners to enhance the capability at all levels of

concept of operation (CONOPS) and event

mechanism to create adaptive decision-making paths

that are robust yet flexible. See the outcomes as

aspects of ResRL proposal description of event

mechanisms in comprised to Figure 6.

Figure 6: Proposal of ResRL metrics.

The operative focus of the term “resilience” was in

monitoring and revising risk models and using of

resources proactively in the face of disruptions or

pressures of ongoing activities such as control,

operations, production, learning, service, or trade-

industry interactions.

The term “resilience” addressed also to an ability

to recover from, or building new positions to,

misfortune or adaption of mandatory change.

Aspects of “resilience” included typically four

aspects: 1) proactive plan and prepare, 2) absorb

disturbance, 3) recover from, and 4) adapt to known

or unknown threats.

Here, outcome for genealogies of the term

“resilience”: empirical and multidisciplinary R&D

results contributed rather to practical-operational

basis and associated necessitate revisions of its

theoretical views such as modular strategy: the

second level of ResRL proposal describes these

factors for modularity in Figure 6.

According to feedback and lessons learnt so far:

study exposed advantages and challenges towards

standardization and maturity validation, mainly

related to the ISO DIS 16290 and authority-based

decision-making interconnections and mechanisms.

Development of ResRL metrics is promising area

of maturity, as remark for future, more studies for

scaling ResRL to nine level model as compatible

with TRL and IRL metrics is needed, hence, it can

make more balanced for comparisons to the overall

scale of information systems maturity metrics.

It is noteworthy that the proposed ResRL metrics

are challenging in global procurement management

such as in national-international agreements and

descriptions of work. Then, more fine grained

descriptions and shared understanding for pre-

operational validation of ResRL metric and

resilience functionalities are needed, such as

terminology development settings in way of a web

ontology and resource description languages.

REFERENCES

Aanestad, M., Jensen, T. B., 2011. Building nation-wide

information infrastructures in healthcare through

modular implementation strategies. Journal of

Strategic Information Systems, 20, 161-176.

Atooh-Okine, N. O., 2016. Resilience Engineering Models

and Analysis. New York: Cambridge University Press.

Beasley, J. S., 2009. Networking, Boston: Pearson

Education, 2nd edition.

Campbell, D. T., Fiske, D. W., 1959. Convergent and

discriminant validation by the multitrait-multimethod

matrix. Psychological Bulletin, 56, 81-105.

Davison, R. M., Martinsons, M. G., Kock, N., 2004.

Principles of canonical action research. Information

Systems Journal, 14, 65-86.

Dubé, L., Paré, G., 2003. Rigor in information systems

positivist case research: Current practices, trends, and

recommendations. MIS Quarterly, 27(4), 597-635.

Eisenhardt, K. M., 1989. Building theories from case

study research. Academy of Management Review,

14(1), 532-550.

Eisner, H., 2011. Systems engineering: Building

successful systems. San Rafael, California: Morgan &

Claypools.

Hanseth, O., Lyytinen, K., 2010. Design theory for

dynamic complexity in information infrastructures:

The case of building internet. Journal of Information

Technology, 28, 1-19.

Kott, A., Abdelzaher, T., 2014. Resiliency and Robustness

of Complex Systems and Networks. In Adaptive,

Dynamic, and Resilient Systems. Boca Raton: Taylor

& Francis Group, pp. 67-85.

Luna, S., Lopes, A., Tao, H., Zapata, F., Pineda, R., 2013.

Integration, verification, validation, test, and

evaluation (IVVT&E) framework for system of

systems (SoS). Procedia Computer Science, 20, 298-

305.

Mantere, E., Pirinen, R., 2014. Utilization of the

Integration Readiness Level in Operative Systems.

Proceedings of IEEE World Engineering Education

Forum (WEEF-2014), Dubai, United Arab Emirates,

726-735.

Patton, M., 1990. Qualitative evaluation and research

methods, London: Sage Publications, 2nd edition.

Peterson, L. L., Davie, B. S., 2012. Computer networks: A

system approach, Burlington: Elsevier, 5th edition.

Pirinen, R., 2014. Studies of Integration Readiness Levels:

Case Shared Maritime Situational Awareness System.

Proceedings of the Joint Intelligence and Security

Informatics Conference (JISIC-2014). The Hague,

Netherlands, 212-215.

Pirinen, R., Sivlén, E., Mantere, E., 2014. Samples of

Externally Funded Research and Development

Projects in Higher Education: Case Integration

Readiness Levels. Proceedings of IEEE World

Engineering Education Forum (WEEF-2014), Dubai,

United Arab Emirates, 691-700.

Robson, C., 2002. Real world research, Oxford: Blackwell

Publishing, 2nd edition.

Sauser, B., Gove, R., Forbes, E., Ramirez-Marquez, J.,

2010. Integration maturity metrics: Development of an

integration readiness level. Information Knowledge

Systems Management, 9(1), 17-46.

Sauser, B., Verma, D., Ramirez-Marquez, J., Gove, R.,

2006. From TRL to SRL: The concept of systems

readiness levels. Conference on Systems Engineering

Research, Los Angeles.

Sivlén, E, Pirinen, R., 2014. Utilization of the Integration

Readiness Level in the Context of Industrial System

Projects. Proceedings of IEEE World Engineering

Education Forum (WEEF-2014), Dubai, United Arab

Emirates, 701-710.

Stake, R., 1995. The art of case study research, Thousand

Oaks: Sage Publications.

Tan, W., Ramirez-Marquez, J., Sauser, B., 2011. A

probabilistic approach to system maturity assessment.

Systems Engineering, 14(3), 279-293.

Yin, R. K., 2009. Case study research design and methods,

Thousand Oaks: Sage Publications, 4th edition.

Zimmermann, H., 1980. OSI reference model: The ISO

model of architecture for open systems

interconnection, IEEE Transactions on

Communications, 24(4), 425-432.