A Literature Review of Evaluation Approaches for Cyber-Physical

Production Systems

Hector Hostos

1,2

, Oscar Avila

and Virginie Goepp

Department of Systems and Computing Engineering, School of Engineering, Universidad de los Andes, Bogota, Colombia

ICube Laboratory, INSA Strasbourg, Strasbourg, France

Keywords: Manufacturing System, Cyber-Physical Production System, Industry 4.0, Evaluation, Meta-Model.

Abstract: The role of Cyber-physical production systems (CPPSs) as Industry 4.0 enablers has raised the interest to

upgrade legacy production systems. However, manufacturers face uncertainty when assessing if the

transformation process is worth it. In this context, the aim of this study is to review the works in the existing

literature that approach the evaluation of CPPSs in a context of production systems’ transformation. To do so,

we adopted a systematic literature review process that comprises the development of a framework of six

review questions that help us to analyze and characterize the literature found. From the literature review, this

paper presents a conceptual model that aims at establishing the basis for a complete approach for CPPS

evaluation.

1 INTRODUCTION

Production systems have been the core of the

manufacturing industry since the appearance of the

steam machine that set off the first industrial

revolution back in the 18

century. Later innovations

like electricity, production lines and information

technologies enabled a continuous evolution in

industry. As of today, technological breakthroughs

like cloud computing, data analytics, the Internet of

things (IoT) and artificial intelligence (AI) have

paved the way for the fourth industrial revolution,

also known as Industry 4.0, in which the fusion of the

physical and virtual worlds has been possible, in part,

by Cyber-Physical Systems (CPS) (Kagermann et al.,

2013). When a CPS is applied to production

environments, we refer to it as a Cyber-Physical

Production System (CPPS) (Wang et al., 2015). We

conceive a CPPS according to the definition of X.

Wu, (2022) as: “A combination of technological

agents, IT agents and humans, collaborating within a

synergistic production environment to carry out

technical, decision-making, or cognitive tasks

autonomously, using the best capabilities of each kind

of agent involved”.

Nowadays, the ever-increasing business demands

like the need for customized products or the decrease

in the product lifecycles (Neugebauer et al., 2016)

drives the transition from legacy manufacturing

systems to CPPSs as a main stake for manufacturers.

However, such transition must be well-founded since

it usually implies high upfront investments for

organizations, development time and it is surrounded

by considerable uncertainty (X. Wu, 2022). Therefore,

the evaluation of the CPPSs to be implemented plays a

crucial role in the transformation context to reduce

uncertainty and mitigate risks. In this context, we

formulate the following research question: “How do

existing methods evaluate a CPPS along its lifecycle,

especially during its design?”. To answer this question,

we perform a systematic literature review on the

existing approaches to evaluate CPPSs to identify the

gaps in this research subject.

This work is structured as follows: Section 2

presents a systematic literature review of the evaluation

approaches for CPPSs. This leads us, in Section 3, to

work out a conceptual model that allows to fulfill the

gaps identified in the analysis of the literature review.

Finally, Section 4 draws some conclusions and sets out

a research agenda for the future.

2 STATE OF THE ART

The literature review followed in this work is based

on the framework proposed by vom Brocke et al.,

(2009), which consists of five phases illustrated on

Fig. 1. Likewise, each phase is explained below.

642

Hostos, H., Avila, O. and Goepp, V.

A Literature Review of Evaluation Approaches for Cyber-Physical Production Systems.

DOI: 10.5220/0011985200003467

In Proceedings of the 25th International Conference on Enterprise Information Systems (ICEIS 2023) - Volume 2, pages 642-650

ISBN: 978-989-758-648-4; ISSN: 2184-4992

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

Figure 1: BPMN illustration of the systematic literature review process followed in this work.

2.1 Definition of Review Scope

The first phase comprises 6 stages, each one is

intended to define one constituent characteristic of the

review: focus, goal, organization, perspective,

audience, and coverage. Our focus is the review of

research outcomes on CPPSs evaluation. The goal is

to describe the state of the art in the evaluation of

CPPSs. The organization is given by a framework of

six review questions to be presented further on.

The review presents different interpretations of

the literature in a neutral perspective. Regarding

the audience, this work is meant to be consulted

by specialized scholars in the field of

production/manufacturing systems. The coverage is

exhaustive, as we are covering most of the literature

about CPPSs evaluation.

2.2 Conceptualization of Topic

The definition of key terms is essential for

conceptualization of the topic to be reviewed. In our

case, 20 keywords are defined and grouped by their

meaning similarity. The first group refers to the

production systems themselves and the different

forms that could be found in the literature: CPPS,

Cyber-physical production systems, CPS, Cyber-

Physical Systems, Manufacturing Systems,

Production system. Next, 6 keywords compose the

second group that refers to evaluation and some other

related concepts: Measure, Measurement, Evaluate,

Evaluation, Assess, Assessment. The third group of

keywords implies the potential transformation of the

production system towards a target stated that can be

named in different ways: Requirement, Request,

Requisite, To-be, Target State, Required State.

Finally, the fourth group refers to the representation

of the designed production system as it could be a

model or a meta-model: Model, Meta-model.

2.3 Literature Search

Given the key terms definition and grouping, we

make use of AND / OR operators to construct a query

that comprises the relevant literature to our topic. For

this work, we employed the Scopus database and

search engine. The query assembly is shown below:

TITLE-ABS-KEY ( ( measure OR measurement

OR evaluate OR evaluation OR assess OR

assessment ) AND ( requirement OR request OR

requisite OR {to-be} OR {required state} OR

{target state} ) AND ( model OR {meta-model} )

AND ( CPPS OR "Cyber-physical production

system" OR "Cyber-physical system" OR

"production system" OR "manufacturing system") )

AND ( LIMIT-TO ( SUBJAREA , "COMP" ) ).

The operator TITLE-ABS-KEY() instructs the

search engine to look for the terms in parenthesis

within the title or the abstract or the keywords of each

of the articles that compose the database. The

operator LIMIT-TO() constrains the results to the

domains of Computer Science only. As a result, 645

articles were listed and passed through a three-stage

filtering process as follows: In the first filtering stage,

we proceeded to assign one of the following

categories to each title according to its relevance and

relationship to our research subject: related (28

articles), somewhat related (49 articles) and not

related (568 articles). Only the articles assigned to the

first and second categories passed to the next stage.

In the second filtering stage, we analyzed the abstract

of each article and selected only the ones most related

to the research subject which accounted for 15

articles. The last filtering stage consisted of

examining their section titles, tables, and figures. We

selected only 3 articles which were most related to the

research subject.

Additionally, 7 works were found by means of the

backward and forward search. The former consists of

looking for relevant articles within the references of

the articles provided by the keyword search. The

latter consists of looking for which relevant articles

cite the articles provided by the keyword search.

Backward and forward searches do not take place in

the first filtering stage since only the title is analyzed

there, not the references. As a result, from the

literature search, a total of 9 articles and one thesis

document are passed for the analysis stage. Table 1

summarizes the results of each filtering stage with

respect to the type of search performed.

A Literature Review of Evaluation Approaches for Cyber-Physical Production Systems

643

Table 1: Number of works from each filtering stage.

Search Type 1

Filter 2

Filter 3

Filter

Keyword Search 77 15 3

Backward Search N/A 5 3

Forward Search N/A 10 4

2.4 Analysis and Synthesis of

Literature

A framework with six review questions is proposed

to review the final set of works (see Table 2).

Table 2: Framework of review questions.

No. Review Question

In which stage of the CPPS lifecycle is the

evaluation performed?

Which criteria are used to perform the

evaluation?

Is the system performance improved through

iterative evaluation?

What kind of notation is used to represent the

roduction s

stem?

Does the article consider the transition from

legacy systems to CPPSs?

Does the article provide a software tool for

stems evaluation?

These questions were set up by identifying the most

relevant dimensions that characterize the evaluation

of a CPPS. The first question refers to the stage where

the evaluation occurs. This is important considering

that, for example, evaluating a production system

while it is being designed differs from evaluating a

system after it has been implemented and it is

operating. The second question examines the criteria

of the evaluation, namely, what it is being measured

and how. Given that an evaluation process expresses

the performance of a system, the third question

determines whether the results of such evaluation are

feedback and used for the sake of system

improvement. The fourth question looks into the

notation, which refers to the way that a model

formally portrays a production system. The fifth

question goes beyond CPPS evaluation as it

investigates which research works consider a

transition from a legacy system to a CPPS. Finally,

the sixth question examines which kind of software

tools have been implemented in the field of CPPS

evaluation. The analysis performed below is

summarized in Table 3.

2.4.1 In Which Stage of the CPPS Lifecycle

Is the Evaluation Performed?

A system lifecycle can be divided into three main

stages according to system engineering principles

(Kossiakoff et al., 2020): The first stage is the concept

development which formulates and defines the

system concept that best satisfies a need. The next

stage is the engineering development which takes the

system concept into hardware and software designs.

The last stage is the post-development stage which

covers the production, deployment, operation and

support of the system. We analyzed the literature to

assign each work within one of the stages and analyze

during which stage evaluation takes place.

With respect to this lifecycle, we found one work

that evaluates the system even before it has been

which the CPPS can be developed (Bunte et al.,

2019). Other works evaluate the production systems

during their design which belongs, in turn, to the

engineering development stage (B. Wu, 1992; X. Wu,

2022). Orellana & Torres, (2019), evaluate the system

during its design and also during a period of 5 months

of its operation. Therefore, it is classified in both

engineering development and post-development

stages. The work of S. Lee & Ryu, (2022), proposes

to perform the evaluation over a digital twin or a

simulation of the system before it is implemented. For

the sake of classification, we consider this simulation

as one phase of the engineering development stage in

which the simulation of the system is part of its design

before going into production. Most of the literature

performs the assessment over production systems

already implemented, namely, during their operation.

For instance, Coelho et al., (2022), apply a survey of

questions on three different manufacturing plants that

are in operation to evaluate their maturity level. These

plants render their services to different industries. On

the other hand, Arjoni et al., (2018); Lins & Oliveira,

(2020), evaluate the performance of assemblies of

robotic arms just after implementation.

From this analysis, we classify the literature by

the criterion Evaluation in the lifecycle of the CPPS

in Table 3.

2.4.2 Which Criteria Are Used to Perform

the Evaluation?

All the works reviewed make use of a certain type of

criteria to evaluate their corresponding systems. We

identified that those types of criteria are qualitative

and/or quantitative. The latter is classified, in turn,

into numeric indicators, and standardized KPIs (Key

ICEIS 2023 - 25th International Conference on Enterprise Information Systems

644

Performance Indicators). The qualitative evaluation

describes the characteristics of the system under

evaluation in terms of natural language. For instance,

Bunte et al., (2019), assess the systems by a survey of

13 questions, each one is related to a requirement.

Some of those requirements are: does the system

learn from experiences? And does it apply the action

on the controller? Similarly, Arjoni et al., (2018),

formulates three questions to evaluate a system: is

there communication between the real machinery and

the virtualized plant? Is there communication among

machinery? and, how is the operation stability of the

new elements added to the system? Coelho et al.,

(2022), propose a framework of 42 questions grouped

into 15 categories.

Although they do not reveal the specific

questions, some of the categories are the organization

of the machines in a network, dashboard and level of

autonomy. X. Wu´s qualitative approach (X. Wu,

2022) presents a framework of 20 questions

categorized into three dimensions: technological,

informational, and organizational. Some of the

questions are: how is the connectivity of machine in

the shop floor? and how is data collected?

The quantitative evaluation is based on

mathematical modeling that provides data that can be

expressed in numbers. The sub-category numeric

indicators refers to indicators that are application-

specific, namely, a production system to be evaluated

is previously analyzed and some performance

indicators are identified and measured for such

specific system. However, those are not referred as

KPIs since they are not directly linked to a target

according to the definition of KPIs (Parmenter, 2015).

One example of this approach is presented by Khan

et al., (2020), who measure only one indicator that is

the latency in the communication between the sensors

and their supervising application when migrating an

industrial SCADA system to the cloud. Likewise,

Lins & Oliveira, (2020), measure two indicators: the

energy consumption and response time of a

manufacturing system. B. Wu, (1992), does not

suggest any specific technical indicator. However, he

considers the internal interest rate as a financial

indicator to consider when implementing a

manufacturing system. Rinker et al., (2021), only

Table 3: Literature classification after applying the framework of review questions.

Criteria Classification References

Evaluation in the

lifecycle of the

CPPS

Concept Development (Bunte et al., 2019)

ngineering Development (Orellana & Torres, 2019; B. Wu, 1992; X. Wu, 2022)

ost-development

(Arjoni et al., 2018; Coelho et al., 2022; Khan et al., 2020; S. Lee & Ryu,

2022; Lins & Oliveira, 2020; Orellana & Torres, 2019

)

Type of

evaluation

criteria

Qualitative (Arjoni et al., 2018; Bunte et al., 2019; Coelho et al., 2022; X. Wu, 2022)

Quantitative

umeric

ndicators

(Khan et al., 2020; Lins & Oliveira, 2020; B. Wu, 1992)

Standardized

(S. Lee & Ryu, 2022; Orellana & Torres, 2019; Rinker et al., 2021; X.

Wu, 2022)

Improvement of

the system

performance

through iterative

evaluation

terative improvement (Orellana & Torres, 2019)

on-iterative improvement

(Arjoni et al., 2018; Bunte et al., 2019; Coelho et al., 2022; Khan et al.,

2020; S. Lee & Ryu, 2022; Lins & Oliveira, 2020; Rinker et al., 2021; B.

Wu, 1992; X. Wu, 2022)

Type of notation

to represent the

system

-hoc

(Arjoni et al., 2018; Bunte et al., 2019; Khan et al., 2020; S. Lee & Ryu,

2022; Lins & Oliveira, 2020; Orellana & Torres, 2019

)

ML (X. Wu, 2022)

ML (Rinker et al., 2021)

Upgrade from

legacy systems

proposed

(Arjoni et al., 2018; Khan et al., 2020; S. Lee & Ryu, 2022; Lins &

Oliveira, 2020; Orellana & Torres, 2019; B. Wu, 1992; X. Wu, 2022)

o (Bunte et al., 2019; Coelho et al., 2022)

Software tool

implementation

odeling tool (Rinker et al., 2021)

Simulation tool (Ferrer et al., 2018)

Custom tool (J. H. Lee et al., 2018)

A Literature Review of Evaluation Approaches for Cyber-Physical Production Systems

645

measure the time as an indicator of performance of a

graphical user interface that they develop to model

production systems. Regarding the standardized

KPIs, X. Wu, (2022), makes use of 26 KPI that come

from the ISO22400 standard (International

Organization for Standardization, 2014). The KPIs

are calculated from a set of supporting data which are

directly measured in the production system. Orellana

& Torres, (2019), also make use of ISO22400. In the

case study that they develop, they select 8 specific

KPIs. S. Lee & Ryu, (2022) use another kind of KPIs

that are oriented to sustainability and are stated in the

Global Reporting Initiative (GRI, 2016).

From this analysis, we classify the literature by

the criterion Type of evaluation criteria in Table 3.

2.4.3 Is the System Performance Improved

Through Iterative Evaluation?

From the literature reviewed, we found three works

that include iteration at least in one point of its

process. We examine below if the iteration in such

works is linked to their corresponding evaluation

processes.

Orellana & Torres, (2019), introduce a 4-phase

maturity framework for Industry 4.0: the first one

refers to isolated digital applications. In the second

one, the enterprise is capable of integrating and

digitizing its machinery, applications and processes.

In the third phase, both suppliers and partners are

digitally integrated by a common and central

architecture. The fourth phase refers to a factory

100% digital. They propose an 8-step iterative

procedure to transit from phase 1 to phase 2. Within

the reviewed articles, this is the only work that

integrates the evaluation step into the iterative process

which allows the manufacturing system to improve its

performance continuously.

The proposal of X. Wu, (2022), consists of the

following steps: modeling, transformation, and

evaluation of the system. Although the modeling and

transformation steps imply iterations, the evaluation

step only happens once in the process, and it is not

linked to the transformation step that takes place

previously. S. Lee & Ryu, (2022), enhance an

existing production system by adding reconfigurable

capabilities. They propose three subprocesses that

must be carried out sequentially. Evaluation takes

place in the third step. Once the three subprocesses

have run, the derived solution is passed through

simulations. If simulation succeeds, then the new

model is configured in the real facilities. Therefore,

no explicit iteration is proposed.

From this analysis, we classify the literature by

the criterion Improvement of the system performance

through iterative evaluation in Table 3.

2.4.4 What Kind of Notation Is Used to

Represent the Production System?

Most of the articles make use of an ad-hoc notation to

represent the production systems, like it is the case of

(S. Lee & Ryu, 2022). This means that no standard

was followed when portraying a representation of the

systems. Nevertheless, two works use standardized

modeling languages. Rinker et al., (2021) make use

of AutomationML (AML) as an XML-based

standardized data exchange format for the storage and

exchange of modeling hierarchical structures

common to production systems. Likewise, the

metamodel that X. Wu, (2022), proposes to

instantiate is expressed in UML class diagrams.

Therefore, these two aforementioned works represent

the exceptions in terms of the use of a notation for

systems representation. From this analysis, we

classify the literature by the criterion Type of notation

to represent the system in Table 3.

2.4.5 Does the Article Consider the

Transition from Legacy Systems to

CPPSs?

Upgrading legacy systems instead of investing in

brand-new systems from scratch represents huge

benefits for organizations (di Carlo et al., 2021).

Therefore, upgrading proposals were frequently

found in the literature.

One of the articles´ case study tackles the

modernization of an assembly line with more than 47

years of operation (Orellana & Torres, 2019). Arjoni

et al., (2018), propose some retrofit techniques at

machinery level to allow old automation and

mechatronic components such as robotic arms and

CNC machines to be adapted to advanced

manufacturing features like communication,

intelligence and sensing capabilities with low

implementation costs. Lins & Oliveira, (2020),

propose a process for upgrading equipment based on

the widely used architecture RAMI 4.0. It comprises

defining the requirements, components, and

technologies necessary to retrofit the industrial

equipment. Khan et al., (2020), take a legacy

industrial SCADA system which operate entirely

local and deploy part of its information systems to the

cloud by implementing secure communication links.

X. Wu, (2022) exhibits a case study of implementing

RFID technology to enable autonomy in a simulated

assembly line. Thus, the products could communicate

ICEIS 2023 - 25th International Conference on Enterprise Information Systems

646

with workstations to decide their next operation.

S. Lee & Ryu, (2022), do not explicitly propose a case

study of a transformation method from a legacy

system into a CPPS. Instead, they propose how to

upgrade a legacy architecture of a CPPS into a self-

reconfigurable one. Finally, B. Wu, (1992), proposes

a methodology that helps a manufacturing system

designer to decide which aspects of advanced

manufacturing technology are required and how they

should be integrated in a legacy system.

From this analysis, we classify the literature by

the criterion Upgrade from legacy system proposed in

Table 3.

2.4.6 Does the Article Provide a Software

Tool for Systems Evaluation?

From the articles analyzed, it was very uncommon to

find a software implementation, however, some

related developments were found. For example, in the

field of modeling tools, Rinker et al., (2021) coded a

prototype that graphically models production systems

whose components belong to different disciplines of

science, for instance, electrical, mechanical or

biological components. As a simulation tool devised

to serve any type of manufacturing application, Ferrer

et al., (2018) developed the FASTory Simulator

platform. In the field of custom software

implementations, J. H. Lee et al., (2018), showcase a

CPPS dashboard that supports the prediction and

operation control of a plant in the metal casting

industry. Although the aforementioned tools are

related to the field of manufacturing systems, they do

not explicitly address the evaluation of such systems.

From this analysis, we classify the literature by

the criterion Software tool implemented in Table 3.

2.5 Gap Analysis

Once the framework of review questions has been

answered in its entirety, it is time to recall the research

question posed: “How do existing methods evaluate a

CPPS along its lifecycle, especially during its design?”.

Thanks to the characterization of the subject performed

with the framework, we are able to identify how well

this problem is addressed by existing propositions from

different perspectives as follows:

The literature showed that the evaluation of CPPSs

can be performed at any given stage of their lifecycle.

However, most of the works focus on the operation

stage (post-development) underestimating the design

stage (engineering development). It is furthermore

necessary to address the focus on the analysis and

design stages since the tuning and changes over the

system are less expensive at those points.

Most of the reviewed works opted for a

quantitative evaluation. This reveals the advantage

that quantitative evaluations have over qualitative

ones in terms of accuracy and objectivity. However,

among the quantitative approaches that relied on

KPIs, very few works were found to set target values

for them. Therefore, it is convenient to set target

values for each KPI considered.

The fact that only one work utilizes the evaluation

results to iteratively improve the production system,

reveals an important gap in the review. Iteration-

based evaluation would bring the benefits of feedback

since the early stages of the system lifecycle. It is

essential for future developments on this subject to

encourage the system improvement by iterating on

the evaluation results.

We can affirm to the best of our knowledge that,

as of today, no software tool has been implemented to

aid the model-based evaluation of CPPSs. The

development of a software tool would be particularly

useful for the researchers working on the CPPS

evaluation subject.

Given that the comprehensive analysis and

synthesis on evaluation of CPPSs have been

performed, we see the need to organize all the concepts

reviewed so far in a formal manner. Consequently, in

the following section we introduce a conceptual model

for the evaluation of CPPSs as a preliminary approach

to develop a solution that bridges the gaps previously

identified in the literature review.

3 A CONCEPTUAL MODEL FOR

EVALUATION OF CPPS

3.1 Model Description

Figure 2 shows a meta-model that is composed by

object classes that represent the concepts and

relationships that must be considered for further

developments on this subject. The description of the

meta-model is as follows:

A Production_system is identified by its name. It is

represented in a Model by instantiating a Meta-model.

In order to represent the Model, a Notation must be

adopted. The Notation could be AD-HOC for each

specific system or it could be STANDARDIZED. As

we are considering the transformation of production

systems, two child classes can be inherited from the

class Model: Legacy_System_Model and

CPPS_Model. The latter refers to the upgraded system

after a transformation sequence has been applied.

A Literature Review of Evaluation Approaches for Cyber-Physical Production Systems

647

Figure 2: Meta-model for the evaluation of CPPSs.

Likewise, the Production_system is developed

through an associated Lifecycle which, in turn, is

composed of one or more stages represented by the

class Stage. The Production_system is also able to be

evaluated. Such Evaluation is represented by a class

that has two attributes. The first one defines the type

of evaluation which is represented by the literals

QUANTITATIVE and QUALITATIVE from the

enumeration EvaluationType. The second attribute

determines whether the evaluation is iterative or not

by means of a boolean value. Evaluation is associated

with the class Stage because, it may be applied to

every stage of the production system lifecycle.

The evaluation procedure can be broken down in

several steps by means of the class Step. One step may

use indicators to measure the performance of a

production system. The class Indicator contains the

attribute formula, which is the mathematical

representation of the indicator. One kind of indicator

is the Key Performance Indicator (KPI) which differs

to a regular indicator in the fact that it is linked to a

target value and may belong to an industry standard.

Therefore, the class KPI extends the class Indicator

and adds the attributes target and standard.

3.2 Discussion

The value of the conceptual model introduced above

lies in how it can be used to address the review

questions in a way that no other proposition did. We

discuss below the case for each one of the review

questions. Regarding the first question, in which stage

of the CPPS lifecycle is the evaluation performed?

The conceptual model associates the evaluation of the

system with any of its lifecycle stages, which means

that a system could be evaluated even before it is

designed (Bunte et al., 2019) and/or during its

operation (Khan et al., 2020). On the contrary, the

reviewed works focus on one specific stage. With

regard to the second question, which criteria are used

to perform the evaluation? The conceptual model

considers not only a qualitative approach but also a

quantitative one with the possibility of using basic

numeric indicators or standardized KPIs. This

represents a step forward in comparison to existing

works in terms of exploiting the use of KPIs.

Standardized KPIs, may be used to compare the

performance of two or more production systems in the

same industry. With respect to the third question, is

the system performance improved through iterative

evaluation? The model includes a boolean attribute

called iterative in the class Evaluation that determines

if the performance of the system improves by iterating

on the evaluation. A true value would mean that the

evaluation results are feedback for system

improvement, like the work of Orellana & Torres,

(2019). Concerning the fourth question, what kind of

notation is used to represent the production system?

The conceptual model includes the class Notation

which is linked to the class Model, meaning that every

single model should have a notation defined as

proposed by Rinker et al., (2021). Respecting the fifth

question, does the article consider the transition from

legacy systems to CPPSs? The conceptual model

suggests two classes called Legacy_System_Model

and the CPPS_Model which extend the class Model

and enable any system to be upgraded unlike some

works reviewed (Bunte et al., 2019; Coelho et al.,

2022). Regarding the sixth question, does the article

provide a software tool for systems evaluation? The

conceptual model, in its entirety, sets the foundation

for the construction of a software tool that allows to

evaluate a CPPS in an intuitive manner.

ICEIS 2023 - 25th International Conference on Enterprise Information Systems

648

4 CONCLUSIONS AND FUTURE

WORK

This paper presents a literature review that identifies

and analyzes research works in the domain of CPPSs

evaluation by employing a framework of six review

questions. A conceptual model was proposed to fill the

gaps identified by the analysis of the literature. A

comprehensive discussion was given to show how the

conceptual model differs from existing propositions

and sets out a path towards an enhanced evaluation

method for CPPSs.

Concerning the research method, the review scope

defined in the first section allowed us to have the

review clearly characterized from the beginning. This

is typically a challenging task since literature reviews

can serve a wide range of very different purposes.

Although the number of articles found by the search

query was initially high, the final number of articles

reviewed was low. This means that the filtering

strategy helped to identify the relevant works hidden

within a large set of results. Likewise, the backward

and forward search helped to expand the final set of

articles. However, it is a matter for further

developments to formulate an enhanced query that

leads to a larger quantity of results.

As future work, the conceptual model can be used

as a base to propose a complete approach of CPPSs

evaluation that considers, for instance, the

improvement of the system by means of an iterative

evaluation method and the use of a custom set of

standardized KPIs. It could also be linked with

existing metamodel proposals like X. Wu, (2022). In

addition, the approach may propose a different

evaluation procedure for each stage of the lifecycle.

REFERENCES

Arjoni, D. H., Madani, F. S., Ikeda, G., Carvalho, G. D. M.,

Cobianchi, L. B., Ferreira, L. F. L. R., & Villani, E.

(2018). Manufacture equipment retrofit to allow usage

in the industry 4.0. Proceedings - 2017 2nd

International Conference on Cybernetics, Robotics and

Control, CRC 2017, 2018-Janua, 155–161.

https://doi.org/10.1109/CRC.2017.46

Bunte, A., Fischbach, A., Strohschein, J., Bartz-Beielstein,

T., Faeskorn-Woyke, H., & Niggemann, O. (2019).

Evaluation of Cognitive Architectures for Cyber-

Physical Production Systems. IEEE International

Conference on Emerging Technologies and Factory

Automation, ETFA, 2019-Septe, 729–736.

https://doi.org/10.1109/ETFA.2019.8869038

Coelho, M. A., de Oliveira, F. A., Dessimoni, L. H., &

Libório, N. S. (2022). Cyber-Physical Production

System Assessment Within the Manufacturing

Industries in the Amazon. International Journal of

Production Management and Engineering, 10(1), 51–

64. https://doi.org/10.4995/ijpme.2022.16130

di Carlo, F., Mazzuto, G., Bevilacqua, M., & Ciarapica, F.

E. (2021). Retrofitting a process plant in an industry 4.0

perspective for improving safety and maintenance

performance. Sustainability (Switzerland), 13(2), 1–18.

https://doi.org/10.3390/su13020646

Ferrer, B. R., Muhammad, U., Mohammed, W. M., &

Lastra, J. L. M. (2018). Implementing and visualizing

ISO 22400 key performance indicators for monitoring

discrete manufacturing systems. Machines, 6(3).

https://doi.org/10.3390/MACHINES6030039

GRI. (2016). Global Reporting Initiative. Gri 101:

foundation 2016 101. In GRI Standards (Vol. GRI101,

Issue 1).

International Organization for Standardization. (2014). ISO

22400-2:2014 Automation systems and integration —

Key performance indicators (KPIs) for manufacturing

operations management — Part 2: Definitions and

descriptions. https://www.iso.org/standard/54497.html.

Kagermann, H., Wahlster, W., & Helbig, J. (2013). Securing

the future of German manufacturing industry initiative

INDUSTRIE 4.0 implementing the strategic

Recommendations for Final report of the Industrie 4.0

Working Group. In Final Report of the Industrie 4.0

Working Group (Issue April).

Khan, R., McLaughlin, K., Kang, B., Laverty, D., & Sezer,

S. (2020). A seamless cloud migration approach to

secure distributed legacy industrial SCADA systems.

2020 IEEE Power and Energy Society Innovative Smart

Grid Technologies Conference, ISGT 2020.

https://doi.org/10.1109/ISGT45199.2020.9087760

Kossiakoff, A., Seymour, S. J., Flanigan, D. A., & Biemer,

S. M. (2020). Systems Engineering Principles and

Practice. In Systems Engineering Principles and

Practice. https://doi.org/10.1002/9781119516699

Lee, J. H., do Noh, S., Kim, H.-J., & Kang, Y.-S. (2018).

Implementation of cyber-physical production systems

for quality prediction and operation control in metal

casting. Sensors (Switzerland), 18

(5). https://doi.org/

10.3390/s18051428

Lee, S., & Ryu, K. (2022). Development of the Architecture

and Reconfiguration Methods for the Smart, Self-

Reconfigurable Manufacturing System. Applied

Sciences (Switzerland), 12(10). https://doi.org/10.3390/

app12105172

Lins, T., & Oliveira, R. A. R. (2020). Cyber-physical

production systems retrofitting in context of industry

4.0. Computers and Industrial Engineering, 139.

https://doi.org/10.1016/j.cie.2019.106193

Neugebauer, R., Hippmann, S., Leis, M., & Landherr, M.

(2016). Industrie 4.0 - From the Perspective of Applied

Research. Procedia CIRP, 57, 2–7. https://doi.org/

10.1016/j.procir.2016.11.002

Orellana, F., & Torres, R. (2019). From legacy-based

factories to smart factories level 2 according to the

industry 4.0. International Journal of Computer

A Literature Review of Evaluation Approaches for Cyber-Physical Production Systems

649

Integrated Manufacturing, 32(4–5), 441–451.

https://doi.org/10.1080/0951192X.2019.1609702

Parmenter, D. (2015). Key Performance Indicators:

Developing, Implementing, and Using Winning KPIs

Third Edition. In Key Performance Indicators.

Rinker, F., Waltersdorfer, L., Schüller, M., Biffl, S., &

Winkler, D. (2021). A Multi-Model Reviewing

Approach for Production Systems Engineering Models.

In Communications in Computer and Information

Science (Vol. 1361). https://doi.org/10.1007/978-3-030-

67445-8_6

vom Brocke, J., Simons, A., Niehaves, B., Riemer, K.,

Plattfaut, R., & Cleven, A. (2009). Reconstructing the

giant: On the importance of rigour in documenting the

literature search process. 17th European Conference on

Information Systems, ECIS 2009.

Wang, L., Törngren, M., & Onori, M. (2015). Current status

and advancement of cyber-physical systems in

manufacturing. Journal of Manufacturing Systems, 37,

517–527. https://doi.org/10.1016/j.jmsy.2015.04.008

Wu, B. (1992). Manufacturing Systems Design and

Analysis. In Manufacturing Systems Design and

Analysis. https://doi.org/10.1007/978-94-011-3128-5

Wu, X. (2022). Transformation of existing production

systems with integrated enterprise information systems

(EISs) into cyber physical production systems (CPPSs)

in the context of Industry 4.0. Hesam University.

ICEIS 2023 - 25th International Conference on Enterprise Information Systems

650