Towards an Evaluation Concept for Business Simulation Games:

Preliminary Work and Piloting in SAP ERP Teaching

Robert Häusler

, Malte Rathjens, Daniel Staegemann

and Klaus Turowski

Magdeburg Research and Competence Cluster Very Large Business Applications, Faculty of Computer Science,

Otto-von-Guericke University Magdeburg, Magdeburg, Germany

Keywords: Game-Based Learning, Business Simulation Games, Teaching, Evaluation, ERP.

Abstract: Business Simulation Games (BSGs) aim to simulate reality and impart knowledge as well as skills in a playful

way. To be able to verify the goal attainment, the first steps towards an evaluation concept were taken in this

paper. With the exemplary evaluation of Global Bike Go, a series of mini BSGs for SAP ERP teaching, initial

indications could be generated about what they (can) achieve. One certain finding is that the games are suitable

for beginners whereas the participants’ knowledge gain only shows tendencies. From the overall results,

development potentials for the BSGs as well as for the evaluation concept used could be identified. However,

due to the small sample and the limiting circumstances, further investigations have to be conducted. In this

context, the self-performed actions as well as interactions with other players as significant game elements

should be focused more, and especially the interdependency between the BSGs and other teaching materials

seems promising. Therefore, an interdisciplinary approach is desirable.

1 INTRODUCTION

Publications on gamification have been accumulating

in recent years (Fischer et al., 2017). Related concepts

such as serious games, especially business simulation

games (BSG), are also being researched in this

context (Faisal et al., 2022). Today, more than 600

business games and serious games are offered in

German-speaking countries alone (Blötz, 2015).

Although forms of case studies were the

prevailing method in the field of Enterprise Resource

Planning (ERP) teaching to impart application

knowledge at German universities, business games

are comparatively rarely used (Leyh, 2017). Possible

explanations for this could be, on the one hand, the

lack of availability of suitable BSGs and, on the other

hand, the controversial benefits of such tools.

However, there are already a few institutions that

have been drivers of past developments, also meeting

the current and expected demand. Among others, the

SAP University Competence Centers (UCC) in

Magdeburg and Munich are Education Service

Providers (ESP) focusing on SAP software (Prifti et

al., 2017; Häusler and Bosse, 2018), and serve as

https://orcid.org/0000-0002-2534-6070

https://orcid.org/0000-0001-9957-1003

exemplary providers for such BSGs. In particular,

they offer so-called teaching and learning

environments (TLEs) “which represent a broad

selection of teaching materials with a high practical

relevance in the field of enterprise software” (Reich

and Häusler, 2019).

Those TLEs usually consist of three elements:

(IT) system, teaching material, and model

organization (Häusler et al., 2019). In this context, the

UCCs emphasize the use of case studies as the

preferred method for teaching system-supported

(business) processes practically as well as

realistically and supplement them with slide sets and

hands-on exercises on the teaching material side. As

part of the portfolio expansion, the UCC Magdeburg

has started to offer three beta-version (mini) BSG

called “Global Bike Go” (GBGo). These are intended

to complement the existing materials in the

“Introduction to SAP S/4HANA” TLE, thus offering

learners methodological variety and contributing to

learning success. Their requirement profile is diverse:

They serve as an introduction to the respective

modules such as purchasing, production planning,

and sales, present an exemplary part of the respective

HÃd’usler, R., Rathjens, M., Staegemann, D. and Turowski, K.

Towards an Evaluation Concept for Business Simulation Games: Preliminary Work and Piloting in SAP ERP Teaching.

DOI: 10.5220/0012144300003552

In Proceedings of the 20th International Conference on Smart Business Technologies (ICSBT 2023), pages 94-103

ISBN: 978-989-758-667-5; ISSN: 2184-772X

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

(business) process, and enable participants to gain

their first experiences with the system’s user

interface. The BSGs have been implemented, tested,

and made available to the community, but to date,

they have not been used or evaluated by the UCC

itself in a teaching context.

In general, the evaluation of teaching materials,

especially of BSGs, is challenging due to their

diversity, and the measurability of learning progress

is complex and effortful because of its subjective

nature. Consequently, the following research

question (RQ) is addressed.

RQ: Which Aspects need to be Considered in a

Possible Evaluation Concept For BSGs?

The exemplary use of GBGo in a university

course generates initial indications about what the

single games (can) achieve. From the overall results

of this study, development potentials for the BSGs as

well as for the evaluation concept will be identified.

The paper at hand aims to do the preliminary work

while achieving the first findings within a pilot

project in ERP teaching. First, the state of the art is

shortly presented, before the basic concept for the

pilot (including subject, goals, and model) is

explained. The following sections describe the

exemplary study and discuss its results. In the end, the

further proceeding is depicted.

2 STATE OF THE ART

Views on whether and how simulation games

influence learning behavior and in which learning

environments they should be used have been

undergoing changes in recent years. It is presently

assumed that playful elements can increase the

motivation of participants (Jacob and Teuteberg,

2017). BSGs as a subset of serious games (Unger et

al., 2015) are understood as tools and used in the

educational context to utilize these effects and thus

impart knowledge (Leyh, 2017; Jacob and Teuteberg,

2017; Rubart and Hartweg, 2019; Matute and Melero,

2016; McGonigal, 2011; Weppel et al., 2012) but also

could enhance engagement, learning achievement,

and higher-order thinking skills (Huang et al., 2022).

At German universities, there are still

comparatively few BSGs used in the field of ERP

teaching and the focus lies on imparting knowledge

through lectures and case study-based hands-on

exercises (Leyh, 2017). An exception is the ERPsim,

which is used in the context of ERP teaching in higher

education. In this BSG, the focus is on economic

processes whereas ERP or SAP prior knowledge is

not necessary (Leyh, 2017; Rubart and Hartweg,

2019). However, ERPsim is fully system-integrated

and also presumably provides practical knowledge on

the use of SAP software. The execution of this BSG

generally leads to a positive response from its

participants (Leyh, 2017; Rubart and Hartweg, 2019;

Utesch et al., 2016). In his publication, Leyh (2017)

goes beyond this general assessment and examines

the knowledge gained by the participants. This is

compared with the knowledge gained from case

studies, whereby in some cases better results are

achieved with ERPsim.

GBGo as a BSG series is another example that can

be considered as a UCC proprietary development in

the context of ERP teaching. Although the universal

research results on BSG can also be assumed for

GBGo, it is unclear at this stage whether its targeted

teaching goals are being achieved. In particular, there

is a lack of systematically generated results with

which the games can be evaluated and, if necessary,

further developed on this basis.

3 CONCEPT

In the following concept section, GBGo is introduced

as evaluation subject. Thereupon, five concrete

questions to be investigated are derived from the

games’ objectives. Subsequently, the evaluation

model is presented.

3.1 Evaluation Subject and Goals

GBGo currently consists of three mini-games –

“Explore Procurement”, “Explore Production” and

“Explore Sales” – which address various operational

(business) processes from the areas of procurement,

production, and sales (Häusler, 2019). In this way,

they complement the teaching materials of the

corresponding modules Materials Management

(MM), Production Planning (PP), and Sales and

Distribution (SD). In Explore Procurement, the

participants (players) compete indirectly with each

other as purchasers of different companies. The goal

is to optimize their procurement strategy based on

offered product combinations, price, and delivery

reliability (supplier selection). In Explore Production,

the participants act as production planners and have

to fulfill a given production target under certain

conditions (working days, personnel, costs) in a cost-

efficient way. In the Explore Sales scenario, parts of

the sales process are simulated. The players have to

maximize their profit through bicycle sales, taking

Towards an Evaluation Concept for Business Simulation Games: Preliminary Work and Piloting in SAP ERP Teaching

into account the shared market and seasonal

influences.

In each scenario, a maximum of 25 companies

compete with each other. The participants take

business decisions for the company assigned to them.

The division of the players can be done individually

or in teams. Several rounds, determined by the

lecturer, are played in sequence, with one round

(period) always corresponding to one month. Before

the game starts, the participants familiarize

themselves with the respective scenario and discuss

any comprehension questions with the lecturers. After

the end of the game, the game leader moderates a joint

debriefing in which the progress of the BSG is

recapitulated and analyzed.

The BSGs may be used independently from each

other and other teaching materials. As an extension

option for lecturers, they link the module’s slide set

and case study by providing a gamified thematic

introduction. The aim is to provide a basic

understanding of simple market mechanisms and the

fundamentals of business activities. The assumption

is that especially learners from a distant business

environment can be given a simplified introduction to

the complex business context with these BSGs.

Furthermore, active and game-based activities as well

as the simulation of real processes are intended to lead

to an increase in motivation among learners (Häusler,

2019).

Participants are offered didactic variety through

the BSGs, which should also lead to increased

motivation. It is further assumed that learners find it

easier to work through the corresponding case studies

if they have previously mastered the business games.

These business games are highly simplified and

consist of quickly understandable scenarios and game

rules. Other features in this context are the flexible

and independent use as well as the short game

duration.

This exemplary study aims to investigate the

concrete use of GBGo in teaching and learning

arrangements. The evaluation concept lays the

foundation for the improvement and further

development of these business games, which can

ideally be extended to other BSGs used in ERP

teaching. The following central questions can be

derived in summary from the (partly implicit)

objectives of the GBGo series:

1. Are these BSGs suitable for beginners?

2. Are these BSGs suitable for imparting basic

knowledge of the respective topics to the

participants?

3. Do the (self-)performed actions of the

participants contribute to knowledge

acquisition?

4. Does the interaction (as a characteristic of

BSGs) with other participants contribute to

knowledge acquisition?

5. Do the participants find it easier to work on the

thematically related case study after the BSGs

have been executed?

This question set forms the frame of reference for

the evaluation concept designed and executed in this

work. From the state of the art, especially from the

field of ERP teaching, there are numerous findings

about the subject area that support the development

and use of a standardized survey for BSG participants

as the central evaluation method. This also enables

the basis for the comparability of future surveys in the

ERP teaching context.

3.2 Evaluation Model

While in the evaluation of educational materials often

different alternatives of the object of evaluation can

be weighed against each other (Mayer, 2010) this is

not the case for GBGo. In the complex ERP curricula

of the UCC, a large number of teaching materials are

not used as alternatives, but rather as supplements.

Consequently, to be analyzed business games should

be considered less as substitutes than as supplements

(for case studies). Therefore, the BSGs have to be

examined and evaluated concerning the achievement

of their learning objectives. However, the

interdependency with the case studies should also be

included in the evaluation. To structure and classify

the previous explanations and to extend them by

further conceptual components, an evaluation model

by Kirkpatrick is used, which is frequently used for

the evaluation of BSGs (Mayer, 2010). The model

distinguishes the four stages or levels of reaction,

learning, behavior, and results, which are situated in

the temporal course of an evaluation.

Level 1 focuses on the reactions of the

participants. Here, subjective assessments and

valuations are to be queried. Questionnaires or

interviews are often used as tools for this purpose

(Mayer, 2010). Both the immediate reaction of the

participants and the subsequent feedback on

organizational and technical content are taken into

account (Birgmayer, 2011). In level 2, the acquisition

of knowledge and skills from the respective

intervention is to be investigated. It should be

determined whether and how the participants’

knowledge level has changed. In order to make the

related change visible, the use of a pre-post-test

ICSBT 2023 - 20th International Conference on Smart Business Technologies

design can be beneficial (Birgmayer, 2011). Level 3

focuses on the evaluation of the transfer of previously

reviewed knowledge and skills into practice. In the

case of occupational topics, this transfer mostly takes

place in the workplace (Birgmayer, 2011). Level 4

focuses on the consequences or expected benefits of

the intervention or program for the implementing

organization. Ideally, the interventions and programs

are intended to achieve predefined goals, e.g., the

evaluation of corporate goals (Birgmayer, 2011) by

measuring key performance indicators and their

change over time (Kriz, 2010).

Levels 1 and 2 are measurable during or directly

after the execution of the BSGs under evaluation,

whereas evaluation in terms of level 3 is only

meaningful or possible after a certain time has

elapsed. Not only will this increase the influence of

confounding factors (Kriz, 2010), but also the

measurement of the transfer itself is challenging and

therefore not investigated in the given scope. Level 4

which is the KPI-oriented proof at the organizational

level can also not be implemented in this study.

Therefore, the evaluation of GBGo concentrates on

the first two levels.

4 STUDY DESCRIPTION

Due to the Covid-19 pandemic, the BSGs to be

evaluated were integrated into a remote course with a

final examination. The framework from the UCC and

the relevant faculty had to be considered when

creating the course. As a result, the concrete

implementation of the evaluation and the method

used depended on these circumstances. A total of 16

participants took part in the study. All participants

were matriculated students from different study

programs and with different study progress.

4.1 Study Design

According to the underlying evaluation model, an

online survey with standardized and open questions

was used as the data collection method. In addition,

the survey was set up as a pre-post design in order to

be able to record knowledge gains in the context of

the BSGs on individual prior knowledge.

Furthermore, the interaction of the games with other

teaching materials was investigated. For this purpose,

the participants were randomly divided into two

groups. The groups conducted the three BSGs within

the GBGo series and the associated case studies in

reverse order. Group 1 (G1) first played the business

game and then worked on the case study, and group 2

(G2) did vice versa. Both groups received the

upstream pre-questionnaire (t

) before playing the

BSGs and the downstream post-questionnaire (t

)

afterward. As a result of the pre-post design, G1 had

to answer a third questionnaire (t

) after the case study

was conducted. An overview of the investigation

procedure can be seen in Figure 1.

This procedure was repeated, maintaining the

subject group composition, for each of the three BSGs

and their corresponding lectures and case studies. The

order of conducting the BSG topic areas followed the

run-through of value-added processes in companies:

Explore Procurement (MM), Explore Production

(PP), and Explore Sales (SD). Slide sets, case studies,

and BSGs in version 3.3 were used (based on SAP

S/4HANA 1809).

4.2 Survey Design

The central questions regarding the impact of GBGo

formed the basis for structuring and operationalizing

the surveys used. Firstly, in addition to general socio-

demographic data, the participants were asked at the

beginning whether they had already come into contact

with BSGs. The data were collected using nominal

scales.

In order to be able to record a possible transfer of

basic knowledge through the respective BSG,

examination questions on thematic basic knowledge

with open answer options were used (e.g., “What is a

bill of materials?”, “What does the term production

control mean?”, “How do a supplier’s bid prices

typically affect the associated demand in the

market?”). These questions were asked in the same

wording both before (t

) and after (t

) the respective

BSG.

To get first indications about whether the

knowledge acquisition is achieved or promoted by

self-performed actions within the BSG as well as by

interaction with other players, one question each was

developed for the action and the interaction of the

participants within the BSG. For these questions,

4-level rating scales were used as response options, in

order not to give too fine-grained, difficult-to-delimit

answer possibilities for this first investigation as well

as to identify a direction.

Towards an Evaluation Concept for Business Simulation Games: Preliminary Work and Piloting in SAP ERP Teaching

Week 0 Week 1 Week 2 Week 3

G1 Lecture

(1) t

-Survey

(2) BSG

(3) t

-Survey

Case Study Case Study t

-Survey

G2 Lecture Case Study Case Study

(1) t

-Survey

(2) BSG

(3) t

-Survey

Figure 1: Investigation procedure overview.

In a further question, the participants were asked

to assess how well they managed the case studies after

completing them (e.g., “Overall, how did you manage

to work on the case study?”, “Which problems did

you have during processing?”). This was intended to

gather initial circumstantial evidence about a possible

connection between participation in the BSGs and

subsequently successful (easier) mastery of the case

studies. Possible answers were listed using a 5-point

rating scale or created as free response options since

neutral answers seem comprehensible for this

question.

5 RESULTS AND DISCUSSION

In the following section, the study is evaluated. First,

some preliminary remarks are made. Then the results

are presented before possible causes and effects are

discussed.

5.1 Preliminary Remarks

The surveys were created, distributed, and answered

using LimeSurvey. The collected data were exported

to Microsoft Excel and subsequently cleansed.

Unfortunately, in the different executions, there were

varying numbers of data sets excluded due to

different causes:

 A participant did not complete at least one

survey within a single pre-post setup.

 The connection parameter between pre and

post-survey was inconsistent.

 There were conflicting responses to essential,

interrelated questions.

The six execution groups and the respective

number of usable response data sets can be seen in

Figure 2.

5.2 Results

In all executions, most respondents stated that they

had never taken part in a BSG before (MM n = 5/9;

PP n = 9/12; SD n = 9/10). Thus, the majority of the

participants are beginners. Previous experience in the

respective module areas (MM, PP, and SD) was also

denied by most people. Accordingly, the respondents

rated their prior thematic knowledge as “poor“ (MM

n = 5/9; PP n = 6/12; SD n = 5/10) and “very poor“

(MM n = 4/9; PP n = 5/12; SD n = 3/10).

5.2.1 Feedback Concerning GBGo

Almost all participants assessed the duration of the

overall games as “appropriate”. Similarly, the

difficulty was mostly assessed as “appropriate” and in

several cases as “(rather) too easy”. Only one

respondent rated Explore Procurement as “rather too

difficult”. The averaged values indicate that Explore

Sales was the easiest game whereas Explore

Procurement was the most difficult.

The majority of respondents enjoyed carrying out

actions by themselves, especially in Explore Sales

and Explore Procurement. Only one of the

participants answered with “rather disagree”

regarding Explore Production. The interaction with

others is assessed lower than in the previous question

but still positive. Regarding action and interaction,

sorting the average values of action and interaction,

the following order emerges: Explore Sales receives

the most approval, followed by Explore Procurement

and Explore Production.

5.2.2 Imparting Knowledge Through GBGo

The questions on specific prior knowledge and the

associated knowledge gained during BSG execution

were posed to the participants in a test-like manner.

MM-G1 MM-G2 PP-G1 PP-G2 SD-G1 SD-G2

Usable Data Sets 5 4 6 6 5 5

Figure 2: Number of usable data sets.

ICSBT 2023 - 20th International Conference on Smart Business Technologies

M Basic Knowledge (t

)

# Participants with

Knowledge Increase (t

)

Total Responses

MM-Q1 0,33 3 9

MM-Q2 0,33 0 9

MM-Q3 0,17 0 9

PP-Q1 0,25 3 12

PP-Q2 0,29 4 12

PP-Q3 0,17 3 12

SD-Q1 0,75 1 10

SD-Q2 0,6 1 10

MM-Q1: What does the term procurement mean?

MM-Q2: What is a bill of materials?

MM-Q3: What does the term procurement strategy mean?

PP-Q1: What does the term production planning mean?

PP-Q2: What does the term production control mean?

PP-Q3: What is the difference between a planned order and a production order?

SD-Q1: How do a supplier’s asking prices typically affect the associated demand in the market?

SD-Q2: How do asking prices of different suppliers in the same market influence each other?

Figure 3: Average values for prior knowledge and absolute frequency for knowledge increase.

The answers to the open-ended questions from the

first survey (t

) on prior knowledge were scored as

follows: Either “0” (incorrect), “0.5” (partially

correct), or “1” (mostly correct) point(s). The answer

to the same question in the second survey after

executing the BSGs (t

) was evaluated according to

the identical scheme and compared to the first answer

on a participant-specific level. The resulting change

in knowledge was coded either “0” (not improved) or

“1” (improved). The results on prior knowledge were

averaged and presented in Figure 3 along with the

absolute frequencies of participants who had an

increase in knowledge.

Overall, it can be seen that the level of knowledge

with regard to the posed questions is relatively

limited. In addition, only some participants showed

an increase in knowledge. The relatively high

increase in knowledge in the PP module is

remarkable.

5.2.3 Mastering the Case Studies

After completing the case studies, the simulation

participants assessed how well they managed to work

through them. Responses were coded as follows:

Very good = 1; good = 2; partly good/partly poor = 3;

poor = 4; very poor = 5.

Figure 4 compares the average values calculated

from the two groups for each BSG.

The available results do not show a consistent

overall picture. Thus, there is no evidence that the

participants coped better with the case studies if the

thematically related BSG had been carried out

beforehand.

5.3 Discussion

The following discussion of the results is structured

according to the evaluation questions that were

derived in the concept section.

5.3.1 Are These BSGs Suitable for

Beginners?

The BSGs are suitable for beginners because the level

of difficulty and the amount of required time were

rated as appropriate by the participants, most of them

inexperienced in BSGs. Only a few reported too little

or too much processing time, which only emphasizes

the heterogeneity of the students regarding learning

requirements and learning behavior.

There were no comprehension problems

regarding rules and scenarios during the execution of

the BSGs. The reason for this could be the

comprehensive support by the simulation instructors

(e.g., introduction to the course of the game and

answering comprehension questions). Support before

and during the implementation, as well as a joint

debriefing afterward, therefore, contribute to a

successful application just as much as the quality of

the teaching materials themselves. In order to

specifically investigate the impact and importance of

Towards an Evaluation Concept for Business Simulation Games: Preliminary Work and Piloting in SAP ERP Teaching

M Mastering the

Case Studies (G1)

Total

Responses G1

M Mastering the

Case Studies (G2)

Total

Responses G2

MM 2,6 5 2,5 4

PP 1,5 6 2,17 6

SD 1,6 5 1,4 5

Figure 4: Average values for mastering the case studies per group.

debriefing as part of the game plan implementation,

the related objectives should be formulated and

operationalized as evaluation questions. In this way,

they can be integrated into future studies.

5.3.2 Are These BSGs Suitable for

Imparting Basic Knowledge of the

Respective Topics to the Participants?

The BSGs aim to impart basic knowledge of the

respective topics to a heterogeneous target group, some

of which is already known to the participants (e.g.,

through job experience). The games seem to be just

partly suitable for this purpose because only a part of

the participants shows a knowledge increase. As an

example, Explore Sales showed the most correct

answers regarding the level of prior knowledge, and at

the same time the smallest increase in knowledge.

Nevertheless, even participants with little or no prior

knowledge showed only a partial improvement. The

reason for this is unclear and could be due to the

selection and formulation of the knowledge questions.

To counteract this “prior knowledge conflict”,

possibilities for experienced players can be created by

alternative game mechanics and incentives. In this

way, they can use their previous experience without

endangering the experience of the other players. In

this regard, possible ideas should be discussed and, if

necessary, integrated. In this context, it should also be

examined whether the BSG is the most suitable

teaching method for the basic knowledge to be

imparted in general.

In order to be able to better evaluate the acquisition

of knowledge and competencies in the context of the

considered BSGs, a profound reflection and discussion

about this topic are needed. Especially the acquisition

of specific competencies through BSGs and their

measurability is challenging and may only be

manageable with an interdisciplinary approach.

5.3.3 Do the (Self-)Performed Actions of the

Participants Contribute to Knowledge

Acquisition?

There are indications that the actions carried out by

the participants contribute to knowledge acquisition.

Participants perceive that they carry out actions

during the BSGs and thus influence the gameplay.

Since results are directly visible, adjusting screws

become comprehensible and cause-and-effect

relationships observable. By analyzing and

evaluating their own actions, future actions become

plannable and calculable. Some participants even

demand further, more extensive action options and

information about their effects. It is unclear whether

such an extension would make the mini-games too

complex or complicated. This could possibly conflict

to provide an easy entry into the respective module

context because there are indications that an (overly)

complex simulation leads to a lower learning success

(Ulrich, 2002).

In this area, it must also be generally ensured that

there are no irregularities between input parameters,

simulation mechanics, and output. The respective

BSG has to react appropriately (close to reality) to the

actions of the participants in order to avoid

misunderstandings and false assumptions among the

players. Since this can also be caused by hard-to-find

bugs, which may become apparent during real use,

extensive and periodic tests are necessary.

5.3.4 Does the Interaction with Other

Participants Contribute to Knowledge

Acquisition?

The interaction with other players within the game

setup is mostly perceived positively, especially in

Explore Sales. There is an obvious explanation: The

scenario of this BSG involves direct competition

between the companies, which increases the

motivating effect. In the other two scenarios (Explore

Procurement and Explore Production), the companies

interact indirectly with each other, which is why

competition presumably is rarely perceived by the

participants.

It should be noted that especially interactions and

their effects are perceived by the participants, as this

increases realism and motivation. Specifically for

Explore Procurement and Explore Production, it

should be examined if and how an increased (more

direct) interaction between the players can be

realized.

ICSBT 2023 - 20th International Conference on Smart Business Technologies

100

The self-performed actions as well as interactions

with others must be included as significant elements

of BSGs appropriately and extensively in further

investigations. This should be done in future

evaluations by adding answer options to the need for

extended action and interaction possibilities, or by

using different data acquisition methods, e.g.,

interviews. This will allow concrete suggestions from

players to be systematically taken up and included in

the further development of the games. In this way, the

players and their intentions become the focus of

attention.

5.3.5 Do the Participants Find it Easier to

Work on the Thematically Related

Case Study after the BSGs Have Been

Executed?

The presumed interdependencies on other teaching

materials, in particular on the processing of the case

studies, could not be proven in this work. Only in the

PP module, the participants find it easier to complete

the case study after completing the BSG. In the MM

and SD modules, a slightly opposite trend can be

discerned. The reasons for this cannot be determined

from the survey. It is possible that the knowledge

gained from the BSGs does not have a positive effect

on facilitating the processing of the case studies, or

that there is no direct interrelation. Nevertheless, it

should be noted that the content of the BSGs only

covers a small part of the business processes

considered in the case studies.

The expansion of existing teaching materials is in

the nature of these BSGs and is therefore of great

interest in this context. Consequently, a suitable

research design for a more in-depth analysis of the

relationship between BSGs and easier mastering of

the case studies needs to be developed and tested.

Possibly, it may not be possible to prove causal

interactions since the overall scenario is too complex.

5.4 Limitations

The following points limit the results of this study,

first and foremost the extremely small number of

participants. Since this was a regular course including

the awarding of ECTS credits, the design of the study

structure had to ensure equal conditions for all

students, i.e., the same content and also a comparable

amount of time for performance. Therefore, no

control groups could be integrated. In addition, the

execution of the BSGs had to be embedded in weekly,

90-minute lectures. This limited the scope of the

study and led to a restriction of the time needed to

conduct the round-based simulations as well as the

extent of the survey.

With regard to the possible execution variants

(online vs. offline, single player vs. group game),

only one was tested, in which the participants

completed the BSGs online as single players. Thus,

there is a lack of comparison possibilities with other

variants, which presumably produce different results.

However, since the present study represents the

first evaluation of GBGo with pilot character, the

previously described circumstances were accepted. In

this way, it is possible to further develop the

evaluation design including the method as well as the

BSGs themselves based on the results of the pilot

phase.

6 CONCLUSION AND FUTURE

WORK

With the evaluation of GBGo, initial indications

could be generated about what the games (can)

achieve in the concrete application context of ERP

teaching and which potentials exist. Together with the

first steps towards an evaluation concept required for

this purpose, this has created the basis for knowledge-

based further developments. In their continuation –

i.e., in the enhancement of the interdependency

between BSG and other materials – is the opportunity

to exploit the hitherto unused potential. Through

continuous evaluation and further development, not

only the BSGs themselves as well as their use can be

improved, but also scientific knowledge about the

subject area can be gained, which is especially

essential for a future “BSG as a Service” approach

(Häusler et al., 2021).

A first step has already been taken by handing

over the BSG documentation including the evaluation

questionnaires to customers (e.g., universities and

vocational schools) of the UCC Magdeburg for

testing. With these and other evaluation results, the

existing data basis can be enlarged and thus made

more solid. An additional gain could be to include the

expertise of disciplines that have not been involved so

far or only marginally, such as didactics and learning

psychology. Thus, the subject area could be

considered and understood even more

comprehensively in order to enrich the further

development of the BSGs with problem-solving ideas

beyond their own disciplines.

Towards an Evaluation Concept for Business Simulation Games: Preliminary Work and Piloting in SAP ERP Teaching

101

REFERENCES

Birgmayer, Renate (2011): Eine praxisnahe Einführung in

Bildungscontrolling. Das Modell von Kirkpatrick und

seine Erweiterungen durch Phillips und Kellner. In

Magazin erwachsenenbildung.at (12), 1-8.

Blötz, Ulrich (2015): Das Planspiel als didaktisches

Instrument. In Ulrich Blötz (Ed.): Planspiele und

Serious Games in der beruflichen Bildung. Auswahl,

Konzepte, Lernarrangements, Erfahrungen; aktueller

Katalog für Planspiele und Serious Games 2015. 5.,

überarb. Aufl. Bielefeld: Bertelsmann (Berichte zur

beruflichen Bildung), pp. 13–25.

Faisal, Nadia; Chadhar, Mehmood; Goriss-Hunter, Anitra;

Stranieri, Andrew (2022): Business Simulation Games

in Higher Education: A Systematic Review of Empirical

Research. In Human Behavior and Emerging

Technologies 2022, pp. 1–28.

Fischer, Helge; Heinz, Matthias; Schlenker, Lars; Münster,

Sander; Follert, Fabiane; Köhler, Thomas (2017): Die

Gamifizierung der Hochschullehre – Potenziale und

Herausforderungen. In Susanne Strahringer, Christian

Leyh (Eds.): Gamification und Serious Games.

Grundlagen, Vorgehen und Anwendungen. Wiesbaden:

Springer Vieweg, pp. 113–125.

Häusler, Robert (2019): Global Bike Go!: Konzept zur

Erweiterung der bestehenden S/4HANA Lehr-und

Lernumgebung. In Karin Gräslund, Dietmar Kilian,

Alexander Redlein (Eds.): Proceedings SAP Academic

User Group Meetings 2019. Wien. Available online at

https://repositum.tuwien.at/handle/20.500.12708/661.

Häusler, Robert; Bernhardt, Chris; Bosse, Sascha;

Turowski, Klaus (2019): A Review of the Literature on

Teaching and Learning Environments. In : New

frontiers in digital convergence. 25th Americas

Conference on Information Systems (AMCIS 2019) :

Cancun, Mexico, 15-17 August 2019. 5 volumes. Red

Hook, NY: Curran Associates Inc (3), pp. 1997–2006.

Häusler, Robert; Bosse, Sascha (2018): Analysis and

Modeling of Learning Systems and Development of a

Process Model for Flexible Orchestration of Learning

Environments. In Paul Drews (Ed.): Data driven X.

Turning data into value : Multikonferenz

Wirtschaftsinformatik (MKWI) 2018 : Leuphana

Universität, 6.-9. März 2018. Lüneburg: Leuphana

Universität, pp. 795–806.

Häusler, Robert; Tröger, Marcus; Staegemann, Daniel;

Volk, Matthias; Turowski, Klaus (2021): Towards a

Systematic Requirements Engineering for IT System-

based Business Simulation Games. In Beno Csapó,

James Uhomoibhi (Eds.): Proceedings of the 13th

International Conference on Computer Supported

Education. 13th International Conference on Computer

Supported Education. Online Streaming, 23.04.2021 -

25.04.2021. 2 volumes: SCITEPRESS - Science and

Technology Publications (1), pp. 386–391.

Huang, Yueh-Min; Silitonga, Lusia Maryani; Wu, Ting-

Ting (2022): Applying a business simulation game in a

flipped classroom to enhance engagement, learning

achievement, and higher-order thinking skills. In

Computers & Education 183, p. 104494.

Jacob, Axel; Teuteberg, Frank (2017): Game-Based

Learning, Serious Games, Business Games und

Gamification –Lernförderliche Anwendungsszenarien,

gewonnene Erkenntnisse und Handlungsempfehlungen.

In Susanne Strahringer, Christian Leyh (Eds.):

Gamification und Serious Games. Grundlagen,

Vorgehen und Anwendungen. Wiesbaden: Springer

Vieweg, pp. 97–112.

Kriz, Willy Christian (2010): Evaluation von ePlanspielen

und digitalen Lernspielen. In Horst O. Mayer, Willy

Christian Kriz (Eds.): Evaluation von eLernprozessen.

Theorie und Praxis. München: Oldenbourg

Wissenschaftsverlag (Ergänzungstitel BWL, VWL,

SoWi 10/2010), pp. 61–96.

Leyh, Christian (2017): Serious Games in der

Hochschullehre: Ein Planspiel basierend auf SAP ERP.

In Susanne Strahringer, Christian Leyh (Eds.):

Gamification und Serious Games. Grundlagen,

Vorgehen und Anwendungen. Wiesbaden: Springer

Vieweg, pp. 151–166.

Matute, Jorge; Melero, Iguácel (2016): Game-based

learning: using business simulators in the university

classroom. In Universia business review (51), pp. 72–

111.

Mayer, Horst O. (2010): Evaluation von eLearning-

Produkten/Prozessen. In Horst O. Mayer, Willy

Christian Kriz (Eds.): Evaluation von eLernprozessen.

Theorie und Praxis. München: Oldenbourg

Wissenschaftsverlag (Ergänzungstitel BWL, VWL,

SoWi 10/2010), pp. 15–24.

McGonigal, Jane (2011): Reality is broken. Why games

make us better and how they can change the world.

New York: Penguin Press.

Prifti, Loina; Knigge, Marlene; Löffler, Alexander; Hecht,

Sonja; Krcmar, Helmut (2017): Emerging Business

Models in Education Provisioning: A Case Study on

Providing Learning Support as Education-as-a-

Service. In Int. J. Eng. Ped. 7 (3), pp. 92–108.

Reich, Chris; Häusler, Robert (2019): Konzept zur

Erstellung von Lehr und Lernumgebungen für die IT

bezogene Hochschullehre. In Karin Gräslund, Dietmar

Kilian, Alexander Redlein (Eds.): Proceedings SAP

Academic User Group Meetings 2019. Wien.

Rubart, Jessica; Hartweg, Elmar (2019): Planspiele in der

Hochschullehre – am Beispiel von Fort Fantastic und

ERPsim. In Tobias Schmohl, Dennis Schäffer, Kieu-

Anh To, Bettina Eller-Studzinsky (Eds.):

Selbstorganisiertes Lernen an Hochschulen. Strategien,

Formate und Methoden. Bielefeld: wbv

(TeachingXchange, 3), pp. 95–103.

Ulrich, Markus (2002): Sind Planspiele langwierig und

kompliziert? Eine Abhandlung über die

Planspielmethodik und die Ausbildung von Planspiel-

Fachleuten. In Ulrich Blötz (Ed.): Planspiele in der

beruflichen Bildung. Aktualisierter Planspielkatalog

2002 sowie Abriss zur Auswahl, Konzeptionierung und

Anwendung von Planspielen. 2., überarb. Aufl.

ICSBT 2023 - 20th International Conference on Smart Business Technologies

102

Bielefeld: Bertelsmann (Schriftenreihe des

Bundesinstituts für Berufsbildung Bonn).

Unger, Thorsten; Goossens, Jannis; Becker, Lisa (2015):

Digitale Serious Games. In Ulrich Blötz (Ed.):

Planspiele und Serious Games in der beruflichen

Bildung. Auswahl, Konzepte, Lernarrangements,

Erfahrungen; aktueller Katalog für Planspiele und

Serious Games 2015. 5., überarb. Aufl. Bielefeld:

Bertelsmann (Berichte zur beruflichen Bildung).

Utesch, Matthias; Heininger, Robert; Krcmar, Helmut

(2016): Strengthening study skills by using ERPsim as

a new tool within the Pupils' academy of serious

gaming. In : 2016 IEEE Global Engineering Education

Conference (EDUCON). 2016 IEEE Global

Engineering Education Conference (EDUCON). Abu

Dhabi, 10.04.2016 - 13.04.2016: IEEE, pp. 592–601.

Weppel, Sheri; Bishop, M. J.; Munoz-Avila, Hector (2012):

The Design of Scaffolding in Game-based Learning: A

Formative Evaluation. In Journal of Interactive

Learning Research 23, pp. 371–402.

Towards an Evaluation Concept for Business Simulation Games: Preliminary Work and Piloting in SAP ERP Teaching

103