Structuring Collaboration Scripts

Optimizing Online Group Work on Classroom Dilemmas in Teacher Education

Hans Hummel

1,2

, Walter Geerts

, Aad Slootmaker

, Derek Kuipers

and Wim Westera

Open University of the Netherlands, Valkenburgerweg 177, 6419AT, Heerlen, The Netherlands

NHL University of Applied Science, Rengerslaan 10, 8917DD, Leeuwarden, The Netherlands

Keywords: Serious Games, Scripted Collaboration, Structure, Classroom Dilemmas, Teacher Education.

Abstract: The optimal structure in collaboration scripts for serious games has appeared to be a key success factor. In

this study we compare a ‘high- structured’ and ‘low-structured’ version of a mastership game where

teachers-in-training discuss solutions on classroom dilemmas. We collected data on the differences in

learning effects and student appreciation. The most interesting result shows that reports delivered by

students that played the low-structured version received significantly higher teacher grades when compared

to the high-structured version. [A shortened version of the paper has been included for copyright reasons.]

1 BACKGROUND

Serious games not only support individual learning

but also foster the acquisition of soft skills like

collaboration and reflection about wicked problems,

that are usually not addressed by other learning

platforms (Gee, 2003). Workplace learning is

shifting focus from individuals acquiring and

updating domain knowledge towards selecting and

using this knowledge for certain problem situations

in daily practice where collaboration plays a crucial

role.

Games are heavily inspired by experiential

learning principles which hold potential for

contextualised workplace learning. Serious games

appear suitable as flexible learning environments

where professional tasks can be carried out with

little or no direct intervention of experts or teachers

(e.g., Bell et al., 2008). How much erroneous or

meaningful learning takes place will depend on the

support that is provided, shared and distributed in

the gaming environment. Collaboration support

within a game has to be enabled by a didactic

‘script’ which we will name ‘scripted collaboration’.

Collaboration scripts (Kobbe et al., 2007) are an

instructional method that structures the collaboration

by guiding the interacting partners online through a

sequence of interaction phases with designated

activities and roles. Such collaboration scripts have

hardly ever been implemented and tested in more

open learning environments like serious games

(Dillenbourg and Hong, 2008). No research has

focused on defining or optimizing the essential

elements (e.g., of structure) or has measured the

learning effects of including such scripting in serious

game play.

Structure is defined here as the amount of

restriction imposed on the freedom that is allowed in

the group collaboration process. An optimal level of

structure appears to be a key success factor for

effective learner support. In a previous study we

found that students complained about the complexity

and task instruction within for the mastership game

(Hummel et al., 2013). Building on Dillenbourg’s

(2002) risks of over-scripting we argue that

segmentation and inter-dependency within the task

constitute the main structure elements. An holistic

task is less structured than a task that has been

segmented in various consecutive subtasks; a task

that can be carried out independently is less

structured than a task that depends on

synchronisation or approval of peers and / or

teachers. We have further operationalized these

structure elements and high/medium/low levels of

structure for this study.

The Mastership game helps students to find

solutions to the most prevailing practical classroom

management dilemmas in a playful and collaborative

way, a way that will help them become better

teachers. The game was originally developed as a

card game to be played face-to-face in small groups

Hummel H., Geerts W., Slootmaker A., Kuipers D. and Westera W..

Structuring Collaboration Scripts - Optimizing Online Group Work on Classroom Dilemmas in Teacher Education.

DOI: 10.5220/0004674300050008

In Proceedings of the 6th International Conference on Computer Supported Education (CSEDU-2014), pages 5-8

ISBN: 978-989-758-022-2

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

(Geerts et al., 2009), and was later transformed into

on online game to be played synchronously with

freedom of place (Hummel et al., 2013).

Figure 1: Screens of the online version of the Mastership

game: selecting three practical dilemmas in phase 1 (upper

left hand), assigning and motivating themes in phase 3

(upper right hand), motivating and discussing declined

themes in phase 4 (lower left hand), and peer assessment

of elaborated assignments in phase 6 (lower right hand).

The online Mastership game (for an impression see

Figure 1) can be played in small groups of two till

six students and does not require any intervention by

teachers. After selecting their avatars, students start

group play both in the role of player (or problem

owner) and of co-player (judging the way that

players solve their problems). The game has a

structure that consists of five consecutive phases,

during which players discuss, elaborate and

negotiate solutions to solve each other’s problems.

Communication is structured by various assignments

and rules during these phases, but is possible by

unstructured group chat as well. During the fifth

phase players select a practical assignment and use

their co-players’ input to further elaborate their

solution in a short advisory report.

The main hypotheses (research questions) to be

answered are twofold: (1) Will less structure lead to

more ‘natural’ and effective collaborative learning?;

and (2) Will less structure in the collaboration be

appreciated more by students?

2 STUDY SET-UP

Third year students of the NHL University of

Applied Science in the Netherlands participated in

this case study as part of their regular curriculum.

Participants are qualifying for a broad variety of first

degree teaching positions, ranging from modern

languages teaching, teaching didactics to science

teaching. All students were approached by their

teacher and invited to be present at a certain place

and time at the university for a two-hour meeting.

Participants were notified in advance that this

meeting would also be used for study purposes, and

were randomly allocated to one of three conditions

(high-structured, low-structured, control).

Participants in the control group had to solve the

practical classroom dilemma individually without

playing the collaboration game. Each gaming

condition contained two groups (of four or five

students each). The players received an e-mail

before the meeting, containing the URL and their

personal account. All playing participants received a

questionnaire about their appreciation of the game

by e-mail a day after playing the game. At the time

of the meeting, playing participants went to a

computer room to work together online. A teacher

was present in this computer room to control for

direct (non)verbal communication beyond the

program. During the time of the meeting, students in

the control group individually worked on their

practical task, without playing the game. For the

purpose of this study we included a sixth and final

phase in which students had to grade the reports of

their peers, in order to enable a comparison of the

assessments by peers (co-players) and teachers

To measure individual learning output, the

quality of the solutions provided for the classroom

dilemmas was assessed by using a learning effect

correction model, that was developed by the topic

expert. The elaborated reports can be assessed on

‘growth in professional productivity’, and the five

criteria to establish this growth were inspired by the

development of ‘design practice’ (or practical

theory) (Copeland and D’Emidio-Caston, 1998): A.

Ownership (to what extent does student commit to

solve this problem); B. Reflection (to what extent

does student reflect on his own actions); C. Focus

(to what extent does student attach the right amount

of context to the problem); D. Nuance / Complexity

(to what extent is applying the solution feasible);

and E. Richness / Correctness (of the elaborated

solution). Sufficient inter-rater reliability of the

instrument was determined in a previous study

(Hummel et al. , 2013).

The student satisfaction questionnaire was

developed for this study by a learning technology

expert. It contains 19 items to establish the students’

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

appreciation of various game aspects, pertaining to

the structure (5 questions), user-friendliness and

clarity (5 questions), the timing of the phases (2

questions), the quality of the dilemmas and

assignments (5 questions), and the interaction during

collaboration (2 questions).

3 RESULTS

We found that most individual reports (76%) could

be graded as sufficient. The average grade for all

participants was M = 6.62, SD = 1.29. We added a

control group to establish if playing the game does

contribute at all to learning. As you see in Table 1

the average teacher grades for the control group

were indeed lowest, so there appears to be an effect

of playing the game. This effect appears significant

when we compare the non-playing group to the low-

structured (t (18) = 2.97, p < 0.01) and the medium-

structured condition (which we left out of the

analyses). However, we could not observe a

significant difference between non-players and those

playing the high-structured version (t (17) = 0.67, p

= 0.51).

Table 1: Average report grades for all conditions, both

from teachers and peers.

High

structure

(n = 9)

Low

structure

(n = 10)

Control

(n = 10)

All

(N = 29)

Assessment

M SD

M SD M SD M SD

Teacher grade

6.44

1.59

7.35

1.03

6.05

0.93

6.62

1.29

Peer rating

7.93 .66 7.52

1.04

7.68

0.89

7.70

0.87

When looking for an overall effect of condition (N =

29) on learning effect we see a clear trend: low-

structure scores best, than high-structure, and finally

the control group. This effect is ‘marginally’

significant (F (2, 26) = 3.072, MSE = 4.428, p =

0.063,



= 0.18), with values of the partial-eta-

squared above .13 showing large effect size

according to Cohen (1988). On top of this and even

more importantly for the central research question, a

significant difference (t (17) = 4,86, p = 0.042) is

found in favour of low-structure when comparing

with high-structure (N= 19). When looking at the

peer ratings, we do not find any significant

differences between conditions.

For most items in the student satisfaction

questionnaire we did not find significant differences

between both versions of the game, with just two

exceptions. The low-structured group showed to be

more satisfied with the amount of time to play (item

6). The high-structured group indicated that the

overall structure was too high (item 11), a finding in

line with what was reported on learning effects. It

did not become clear that low-structure was

appreciated more by students on various aspects.

We may conclude that collaboration can be

successfully facilitated by scripting serious games

when we take into account the importance of good

instruction and optimal structure. This study found

that over-scripting may indeed have disruptive

learning effects. Players of the low-structured

version of the mastership game produced reports that

were graded significantly higher than the ones of

those playing the high-structured version (and of

those not playing the game).

For the generalizability of these findings it will

be useful to carry out studies that research the

effectiveness of other types of collaboration scripts

and implementations in other domains.

ACKNOWLEDGEMENTS

This study was carried out as research activity

within the ‘Learning Media’ program of CELSTEC

(Open University of the Netherlands), and within the

Lectorate ‘Workplace Learning and ICT’ (NHL

University of Applied Science, The Netherlands).

Authors and developers from both institutes worked

closely together during development of the game

and this study. We express our thanks to both

institutes for their funding, and to NHL students for

participating.

REFERENCES

Bell, B. S., Kanar, A. M., & Kozlowski, S. W. J. (2008).

Current issues and future directions in simulation-

based training in North America. The International

Journal of Human Resource Management, 19(8),

1416-1434.

Copeland, W. D., & D'Emidio-Caston, M. (1998).

Indicators of development of practical theory in pre-

service teacher education students. Teaching and

Teacher Education, 14(5), 513-534.

Dillenbourg, P. (2002). Over-scripting CSCL: the risks of

blending collaborative learning with instructional

design. Three worlds of CSCL.

Dillenbourg, P., & Hong, F. (2008). The mechanics of

CSCL macro scripts. International Journal of

Computer-Supported Collaborative Learning, 3(1),

StructuringCollaborationScripts-OptimizingOnlineGroupWorkonClassroomDilemmasinTeacherEducation

5–23. Springer.

Gee, J.P. (2003). What video games have to teach us about

learning and literacy? New York: Palgrave

MacMillan.

Geerts, W., Mitzsche, M., & Van Laeken, M. (Red.).

(2009). Wat zou je doen? Dilemma’s in de

onderwijspraktijk. Bussum: Coutinho.

Hummel, H.G.K., Geerts, W.M., Slootmaker, A., Kuipers,

D., & Westera, W. (2013, online). Collaboration

scripts for mastership skills: online game about

classroom dilemmas in teacher education. Interactive

Learning Environments, 2013.

Kobbe, L., Weinberger, A., Dillenbourg, P., Harrer, A.,

Hämäläinen, Häkkinen, P., & Fisher, F. (2007).

Specifying computer-supported collaboration scripts.

Computer-Supported Collaborative Learning, 2, 211-

224.

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation