Library Not Found

The Disconnect between Gaming Research and Development

Marcello Passarelli

, Jeffrey Earp

, Francesca Maria Dagnino

, Flavio Manganello

Donatella Persico

, Francesca Pozzi

, Thomas Buijtenweg

, Mata Haggis

, Chris Bailey

and Carlo Perrotta

Institute for Educational Technologies, National Research Council of Italy, Genoa, Italy

Faculty of Education, Social Sciences and Law, University of Leeds, Leeds, U.K.

Academy of Digital Entertainment, NHTV: Breda University of Applied Science, Breda, The Netherlands

{C.Perrotta, C.J.Bailey}@leeds.ac.uk

Keywords: Video Games, Game Research, Serious Games, Entertainment Games, Game Development.

Abstract: Video games clearly have great educational potential, both for formal and informal learning, and this avenue

is being thoroughly investigated in the psychology and education literature. However, there appears to be a

disconnect between social science academic research and the game development sector, in that research and

development practices rarely inform each other. This paper presents a two-part analysis of this

communicative disconnect based on investigations carried out within the H2020 Gaming Horizons project.

The first part regards a literature review that identified the main topics of focus in the social sciences

literature on games, as well as the chief recommendations authors express. The second part examines 73

interviews with 30 developers, 14 researchers, 13 players, 12 educators, and 4 policy makers, investigating

how they perceived games and gaming. The study highlights several factors contributing to the disconnect:

different priorities and dissemination practices; the lag between innovation in the games market and

research advancements; low accessibility of academic research; and disproportionate academic focus on

serious games compared to entertainment games. The authors suggest closer contact between researchers

and developers might be sought by diversifying academic dissemination channels, promoting conferences

involving both groups, and developing research partnerships with entertainment game companies.

1 INTRODUCTION

Video gaming is a popular recreational activity that

millions of people worldwide engage in

enthusiastically (Przybylski, Rigby and Ryan, 2010).

This enjoyment factor can be purposefully harnessed

to enhance formal education or to train specific

skills (Randel et al., 1992). Indeed, there has long

been interest in the relationship between games and

learning, giving rise to two main social sciences

research lines.

The first considers the use of games in formal

educational contexts. In this case, video games are

intentionally used as a tool to enhance the learning

experience by motivating and engaging learners

(Dickey, 2005) and by facilitating learning through

suitable game mechanics (e.g., by providing

immediate feedback, hints, and adaptive difficulty

for exercises; Orvis, Horn and Belanich, 2008). This

perspective considers both serious games, i.e. games

specifically created for the attainment of specific

outcomes like learning, and commercial off-the-

shelf games originally developed for entertainment

but adopted for the purposes of some kind of

educational activity.

The second research line considers learning in

informal or non formal contexts, i.e. when gaming is

largely recreational. In this case, learning is either

self-directed or unintentional, and it strictly

intertwines with entertainment. Studies considering

this kind of learning stem mostly from the

psychological literature, and consider the cognitive

and perceptual benefits of frequent video gaming

(see Powers et al., 2013, for a meta-analysis). Due to

the kind of context considered, these studies

overwhelmingly focus on entertainment games.

While these lines are distinct in many respects,

they do share a number of common aspects. Firstly,

134

Passarelli, M., Earp, J., Dagnino, F., Manganello, F., Persico, D., Pozzi, F., Buijtenweg, T., Haggis, M., Bailey, C. and Perrotta, C.

Library Not Found.

DOI: 10.5220/0006773601340141

In Proceedings of the 10th International Conference on Computer Supported Education (CSEDU 2018), pages 134-141

ISBN: 978-989-758-291-2

they both seek to determine the beneficial outcomes

of gaming. At the same time, they sometimes

consider broad typologies of games, without

differentiating much between game genres (e.g.,

Boot et al., 2011). Similarly, they rarely consider

individual differences in game preferences or

learning styles (Gros, 2007; Papastergiou, 2009) and

give relatively low priority to investigating the

gaming experience from the viewpoint of the

individual player (see Wouters et al., 2013, for a

critique). Finally, as will be argued in this paper,

they both seem to suffer a degree of isolation from

game development practitioners and their activity

(Ondrejka, 2006).

This last issue is especially troubling, since it

carries the risk of making social sciences academic

research self-referential, limiting the impact of

scientific findings on stakeholders other than

researchers, and confining the dissemination of

results to the academic community. At the same

time, a lack of communication between game

developers and researchers can lead to academic

research that is misinformed about the world of

commercial video games, and especially about the

constraints and opportunities in for-profit game

development practice (Blow, 2004; Wender &

Watson, 2012). The very different paces at which

the game industry and game research evolve

exacerbate this factor (Kultima, 2015).

This paper reports the findings on this research-

development disconnect that emerged from data

collected in the H2020 Gaming Horizons project

Gaming Horizons has investigated the effects of

gaming in society in a broad sense and from a

variety of perspectives (psychological, educational,

ethical, sociocultural/artistic), with the eventual aim

of proposing alternative framings for considering the

role of video gaming in society. In the course of the

project, which has actively involved a range of

stakeholders (researchers, developers, policy

makers, educators, and players), the disconnect

between social sciences academic research and the

game development world emerged as a clear and

important issue to be addressed. In this paper, we

will closely examine the findings from two phases of

the project (literature review and interviews with

stakeholders) with the dual aim of identifying the

main contributing factors to the disconnect and

producing recommendations for bridging it.

www.gaminghorizons.eu

2 METHOD

Gaming Horizons started at the end of 2016 and is

due to end at the beginning of 2019. The project

entailed a number of tasks, extensively described in

the project deliverables (Persico et al., 2017a;

Persico et al., 2017b) and had a much wider focus

than the one considered in the present paper. Here,

we will focus on the two tasks of the project most

relevant for exploring and addressing the research-

development disconnect: the literature review and

the interviews with stakeholders.

2.1 Literature Review

The literature review (Persico et al, 2017a) had three

closely interconnected goals: (i) identifying the main

topics addressed within social sciences research into

video games and gamification; (ii) highlighting the

most influential contributions and results in order to

obtain a broad overview of the 'state of the art' in

this research field; (iii) collecting the

recommendations made in those investigations,

noting what is being recommended and to whom.

The review considered all the academic journal

papers dedicated to games and gamification

published since 2010 that were indexed on Scopus

and Web of Science at the beginning of 2017. These

were retrieved using sets of keywords specifically

targeting three social science oriented perspectives,

which we labelled 'psychological', 'educational', and

'ethical'; therefore, articles pertaining to the

technological aspects of game development were

excluded (for more details on the adopted

methodology, see Persico et al, 2017a).

The full set of 9,157 retrieved papers was used

to inform goal (i) through analysis of frequency and

co-occurrence of paper keywords. Goals (ii) and (iii)

were achieved by considering a selected subset of

papers, since the large number of contributions made

complete examination unfeasible. We selected the

most 'influential' papers by using year-adjusted

citation rates as a proxy for impact. Only papers one

deviation standard or more above the mean citation

rate of their publication year (for the full set of

papers) were then taken into consideration (n=674).

Subsequent manual selection of relevant papers

through abstract reading led to the selection of 47

literature reviews and meta-analyses, which

comprised the final set of papers for full reading.

This set was used to meet goals (ii) and (iii).

Library Not Found

135

2.2 Interviews with Relevant

Stakeholders

A second phase of the project involved a total of 73

one-on-one interviews with stakeholders concerned

with the role of video games in society and their

potential for practical applications (Persico et al,

2017b). The stakeholders considered included 30

game developers (20 of whom were interviewed at

two non-academic conferences: the Game

Developers’ Conference (GDC)

in the USA and

Game Happens

2017 in Italy), 14 social sciences

academic researchers, 13 players, 12 educators with

experience in using games / gamification in class,

and 4 policy makers involved with games.

The unstructured interviews, which were

transcribed and analysed using a purposely-designed

codebook, considered a wide range of topics, from

the ethics of video game mechanics and contents to

the potential of video games for learning. Both

developers and researchers were explicitly asked

about how they saw the relationship between video

games research and development. Additionally,

game developers were asked whether they were

personally informed about academic social science

research and, if so, whether it influenced their

development practice. Educators, players and policy

makers were not directly asked about the topic, and

any mention of the issue on their part was

unsolicited.

3 RESULTS

3.1 Literature Review

Outcomes regarding goal (i) of the literature review

were fairly clear for the psychological and ethical

perspectives but less so for the educational

perspective. For psychology, six clear subfields of

active investigation emerged: immersion / presence,

motivation / flow, video game addiction, cognitive

benefits of gaming (especially for older adults),

health impact of games (e.g., exergames), and links

between violent video games and aggression.

Identified areas in the 'ethical' perspective were

violence and aggression, identity and inclusion (e.g.,

regarding gender, race, special needs), and game

monetization / manipulation. By contrast, keywords

in the educational perspective did not neatly cluster

around prevailing themes.

http://www.gdconf.com/

http://2017.gamehappens.com/

Results of goal (ii) fall outside the scope of the

present paper, which is focused on the disconnect

between social science research and game

development. However, it is worth noting here that

even extensively investigated research questions,

such as whether violent video games cause

aggressive behaviour, generated inconclusive

answers, sometimes with meta-analyses reporting

conflicting results (e.g. Ferguson, 2015, and

Greitemeyer and Mügge, 2014).

Regarding goal (iii) of the literature review, the

most striking result is the sheer amount of

recommendations directed from academic

researchers to other academic researchers. Of the 81

recommendations gleaned from the analysis of full

papers, 38 (46%) were directed to researchers, while

only 11 (12%) were addressed to developers. This

result is most evident for the psychological

perspective, in which recommendations directed to

other stakeholders are virtually absent, and most

recommendations focus on future lines of research.

This may in part stem from the fact that the 'future

research' section is often a requirement for papers

published in psychology journals (see, e.g.,

Sampson, 2012) and even where this is not the case

authors tend to qualify inconclusive results with the

caveat “more research is needed”.

The set of papers in the educational perspective

did yield recommendations directed to developers

and educators. These mostly regarded developing

more effective games for education (e.g. Merchant et

al., 2014) and using existing ones more effectively,

for example by making sure they align with learning

goals and that they are calibrated on students' needs

(Abdul Jabbar and Felicia, 2015; Tsekleves et al.,

2016). Additionally, Tsekleves et al. (2016) address

recommendations to policy makers for the adoption

of serious games in education.

Lastly, addressees for recommendations from the

ethical perspective vary by topic: recommendations

on violence and aggression are addressed to both

researchers and policy makers (e.g., Ferguson,

2015); recommendations on inclusion are addressed

to both researchers and developers (e.g., Ratan et al.,

2015); recommendations on monetization and

exploitation are directed to researchers and

developers (e.g., Heimo et al., 2016).

3.2 Interviews

As might be expected, most of the comments that

interviewees made regarding the research-

development relationship came from developers and

researchers themselves. Developers interviewed in

CSEDU 2018 - 10th International Conference on Computer Supported Education

136

the course of the project almost unanimously

reported feeling very distant from academic

research, and expressed considerable difficulty in

locating and accessing findings (e.g., “I must admit

I don’t read academic papers on games. To be

honest, I’m not sure where I’d go to find them if I

did”). When these developers did manage to source

research output, their unfamiliarity with social

sciences jargon made the contents difficult to grasp

(“A lot of academic language is also very, how

should I say this, specific. Almost to the point of

inaccessibility”). Those few developers who did

report having knowledge of social sciences

academic research were actually directly involved

with academia, either as researchers themselves or

as teacher/students of game design (“I ended up

mostly focusing on design but from a somewhat

academic angle”).

Furthermore, academic research was sometimes

written off because it didn’t focus on aspects

relevant to the commercialization of video games (“I

needed the research to prove people would buy this,

people would support it, we would see an increase in

sales, […] but there wasn't any research that I could

find for games that showed this”; “The studies that

are most useful to game developers from academia

seem to rarely be actually directly related to games;

they seem to be more related to human psychology

and general human computer interaction”).

However, one interviewee reported that this view

may be biased by unfamiliarity (“Maybe I might

start to get more into it, if I find something that

appeals to the kind of thing I like to read about”).

Aside from market concerns, when interviewees

made explicit the kind of research they would be

interested in, they reported wanting more research

from a narrative / artistic perspective (“I would love

to bring the humanities into what we’re doing

more”; “Narrative games and psychologically

challenging games would be the next step”). They

also expressed interest in user psychology

(“[Developers have] questions about psychology,

there are questions about human physiology and

how we respond to visual cues, things like that”;

“obviously psychology is one big area that is useful

and can be applied to games; if that research is

focused on the psychology within games,

entertainment games, that could really enhance

that”).

In general, the interviewed developers did report

being interested in research and innovation and

being kept up to date by reading non-academic

websites (“I try to stay up to date with research. I

couldn't name a specific group that does research.

[…] I will frequently go to Gamasutra to see if

there's anything posted there, or the GDC Vaults”; “I

don’t [remember any academic influences on my

work], unless it’s articles that come up on places like

Gamasutra”). They also expressed strong enthusiasm

for attending game developers’ conferences, which

were characterized as essential for exchanging

practical research information (“the most direct

influence comes from places like GDC when we’ve

come here to soak up the knowledge of other

developers”; “[GDC is] an inspiring place and it’s an

invaluable resource”; “if you want to learn the best

of what’s going on in the craft, you’ve got to go to

GDC San Francisco”). This positive opinion of non-

academic conferences was to be expected, since

most of the developer interviewees were recruited in

that context. An interesting point on the preference

for trade conferences over academic papers was

offered by an interviewee when commenting on the

pace of change of game industry (“I feel like we can

move faster at conferences than you can with

writing, writing and reading papers and books”). The

cautious pace of academia was also cited by another

interviewee as clashing somewhat with personal

priorities (“I’m less interested in the traditional

academic language of things and the lengthy writing

and sourcing and stuff, that I see the value of but

that costs me a lot of energy”).

Analysis of the researcher interviews revealed a

similar view of there being a separation between the

research priorities of academia and industry, with

industry being seen as more interested in the

commercial implications of design decisions.

However, a picture of compartmentalisation within

social science games research itself emerged. In this

regard, three distinct research trends can be

identified: one investigating causal effects of mainly

commercial video games (mostly coming from

psychological research); another that is outcome

oriented and focused on the use of applied or serious

games for educational purposes; and thirdly, an

emerging field that considers games as media and is

interested in how commercial games influence

cultural practices, identities and politics. This

partitioning of academic research was characterized

by some as problematic and stemming from a lack of

communication between different traditions,

especially between better funded outcome-oriented

research and less quantitatively-oriented

sociocultural investigations. (“Unlike some of the

games research that I tend to come across where

researchers are really interested in 'does a game

increase this, does a game lead to more motivation

or better outcomes', what I'm more interested in

Library Not Found

137

asking about games is more a process point of view

[…] actually really looking more closely at the role

of interactions, social interactions that are happening

around games. So, more a sociocultural perspective,

perhaps [...] [the dominant academic perspective]

would be the outcome oriented perspective. [...] The

kind of evidence that many funders are asking to

really prove is, I guess, that games are effective

learning tools”).

One area where social science research and game

development seem somewhat less disconnected is in

serious games R&D. It is not uncommon for this to

be undertaken within publicly-funded research

initiatives. However, economic pressures can make

it a harsh environment, especially for research

purposes (“there are companies that make a living

out of that. [...] you have to go in a short period,

understand something, turn it into a game, deliver it

in a way that it works. Otherwise you’re not going to

get the next gig. And that all happens in three, six

months if you’re lucky. [...] when you’re working

commercially you’re not pushing the boundaries,

you are literally trying to squeeze out”).

Whereas the spheres of research and game

development appear somewhat disconnected, social

sciences academic research seems more suited for

addressing the needs of educational innovation. This

was clearly borne out in the interviews with

educators, many of whom mentioned being directly

engaged in research activities and roles (“I first tried

out using a digital game in my school class […] I got

interested in games, also from educator’s point of

view, and later on from a researcher’s point of

view”). Some educators also mentioned sourcing

games-related research findings (“we wanted to play

a World War II based first person shooter with the

kids because we’d read some research that said that

the visual processes when you’re playing FPS, first

person shooters, are quite similar to the visual

processes that go on when you’re learning to read”).

This fluidity of pursuits and roles is not unusual in

the game based learning field.

However, similarly to the divergence in priorities

that characterizes the relationship between academic

research and commercial game development, some

interviewed educators called for support

mechanisms that are less theoretical and research

oriented and more practical in nature(“it’s more than

just bringing a game and playing in the classroom.

So we need to really create some form of framework

[...] not an academic tool [...] [but] a practical

tool”).

Some educators expressed the view that

research-led development of successful serious

games for learning is inherently fraught with

limitations (“quality obviously suffers, even when

the know-how is there, [unless] researchers can team

up with well-heeled developers”). Since a number of

educators interviewed expressed greater enthusiasm

for the employment of entertainment games, the

need of a closer relationship between developers and

educators was stated (“developers [should work]

with teachers […] and with the focus not to create

serious games, but to create games that enable

learning processes that can be [of] benefit to

teachers and students”).

When it came to making recommendations,

educators not only advocated closer collaboration

among the various stakeholder groups, they also

called for greater sharing of research results

(“Dissemination should be as broad as possible,

addressing institutions, players, developers, parents,

educators, etc.”)

During analysis of the player interviews there

was no expectation that interviewees would make

mention of academic research However, two of

them reported being familiar with academic research

on the use of entertainment video games for

rehabilitation (“I think there’s a study, they

compared Lumosity which was that game made by

neuro-scientists or whatever to help people with

memory and things like that. They did a kind of test

between people who did that and people who played

Portal”; “There was research at my school about

using Guitar Hero for hand rehabilitation, and I’m so

down with that […] There’s been a lot of research

on [using games for post-traumatic stress disorder].

I’ve tried to find some, but it’s really hard.”).

Another player was familiar with the psychological

research literature on addiction and very critical of

its stance (“addiction models in psychology in

general are tremendously flawed and written through

with a lot of implicit ideological assumptions about

how people should spend their leisure time

[…][There are] some very condescending opinions

on […] free will and [managing] one’s own time”).

None of the players commented directly on the

relationship between game development and

academic research, but some did report the feeling

that there was a great deal of research behind some

game design decisions (“[some free-to-play games]

are carefully studied and designed to be addictive”;

“they give an immediate visual and auditory

reward… there is for sure some careful study behind

[it]”).

CSEDU 2018 - 10th International Conference on Computer Supported Education

138

4 DISCUSSION

Both the literature review and the interviews yielded

valuable insights regarding factors contributing to

the disconnect between game research and game

development.

The first such factor is that social sciences

research has a tendency to be more tentative than

categorical in relaying its findings. When

considering games-related research, definite

conclusions are rare even for the most extensively

investigated topics, such as whether frequent gaming

activity leads to improvements in memory, attention,

etc. While caution in drawing conclusions from

(uncertain) results is certainly a good practice from

the scientific viewpoint, it may well prove

frustrating for a game developer seeking clear-cut

answers to inform practical decisions. Moreover,

researchers’ tentativeness in interpreting outcomes

often blinkers them from making pragmatic

considerations about the potential applicability of

those findings. This is compounded by the fact that

most of the recommendations we identified,

especially in the psychology literature, were

addressed to other researchers. Indeed, the relative

lack of recommendations directed to game

developers, educators, or policy makers may give

the impression that the social sciences academic

community is largely self-referential and

unconcerned with generating outcomes of practical

significance. However, it should be acknowledged

here that the full papers considered in the literature

review mainly comprised highly cited literature

reviews and meta-analyses, which could be seen as

part of the internal discourse of academia.

The interviews with developers confirmed this

characterization, as they believe that academic

research doesn’t offer answers to their questions.

This is not only a matter of overall inconclusiveness

of results; it is also a matter of the kind of questions

being asked. Obviously, those emerging from the

games industry, especially in the AAA sector, are

driven by the need to meet market demands for new

and better products. By contrast, the overriding goal

of social science research is to gain understanding of

gaming related phenomena, and is less concerned

about market applicability. However, a desire for

research with clearer practical applications was also

expressed by educators, who, as stakeholders, are

less bound by commercial considerations than

developers are.

A second factor contributing to the disconnect is

the different pace at which game development and

academic research advance. This, too, is connected

with the divergence in the respective priorities of

development and research: the former prioritises

rapid innovation and exploitation of market

opportunities, while the latter pursues steady

accumulation of knowledge and the careful drawing

of conclusions. From the viewpoint of game

developers, research risks being perpetually

outdated, investigating specific games or game

mechanics that have since lost their leading edge

position on the market. The (often considerable)

time lapse between a manuscript’s completion and

its publication exacerbates the untimeliness of

research (Björk and Solomon, 2013).

A third factor contributing to the disconnect is

the relatively low accessibility of much research

output for stakeholders outside academia. This

regards both the language used, described by some

interviewees as overly technical, and the channels

adopted for dissemination, which appear unfamiliar

to many. These accessibility factors seem to have

afflicted the interviewed developers in particular,

whereas comments from the educators we

interviewed - and even some of the players - suggest

that the language and culture of social science

research is more familiar to them. Accessibility

issues, combined with the other factors mentioned

above, may further discourage the game developer

from keeping up to date with current academic

understanding of game-related phenomena.

Lastly, the Gaming Horizons literature review

and the interviews revealed how strongly social

science research focuses on serious games, which is

perhaps not surprising given that they are designed

with the explicit intent of generating specific – and

presumably measurable – outcomes. Considered

collectively as a stakeholder group, however, game

developers are more commonly concerned with the

creation of games for entertainment. The impression

that social science research is ‘looking elsewhere’,

so to speak, limits developers’ potential interest in it.

On this point, it’s important to note that some of the

educators we interviewed were chiefly interested in

the use of commercial entertainment games for

educational purposes. This could well represent a

research line that serves as a meeting point for game

developers, researchers, and educators alike.

Before proposing concrete steps to address the

research-development disconnect, it should be

acknowledged that these two spheres embody

priorities and pursue aims that are quite distinct from

each other. So the spaces that they occupy in the

games landscape cannot be totally bridged, nor is it

desirable to do so. That said, however, steps may be

taken to ameliorate connections for mutual benefit.

Library Not Found

139

One possible step is the promotion of

information sources that sit in between the academic

and development worlds, such as conferences

addressed to and involving both researchers and

developers. Blended conferences present four main

advantages: one, they allow relatively rapid

dissemination of scientific results; two, participants

from both backgrounds have an opportunity, and a

strong incentive, to communicate in a way that is

readily understandable outside their own cohort;

three, conferences are generally more familiar to

developers as a dissemination channel than

academic publishing is; and four, they present an

opportunity for personal interaction that may spark

collaboration and the eventual formation of

multidisciplinary endeavours.

Another possible area of action is to encourage

social science researchers who are seeking to

disseminate findings on games and gaming to target

channels like websites, blogs and social networks

that are popular with the game development

community, adjusting their message and language to

suit this audience. For example, the Gamasutra

website

was mentioned by a number of interviewed

developers as the primary hub for information on

innovation in game development.

Another step that may be taken to reduce the

disconnect is to encourage greater and more diverse

social science research into entertainment games and

gaming, seen as opportunities for leisure, as socio-

cultural phenomena and as potential tools for

application in fields like education, health, and

rehabilitation. Gaming Horizons interviews with

educators suggest that the employment of

entertainment games for learning may avoid some of

the problems they encounter with serious games,

especially regarding the effect of production values

on student engagement (e.g., outdated graphics

limiting engagement). Shifting the focus of social

science research away from the development of

games with specific purposes and more towards

investigation of the various ways entertainment-

oriented games might be geared and/or harnessed for

learning could well help to bring developers,

researchers and educators together for mutual

benefit. Additionally, considering games as socio-

cultural tools can open new ways of using them in

education, such as using game narratives and

mechanics for introducing discussions on ethics.

Furthermore, much stands to be gained from

initiatives and mutual agreements supporting wider

access to data sources. The Open Science movement

https://www.gamasutra.com

is producing important effects in terms of policies

about data and publication openness which are

making research results more and more accessible.

On the other hand, the industry gathers a wealth of

critical data from gaming ‘in the wild’. This could

give social science research new insight into areas

like gamer behaviour and group dynamics in virtual

environments, knowledge that would prove

particularly beneficial for designing and developing

game experiences that have strong market potential.

However, our interviews reveal that business

competitiveness is still hindering this side of the

communication channel between research and game

development world.

The signs of interest in academic research, both

from a social sciences and a humanities perspective,

expressed by game developers we interviewed

strengthens our belief that there is indeed room for

fruitful collaboration between the two ‘worlds’.

ACKNOWLEDGEMENTS

The Gaming Horizon project has received funding

from the European Union's Horizon 2020 research

and innovation programme under grant agreement

No 732332.

REFERENCES

Abdul Jabbar, A.I. & Felicia, P., 2015. Gameplay

engagement and learning in game-based learning: A

systematic review. Review of Educational Research,

85(4), pp.740–779. Available at: http://rer.

sagepub.com/cgi/doi/10.3102/0034654315577210.

Björk, B.-C. & Solomon, D., 2013. The publishing delay

in scholarly peer reviewed journals. Journal of

Informetrics, In Proof(4), pp.1–25. Available at:

http://openaccesspublishing.org/oa11/article.pdf.

Blow, J., 2004. Game Development: Harder Than You

Think. Queue, 1(10), p.28. Available at: http://

portal.acm.org/citation.cfm?doid=971564.971590.

Boot, W.R., Blakely, D.P. and Simons, D.J., 2011. Do

action video games improve perception and cognition?

Frontiers in Psychology, 2(226), p.226. Available at:

http://dx.doi.org/10.3389/fpsyg.2011.00226%5Cnhttp:

//www.frontiersin.org/cognition/10.3389/fpsyg.2011.0

0226/abstract

Dickey, M.D., 2005. Engaging by design: How

engagement strategies in popular computer and video

games can inform instructional design. Educational

Technology Research and Development, 53(2), pp.67–

83. Available at: http://link.springer.com/10.1007/

BF02504866.

CSEDU 2018 - 10th International Conference on Computer Supported Education

140

Ferguson, C.J., 2015. Do Angry Birds make for angry

children? A meta-analysis of video game influences on

children’s and adolescents’ aggression, mental health,

prosocial behavior, and academic performance.

Perspectives on Psychological Science, 10(5), pp.646–

666. Available at: http://journals.sagepub.com/doi/

10.1177/1745691615592234.

Greitemeyer, T. and Mügge, D.O., 2014. Video games do

affect social outcomes: a meta-analytic review of the

effects of violent and prosocial video game play.

Personality & Social Psychology Bulletin, 40(5),

pp.578–89. Available at: http://psp.sagepub.com/cgi/

content/long/40/5/578.

Gros, B., 2007. Digital games in education: The design of

games-based learning environments. Journal of

Research on Technology in Education, 40(1), pp.23–

38. Available at: https://doi.org/10.1080/

15391523.2007.10782494.

Heimo, O.I., Harviainen, J.T., Kimppa, K.K. and Mäkilä,

T., 2016. Virtual to virtuous money: A virtue ethics

perspective on video game business logic. Journal of

Business Ethics, pp.1-9. Available at: https://doi.org/

0.1007/s10551-016-3408-z

Kultima, A., 2015. Game Design Research. Academic

Mindtrek ’15, pp.18–25. Available at:

http://dl.acm.org/citation.cfm?doid=2818187.2818300.

Merchant, Z., Goetz, E.T., Cifuentes, L., Keeney-

Kennicutt, W. and Davis, T.J., 2014. Effectiveness of

virtual reality-based instruction on students' learning

outcomes in K-12 and higher education: A meta-

analysis. Computers & Education, 70, pp.29-40.

Available at:

https://doi.org/10.1016/j.compedu.2013.07.033

Ondrejka, C., 2006. Finding Common Ground in New

Worlds. Games and Culture, 1(1), pp.111–115.

Available at: http://gac.sagepub.com/cgi/content/

abstract/1/1/111%5Cnhttp://gac.sagepub.com/content/

1/1/111.full.pdf.

Orvis, K.A., Horn, D.B. & Belanich, J., 2008. The roles of

task difficulty and prior videogame experience on

performance and motivation in instructional

videogames. Computers in Human Behavior, 24(5),

pp.2415–2433. Available at: https://doi.org/10.1016/

j.chb.2008.02.016

Papastergiou, M., 2009. Exploring the potential of

computer and video games for health and physical

education: A literature review. Computers &

Education, 53(3), pp.603–622. Available at:

http://linkinghub.elsevier.com/retrieve/pii/S03601315

09000840.

Persico, D., Bailey, C., Buijtenweg, T., Dagnino, F., Earp,

J., Haggis, M., Manganello, F., Passarelli, M.;

Perrotta, C. and Pozzi, F. (2017a). Systematic Review

and Methodological Framework, Gaming Horizons

Deliverable D2.1. Available at: http://

www.gaminghorizons.eu/wp-content/uploads/sites/18/

2017/05/D2.1-State-of-the-Art-Literature-review.pdf

Persico, D., Dagnino, D., Earp, J., Manganello, F.,

Passarelli, M., Pozzi, F., Haggis, M. Buijtenweg, T.,

Perrotta, C. and Bailey, C. (2017b). Report on

interviews with experts and informants, Gaming

Horizons Deliverable D2.3. Available at: https://

www.gaminghorizons.eu/wp-content/uploads/sites/18/

2017/09/D2.3_Interviews-report.pdf

Powers, K.L. et al., 2013. Effects of video-game play on

information processing: A meta-analytic investigation.

Psychonomic Bulletin & Review, 20(6), pp.1055–

1079. Available at: http://link.springer.com/10.3758/

s13423-013-0418-z.

Przybylski, A.K., Rigby, C.S. & Ryan, R.M., 2010. A

motivational model of video game engagement.

Review of General Psychology, 14(2), pp.154–166.

Available at: http://doi.apa.org/getdoi.cfm?

doi=10.1037/a0019440.

Randel, J.M., Morris, B.A., Wetzel, C.D. and Whitehill,

B.V., 1992. The effectiveness of games for

educational purposes: A review of recent research.

Simulation & gaming, 23(3), pp.261-276. Available at:

http://journals.sagepub.com/doi/10.1177/10468781922

33001.

Ratan, R.A. et al., 2015. Stand by your man: An

examination of gender disparity in League of Legends.

Games and Culture, 10(5), pp.438–462. Available at:

http://gac.sagepub.com/cgi/doi/10.1177/15554120145

67228.

Sampson, J.P., 2012. A guide to quantitative and

qualitative dissertation research. Educational

Psychology and Learning Sytems Faculty

Publications, pp.1–93. Available at: http://diginole.

lib.fsu.edu/edpsy_faculty_publicatons/1

Tsekleves, E., Cosmas, J. and Aggoun, A., 2016. Benefits,

barriers and guideline recommendations for the

implementation of serious games in education for

stakeholders and policymakers. British Journal of

Educational Technology, 47(1), pp.164–183.

Available at: https://doi.org/10.1111/bjet.12223

Wender, S. and Watson, I., 2012. Applying reinforcement

learning to small scale combat in the real-time strategy

game StarCraft: Broodwar. In 2012 IEEE Conference

on Computational Intelligence and Games, CIG 2012.

pp. 402–408. Available at: https://doi.org/10.1109/

CIG.2012.6374183

Wouters, P., Van Nimwegen, C., Van Oostendorp, H. and

Van Der Spek, E.D., 2013. A meta-analysis of the

cognitive and motivational effects of serious games.

Journal of Educational Psychology, 105(2), pp.249–

265. Available at: http://doi.apa.org/

getdoi.cfm?doi=10.1037/a0031311.

Library Not Found

141