Crowdsourced Software Development in Civic Apps

Motivations of Civic Hackathons Participants

Kiev Gama

Centro de Inform

atica, Universidade Federal de Pernambuco (UFPE), Recife, Brazil

Keywords:

Software Crowdsourcing, Hackathon, Motivation, Civic Apps.

Abstract:

Hackathons are intensive events that typically last from 1 to 3 days, where programmers and sometimes peo-

ple with interdisciplinary backgrounds (e.g., designers, journalists, activists) collaborate to develop software

applications to overcome a challenge proposed by the event organizers. Civic hackathons are a particular type

of hackathon that gained momentum in the last years, mainly propelled by city halls and government agencies

throughout the world as a way to explore public data repositories. These initiatives became an attempt to

crowdsource the development of software applications targeting civic issues. Some articles in academic litera-

ture have conﬂicting arguments about factors that motivate developers to create such apps. Claims are mostly

based on anecdotal evidence since research is still scarce in the empirical analysis of civic hackathons. Thus,

we decided to do a study to gather data under the perspective of hackathon participants, focusing on what

motivation factors make them join such competitions. We conducted a survey research where we intended to

provide empirical evidence for a diverse audience (e.g., hackathon organizers, open data specialists) interested

in civic hackathons as a form of software crowdsourcing. In this work, we present preliminary results.

1 INTRODUCTION

Hackathons are continuous events that engage peo-

ple in small groups to produce software in a limited

amount of time, typically lasting from 1 to 3 days

(Komssi et al., 2015). They gained popularity in

technology companies in the 2000s as a way to pro-

mote exploratory coding, new idea generation, and

prototyping with low-risk (Carr and Lassiter, 2017).

Leveraged by many government open data initiatives,

which aim at the increase of public transparency, civic

hackathons or ”app contests” became an opportunity

for governments to invest in crowdsourced software

as a new form of procurement (Johnson and Robin-

son, 2014). These are issue-oriented hackathons, fo-

cused on governance and public life, that go in a dif-

ferent direction of corporate hackathons, which con-

centrate on speciﬁc technologies and business oppor-

tunities (DiSalvo et al., 2014).

Initial competitions for ”civic apps” development

brought the argument that a few thousands of dollars

invested in prizes would generate crowdsourced soft-

ware that altogether would cost millions of dollars to

develop (Lee et al., 2015). However, in recent years,

the potential of these contests started to be criticized

due to numerous issues such as the apps’ utility (Carr

and Lassiter, 2017) or quality (Johnson and Robinson,

2014), and the hackathons process (Ferrario et al.,

2014) or scope deﬁnition (Lee et al., 2015).

Apps developed in civic hackathons are said to be

quickly abandoned by users because of a lack of fo-

cus on the citizen (Townsend, 2013). These claims are

in part conﬁrmed by a study(Lee et al., 2015) report-

ing that developers participating in these competitions

have a limited perspective of the city problems and lit-

tle experience with city services. Therefore, they usu-

ally create what they think to be appealing, guided by

their personal experience. However, that study also

brings into discussion other aspects such the appli-

cations being of low quality. Johnson and Robinson

(Johnson and Robinson, 2014) also mention the vari-

able quality and highlight that the gap between design

and implementation signals the importance of track-

ing the outcomes of civic hackathons.

Besides these claims about the results of civic

hackathons, we also found contradictions in the lit-

erature about the motivation of participants to join

these events. While some studies say participants

join these competitions because of money and prizes

(Almirall et al., 2014), others allege that those are the

least important motivation factors (Hartmann et al.,

2016) (Juell-Skielse et al., 2014). Briscoe and Mul-

550

Gama, K.

Crowdsourced Software Development in Civic Apps - Motivations of Civic Hackathons Participants.

DOI: 10.5220/0006377005500555

In Proceedings of the 19th International Conference on Enterprise Information Systems (ICEIS 2017) - Volume 2, pages 550-555

ISBN: 978-989-758-248-6

ligan (Briscoe and Mulligan, 2014) present the top

three reasons for attending a hackathon as learning,

networking, and social change, while prizes come in

fourth place.

After looking for empirical studies with deeper

analysis and evidence to support all of those claims,

we found academic research about civic hackathons

to be scarce and limited. Johnson ad Robinson (John-

son and Robinson, 2014) mention that fact which is

reinforced by more recent work from Carr and Las-

siter (Carr and Lassiter, 2017). Even though crowd-

sourced labor is one of the most common features of

civic hackathons (DiSalvo et al., 2014), we still found

limited literature on software engineering research

focusing on civic hackathons as a way of software

crowdsourcing. Currently, studies of crowdsourcing

in software engineering (Mao et al., 2016) are rather

focused on a perspective where global software engi-

neers are recruited through open calls in online plat-

forms. Concerning motivation in civic hackathons,

most of the work is either addressed to more gen-

eral forms of crowdsourcing (Leimeister et al., 2009)

(Hossain, 2012) or brings limited studies sampling

the participants of just one competition (Juell-Skielse

et al., 2014) (Decker et al., 2015).

We decided to do a preliminary study to gather

data under the perspective of hackathon participants,

focusing on two points: (1) how they are dealing with

software engineering activities and (2) what motiva-

tion factors make they join such competitions. Based

on that, we did an online survey with civic hackathon

participants from different countries. We presented

the results around point (1) in another work (Gama,

2017), while the results around point (2) we detail

here. What we explore the current article is based

on the following research question: ”What are the

main motivation factors of participants in civic

hackathons?”. With this study, we want to gain an

initial understanding on that topic as well as to verify

if our preliminary results can support the claims we

found in literature. It also allows us to provide empir-

ical evidence for a diverse audience (e.g., hackathon

organizers, open data specialists) interested in civic

hackathons as a form of software crowdsourcing.

The remainder of this paper presents related work

in section 2, section 3 presents the methodology we

employed, section 4 discusses the results of our sur-

vey, followed by our conclusions in section 5.

2 RELATED WORK

From a general perspective on crowdsourcing, there

are some studies highlighting aspects that motivate

people to join such practices. Leimeister et al.

(Leimeister et al., 2009) analyze motives and in-

centives that lead to participation in information

technology-based ideas competition. Intrinsic moti-

vation is related to behavior initiated without exter-

nal incentives (e.g. a hobby). External motivation is

activated by external incentives (e.g., direct or indi-

rect monetary compensation, recognition by others).

They allege that competitions organizers are not able

to inﬂuence intrinsic motivation. In a crowdsource

competition organized by a large technology com-

pany, Leimester and colleagues found that prizes were

an important motivation aspect after ”Appreciation by

the organizer.”

These ﬁndings go in a different direction from

other studies, like (Seltzer and Mahmoudi, 2012) who

emphasize that any compensation by money or re-

wards is less important than more subjective aspects

such as the prospect for public recognition or satis-

fying inner desires. Hossain (Hossain, 2012) per-

formed a study on motivation in crowdsourcing where

he mentions intrinsic incentives as the dominant mo-

tivational factor in open source software. When ana-

lyzing crowdsourcing platforms, the author arrives at

the conclusion that extrinsic motivation is more domi-

nant than intrinsic motivation. However, that analysis

is limited since it is solely based on the analysis of

platforms, without ever collecting opinions of people

that participate in crowdsourcing.

If we narrow down the scope to studies focusing

on civic apps development on different types of civic

app competitions (e.g., civic hackathons), we also see

contradiction about the importance of prizes. Almi-

rall et al. (Almirall et al., 2014) support the argument

that instead of being motivated by civic engagement,

hackathon and application development contest de-

velopers are driven by entrepreneurship and the re-

wards of the contests. There is no data presented

by the authors to support those claims. Decker et

al. (Decker et al., 2015) found that the social as-

pect and the community involvement aspect were rel-

evant to some of the participants of a non-competitive

hackathon they organized. (Lee et al., 2015) mention

that developers go to civic app contests not only af-

ter money but also looking for visibility that would

help them get investments. In a study of 24 civic

app contests (Hartmann et al., 2016), researchers con-

ﬁrmed that civic hackathons are typically centered on

the support of open data usage and development of

useful services for citizens. Participants are driven

by motivation around engaging in the construction of

that as well as meeting new people, discussing ideas

and improving skills. Juell et al. (Juell-Skielse et al.,

2014) performed a survey with participants of a 24-

Crowdsourced Software Development in Civic Apps - Motivations of Civic Hackathons Participants

551

hour civic hackathon held in Sweden; the authors

found that the top three triggers of motivation were

all related to intrinsic motivation: fun and enjoyment,

intellectual challenge, and status and reputation.

Briscoe and Mulligan (Briscoe and Mulligan,

2014) on a survey with 150 participants from the USA

reported that the motivation for these people to partic-

ipate in hackathons was learning, networking, social

change, win prizes, free pizza, build a product, glory,

ﬁnd a team, ﬁnd employment, and attract investors, in

that order.

Hackathons, in general, have been used as a learn-

ing platform (Nandi and Mandernach, 2016), where

students can teach and learn from their peers. Partic-

ipating students spawned creativity and enthusiasm.

The hackathons established valuable mentorship con-

nections with alumni. The authors mention that stu-

dents involved in these hackathons tend to have better

grades than those who did not participate. We found

this related to the different forms of intrinsic motiva-

tions, from an interpersonal level, such as cooperation

and recognition, which are supported by (Malone and

Lepper, 1987) in the context of learning.

3 METHOD

In our study, we used a survey to gain a broad un-

derstanding on how participants of civic hackathons

perceive the software development practices they em-

ploy in the different project lifecycle activities and

what factors they consider more important as moti-

vation for joining such competitions.

3.1 Research Question

When performing survey research, having clear re-

search questions is a precondition (Easterbrook et al.,

2008). Before deﬁning the questionnaire, we deﬁned

our research questions that would be broken down

into a more detailed questionnaire. Our research fo-

cuses on two concerns: (1) how civic hackathons are

dealing with software engineering activities and (2)

what motivation factors make they join such competi-

tions. In complementary work (Gama, 2017), we dis-

cuss research questions concentrated on the ﬁrst con-

cern (1) while the second one is depicted below:

RQ: ”What are the main motivation factors of

participants in civic hackathons?”. We try to un-

derstand the motivations of people that participate in

civic hackathons. For instance, participants of civic

hackathons are driven by motivations such as meeting

new people, discussing ideas, and improving skills

rather than going after money (Hartmann et al., 2016).

Table 1: Motivation factors asked to civic hackathon par-

ticipants. The answer was measured in a ﬁve-point Likert

scale, ranging from ”not important at all” (1) to ”very im-

portant” (5).

Motivation Factor

MF1 Prizes (cash, products)

MF2 Engaging in the resolution of civic problems

MF3 Learning and developing new skills

MF4 Performing teamwork

MF5 Networking (make contacts, meet new people)

MF6 Increase your visibility in the community

3.2 Questionnaire Design

In agreement with the research question, the question-

naire was formulated based on our ﬁeld observation

from previous experiences in the organization of civic

hackathons. We validated it with other two hackathon

organizers who suggested rephrasing some of the sen-

tences to avoid ambiguity.

In a previous attempt of an exploratory survey

employing open questions, we did not get many re-

sponses after sending out emails to participants from

civic hackathons. Thus, instead of open-ended ques-

tions, in the survey instrument described here we

focused on multiple choice questions, which were

grouped into three sections:

• Participant proﬁle: Since teams in civic

hackathons may mix people with different

backgrounds, we asked about the participant’s ex-

perience and their roles in the project (developer,

designer, project manager, activist).

• Software Engineering Practices: This section

contained questions about different project activ-

ities: requirements gathering, software design,

project management, development, conﬁguration

management, testing, software release, and main-

tenance. The results from this section were

explored and detailed further in another work

(Gama, 2017).

• Motivation: In this ﬁnal section, which is the fo-

cus of this article, we used a ﬁve-point Likert scale

(ranging from ”not important at all” to ”very im-

portant”) to ask participants about the importance

of different factors that may have motivated them

in the hackathon. Table 1 details the motivation

factors we asked participants.

3.3 Population and Sampling

Certain scenarios make difﬁcult to gather probabil-

ity samples from a signiﬁcant part of the target pop-

ulation. Trying to reach out participants of civic

hackathons who were willing to answer a survey

ICEIS 2017 - 19th International Conference on Enterprise Information Systems

552

questionnaire was not an easy task. Therefore, our

best option was to gather non-probability samples,

which is supported by Kitchenham (Kitchenham and

Pﬂeeger, 2008) in situations when the target popula-

tion is very speciﬁc and of limited availability. Re-

sponses from a convenience sample might be useful

in developing research hypotheses in early stages of

research (Fricker, 2008), which is our case.

We use a Google Form as the online question-

naire that collected the data. The sampling was per-

formed by sending the survey to participants of three

civic hackathons: Hackacity

, GovHack

and Hacker

Cidad

. These competitions were held in May, July

and August of 2016, and January of 2017 respec-

tively. The ﬁrst one took place held in Australia/New

Zealand while the other two happened in Brazil.

GovHack is one the worlds biggest annual open

data hackathon, attracting over 3,000 participants

from 40 different locations in the 2016 in Australia

and New Zealand

. The access to them was done

online during the last day of the hackathon through

messages posted on Twitter by one the organizers and

through Slack

In the other two of civic hackathons we were the

organizers: Hackacity and Hacker Cidad

ao). The for-

mer was a 24-hour event that was held simultaneously

in four cities worldwide, focused on solutions that

will have an impact in the city. We applied the sur-

vey to 45 participants in the Brazilian side. The latter

is the ofﬁcial hackathon from the City Hall of Recife,

Brazil, which was held in two rounds on two consec-

utive weekends, involving 46 competitors. The goal

was to create civic applications that use data from the

city’s Open Data Portal as well as from connected

sensors. All participants from both hackathons re-

ceived the survey link by email in the week after the

event. We sent a reminder email a week later.

We also used the Facebook social network to post

the link to the survey in 12 civic hackathon communi-

ties and also sent it to some civic hackers mailing lists.

However, the data collected was not representative of

any demographics (e.g., region, country) therefore it

was not used in our analysis.

https://www.hackacity.eu/

https://www.govhack.org/

http://hackercidadao.rec.br/

https://accgh16.alan.id.au/wp-

content/uploads/2016/11/GovHack-2016-Review.pdf

http://govhackhq.slack.com

4 RESULTS

Our questionnaire had 123 respondents from three

civic hackathons: GovHack (55 responses), Hackac-

ity (32 responses) and Hacker Cidad

ao (36 re-

sponses). We used the same set of responses in

our analysis of software engineering practices (Gama,

2017), where we decided to consider a subset of re-

sponses only from participants who marked ”devel-

oper” as their role in the team. However, in this

study of motivation factors presented here, we exam-

ine the responses from all participants, from all roles.

We decided to partition our responses grouping them

by hackathons. This partitioning approach is useful

when one may want to compare responses from dif-

ferent subgroups, reporting it separately, and can also

be used to alleviate initial design errors of the survey

(Kitchenham and Pﬂeeger, 2008).

4.1 Motivation Factors

The motivation factors were analyzed for all re-

sponses, since we were interested in the motivation

of all participants, regardless of their proﬁle or back-

ground. A summary of the results is presented in

Table 2. We chose to consider the values from the

Likert scale as intervals (Brown, 2011), which al-

lowed to use descriptive statistics that helped sum-

marize the data from our sample and understand the

central tendencies. A similar approach was used in

(Juell-Skielse et al., 2014) to analyze the main moti-

vation factors of participants in a 24-hour hackathon.

4.2 Discussion

This section discusses survey results from the per-

spective of the research question ”What are the

main motivation factors of participants in civic

hackathons?”. The discussion is centered around the

data presented in Table 2.

Contradicting claims we found concerning the

motivation around prizes (Almirall et al., 2014), our

results show it was the least important motivation

factor for the civic hackathon participants we sur-

veyed, as illustrated in the answers of MF1. If we

look at the top three motivation factors for respon-

dents of those competitions, learning and develop-

ing new skills (MF3) was rated as the most impor-

tant factor by respondents of two competitions (Gov-

Hack and Hackacity) and as the third most impor-

tant in Hacker Cidad

ao. This reinforces the perspec-

tive of a hackathon being also a learning environ-

ment (Nandi and Mandernach, 2016). Networking

(MF5) was considered as the second most important

Crowdsourced Software Development in Civic Apps - Motivations of Civic Hackathons Participants

553

Table 2: Mean, standard deviation, mode and median of the results about motivation factors (see Table 1) in civic hackathon

participants.

Motivation

GovHack (N=55) Hackacity (N=32) Hacker Cidadao (N=36)

Mean SD Mode Median Mean SD Mode Median Mean SD Mode Median

MF1 2.73 1.08 3 3 3.34 1.1 3 3 3.81 0.95 4 4

MF2 3.85 0.99 4 4 4.47 0.67 5 5 4.75 0.65 5 5

MF3 4.22 0.83 5 4 4.78 0.42 5 5 4.72 0.57 5 5

MF4 4.11 0.98 5 4 4.44 0.8 5 5 4.64 0.68 5 5

MF5 4.05 1.03 5 4 4.56 0.91 5 5 4.75 0.50 5 5

MF6 4 1 4 4 4.22 0.97 5 4.5 4.53 0.77 5 5

factor for respondents from two hackathons (Hackac-

ity and Hacker Cidad

ao) and as the third most im-

portant factor for GovHack respondents. Engaging

in the resolution of civic problems (MF2) was the

most important factor in the responses from Hacker

Cidad

ao. Performing teamwork (MF4) was the sec-

ond most important factor in the answers from Gov-

Hack but was less important for respondents of the

other hackathons.

4.3 Threats to Validity

Like any research, this one poses some threats to va-

lidity. Easterbrook et al. (Easterbrook et al., 2008)

state that in survey research one of the major chal-

lenges is to control for sampling bias. Due to that risk,

it is dangerous to draw strong inferences from the

samples (Kitchenham and Pﬂeeger, 2008). It applies

to this survey, which has a limited representation of

the target population. The samples have only the per-

spective of participants from three hackathons, which

is a limiting factor. The professional background of

participants was not collected, which may lead to an-

other bias. For instance, students may be more mo-

tivated to learning. According to general estimates

from hackathon organizers, the overall proﬁle of Gov-

Hack participants is a fairly distributed proportion of

both professionals and students, while Hackacity and

Hacker Cidad

ao had a slight majority of students.

The low response rate from GovHack also is a ma-

jor limitation. Researcher bias also poses threats to

validity in the formulation of questions. Our experi-

ence in the organization of civic hackathons may have

inﬂuenced the questions regarding the practices ob-

served in hackathon participants, instead of going to a

broader view. The survey has a probability of having

responses from members of the same team, however,

we intended to measure individual perceptions.

5 CONCLUSIONS

Many government agencies have been exploring

hackathons as a way to foster the usage of public open

data and as an attempt to crowdsource the develop-

ment of civic applications. We can ﬁnd in literature

criticism about the resulting applications (e.g., bad re-

quirements deﬁnition, lack of quality in the applica-

tions, incomplete work) and contradictions about the

real motivations of participants in those competitions.

However, those claims are mostly based on anecdo-

tal evidence. We collected data in a survey with civic

hackathon participants in order to ﬁnd empirical evi-

dence. Results concerning applications are presented

in another work (Gama, 2017), while motivation fac-

tors are detailed and analyzed in the current work.

After examining the data collected with partici-

pants from three civic hackathons, we could ﬁnd sup-

port to some of the claims reported in the literature.

When looking into the motivation of participants, we

found that prizes are the least important factor. The

top three motivation factors were learning and devel-

oping new skills, networking with other people, and

engagement in the resolution of civic problems. This

is aligned with the ﬁndings of (Briscoe and Mulligan,

2014), who reported similar results in another study

involving 150 hackathon participants from the USA,

where the top three reasons for attending a hackathon

were learning, networking and social change.

Among future work to be pursued, we plan to per-

form a broader study involving more competitions

with pre- and post-hackathon interviews, to gather

data that can reveal motivation factors we could not

ﬁnd or were not apparent in literature.

ACKNOWLEDGEMENTS

This research was partially funded by INES 2.0,

FACEPE PRONEX project APQ 0388-1.03/14. The

author acknowledges support from the organiza-

tion of GovHack (Richard Tubb and Kathy Reid),

Hacker Cidad

ao (Breno Alencar and Eugenio An-

ICEIS 2017 - 19th International Conference on Enterprise Information Systems

554

tunes) and Hackacity (Margarida Campolargo and

Claudio Nascimento).

REFERENCES

Almirall, E., Lee, M., and Majchrzak, A. (2014). Open in-

novation requires integrated competition-community

ecosystems: Lessons learned from civic open innova-

tion. Business Horizons, 57(3):391–400.

Briscoe, G. and Mulligan, C. (2014). Digital innovation:

The hackathon phenomenon. London: Creativeworks

London Work Paper, 6.

Brown, J. D. (2011). Likert items and scales of mea-

surement. Shiken: JALT Testing & Evaluation SIG

Newsletter, 15(1):10–14.

Carr, S. J. and Lassiter, A. (2017). Big data, small

apps: premises and products of the civic hackathon.

In Seeing Cities Through Big Data, pages 543–559.

Springer.

Decker, A., Eiselt, K., and Voll, K. (2015). Understand-

ing and improving the culture of hackathons: Think

global hack local. In Frontiers in Education Con-

ference (FIE), 2015. 32614 2015. IEEE, pages 1–8.

IEEE.

DiSalvo, C., Gregg, M., and Lodato, T. (2014). Building

belonging. interactions, 21(4):58–61.

Easterbrook, S., Singer, J., Storey, M.-A., and Damian, D.

(2008). Selecting empirical methods for software en-

gineering research. In Guide to advanced empirical

software engineering, pages 285–311. Springer.

Ferrario, M. A., Simm, W., Newman, P., Forshaw, S., and

Whittle, J. (2014). Software engineering for’social

good’: integrating action research, participatory de-

sign, and agile development. In Companion Proceed-

ings of the 36th International Conference on Software

Engineering, pages 520–523. ACM.

Fricker, R. D. (2008). Sampling methods for web and e-

mail surveys. N. Fielding, pages 195–216.

Gama, K. (2017). Preliminary ﬁndings on software engi-

neering practices in civic hackathons. In Proceedings

of the 4th International Workshop on CrowdSourcing

in Software Engineering.

Hartmann, S., Mainka, A., and Stock, W. G. (2016). Op-

portunities and challenges for civic engagement: A

global investigation of innovation competitions. In-

ternational Journal of Knowledge Society Research

(IJKSR), 7(3):1–15.

Hossain, M. (2012). Crowdsourcing: Activities, incentives

and users’ motivations to participate. In Innovation

Management and Technology Research (ICIMTR),

2012 International Conference on, pages 501–506.

IEEE.

Johnson, P. and Robinson, P. (2014). Civic hackathons: In-

novation, procurement, or civic engagement? Review

of Policy Research, 31(4):349–357.

Juell-Skielse, G., Hjalmarsson, A., Johannesson, P., and

Rudmark, D. (2014). Is the public motivated to engage

in open data innovation? In International Conference

on Electronic Government, pages 277–288. Springer.

Kitchenham, B. A. and Pﬂeeger, S. L. (2008). Personal

opinion surveys. In Guide to Advanced Empirical

Software Engineering, pages 63–92. Springer.

Komssi, M., Pichlis, D., Raatikainen, M., Kindstr

om, K.,

and J

arvinen, J. (2015). What are hackathons for?

IEEE Software, 32(5):60–67.

Lee, M., Almirall, E., and Wareham, J. (2015). Open

data and civic apps: ﬁrst-generation failures, second-

generation improvements. Communications of the

ACM, 59(1):82–89.

Leimeister, J. M., Huber, M., Bretschneider, U., and

Krcmar, H. (2009). Leveraging crowdsourcing:

activation-supporting components for it-based ideas

competition. Journal of management information sys-

tems, 26(1):197–224.

Malone, T. W. and Lepper, M. R. (1987). Making learning

fun: A taxonomy of intrinsic motivations for learning.

Aptitude, learning, and instruction, 3(1987):223–253.

Mao, K., Capra, L., Harman, M., and Jia, Y. (2016). A

survey of the use of crowdsourcing in software engi-

neering. Journal of Systems and Software.

Nandi, A. and Mandernach, M. (2016). Hackathons as

an informal learning platform. In Proceedings of the

47th ACM Technical Symposium on Computing Sci-

ence Education, pages 346–351. ACM.

Seltzer, E. and Mahmoudi, D. (2012). Citizen participation,

open innovation, and crowdsourcing: Challenges and

opportunities for planning. Journal of Planning Liter-

ature, page 0885412212469112.

Townsend, A. M. (2013). Smart cities: Big data, civic hack-

ers, and the quest for a new utopia. WW Norton &

Company.

Crowdsourced Software Development in Civic Apps - Motivations of Civic Hackathons Participants

555