An Exploratory Study on the Evidence of Hackathons’ Role in Solving

OSS Newcomers’ Challenges

Ahmed Samir Imam Mahmoud

1 a

, Alexander Nolte

1,2 b

and Dietmar Pfahl

1 c

University of Tartu, Estonia

Carnegie Mellon University, Pittsburgh, PA, U.S.A.

Keywords:

OSS, Open-Source, Hackathon, Newcomers, Challenges, Barriers, Evidence.

Abstract:

Background: OSS projects face various challenges. One major challenge is to onboard and integrate newcom-

ers to the project. Aim: We aim to understand and discuss the challenges newcomers face when joining an

OSS project and present evidence on how hackathons can mitigate those challenges. Method: We conducted

two searches on digital libraries to (1) explore challenges faced by newcomers to join OSS projects, and (2)

collect evidence on how hackathons were used to address them. We deﬁned four evidence categories (posi-

tive, inconclusive, and no evidence) to classify evidence how hackathons address challenges. In addition, we

investigated whether a hackathon event was related to an OSS project or not. Result: We identiﬁed a range of

newcomer challenges that were successfully addressed using hackathons. However, not all of the solutions we

identiﬁed were applied in the context of OSS. Conclusion: There seems to be potential in using hackathons to

overcome newcomers’ challenges in OSS projects and allow them to integrate faster into the project.

1 INTRODUCTION

Open source software (OSS) projects have proven

their success through projects like Debian, Linux

kernel, and many more. Researchers have stud-

ied their success factors (Mateos-Garcia and Stein-

mueller, 2008; Sadowski et al., 2008; Norris, 2004)

and presented lessons learned (M

uller, 2018) when

creating and maintaining successful OSS projects.

The management of OSS projects faces chal-

lenges. Maintaining engagement, attracting new de-

velopers (Sen et al., 2012), and integrating them such

that they become contributors to a project are exam-

ples of such challenges. The range of challenges that

exist in OSS projects has been discussed in litera-

ture (Lee et al., 2017; Hannebauer et al., 2014; Bal-

ali et al., 2018; Hannebauer and Gruhn, 2017). Re-

searchers have proposed and evaluated various ap-

proaches (Steinmacher et al., 2016) that aim at help-

ing the OSS communities to onboard newcomers and

reduce the time it takes until they become productive.

Hackathons have previously been used to over-

come challenges related to networking, sharing ideas,

https://orcid.org/0000-0002-9898-2158

https://orcid.org/0000-0003-1255-824X

https://orcid.org/0000-0003-2400-501X

learning, and creating prototypes (Nolte et al., 2020b).

Hackathons might thus have the potential to address

problems related to onboarding newcomers to OSS

projects and fostering their contribution. Hackathons

are time-bounded events where individuals form ad-

hoc teams and engage in intensive collaboration on

a project idea of their interest (Falk et al., 2022).

These events have been organized and studied in var-

ious contexts, including corporations (Pe-Than et al.,

2019; Nolte et al., 2018; Komssi et al., 2015), en-

trepreneurship (Cobham et al., 2017; Nolte, 2019),

and education (Porras et al., 2019; Gama et al.,

2018; Kienzler and Fontanesi, 2017). Despite their

widespread use, research focusing on hackathons in

the context of OSS projects in general and on how

they can aid the onboarding of newcomers, in partic-

ular, is scarce.

To address this gap, we conducted a review of ex-

isting literature on challenges that affect OSS new-

comers. We contrasted our ﬁndings with prior work

on how hackathons were used to tackle such chal-

lenges. In this position paper, we report on ﬁndings

from this initial analysis. These ﬁndings will sub-

sequently serve as a basis for an empirical study on

how hackathons can support newcomers to join and

become productive members of OSS projects.

We provide the following contribution: We

420

Mahmoud, A., Nolte, A. and Pfahl, D.

An Exploratory Study on the Evidence of Hackathons’ Role in Solving OSS Newcomers’ Challenges.

DOI: 10.5220/0012091200003538

In Proceedings of the 18th International Conference on Software Technologies (ICSOFT 2023), pages 420-427

ISBN: 978-989-758-665-1; ISSN: 2184-2833

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

present an overview of challenges affecting newcom-

ers to start contributing to OSS projects reported in

the literature. We also collect evidence about how

hackathons were used to tackle these challenges.

2 BACKGROUND

2.1 Challenges for OSS Projects

OSS projects are an important piece of the larger soft-

ware ecosystem contributing software, libraries, and

packages. Typical challenges of OSS projects that

need continuous attention and monitoring include the

risk of underproduction (Champion and Hill, 2021),

the difﬁculty to attract and maintain developers (Sen

et al., 2012), knowledge management (KM) (Dai

et al., 2020), and the handling of community dynam-

ics in agile OSS projects (M

uller, 2018).

In this paper, we focus on challenges related

to onboarding newcomers to OSS projects and how

hackathons were used to overcome them.

2.2 Developers Joining OSS Projects

Prior research has been conducted to understand the

onboarding of new developers joining OSS projects.

This work includes understanding their motivation

join OSS projects (Ye and Kishida, 2003). Based on

this understanding, researchers have proposed scripts

for new developers to start contributing (von Krogh

et al., 2003), proposed a joining model (Steinmacher

et al., 2014), and proposed an approach to identify and

recommend mentors in OSS projects (Canfora et al.,

2012). In addition, there is also work that aims to un-

derstand the impact of globalization and offshoring in

OSS projects and provide tools to facilitate newcom-

ers’ learning (Zhou and Mockus, 2010).

2.3 Hackathons and OSS

It is a common practice for OSS projects to arrange

in-person events like conferences or hackathons. Re-

search on such events has found that they can aid the

development of trust and build relationships (Geiger

et al., 2021). There are empirical studies on the

Google Summer of Code (GSoC) – a community code

engagement event similar to hackathons. Findings

indicate that such events can be used to attract new

developers (Silva et al., 2017). Similarly, there are

studies on how project characteristics can affect the

onboarding of developers by analyzing data from a

kick-start hackathon at Facebook. Findings indicate

positive effects of mentoring on the onboarding pro-

cess (Fagerholm et al., 2014).

3 METHODOLOGY

Our work is divided into two main steps. First, we

conducted a literature review on the challenges and

barriers affecting newcomers in OSS projects and cat-

egorized them into different groups (section 3.1). Sec-

ond, we extracted evidence from existing work about

how hackathons were used to overcome these chal-

lenges and barriers.

3.1 Identifying Challenges that Affect

Newcomers in OSS Projects

We ﬁrst searched for secondary studies on newcomer

barriers in OSS projects, i.e., Systematic Literature

Reviews (SLRs) in the ACM digital library, Google

scholar, and IEEE Xplore. As keywords we used OSS,

newcomers, barriers, challenges, SLR, and systematic

literature review. One of the identiﬁed papers (Stein-

macher et al., 2015b) speciﬁcally discussed barriers

faced by newcomers in OSS projects and grouped

them into different categories and subcategories.

Since SLRs do not cover the most recent studies,

we expanded our search on the same digital libraries

for the time period after the publication of the afore-

mentioned paper (Steinmacher et al., 2015b) exclud-

ing the keywords SLR and systematic literature re-

view. The goal was to identify additional challenges

and barriers that were discussed after the SLR was

published. This new search yielded 7 additional pa-

pers discussing newcomers’ challenges and barriers

in OSS projects. We only included papers that dis-

cussed developers joining OSS projects and excluded

papers discussing newcomers in other settings.

From the ﬁnal list of papers, we extracted all new-

comers’ challenges and barriers in OSS projects and

classiﬁed them based on the categories proposed in

the aforementioned SLR (Steinmacher et al., 2015b).

Because we found new types of barriers and chal-

lenges, we had to extend the original set of categories.

3.2 Finding Evidence on Hackathons

Addressing Newcomer Challenges

Before searching for prior work on how hackathons

were used to overcome newcomer challenges, we de-

ﬁned categories that describe the state of evidence

based on two criteria. The ﬁrst criterion covers evi-

dence that hackathons helped address newcomer chal-

An Exploratory Study on the Evidence of Hackathons’ Role in Solving OSS Newcomers’ Challenges

421

lenges. The second criterion covers the context of the

hackathon events, i.e., if they were related to OSS or

not. Based on these criteria, we deﬁned four cate-

gories. Positive evidence in open source indicates that

there is positive evidence in literature that hackathons

have helped to overcome a speciﬁc OSS challenge.

Positive evidence in different context indicates there is

positive evidence in literature that hackathons helped

overcome a speciﬁc challenge outside of OSS. Incon-

clusive evidence indicates that there is contradicting

evidence in literature, i.e., that there is evidence in

favor and against hackathons helping to overcome a

speciﬁc challenge. No evidence indicates a lack of ev-

idence that hackathons help overcome a speciﬁc chal-

lenge. Table 1 provides an overview.

Table 1: Categories of evidence (including abbreviations).

Evidence category Abbreviation

Positive Evidence in OSS Projects P-E-OSS

Positive Evidence in Different Context P-E-OTH

Inconclusive Evidence INC-E

No Evidence NO-E

After deﬁning these categories, we conducted a

search in digital libraries, including the ACM digital

library, Google scholar, and IEEE Xplore, for papers

that discuss newcomers and hackathon outcomes in

general. There are no secondary studies on the topic

so we used the following keywords in our search:

hackathon, newcomers, outcome, challenges, engage-

ment, and collaboration. In addition, we included

newcomer challenges collected in the ﬁrst step as key-

words in our search.

Based on our search, we identiﬁed 8 papers.

We analyzed these papers to ﬁnd evidence that

hackathons overcome or solve any of the previously

identiﬁed challenges and barriers in the context of

OSS and beyond. We focused on the method and ap-

proach described in the paper before extracting how

newcomer challenges were addressed.

4 RESULTS

Our main goal was to ﬁnd evidence that hackathons

helped overcome newcomer challenges in OSS

projects. To achieve this goal, we collected and an-

alyzed existing literature as discussed in 3. In this

section, we present our ﬁndings regarding newcomer

challenges (section 4.1) and regarding evidence about

how hackathons helped overcome them (section 4.2).

4.1 Challenges Affecting Newcomers in

OSS Projects

Several publications report on barriers and challenges

of newcomers joining an OSS project that can be cat-

egorized into ﬁve main groups (Steinmacher et al.,

2015b). These include Finding a way to start, Tech-

nical hurdles, Poorly documented code, Newcomers’

previous knowledge, Social interaction, and Individ-

uals problems. We extended these categories by in-

cluding more recent work. Table 2 shows the ex-

tended set of categories of challenges faced by new-

comers in OSS projects. It also includes the results

of the evidence where hackathons have been used to

overcome newcomer challenges in OSS and beyond.

4.1.1 Finding a Way to Start

This category is related to newcomers’ challenges to

ﬁnd the ﬁrst steps to become engaged in a project and

ﬁnd their way to interact with the existing team of

contributors (Steinmacher et al., 2015b). One prob-

lem reported by Steinmacher et al. (2015b); Balali

et al. (2018) is to “ﬁnd an appropriate task to start

with”. Steinmacher et al. (2015b,a) also report chal-

lenges to “ﬁnd a mentor” who guides a newcomer

through onboarding and integration them into the

team. “Difﬁculties locating bugs they choose to ﬁx”

is a related challenge reported by Lee et al. (2017);

Hannebauer and Gruhn (2017). Also related to bugs

are challenges related to “Bug reproduction” which

is reported by Hannebauer and Gruhn (2017). New-

comers struggle to reproduce bugs and test if their ﬁx

resolved them. One last challenge reported by Bal-

ali et al. (2018) is that newcomers sometimes “start

with a too complex task” for their skillset. They might

struggle to understand the task and are not able to ac-

complish what they desire with their current experi-

ence in the OSS project.

4.1.2 Technical Hurdles

This category includes technical challenges related to

handling code. One of the main issues for newcom-

ers reported by Steinmacher et al. (2015b) is “set-

ting up the local workspace”. This is always a chal-

lenge for new developers joining a project. They of-

ten face problems building the project (Hannebauer

and Gruhn, 2017), setting up the development envi-

ronment (Hannebauer and Gruhn, 2017), and using

different devices (Balali et al., 2018). Other barri-

ers reported by Steinmacher et al. (2015b) are “Code

complexity” and “Software architecture complexity”

which always need time and effort by newcomers to

get a grasp of the project and its speciﬁcs. One main

ICSOFT 2023 - 18th International Conference on Software Technologies

422

Table 2: Evidence in literature where hackathons were used to overcome newcomer challenges in OSS projects.

Category Sub Category Sub Sub Category Classiﬁcation

Finding a way to start (Steinmacher et al., 2015b)

Find appropriate task to start with (Steinmacher et al., 2015b; Balali et al., 2018) P-E-OTH (Nolte et al., 2020a; Hogan, 2022)

Find a mentor (Steinmacher et al., 2015b,a) P-E-OTH (Nolte et al., 2020a; Hogan, 2022)

Difﬁculties locating the bug they chose to ﬁx (Lee et al., 2017) P-E-OTH (Nolte et al., 2020a; Hogan, 2022)

Bug reproduction (Hannebauer and Gruhn, 2017) P-E-OTH (Nolte et al., 2020a; Hogan, 2022)

Willingness to start with a complex task (Balali et al., 2018) P-E-OTH (Nolte et al., 2020a; Hogan, 2022)

Technical hurdles (Steinmacher et al., 2015b)

Issue setting up local workspace (Steinmacher et al., 2015b)

Problems to build the project (Hannebauer and Gruhn, 2017) P-E-OTH (Hogan, 2022; Noguera Salinas et al., 2019; Steglich et al., 2020)

Setup the development environment (Hannebauer and Gruhn, 2017) P-E-OTH (Hogan, 2022; Noguera Salinas et al., 2019; Steglich et al., 2020)

Difference in the devices that mentors and mentees use (Balali et al., 2018) NO-E

Code Complexity (Steinmacher et al., 2015b) P-E-OTH (Hogan, 2022; Noguera Salinas et al., 2019; Steglich et al., 2020)

Software architecture complexity (Steinmacher et al., 2015b) P-E-OTH (Noguera Salinas et al., 2019; Steglich et al., 2020)

Submission process is too long and too complex (Lee et al., 2017)

Submission technique (Hannebauer and Gruhn, 2017) P-E-OTH (Hogan, 2022)

Issue tracker complexity (Hannebauer and Gruhn, 2017) NO-E

Bureaucracy (Hannebauer and Gruhn, 2017) NO-E

Long project processes (Balali et al., 2018) NO-E

Poorly documented code (Steinmacher et al., 2015b)

Too much documentation (Steinmacher et al., 2015b) NO-E

Outdated documentation (Steinmacher et al., 2015b) NO-E

Unclear code comments (Steinmacher et al., 2015b) NO-E

Lack of documentation (Steinmacher et al., 2015b) NO-E

Newcomer previous knowledge (Steinmacher et al., 2015b)

Lack of technical experience (Steinmacher et al., 2015b) P-E-OTH (Nolte et al., 2020a; Noguera Salinas et al., 2019)

Lack of domain experience (Steinmacher et al., 2015b) P-E-OTH (Nolte et al., 2020a; Noguera Salinas et al., 2019)

Lack of knowledge of project practices (Steinmacher et al., 2015b) P-E-OTH (Nolte et al., 2020a; Noguera Salinas et al., 2019)

Unfamiliar project management schemes (Lee et al., 2017) P-E-OTH (Nolte et al., 2020a; Noguera Salinas et al., 2019)

Lack of knowledge of the project’s programming language (Hannebauer et al., 2014) P-E-OTH (Nolte et al., 2020a; Noguera Salinas et al., 2019)

Social Interaction (Steinmacher et al., 2015b) Not receiving a (timely) answer (Steinmacher et al., 2015b,a)

Delayed answers (Steinmacher et al., 2015a)

Low responsiveness (Steinmacher et al., 2015a)

P-E-OTH (Nolte et al., 2020a; Lyonnet, 2022)

Receive an improper answer (Steinmacher et al., 2015b)

Impolite answers (Steinmacher et al., 2015a)

Receiving answers with too advanced/complex context (Steinmacher et al., 2015a)

P-E-OTH (Nolte et al., 2020a; Lyonnet, 2022)

Communication barriers

English level (Steinmacher et al., 2015a; Balali et al., 2018) NO-E

Newcomer not sending meaningful messages (Steinmacher et al., 2015a) P-E-OTH (Lyonnet, 2022)

Making useless comments in the mailing list (Steinmacher et al., 2015a) P-E-OTH (Lyonnet, 2022)

Time zone and place barriers (Balali et al., 2018) P-E-OTH (Lyonnet, 2022)

Lack of interpersonal skills in mentors (Balali et al., 2018) P-E-OTH (Lyonnet, 2022)

Cultural differences (Balali et al., 2018) P-E-OTH (Lyonnet, 2022)

Some newcomers need to contact a real person (Steinmacher et al., 2015a) P-E-OTH (Steglich et al., 2020; Lyonnet, 2022)

Individuals problems Newcomers’ personal issues

Lack of clear professional goals (Balali et al., 2018)

Fear of judgment (Balali et al., 2018)

Low self-efﬁcacy (Balali et al., 2018)

Performance anxiety (Balali et al., 2018)

Newcomer’s personality conﬂicts with the role (Balali et al., 2018)

Newcomer’s inability to improve upon criticism (Balali et al., 2018)

Difﬁculty in time-management (Balali et al., 2018)

Difﬁculty in managing different accounts (Balali et al., 2018)

Shyness (Steinmacher et al., 2015a)

NO-E

Mentors’ issues

Difﬁculty in time-management (Balali et al., 2018) INC-E (Silva et al., 2020)

Handling a large number of mentees (Balali et al., 2018) INC-E (Silva et al., 2020)

Difﬁculty in switching context (Balali et al., 2018) INC-E (Silva et al., 2020)

Not having a formal procedure for introducing the community (Balali et al., 2018) NO-E

Difﬁculty in managing different accounts (Balali et al., 2018) NO-E

An Exploratory Study on the Evidence of Hackathons’ Role in Solving OSS Newcomers’ Challenges

423

issue reported by Lee et al. (2017) is that the “Sub-

mission process is too long and too complex”. This

relates to issue tracker complexity (Hannebauer and

Gruhn, 2017), bureaucracy (Hannebauer and Gruhn,

2017) and submission techniques (Hannebauer and

Gruhn, 2017) (ex. patch-based or pull requests) which

can be different for each project.

4.1.3 Poorly Documented Code

This category is related to documentation, an im-

portant aspect of any software development project.

Steinmacher et al. (2015b) categorized barriers re-

lated to documentation into “Too much documen-

tation”, “Outdated documentation”, “Unclear code

comments”, and “Lack of documentation”. Lack of

documentation was also reported by Hannebauer and

Gruhn (2017) referring to newcomers having to use

scripts that are not fully documented and must be

learned by observation or trial-and-error.

4.1.4 Newcomers’ Previous Knowledge

This category is related to inadequate previous knowl-

edge that could become a challenge for newcom-

ers when joining an OSS project. “Lack of techni-

cal experience”, “Lack of domain experience”, and

“Lack of knowledge of project practices” are re-

ported by Steinmacher et al. (2015b). Other newcom-

ers’ previous knowledge challenges are “Unfamiliar

project management schemes” (Lee et al., 2017) and

“Lack of knowledge of the project’s programming

language” (Hannebauer et al., 2014).

4.1.5 Social Interaction

This category refers to one of the most common

groups of challenges faced by newcomers in OSS

projects. Subcategories include “Not receiving a

timely answer” (Steinmacher et al., 2015b,a), e.g.,

in the form of delayed answers, and “Receiving an

improper answer” (Steinmacher et al., 2015b), e.g.,

in the form of impolite answers or answers with ad-

vanced or complex content. A subcategory that we

derived covers Communication barriers. It includes

barriers related due to insufﬁcient command of the

English language (Steinmacher et al., 2015a; Bal-

ali et al., 2018), newcomers not sending meaningful

messages (Steinmacher et al., 2015a), making use-

less comments in mailing lists (Steinmacher et al.,

2015a), issues related to timezone and geographical

location (Balali et al., 2018), lack of interpersonal

skills of mentors (Balali et al., 2018), cultural differ-

ences (Balali et al., 2018), and the need of newcomers

to contact a person face-to-face (Steinmacher et al.,

2015a).

4.1.6 Individuals Problems

This category was derived by us, and it consists of var-

ious problems of individuals in OSS projects, arising

either from newcomers themselves or from mentors,

and affecting the newcomers’ contributions.

The ﬁrst subcategory contains Newcomers’ per-

sonal issues that could affect their integration in an

OSS project, including lack of clear professional

goals (Balali et al., 2018), fear of judgement (Balali

et al., 2018), low self-efﬁcacy (Balali et al., 2018),

performance anxiety (Balali et al., 2018), newcomer’s

personality conﬂicts with the role (Balali et al., 2018),

newcomer’s inability to improve upon criticism (Bal-

ali et al., 2018), difﬁculty in time management (Bal-

ali et al., 2018), difﬁculty in managing different ac-

counts (Balali et al., 2018), and shyness (Steinmacher

et al., 2015a).

The second subcategory contains Mentor issues

that could become barriers for newcomers in OSS

projects. Mentors are often needed by newcomers to

integrate into the team quickly. Mentor issues relate

to difﬁculties with time-management (Balali et al.,

2018), handling a large number of mentees (Balali

et al., 2018), switching context (Balali et al., 2018),

not having a formal procedure for introducing the

community (Balali et al., 2018), and managing dif-

ferent accounts (Balali et al., 2018).

4.2 Evidence on How Hackathons

Address Newcomer Challenges in

OSS Projects

In this subsection, we present evidence on how

hackathons have addressed newcomers’ challenges in

OSS projects and beyond (see the right-most column

of Table 2).

4.2.1 Finding a Way to Start

This category is related to newcomer challenges while

becoming engaged with the project and ﬁnding their

way to integrate with the existing team. The chal-

lenges in this category are directly related to learn-

ing and coaching when starting participation in a

project. This matches well with hackathons set-up

as time-bounded events focusing on participants as

potential project newcomers, with assigned mentors

for each team. Nolte et al. (2020a) reports about

supporting newcomers in scientiﬁc hackathons and

shares recommendations about how mentors should

focus on mentoring the team, and their learning rather

than the project completion. They also found that

teams taking ownership of their projects, receiving

ICSOFT 2023 - 18th International Conference on Software Technologies

424

proper support from mentors, and receiving learning-

oriented support reported positive outcomes. Besides,

Hogan (2022) also found positive outcomes for learn-

ing through hackathons. They used mentors to sup-

port teams, guide them, and carefully structure the

hackathon events in order to improve their authentic

learning.

4.2.2 Technical Hurdles

This category refers to technical challenges related to

the handling of program code by newcomers. Similar

to the ﬁrst category, newcomers’ challenges related to

building the project, setting up the development envi-

ronment, code complexity, submission technique, and

software architecture complexity can also be solved

by learning (Hogan, 2022) and coaching by mentors.

Researchers Steglich et al. (2020) used hackathons for

learning and to engage students to learn and adapt

software engineering practices, highlighting positive

outcomes when hackathon participants learn from

their peers, share knowledge and simply have an ex-

perience different from that provided in a classroom

setting. On the other hand, we could not ﬁnd studies

that speciﬁcally mention challenges like issue tracker

complexity, bureaucracy, long project processes, and

difference in the devices that mentors and mentees

use.

4.2.3 Poorly Documented Code

This category relates to documentation, which is

an important aspect of any software development

project. We could not ﬁnd any evidence that

hackathons were used to overcome any of the sub-

categories related to poorly documented code, includ-

ing “too much documentation”, “outdated documen-

tation”, “unclear code comments”, and “lack of doc-

umentation”. We found in one existing SLR (Med-

ina Angarita and Nolte, 2020) that documentation

could be one outcome of a hackathon, implying that

hackathons may be used to improve the documenta-

tion or generate new documentation and, thus, OSS

projects might use hackathons for that goal. How-

ever, this ﬁnding is not related to our focus on helping

newcomers integrate faster in an OSS project.

4.2.4 Newcomers’ Previous Knowledge

Newcomers’ challenges are related to the lack of

knowledge about the project as a whole or aspects

such as project management, domain, or program-

ming language. The usage of hackathons for improv-

ing the learning curve is discussed by Steglich et al.

(2020) and positive evidence has been achieved by al-

lowing participants to interact and talk with mentors

and stakeholders. Other researchers Noguera Sali-

nas et al. (2019) report on improving learning skills

through datathon events. Researchers Nolte et al.

(2020a) also provided several recommendations for

choosing experienced mentors with previous knowl-

edge about the domain, community, and project prac-

tices that can help the hackathon participants to

achieve their goal and improve their engagement.

4.2.5 Social Interaction

Hackathons are mainly social events where partici-

pants interact and work together to achieve a goal set

during the event, so it can be used to overcome most

of the challenges in the category. Challenges related

to “Not receiving a (timely) answer” and “Receive

an improper answer” are mainly discussed by Nolte

et al. (2020a) suggesting that assigning a mentor with

previous knowledge about the community may help

with guiding the team towards the idea and the solu-

tion and eliminating wrong communications between

newcomers and mentors. Researchers Lyonnet (2022)

found positive evidence for hackathons improving so-

cial interactions and communication which also cov-

ers the challenges faced by newcomers related to

Communication barriers - except the English level

barrier for which we were not able to ﬁnd any evi-

dence of hackathons being used to solve it.

4.2.6 Individuals Problems

This category is connected to the individual problems

in the project that could arise from the newcomers

or from the mentors. Few of the challenges in this

category related to Mentors’ issues are mentioned in

the literature, however, no conclusive evidence was

mentioned. Researchers Silva et al. (2020) recom-

mend assigning 2 mentors (one experienced and an-

other new mentor) can help in balancing the load on

the assigned mentor, thus can help to overcome men-

tors’ challenges like difﬁculty in time-management,

handling a large number of mentees, and difﬁculty

in switching context. No evidence was found of the

usage of hackathons to overcome other challenges in

this category related to newcomers or mentors.

5 LIMITATIONS AND THREATS

TO VALIDITY

Our ﬁndings are based on a literature search utilizing

speciﬁc search terms. This approach does not guaran-

tee that we found all relevant papers. Moreover, our

selection of digital libraries might not have been com-

prehensive.

An Exploratory Study on the Evidence of Hackathons’ Role in Solving OSS Newcomers’ Challenges

425

Since we only used refereed research publications

and did not include gray literature, we might have

missed challenges and barriers and evidence – both

positive and negative – of using hackathons to address

them. Moreover, the data extraction from the relevant

literature as well as the analysis of evidence is subject

to interpretation bias.

6 CONCLUSION AND FUTURE

WORK

Based on our ﬁndings, several of the newcomers’

challenges and barriers might be solved by con-

ducting hackathons and there is potential in using

hackathons to overcome newcomers’ challenges in

OSS projects, allowing them to integrate faster into

the project.

The work presented in this paper is the ﬁrst step

in a larger research undertaking. It should thus be

perceived as a position paper paving the ground for

future empirical work on how hackathons can support

newcomers to join and become productive members

of OSS projects.

ACKNOWLEDGEMENTS

Part of this work was funded by grant PRG1226 of

the Estonian Research Council.

REFERENCES

Balali, S., Steinmacher, I., Annamalai, U., Sarma, A., and

Gerosa, M. A. (2018). Newcomers’ barriers. . . is that

all? an analysis of mentors’ and newcomers’ barri-

ers in oss projects. Computer Supported Cooperative

Work (CSCW), 27.

Canfora, G., Di Penta, M., Oliveto, R., and Panichella, S.

(2012). Who is going to mentor newcomers in open

source projects? In Proceedings of the ACM SIG-

SOFT 20th International Symposium on the Founda-

tions of Software Engineering, FSE ’12, New York,

NY, USA. Association for Computing Machinery.

Champion, K. and Hill, B. (2021). Underproduction:

An approach for measuring risk in open source

software. In 2021 IEEE International Conference

on Software Analysis, Evolution and Reengineering

(SANER), pages 388–399, Los Alamitos, CA, USA.

IEEE Computer Society.

Cobham, D., Jacques, K., Gowan, C., Laurel, J., Ringham,

S., et al. (2017). From appfest to entrepreneurs: us-

ing a hackathon event to seed a university student-led

enterprise. In 11th annual International Technology,

Education and Development Conference.

Dai, J. X., Boujut, J.-F., Pourroy, F., and Marin, P. (2020).

Issues and challenges of knowledge management in

online open source hardware communities. Design

Science, 6:e24.

Fagerholm, F., Guinea, A. S., M

unch, J., and Borenstein, J.

(2014). The role of mentoring and project characteris-

tics for onboarding in open source software projects.

In Proceedings of the 8th ACM/IEEE International

Symposium on Empirical Software Engineering and

Measurement, ESEM ’14, New York, NY, USA. As-

sociation for Computing Machinery.

Falk, J., Nolte, A., Huppenkothen, D., Weinzierl, M., Gama,

K., Spikol, D., Tollerud, E., Hong, N. C., Kn

apper, I.,

and Hayden, L. B. (2022). The future of hackathon re-

search and practice. arXiv preprint arXiv:2211.08963.

Gama, K., Alencar, B., Calegario, F., Neves, A., and

Alessio, P. (2018). A hackathon methodology for un-

dergraduate course projects. In 2018 IEEE Frontiers

in Education Conference (FIE), pages 1–9. IEEE.

Geiger, R. S., Howard, D., and Irani, L. (2021). The labor

of maintaining and scaling free and open-source soft-

ware projects. Proc. ACM Hum.-Comput. Interact.,

5(CSCW1).

Hannebauer, C., Book, M., and Gruhn, V. (2014). An ex-

ploratory study of contribution barriers experienced

by newcomers to open source software projects.

In Proceedings of the 1st International Workshop

on CrowdSourcing in Software Engineering, CSI-SE

2014, page 11–14, New York, NY, USA. Association

for Computing Machinery.

Hannebauer, C. and Gruhn, V. (2017). On the relation-

ship between newcomer motivations and contribution

barriers in open source projects. In Proceedings of

the 13th International Symposium on Open Collabo-

ration, OpenSym ’17, New York, NY, USA. Associa-

tion for Computing Machinery.

Hogan, M. (2022). Hackathons as a tool for authentic learn-

ing. In Proceedings of the 27th ACM Conference on

on Innovation and Technology in Computer Science

Education Vol. 2, ITiCSE ’22, page 582–584, New

York, NY, USA. Association for Computing Machin-

ery.

Kienzler, H. and Fontanesi, C. (2017). Learning through in-

quiry: A global health hackathon. Teaching in Higher

Education, 22(2):129–142.

Komssi, M., Pichlis, D., Raatikainen, M., Kindstr

om, K.,

and J

arvinen, J. (2015). What are hackathons for?

IEEE Software, 32(5):60–67.

Lee, A., Carver, J. C., and Bosu, A. (2017). Understand-

ing the impressions, motivations, and barriers of one

time code contributors to ﬂoss projects: A survey.

In 2017 IEEE/ACM 39th International Conference on

Software Engineering (ICSE), pages 187–197.

Lyonnet, B. (2022). Hackathon approach: Its contributions

on collaboration and teamwork skills: A case study in

maritime sector. In 2021 4th International Conference

on Education Technology Management, ICETM’21,

page 91–98, New York, NY, USA. Association for

Computing Machinery.

Mateos-Garcia, J. and Steinmueller, W. E. (2008). The in-

stitutions of open source software: Examining the de-

bian community. Information Economics and Policy,

ICSOFT 2023 - 18th International Conference on Software Technologies

426

20(4):333–344. Empirical Issues in Open Source Soft-

ware.

Medina Angarita, M. A. and Nolte, A. (2020). What do we

know about hackathon outcomes and how to support

them? - a systematic literature review. In Collabora-

tion Technologies and Social Computing. Springer.

uller, M. (2018). Agile challenges and chances for

open source: Lessons learned from managing a ﬂoss

project. In 2018 IEEE Conference on Open Systems

(ICOS), pages 1–6.

Noguera Salinas, M. R., Figueiredo Pereira Emer, M. C.,

and Serra Seca Neto, A. G. (2019). Short datathon for

the interdisciplinary development of data analysis and

visualization skills. In 2019 IEEE/ACM 12th Interna-

tional Workshop on Cooperative and Human Aspects

of Software Engineering (CHASE), pages 95–98.

Nolte, A. (2019). Touched by the hackathon: a study

on the connection between hackathon participants

and start-up founders. In Proceedings of the 2nd

ACM SIGSOFT International Workshop on Software-

Intensive Business: Start-ups, Platforms, and Ecosys-

tems, pages 31–36.

Nolte, A., Hayden, L. B., and Herbsleb, J. D. (2020a).

How to support newcomers in scientiﬁc hackathons-

an action research study on expert mentoring. Pro-

ceedings of the ACM on Human-Computer Interac-

tion, 4(CSCW1):1–23.

Nolte, A., Pe-Than, E. P. P., Afﬁa, A.-A. O., Chaihirunkarn,

C., Filippova, A., Kalyanasundaram, A., Angarita, M.

A. M., Trainer, E. H., and Herbsleb, J. D. (2020b).

How to organize a hackathon - a planning kit. ArXiv,

abs/2008.08025.

Nolte, A., Pe-Than, E. P. P., Filippova, A., Bird, C., Scallen,

S., and Herbsleb, J. D. (2018). You hacked and now

what? -exploring outcomes of a corporate hackathon.

Proceedings of the ACM on Human-Computer Inter-

action, 2(CSCW):1–23.

Norris, J. (2004). Mission-critical development with open

source software: lessons learned. IEEE Software,

21(1):42–49.

Pe-Than, E. P. P., Nolte, A., Filippova, A., Bird, C., Scallen,

S., and Herbsleb, J. D. (2019). Designing corporate

hackathons with a purpose: The future of software de-

velopment. IEEE Software, 36(1):15–22.

Porras, J., Knutas, A., Ikonen, J., Happonen, A., Khakurel,

J., and Herala, A. (2019). Code camps and hackathons

in education-literature review and lessons learned. In

Proceedings of the 52nd Hawaii International Confer-

ence on System Sciences.

Sadowski, B. M., Sadowski-Rasters, G., and Duysters,

G. (2008). Transition of governance in a mature

open software source community: Evidence from

the debian case. Information Economics and Policy,

20(4):323–332. Empirical Issues in Open Source Soft-

ware.

Sen, R., Singh, S. S., and Borle, S. (2012). Open source

software success: Measures and analysis. Decision

Support Systems, 52(2):364–372.

Silva, J., Wiese, I., German, D. M., Treude, C., Gerosa,

M. A., and Steinmacher, I. (2020). A theory of the

engagement in open source projects via summer of

code programs. In Proceedings of the 28th ACM Joint

Meeting on European Software Engineering Confer-

ence and Symposium on the Foundations of Software

Engineering, ESEC/FSE 2020, page 421–431, New

York, NY, USA. Association for Computing Machin-

ery.

Silva, J. O., Wiese, I. S., Steinmacher, I., and Gerosa, M. A.

(2017). Students’ engagement in open source projects:

An analysis of google summer of code. In Proceed-

ings of the XXXI Brazilian Symposium on Software

Engineering, SBES ’17, page 224–233, New York,

NY, USA. Association for Computing Machinery.

Steglich, C., Salerno, L., Fernandes, T., Marczak, S.,

Dutra, A., Bacelo, A. P., and Trindade, C. (2020).

Hackathons as a pedagogical strategy to engage stu-

dents to learn and to adopt software engineering prac-

tices. In Proceedings of the XXXIV Brazilian Sym-

posium on Software Engineering, SBES ’20, page

670–679, New York, NY, USA. Association for Com-

puting Machinery.

Steinmacher, I., Conte, T., Gerosa, M. A., and Redmiles,

D. (2015a). Social barriers faced by newcomers plac-

ing their ﬁrst contribution in open source software

projects. In Proceedings of the 18th ACM Conference

on Computer Supported Cooperative Work & Social

Computing, CSCW ’15, page 1379–1392, New York,

NY, USA. Association for Computing Machinery.

Steinmacher, I., Conte, T. U., Treude, C., and Gerosa, M. A.

(2016). Overcoming open source project entry barri-

ers with a portal for newcomers. In 2016 IEEE/ACM

38th International Conference on Software Engineer-

ing (ICSE), pages 273–284.

Steinmacher, I., Gerosa, M. A., and Redmiles, D. F.

(2014). Attracting, onboarding, and retaining new-

comer developers in open source software projects.

In Workshop on Global Software Development in a

CSCW Perspective held in conjunction with the17th

ACM Conference on Computer Supported Cooper-

ative Work & Social Computing (CSCW’14), Balti-

more, Maryland.

Steinmacher, I., Graciotto Silva, M. A., Gerosa, M. A.,

and Redmiles, D. F. (2015b). A systematic literature

review on the barriers faced by newcomers to open

source software projects. Information and Software

Technology, 59:67–85.

von Krogh, G., Spaeth, S., and Lakhani, K. R. (2003).

Community, joining, and specialization in open source

software innovation: a case study. Research Policy,

32(7):1217–1241. Open Source Software Develop-

ment.

Ye, Y. and Kishida, K. (2003). Toward an understanding

of the motivation open source software developers.

In Proceedings of the 25th International Conference

on Software Engineering, ICSE ’03, page 419–429,

USA. IEEE Computer Society.

Zhou, M. and Mockus, A. (2010). Growth of newcomer

competence: Challenges of globalization. In Proceed-

ings of the FSE/SDP Workshop on Future of Software

Engineering Research, FoSER ’10, page 443–448,

New York, NY, USA. Association for Computing Ma-

chinery.

An Exploratory Study on the Evidence of Hackathons’ Role in Solving OSS Newcomers’ Challenges

427