Modern Code Reviews: Preliminary Results of an Analysis of the State of

the Art with Respect to the Role Played by Human Factors

Aygul Malikova

and Giancarlo Succi

Innopolis University, Innopolis, Russia

Keywords:

Modern Code Review, Social Interactions, Problems, Quality.

Abstract:

Modern Code Reviewing has shown to be an effective mechanism to identify bugs in the code; however,

given their intrinsic subjectivity, they can be signiﬁcantly affected by human factors such as interpersonal

relationships. This paper focuses on exploring such issues, with speciﬁc attention to social iterations and

personal factors. Future work includes experimental evaluations to verify the research hypothesis related to

improving the quality of the process under the study.

1 INTRODUCTION

Code reviews have been a common software engi-

neering practice for the last four decades (Fagan,

1999). An improved version of it, Modern Code Re-

views (MCR), has been proven particularly effective

(Bacchelli, 2013), and it is the subject of this work.

The effects of MCR have been variously ex-

plored in the research, with experiments also done in

large software companies, such as Microsoft (Bac-

chelli, 2013; Bosu et al., 2017; Rigby, 2013),

Google (Sadowski et al., 2018; Rigby, 2013),

Mozilla (Kononenko et al., 2016), and in Open

Source projects (Rigby, 2013).

In such works, a signiﬁcant number of prob-

lems are highlighting. For example, misunderstand-

ings, distance, social interactions, and customization

among developers (Bacchelli, 2013; Sadowski et al.,

2018). And these kinds of people issues is the area

of the present research. Speciﬁcally, considering how

social interactions and human factors can lead to not

objective and misleading reviews.

Previous research focused mainly on the problems

of the modern code review and their consequences,

while the possible solutions to these issues are not

well-studied. In this regard, the our study aims not

only at investigating social problems in MCR de-

scribed above, but it also aims at providing possible

ways to prevent them and improve the overall qual-

ity of Modern Code Review. To this end, our key re-

https://orcid.org/0000-0002-8757-5282

https://orcid.org/0000-0001-8847-0186

search questions are:

1. How do social interactions affect the code review-

ing process?

2. How to prevent the artefacts induced by social in-

teractions and to improve the quality of Modern

Code Review?

For this study, we are going to apply various re-

search techniques:

• Systematic literature review for studying the

previous related works and systematize the ﬁnd-

ings (Siddaway et al., 2019), including

• Forward and backward snowballing for the

search process of papers (Wohlin, 2014);

• Qualitative research approach for collecting the

data from the interviews (Bolderston, 2012);

As a methodology, we chose face-to-face semi-

structural interviews with developers of different

companies to investigate the topic and gather the

statistics. The goal behind conducting the survey was

to:

• understand how social interactions affect the code

reviewing process ,

• deﬁne a strategy to prevent the artifacts induced

by social and negative interactions and to improve

the quality of Modern Code Review .

This work is structured as follows. Section 2

presents the background and related works review.

Section 3 describes the Systematic Literature Review

and section 4 an arguable opinion about the non-

technical Modern Code Reviews issues. Description

Malikova, A. and Succi, G.

Modern Code Reviews: Preliminary Results of an Analysis of the State of the Art with Respect to the Role Played by Human Factors.

DOI: 10.5220/0010576302530260

In Proceedings of the 16th International Conference on Software Technologies (ICSOFT 2021), pages 253-260

ISBN: 978-989-758-523-4

253

of future work presented in Section 5. And ﬁnally,

the conclusion is placed in 6.

2 BACKGROUND

2.1 Modern Code Review

Modern Code review (MCR) is a multistage process

where developers evaluate source code written by oth-

ers to enhance the software quality (Fatima et al.,

2019). The term MCR appeared recently in 2013

and represents the lightweight variant of Fagan’s code

(Bacchelli, 2013). Other distinguishing features of

MCR from other types of review processes are tool-

based approach and asynchrony (Bacchelli, 2013).

Applying tool-based review assumes adapting some

instrument for bringing structure to the process of re-

viewing patches and supporting the logistics of re-

view (Sadowski et al., 2018). There many different

tools which are used by OSS and industrial projects,

for example, CodeFlow (used by Microsoft), Ger-

rit (Google’s Chromium and OSS projects), Review-

Board (VMware), Phabricator (Facebook), and others

(Sadowski et al., 2018). Asynchrony allows partici-

pants to conduct code review independently of time

and space (Stein et al., 1997).

The software development process involves indi-

vidual or collaborative work on it. Its vital part, MCR,

requires at least two people: the author and the re-

viewer. In some companies, it requires more than one

reviewer, for example, VMware involves two inde-

pendent reviewers (Rigby and Germ

an, 2005) and

Microsoft required an average four people (Rigby,

2013). The ﬂow of process consists of several steps,

which are general for many different companies,

such as creating, previewing, commenting, address-

ing feedback, and approving (Sadowski et al., 2018).

2.2 Rationale for Performing MCR

One of the most compelling reasons for perform-

ing MCR is to prevent developers from inappropri-

ately “protect” the code that they develop (Bacchelli,

2013). “Protecting” means organizing the process to

prevent none apart from them can modify or even

use, in the most extreme cases, such as their code.

In addition, review also amounts giving insight about

the code to other developers, sharing the information

across the team, supporting them, and improving the

overall process and quality of code (Bacchelli, 2013).

We have then decided to perform an empirical

study on the matter (Vernazza et al., 2000; Succi et al.,

2001a; Succi et al., 2001b; Mus

ılek et al., 2002; Sil-

litti et al., 2004; Scotto et al., 2004; Pedrycz and

Succi, 2005; Ronchetti et al., 2006; Scotto et al.,

2006; Moser et al., 2008b; Moser et al., 2008a; Rossi

et al., 2010; Pedrycz et al., 2011; Pedrycz et al., 2012;

Sillitti et al., 2012; Corral et al., 2015). A detailed

empirical study conducted at Google evidenced that

from the perspective of developers, the main reasons

for doing MCR are education (teaching or learning),

maintaining the organization norms, establishing the

boundaries around source code, and ﬁnally, preven-

tion of bugs, defects, and other quality issues (Sad-

owski et al., 2018).

2.3 Issues in Modern Code Reviews

There are several issues that developers face during

modern code reviews. Understanding the code and

the reasons for changing it is considered as the main

problem and one of the hardest to solve (Bacchelli,

2013). From an interview with twelve Google de-

velopers, breakdowns concerning aspects of the pro-

cess were identiﬁed, which are social interactions,

distance, review subject, context, and customization.

The social interactions will be explained in more de-

tails in the next paragraphs. The distance can be

treated as a physically or between different teams or

different roles. Review subject comes from a lack of

understanding of the code. Context problem means

misunderstanding of reasons for changing the code.

Finally, the customization is a problem of various re-

quirements of different companies (Sadowski et al.,

2018).

Since a human is a social being and while getting

in touch with other ones the communication occurs,

thereby the positive and negative effects can appear

in the results of working together, in this case, during

the peer review. Social inﬂuences include the trust re-

lationship between the author (Zhang et al., 2020),

interaction among the MCR workforce (history of in-

teractions, its volume, the sequence, mode, and so on)

(Bosu et al., 2017; Fatima et al., 2019), relationships

between the group members (Bosu et al., 2017) and

the impression of the individual author or reviewer

(Bosu et al., 2017; Fatima et al., 2019). Other non-

technical issues inﬂuencing the code review process

are the personnel factors, which are the team fac-

tors, team interaction, and reviewer response (Fatima

et al., 2019). Finally, the individual factors including

skill, characteristics, emotions, knowledge and expe-

rience, historical factors, psychological safety, work

style, and individual biasness (Fatima et al., 2019).

This work is based on examining the social aspects.

The problems that arise from social communica-

ICSOFT 2021 - 16th International Conference on Software Technologies

254

tions between developers and affect the code review-

ing process are common in different kind of team:

distributed and co-located (Bosu et al., 2017). But

it is worth considering that with increasing the team

size, the social networks become less close (Crow-

ston and Howison, 2003). Moreover, the researchers

found that a few individuals have a large number of

interactions, while most have only a few (Crowston

and Howison, 2003). The surveys to identify the ef-

fects of social factors were conducted with OSS and

Microsoft teams. The results showed that constructs

such as trust, perception of expertise, reliability and

friendship have a large impact on code review pro-

cesses (Bosu et al., 2017).

3 SYSTEMATIC LITERATURE

REVIEW

As mentioned, we have been performed a Systematic

Literature Review (SLR) for gathering a comprehen-

sive understanding of the state of the art (Kitchen-

ham, 2004). SLR aims to address the problems of

conﬂicting ﬁndings by identifying, critically evaluat-

ing, and integrating the sources of all relevant, high-

quality individual studies (Siddaway et al., 2019). In

our SLR we have followed the steps coming from the

original work of Kitchenham, 2004 (Table 1). In the

remaining of this section we will detail such steps.

Table 1: Systematic literature review process (Brereton

et al., 2007).

Phase 1.

Plan Review

1. Specify Research Ques-

tions

2. Develop Review Protocol

3. Validate Review Protocol

Phase 2.

Conduct

Review

4. Identify Relevant Re-

5. Select Primary Studies

6. Assess Study Quality

7. Extract Required Data

8. Synthesise Data

Phase 3.

Document

Review

9. Write Review Report

10. Validate Report

3.1 Developing the Review Protocol

The development of the review protocol is a signiﬁ-

cant part of performing a systematic literature review,

and it aims to minimize possible inconsistencies in the

analysis of existing work, detailing in advance how

the systematic review is to be conducted (Brereton

et al., 2007). Table 2 shows the steps of this phase.

Table 2: Process to develop review protocol (Galster et al.,

2014).

Step 1 Deﬁne search strategy

Step 2 Deﬁne inclusion + exclusion

criteria

Step 3 Deﬁne research process

Step 4 Deﬁne quality criteria

Step 5 Design data extraction form

Step 6 Deﬁne data analysis + pre-

sentation

3.2 Search Strategy

The search for the necessary literature took place in

two ways: automatic by the research string and snow-

balling. For setting the research string, the keywords

and their alternative have to be deﬁned.

The keywords are modern code review, social in-

teractions, problems, quality.

Table 3: Keywords.

Modern

Code

Review

Social

Interac-

tions

Problems Quality

Modern

Code In-

spection,

Contem-

porary

Code

Review

Group

Interac-

tions,

Human

Factors

Challenges,

Issues

Capacity

The search string is the following:

((”Modern code review” OR ”Modern Code In-

spection” OR ”Contemporary Code Review”) AND

((”Social Interactions” OR ”Group Interactions” OR

”Human Factors”) AND (”Problems” OR ”Chal-

lenges” OR ”Issues”)) AND (”Quality” OR ”Capac-

ity”))

Table 4 presents the results of the automatic search

by the research string.

Table 4: Results of automatic search.

Database Number of

found works

IEEE XPlore Digital Library 10

ACM Digital Library 14

Google Scholar 214

The snowballing technique has been used to ex-

tend the search for the reviewing literature. Snow-

Modern Code Reviews: Preliminary Results of an Analysis of the State of the Art with Respect to the Role Played by Human Factors

255

balling implies using the reference list of a pa-

per or the citations to the studies to identify addi-

tional sources (Wohlin, 2014). The start set of pa-

pers are found by deﬁned search strings and inclu-

sion/exclusion criteria speciﬁed in the next section.

This work includes both backward and forward snow-

balling, which are two techniques of the snowballing

approach. Detailed steps of each method are de-

scribed later in this chapter.

Our inclusion (I) and exclusion (E) criteria are:

• I.1 The year of publication of works related to

MCR is not earlier than 2013 since this year the

term and concept of Modern Code Review ap-

peared

• I.2 The year of other publications that is not re-

lated to MCR are not limited

• I.3 The work is related to modern code review

topic

• I.4 The number of citations is not less than ten

since this topic has a narrow scope but at the same

time already has many articles

• I.5 The language in which the work is written in

English

• E.1 The source is outdated

• E.2 The work is not related to one of the research

questions

• E.3 The article has a few citations

• E.4 The language of the work is not English

3.3 Quality Check

The crucial part of the systematic literature review is

evaluating the found articles by the quality check. The

checklist for assessing is presented in Table 5.

3.4 Data Extraction

The data extraction forms must be designed to col-

lect all the information needed to address the review

questions and the study quality criteria (Kitchenham,

2004). Table 6 shows the data collection forms and

Table 7 contains the ﬁnal studies included in the ﬁnal

review.

4 PRELIMINARY OF THE

RESULTS

After the review of the literature on the topic of non-

technical problems, it is possible to note intermediate

results on the posed research questions.

Table 5: Quality assessment checks (Ali et al., 2010).

Q1 Is there a rationale for why the

study was undertaken?

Q2 Is there an adequate description of

the context (e.g. industry,laboratory

setting, products used, etc.) in

which the research was carried out?

Q3 Has the researcher explained how

the study sample (participants or

cases) were identiﬁed and selected,

and what was the justiﬁcation for

such selection?

Q4 Does the study provide description

and justiﬁcation of the data analysis

approaches?

Q5 Are limitations of the study dis-

cussed explicitly?

Table 6: Data collection forms (Galster et al., 2014) (Fatima

et al., 2019).

Data attribute Research

Question

F1 Title

F2 Author(s)

F3 Year

F4 Paper Category (Confer-

ence / Report / Workshop

/ Journal)

F5 Keywords

F6 Social and human factors

stated

RQ1

F7 Methods for improving

MCR

RQ2

RQ1: Social Interactions and the Code Reviews.

Several studies have been conducted on how social

relationships between members of software teams can

inﬂuence the review process. It cannot be denied that

teamwork always involves social interaction and com-

munication. The software development team is no

exception. The process of checking someone else’s

code is one of the methods of communication when

one person communicates with another through the

code. As in real life, communication can be differ-

ent - calm, irritable, funny, etc. But unlike regular

communication, the code review carries several con-

sequences, such as ﬁxing the code, skipping critical

errors. These factors directly affect the quality of the

product. Based on this logic, as well as studies that

also showed the relationship between human factors

and the quality of the code review, we can conclude

that interpersonal relationships affect the code review.

ICSOFT 2021 - 16th International Conference on Software Technologies

256

Table 7: List of reviewed studies.

Title Author(s) Year Category

1 Individual, Social and Personnel

Factors Inﬂuencing Modern Code

Review Process

N. Fatima, S. Nazir,

S. Chuprat

2019 Conference

2 Expectations, Outcomes, and Chal-

lenges of Modern Code Review

A. Bacchelli, C. Bird 2013 Conference

3 Modern code review: A case study

at google

C. Sadowski, E.

oderberg, L.

Church, M. Sipko,

A. Bacchell

2018 Conference

4 Convergent software peer review

practices

P. C. Rigby 2013 Conference

5 Process aspects and social dynam-

ics of contemporary code review:

Insights from open source develop-

ment and industrial practice at mi-

crosoft

A. Bosu, J. C. Carver,

C. Bird, J. Orbeck, C.

Chockley

2016 Conference

6 A case study of distributed, asyn-

chronous software inspection

M. Stein, J. Riedl,

S. J. Harner, V.

Mashayekhi

1997 Conference

7 A preliminary examination of code

review processes in open source

projects

P. C. Rigby and D.

Germ

2005 Journal Article

8 On the shoulders of giants: A new

dataset for pull-based development

research

X. Zhang, A. Ras-

togi, Y. Yu

2020 Conference

9 The social structure of open source

software development teams

K. Crowston and J.

Howison,

2003 Article

RQ2: Prevention of Negative Impact of Non Tech-

nical Issues on Modern Code Reviews. The topic

of social problems and their impacts are well re-

searched. However, there is a literature gap on pre-

venting them and on possible ways to improve the

quality of the review. It follows that this work should

be more focused on possible ways to solve social

problems in teams.

It seems impossible to avoid interpersonal prob-

lems in a process where people are present. But from

the ﬁrst interviews with the developers, we learned

that there are still possible options. The ﬁrst one is

to involve several team members in the code review.

Another possible solution is to develop a list of crite-

ria by which to check the code inside the team. There

are also other options, but we will have to study their

effectiveness in more detail in our future work.

5 FUTURE WORK

Qualitative research is an procedure that involves col-

lecting and analyzing the data (e.g., images, sounds,

words, and numbers) (Rossman and Rallis, 2003).

Such a strategy employs different philosophical as-

sumptions; strategies of inquiry; and methods of

data collection, analysis, and interpretation (Creswell,

2009). Its purpose is to learn about some facet of

the social world by understanding concepts, opinions,

or experiences (Rossman and Rallis, 2003). Quali-

tative research has different speciﬁc approaches such

as grounded theory, case study, ethnography, phe-

nomenology, and narrative research. For this work,

the grounded theory is most suitable since it helps

to study the process of human interaction and gen-

erate theories to explain human behavior (Bolderston,

2012).

There are different data collection methods that

might be used withing qualitative research approach.

One of them is an interview. As was mentioned in the

previous chapter, many studies use interviews to learn

more about software processes, and in particular code

reviews (Bacchelli, 2013) (Bosu et al., 2017) (Rigby,

2013) (Sadowski et al., 2018) (Kononenko et al.,

2016). We also decided to survey with professional

developers to collect the data regarding the attitude to

Modern Code Reviews: Preliminary Results of an Analysis of the State of the Art with Respect to the Role Played by Human Factors

257

Modern Code Review of developers among varying

teams and their opinion regarding the objectivity and

human factors inﬂuencing the review.

The survey consists of several steps, including the

preparation phase, execution, and analyzing the re-

sults

Survey Design. During the preparation, the inter-

view protocol is set up. A protocol usually includes

• Instructions for the interviewer

• Date, place, interviewer, interviewee

• The questions

• Pilot tests

• A ﬁnal thank-you statement (Creswell, 2009)

Participant Selection. The target group is develop-

ers from the software teams. We suggested to in-

volve participants from the heterogeneous teams so

that their work processes may differ from each other.

It makes it possible to study the opinion of various

categories of developers.

To ensure valid results, is was decided to follow

the criteria of one of the previous studies (Bosu 2016).

The restriction is to survey developers with sufﬁcient

experience. Namely, to interview only those devel-

opers who had participated in at least 30 code review

requests (Bosu et al., 2017). In connection with the

speciﬁcs of our research, the study of interpersonal re-

lationships, it was also decided take into account the

amount of time during which the survey participant

works in the current team. We have set the minimum

working time to the six months.

Data Collection. The execution phase requires ad-

herence to the established protocol during the prepa-

ration phase. The participants will be asked individu-

ally by the established format. The set of questions is

the same for all interviewees. But questions may vary

according to the semi-structured format to understand

and learn more about the participant’s opinion.

Data Analysis. Data analysis consists of several

consecutive steps such as collecting open-ended data,

based on asking general questions, and analysing

the information provided by participants (Creswell,

2009).

The results are analyzed in the following order:

• Transcribing interview by organizing and prepar-

ing the data for analysis

• Read all the data

• Code the data by classifying the data by words.

Coding is the procedure of organizing material

into text segments before making sense of infor-

mation (Rossman and Rallis, 2003).

• Generate a description of the setting or people and

categories or themes for analysis by codes

• Represent the description and themes

• Produce qualitative research of the results

(Creswell, 2009)

The results of conducted interviews will help us

to understand the processes and problems of the code

reviewers. Also, we will take into consideration the

possible solutions that participants may suggest for

their speciﬁc team. Depending on the outcomes, it

will be possible to conclude the initially set research

questions.

6 CONCLUSION

This paper presented the position regarding the non-

technical issues in the software code reviewing pro-

cess. It contains the observations from related re-

searches, the preliminary evaluations, and the pro-

posal. The selected methodology to use is the sys-

tematic literature review and the interview. Future

work requires data collection and analysis using pro-

posed approaches. Moreover, it would be interesting

to explore further the effect of MCR in Open Source

(Paulson et al., 2004; Kov

acs et al., 2004; Petrinja

et al., 2010; Fitzgerald et al., 2011; Rossi et al.,

2012; Di Bella et al., 2013) and in Agile environ-

ments (Maurer et al., 1999; Kivi et al., 2000; Succi

et al., 2002; Coman et al., 2014; Janes and Succi,

2014), and when different programming approach are

in place, such as mobile (Corral et al., 2011; Corral

et al., 2013; Corral et al., 2014) or functional/logic

(Marino and Succi, 1989; Valerio et al., 1997; Sillitti

et al., 2002; Clark et al., 2004).

ACKNOWLEDGMENTS

We thank Innopolis University for generously funding

this research.

REFERENCES

Ali, M. S., Ali Babar, M., Chen, L., and Stol, K.-J.

(2010). A systematic review of comparative evidence

of aspect-oriented programming. Information and

Software Technology, 52:871–887.

ICSOFT 2021 - 16th International Conference on Software Technologies

258

Bacchelli, A. (2013). Expectations, outcomes, and chal-

lenges of modern code review.

Bolderston, A. (2012). Conducting a research interview.

Journal of Medical Imaging and Radiation Sciences,

43:66–76.

Bosu, A., Carver, J. C., Bird, C., Orbeck, J., and Chock-

ley, C. (2017). Process aspects and social dynam-

ics of contemporary code review: Insights from open

source development and industrial practice at mi-

crosoft. IEEE Transactions on Software Engineering,

43(1):56–75.

Brereton, P., Kitchenham, B., Budgen, D., Turner, M., and

Khalil, M. (2007). Lessons from applying the sys-

tematic literature review process within the software

engineering domain. j syst softw. Journal of Systems

and Software, 80:571–583.

Clark, J., Clarke, C., De Panﬁlis, S., Granatella, G., Predon-

zani, P., Sillitti, A., Succi, G., and Vernazza, T. (2004).

Selecting components in large cots repositories. Jour-

nal of Systems and Software, 73(2):323–331.

Coman, I. D., Robillard, P. N., Sillitti, A., and Succi,

G. (2014). Cooperation, collaboration and pair-

programming: Field studies on backup behavior.

Journal of Systems and Software, 91:124–134.

Corral, L., Georgiev, A. B., Sillitti, A., and Succi, G. (2013).

A method for characterizing energy consumption in

Android smartphones. In Green and Sustainable Soft-

ware (GREENS 2013), 2nd International Workshop

on, pages 38–45. IEEE.

Corral, L., Georgiev, A. B., Sillitti, A., and Succi, G. (2014).

Can execution time describe accurately the energy

consumption of mobile apps? An experiment in An-

droid. In Proceedings of the 3rd International Work-

shop on Green and Sustainable Software, pages 31–

37. ACM.

Corral, L., Sillitti, A., and Succi, G. (2015). Software As-

surance Practices for Mobile Applications. Comput-

ing, 97(10):1001–1022.

Corral, L., Sillitti, A., Succi, G., Garibbo, A., and Ramella,

P. (2011). Evolution of Mobile Software Development

from Platform-Speciﬁc to Web-Based Multiplatform

Paradigm. In Proceedings of the 10th SIGPLAN Sym-

posium on New Ideas, New Paradigms, and Reﬂec-

tions on Programming and Software, Onward! 2011,

pages 181–183, New York, NY, USA. ACM.

Creswell, J. (2009). Research Design: Qualitative, Quanti-

tative, and Mixed-Method Approaches.

Crowston, K. and Howison, J. (2003). The social structure

of open source software development teams.

Di Bella, E., Sillitti, A., and Succi, G. (2013). A multivari-

ate classiﬁcation of open source developers. Informa-

tion Sciences, 221:72–83.

Fagan, M. E. (1999). Design and code inspections to reduce

errors in program development. IBM Syst. J., 38:258–

287.

Fatima, N., Nazir, S., and Chuprat, S. (2019). Individual,

social and personnel factors inﬂuencing modern code

review process. In 2019 IEEE Conference on Open

Systems (ICOS), pages 40–45.

Fitzgerald, B., Kesan, J. P., Russo, B., Shaikh, M., and

Succi, G. (2011). Adopting open source software: A

practical guide. The MIT Press, Cambridge, MA.

Galster, M., Weyns, D., Tofan, D., Michalik, B., and Avge-

riou, P. (2014). Variability in software systems—a

systematic literature review. IEEE Transactions on

Software Engineering, 40(3):282–306.

Janes, A. and Succi, G. (2014). Lean Software Development

in Action. Springer, Heidelberg, Germany.

Kitchenham, B. (2004). Procedures for performing system-

atic reviews. Keele, UK, Keele Univ., 33.

Kivi, J., Haydon, D., Hayes, J., Schneider, R., and Succi,

G. (2000). Extreme programming: a university team

design experience. In 2000 Canadian Conference

on Electrical and Computer Engineering. Confer-

ence Proceedings. Navigating to a New Era (Cat.

No.00TH8492), volume 2, pages 816–820 vol.2.

Kononenko, O., Baysal, O., and Godfrey, M. (2016). Code

review quality: how developers see it. pages 1028–

1038.

Kov

acs, G. L., Drozdik, S., Zuliani, P., and Succi, G.

(2004). Open Source Software for the Public Ad-

ministration. In Proceedings of the 6th Interna-

tional Workshop on Computer Science and Informa-

tion Technologies.

Marino, G. and Succi, G. (1989). Data Structures for Par-

allel Execution of Functional Languages. In Pro-

ceedings of the Parallel Architectures and Languages

Europe, Volume II: Parallel Languages, PARLE ’89,

pages 346–356. Springer-Verlag.

Maurer, F., Succi, G., Holz, H., K

otting, B., Goldmann, S.,

and Dellen, B. (1999). Software Process Support over

the Internet. In Proceedings of the 21st International

Conference on Software Engineering, ICSE ’99, pages

642–645. ACM.

Moser, R., Pedrycz, W., and Succi, G. (2008a). A Compara-

tive Analysis of the Efﬁciency of Change Metrics and

Static Code Attributes for Defect Prediction. In Pro-

ceedings of the 30th International Conference on Soft-

ware Engineering, ICSE 2008, pages 181–190. ACM.

Moser, R., Pedrycz, W., and Succi, G. (2008b). Analysis of

the reliability of a subset of change metrics for defect

prediction. In Proceedings of the Second ACM-IEEE

International Symposium on Empirical Software En-

gineering and Measurement, ESEM ’08, pages 309–

311. ACM.

Mus

ılek, P., Pedrycz, W., Sun, N., and Succi, G. (2002).

On the Sensitivity of COCOMO II Software Cost Es-

timation Model. In Proceedings of the 8th Interna-

tional Symposium on Software Metrics, METRICS

’02, pages 13–20. IEEE Computer Society.

Paulson, J. W., Succi, G., and Eberlein, A. (2004). An em-

pirical study of open-source and closed-source soft-

ware products. IEEE Transactions on Software Engi-

neering, 30(4):246–256.

Pedrycz, W., Russo, B., and Succi, G. (2011). A model

of job satisfaction for collaborative development pro-

cesses. Journal of Systems and Software, 84(5):739–

752.

Modern Code Reviews: Preliminary Results of an Analysis of the State of the Art with Respect to the Role Played by Human Factors

259

Pedrycz, W., Russo, B., and Succi, G. (2012). Knowl-

edge Transfer in System Modeling and Its Realization

Through an Optimal Allocation of Information Gran-

ularity. Appl. Soft Comput., 12(8):1985–1995.

Pedrycz, W. and Succi, G. (2005). Genetic granular classi-

ﬁers in modeling software quality. Journal of Systems

and Software, 76(3):277–285.

Petrinja, E., Sillitti, A., and Succi, G. (2010). Compar-

ing OpenBRR, QSOS, and OMM assessment models.

In Open Source Software: New Horizons - Proceed-

ings of the 6th International IFIP WG 2.13 Confer-

ence on Open Source Systems, OSS 2010, pages 224–

238, Notre Dame, IN, USA. Springer, Heidelberg.

Rigby, P. (2013). Convergent software peer review prac-

tices.

Rigby, P. C. and Germ

an, D. (2005). A preliminary ex-

amination of code review processes in open source

projects.

Ronchetti, M., Succi, G., Pedrycz, W., and Russo, B.

(2006). Early estimation of software size in object-

oriented environments a case study in a cmm level 3

software ﬁrm. Information Sciences, 176(5):475–489.

Rossi, B., Russo, B., and Succi, G. (2010). Modelling Fail-

ures Occurrences of Open Source Software with Reli-

ability Growth. In Open Source Software: New Hori-

zons - Proceedings of the 6th International IFIP WG

2.13 Conference on Open Source Systems, OSS 2010,

pages 268–280, Notre Dame, IN, USA. Springer, Hei-

delberg.

Rossi, B., Russo, B., and Succi, G. (2012). Adoption of

free/libre open source software in public organiza-

tions: factors of impact. Information Technology &

People, 25(2):156–187.

Rossman, G. and Rallis, S. (2003). Learning in the Field:

An Introduction to Qualitative Research.

Sadowski, C., S

oderberg, E., Church, L., Sipko, M., and

Bacchelli, A. (2018). Modern code review: A case

study at google. In Proceedings of the 40th Inter-

national Conference on Software Engineering: Soft-

ware Engineering in Practice, ICSE-SEIP ’18, page

181–190, New York, NY, USA. Association for Com-

puting Machinery.

Scotto, M., Sillitti, A., Succi, G., and Vernazza, T. (2004). A

Relational Approach to Software Metrics. In Proceed-

ings of the 2004 ACM Symposium on Applied Comput-

ing, SAC ’04, pages 1536–1540. ACM.

Scotto, M., Sillitti, A., Succi, G., and Vernazza, T. (2006). A

non-invasive approach to product metrics collection.

Journal of Systems Architecture, 52(11):668–675.

Siddaway, A., Wood, A., and Hedges, L. (2019). How

to do a systematic review: A best practice guide

for conducting and reporting narrative reviews, meta-

analyses, and meta-syntheses. Annual Review of Psy-

chology, 70.

Sillitti, A., Janes, A., Succi, G., and Vernazza, T. (2004).

Measures for mobile users: an architecture. Journal

of Systems Architecture, 50(7):393–405.

Sillitti, A., Succi, G., and Vlasenko, J. (2012). Understand-

ing the Impact of Pair Programming on Developers

Attention: A Case Study on a Large Industrial Exper-

imentation. In Proceedings of the 34th International

Conference on Software Engineering, ICSE ’12, pages

1094–1101, Piscataway, NJ, USA. IEEE Press.

Sillitti, A., Vernazza, T., and Succi, G. (2002). Service

Oriented Programming: A New Paradigm of Software

Reuse. In Proceedings of the 7th International Con-

ference on Software Reuse, pages 269–280. Springer

Berlin Heidelberg.

Stein, M., Riedl, J., Harner, S. J., and Mashayekhi, V.

(1997). A case study of distributed, asynchronous

software inspection. Proceedings of the (19th) Inter-

national Conference on Software Engineering, pages

107–117.

Succi, G., Benedicenti, L., and Vernazza, T. (2001a). Anal-

ysis of the effects of software reuse on customer sat-

isfaction in an RPG environment. IEEE Transactions

on Software Engineering, 27(5):473–479.

Succi, G., Paulson, J., and Eberlein, A. (2001b). Prelim-

inary results from an empirical study on the growth

of open source and commercial software products. In

EDSER-3 Workshop, pages 14–15.

Succi, G., Pedrycz, W., Marchesi, M., and Williams, L.

(2002). Preliminary analysis of the effects of pair pro-

gramming on job satisfaction. In Proceedings of the

3rd International Conference on Extreme Program-

ming (XP), pages 212–215.

Valerio, A., Succi, G., and Fenaroli, M. (1997). Domain

analysis and framework-based software development.

SIGAPP Appl. Comput. Rev., 5(2):4–15.

Vernazza, T., Granatella, G., Succi, G., Benedicenti, L.,

and Mintchev, M. (2000). Deﬁning Metrics for Soft-

ware Components. In Proceedings of the World Mul-

ticonference on Systemics, Cybernetics and Informat-

ics, volume XI, pages 16–23.

Wohlin, C. (2014). Guidelines for snowballing in system-

atic literature studies and a replication in software en-

gineering. ACM International Conference Proceeding

Series.

Zhang, X., Rastogi, A., and Yu, Y. (2020). On the shoul-

ders of giants: A new dataset for pull-based develop-

ment research. In Proceedings of the 17th Interna-

tional Conference on Mining Software Repositories,

MSR ’20, page 543–547, New York, NY, USA. Asso-

ciation for Computing Machinery.

ICSOFT 2021 - 16th International Conference on Software Technologies

260