Evaluating Open Source Project Management Tools using OSSPal

Methodology

Antonio Oliveira

1 a

and Jorge Bernardino

1,2 b

Polytechnic of Coimbra – ISEC, Rua Pedro Nunes, Quinta da Nora, 3030-199 Coimbra, Portugal

CISUC - Centre of Informatics and Systems of University of Coimbra, Pinhal de Marrocos, 3030-290 Coimbra, Portugal

Keywords: Project Management Tools, Open Source Tools, OSSPal Methodology, Open Project, ProjectLibre,

OrangeScrum.

Abstract: One of the major differences between a successful project and a failed one is the project management abilities.

Project management leads to better alignment of projects within the business’ strategy, so that companies can

reduce their costs, accelerate product development, and focus on meeting their customers’ needs. To help with

that, project management tools are highly recommended, once they ease planning, scheduling, resource

allocation, communication and documentation tasks. In this paper, we assess three popular open source project

management tools: OpenProject, Orangescrum and ProjectLibre with the help of the OSSPal methodology.

This study can help project managers and programmers on choosing an adequate, current, high quality and

affordable tool to perform their projects.

1 INTRODUCTION

According PMBOK Guide (2017), projects are a

source of value and benefits for organizations.

Technologies changing in a fast-pace, reduced

budgets, shorter deadlines, and limited amount of

resources are challenges normally faced by business

leaders currently. That reveals a dynamic corporate

environment with increasing rate of change. Thus, in

order to remain competitive, companies have adopted

project management consistently. Besides that,

software projects tend to present several issues and

problems along their lifecycles.

The Project Management Institute, a worldwide

leading non-profit professional association in the area

of project management, defines project management

as the application of knowledge, skills, tools and

techniques to project activities to meet the project

requirements (PMI, 2019). It includes 5 phases:

initiating, planning, executing, monitoring and

controlling and closing. Besides that, it draws on ten

areas: integration, scope, time, cost, quality,

procurement, human resources, communications, risk

management and stakeholder management.

https://orcid.org/0000-0002-0078-1033

https://orcid.org/0000-0001-9660-2011

Consequently, project management is no

elementary task and has become more and more

essential. It embraces phases, areas, skills, resources

and techniques which should be properly

orchestrated, so that they work well together to

minimize the risk of failure and ensure that is being

offered the highest level of efficiency and

effectiveness while carrying out a project.

To help perform that indispensable work

successfully, there is available nowadays a large set

of good open source project management tools. This

kind of software can be one of the most helpful

resources to assist project managers in being as

competent as possible, regardless of the industry.

What is justified by the fact that it aims to deliver

benefits related to effective team collaboration,

monitoring for scope control, document sharing,

standard management approach implementation,

efficient project pipeline management, optimized

resource allocation and decision making, enhanced

customer satisfaction, centralized project reporting,

and improvement of task handling and visibility.

Considering that, it is appropriate to choose the

most suitable tool to our needs the best way possible.

Oliveira, A. and Bernardino, J.

Evaluating Open Source Project Management Tools using OSSPal Methodology.

DOI: 10.5220/0008351003430350

In Proceedings of the 15th International Conference on Web Information Systems and Technologies (WEBIST 2019), pages 343-350

ISBN: 978-989-758-386-5

343

For that, we will make use of the OSSPal

methodology.

The OSSpal methodology has recently emerged

as a successor of the Business Readiness Rating

(OpenBRR). It combines quantitative and qualitative

measures for evaluating software in several

categories, resulting in a score that allows the

comparison between tools (Wasserman et al., 2017).

By using that methodology, we evaluate three

popular open source project management tools:

OpenProject, Orangescrum, and ProjectLibre. These

tools will be scored taking into account the features

we considered to be essential for an open source

project management tool.

The present paper is organized as follows. Section

2 describes the three open source project management

tools. Section 3 explains the OSSpal methodology.

Section 4 presents the evaluation of the tools with the

application of OSSpal methodology. Finally, Section

5 presents the conclusions and some future work.

2 RELATED WORK

Mishra and Mishra (2013) observed something still

very current: the evolution of project management

tools for both software and non-software applications

had been increasing at a rapid pace, and the number

of available products had grown significantly.

Considering that, they compared and discussed 20

popular project management tools used by

professionals, on a feature basis. Those software tools

were selected from both open source and proprietary

software groups. Among their conclusions, it is to be

highlighted that those tools were indeed useful to

access information about the project when many

people are working together on one project, or when

multiple teams are simultaneously working on

different parts of the same project and coordination

among them is important.

Deprez and Alexandre (2008) list advantages and

disadvantages of two Free/Libre Open Source

Software assessment models: the Open Business

Readiness Rating (OpenBRR), OSSPal’s

predecessor, and the Qualification and Selection of

Open Source software (QSOS), being stated by the

authors that OpenBRR had the advantages of

allowing criteria selection to adapt the model to a

context and clearer scoring procedure with fewer

ambiguities when compared to QSOS.

In Marinheiro and Bernardino (2013), they used

OpenBRR to assess the Open Source Business

Intelligence Suite Pentaho. After applying this

evaluation model, the authors came to the conclusion

that Pentaho Community Edition could be rated as a

“good” software.

Marinheiro and Bernardino (2015) compared five

remarkable Open Source Business Intelligence suites:

Jaspersoft, Palo, Pentaho, SpagoBI and Vanilla, in

order to verify if relevant features to BI were present

on them. To do that, they adopted OpenBRR and

could then conclude that, among the assessed BI

solutions, SpargoBI was the one with the greatest

number important functionalities according to chosen

criteria set, once it obtained the highest score

provided by the assessment model.

In Ferreira, Pedrosa and Bernardino (2017), to

compare four of the top business intelligence

applications: BIRT, Jaspersoft, Pentaho, and

SpagoBI, it was used OSSPal by the authors. Thus, as

a result of the tools’ assessment, Pentaho had the best

score and BIRT the lowest one, mainly because the

latter focus less on business intelligence and more on

report construction.

3 PROJECT MANAGEMENT

TOOLS

As there is currently available a huge and growing

number of open source project management tools,

building a top 3 was no small endeavor.

Therewith, the very first criterion to build had to

be popularity stated on good-reputation websites such

as Capterra and Pat Research, which have made their

lists of 2018 best project management tools based not

only on customers’ ratings but also on some unbiased

methodology.

After that, the obtained list of top tools was still

longer than three. Then, to shorter that, it was

observed how often those programs were classified as

best ones on more websites, even if the webpage was

not so famous as the ones mentioned above.

Thereafter, the next criteria were cost and if it

works on a client-server basis. Thus, we could

evaluate a highly rated, robust, open source set of

applications with at least one of these characteristics:

free, partially-free, client-server application or

desktop application. That would be useful for project

managers who have different limitations regarding

cost, installation difficulty level or teamwork form.

As a result, the top 3 became the following project

management tools: OpenProject, Orangescrum, and

ProjectLibre.

In the following sections, we describe the main

characteristics of each solution. Besides that, major

advantages and drawbacks of each tool are outlined.

WEBIST 2019 - 15th International Conference on Web Information Systems and Technologies

344

3.1 OpenProject

Licensed under General Public License GNU Version

3, OpenProject is a web-based open source project

management solution established in 2012.

It was initially developed due to performance,

security and accessibility requirements not yet

available on software of its type at that point and it

has had the goal to become a powerful yet easy-to-use

software from its very beginning.

It is written in Ruby on Rails and requires Linux

operating system, Postgres or MySQL database and

Apache web server to work. In addition, it puts out

new releases at least once a month and is under

consistent development by the open source

community. It is recommended to any project size.

The community, free version includes task

management, time tracking, team collaboration,

project planning using Gantt charts, budgeting and

reporting. It supports not only classical project

management, but also agile. Besides that, it offers

task boards, backlogs, bug tracking, and road

mapping. The cloud and enterprise versions are paid

and add capabilities of customization, security and

support.

The main advantages of OpenProject are:

 All the features for project management itself are

free;

 Paid features are inexpensive;

 Very good for teamwork with members in

different geographic locations;

 Excellent documentation.

The main limitations of the tool are:

 Windows OS is not supported and to run the tool

on OS X is required to set up a development

environment;

 It is necessary IT staff support for installation,

troubleshoot and maintenance on Linux OS or to

be tech savvy enough for that.

Very intuitive and appealing user interface,

powerful documentation, and enhanced features give

to OpenProject remarkable popularity. It is highly

suitable for location-independent team collaboration

and ideal for project teams to work throughout the

project’s lifecycle with robust and flexible features.

Figure 1 shows the OpenProject interface.

Figure 1: OpenProject user interface.

3.2 OrangeScrum

OrangeScrum is a web-based open source project

management solution established in 2014 and

licensed under General Public License GNU Version

3. It has released several patches and new features

monthly since 2015.

It is based on the CakePHP framework and

compatible with Windows, OS X, and Linux

operating systems. It is available as a downloadable

desktop application, which installs automatically

Apache, PHP and MySQL to work.

Its self-hosted free version is called community

and includes integration with Google Drive and

Dropbox, task management using lists or a Kanban

board, resource utilization, task and resource reports

and analytics, mobile app, progress tracking. Besides

that, agile and traditional project management are

supported.

Features such as time tracking, recurring tasks,

Gantt charts, project templates, client management,

and user role management are premium and therefore

paid. Users can purchase such features as add-ons to

the free plan or they can upgrade to a paid plan or

cloud, which have them already included. It is then

recommended to small, medium and large

enterprises.

The main advantages of OrangeScrum are:

 It runs on Windows, Linux or OS X;

 Overall good documentation;

 Global forum to help community users with

troubleshoot issues;

 It allows real-time project collaboration for

teams.

The main drawbacks of the tool are:

 Self-hosted installation guide not very updated

and easy to follow;

 Charging for commonly free and essential

features such as Gannt Charts.

OrangeScrum is one of the most feature-rich tools

available currently and has hence more than enough

functions for effective project management.

Evaluating Open Source Project Management Tools using OSSPal Methodology

345

Furthermore, it tries to bring all communication about

the project being handled together and team members

can use a mobile app to stay productive on the go.

That facilitates the interaction between project

managers and the whole team, speeds up the

distribution of tasks and allows managers to identify

possible problems more quickly.

Figure 2 shows the Orangescrum interface.

Figure 2: Orangescrum user interface.

3.3 ProjectLibre

ProjectLibre is a fully free open source project

management tool established in 2012 and released

under Common Public Attribution License. New

releases happen at least a year.

It is a desktop application written in Java and

provides a similar user interface as Microsoft Project,

once it was created to be an open source replacement

of Microsoft Project desktop. It is suitable to projects

of any size.

Its main features are: compatibility with

Microsoft Project, Gantt charts, network diagrams,

Work Breakdown Structure/ Risk Breakdown

Structure charts, earned value costing and resource

histograms.

According to the tool’s official page, it will soon

offer a cloud version, so that users can manage and

create projects anytime and anywhere and do real-

time team collaboration. That will be best for teams

and multiple project management.

The main advantages of ProjectLibre are:

 It runs on Windows, Linux or OS X;

 Very easy installation process;

 It is compatible with OpenOffice, LibreOffice,

and Microsoft Project 2003, 2007, and 2010.

The main drawbacks of the tool are:

 Not fully compatible with Microsoft Project

2013 or 2016 yet;

 It lacks readily available direct online customer

service support as it is freeware;

 User interface is not inviting for those unfamiliar

with Microsoft Project or similar tools and looks

a little old;

 Due to being a desktop application, it does not

allow real-time team collaboration;

 It does not have specific features for agile

projects;

 It can take some time to load when the project

sizes are larger.

The Projectlibre tool is light, easy to install on all

3 compatible operating systems, intuitive and

absolutely a good substitute for its Microsoft

competitor. It is suitable to project managers who

want to escape from dependence on other bloated,

expensive and complicated project management

solutions. Furthermore, it will be an excellent option

specially for those learning how to use a project

management software for the first time.

Figure 3 shows the ProjectLibre interface.

Figure 3: ProjectLibre user interface.

4 OSSpal METHODOLOGY

OSSpal, originally the Business Reading Rating, is an

assessment methodology which eases the work of

identifying high quality open source software

according to an organization’s needs.

The OSSpal approach distinguishes itself from

other assessment methodologies because it uses

metrics to find qualified open source software

projects in several categories. However, it leaves the

evaluation of the quality and functionality of

individual projects to external reviewers, who are also

allowed to add informal comments to their scores. It

adopts seven categories to rate a software

(Wasserman et al., 2017):

 Functionality: How well will the software meet

the average user’s requirements?

 Operational Software Characteristics: How

secure is the software? How well does the

software perform? How well does the software

scale to a large environment? How good is the

UI? How easy to use is the software for end-

users? How easy is the software to install,

configure, deploy and maintain?

WEBIST 2019 - 15th International Conference on Web Information Systems and Technologies

346

 Support and Services: How well is the software

component supported? Is there commercial

and/or community support? Are there people and

organizations that can provide training and

consulting services?

 Documentation: Is there adequate tutorials and

reference documentations for the software?

 Software Technology Attributes: How well is

the software architected? How modular,

portable, flexible, extensible, open, and easy to

integrate is it? Is the design, the code, and the

tests of high quality? How complete and error

free are they?

 Community and Adoption: How well is the

component adopted by community, market, and

industry? How active and lively is the

community for the software?

 Development Process: What is the level of the

professionalism of the development process and

of the project organization as a whole?

The assessment process consists of four phases:

Phase 1. First of all, construct a software component

list to be analysed, to measure each component in

relation to the evaluation criteria and removing from

the analysis any software component that does not

meet the user requirements.

Phase 2. Secondly, it should be attributed weights for

the categories and for the measures:

a) Assign a percentage of importance to each

category, totalling 100%;

b) For each measure within a category, it is necessary

to rank the measure in accordance with its importance

and assign the importance;

c) For each measure within a category assign the

importance by percentage, totalling all the measures

100% of the category.

Phase 3. Then, gather data for each measure used in

each category and calculate its weighting in a range

between 1 to 5 (1 - Unacceptable, 2 - Poor, 3 -

Acceptable, 4 - Very Good, 5 - Excellent);

Phase 4. Lastly, it is calculated the OSSpal final score

based on the qualification of the category and the

weighting factors that were mentioned above.

The calculation for the category ‘Functionality’

works in a different fashion from the rest. It aims to

analyse and evaluate the capabilities that the program

features or should have, as follows:

a) Select the characteristics to analyse, scoring

them from 1 to 3 (less important to very

important);

b) Classify the characteristics in a cumulative sum

(from 1 to 3);

c) By using weighted average, standardize the

previous result on a scale of 1 to 5 with the

weights selected in a).

The functionality category will have the

following scale:

 Under 65%, Score = 1 (Unacceptable);

 65% - 80%, Score = 2 (Poor);

 80% - 90%, Score = 3 (Acceptable);

 90% - 96%, Score = 4 (Good);

 Over 96%, Score = 5 (Excellent).

5 EVALUATION

First of all, we determined a weight for each category

of this methodology in order of importance (see Table

1). That needed to be based on the most important

characteristics of a good software (Kohli, 2014), and

characteristics that people expect from source project

management tools (Giraud-Carrier and Povel, 2003).

Table 1: Assigned weights to the categories.

Category

Score

OpenProject

ProjectLibre

Orangescrum

Functionality

3,6

3,3

Operational Software

Characteristics

Software Technology Attributes

Documentation

Community and Adoption

Support and Service

Development Process

Now, after gathering data for each measure used in

each category, we calculate its score in a range

between 1 to 5 (see Table 3).

As we can see in Table 3, OpenProject stood out

from the rest in the “Functionality” category. It has

the best performance with the maximum score of 5.

In other words, it has all the considered characteristics

in the that category.

It is also worth highlighting that ProjectLibre had

a lower score in “Functionality” category as it does

not feature wiki and real-time collaboration

functionalities. Orangescrum has the lowest score

because it does not offer cost reporting, budgeting and

bug tracking features. Furthermore, it charges for

very common and basic features like Gantt charts,

what falls into the characteristic “Relevance of free

functionalities” from Table 2.

Concerning the remaining categories,

ProjectLibre and Orangescrum have almost the same

score. Except for ProjectLibre in “Support and

Service” category due to the difficulty to find

professional support or updated documentation on the

internet. Moreover, Orangescrum has the lowest

score in “Operational Software Characteristics and

Support” as it was not easy to install it by making use

of the tutorial available on its official webpage.

After this step of scoring every category for all

tools, the next and last part in this methodology is to

calculate each tool’s final score by multiplying each

score by its respective category weight. Thus, it was

obtained the Table 4.

OpenProject = 5 x 0.30 + 4 x 0.15 + 5 x 0.10 + 5 x 0.15 +

4 x 0.15 + 4 x 0.10 + 3 x 0.05 = 4.50

ProjetLibre = 3.58 x 0.30 + 5 x 0.15 + 4 x 0.10 + 5 x 0.15

+ 4 x 0.15 + 1 x 0.10 + 3 x 0.05 = 3.82

OrangeScrum = 3.34 x 0.30 + 3 x 0.15 + 5 x 0.10 + 4 x

0.15 + 4 x 0.15 + 4 x 0.10 + 3 x 0.05 = 3.70

Table 4: OSSpal final score.

Score

OpenProject

ProjectLibre

Orangescrum

Total

4.50

3.82

3.70

Overall, as we can see in Table 4, OpenProject has the

best final score of 4.5 (out of 5) through the

application of the OSSpal methodology. ProjectLibre

has next best score of 3.82, and Orangescrum the

lowest score of 3.70.

WEBIST 2019 - 15th International Conference on Web Information Systems and Technologies

348

6 CONCLUSIONS AND FUTURE

WORK

In this paper were assessed three of the most used

open source project management tools. This

evaluation work made use our experience,

information from technical documentation, usability

of the tools, the tools’ official webpages, and also

third-party websites which publish reviews and

rankings about the top current project management

programs.

After applying the OSSPal methodology, we

could classify OpenProject as “Good”, due to its final

score. This justifies its high popularly. Although it is

not totally free, it has the great advantage of keeping

for free all the necessary features to manage a project.

Next and classified as “Acceptable” is

ProjectLibre. However, from our usability

experience, we can see it is a good and fully free open

source solution that is mostly to be penalized because

it lacks support for agile projects and real-time team

collaboration. Nonetheless, that is to be solved in the

coming cloud version and that will increase its score.

In addition, it does not have specific functionalities

for agile projects.

With the lowest score and also categorized as

“Acceptable” is Orangenscrum. That mainly because

it was not easy to install it and the available free

features are not the most relevant ones to conduct a

project. Nonetheless, it was almost so good scored as

ProjectLibre, which also confirms its good

acceptance on the market.

As future work, we plan to broadening this study

by increasing the amount of open source tools to

assess, so that we can have consequently more

already-evaluated ones to choose according to the

kind of project we intend to manage.

REFERENCES

21 Best Free Project Management Software Solutions for

2019. (n.d.). Retrieved April 6, 2019, from

https://learn.g2crowd.com/free-project-management-

software

5 open source project management tools – IssueHunt –

Medium. (n.d.). Retrieved April 5, 2019, from

https://medium.com/issuehunt/5-open-source-project-

management-tools-c33a19132ed6

73 Free, Open Source and Top Project Management

Software - Compare Reviews, Features, Pricing in 2019

- PAT RESEARCH: B2B Reviews, Buying Guides

& Best Practices. (n.d.). Retrieved April 4, 2019,

from https://www.predictiveanalyticstoday.com/top-

free-premium-project-management-software/

8 Benefits of using a Project Management Software. (n.d.).

Retrieved June 8, 2019, from

https://blog.forecast.it/benefits-of-using-project-

management-software

Almeida, P. and Bernardino, J. (2016) ‘A survey on open

source data mining tools for SMEs’, Advances in

Intelligent Systems and Computing, 444, pp. 253–262.

doi: 10.1007/978-3-319-31232-3_24.

Chauhan, N. and Gautam, N. (2015) ‘Parametric

Comparison of Data Mining Tools’, v, pp. 291–298.

Courses (2015) Software Quality Characteristics. Available

at: https://courses.cs.vt.edu/csonline/SE/Lessons/

Qualities/index.html.

Deprez, J. C. and Alexandre, S. (2008) ‘Comparing

Assessment Methodologies for Free/Open Source

Software: OpenBRR and QSOS’, Lecture Notes in

Computer Science, 5089 LNCS, pp. 189–203.

Ferreira, T., Pedrosa, I. and Bernardino, J. (2017)

‘Evaluating Open Source Business Intelligence Tools

using OSSpal Methodology’, in, pp. 283–288.

Giraud-Carrier, C. and Povel, O. (2003) ‘Characterising

Data Mining Software’, Intelligent Data Analysis, 7(3),

pp. 181–192.

Kohli, T. (2014) What are the five most important

characteristics of a good software? Available at:

https://www.quora.com/What-are-the-five-most-

important-characteristics-of-a-good-software.

Marinheiro, A. and Bernardino, J. (2013) ‘OpenBRR

evaluation of an open source BI suite’, Proceedings of

Marinheiro, A. and Bernardino, J. (2015) ‘Experimental

Evaluation of Open Source Business Intelligence Suites

Mishra, A., & Mishra, D. (2013). Software Project

Management Tools: A Brief Comparative View.

https://doi.org/10.1145/2464526.2464537

OpenProject Reviews: Overview, Features & Pricing |

ShareCodex. (n.d.). Retrieved June 8, 2019, from

https://sharecodex.com/openproject-reviews/

OpenProject Software Review: Overview - Features -

Pricing. (n.d.). Retrieved June 8, 2019, from

https://project-management.com/openproject-

software-review/

Orangescrum Reviews: Overview, Features & Pricing

| ShareCodex. (n.d.). Retrieved June 8, 2019, from

https://sharecodex.com/orangescrum-reviews/

Project Management Institute. A Guide to the Project

Management Body of Knowledge (PMBOK Guide).

6th ed. Newtown Square, Pa: Project Management

Institute, 2017.

Project Management Software | Task Management Tool |

Orangescrum. (n.d.). Retrieved April 6, 2019, from

https://www.orangescrum.com/

Projectlibre open source | Projectlibre. (n.d.). Retrieved

April 5, 2019, from https://www.projectlibre.com/

product/projectlibre-open-source

ProjectLibre Reviews: Overview, Features & Pricing |

ShareCodex. (n.d.). Retrieved June 8, 2019, from

https://sharecodex.com/projectlibre-reviews/the Int. C*

Conference on Computer Science and Software

Engineering - C3S2E. doi: 10.1145/249444 .2494463.

Evaluating Open Source Project Management Tools using OSSPal Methodology

349

The Top 29 Free and Open Source Project Management

Software for Your Small Business. (n.d.). Retrieved

June 8, 2019, from https://wp-stage.capstage.net/free-

open-source-project-management-software/

The Top Five Open Source Project Management Tools for

Your Business. (n.d.). Retrieved April 5, 2019, from

https://opensourceforu.com/2018/03/the-top-five-

open-source-project-management-tools-for-your-

business/

Top 7 open source project management tools for agile

teams | Opensource.com. (n.d.). Retrieved April 5,

2019, from https://opensource.com/article/18/2/agile-

project-management-tools

Top 7 Web Open Source Project Management Tools. (n.d.).

Retrieved April 5, 2019, from https://project-

management.com/top-7-web-open-source-project-

management-tools/using OpenBRR’, IEEE Latin

America transactions, 13(3), pp. 810–817.

Wasserman, A. I. et al. (2017) ‘Open Source Systems:

Towards Robust Practices’, 496, pp. 193–203. doi:

10.1007/978-3-319-57735-7.

What is Project Management | PMI. (n.d.). Retrieved April

5, 2019, from https://www.pmi.org/about/learn-about-

pmi/what-is-project-management

WEBIST 2019 - 15th International Conference on Web Information Systems and Technologies

350