Evaluating the Teaching of Project Management Tools

through a Series of Case Studies

Rafael Queiroz Gonçalves and Christiane Gresse von Wangenheim

Department of Informatics and Statistics, Graduate Program on Computer Science,

Federal University of Santa Catarina (UFSC), Florianópolis, SC, Brazil

Keywords: Project Management, PMBOK, Instructional Design, Instructional Unit, Dotproject+.

Abstract: Project management (PM) tools are mandatory to properly manage software projects. The usage of these tools

is an important competence for professionals in the computer area, and its teaching is addressed in superior

computer courses. In this context, general usage tools are usually adopted, such as MS-Project, but the lack

of educational features of in these tools has motivated the development of several educational PM tools.

However, previous studies have shown that these tools still do not cover the whole PM process, as defined by

PMBOK. As a result, this study aims at presenting an instructional unit to assist in the teaching of

functionalities that support an extensive part of this process, covering the initiating and planning processes

groups for all knowledge areas. It adopts an open-source and educational PM tool – dotProject+, and other

instructional materials. The instructional unit was applied in several case studies in undergraduate computer

courses. Its results demonstrated students were able to learn how to use the PM tool to carry out that part of

PM process, and teachers state students learning was facilitated by the instructional materials adoption.

1 INTRODUCTION

Project Management (PM) is an important area for

many organizations in the software industry. It is so

because several projects still fail due to a lack of

proper management, leading to problems related to

unaccomplished deadlines, budget overrun, or scope

coverage (The Standish Group, 2013). In this context,

a project is defined as a temporary endeavor to

achieve a single result, and PM is the use of

knowledge, abilities, tools, and techniques that enable

a project to reach its goals (PMI, 2013).

Project problems take place mainly because of the

absence of a PM process (Keil et al., 2003), resulting

in a narrow control over project restrictions (The

Standish Group, 2013). The adoption of a PM process

may be aided by the usage of a PM tool (Fabac et al.,

2010). Despite many organizations still not using any

PM tool, the positive contributions that these tools

have brought about have increased the interest in their

usage (Cicibas et al., 2010).

Given that the usage of PM tools is not well-

established in organizations and that projects still fail,

a possible cause for this may be the lack of teaching

project managers and team members in the usage of

these tools (The Standish Group, 2013; Fabac et al.,

2010; Reid and Wilson, 2007).

The teaching of PM has to address the knowledge

on PM, beyond general knowledge on administration,

project environment, and interpersonal abilities (PMI,

2013). However, the teaching of PM should not just

be focused on theoretical knowledge, because this is

not enough to employ the PM effectively. It is crucial

to develop the project manager competencies, which

include knowledge (theoretical), abilities (practical),

and attitudes (Branch., 2009). In addition to this, the

PM is infeasible without the support of a PM tool, due

to the complexity of contemporary software projects.

Furthermore, the usage of these tools is among the

project manager competencies (PMI, 2013; Salas-

Morera et al., 2013). A PM tool is a software that

supports the whole PM process or just a specific part

of it. Among its supported functionalities are:

schedule development, cost planning, risk analysis,

etc. (Car et al., 2007).

However, there are a wide variety of PM tools,

and most of them are not suitable for teaching,

complicating the learning of their usage (PMI, 2013;

Keil et al., 2003). For instance, some PM tools demand

an initial effort to setup the environment and to learn

about its usage, thus being rejected by some students

Gonçalves, R. and Wangenheim, C.

Evaluating the Teaching of Project Management Tools through a Series of Case Studies.

In Proceedings of the 18th International Conference on Enterprise Information Systems (ICEIS 2016) - Volume 1, pages 393-404

ISBN: 978-989-758-187-8

393

during their first contact with them (Salas-Morera et

al., 2013).

As an attempt to improve this scenario, some

research (Reid and Wilson, 2007; Gregoriou et al.,

2010) has identified that MS-Project is the most

adopted PM tool for teaching. However, the lack of

didactic features in this tool has motivated these

research which proposes new educational PM tools

(Salas-Morera et al., 2013). Most of these tools are

focused on specific techniques, such as CPM, PERT,

RACI Matrix. Nonetheless, when considering the PM

process, none of these tools have focused on

addressing of all PM knowledge areas. Thus, this

paper presents an Instructional Unit (IU) that adopts

the educational PM tool – dotProject+ – for teaching

the usage of functionalities that supporting the

execution of all PM knowledge areas for the initiating

and planning processes groups.

An IU is a set of classes designed to teach certain

learning objectives for a specific target audience. It

consists of a set of instructional materials, for teachers

and students, which are developed to enable the

learning in a specific educational context (Hill et al.,

2005).

The paper structure presents, in the background

section, the main concepts related to software PM,

PM tools, and teaching of PM tools. Section 3

presents related studies, and Section 4 presents the IU

for teaching PM tools, including its instructional

materials, such as the educational PM tool

dotProject+. Section 5 presents the case study

definition, which instances are presented in Section 6.

In Section 7, we present the IU evaluation, along with

a discussion of the research results, leading to the

paper’s conclusions in Section 8.

2 BACKGROUND

Concepts that are relevant to this research are

presented in this section. All these concepts are

utilized during the presentation of the IU and in the

discussion of the case studies results.

2.1 Project Management

The PM directs the project activities and its resources

in order to meet the project requirements. It is

organized in 5 processes groups, which guide the PM

process from its initiating to its closing (Figure 1).

Orthogonally to the processes groups, the PM

processes are organized in 10 knowledge areas (Table

1), which may be addressed to effectively manage a

project.

Figure 1: PM processes groups (PMI, 2013).

Table 1: PM knowledge areas (PMI, 2013).

Knowledge

area

Processes to:

Integration Identify and coordinate PM processes

and PM activities.

Scope Ensure that the project addresses the

entire work to meet its requirements.

Time Plan and control the activities that will

be carried out during the project, so it

concludes within the deadline.

Cost Plan, estimate, and control project costs,

so it concludes within the approved

budget.

Quality Define the goals, and quality policies,

so the project meets the needs that have

initiated it.

HR Organize and manage the project team.

Communicat

ion

Ensure the generation, collection, and

distribution of project information.

Risk Identify and control the project risks.

Acquisition Buy or contract products, services or

any resources that are not available as

project internal resources.

Stakeholder Identify and manage the stakeholders

and its expectations.

In the context of this study, the PM process refers

to the one defined by PMBOK (PMI, 2013), which is

the main reference in this area and is widely accepted

worldwide (Ojeda and Reusch, 2013). The

application of a PM process is aided by the usage of

PM tools, which take advantage from technology

either to support the whole PM process, or a specific

part of it. This support may semi-automatize a few

activities of the PM process, such as the schedule

development, registering the project activities and its

sequencing, and providing online forms to record

their estimated durations and resources, then

compiling its result in a gantt chart, instead of

performing all the work manually (Cicibas et al.,

2010). On the other hand, some PM process activities

may be totally automated by PM tools, for instance,

the calculi of project total cost, the critical path

method identification, or its over allocated resources

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

394

(Fabac et al., 2010; Gregoriou et al., 2010).

2.2 PM Tools

Carrying out the PM process may be very complex,

and demand many organizational resources. To assist

its execution, many PM tools have been developed.

Examples of PM tools are: MS-Project

(microsoft.com/project), GanttProject

(ganttproject.biz), DotProject (dotproject.net),

Project.net (project.net), etc. (Fabac et al., 2010;

Mishra, 2013).

However, due to the wide variety of PM tools,

their functionalities and characteristics are very

heterogenic (Pereira et al. 2013). The supported

functionalities, for example, may cover the whole PM

process, or just one or a few PM knowledge areas, or

even more specifically just some activities, such as

the tracking of worked hours or registering the project

stakeholders.

Beyond its functionalities, other features may also

influence the choice of the PM tool to be adopted for

teaching. According to its features, some

particularities of computational environment may be

demanded, besides economic investments. Among

these features, the most relevant are: availability,

platform and usage propose.

The PM tools availability may be proprietary (the

use of a license or acquisition is mandatory and it is

maintained exclusively by a single organization) or

open-source (free usage and maintained by users

community). The proprietary PM tools may be

adopted just by organizations that are prepared to

perform its acquisition, while others may prefer to

adopt an open-source tool, as a more economically

savy alternative.

In terms of platform, there are the stand-alone

tools (mono-user and desktop access) or web-based

(multi-user and web browser access). In practice, a

web-based PM tool has to be adopted to properly

manage a software project, because it promotes

collaborative work and information sharing (Cicibas

et al., 2010). Thus, the teaching of these tools better

prepares the student for a professional career (Reid

and Wilson, 2007). However, the adoption of a PM

web-based tool requires its installation in a web

server that complies with the tool specification, and

where internet access is provided to students.

Beyond the general usage PM tools, such as MS-

Project or DotProject, that are focused on the

professional daily routine, there are educational PM

tools, which focus on student learning. These tools

include didactic features, such as instructions about

the usage of its functionalities, and simulations which

create scenarios that facilitate the usage of specific

PM techniques. Some examples of educational PM

tools are DrProject, ProMES and PpcProject

(Gregoriou et al., 2010).

2.3 Teaching of PM Tools

The usage of PM tools figures among the project

manager competencies (PMI, 2013). The need for

teaching this competency is addressed by the

ACM/IEEE reference curriculum for Computer

Science (ACM and IEEE, 2013). It specifies that

students have to develop knowledge in all PM

knowledge areas, and have to learn the usage of a PM

tool to develop a project schedule, to perform risk

analysis, to monitor the project performance, etc.

Often the teaching of PM tools includes the

application of the following techniques (PMI, 2013;

Reid and Wilson, 2007; Gregoriou et al., 2010): the

Critical Path Method (CPM) – that identifies the

project activities that cannot be delayed without

affecting the project deadline; the Program

Evaluation and Review Technique (PERT) – that

calculates the estimated effort to carry out an activity

based on three other estimates (worst case, most

common case, and best case); the RACI Matrix -

describes the participation by various roles in

completing project activities; the Resources

Levelling - technique in which start and finish dates

are adjusted based on resource constraints, with the

goal of balancing demand for resources with the

available supply; amongst others.

3 RELATED STUDIES

Related studies have been identified by previous

research (Gonçalves and Wangenheim, 2015) which

has presented some IUs that adopts educational PM

tools. Among all the studies that have been found, we

have selected just the ones which present the IU

evaluation through some case study with students in

undergraduate computer courses.

3.1 DrProject

This related study (Reid and Wilson, 2007) presents

an IU that make use of an educational PM tool,

DrProject, that is open-source and web-based. This

PM tool proposes to be simple enough to be learned

in just a few hours, but covering several

functionalities of PM tools. The employed strategy is

focused on collaborative project development,

involving students groups through PM tools

Evaluating the Teaching of Project Management Tools through a Series of Case Studies

395

functionalities. This IU was applied during 4

semesters in Software Engineering disciplines. It

begins with an expositive class, providing theoretical

instructions about time, human resources, and

communication management, and its support through

PM tool functionalities. Afterwards, the class is

organized into groups composed of 4 to 5 students,

that have to develop a software project. The groups

work on the project during the semester, and use

DrProject to develop the project schedule, organize

and share the project artifacts, and also to carry out

the whole communication among group members.

This IU was evaluated to identify whether the

students considered it was simple to learn about the

usage of a PM tool with the support of DrProject. The

data collection occurred at the end of each semester,

by students answering a questionnaire. Its analysis

demonstrated that 2/3 of students considered PM tool

usage simple to learn. Some general comments

highlighted that students complained about the lack

of a tutorial explaining DrProject functionalities.

Another part of the evaluation was based on teacher

observation, that highlighted the PM tool has

motivated the students to produce the project artifacts

with more quality, and has facilitated the

collaborative work among the students.

3.2 ProMES

This related study (Gregoriou et al., 2010) presents an

IU that uses the educational PM tool, ProMES, which

is open-source and stand-alone, for teaching CPM,

PERT and RACI matrix techniques. The instructional

strategy is based on scenarios (problems) resolution.

In each scenario the student has to solve a problem

using a specific technique, and when it is solved,

another one is presented with a higher level of

difficulty. This tool includes some educational

features, such as the configuration of student level of

experience, namely: trainee and professional. At the

trainee level the PM tool presents feedback, assisting

the student to identify each error, conducting him to

the scenario resolution. On the other hand, the

professional level does not provide any assistance.

Another instructional feature of this tool is the tutorial

video that is presented when the student first accesses

the tool, explaining how to use its functionalities. The

usage of this tool had been evaluated by teacher

observation, and also by collecting verbal feedback

from students. It was applied with 121 students during

3 semesters. It leads to conclusions that the ProMES

promoted PM learning, highlighting the benefits of its

educational features.

3.3 PpcProject

This related study (Salas-Morera et al., 2013) presents

an IU that adopts the educational PM tool,

PpcProject, which is open-source and stand-alone,

and is focused on the teaching of CPM, PERT and

resources allocation techniques. This tool has been

developed to fulfil the same requirements provided by

MS-Project when it is adopted for teaching, but to be

superior for educational proposes. This IU was

evaluated to verify whether students prefer to learn

using PpcProject or MS-Project. It has been

conducted through an experiment involving a total of

54 students. They were organized in two groups,

control and experimental groups. Each group carried

out the same activities, one using PpcProject, and

other using MS-Project. In a second stage, each group

carried out again the same activities, but using the

other tool. Thus, each student has responded to 24

questions (12 for each PM tool). Their answers have

been analysed by a non-parametric statistic test. This

analysis has demonstrated that PpcProject is more

suitable for teaching then MS-Project, except for the

resources allocation process.

Analysing the IUs presented in the related studies,

it is identified that the adopted educational PM tools

have contributed for students learning. The assistance

these tools provide have facilitated the content

understanding, beyond facilitating the PM tool usage

in class room. However, the IUs learning goals are

generally focused on time and human resources

management. Thus, considering the whole PM

process, still there is a huge gap of what is currently

been taught and all PM knowledge areas.

4 IU FOR TEACHING PM TOOLS

TO SUPPORT THE PM

PROCESS

In this context, this section presents an IU for

teaching PM tools focused on initiating and planning

processes groups, covering all knowledge areas. We

have focused on these processes groups because they

may be carried out within the IU discipline hours.

The execution of the planned projects may

demand more hours than available, especially in the

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

396

Figure 2: Execution of ADDIE process.

IU context, which projects are related to students term

paper, demanding about a year to be concluded.

Effective and motivating IUs are developed

following an Instructional Design process, for

instance, ADDIE (Branch, 2009). An overview of the

ADDIE process for the development of the proposed

IU is presented in Figure 2.

As presented in Figure 2, the ADDIE process has

5 phases. Firstly, in analysis phase it is identified the

target audience and the learning environment. This

phase also includes the IU educational goals

definition. Then, in analysis phase, it is defined the

content to be addressed and its sequencing. This

content is grouped in one or more meetings, and with

the definition of instructional materials and activities,

it composes the IU teaching plan. In the development

phase, the instructional materials are developed, then

leading to the implementation phase, which performs

the IU application in class room. To evaluate the IU

quality it is necessary to perform observations and

data collection about teachers and students perception

about the IU.

The next sections present details of instructional

materials and about the IU evaluation process.

4.1 DotProject+

DotProject is one of most popular open-source tools

for PM (Mishra, 2013). And previous studies have

identified that among open-source alternatives, it is

the most aligned with PMBOK (Pereira et al. 2013).

DotProject architecture is organized in core modules,

developed by its core team, and add-on modules,

developed by users’ community, which, may be

installed on demand. Thus, we decided to adopt this

tool, not only because its wide coverage of the PM

process, but also because its functionalities may be

extended via add-on modules. In this context,

dotProject+ was developed, being composed by

dotProject core modules, and several add-on modules

that have been developed to enhance dotProject

functionalities to cover all PM knowledge areas and

also include didactic features (Figure 3). An example

of these features is the related to the organization of

its functionalities, which are hierarchically grouped

by processes groups, and then by knowledge areas.

Thus, when the student is using a certain

functionality, it is easy to identify what part of PM

process are been supported by each functionality.

Evaluating the Teaching of Project Management Tools through a Series of Case Studies

397

Figure 3: dotProject+ architecture.

Beyond adopting dotProject+, the IU also adopts

the dotProject+ usage guide. This material is

organized as presentation slides, which may serve to

assist the teachers in expositive classes and students

as a reference material. This material presents a

process designed in BPMN notation (Weske, 2012),

addressing all knowledge areas for the initiating and

planning processes groups. Thus, this process defines

the correct sequence to use dotProject+

functionalities, satisfying the requirements to execute

each process activity before initiating it. Also, it

presents print screens for each process activity,

explaining how to use dotProject+ functionalities to

support that part of the process (Figure 4).

Figure 4: dotProject+ - Usage Guide.

All instructional materials are freely available and

may be downloaded from dotProject+ web site

(http://www.gqs.ufsc.br/evolution-of-dotproject/).

Thus, any teacher interested may download all

material and then apply the IU.

This evaluation process has been approved by CEPSH/UFSC – an

ethic committee for researches with human beings, and is

registered under the number - 47734215.9.0000.0121.

4.2 IU Evaluation Process

The IU evaluation aims to identify its quality in

relation to its content, instructional materials, user

experience, students learning, and instructional

strategy, based on students and teachers perspectives.

It is carried out by a series of case studies, based on

the empiric study process defined by Wohlin (2012)

(Figure 5)

. This evaluation process is integrated with

the GQM approach (Basili et al., 1994), which is

utilized to define the evaluation goal, the analysis

questions which evaluate this goal, and metrics which

support answering these questions.

Figure 5: IU evaluation process (Wohlin et al., 2012; Basile

et al., 1994).

Derived from the GQM metrics, data collection

instruments were developed. There is a questionnary

for stundents and other for teachers, both containing

the same structure: demographic questions followed

by a set of affirmations using a likert scale to evaluate

their perception of each IU dimension. These

dimensions were chossen based on its compatibility

with the proposed IU, and in accordance with

previous studies (Arcuri and Fraser, 2012; Chen et al.,

2013). At the end, there are open questions to collect

points regarding strengths and improvements to be

made, as well as other comments. The individual

participation in the IU evaluation is voluntary and

anonymous.

The perception about the IU quality is also

evaluated by observation, analyzing the students and

teachers behavior when interacting with the

instructional materials, and when carrying out the

instructional activities. In this case the data is

collected by verbal or written feedback, that may be

provided by e-mail or using an online form that was

available for students and teachers along all the IU

application.

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

398

5 CASE STUDY DEFINITION

Aiming to analyze the IU for teaching PM tools using

dotProject+, a case study has been defined (Figure 6).

The case study definition presents all steps necessary

for this study to be reproduced in future situations

(Wohlin et al., 2012).

Figure 6: Case study definition (Wohlin et al., 2012; Basile

et al., 1994).

The case study is organized in 3 stages. The first

one is related to its definition, where the research

goals are defined, and the IU is developed, addressing

the analysis, design and development phase of

ADDIE process. In this phase the IU evaluation is

also defined, and the data collection instruments are

developed. Details about the follow stages are

presented in the next section.

6 CASE STUDIES EXECUTION

The case study has been reproduced during six

consecutive semesters. It has been used in 3 different

Brazilian educational institutions, applied by 6

different teachers, in a total of 13 classes, teaching

more than 300 students (Table 2).

These case studies execution begins with teacher

preparing the computational environment, installing

dotProject+ and creating student accounts. The

teacher also has to get familiarized with the

instructional plan, with dotProject+ usage and with

the usage guide.

At the beginning of classes, the students are

organized in groups, receiving their credentials for

Table 2: Cases studies execution.

Semester Educational

institution

Course Discipline Teacher* Number of

students

2013-1 UFSC Computer Science Planning and management of

software projects

Teacher A 19

2013-2 UFSC Computer Science Planning and management of

software projects

Teacher A 21

2013-2 UFSC Information Systems Project management Teacher B 23

2014-1 UFSC Computer Science Planning and management of

software projects

Teacher A 30

2014-1 UFSC Information Systems Project management Teacher B 22

2014-1 SEBRAE –

Espirito Santo

Environmental

Technician

Project management Teacher C 19

2014-2 UFSC Computer Science Planning and management of

software projects

Teacher A 17

2015-1 UFSC Computer Science Planning and management of

software projects

Teacher A 24

2015-1 UFSC Information Systems Project management Teacher D 19

2015-1 SENAC –

Jaraguá do Sul

Information Technology

Management

Fundamentals in Project

Management

Teacher E 21

2015-2 SENAC –

Jaraguá do Sul

Information Technology

Management

Fundamentals in Project

Management

Teacher E 24

2015-2 UFSC Information Systems Project management Teacher D 37

2015-2 UFSC Computer Science Planning and management of

software projects

Teacher E 28

Legend:

UFSC – Federal University of Santa Catarina.

SENAC – National Service of Commercial Learning.

SEBRAE - Brazilian Service of Assistance of Small and Medium Enterprises.

* Teacher names have been replaced for privacy reasons.

Evaluating the Teaching of Project Management Tools through a Series of Case Studies

399

dotProject+ access, and the teacher carries out

expositive classes explaining how to access

dotProject+, as well as general navigation rules.

During the next meetings the teacher employs the

usage guide to explain how to use dotProject+ to

support the PM process for one or more knowledge

areas, until the students complete the whole PM

process for the initiating and planning processes

groups. At the end of the classes, the students may

export the project charter and the project plan in PDF

format, and it is delivered to the teacher for

evaluation.

After the IU application, the students and teachers

are invited to answer an evaluation questionnaire. The

answer of this questionnaire is non-mandatory,

anonymous and online. Once the data are collected, it

is analyzed and discussed, identifying the IU quality

for each dimension, and its strengths and

improvement points.

It is important to highlight that the case studies

were not exactly reproduced, because the IU

materials have been improved each semester, based

on the feedback we have received, both by students

and teachers. This feedback is normally related to

improvement suggestions or from reporting some

implementation issue. The feedback was provided

verbally or in writing, using an online form or e-mail.

In regards to data collection instruments, we have

developed and applied the complete evaluation

questionnaire (derivated from GQM), but just in the

case studies carried out after the second semester of

2014. In the previous semesters we applied a

questionnaire with open questions for individuals to

inform their improvement suggestions for the IU and

its perceived strengths, as well as other general

comments.

7 ANALYSIS

In this section we present the collected data from the

students perspective, based on the received answers

of the evaluation questionnaire. The analysis is

segmented by each evaluation dimension, presenting

the median of each questionnaire item, considering

the 26 students that have answered it.

Firstly, concerning the content dimension

(Figure 7), among the affirmations there are items

about the content relevance for computer

professionals, and whether it is addressed in proper

depth and extension.

The materials dimension aims to evaluate the

students perception about dotProject+ and its usage

guide. The dotProject+ (Figure 8) was evaluated

Figure 7: Content dimension evaluation data.

based on affirmations related to its contribution to the

understanding about the practical application of the

PM process, and also if it also assisted students during

the instructional activities.

Figure 8: dotProject+ evaluation data for PM process

coverage.

Yet, related to dotProject+, it was collected data

about the students perception of its usability (Figure

9), and how much it stimulate students and the

dificults they may had faced during its usage.

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

400

Figure 9: dotProject+ usability evaluation data.

Furthermore, we collected data to evaluate how

the usage guide has contributed to students learning

(Figure 10); identifying how it was consulted during

the classes, and whether its content and structure are

suitable for students learning.

Figure 10: Usage guide slides evaluation data.

Figure 11: User experience dimension evaluation data.

The user experience dimension (Figure 11) was

evaluated based on affirmations that attempted to

identify how students become motivated when

carrying out the instructional activities.

Figure 12: Learning dimension evaluation data.

Then, in relation to the learning dimension

(Figure 12), we have utilized affirmations to

understand the knowledge about PM tools usage in

the beginning of the IU, and how it was after the IU.

Figure 13: Instructional strategy dimension evaluation data.

The instructional strategy dimension (Figure

13) was evaluated based on affirmations about the

contribution of theoretical classes and instructional

Evaluating the Teaching of Project Management Tools through a Series of Case Studies

401

activities in students learning. In addition to its

suitability to students learning preferences.

From the teachers’ perspective, 5 of the 6 teachers

have answered the IU evaluation questionnaire. Since

we have more data about students evaluation than

about teachers evaluation, we opted to present only

the former. Nevertheless, in the discussion section we

are addressing both perspectives.

8 DISCUSSION

In this section we present a discussion about the IU

quality focusing on each dimension we have defined

to be evaluated. We are considering the students and

the teachers perspectives, collected both by the

evaluation questionnaires and by observation.

Firstly, about the learning dimension, the

students stated that the IU assists them to correlate the

theoretical content with the professional practice,

through the functionalities they learned to use on

dotProject+. Also, the majority of students have

finished all activities they were delegated, indicating

they effectively have learned how to use a PM tool to

elaborate a project charter and a project plan,

covering all PM knowledge areas. From the teachers

perspective they highlighted that the IU assisted the

student learning, mainly because the employed

instructional strategy and materials are strictly

designed for the IU learning objectives.

Regarding the instructional materials

dimension, the students stated that the usage guide

assisted in their understanding about the whole PM

process, explaining how to use dotProject+

functionalities to carry out each process activity.

However, specifically about dotProject+, the most

addressed issue was related to its usability, which in

a few cases was considered complex to use. From the

teachers perspective, they considered the entire

instructional material, ready to use, assisted in classes

preparation. They also considered the usage guide

very important, especially because of the lack of

experience of most students with PM tools. They also

considered that the usage of dotProject+ assists in the

learning of the whole discipline, because the students

have the opportunity to apply the theoretical content

through the tool functionalities. As regards to

dotProject+ functionalities, they highlighted that it is

very positive in supporting all of these processes in a

single tool. It avoids the usage of several tools to

cover all these functionalities, thus facilitating the

integration of its results in a complete project plan

generated by the tool. However, some teachers

complained about the support provided by HR

allocation process, because it demands many steps,

making it complex to be used. Other teachers also

complained about the complexity for the installation

of dotProject+, including many add-on modules, a

complication when the teacher does not master the

related technologies.

The issue about dotProject+ usability was

drastically reduced after the 2015 first semester, when

the dotProject+ version 2.0 was adopted, which had

its usability improved based on an analysis carried out

by a software usability researcher. In this same

version we also have included a new theme, to make

dotProject+ more attractive (in relation to the

standard dotProject theme), also assisting in usability

issues and facilitating student receptiveness.

Regarding the issue related to the HR allocation

process, it also has been improved in the dotProject+

version 2.0, which has simplified this process.

About the content dimension, the IU has received

a positive feedback about the content coverage and

depth, both by students and teachers. It was because

dotProject+ supports several functionalities, enabling

to apply on practice many PM techniques that are

taught in theoretical classes, covering all PM

knowledge areas. Most students also considered the

content relevant for a computer professional.

In relation to the user experience dimension, the

students highlighted that the most motivational aspect

of the IU is that it enables them to have a clear

comprehension about the practical application of the

PM content. Although their motivation had been

affected by the usability issues of dotProject+, this

negativity has been reduced after the dotProject+

version 2.0 was adopted. From the teachers perspective

they also considered that the IU prepares the students

for their professional career, and all teachers have

informed that they would recommend the IU to other

teachers that need to teach about PM tools.

In relation to the instructional strategy

dimension, the students considered that the

intercalation between the theoretical and the practical

classes has facilitated the content understanding, and

also the comprehension of its practical applicability.

Regarding the teachers perspective, they highlighted

that the instructional activities guided by the PM

process facilitated student understanding about the

correct order to use dotProject+ functionalities. Thus,

the result being, that most students concluded the

elaboration of the project charter and the project plan.

Based on the presented discussion, it is evident

that the proposed IU makes a positive contribution to

student learning, and meets for teacher demands for

all evaluated dimensions. During its application

several improvement suggestions have been

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

402

collected, and many of them have already been

implemented. Thus, at the current stage the IU

reached a maturity level that allows it to be adopted

by other teachers that need to teach about PM tools,

aligned with the PM process as defined by PMBOK.

In comparison to related studies, thought the

evaluation of the presented IU, it has demonstrated to

assist in the teaching of a more extensive part of PM

process than any other related studies. However,

some studies, such as Gregoriou et al. (2010), besides

covering just time and human resources knowledge

areas for planning processes group, it offers several

specific instructional features to assist students.

Among these features are the configuration of

difficulty level, automatic feedback, and tutorial

videos. On the other hand, only the presented IU has

adopted a material as the usage guide, which is

oriented by the PM process, and provides instructions

about how each step of this process may be supported

by PM tools functionalities.

8.1 Threats to Validity

As any research there are some threats to validity

(Wohlin et al., 2012). They are analyzed for

conclusion, construction, and external threats to

validity.

Threats to conclusion validity may occur due to

inconsistences in the data collected. In this research

the individuals may lack some knowledge related to

PM, even while being taught during the discipline. It

may lead to wrong interpretation of questionnaire

items and as a consequence lead to inconsistent

answers. To reduce this threat, the questionnaire was

designed carefully analyzing the employed

terminology, bringing it as near as possible to the

student language. Also, when the students answered

the questionnaire still in the context of the discipline,

they may have been afraid to be punished for their

answers, especially when criticizing some IU aspects.

This was mitigated by anonymising the answers,

applying the questionnaire only after all student

evaluations had been concluded and having this final

process conducted by an external researcher, instead

of the teacher. However, especially from a students

perspective, a significant part of our evaluation has

been based on data collected in an ad-hoc manner,

based on verbal and written feedback provided during

the case studies instances.

Threats to construction validity are related to the

data collection instrument, which may not contain the

necessary set of questions to reach the evaluation

goal. We have employed the GQM approach to

design the questionnaire, thus the evaluation goal was

systematically deployed in question analysis and

metrics, which were represented by questionnaire

items.

Threats to external validity may occur by not

obtaining a significant sample. In fact, we still do

have not collected a significant amount of structured

data; consequentially performing the statistical study

with only 26 students answers. However, we have

mitigated that by applying the IU in different

semesters, involving 6 teachers and 304 students,

which are significant events for a general evaluation

of the IU for teaching PM tools with the support of

dotProject+.

9 CONCLUSIONS

This study has presented the evaluation of an IU for

teaching the usage of PM tools. This IU has

introduced the educational PM tool - dotProject+,

which is an enhancement of one of the most popular

open-source tools for PM. DotProject+ includes a

more comprehensive support to the PM process and

educational features. After carrying out a series of

case studies, and analysing the collected data from

students and teachers perspectives, the IU has

demonstrated to be effective for teaching the usage of

PM tools for the initiating and planning processes

groups, covering all PM knowledge areas. Students

highlighted that they have learned the content, and

consider they are able to reproduce it in their

professional activities. Teachers have highlighted

they would like to use this IU again, and would

recommend it to other teachers. Future studies may

expand upon the instructional feedback of

dotProject+, beyond the creation of other IUs to

address other processes groups that were not included

in this research.

ACKNOWLEDGEMENTS

This work was supported by the CNPq (Conselho

Nacional de Desenvolvimento Científico e

Tecnológico – www.cnpq.br), an entity of the

Brazilian government focused on scientific and

technological development.

REFERENCES

ACM, IEEE Computer Society, 2013. Computer Science

Curricula 2013.

Evaluating the Teaching of Project Management Tools through a Series of Case Studies

403

Arcuri A., Fraser G., 2012. Sound Empirical Evidence in

Software Testing. In: Proc. of 34th International

Conference on Software Engineering, Zurich/

Switzerland.

Basili, V., Caldier, G., Rombach, D., 1994. The Goal

Question Metric Approach. Encyclopedia of software

engineering, pp. 528–532.

Branch R., 2009. Instructional Design: The ADDIE

Approach. Springer, 2

edition.

Car Ž., Belani H., Pripužić K., 2007. Teaching Project

Management in Academic ICT Environments. In: Proc.

of the Int. Conf. on computer as a tool, Warsaw.

Cicibas H., Unal O., Demir K, 2010. A comparison of

project management software tools (PMST). In: Proc.

of the 9th Software Engineering Research and Practice,

Las Vegas.

Chen H., Chen Y., Chen K., 2013. The Design and Effect

of a Scaffolded Concept Mapping Strategy on Learning

Performance in an Undergraduate Database Course.

IEEE Transactions on Education, vol. 56, n. 3, pp. 300-

307.

Fabac R., Radoševic D., Pihir I., 2010. Frequency of use

and importance of software tools in project

management practice in Croatia. In: Proc. of 32nd Int.

Conf. on Information Technology Interfaces, Cavtat.

Gregoriou G., Kirytopoulos K., Kiriklidis C., 2010. Project

Management Educational Software (ProMES).

Computer Applications in Engineering Education, vol.

21, n. 1, pp. 46–59.

Gonçalves R., Wangenheim C., 2015. How to Teach the

Usage of Project Management Tools in Computer

Courses: A Systematic Literature Review. In: Proc. of

the Int. Conf. on Software Engineering and Knowledge

Engineering, Pittsburgh.

Hill, H., Rowan, B., Ball, D., 2005. Effects of Teachers'

Mathematical Knowledge for Teaching on Student

Achievement. American Educational Research

Journal, 42(2), p. 371-406.

Keil, M., Rai, A., Mann J., 2003. Why software projects

escalate: The importance of project management

constructs. IEEE Transactions on Engineering

Management, vol. 50, n.3, pp. 251–261.

Kirkpatrick, D., kirkpatrick, J., 2012. Evaluating Training

Programs: The Four Levels. Berrett-Koehler

Publishers, 4

edition.

Mishra A., Mishra D., 2013. Software Project Management

Tools: A Brief Comparative View, ACM SIGSOFT

Software Engineering Notes, 38 (3), pp. 1-4.

Ojeda O., Reusch P., 2013. Sustainable procurement -

Extending project procurement concepts and processes

based on PMBOK. In: Proc. of 7th International

Conference on Intelligent Data Acquisition and

Advanced Computing Systems, Berlin/Germany, pp.

530 – 536.

Pereira, A., Gonçalves R., Wangenheim, C., 2013.

Comparison of open source tools for project

management. International Journal of Software

Engineering and Knowledge Engineering, vol. 23, n. 2,

2013, pp. 189-209.

PMI – Project Management Institute, 2013. A Guide to the

Project Management Body of Knowledge, 5. ed.,

Newtown Square.

Reid K., Wilson G., 2007 DrProject: A Software Project

Management Portal to Meet Educational Needs. In:

Proc. of the Special Interest Group on Computer

Science Education, Covington.

Salas-Morera L., Arauzo-Azofra A., García-Hernández, L.,

2013. PpcProject: An educational tool for software

project management. Computers & Education, vol. 69,

n. 1, pp. 181-188.

The Standish Group, 2013. Chaos Manifesto 2013, Boston.

Wohlin C., Runeson P., Höst M., 2012. Experimentation in

Software Engineering: An Introduction, Springer.

Weske, M., 2012. Business Process Management:

Concepts, Languages, Architectures. Springer, 2

edition.

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

404