Evaluating the Teaching of Project Management Tools
through a Series of Case Studies
Rafael Queiroz Gonçalves and Christiane Gresse von Wangenheim
Department of Informatics and Statistics, Graduate Program on Computer Science,
Federal University of Santa Catarina (UFSC), Florianópolis, SC, Brazil
Keywords: Project Management, PMBOK, Instructional Design, Instructional Unit, Dotproject+.
Abstract: Project management (PM) tools are mandatory to properly manage software projects. The usage of these tools
is an important competence for professionals in the computer area, and its teaching is addressed in superior
computer courses. In this context, general usage tools are usually adopted, such as MS-Project, but the lack
of educational features of in these tools has motivated the development of several educational PM tools.
However, previous studies have shown that these tools still do not cover the whole PM process, as defined by
PMBOK. As a result, this study aims at presenting an instructional unit to assist in the teaching of
functionalities that support an extensive part of this process, covering the initiating and planning processes
groups for all knowledge areas. It adopts an open-source and educational PM tool – dotProject+, and other
instructional materials. The instructional unit was applied in several case studies in undergraduate computer
courses. Its results demonstrated students were able to learn how to use the PM tool to carry out that part of
PM process, and teachers state students learning was facilitated by the instructional materials adoption.
1 INTRODUCTION
Project Management (PM) is an important area for
many organizations in the software industry. It is so
because several projects still fail due to a lack of
proper management, leading to problems related to
unaccomplished deadlines, budget overrun, or scope
coverage (The Standish Group, 2013). In this context,
a project is defined as a temporary endeavor to
achieve a single result, and PM is the use of
knowledge, abilities, tools, and techniques that enable
a project to reach its goals (PMI, 2013).
Project problems take place mainly because of the
absence of a PM process (Keil et al., 2003), resulting
in a narrow control over project restrictions (The
Standish Group, 2013). The adoption of a PM process
may be aided by the usage of a PM tool (Fabac et al.,
2010). Despite many organizations still not using any
PM tool, the positive contributions that these tools
have brought about have increased the interest in their
usage (Cicibas et al., 2010).
Given that the usage of PM tools is not well-
established in organizations and that projects still fail,
a possible cause for this may be the lack of teaching
project managers and team members in the usage of
these tools (The Standish Group, 2013; Fabac et al.,
2010; Reid and Wilson, 2007).
The teaching of PM has to address the knowledge
on PM, beyond general knowledge on administration,
project environment, and interpersonal abilities (PMI,
2013). However, the teaching of PM should not just
be focused on theoretical knowledge, because this is
not enough to employ the PM effectively. It is crucial
to develop the project manager competencies, which
include knowledge (theoretical), abilities (practical),
and attitudes (Branch., 2009). In addition to this, the
PM is infeasible without the support of a PM tool, due
to the complexity of contemporary software projects.
Furthermore, the usage of these tools is among the
project manager competencies (PMI, 2013; Salas-
Morera et al., 2013). A PM tool is a software that
supports the whole PM process or just a specific part
of it. Among its supported functionalities are:
schedule development, cost planning, risk analysis,
etc. (Car et al., 2007).
However, there are a wide variety of PM tools,
and most of them are not suitable for teaching,
complicating the learning of their usage (PMI, 2013;
Keil et al., 2003). For instance, some PM tools demand
an initial effort to setup the environment and to learn
about its usage, thus being rejected by some students
Gonçalves, R. and Wangenheim, C.
Evaluating the Teaching of Project Management Tools through a Series of Case Studies.
In Proceedings of the 18th International Conference on Enterprise Information Systems (ICEIS 2016) - Volume 1, pages 393-404
ISBN: 978-989-758-187-8
Copyright
c
2016 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
393
during their first contact with them (Salas-Morera et
al., 2013).
As an attempt to improve this scenario, some
research (Reid and Wilson, 2007; Gregoriou et al.,
2010) has identified that MS-Project is the most
adopted PM tool for teaching. However, the lack of
didactic features in this tool has motivated these
research which proposes new educational PM tools
(Salas-Morera et al., 2013). Most of these tools are
focused on specific techniques, such as CPM, PERT,
RACI Matrix. Nonetheless, when considering the PM
process, none of these tools have focused on
addressing of all PM knowledge areas. Thus, this
paper presents an Instructional Unit (IU) that adopts
the educational PM tool – dotProject+ – for teaching
the usage of functionalities that supporting the
execution of all PM knowledge areas for the initiating
and planning processes groups.
An IU is a set of classes designed to teach certain
learning objectives for a specific target audience. It
consists of a set of instructional materials, for teachers
and students, which are developed to enable the
learning in a specific educational context (Hill et al.,
2005).
The paper structure presents, in the background
section, the main concepts related to software PM,
PM tools, and teaching of PM tools. Section 3
presents related studies, and Section 4 presents the IU
for teaching PM tools, including its instructional
materials, such as the educational PM tool
dotProject+. Section 5 presents the case study
definition, which instances are presented in Section 6.
In Section 7, we present the IU evaluation, along with
a discussion of the research results, leading to the
paper’s conclusions in Section 8.
2 BACKGROUND
Concepts that are relevant to this research are
presented in this section. All these concepts are
utilized during the presentation of the IU and in the
discussion of the case studies results.
2.1 Project Management
The PM directs the project activities and its resources
in order to meet the project requirements. It is
organized in 5 processes groups, which guide the PM
process from its initiating to its closing (Figure 1).
Orthogonally to the processes groups, the PM
processes are organized in 10 knowledge areas (Table
1), which may be addressed to effectively manage a
project.
Figure 1: PM processes groups (PMI, 2013).
Table 1: PM knowledge areas (PMI, 2013).
Knowledge
area
Processes to:
Integration Identify and coordinate PM processes
and PM activities.
Scope Ensure that the project addresses the
entire work to meet its requirements.
Time Plan and control the activities that will
be carried out during the project, so it
concludes within the deadline.
Cost Plan, estimate, and control project costs,
so it concludes within the approved
budget.
Quality Define the goals, and quality policies,
so the project meets the needs that have
initiated it.
HR Organize and manage the project team.
Communicat
ion
Ensure the generation, collection, and
distribution of project information.
Risk Identify and control the project risks.
Acquisition Buy or contract products, services or
any resources that are not available as
project internal resources.
Stakeholder Identify and manage the stakeholders
and its expectations.
In the context of this study, the PM process refers
to the one defined by PMBOK (PMI, 2013), which is
the main reference in this area and is widely accepted
worldwide (Ojeda and Reusch, 2013). The
application of a PM process is aided by the usage of
PM tools, which take advantage from technology
either to support the whole PM process, or a specific
part of it. This support may semi-automatize a few
activities of the PM process, such as the schedule
development, registering the project activities and its
sequencing, and providing online forms to record
their estimated durations and resources, then
compiling its result in a gantt chart, instead of
performing all the work manually (Cicibas et al.,
2010). On the other hand, some PM process activities
may be totally automated by PM tools, for instance,
the calculi of project total cost, the critical path
method identification, or its over allocated resources
ICEIS 2016 - 18th International Conference on Enterprise Information Systems
394
(Fabac et al., 2010; Gregoriou et al., 2010).
2.2 PM Tools
Carrying out the PM process may be very complex,
and demand many organizational resources. To assist
its execution, many PM tools have been developed.
Examples of PM tools are: MS-Project
(microsoft.com/project), GanttProject
(ganttproject.biz), DotProject (dotproject.net),
Project.net (project.net), etc. (Fabac et al., 2010;
Mishra, 2013).
However, due to the wide variety of PM tools,
their functionalities and characteristics are very
heterogenic (Pereira et al. 2013). The supported
functionalities, for example, may cover the whole PM
process, or just one or a few PM knowledge areas, or
even more specifically just some activities, such as
the tracking of worked hours or registering the project
stakeholders.
Beyond its functionalities, other features may also
influence the choice of the PM tool to be adopted for
teaching. According to its features, some
particularities of computational environment may be
demanded, besides economic investments. Among
these features, the most relevant are: availability,
platform and usage propose.
The PM tools availability may be proprietary (the
use of a license or acquisition is mandatory and it is
maintained exclusively by a single organization) or
open-source (free usage and maintained by users
community). The proprietary PM tools may be
adopted just by organizations that are prepared to
perform its acquisition, while others may prefer to
adopt an open-source tool, as a more economically
savy alternative.
In terms of platform, there are the stand-alone
tools (mono-user and desktop access) or web-based
(multi-user and web browser access). In practice, a
web-based PM tool has to be adopted to properly
manage a software project, because it promotes
collaborative work and information sharing (Cicibas
et al., 2010). Thus, the teaching of these tools better
prepares the student for a professional career (Reid
and Wilson, 2007). However, the adoption of a PM
web-based tool requires its installation in a web
server that complies with the tool specification, and
where internet access is provided to students.
Beyond the general usage PM tools, such as MS-
Project or DotProject, that are focused on the
professional daily routine, there are educational PM
tools, which focus on student learning. These tools
include didactic features, such as instructions about
the usage of its functionalities, and simulations which
create scenarios that facilitate the usage of specific
PM techniques. Some examples of educational PM
tools are DrProject, ProMES and PpcProject
(Gregoriou et al., 2010).
2.3 Teaching of PM Tools
The usage of PM tools figures among the project
manager competencies (PMI, 2013). The need for
teaching this competency is addressed by the
ACM/IEEE reference curriculum for Computer
Science (ACM and IEEE, 2013). It specifies that
students have to develop knowledge in all PM
knowledge areas, and have to learn the usage of a PM
tool to develop a project schedule, to perform risk
analysis, to monitor the project performance, etc.
Often the teaching of PM tools includes the
application of the following techniques (PMI, 2013;
Reid and Wilson, 2007; Gregoriou et al., 2010): the
Critical Path Method (CPM) – that identifies the
project activities that cannot be delayed without
affecting the project deadline; the Program
Evaluation and Review Technique (PERT) – that
calculates the estimated effort to carry out an activity
based on three other estimates (worst case, most
common case, and best case); the RACI Matrix -
describes the participation by various roles in
completing project activities; the Resources
Levelling - technique in which start and finish dates
are adjusted based on resource constraints, with the
goal of balancing demand for resources with the
available supply; amongst others.
3 RELATED STUDIES
Related studies have been identified by previous
research (Gonçalves and Wangenheim, 2015) which
has presented some IUs that adopts educational PM
tools. Among all the studies that have been found, we
have selected just the ones which present the IU
evaluation through some case study with students in
undergraduate computer courses.
3.1 DrProject
This related study (Reid and Wilson, 2007) presents
an IU that make use of an educational PM tool,
DrProject, that is open-source and web-based. This
PM tool proposes to be simple enough to be learned
in just a few hours, but covering several
functionalities of PM tools. The employed strategy is
focused on collaborative project development,
involving students groups through PM tools
Evaluating the Teaching of Project Management Tools through a Series of Case Studies
395
functionalities. This IU was applied during 4
semesters in Software Engineering disciplines. It
begins with an expositive class, providing theoretical
instructions about time, human resources, and
communication management, and its support through
PM tool functionalities. Afterwards, the class is
organized into groups composed of 4 to 5 students,
that have to develop a software project. The groups
work on the project during the semester, and use
DrProject to develop the project schedule, organize
and share the project artifacts, and also to carry out
the whole communication among group members.
This IU was evaluated to identify whether the
students considered it was simple to learn about the
usage of a PM tool with the support of DrProject. The
data collection occurred at the end of each semester,
by students answering a questionnaire. Its analysis
demonstrated that 2/3 of students considered PM tool
usage simple to learn. Some general comments
highlighted that students complained about the lack
of a tutorial explaining DrProject functionalities.
Another part of the evaluation was based on teacher
observation, that highlighted the PM tool has
motivated the students to produce the project artifacts
with more quality, and has facilitated the
collaborative work among the students.
3.2 ProMES
This related study (Gregoriou et al., 2010) presents an
IU that uses the educational PM tool, ProMES, which
is open-source and stand-alone, for teaching CPM,
PERT and RACI matrix techniques. The instructional
strategy is based on scenarios (problems) resolution.
In each scenario the student has to solve a problem
using a specific technique, and when it is solved,
another one is presented with a higher level of
difficulty. This tool includes some educational
features, such as the configuration of student level of
experience, namely: trainee and professional. At the
trainee level the PM tool presents feedback, assisting
the student to identify each error, conducting him to
the scenario resolution. On the other hand, the
professional level does not provide any assistance.
Another instructional feature of this tool is the tutorial
video that is presented when the student first accesses
the tool, explaining how to use its functionalities. The
usage of this tool had been evaluated by teacher
observation, and also by collecting verbal feedback
from students. It was applied with 121 students during
3 semesters. It leads to conclusions that the ProMES
promoted PM learning, highlighting the benefits of its
educational features.
3.3 PpcProject
This related study (Salas-Morera et al., 2013) presents
an IU that adopts the educational PM tool,
PpcProject, which is open-source and stand-alone,
and is focused on the teaching of CPM, PERT and
resources allocation techniques. This tool has been
developed to fulfil the same requirements provided by
MS-Project when it is adopted for teaching, but to be
superior for educational proposes. This IU was
evaluated to verify whether students prefer to learn
using PpcProject or MS-Project. It has been
conducted through an experiment involving a total of
54 students. They were organized in two groups,
control and experimental groups. Each group carried
out the same activities, one using PpcProject, and
other using MS-Project. In a second stage, each group
carried out again the same activities, but using the
other tool. Thus, each student has responded to 24
questions (12 for each PM tool). Their answers have
been analysed by a non-parametric statistic test. This
analysis has demonstrated that PpcProject is more
suitable for teaching then MS-Project, except for the
resources allocation process.
Analysing the IUs presented in the related studies,
it is identified that the adopted educational PM tools
have contributed for students learning. The assistance
these tools provide have facilitated the content
understanding, beyond facilitating the PM tool usage
in class room. However, the IUs learning goals are
generally focused on time and human resources
management. Thus, considering the whole PM
process, still there is a huge gap of what is currently
been taught and all PM knowledge areas.
4 IU FOR TEACHING PM TOOLS
TO SUPPORT THE PM
PROCESS
In this context, this section presents an IU for
teaching PM tools focused on initiating and planning
processes groups, covering all knowledge areas. We
have focused on these processes groups because they
may be carried out within the IU discipline hours.
The execution of the planned projects may
demand more hours than available, especially in the
ICEIS 2016 - 18th International Conference on Enterprise Information Systems
396
Figure 2: Execution of ADDIE process.
IU context, which projects are related to students term
paper, demanding about a year to be concluded.
Effective and motivating IUs are developed
following an Instructional Design process, for
instance, ADDIE (Branch, 2009). An overview of the
ADDIE process for the development of the proposed
IU is presented in Figure 2.
As presented in Figure 2, the ADDIE process has
5 phases. Firstly, in analysis phase it is identified the
target audience and the learning environment. This
phase also includes the IU educational goals
definition. Then, in analysis phase, it is defined the
content to be addressed and its sequencing. This
content is grouped in one or more meetings, and with
the definition of instructional materials and activities,
it composes the IU teaching plan. In the development
phase, the instructional materials are developed, then
leading to the implementation phase, which performs
the IU application in class room. To evaluate the IU
quality it is necessary to perform observations and
data collection about teachers and students perception
about the IU.
The next sections present details of instructional
materials and about the IU evaluation process.
4.1 DotProject+
DotProject is one of most popular open-source tools
for PM (Mishra, 2013). And previous studies have
identified that among open-source alternatives, it is
the most aligned with PMBOK (Pereira et al. 2013).
DotProject architecture is organized in core modules,
developed by its core team, and add-on modules,
developed by users’ community, which, may be
installed on demand. Thus, we decided to adopt this
tool, not only because its wide coverage of the PM
process, but also because its functionalities may be
extended via add-on modules. In this context,
dotProject+ was developed, being composed by
dotProject core modules, and several add-on modules
that have been developed to enhance dotProject
functionalities to cover all PM knowledge areas and
also include didactic features (Figure 3). An example
of these features is the related to the organization of
its functionalities, which are hierarchically grouped
by processes groups, and then by knowledge areas.
Thus, when the student is using a certain
functionality, it is easy to identify what part of PM
process are been supported by each functionality.
Evaluating the Teaching of Project Management Tools through a Series of Case Studies
397
Figure 3: dotProject+ architecture.
Beyond adopting dotProject+, the IU also adopts
the dotProject+ usage guide. This material is
organized as presentation slides, which may serve to
assist the teachers in expositive classes and students
as a reference material. This material presents a
process designed in BPMN notation (Weske, 2012),
addressing all knowledge areas for the initiating and
planning processes groups. Thus, this process defines
the correct sequence to use dotProject+
functionalities, satisfying the requirements to execute
each process activity before initiating it. Also, it
presents print screens for each process activity,
explaining how to use dotProject+ functionalities to
support that part of the process (Figure 4).
Figure 4: dotProject+ - Usage Guide.
All instructional materials are freely available and
may be downloaded from dotProject+ web site
(http://www.gqs.ufsc.br/evolution-of-dotproject/).
Thus, any teacher interested may download all
material and then apply the IU.
1
This evaluation process has been approved by CEPSH/UFSC – an
ethic committee for researches with human beings, and is
registered under the number - 47734215.9.0000.0121.
4.2 IU Evaluation Process
The IU evaluation aims to identify its quality in
relation to its content, instructional materials, user
experience, students learning, and instructional
strategy, based on students and teachers perspectives.
It is carried out by a series of case studies, based on
the empiric study process defined by Wohlin (2012)
(Figure 5)
1
. This evaluation process is integrated with
the GQM approach (Basili et al., 1994), which is
utilized to define the evaluation goal, the analysis
questions which evaluate this goal, and metrics which
support answering these questions.
Figure 5: IU evaluation process (Wohlin et al., 2012; Basile
et al., 1994).
Derived from the GQM metrics, data collection
instruments were developed. There is a questionnary
for stundents and other for teachers, both containing
the same structure: demographic questions followed
by a set of affirmations using a likert scale to evaluate
their perception of each IU dimension. These
dimensions were chossen based on its compatibility
with the proposed IU, and in accordance with
previous studies (Arcuri and Fraser, 2012; Chen et al.,
2013). At the end, there are open questions to collect
points regarding strengths and improvements to be
made, as well as other comments. The individual
participation in the IU evaluation is voluntary and
anonymous.
The perception about the IU quality is also
evaluated by observation, analyzing the students and
teachers behavior when interacting with the
instructional materials, and when carrying out the
instructional activities. In this case the data is
collected by verbal or written feedback, that may be
provided by e-mail or using an online form that was
available for students and teachers along all the IU
application.
ICEIS 2016 - 18th International Conference on Enterprise Information Systems
398
5 CASE STUDY DEFINITION
Aiming to analyze the IU for teaching PM tools using
dotProject+, a case study has been defined (Figure 6).
The case study definition presents all steps necessary
for this study to be reproduced in future situations
(Wohlin et al., 2012).
Figure 6: Case study definition (Wohlin et al., 2012; Basile
et al., 1994).
The case study is organized in 3 stages. The first
one is related to its definition, where the research
goals are defined, and the IU is developed, addressing
the analysis, design and development phase of
ADDIE process. In this phase the IU evaluation is
also defined, and the data collection instruments are
developed. Details about the follow stages are
presented in the next section.
6 CASE STUDIES EXECUTION
The case study has been reproduced during six
consecutive semesters. It has been used in 3 different
Brazilian educational institutions, applied by 6
different teachers, in a total of 13 classes, teaching
more than 300 students (Table 2).
These case studies execution begins with teacher
preparing the computational environment, installing
dotProject+ and creating student accounts. The
teacher also has to get familiarized with the
instructional plan, with dotProject+ usage and with
the usage guide.
At the beginning of classes, the students are
organized in groups, receiving their credentials for
Table 2: Cases studies execution.
Semester Educational
institution
Course Discipline Teacher* Number of
students
2013-1 UFSC Computer Science Planning and management of
software projects
Teacher A 19
2013-2 UFSC Computer Science Planning and management of
software projects
Teacher A 21
2013-2 UFSC Information Systems Project management Teacher B 23
2014-1 UFSC Computer Science Planning and management of
software projects
Teacher A 30
2014-1 UFSC Information Systems Project management Teacher B 22
2014-1 SEBRAE
Espirito Santo
Environmental
Technician
Project management Teacher C 19
2014-2 UFSC Computer Science Planning and management of
software projects
Teacher A 17
2015-1 UFSC Computer Science Planning and management of
software projects
Teacher A 24
2015-1 UFSC Information Systems Project management Teacher D 19
2015-1 SENAC
Jaraguá do Sul
Information Technology
Management
Fundamentals in Project
Management
Teacher E 21
2015-2 SENAC
Jaraguá do Sul
Information Technology
Management
Fundamentals in Project
Management
Teacher E 24
2015-2 UFSC Information Systems Project management Teacher D 37
2015-2 UFSC Computer Science Planning and management of
software projects
Teacher E 28
Legend:
UFSC Federal University of Santa Catarina.
SENAC – National Service of Commercial Learning.
SEBRAE - Brazilian Service of Assistance of Small and Medium Enterprises.
* Teacher names have been replaced for privacy reasons.
Evaluating the Teaching of Project Management Tools through a Series of Case Studies
399
dotProject+ access, and the teacher carries out
expositive classes explaining how to access
dotProject+, as well as general navigation rules.
During the next meetings the teacher employs the
usage guide to explain how to use dotProject+ to
support the PM process for one or more knowledge
areas, until the students complete the whole PM
process for the initiating and planning processes
groups. At the end of the classes, the students may
export the project charter and the project plan in PDF
format, and it is delivered to the teacher for
evaluation.
After the IU application, the students and teachers
are invited to answer an evaluation questionnaire. The
answer of this questionnaire is non-mandatory,
anonymous and online. Once the data are collected, it
is analyzed and discussed, identifying the IU quality
for each dimension, and its strengths and
improvement points.
It is important to highlight that the case studies
were not exactly reproduced, because the IU
materials have been improved each semester, based
on the feedback we have received, both by students
and teachers. This feedback is normally related to
improvement suggestions or from reporting some
implementation issue. The feedback was provided
verbally or in writing, using an online form or e-mail.
In regards to data collection instruments, we have
developed and applied the complete evaluation
questionnaire (derivated from GQM), but just in the
case studies carried out after the second semester of
2014. In the previous semesters we applied a
questionnaire with open questions for individuals to
inform their improvement suggestions for the IU and
its perceived strengths, as well as other general
comments.
7 ANALYSIS
In this section we present the collected data from the
students perspective, based on the received answers
of the evaluation questionnaire. The analysis is
segmented by each evaluation dimension, presenting
the median of each questionnaire item, considering
the 26 students that have answered it.
Firstly, concerning the content dimension
(Figure 7), among the affirmations there are items
about the content relevance for computer
professionals, and whether it is addressed in proper
depth and extension.
The materials dimension aims to evaluate the
students perception about dotProject+ and its usage
guide. The dotProject+ (Figure 8) was evaluated
Figure 7: Content dimension evaluation data.
based on affirmations related to its contribution to the
understanding about the practical application of the
PM process, and also if it also assisted students during
the instructional activities.
Figure 8: dotProject+ evaluation data for PM process
coverage.
Yet, related to dotProject+, it was collected data
about the students perception of its usability (Figure
9), and how much it stimulate students and the
dificults they may had faced during its usage.
ICEIS 2016 - 18th International Conference on Enterprise Information Systems
400
Figure 9: dotProject+ usability evaluation data.
Furthermore, we collected data to evaluate how
the usage guide has contributed to students learning
(Figure 10); identifying how it was consulted during
the classes, and whether its content and structure are
suitable for students learning.
Figure 10: Usage guide slides evaluation data.
Figure 11: User experience dimension evaluation data.
The user experience dimension (Figure 11) was
evaluated based on affirmations that attempted to
identify how students become motivated when
carrying out the instructional activities.
Figure 12: Learning dimension evaluation data.
Then, in relation to the learning dimension
(Figure 12), we have utilized affirmations to
understand the knowledge about PM tools usage in
the beginning of the IU, and how it was after the IU.
Figure 13: Instructional strategy dimension evaluation data.
The instructional strategy dimension (Figure
13) was evaluated based on affirmations about the
contribution of theoretical classes and instructional
Evaluating the Teaching of Project Management Tools through a Series of Case Studies
401
activities in students learning. In addition to its
suitability to students learning preferences.
From the teachers’ perspective, 5 of the 6 teachers
have answered the IU evaluation questionnaire. Since
we have more data about students evaluation than
about teachers evaluation, we opted to present only
the former. Nevertheless, in the discussion section we
are addressing both perspectives.
8 DISCUSSION
In this section we present a discussion about the IU
quality focusing on each dimension we have defined
to be evaluated. We are considering the students and
the teachers perspectives, collected both by the
evaluation questionnaires and by observation.
Firstly, about the learning dimension, the
students stated that the IU assists them to correlate the
theoretical content with the professional practice,
through the functionalities they learned to use on
dotProject+. Also, the majority of students have
finished all activities they were delegated, indicating
they effectively have learned how to use a PM tool to
elaborate a project charter and a project plan,
covering all PM knowledge areas. From the teachers
perspective they highlighted that the IU assisted the
student learning, mainly because the employed
instructional strategy and materials are strictly
designed for the IU learning objectives.
Regarding the instructional materials
dimension, the students stated that the usage guide
assisted in their understanding about the whole PM
process, explaining how to use dotProject+
functionalities to carry out each process activity.
However, specifically about dotProject+, the most
addressed issue was related to its usability, which in
a few cases was considered complex to use. From the
teachers perspective, they considered the entire
instructional material, ready to use, assisted in classes
preparation. They also considered the usage guide
very important, especially because of the lack of
experience of most students with PM tools. They also
considered that the usage of dotProject+ assists in the
learning of the whole discipline, because the students
have the opportunity to apply the theoretical content
through the tool functionalities. As regards to
dotProject+ functionalities, they highlighted that it is
very positive in supporting all of these processes in a
single tool. It avoids the usage of several tools to
cover all these functionalities, thus facilitating the
integration of its results in a complete project plan
generated by the tool. However, some teachers
complained about the support provided by HR
allocation process, because it demands many steps,
making it complex to be used. Other teachers also
complained about the complexity for the installation
of dotProject+, including many add-on modules, a
complication when the teacher does not master the
related technologies.
The issue about dotProject+ usability was
drastically reduced after the 2015 first semester, when
the dotProject+ version 2.0 was adopted, which had
its usability improved based on an analysis carried out
by a software usability researcher. In this same
version we also have included a new theme, to make
dotProject+ more attractive (in relation to the
standard dotProject theme), also assisting in usability
issues and facilitating student receptiveness.
Regarding the issue related to the HR allocation
process, it also has been improved in the dotProject+
version 2.0, which has simplified this process.
About the content dimension, the IU has received
a positive feedback about the content coverage and
depth, both by students and teachers. It was because
dotProject+ supports several functionalities, enabling
to apply on practice many PM techniques that are
taught in theoretical classes, covering all PM
knowledge areas. Most students also considered the
content relevant for a computer professional.
In relation to the user experience dimension, the
students highlighted that the most motivational aspect
of the IU is that it enables them to have a clear
comprehension about the practical application of the
PM content. Although their motivation had been
affected by the usability issues of dotProject+, this
negativity has been reduced after the dotProject+
version 2.0 was adopted. From the teachers perspective
they also considered that the IU prepares the students
for their professional career, and all teachers have
informed that they would recommend the IU to other
teachers that need to teach about PM tools.
In relation to the instructional strategy
dimension, the students considered that the
intercalation between the theoretical and the practical
classes has facilitated the content understanding, and
also the comprehension of its practical applicability.
Regarding the teachers perspective, they highlighted
that the instructional activities guided by the PM
process facilitated student understanding about the
correct order to use dotProject+ functionalities. Thus,
the result being, that most students concluded the
elaboration of the project charter and the project plan.
Based on the presented discussion, it is evident
that the proposed IU makes a positive contribution to
student learning, and meets for teacher demands for
all evaluated dimensions. During its application
several improvement suggestions have been
ICEIS 2016 - 18th International Conference on Enterprise Information Systems
402
collected, and many of them have already been
implemented. Thus, at the current stage the IU
reached a maturity level that allows it to be adopted
by other teachers that need to teach about PM tools,
aligned with the PM process as defined by PMBOK.
In comparison to related studies, thought the
evaluation of the presented IU, it has demonstrated to
assist in the teaching of a more extensive part of PM
process than any other related studies. However,
some studies, such as Gregoriou et al. (2010), besides
covering just time and human resources knowledge
areas for planning processes group, it offers several
specific instructional features to assist students.
Among these features are the configuration of
difficulty level, automatic feedback, and tutorial
videos. On the other hand, only the presented IU has
adopted a material as the usage guide, which is
oriented by the PM process, and provides instructions
about how each step of this process may be supported
by PM tools functionalities.
8.1 Threats to Validity
As any research there are some threats to validity
(Wohlin et al., 2012). They are analyzed for
conclusion, construction, and external threats to
validity.
Threats to conclusion validity may occur due to
inconsistences in the data collected. In this research
the individuals may lack some knowledge related to
PM, even while being taught during the discipline. It
may lead to wrong interpretation of questionnaire
items and as a consequence lead to inconsistent
answers. To reduce this threat, the questionnaire was
designed carefully analyzing the employed
terminology, bringing it as near as possible to the
student language. Also, when the students answered
the questionnaire still in the context of the discipline,
they may have been afraid to be punished for their
answers, especially when criticizing some IU aspects.
This was mitigated by anonymising the answers,
applying the questionnaire only after all student
evaluations had been concluded and having this final
process conducted by an external researcher, instead
of the teacher. However, especially from a students
perspective, a significant part of our evaluation has
been based on data collected in an ad-hoc manner,
based on verbal and written feedback provided during
the case studies instances.
Threats to construction validity are related to the
data collection instrument, which may not contain the
necessary set of questions to reach the evaluation
goal. We have employed the GQM approach to
design the questionnaire, thus the evaluation goal was
systematically deployed in question analysis and
metrics, which were represented by questionnaire
items.
Threats to external validity may occur by not
obtaining a significant sample. In fact, we still do
have not collected a significant amount of structured
data; consequentially performing the statistical study
with only 26 students answers. However, we have
mitigated that by applying the IU in different
semesters, involving 6 teachers and 304 students,
which are significant events for a general evaluation
of the IU for teaching PM tools with the support of
dotProject+.
9 CONCLUSIONS
This study has presented the evaluation of an IU for
teaching the usage of PM tools. This IU has
introduced the educational PM tool - dotProject+,
which is an enhancement of one of the most popular
open-source tools for PM. DotProject+ includes a
more comprehensive support to the PM process and
educational features. After carrying out a series of
case studies, and analysing the collected data from
students and teachers perspectives, the IU has
demonstrated to be effective for teaching the usage of
PM tools for the initiating and planning processes
groups, covering all PM knowledge areas. Students
highlighted that they have learned the content, and
consider they are able to reproduce it in their
professional activities. Teachers have highlighted
they would like to use this IU again, and would
recommend it to other teachers. Future studies may
expand upon the instructional feedback of
dotProject+, beyond the creation of other IUs to
address other processes groups that were not included
in this research.
ACKNOWLEDGEMENTS
This work was supported by the CNPq (Conselho
Nacional de Desenvolvimento Científico e
Tecnológico – www.cnpq.br), an entity of the
Brazilian government focused on scientific and
technological development.
REFERENCES
ACM, IEEE Computer Society, 2013. Computer Science
Curricula 2013.
Evaluating the Teaching of Project Management Tools through a Series of Case Studies
403
Arcuri A., Fraser G., 2012. Sound Empirical Evidence in
Software Testing. In: Proc. of 34th International
Conference on Software Engineering, Zurich/
Switzerland.
Basili, V., Caldier, G., Rombach, D., 1994. The Goal
Question Metric Approach. Encyclopedia of software
engineering, pp. 528–532.
Branch R., 2009. Instructional Design: The ADDIE
Approach. Springer, 2
nd
edition.
Car Ž., Belani H., Pripužić K., 2007. Teaching Project
Management in Academic ICT Environments. In: Proc.
of the Int. Conf. on computer as a tool, Warsaw.
Cicibas H., Unal O., Demir K, 2010. A comparison of
project management software tools (PMST). In: Proc.
of the 9th Software Engineering Research and Practice,
Las Vegas.
Chen H., Chen Y., Chen K., 2013. The Design and Effect
of a Scaffolded Concept Mapping Strategy on Learning
Performance in an Undergraduate Database Course.
IEEE Transactions on Education, vol. 56, n. 3, pp. 300-
307.
Fabac R., Radoševic D., Pihir I., 2010. Frequency of use
and importance of software tools in project
management practice in Croatia. In: Proc. of 32nd Int.
Conf. on Information Technology Interfaces, Cavtat.
Gregoriou G., Kirytopoulos K., Kiriklidis C., 2010. Project
Management Educational Software (ProMES).
Computer Applications in Engineering Education, vol.
21, n. 1, pp. 46–59.
Gonçalves R., Wangenheim C., 2015. How to Teach the
Usage of Project Management Tools in Computer
Courses: A Systematic Literature Review. In: Proc. of
the Int. Conf. on Software Engineering and Knowledge
Engineering, Pittsburgh.
Hill, H., Rowan, B., Ball, D., 2005. Effects of Teachers'
Mathematical Knowledge for Teaching on Student
Achievement. American Educational Research
Journal, 42(2), p. 371-406.
Keil, M., Rai, A., Mann J., 2003. Why software projects
escalate: The importance of project management
constructs. IEEE Transactions on Engineering
Management, vol. 50, n.3, pp. 251–261.
Kirkpatrick, D., kirkpatrick, J., 2012. Evaluating Training
Programs: The Four Levels. Berrett-Koehler
Publishers, 4
nd
edition.
Mishra A., Mishra D., 2013. Software Project Management
Tools: A Brief Comparative View, ACM SIGSOFT
Software Engineering Notes, 38 (3), pp. 1-4.
Ojeda O., Reusch P., 2013. Sustainable procurement -
Extending project procurement concepts and processes
based on PMBOK. In: Proc. of 7th International
Conference on Intelligent Data Acquisition and
Advanced Computing Systems, Berlin/Germany, pp.
530 – 536.
Pereira, A., Gonçalves R., Wangenheim, C., 2013.
Comparison of open source tools for project
management. International Journal of Software
Engineering and Knowledge Engineering, vol. 23, n. 2,
2013, pp. 189-209.
PMI – Project Management Institute, 2013. A Guide to the
Project Management Body of Knowledge, 5. ed.,
Newtown Square.
Reid K., Wilson G., 2007 DrProject: A Software Project
Management Portal to Meet Educational Needs. In:
Proc. of the Special Interest Group on Computer
Science Education, Covington.
Salas-Morera L., Arauzo-Azofra A., García-Hernández, L.,
2013. PpcProject: An educational tool for software
project management. Computers & Education, vol. 69,
n. 1, pp. 181-188.
The Standish Group, 2013. Chaos Manifesto 2013, Boston.
Wohlin C., Runeson P., Höst M., 2012. Experimentation in
Software Engineering: An Introduction, Springer.
Weske, M., 2012. Business Process Management:
Concepts, Languages, Architectures. Springer, 2
nd
edition.
ICEIS 2016 - 18th International Conference on Enterprise Information Systems
404