Understanding Approval Rating of Agile Project Management Tools
using Twitter
Martina Matta and Michele Marchesi
Universita’ degli Studi di Cagliari, Piazza d’Armi, 09123 Cagliari, Italy
Keywords:
Agile, Sentiment Analysis, Twitter API, Agile Project Management Tools, Google Trends.
Abstract:
The role of managing a software project can be extremely complicated, requiring many teams and organi-
zational resources. Many people are acknowledging that Agile development is helpful to business, with an
high increase over the last years in the number of people who believe that Agile helps companies to com-
plete projects faster. Since there is a multitude of available tools that supports Agile methods, in this position
paper we tried to understand which tools are most looked for using Web Search media, most mentioned and
most appreciated through Social media. Furthermore, we applied automated Sentiment Analysis on shared
short messages of users on Twitter, one of the most popular social networks, in order to analyze automatically
developers’ opinions, sentiments, evaluations and attitudes.
1 INTRODUCTION
The role of managing a software project can be ex-
tremely complicated, requiring many teams and orga-
nizational resources. Software projects tend to raise
many issues and problems throughout their life cy-
cles. The quality of the final software product is re-
lated to how the project has been managed (Mishra
and Mishra, 2013). Project Management, then, is the
application of knowledge, skills, and techniques to
execute project effectively and efficiently. This action
typically includes facing the needs and expectations
of the project stakeholders as the project is sched-
uled and built up, as well as identifying the require-
ments of the project and balancing the project con-
straints (Heagney, 2011). Recently, the evolution of
Project Management tools for both software and non-
software applications is speeding up at a rapid pace
and, then, the number of available products is grow-
ing considerably (Fortune et al., 2011).
Many people are acknowledging that Agile devel-
opment is helpful to business, with an high increase
over the last years in the number of people who be-
lieve that Agile helps companies to complete projects
faster. Teams and organizations often support their
Agile practices using an Agile Project Management
tool, defined as a project and work management tool
that helps a team or an organization to improve their
quality and enhance project agility (Beck et al., 2001).
More and more software companies shift towards ag-
ile methodologies to achieve speed, efficiency and
quality of the software. They take advantage of the
various agile methods such as Extreme Programming
(XP), Scrum(Schwaber, 2004) and Lean (Mary and
Tom, 2003). Since there is a multitude of available
tools that supports Agile methods, we decided to eval-
uate directly which tools are most looked for using
Web Search media, most mentioned and most appre-
ciated through Social media.
Nowadays, Web 2.0 services such as blogs,
tweets, forums, chats, email etc. are widely used as
media for communication, with great results. Re-
search has established that software engineers use
Twitter, one of the most popular social network, in
their work to communicate about software engineer-
ing topics. Through use of social media services,
team members have opportunities to acquire more de-
tailed information about their peers’ expertise (Dessai
et al., 2012).
The popularity of Twitter is attracting the atten-
tion of researchers. Several recent studies examined
Twitter from different point of views, including the
sentiment prediction power (Tumasjan et al., 2010),
the topological characteristics of Twitter (Kwak et al.,
2010) or tweets as social sensors of real-time events
(Sakaki et al., 2010). Another stream of research
focuses on corporate applications of microblogging
such as the company internal use for project manage-
ment (B
¨
ohringer and Richter, 2009) or the analysis
of Twitter as electronic word of mouth in the area of
168
Matta M. and Marchesi M..
Understanding Approval Rating of Agile Project Management Tools using Twitter.
DOI: 10.5220/0005557201680173
In Proceedings of the 10th International Conference on Software Engineering and Applications (ICSOFT-EA-2015), pages 168-173
ISBN: 978-989-758-114-4
Copyright
c
2015 SCITEPRESS (Science and Technology Publications, Lda.)
product marketing (Jansen et al., 2009). Our aim is
to evaluate the power of Twitter in order to under-
stand the level of popularity and appreciation of Ag-
ile Project Management tools. We decided to apply
automated Sentiment Analysis on shared short mes-
sages of users on Twitter in order to analyze auto-
matically people’s opinions, sentiments, evaluations
and attitudes. We compared our results with one of
the most used web search media, Google Trends, and
with the related results of published surveys (Ver-
sionOne, 2014) (Azizyan et al., 2011) (Mishra and
Mishra, 2013).
The body of this paper is organized in five major
sections. Section 2, describes the background, section
3 presents the research steps of our study and section
4 summarizes and discusses our results. Finally, Sec-
tions 5 presents the conclusions and suggestions for
future work.
2 BACKGROUND
We analyzed the existing literature and surveys on
tool usage in the context of Agile development, look-
ing also for ”common” sources such as websites,
white-papers and published surveys. In the last years,
two relevant surveys concerning Agile Project Man-
agement tools have been published.
The first is a study conducted by the tool vendor
VersionOne in 2013. This is the latest of a series of
similar surveys made every year by VersionOne, that
is also a tool producer. Most of the survey is fo-
cused on the state of Agile development itself (Ver-
sionOne, 2014). A total of 3500 responses were
collected, analyzed and described in a summary re-
port. The survey is very detailed, including infor-
mation such as reasons for adopting Agile methods,
resulting benefits, roles and so on. They declare
that the most commonly used tools still are standard
office productivity tools ,such as Excel (66%) fol-
lowed by tools like Microsoft Project (48%), Ver-
sionOne (41%), Atlassian/Jira (36%), Microsoft TFS
(26%), IBM ClearCase (10%), LeanKit (5%), Xplan-
ner (4%) and Trello (outside the survey choices). In
addition to tool usage, the respondents were asked
whether they would recommend the tools they are
using based on their past or present use. Versio-
neOne had the highest satisfaction of any other tool
evaluated (93%), followed by Atlassian/Jira (87%),
LeanKit (84%), TargetProcess (83%), Microsoft TFS
(79%) and ThoughtWorks Mingle (69%).
The other relevant survey is a study conducted in
2011(Azizyan et al., 2011). It is an Agile Project
Management Tool survey with 121 answers from 35
countries. The survey reports also features, a list of fa-
vored tool types and a list of tools that the respondents
felt most satisfied with. This survey reports that the
most commonly used tool within the companies stud-
ied is the physical wall (26%) followed by Microsoft
Project (8%), Rally (5%), Mingle (3%), VersionOne
(2%), JIRA (2%) and Team Foundation Server (2%).
In order to verify the accuracy of the informa-
tion reported on the cited surveys, we decided to
check these data through a public web service, called
”Google Trends”
1
. It is a feature of the search engine
that illustrates how frequently a fixed search term was
looked for using Google. Using Google Trends you
can compare up to five topics at one time to view rel-
ative popularity, allowing you to gain an understand-
ing of the hottest search trends of the moment, along
with those developing in popularity over time. Fol-
lowing this kind of approach and taking into consid-
eration the list of previously mentioned tools, we are
able to evaluate which ones are the most looked for
using Google’s search engine.
Figure 1 shows the top five of the most mentioned
Agile Project Management tools in Google Trends.
We found that, nowadays, the most searched tools are,
in descending order of search, Jira, Trello, Mingle,
Microsoft Project and Team Foundation Server. This
graph shows also that, in the time period from 2004 to
today, Microsoft Project has been losing popularity in
favor of a more and more increasing fame of Jira and
Trello.
Figure 1: Most mentioned Agile Project Management tools
in Google Trends.
3 METHODOLOGY
3.1 Sentiment Analysis and Twitter
Besides analyzing which are the most searched tools,
we wanted to examine what customers really think
about them. In the last years, the social web has been
commercially exploited for goals such as automati-
cally extracting customer opinions about products or
brands, to find which aspects are liked and which are
disliked (Thelwall et al., 2011).
1
http://www.google.it/trends/
UnderstandingApprovalRatingofAgileProjectManagementToolsusingTwitter
169
Twitter is an online social networking website and
microblogging service that allows users to post and
read text-based messages of up to 140 characters,
known as ”tweets”. It seems to be used to share in-
formation and to describe minor daily activities (Java
et al., 2007). The short format of tweet is a defined
characteristic of the service, allowing informal col-
laboration and quick information sharing. Black et
al. presented a survey conducted to collect informa-
tion on social media use in global software systems
development. Twitter was found to be the most popu-
lar media and respondents affirmed that specification,
source codes and design information were shared over
social media (Black et al., 2010). Singer et al. showed
that Twitter helps developers keep up with the fast-
paced development landscape. They use it to stay
aware of industry changes, for learning and for build-
ing work relationships (Singer et al., 2014). Romero
et al. showed that the aspect responsible for the pop-
ularity of certain topics is the influence of users of
the network on the spread of content. Some mem-
bers produce content that resonates very strongly with
their followers thus causing the content to propagate
and gain popularity (Romero et al., 2011).
So, the tweets sometimes express opinions about
different topics, and for this reason we decided to
evaluate how much users speak about Agile Project
Management tools. In order to evaluate if a user re-
ally appreciates the tool, we tried to predict the sen-
timent analyzing the collection of tweets. By recent
years, there is a wide collection of research surround-
ing machine learning techniques, in order to extract
and identify subjective information in texts. This area
is known as sentiment analysis or opinion mining.
The research field of sentiment analysis has devel-
oped many algorithms to identify if the opinion ex-
pressed is positive or negative (Pang and Lee, 2008).
The strength of the sentiment analysis applied to the
Twitter domain by applying similar machine learning
techniques to classifying the sentiment of tweets (Go
et al., 2009).
For these reasons, we chose to use automated sen-
timent analysis techniques to identify the sentiments
of tweets regarding Agile Project Management tools.
Since the goal of this research is neither to develop
a new sentiment analysis nor to improve an existing
one, we to used ”SentiStrenght”, a tool developed by a
team of researchers in the UK (Thelwall et al., ). This
tool was implemented to analyze informal short mes-
sages. Based on the formal evaluation of this system
on a large sample of comments from MySpace.com,
the accuracy of predicting positive and negative emo-
tions was something similar to that of other systems
(72.8% for negative emotions and 60.6% for positive
emotions, based on a scale of 1-5), and compared to
other methods, SentiStrenght showed the highest cor-
relation with human coders (Thelwall et al., 2010).
The tool SentiStrenght is able to assess each mes-
sage separately and, at the end, it returns one singular
value: a positive (1), a negative (-1) or a neutral sen-
timent (0).
3.2 Data Collection
Starting from the list of the most used Agile Project
Management tools found in the existing literature, we
evaluated how many people speak on Twitter about
these tools, and what they think about them. The
tweets are available and are easily retrieved mak-
ing use of Twitter Application Programming Interface
(API). Composing the hashtags # or @ with the name
of the tool, we can see all the tweets that mentioned
the analyzed system. The system architecture consists
of four components:
Twitter Streaming API: it provides access to Twit-
ter data, both public and protected, on a nearly
real-time basis. A persistent connection is cre-
ated between our system and Twitter. As soon as
tweets come in, Twitter notifies our system in real
time, allowing us to store them into our database.
DataStore: our datastore consists of a back-end
database engine, using MySQL as RDBMS, that
repeatedly saves the incoming tweets from the
Twitter Streaming API.
SentiStrenght Tool
Java Module: this component allows us to send
automated requests to Twitter Streaming API, to
recover new tweets about analyzed tools, to parse
data gained and to store them into our datas-
tore. In a later stage, these data are sent to Sen-
tiStrenght tool in order to evaluate automatically
the users opinion.
We analyzed a collection of tweets posted on Twit-
ter between September 2014 and March 2015 regard-
ing the Agile project management tools most men-
tioned. We found a total of 84837 tweets. We then
used the SentiStrenght tool to evaluate the comments
extracted from Twitter. Given as input all tweets of
every tool previously mentioned, a score for each
comment was assigned.
1 if the comment is positive
-1 if the comment is negative
0 if the comment is neutral
ICSOFT-EA2015-10thInternationalConferenceonSoftwareEngineeringandApplications
170
Table 1: Most quoted Agile Project Management tools or-
dered by number of tweets.
Number of Tweets
Trello 32613
Jira 21903
VersionOne 7887
Microsoft TFS 6730
Rally 5684
Mingle 4629
Microsoft Project 2249
LeanKit 1595
TargetProcess 1547
4 RESULTS AND DISCUSSION
4.1 Number of Tweets Analysis
Of the total 84837 tweets, 39756 were neutral (46%),
35409 were positive (42%) and 9672 were negative
(11%). In Table 1, the most quoted Agile Project
Management tools, ordered by number of tweets, are
shown. XPlanner and IBM ClearCase were excluded
since few comments were found about them.
Going back to the results of Google Trends about
the Agile Project Management tools reported in Fig-
ure 1, we observed a striking similarity between it
and the number of tweets we found. In fact, Google
Trends reported, in order of search, Jira, Trello, Min-
gle, Microsoft Project and Team Foundation Server.
Taking a look to Table 1, we can see that the rank
is similar, with the exception of VersionOne. So,
we found that there is a relation between the num-
ber of tweets posted for each tool and the Google
searches about it. TargetProcess turns out to be the
least tweeted, maybe due to the fact that it is a re-
cent tool and it is still little known worldwide. Con-
sequently, it is possible to stress that the most used
and quoted Agile Project Management tools on Twit-
ter and Google are Jira, Trello and VersionOne.
4.2 Comparison Between Negative and
Positive Tweets
From the list of tweets, we evaluated the sentiment
using SentiStrenght tool, in order to understand if the
users really appreciated these Agile Project Manage-
ment tools. Among all tweets, we observed that, on
average, 40% of the tweets don’t represent a senti-
ment, or Sentistrenght is not able to identify it. After
a careful analysis, it was observed that a lot of tweets
are neutral because often people wrote texts asking
help, non-expressive comments, tweets in a different
Table 2: Comparison between negative and positive tweets.
Tool
Negative Positive
Total % Total %
Trello 2760 8 % 16569 51 %
Jira 3445 16 % 7043 32 %
VersionOne 808 10 % 2612 33 %
MicrosoftTFS 859 13 % 2290 34 %
Rally 673 12 % 2612 46 %
Mingle 554 12 % 1971 43 %
MicrosoftProj 247 11 % 1055 47 %
LeanKit 89 6 % 829 52 %
TargetProcess 237 15 % 428 28 %
language than English, or simple links that lead to
other web pages.
An Agile Project Management tool that has more
positive than negative tweets is likely to be success-
ful. After this evaluation, we determined that, for all
tools, there were more positive messages than neg-
ative ones, and that positive messages were almost
3 times more likely to be forwarded than negative
messages. In Table 2 we show the data found about
positive and negative tweets using SentiStrenght. We
observed that, in general, the percentage of negative
tweets found in each tool is lower than 14%; Jira,
however, achieved 16% of negative comments. In
fact, users of Jira sometimes posted tweets in which
they explained their problems using the tool.
Based on this result, we found that Trello is clearly
the most appreciated tool, getting a high percentage
of positive comments (51%) and only 8% of nega-
tive comments. Beyond this tool, the other tools with
a high percentage of positive comments are LeanKit
(52%) and Microsoft Project (47%).
To better quantify the sentiments, we defined the
score PNRatio as the ratio of positive versus negative
tweets on each tool.
PNRatio =
|Tweets with Positive Sentiment|
|Tweets with Negative Sentiment|
(1)
The indicator of eq.1 was applied to all tools and
Table 3 shows the results. We compared our re-
sults to the satisfaction achievement obtained by Ver-
sionOne survey, where it is possible. In our study the
most quoted (excluding Trello and Rally, since they
were not included in VersionOne survey) are LeanKit,
Microsoft Project and VersionOne. We noticed that
LeanKit and VersionOne also exhibit a high percent-
age of satisfaction from the survey, greater than 84%.
Nevertheless, Trello turns out to be one of the most
popular tool, showing a ratio P/N of 6.00, with a total
of 16569 positive comments.
In Table 3 we observed a relation between PN ra-
tio and its satisfaction, for most tools. As a matter of
UnderstandingApprovalRatingofAgileProjectManagementToolsusingTwitter
171
Table 3: Comparison between PN ratio of eq.1 and Ver-
sionOne satisfaction survey results by tools.
Tool Name Ratio P/N (VersionOne, 2014)
LeanKit 9.31 84%
Trello 6.00 -
Microsoft Project 4.27 53%
Rally 3.88 -
VersionOne 3.23 93%
Mingle 3.55 69%
Microsoft TFS 2.66 79%
Jira 2.04 87%
TargetProcess 1.80 83%
fact, the majority of the tweets represents a positive
sentiment while the negative comments are less than
16% and PN Ratio is always greater than 2. Also in
this case, we can confirm that the favorite tools are
Trello and VersionOne.
4.3 Relationship with Agile
Methodologies and Approaches
We decided to analyze all tweets in order to assess
whether some Agile methodology was mentioned by
someone and, in the case, which one of these and
in which tool. So, for each tweet, we evaluated
if the user cited one of the major agile method-
ologies and approaches. We chose to test Scrum,
Kanban approach, Lean and eXtreme Programming
(XP) (Schwaber, 2004) (Mary and Tom, 2003). Ta-
ble 4 shows the most cited methodologies within the
tweets; we can notice that the methodology most
mentioned is Scrum, followed by Kanban, Lean and
finally eXtreme Programming. About Scrum, we
found that the main contribution is given by Ver-
sionOne, Rally, Jira and Trello. On the other hand
Trello, Jira and LeanKit quoted Kanban approach.
Table 4: Most mentioned Agile metodologies in the tweets.
Methodology Tot tweets Tool contribution
Scrum
2021 VersionOne 53%
Rally 20%
Jira 10%
Trello 6%
Kanban
807 LeanKit 34%
Trello 40%
Jira 12%
Lean
439 VersionOne 30%
LeanKit 21%
eXtreme
111 VersionOne 42%
Jira 27%
Programming Trello 14%
Figure 2: Most mentioned Agile metodologies in Google
Trends.
We checked these results using Google Trends, to
confirm the ranking found using Twitter. Figure 2
shows the results. Scrum is clearly the most searched
word, so we can say that it’s the most popular method-
ology, and its fame is still growing. The popularity
of Lean and Kanban are quite similar while eXtreme
Programming, over the years, lost most of its fame.
In the end, Google Trends confirmed the rankings re-
lated to Agile methodologies and approaches found
using Twitter. It affirms the growing popularity of
Scrum in the last years, and the effectiveness of us-
ing tweets to assess the popularity of something.
5 CONCLUSION
This position paper presents an analysis of a set of
tweets, starting from the results of different surveys
about Agile project management tools most used by
companies. We analyzed a total amount of 84837
tweets about this kind of tools, covering a period be-
tween September 2014 and March 2015. We applied
automated Sentiment Analysis on tweets, in order to
assess whether the opinions of the users about the ex-
amined tool were positive or negative. The aim of this
work was to understand which tools are most men-
tioned and looked for, and, if those who use it re-
ally appreciate it. Using Twitter, it was possible to
recognize the most mentioned tools, and to evaluate
their level of appreciation. This approach is able to
give immediate results, reducing the need to submit
surveys to users. We can conclude that Jira, Trello
and VersionOne are the most twitted and, at the same
time, the most appreciated Agile Project Management
tools. Google Trends confirms that the fame of Jira
and Trello is ever increasing in these years.
In future we plan to extend the increase the num-
ber of analyzed tweets, and to perform a complete
correlation analysis of the results. We also plan to
gather tweets along a much longer time interval, per-
forming trend analysis on the tweets. In this way, we
could assign a specific weight to each tweet and check
whether the results remain unchanged with the addi-
tion of this variable.
ICSOFT-EA2015-10thInternationalConferenceonSoftwareEngineeringandApplications
172
REFERENCES
Azizyan, G., Magarian, M. K., and Kajko-Matsson, M.
(2011). Survey of agile tool usage and needs. In Agile
Conference (AGILE), 2011, pages 29–38. IEEE.
Beck, K., Beedle, M., Van Bennekum, A., Cockburn, A.,
Cunningham, W., Fowler, M., Grenning, J., High-
smith, J., Hunt, A., Jeffries, R., et al. (2001). Man-
ifesto for agile software development.
Black, S., Harrison, R., and Baldwin, M. (2010). A survey
of social media use in software systems development.
In Proceedings of the 1st Workshop on Web 2.0 for
Software Engineering, pages 1–5. ACM.
B
¨
ohringer, M. and Richter, A. (2009). Adopting social soft-
ware to the intranet: a case study on enterprise mi-
croblogging. In Proceedings of the 9th Mensch &
Computer Conference, pages 293–302.
Dessai, K. G., Kamat, M. S., and Wagh, R. (2012). Appli-
cation of social media for tracking knowledge in agile
software projects. Available at SSRN 2018845.
Fortune, J., White, D., Jugdev, K., and Walker, D. (2011).
Looking again at current practice in project manage-
ment. International Journal of Managing Projects in
Business, 4(4):553–572.
Go, A., Bhayani, R., and Huang, L. (2009). Twitter senti-
ment classification using distant supervision. CS224N
Project Report, Stanford, pages 1–12.
Heagney, J. (2011). Fundamentals of project management.
AMACOM Div American Mgmt Assn.
Jansen, B. J., Zhang, M., Sobel, K., and Chowdury, A.
(2009). Twitter power: Tweets as electronic word of
mouth. Journal of the American society for informa-
tion science and technology, 60(11):2169–2188.
Java, A., Song, X., Finin, T., and Tseng, B. (2007). Why we
twitter: understanding microblogging usage and com-
munities. In Proceedings of the 9th WebKDD and 1st
SNA-KDD 2007 workshop on Web mining and social
network analysis, pages 56–65. ACM.
Kwak, H., Lee, C., Park, H., and Moon, S. (2010). What is
twitter, a social network or a news media? In Proceed-
ings of the 19th International Conference on World
Wide Web, WWW ’10, pages 591–600, New York,
NY, USA. ACM.
Mary, P. and Tom, P. (2003). Lean software development:
An agile toolkit for software development managers.
Lean Software Development: An Agile Toolkit for
Software Development Managers.
Mishra, A. and Mishra, D. (2013). Software project man-
agement tools: a brief comparative view. ACM SIG-
SOFT Software Engineering Notes, 38(3):1–4.
Pang, B. and Lee, L. (2008). Opinion mining and senti-
ment analysis. Foundations and trends in information
retrieval, 2(1-2):1–135.
Romero, D. M., Galuba, W., Asur, S., and Huberman,
B. A. (2011). Influence and passivity in social media.
In Proceedings of the 20th International Conference
Companion on World Wide Web, WWW ’11, pages
113–114, New York, NY, USA. ACM.
Sakaki, T., Okazaki, M., and Matsuo, Y. (2010). Earth-
quake shakes twitter users: Real-time event detection
by social sensors. In Proceedings of the 19th Inter-
national Conference on World Wide Web, WWW ’10,
pages 851–860, New York, NY, USA. ACM.
Schwaber, K. (2004). Agile project management with
Scrum. Microsoft Press.
Singer, L., Figueira Filho, F., and Storey, M.-A. (2014).
Software engineering at the speed of light: How de-
velopers stay current using twitter. In Proceedings of
the 36th International Conference on Software Engi-
neering, pages 211–221. ACM.
Thelwall, M., Buckley, K., and Paltoglou, G. Sentistrength,
2011. available online.
Thelwall, M., Buckley, K., and Paltoglou, G. (2011).
Sentiment in twitter events. Journal of the Ameri-
can Society for Information Science and Technology,
62(2):406–418.
Thelwall, M., Buckley, K., Paltoglou, G., Cai, D., and Kap-
pas, A. (2010). Sentiment strength detection in short
informal text. Journal of the American Society for
Information Science and Technology, 61(12):2544–
2558.
Tumasjan, A., Sprenger, T. O., Sandner, P. G., and Welpe,
I. M. (2010). Predicting elections with twitter:
What 140 characters reveal about political sentiment.
ICWSM, 10:178–185.
VersionOne (2014). Inc. 8th annual state of agile develop-
ment survey. http://www.versionone.com/pdf/2013-
state-of-agile-survey.pdf. [Online; accessed 8-April-
2015].
UnderstandingApprovalRatingofAgileProjectManagementToolsusingTwitter
173