Understanding Approval Rating of Agile Project Management Tools

using Twitter

Martina Matta and Michele Marchesi

Universita’ degli Studi di Cagliari, Piazza d’Armi, 09123 Cagliari, Italy

Keywords:

Agile, Sentiment Analysis, Twitter API, Agile Project Management Tools, Google Trends.

Abstract:

The role of managing a software project can be extremely complicated, requiring many teams and organi-

zational resources. Many people are acknowledging that Agile development is helpful to business, with an

high increase over the last years in the number of people who believe that Agile helps companies to com-

plete projects faster. Since there is a multitude of available tools that supports Agile methods, in this position

paper we tried to understand which tools are most looked for using Web Search media, most mentioned and

most appreciated through Social media. Furthermore, we applied automated Sentiment Analysis on shared

short messages of users on Twitter, one of the most popular social networks, in order to analyze automatically

developers’ opinions, sentiments, evaluations and attitudes.

1 INTRODUCTION

The role of managing a software project can be ex-

tremely complicated, requiring many teams and orga-

nizational resources. Software projects tend to raise

many issues and problems throughout their life cy-

cles. The quality of the ﬁnal software product is re-

lated to how the project has been managed (Mishra

and Mishra, 2013). Project Management, then, is the

application of knowledge, skills, and techniques to

execute project effectively and efﬁciently. This action

typically includes facing the needs and expectations

of the project stakeholders as the project is sched-

uled and built up, as well as identifying the require-

ments of the project and balancing the project con-

straints (Heagney, 2011). Recently, the evolution of

Project Management tools for both software and non-

software applications is speeding up at a rapid pace

and, then, the number of available products is grow-

ing considerably (Fortune et al., 2011).

Many people are acknowledging that Agile devel-

opment is helpful to business, with an high increase

over the last years in the number of people who be-

lieve that Agile helps companies to complete projects

faster. Teams and organizations often support their

Agile practices using an Agile Project Management

tool, deﬁned as a project and work management tool

that helps a team or an organization to improve their

quality and enhance project agility (Beck et al., 2001).

More and more software companies shift towards ag-

ile methodologies to achieve speed, efﬁciency and

quality of the software. They take advantage of the

various agile methods such as Extreme Programming

(XP), Scrum(Schwaber, 2004) and Lean (Mary and

Tom, 2003). Since there is a multitude of available

tools that supports Agile methods, we decided to eval-

uate directly which tools are most looked for using

Web Search media, most mentioned and most appre-

ciated through Social media.

Nowadays, Web 2.0 services such as blogs,

tweets, forums, chats, email etc. are widely used as

media for communication, with great results. Re-

search has established that software engineers use

Twitter, one of the most popular social network, in

their work to communicate about software engineer-

ing topics. Through use of social media services,

team members have opportunities to acquire more de-

tailed information about their peers’ expertise (Dessai

et al., 2012).

The popularity of Twitter is attracting the atten-

tion of researchers. Several recent studies examined

Twitter from different point of views, including the

sentiment prediction power (Tumasjan et al., 2010),

the topological characteristics of Twitter (Kwak et al.,

2010) or tweets as social sensors of real-time events

(Sakaki et al., 2010). Another stream of research

focuses on corporate applications of microblogging

such as the company internal use for project manage-

ment (B

ohringer and Richter, 2009) or the analysis

of Twitter as electronic word of mouth in the area of

168

Matta M. and Marchesi M..

Understanding Approval Rating of Agile Project Management Tools using Twitter.

DOI: 10.5220/0005557201680173

In Proceedings of the 10th International Conference on Software Engineering and Applications (ICSOFT-EA-2015), pages 168-173

ISBN: 978-989-758-114-4

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

product marketing (Jansen et al., 2009). Our aim is

to evaluate the power of Twitter in order to under-

stand the level of popularity and appreciation of Ag-

ile Project Management tools. We decided to apply

automated Sentiment Analysis on shared short mes-

sages of users on Twitter in order to analyze auto-

matically people’s opinions, sentiments, evaluations

and attitudes. We compared our results with one of

the most used web search media, Google Trends, and

with the related results of published surveys (Ver-

sionOne, 2014) (Azizyan et al., 2011) (Mishra and

Mishra, 2013).

The body of this paper is organized in ﬁve major

sections. Section 2, describes the background, section

3 presents the research steps of our study and section

4 summarizes and discusses our results. Finally, Sec-

tions 5 presents the conclusions and suggestions for

future work.

2 BACKGROUND

We analyzed the existing literature and surveys on

tool usage in the context of Agile development, look-

ing also for ”common” sources such as websites,

white-papers and published surveys. In the last years,

two relevant surveys concerning Agile Project Man-

agement tools have been published.

The ﬁrst is a study conducted by the tool vendor

VersionOne in 2013. This is the latest of a series of

similar surveys made every year by VersionOne, that

is also a tool producer. Most of the survey is fo-

cused on the state of Agile development itself (Ver-

sionOne, 2014). A total of 3500 responses were

collected, analyzed and described in a summary re-

port. The survey is very detailed, including infor-

mation such as reasons for adopting Agile methods,

resulting beneﬁts, roles and so on. They declare

that the most commonly used tools still are standard

ofﬁce productivity tools ,such as Excel (66%) fol-

lowed by tools like Microsoft Project (48%), Ver-

sionOne (41%), Atlassian/Jira (36%), Microsoft TFS

(26%), IBM ClearCase (10%), LeanKit (5%), Xplan-

ner (4%) and Trello (outside the survey choices). In

addition to tool usage, the respondents were asked

whether they would recommend the tools they are

using based on their past or present use. Versio-

neOne had the highest satisfaction of any other tool

evaluated (93%), followed by Atlassian/Jira (87%),

LeanKit (84%), TargetProcess (83%), Microsoft TFS

(79%) and ThoughtWorks Mingle (69%).

The other relevant survey is a study conducted in

2011(Azizyan et al., 2011). It is an Agile Project

Management Tool survey with 121 answers from 35

countries. The survey reports also features, a list of fa-

vored tool types and a list of tools that the respondents

felt most satisﬁed with. This survey reports that the

most commonly used tool within the companies stud-

ied is the physical wall (26%) followed by Microsoft

Project (8%), Rally (5%), Mingle (3%), VersionOne

(2%), JIRA (2%) and Team Foundation Server (2%).

In order to verify the accuracy of the informa-

tion reported on the cited surveys, we decided to

check these data through a public web service, called

”Google Trends”

. It is a feature of the search engine

that illustrates how frequently a ﬁxed search term was

looked for using Google. Using Google Trends you

can compare up to ﬁve topics at one time to view rel-

ative popularity, allowing you to gain an understand-

ing of the hottest search trends of the moment, along

with those developing in popularity over time. Fol-

lowing this kind of approach and taking into consid-

eration the list of previously mentioned tools, we are

able to evaluate which ones are the most looked for

using Google’s search engine.

Figure 1 shows the top ﬁve of the most mentioned

Agile Project Management tools in Google Trends.

We found that, nowadays, the most searched tools are,

in descending order of search, Jira, Trello, Mingle,

Microsoft Project and Team Foundation Server. This

graph shows also that, in the time period from 2004 to

today, Microsoft Project has been losing popularity in

favor of a more and more increasing fame of Jira and

Trello.

Figure 1: Most mentioned Agile Project Management tools

in Google Trends.

3 METHODOLOGY

3.1 Sentiment Analysis and Twitter

Besides analyzing which are the most searched tools,

we wanted to examine what customers really think

about them. In the last years, the social web has been

commercially exploited for goals such as automati-

cally extracting customer opinions about products or

brands, to ﬁnd which aspects are liked and which are

disliked (Thelwall et al., 2011).

http://www.google.it/trends/

UnderstandingApprovalRatingofAgileProjectManagementToolsusingTwitter

169

Twitter is an online social networking website and

microblogging service that allows users to post and

read text-based messages of up to 140 characters,

known as ”tweets”. It seems to be used to share in-

formation and to describe minor daily activities (Java

et al., 2007). The short format of tweet is a deﬁned

characteristic of the service, allowing informal col-

laboration and quick information sharing. Black et

al. presented a survey conducted to collect informa-

tion on social media use in global software systems

development. Twitter was found to be the most popu-

lar media and respondents afﬁrmed that speciﬁcation,

source codes and design information were shared over

social media (Black et al., 2010). Singer et al. showed

that Twitter helps developers keep up with the fast-

paced development landscape. They use it to stay

aware of industry changes, for learning and for build-

ing work relationships (Singer et al., 2014). Romero

et al. showed that the aspect responsible for the pop-

ularity of certain topics is the inﬂuence of users of

the network on the spread of content. Some mem-

bers produce content that resonates very strongly with

their followers thus causing the content to propagate

and gain popularity (Romero et al., 2011).

So, the tweets sometimes express opinions about

different topics, and for this reason we decided to

evaluate how much users speak about Agile Project

Management tools. In order to evaluate if a user re-

ally appreciates the tool, we tried to predict the sen-

timent analyzing the collection of tweets. By recent

years, there is a wide collection of research surround-

ing machine learning techniques, in order to extract

and identify subjective information in texts. This area

is known as sentiment analysis or opinion mining.

The research ﬁeld of sentiment analysis has devel-

oped many algorithms to identify if the opinion ex-

pressed is positive or negative (Pang and Lee, 2008).

The strength of the sentiment analysis applied to the

Twitter domain by applying similar machine learning

techniques to classifying the sentiment of tweets (Go

et al., 2009).

For these reasons, we chose to use automated sen-

timent analysis techniques to identify the sentiments

of tweets regarding Agile Project Management tools.

Since the goal of this research is neither to develop

a new sentiment analysis nor to improve an existing

one, we to used ”SentiStrenght”, a tool developed by a

team of researchers in the UK (Thelwall et al., ). This

tool was implemented to analyze informal short mes-

sages. Based on the formal evaluation of this system

on a large sample of comments from MySpace.com,

the accuracy of predicting positive and negative emo-

tions was something similar to that of other systems

(72.8% for negative emotions and 60.6% for positive

emotions, based on a scale of 1-5), and compared to

other methods, SentiStrenght showed the highest cor-

relation with human coders (Thelwall et al., 2010).

The tool SentiStrenght is able to assess each mes-

sage separately and, at the end, it returns one singular

value: a positive (1), a negative (-1) or a neutral sen-

timent (0).

3.2 Data Collection

Starting from the list of the most used Agile Project

Management tools found in the existing literature, we

evaluated how many people speak on Twitter about

these tools, and what they think about them. The

tweets are available and are easily retrieved mak-

ing use of Twitter Application Programming Interface

(API). Composing the hashtags # or @ with the name

of the tool, we can see all the tweets that mentioned

the analyzed system. The system architecture consists

of four components:

• Twitter Streaming API: it provides access to Twit-

ter data, both public and protected, on a nearly

real-time basis. A persistent connection is cre-

ated between our system and Twitter. As soon as

tweets come in, Twitter notiﬁes our system in real

time, allowing us to store them into our database.

• DataStore: our datastore consists of a back-end

database engine, using MySQL as RDBMS, that

repeatedly saves the incoming tweets from the

Twitter Streaming API.

• SentiStrenght Tool

• Java Module: this component allows us to send

automated requests to Twitter Streaming API, to

recover new tweets about analyzed tools, to parse

data gained and to store them into our datas-

tore. In a later stage, these data are sent to Sen-

tiStrenght tool in order to evaluate automatically

the users opinion.

We analyzed a collection of tweets posted on Twit-

ter between September 2014 and March 2015 regard-

ing the Agile project management tools most men-

tioned. We found a total of 84837 tweets. We then

used the SentiStrenght tool to evaluate the comments

extracted from Twitter. Given as input all tweets of

every tool previously mentioned, a score for each

comment was assigned.

• 1 if the comment is positive

• -1 if the comment is negative

• 0 if the comment is neutral

ICSOFT-EA2015-10thInternationalConferenceonSoftwareEngineeringandApplications

170

Table 1: Most quoted Agile Project Management tools or-

dered by number of tweets.

Number of Tweets

Trello 32613

Jira 21903

VersionOne 7887

Microsoft TFS 6730

Rally 5684

Mingle 4629

Microsoft Project 2249

LeanKit 1595

TargetProcess 1547

4 RESULTS AND DISCUSSION

4.1 Number of Tweets Analysis

Of the total 84837 tweets, 39756 were neutral (46%),

35409 were positive (42%) and 9672 were negative

(11%). In Table 1, the most quoted Agile Project

Management tools, ordered by number of tweets, are

shown. XPlanner and IBM ClearCase were excluded

since few comments were found about them.

Going back to the results of Google Trends about

the Agile Project Management tools reported in Fig-

ure 1, we observed a striking similarity between it

and the number of tweets we found. In fact, Google

Trends reported, in order of search, Jira, Trello, Min-

gle, Microsoft Project and Team Foundation Server.

Taking a look to Table 1, we can see that the rank

is similar, with the exception of VersionOne. So,

we found that there is a relation between the num-

ber of tweets posted for each tool and the Google

searches about it. TargetProcess turns out to be the

least tweeted, maybe due to the fact that it is a re-

cent tool and it is still little known worldwide. Con-

sequently, it is possible to stress that the most used

and quoted Agile Project Management tools on Twit-

ter and Google are Jira, Trello and VersionOne.

4.2 Comparison Between Negative and

Positive Tweets

From the list of tweets, we evaluated the sentiment

using SentiStrenght tool, in order to understand if the

users really appreciated these Agile Project Manage-

ment tools. Among all tweets, we observed that, on

average, 40% of the tweets don’t represent a senti-

ment, or Sentistrenght is not able to identify it. After

a careful analysis, it was observed that a lot of tweets

are neutral because often people wrote texts asking

help, non-expressive comments, tweets in a different

Table 2: Comparison between negative and positive tweets.

Tool

Negative Positive

Total % Total %

Trello 2760 8 % 16569 51 %

Jira 3445 16 % 7043 32 %

VersionOne 808 10 % 2612 33 %

MicrosoftTFS 859 13 % 2290 34 %

Rally 673 12 % 2612 46 %

Mingle 554 12 % 1971 43 %

MicrosoftProj 247 11 % 1055 47 %

LeanKit 89 6 % 829 52 %

TargetProcess 237 15 % 428 28 %

language than English, or simple links that lead to

other web pages.

An Agile Project Management tool that has more

positive than negative tweets is likely to be success-

ful. After this evaluation, we determined that, for all

tools, there were more positive messages than neg-

ative ones, and that positive messages were almost

3 times more likely to be forwarded than negative

messages. In Table 2 we show the data found about

positive and negative tweets using SentiStrenght. We

observed that, in general, the percentage of negative

tweets found in each tool is lower than 14%; Jira,

however, achieved 16% of negative comments. In

fact, users of Jira sometimes posted tweets in which

they explained their problems using the tool.

Based on this result, we found that Trello is clearly

the most appreciated tool, getting a high percentage

of positive comments (51%) and only 8% of nega-

tive comments. Beyond this tool, the other tools with

a high percentage of positive comments are LeanKit

(52%) and Microsoft Project (47%).

To better quantify the sentiments, we deﬁned the

score PNRatio as the ratio of positive versus negative

tweets on each tool.

PNRatio =

|Tweets with Positive Sentiment|

|Tweets with Negative Sentiment|

(1)

The indicator of eq.1 was applied to all tools and

Table 3 shows the results. We compared our re-

sults to the satisfaction achievement obtained by Ver-

sionOne survey, where it is possible. In our study the

most quoted (excluding Trello and Rally, since they

were not included in VersionOne survey) are LeanKit,

Microsoft Project and VersionOne. We noticed that

LeanKit and VersionOne also exhibit a high percent-

age of satisfaction from the survey, greater than 84%.

Nevertheless, Trello turns out to be one of the most

popular tool, showing a ratio P/N of 6.00, with a total

of 16569 positive comments.

In Table 3 we observed a relation between PN ra-

tio and its satisfaction, for most tools. As a matter of

UnderstandingApprovalRatingofAgileProjectManagementToolsusingTwitter

171

Table 3: Comparison between PN ratio of eq.1 and Ver-

sionOne satisfaction survey results by tools.

Tool Name Ratio P/N (VersionOne, 2014)

LeanKit 9.31 84%

Trello 6.00 -

Microsoft Project 4.27 53%

Rally 3.88 -

VersionOne 3.23 93%

Mingle 3.55 69%

Microsoft TFS 2.66 79%

Jira 2.04 87%

TargetProcess 1.80 83%

fact, the majority of the tweets represents a positive

sentiment while the negative comments are less than

16% and PN Ratio is always greater than 2. Also in

this case, we can conﬁrm that the favorite tools are

Trello and VersionOne.

4.3 Relationship with Agile

Methodologies and Approaches

We decided to analyze all tweets in order to assess

whether some Agile methodology was mentioned by

someone and, in the case, which one of these and

in which tool. So, for each tweet, we evaluated

if the user cited one of the major agile method-

ologies and approaches. We chose to test Scrum,

Kanban approach, Lean and eXtreme Programming

(XP) (Schwaber, 2004) (Mary and Tom, 2003). Ta-

ble 4 shows the most cited methodologies within the

tweets; we can notice that the methodology most

mentioned is Scrum, followed by Kanban, Lean and

ﬁnally eXtreme Programming. About Scrum, we

found that the main contribution is given by Ver-

sionOne, Rally, Jira and Trello. On the other hand

Trello, Jira and LeanKit quoted Kanban approach.

Table 4: Most mentioned Agile metodologies in the tweets.

Methodology Tot tweets Tool contribution

Scrum

2021 VersionOne 53%

Rally 20%

Jira 10%

Trello 6%

Kanban

807 LeanKit 34%

Trello 40%

Jira 12%

Lean

439 VersionOne 30%

LeanKit 21%

eXtreme

111 VersionOne 42%

Jira 27%

Programming Trello 14%

Figure 2: Most mentioned Agile metodologies in Google

Trends.

We checked these results using Google Trends, to

conﬁrm the ranking found using Twitter. Figure 2

shows the results. Scrum is clearly the most searched

word, so we can say that it’s the most popular method-

ology, and its fame is still growing. The popularity

of Lean and Kanban are quite similar while eXtreme

Programming, over the years, lost most of its fame.

In the end, Google Trends conﬁrmed the rankings re-

lated to Agile methodologies and approaches found

using Twitter. It afﬁrms the growing popularity of

Scrum in the last years, and the effectiveness of us-

ing tweets to assess the popularity of something.

5 CONCLUSION

This position paper presents an analysis of a set of

tweets, starting from the results of different surveys

about Agile project management tools most used by

companies. We analyzed a total amount of 84837

tweets about this kind of tools, covering a period be-

tween September 2014 and March 2015. We applied

automated Sentiment Analysis on tweets, in order to

assess whether the opinions of the users about the ex-

amined tool were positive or negative. The aim of this

work was to understand which tools are most men-

tioned and looked for, and, if those who use it re-

ally appreciate it. Using Twitter, it was possible to

recognize the most mentioned tools, and to evaluate

their level of appreciation. This approach is able to

give immediate results, reducing the need to submit

surveys to users. We can conclude that Jira, Trello

and VersionOne are the most twitted and, at the same

time, the most appreciated Agile Project Management

tools. Google Trends conﬁrms that the fame of Jira

and Trello is ever increasing in these years.

In future we plan to extend the increase the num-

ber of analyzed tweets, and to perform a complete

correlation analysis of the results. We also plan to

gather tweets along a much longer time interval, per-

forming trend analysis on the tweets. In this way, we

could assign a speciﬁc weight to each tweet and check

whether the results remain unchanged with the addi-

tion of this variable.

ICSOFT-EA2015-10thInternationalConferenceonSoftwareEngineeringandApplications

172

REFERENCES

Azizyan, G., Magarian, M. K., and Kajko-Matsson, M.

(2011). Survey of agile tool usage and needs. In Agile

Conference (AGILE), 2011, pages 29–38. IEEE.

Beck, K., Beedle, M., Van Bennekum, A., Cockburn, A.,

Cunningham, W., Fowler, M., Grenning, J., High-

smith, J., Hunt, A., Jeffries, R., et al. (2001). Man-

ifesto for agile software development.

Black, S., Harrison, R., and Baldwin, M. (2010). A survey

of social media use in software systems development.

In Proceedings of the 1st Workshop on Web 2.0 for

Software Engineering, pages 1–5. ACM.

ohringer, M. and Richter, A. (2009). Adopting social soft-

ware to the intranet: a case study on enterprise mi-

croblogging. In Proceedings of the 9th Mensch &

Computer Conference, pages 293–302.

Dessai, K. G., Kamat, M. S., and Wagh, R. (2012). Appli-

cation of social media for tracking knowledge in agile

software projects. Available at SSRN 2018845.

Fortune, J., White, D., Jugdev, K., and Walker, D. (2011).

Looking again at current practice in project manage-

ment. International Journal of Managing Projects in

Business, 4(4):553–572.

Go, A., Bhayani, R., and Huang, L. (2009). Twitter senti-

ment classiﬁcation using distant supervision. CS224N

Project Report, Stanford, pages 1–12.

Heagney, J. (2011). Fundamentals of project management.

AMACOM Div American Mgmt Assn.

Jansen, B. J., Zhang, M., Sobel, K., and Chowdury, A.

(2009). Twitter power: Tweets as electronic word of

mouth. Journal of the American society for informa-

tion science and technology, 60(11):2169–2188.

Java, A., Song, X., Finin, T., and Tseng, B. (2007). Why we

twitter: understanding microblogging usage and com-

munities. In Proceedings of the 9th WebKDD and 1st

SNA-KDD 2007 workshop on Web mining and social

network analysis, pages 56–65. ACM.

Kwak, H., Lee, C., Park, H., and Moon, S. (2010). What is

twitter, a social network or a news media? In Proceed-

ings of the 19th International Conference on World

Wide Web, WWW ’10, pages 591–600, New York,

NY, USA. ACM.

Mary, P. and Tom, P. (2003). Lean software development:

An agile toolkit for software development managers.

Lean Software Development: An Agile Toolkit for

Software Development Managers.

Mishra, A. and Mishra, D. (2013). Software project man-

agement tools: a brief comparative view. ACM SIG-

SOFT Software Engineering Notes, 38(3):1–4.

Pang, B. and Lee, L. (2008). Opinion mining and senti-

ment analysis. Foundations and trends in information

retrieval, 2(1-2):1–135.

Romero, D. M., Galuba, W., Asur, S., and Huberman,

B. A. (2011). Inﬂuence and passivity in social media.

In Proceedings of the 20th International Conference

Companion on World Wide Web, WWW ’11, pages

113–114, New York, NY, USA. ACM.

Sakaki, T., Okazaki, M., and Matsuo, Y. (2010). Earth-

quake shakes twitter users: Real-time event detection

by social sensors. In Proceedings of the 19th Inter-

national Conference on World Wide Web, WWW ’10,

pages 851–860, New York, NY, USA. ACM.

Schwaber, K. (2004). Agile project management with

Scrum. Microsoft Press.

Singer, L., Figueira Filho, F., and Storey, M.-A. (2014).

Software engineering at the speed of light: How de-

velopers stay current using twitter. In Proceedings of

the 36th International Conference on Software Engi-

neering, pages 211–221. ACM.

Thelwall, M., Buckley, K., and Paltoglou, G. Sentistrength,

2011. available online.

Thelwall, M., Buckley, K., and Paltoglou, G. (2011).

Sentiment in twitter events. Journal of the Ameri-

can Society for Information Science and Technology,

62(2):406–418.

Thelwall, M., Buckley, K., Paltoglou, G., Cai, D., and Kap-

pas, A. (2010). Sentiment strength detection in short

informal text. Journal of the American Society for

Information Science and Technology, 61(12):2544–

2558.

Tumasjan, A., Sprenger, T. O., Sandner, P. G., and Welpe,

I. M. (2010). Predicting elections with twitter:

What 140 characters reveal about political sentiment.

ICWSM, 10:178–185.

VersionOne (2014). Inc. 8th annual state of agile develop-

ment survey. http://www.versionone.com/pdf/2013-

state-of-agile-survey.pdf. [Online; accessed 8-April-

2015].

UnderstandingApprovalRatingofAgileProjectManagementToolsusingTwitter

173