Distributed Crowd-based Annotation of Soccer Games using Mobile

Devices

Bruno Barros

, Carlos Serrão

and Rui Lopes

Instituto Universitário de Lisboa (ISCTE-IUL), Av. das Forças Armadas, 1649-026 Lisboa, Portugal

Information Sciences, Technologies and Architecture Research Centre (ISTAR-IUL), (ISCTE-IUL), Lisboa, Portugal

Instituto de Telecomunicações (IT), (ISCTE-IUL), Lisboa, Portugal

Keywords: Soccer, Annotation, Mobile Application, Distributed Crowd-Sourcing.

Abstract: Soccer is one of the most loved sports in the world. Millions of people either follow the sport or are actually

involved in its practice. Soccer also moves huge financial amounts every year and therefore teams always

thrive to be better than the competition. New technologies have become a common place both in the

preparation of the games and on the analysis of the games after they are concluded. In this paper, the authors

will present a developed system, based on the usage of distributed mobile devices, that will enable the

annotation of soccer matches, either in real time or after the matched is concluded (through the observation

of other media). The capture of relevant events in the game can be used to better analyse the game and the

performance of individual players fostering improvements and better decisions in the future. The application

is implemented in the Android platform so that it can be easily installed by typical soccer fans empowering

them as match annotators. This crowd of annotators, although not experts, can collectively provide a robust

and rich annotation for soccer matches.

1 INTRODUCTION

Information technology is present in many aspects of

our live. Sports are a very special case of technology

application where mobile smart devices can play an

important role. On the individual and personal side, it

is more and more common to support the practice of

a given specific sport activity, such as running,

walking or swimming, on the data collected by smart

devices, equipped with several sensors. Smartphones

and smartwatches are charged with multiple motion

(e.g., accelerometers, altitude), GPS/location and

heartbeat sensors that track their users sport activities

and produce detailed reports about their performance.

This is a common trend that can be easily confirmed

simply by looking to every person conducting some

type of exercise.

Although these personal devices are adequate to

capture individual performance, they lack the

capability of analyzing collective sports such as

soccer, or better, they are able to determine the

individual physical performance of the players but

they are unable to capture their actions in the match

field. In order to acquire the capability of capturing

the players actions and analyze a soccer match as a

whole, annotation tools are often used.

Moreover, these annotation tools are often used by

soccer clubs that have the capability of making large

investments in specialized hardware and software

(e.g., GPS tracking (STATsports, 2018)) and

dedicated team members to annotate the games

(either through automated or manual video analysis,

e.g., data streams (Liu et al., 2013; OPTAsports,

2018)) and extract information from them. However,

teams without the necessary financial resources to

acquire such systems do not have the opportunity to

annotate their games and thus extract the required

knowledge to improve their players' actions on the

field and improve as a team. A particular example of

this are young age soccer teams, in particular those

involved in the development/formation of young

players, where both team performance and young

talent scouting can capitalize from the annotation of

team and players' performance (Fernandez-Rio and

Méndez-Giménez 2014; Müller, Simons, and

Weinmann 2017; Pastor-Vicedo, Contreras-Jordán,

and Prieto-Ayuso 2017).

On the other hand, using large communities (i.e.,

crowd-sourcing (Howe, 2006; Zhao and Zhu, 2014)

for solving large complex tasks has proven to be a

valuable source of data. This concept has already

successfully been applied in the real of sport events

Barros, B., Serrão, C. and Lopes, R.

Distributed Crowd-based Annotation of Soccer Games using Mobile Devices.

DOI: 10.5220/0006927000400048

In Proceedings of the 6th International Congress on Sport Sciences Research and Technology Support (icSPORTS 2018), pages 40-48

ISBN: 978-989-758-325-4

and media (Perin, et al., 2013; Sulser, et al., 2014). In

the particular case of young players' teams there is a

particular interest on their progress and achievements

from parents and other relatives, which usually attend

to their matches and are thus a significant, and cheap,

resource pool (crowd) of annotators (Fernandez-Rio

and Méndez-Giménez, 2014; Pastor-Vicedo, et al.,

2017). That is, using the potential of distributed and

mobile annotation applications can help young player

team managers/coaches to develop training processes

to improve the individual skills of each individual

player and the team as an all. Additionally, the need

to have specific team members to annotate the games

is mitigated, due to the fact that any of the users of the

distributed mobile annotation application, can

annotate the games - for instance, parents of the

players can annotate the games, contributing to the

collection of game events for the team or club.

In this paper, a tool for the distributed annotation

of soccer matches is presented and described, based

on the usage of mobile devices, such as smartphones

or tablets. As first section of this paper, an

introduction to the usage of technologies in sports and

of the different approaches that are used for

annotation on soccer matches is provided,

highlighting their major advantages and

disadvantages. In the second section of the paper the

description of the distributed soccer match annotation

tool is provided, with information regarding the

different choices that were made concerning the

design of the mobile application and the implemen-

tation of the distributed annotation algorithm.

Following this, the description of a specific soccer

match annotation use-case is presented as a way to

evaluate the developed solution. Finally, the last

section presents the conclusions of this work and

draws the directions for some future work.

2 OVERVIEW OF SOCCER

MATCH ANNOTATION

SOLUTIONS

Annotation of soccer matches is an important tool for

the soccer game intelligence collection, both in terms

of the evaluation of the individual players and the

overall team performance. There are already

numerous solutions that are used for game annotation.

In this analysis, it is possible to categorize the existing

tools according to three different axes: focus,

collection and context. Focus is concerned about the

object where the annotation events are collected

(team or individual player). Collection refers to the

type of data collection being made, and it can be

manual or automated. Context refers to where the data

is collected, if the collection is made in real time, from

the field, or from some pre-recorded video.

For the sake of the analysis presented here, we

have considered just one dimension - collection. For

this dimension it is necessary to consider systems that

collected data from games manually or automatically.

2.1 Manual Annotation Solutions

Opta is a commercial system that allows the manual

collection of events from a soccer match

(OPTAsports, 2018). It uses is own computer

application and requires specialized trained operators

to manually collect the events from a video live feed

or from recorded video feed. It requires two

annotators, one for each of the teams. Opta is a

professional solution that produces statistics that are

afterwards analysed by professional soccer leagues

(Liu et al., 2013).

Another interesting solution is SAP Sports One

for Football (SAP News Center, 2017), although not

targeting specifically the annotation of games. This

solution is designed for football clubs and

associations and is capable of handling different types

of data, such as match statistics, player fitness data,

information about injuries, medication, and recovery,

training data, match analyses, and scouting.

A popular application is Longomatch

(Longomatch, 2011). Longomatch is also a system for

manual collection of soccer match events, mostly

based on either recorded or live video. It also allows

the edition and annotation of videos to prepare

specific trainings or instruct the players about some

of the movements they should conduct on the field

during the game.

In the mobile realm, the Muithu touch system

(Stenhaug et al., 2014) allows to annotate sport events

using mobile devices. The different interaction

screens are organized in a hierarchical way where, by

selection and dragging, players can be associated to

their actions.

2.2 Automated Annotation Solutions

There are currently several automated soccer match

annotation solutions, all of them targeting

professional soccer teams, mostly because of their

costs. Most of these solutions involve the analysis of

images captured from the field. An example of such

is the Bagadus sports analytics application that uses

soccer as a case study (Halvorsen et al., 2013). In the

Bagadus system, annotations provided via the Muithu

Distributed Crowd-based Annotation of Soccer Games using Mobile Devices

touch system, are combined with the result of a sports

analytics module that processes data from on a video

processing system using a video camera array.

(Pettersen et al., 2014) (the ZXY Sport Tracking

system).

(Duh et al., 2013) also propose an automatic

process to analyse soccer matches broadcast video for

the detection and tracking of players based on the

automatic classification of the video into different

scenes based on 2-D Gaussian colour model of hue

and saturation, and the analysis of the player actions.

A framework for semantic annotation of soccer

videos that exploits an ontology model referred to as

Dynamic Pictorially Enriched Ontology is proposed

by (Ballan et al., 2010). In this framework, visual

instances are used as matching references for the

visual descriptors of the entities to be annotated on

the match.

Interplay (Interplay-sports., 2017) is a private

company that offer a set of products that are also used

to conduct automated analysis of live and video from

different types of sports - soccer included.

3 REQUIREMENTS FOR A

CROWD-SOURCE

ANNOTATION APPLICATION

From the analysis conducted in the previous section

of this paper, it was possible to conclude that most of

well-known and financial powerful soccer clubs in

the World are already incorporating IT on their

players training processes, both to capture data about

their performance, but also to discover ways on how

to improve their results. Also, there are currently

different soccer matches annotations solutions that

are a mix of expensive and perfectly calibrated

hardware on the field with proprietary software

analysis tools. There are others that extremely

personal intensive, demanding a permanent team to

capture data from recorded videos in order to analyse

the soccer matches. These are solutions that are not

accessible to most soccer clubs and are completely off

limits in the specific case of young players formation

and training.

Therefore, current solutions are not adequate for

the purpose referred above and different solutions

need to be used. In this specific case, inexpensive and

lightweight match annotation solutions are required

to enable smaller soccer teams/clubs and young

players formation to use low-cost technology, such as

smartphones and tablets, to annotate live or pre-

recorded matches, in a distributed manner.

Annotation can be performed not only by the team

staff, but also by other external elements. For

instance, team fans while watching the game can

contribute to the annotation of the game using their

own mobile devices (Zhao and Zhu, 2014). Also,

family members of the young players can also

annotate the games where their relatives participate.

The solution presented in this paper is based on

using distributed mobile devices to annotate soccer

games. Multiple users can annotate the same game,

and some users can annotate one of the teams, while

other users can annotate the adversary. This way, it

will be possible to collect a higher number of game

events with a richer detail. This information is

collected, stored post processed. The post processing

of the collected data can be used to conduct a set of

actions:

• Analyse the overall team performance and

individual performance of a given player -

identify the team behaviour in different moments

and aspects of the game, in particular, comparing

that performance with the adversary opposition;

• Create game statistics - offer a statistical

treatment of all the data collected to provide

better ways to analyse and visualize the game;

• Improve training processes - identify which are

the specific needs in terms of training to help the

coach to design better precise training techniques

focusing on the group or on individual players;

• Discover new talents - finally, the solution can

also be used to identify new and talented players

(scouting).

4 IMPLEMENTATION OF A

CROWD-BASED DISTRIBUTED

SOCCER GAME ANNOTATION

SYSTEM

As a way to achieve the overall objectives of an

effective distributed game solution the authors opted

by designing a system that would take advantage of

the extreme mobility presented by the nowadays end-

user devices and developed a mobile application (for

the Android platform) that would provide the

annotation capabilities required by a soccer match.

The system is composed by a set of different

components that enable data collection at the end-user

side and posterior storage at a backend system. Figure

1 displays the different architectural components of

the annotation system developed.

icSPORTS 2018 - 6th International Congress on Sport Sciences Research and Technology Support

Figure 1: Architecture overview of the distributed

annotation system.

The fact that the system is designed deliberately

taking into consideration multiple, simultaneous

annotations (from the crowd), distinguishes it from

other mobile solutions.

The system is based on a client-server

architecture. On the client-side, the users will interact

with the mobile application, which will present an

interface that will adapt according to the different

conditions of the game. The mobile application uses

a REST API that is implemented and the server side,

and that allows the application to retrieve all the

information it needs about the soccer match and the

different players information (the REST API was

developed using PHP). Moreover, this API will

enable the mobile application to communicate the

annotations captured by the user on a specific soccer

match. All of this information is stored on a database

(a MySQL database).

Another important function on the server is the

ability of aggregating all the different annotation

events captured by different users (different users

may be annotating the same match and the same

team), creating an integrated and consolidated "view"

on the data of the events occurred during the game.

This consolidated view is also stored on the database

for further statistical analysis, that will provide the

necessary feedback to improve the players and

training performance.

4.1 The Data Collection Mobile

Application

The application was developed for Android

smartphones and tablets. Special focus was placed on

the design and responsiveness of the user-interface

since it is one of the most critical components of the

system. Simplicity was thus one of the major

concerns on the design of the interface. A proper-

designed user interface will allow the human

annotator to capture more and better data about the

soccer match events (which is not an easy task, taking

into consideration the event speed occurrence on

some matches).

As it is possible to notice in Figure 2, the mobile

application presents on the initial screen, the different

elements necessary for the user to start taking

annotations. On the beginning of the game, the

annotator selects the team that he will be annotating,

and it is possible for other users to select the same

team or the opponent team.

Figure 2: Initial screen of the annotation application.

As soon as the user starts annotating the events on

the game, the mobile application user interface will

change and adapt to the new event that needs to be

annotated. For instance, Figure 3 displays the screen

that is presented to the user when a player attempts a

shot on goal. In this screen it is presented all the

options that may occur on that situation, including the

direction of the shot towards the goal.

Figure 3: Shot screen displaying the different annotation

events.

Distributed Crowd-based Annotation of Soccer Games using Mobile Devices

This sequence of screens, resulting from a

sequence of events occurring on a soccer match were

defined and implemented on the annotation state

machine, presented on the next section, that allows

the user, to naturally capture the different events that

occur inside the field. This way, the usage of a

distributed system, will allow the collection of larger

number of events for both of the opposing teams, that

will be afterwards sent to the server and stored, and

aggregated and consolidated to provide an integrated

and faithful textual annotated view of the game. All

of this data collected will be used posteriorly to

produce detailed statistical information about the

team and individual players, allowing the trainers and

the players themselves to work towards the

improvement of their performance and development.

4.2 Annotation State Machine

One of the concerns that the team had during the

development of the system, was to adapt it to be as

simple as possible for the user to annotate the games.

A state machine was created to mimic the most

common situations of a soccer match to help the

annotator to capture a larger number of events. It was

possible to identify from other systems and from the

existing literature, which were the most relevant

events on a football match (Carling, Reilly, and

Williams 2007). In order to simplify the process, each

annotator only follows a specific team so that the user

interface only details the specifics of that team on a

specific situation. The most basic part of this state

machine refers to the events occurring between the

start and the end of the match (Figure 4). The focus

on simplicity of interaction lead us to three design

options that differentiate our system: i) the annotation

state machine is based on atomic events/actions (e.g.,

player A has the ball), ii) only these atomic events are

stored in the database, iii) non-atomic events (e.g., a

pass from player A to B) are obtained from atomic

events during the post-processing and not annotated

explicitly in the mobile device.

The first event to occur on a game is the "kick off"

event, so the first state is to show the user the “Kick

Off Screen”. After this, a specific player has the ball,

and pressing on the player button, will bring the

“Offensive Screen” to the user, where it will annotate

the event, by switching between different players,

until the game reaches halftime, where the “Halftime

Screen” is presented, or the game ends, bringing the

“Initial Screen”. During this annotation period, the

player may change roles, between offensive and

defensive actions, therefore switching between the

“Offensive” and “Defensive Screens”.

Figure 4: Basic soccer game state machine.

The state machine will result on a set of events

and screens that are presented to the users. The

following is a list of screens presented to the

annotators:

• Initial Screen: this screen is displayed when the

application starts or when a game ends;

• Kick Off Screen: the initial state, waiting for one

of the teams to kick off the game;

• Offensive Screen: this screen displays the team

while possessing the ball, to collect offensive

events;

• Halftime End Screen: the screen is displayed at

halftime and waits for a new kick off;

• Defensive Screen: the screen is displayed when

the team doesn't have the ball, used to collect the

defensive events;

• Goal Kick Screen: this screen collects all the

possible outcomes of a shot, necessary for the

annotator to select the type of goal kick that

occurred (shot off, goal kick, and others);

• Foul Screen: the screen displays the offensive

and defensive infractions (for instance, foul,

penalty, and others);

• Select Player Screen: displays all the players on

the field and allows to select a player that will

take an action, when the ball is stopped (corner

kicks, throw ins, fouls, penalties, and others);

• Select the Faulty Player Screen: displays all the

players on the field to allow the selection of the

player that as committed a foul;

icSPORTS 2018 - 6th International Congress on Sport Sciences Research and Technology Support

• Card Screen: displays all the players on the

screen that allows the registration of cards the

player has received (yellow or red cards).

In the presented screens, different events may

occur. The way for the annotator user to record such

events is through different buttons:

• Player Button: records the player that received

the ball - has possession of the ball;

• Halftime Start: registers the beginning of the

game or the beginning of the second half;

• Halftime End: signals the end of one of the game

halftimes;

• End of the Game: signals the end of the game;

• Own Goal: registers the player that scored an

own goal;

• Ball Loss: registers the player that has lost the

ball possession;

• Yellow (Red) Card: registers that a specific

player has received a yellow (red) card;

• Corner (Goal) Kick: registers that the team has

been awarded a corner (goal) kick;

• Throw in: records a throw in;

• Foul: brings the Foul Screen, selects the foul and

records if it is a defensive or offensive foul;

• Goal: registers the player that scored a goal,

awarding the ball possession to the opponent

team;

• Blocked Kick: records the player that had a kick

attempt blocked;

• Shot off: records the player that attempted a shot

to the goal and the ball goes off;

• Post (Bar) Shot: records the player that shoot the

ball to the post (bar);

• Foul Awarded: records the foul that was

committed on a player;

• Foul Committed: records the foul that was

committed by a player;

• Penalty Awarded: register the penalty that was

committed on a given player.

The state machine allows the modelling of all the

possible situations on the soccer match (Figure 5),

allowing the annotation user to capture a larger

number of events occurring on the game. The state

machine, implemented on the mobile application

allows the different appropriated annotation screens

to be presented to the end user, according to the

different events that occur on the game.

5 SOCCER MATCH

ANNOTATION USE-CASE

After the development of the distributed annotated

mobile application the system was tested with real

users in a controlled environment. Users were invited

to annotate a soccer match sequence, based on a

video, where all the relevant events have been

previously captured. The basic idea with these tests

was to evaluate how the users, using the mobile

application, were able to accurately identify the

events that were displayed on the match video

sequence.

Figure 5: State machine that describes the different

moments on a soccer match.

The video used on the tests with the users, with a

duration of around 10 minutes, was composed of

several other shorter video sequences containing

different types of events on a soccer match. The idea

of the tests was to simulate the real environment

conditions that an annotator user would find on a real

match. Therefore, the video was visualized

uninterruptedly, without the possibility of rewinding

it, and the events observed by the user on the video

were annotated, in real time, on the application. The

results from the tests conducted with several different

types of users are presented on Table 1, where que

Distributed Crowd-based Annotation of Soccer Games using Mobile Devices

number of different events that are presented by the

video to the testing user are recorded (E1), the

average number of recorded events by the different

users on the annotation application developed are

presented (E2) and the average number of correctly

identified and recorded match events are also

depicted (E3). On the table it is also presented the

accuracy level for each of the correct identified events

(AC). The tests were conducted on different users,

representing a different skillset on the usage of

mobile technologies (smartphones and tablets) and

mobile applications. Also, the users did not have any

previous contact with the developed application or

with any other existing soccer match annotation

technologies.

Table 1: Results of the tests conducted on the system to

evaluate how users were collecting the events on a sample

soccer match video. E1: Refers to the number of events that

actually exist on the sample video; E2: refers to the average

number of events annotated by the users testing the

application on the sample video; E3: refers to the average

number of events that were correctly identified by the users

testing the application; AC: refers to the accuracy of the

correct event identification by the users.

Events E1 E2 E3 AC

Kick-off 1 1 1 100%

Failed pass 16 13 10 63%

Throw-in awarded 6 5 5 83%

Passes 116 102 77 66%

Foul awarded 2 3 2 100%

Goal 3 3 3 100%

Blocked shot 5 2 2 40%

Ball recovery 24 21 19 79%

Committed foul 3 3 3 100%

Corner kick awarded 3 3 3 100%

Shot off 3 5 3 100%

Post shot 2 1 1 50%

Off-side 1 1 1 100%

Halftime end 1 1 1 100%

Ball possession lost 4 2 3 75%

Game end 1 1 1 100%

From the analysis of the results obtained and

presented on Table 1, the most obvious conclusion

that can be drawn is that there is a slight discrepancy

between the number of events actually existing in the

soccer match video and the number of events that

were annotated by the users. The average accuracy

percentage is 85%, allowing us to conclude that the

developed solution can indeed contribute to the

collection of a large number of events happening on

a soccer match. The obtained value can even be

improved if we consider the consolidation of the

annotated events by the different users that

independently can annotate the same match and the

same team (which is actually one of the

functionalities of the developed system).

Another interesting conclusion from the

evaluation and tests conducted on the developed

solution is that there are some events in the match

game that are more easily recorded by the user

annotators than others. For instance, failed passes,

blocked shots and post shots are among the most

difficult events for users to record on the application.

On the other side, kick-off, fouls, corner kicks, off-

sides, goals and game end are some of the easiest

events to record on the application. There is a factor

that contributes to this difference in the identification

of events - time. Most of the events with a higher

accuracy ratio are events that require the soccer match

to stop (or at least pause for some time) making the

task easier for the users, that have more time to record

such events.

These results allow us to conclude that the

developed solution, if we consider the aggregation of

the different users annotating the same game, can

contribute to an accurate annotation of a soccer

match. Considering that the tests were conducted on

a controlled environment, using a soccer match video

sequence and not the real match on the field, the

results may differ. However, it was important to

conduct these tests on a controlled environment to

have the opportunity to capture the opinion of the

users annotating the game, through interviews after

the tests, to learn the opinion of the user about the

system.

6 CONCLUSIONS AND FURTHER

WORK

The technology is more and more present in almost

every sport. With technology, it is possible to better

analyse the game and the players and find

opportunities to improve players performance. The

presented system is an example of how technology

can play an important role on the annotation of soccer

matches, as a way to discover new talented players,

study the behaviour of the players and team on the

field, and find individual or group improvement

opportunities.

The presented system was developed using a

client-server paradigm, where one of the most

important components of the system was a mobile

application capable of running on smartphones and

tablets and representing the major communication

interface with the user that plays an important role on

icSPORTS 2018 - 6th International Congress on Sport Sciences Research and Technology Support

the system - the annotation of soccer matches. Special

care was dedicated to the design of this mobile

application, because, in order to be efficient, it would

have to represent the different moments and events

that occur during a soccer match as best as possible.

The state machine that makes this possible is also

presented on this paper, where it is possible to

understand the different flow of screens (and the

corresponding user interface) appearing to the user

annotating the games, reacting to the different choices

the user makes on each individual screen. The authors

of this system opted by allowing each user to follow

only one of the teams on the match field (to maximize

efficiency) and also to allow multiple different users

to annotate the same team concurrently. This

concurrent annotation of the events on the soccer

match decisively contributes to a much complete

identification of all of them, through a posterior

aggregation and consolidation of all the collected

data.

The system was tested using different users, with

different mobile application usage backgrounds, and

without no knowledge of sports annotation tools.

From the evaluation conducted, it was possible to

conclude that the system is able to accurately allow

the collection of the soccer match events, although

some limitations were also identified. The tests

focused more on the individual annotation task of

each of the users, and therefore it was not yet possible

to conclude if the aggregation of data from the

different users could increase the event collection

accuracy levels.

The work described on this paper focused

primarily on the quality of the collection of data of

events occurring on a soccer match. The second stage

of this work, that was not presented here, relates to

the processing of the collected data to produce

accurate statistics about the team and individual

players performance. This statistical information will

be of great value for all the different stakeholders

involved in the soccer team, mainly for the team

managers and players, and will help the

democratization of the usage of advanced IT to help

smaller professional soccer teams, or even for young

players training teams.

There are still some open issues that will need to

be tackle while we continue the research and

implementation of this system. The first one refers to

the huge amount of data will be collected either by

this system (where a crowd-based approach is used)

or by other sensor-based systems (Rein and

Memmerte 2016). The second issue to consider refers

to the quality of the collected data, since most of the

annotators have no or little experience on soccer

match annotations (Hsueh, Melville, and Sindhwani

2009; Nowak and Rüger 2010).

ACKNOWLEDGEMENTS

The authors would like to acknowledge the FCT

Project UID/MULTI/4466/2016.

REFERENCES

Ballan, Lamberto, Marco Bertini, Alberto Del Bimbo, and

Giuseppe Serra. 2010. “Semantic Annotation of Soccer

Videos by Visual Instance Clustering and

Spatial/Temporal Reasoning in Ontologies.”

Multimedia Tools and Applications 48(2): 313–37.

https://doi.org/10.1007/s11042-009-0342-4.

Carling, Christopher, Thomas Reilly, and A Mark

Williams. 2007. Handbook of Soccer Match Analysis:

A Systematic Approach to Improving Performance.

Routledge.

Duh, Der-Jyh, Shu-Yu Chang, Shu-Yuan Chen, and Cheng-

Chung Kan. 2013. “Automatic Broadcast Soccer Video

Analysis, Player Detection, and Tracking Based on

Color Histogram.” In Intelligent Technologies and

Engineering Systems, , 123–30.

Fernandez-Rio, Javier, and Antonio Méndez-Giménez.

2014. “Talent Detection and Development in Soccer: A

Review.” Journal of Sport and Health Research 6(1):

7–18.

Halvorsen, Pål et al. 2013. “Bagadus: An Integrated System

for Arena Sports Analytics: A Soccer Case Study.” In

Proceedings of the 4th ACM Multimedia Systems

Conference, MMSys ’13, 48–59. http://doi.acm.org/

10.1145/2483977.2483982.

Howe, Jeff. 2006. “The Rise of Crowdsourcing.” Wired

Mag. 14(6): 1–4. https://www.wired.com/2006/06/

crowds/.

Hsueh, Pei-Yun, Prem Melville, and Vikas Sindhwani.

2009. “Data Quality from Crowdsourcing: A Study of

Annotation Selection Criteria.” In Proceedings of the

NAACL HLT 2009 Workshop on Active Learning for

Natural Language Processing, , 27–35. http://dl.acm.

org/citation.cfm?id=1564131.1564137.

Interplay-sports. 2017. “Interplay-Sports.” http://www.

interplay-sports.com (June 30, 2018).

Liu, Hongyou, Will Hopkins, A Miguel Gómez, and S

Javier Molinuevo. 2013. “Inter-Operator Reliability of

Live Football Match Statistics from OPTA Sportsdata.”

International Journal of Performance Analysis in Sport

13(3): 803–21.

Longomatch. 2011. “LongoMatch: Sports Video Analysis.”

http://www.longomatch.org/documentation/ (June 30,

2018).

Müller, Oliver, Alexander Simons, and Markus Weinmann.

2017. “Beyond Crowd Judgments: Data-Driven

Estimation of Market Value in Association Football.”

Distributed Crowd-based Annotation of Soccer Games using Mobile Devices

European Journal of Operational Research 263(2):

611–24. http://www.sciencedirect.com/science/article/

pii/S0377221717304332.

Nowak, Stefanie, and Stefan Rüger. 2010. “How Reliable

Are Annotations via Crowdsourcing: A Study About

Inter-Annotator Agreement for Multi-Label Image

Annotation.” In Proceedings of the International

Conference on Multimedia Information Retrieval,

557–66. http://doi.acm.org/10.1145/1743384.1743478.

OPTAsports. 2018. “OPTAsports Web Site.”

https://www.optasports.com (June 30, 2018).

Pastor-Vicedo, Juan Carlos, Onofre Contreras-Jordán, and

Alejandro Prieto-Ayuso. 2017. “Performance

Indicators as a Resource for the Selection of Talented

Football Players.” Journal of Human Sport and

Exercise 12(3proc): S797–806.

Perin, Charles, Romain Vuillemot, and Jean-Daniel Fekete.

2013. “Real-Time Crowdsourcing of Detailed Soccer

Data. What’s the Score?” In Proceedings of the 1st

Workshop on Sports Data Visualization, , 18–23.

Pettersen, Svein Arne et al. 2014. “Soccer Video and Player

Position Dataset.” In Proceedings of the 5th ACM

Multimedia Systems Conference, MMSys ’14, , 18–23.

http://home.ifi.uio.no/paalh/dataset/alfheim/.

Rein, Robert, and Daniel Memmerte. 2016. “Big Data and

Tactical Analysis in Elite Soccer: Future Challenges

and Opportunities for Sports Science.” SpringerPlus

5(1): 1410.

SAP News Center. 2017. “Get the Play-by-Play on SAP

Sports One.” https://www.sap.com/products/sports-

one.html (June 30, 2018).

STATsports. 2018. “STATsports Web Site.” http://stat

sports.com (June 30, 2018).

Stenhaug, Magnus, Yang Yang, Cathal Gurrin, and Dag

Johansen. 2014. “Muithu: A Touch-Based Annotation

Interface for Activity Logging in the Norwegian

Premier League.” In MultiMedia Modeling, , 365–68.

Sulser, Fabio, Ivan Giangreco, and Heiko Schuldt. 2014.

“Crowd-Based Semantic Event Detection and Video

Annotation for Sports Videos.” In Proceedings of the

International Workshop on Crowdsourcing for

Multimedia (CrowdMM ’14), , 63–68.

Zhao, Yuxiang, and Qinghua Zhu. 2014. “Evaluation on

Crowdsourcing Research: Current Status and Future

Direction.” Information Systems Frontiers 16(3): 417–

34. https://doi.org/10.1007/s10796-012-9350-4.

icSPORTS 2018 - 6th International Congress on Sport Sciences Research and Technology Support