Exploring Human Computation and Social Computing to Inform

the Design Process

Roberto Romani

and Maria Cecília Calani Baranauskas

Institute of Computing, Unicamp, Av. Albert Einstein, 1251 Campinas, SP, Brazil

Keywords: GWAP, HCI, GWIDO, Social Network, Human-Centered Computing, Culture of Participation.

Abstract: Although several standards, recommendations and guidelines have been used to assist designers in their

tasks, much of the design choices still rely heavily on the designer’s experience. In this work we argue that

complex choices about interface elements (e.g. images, icons, sounds) could have the help of the users

themselves to inform the designer´s choices. The paper situates the contribution in the intersection of the

human computation and social computing fields, showing a preliminary survey of related work. Moreover,

we illustrate the idea with an instantiation of an environment for designers, within the frontiers of human

computation and social computing.

1 INTRODUCTION

Since the HCI beginning, designers of interactive

applications have been using several techniques to

understand the users' tasks, their needs and potential

new features that might improve the users’ activities.

Although these techniques are constantly being

improved, the challenge has increased especially

because of the development of new electronic

mobile devices, the web evolution and consequent

diversity of users.

When the scope of an application is well-defined

and the set of potential users is limited and

homogenous, designers may use traditional HCI

techniques to work the user interface elements

representations in tune with the users’ profile.

However, the interaction design becomes more

difficult as the number of user classes and systems

requirements increase. In this context new support to

the design process must be provided to capture this

diversity.

The design for all (University N.C.S, 2008)

approach proposes that systems should be projected

for a huge variety of users with different conditions

and needs. According to the HCI practice, the user is

the most indicated stakeholder to validate the

interfaces projected by designers, as well as to

contribute during the design process. In several

design projects it is difficult to involve a large and

varied number of users through conventional user

centered or participatory design methods. Thus,

designers solve such difficulties adapting techniques

and using their own experience. However, the web

provides resources which can be used as an efficient

mechanism of “unlimited” access to different users’

classes worldwide.

This idea of using applications and services that

facilitate collective action and online social

interaction is associated to the term “social

computing”. Several technologies such as blogs,

wikis and online communities are examples of social

computing. Although the scope of the term is broad,

it includes humans in a social role where technology

mediates the human communication. Thus, these

aspects of the web could be used to provide a virtual

space for designers to share experiences in order to

propose design elements more suitable to users.

Although several standards, recommendations

and guidelines are used to assist in the user

interfaces development, much of the design still

relies heavily on the designer’s experience and

knowledge. In other words, important decisions on

specific parts of interface design such as choices

about images, icons, sounds and other interface

elements are complex tasks that could have the help

of humans themselves to inform the designer´s

choices. For example, for a computer system,

defining the best image among many to represent a

concept such as “schedule meeting”, is unthinkable.

However, users can quickly choose which image is

most representative for the concept. The paradigm

that relates to the use of human endeavor to

Romani R. and Calani Baranauskas M..

Exploring Human Computation and Social Computing to Inform the Design Process.

DOI: 10.5220/0004434300670074

In Proceedings of the 15th International Conference on Enterprise Information Systems (ICEIS-2013), pages 67-74

ISBN: 978-989-8565-61-7

 2013 SCITEPRESS (Science and Technology Publications, Lda.)

accomplish tasks that computers can not yet perform

is defined as “Human Computation”.

According Quinn and Bederson (2011) the

difference between human computation and social

computing is that social computing facilitates

relatively natural human behavior that happens to be

mediated by technology, whereas participation in a

human computation is directed primarily by the

human computation system. However, the same

authors show that there is an intersection field

between human computation and social computing.

This paper aims at shedding light on that

intersection field by presenting possibilities of

taking advantage of both fields to contribute to HCI

research. In this scenario, this work proposes a new

approach for supporting the choices of designers

with users’ contribution through enjoyable activities

such as games. In addition, an environment where

designers can socialize knowledge and information

in the design process is suggested.

The work is organized as follows: Section 2

presents the background research fields, delineating

the focus of interest of this paper; Section 3

organizes a preliminary literature overview in the

intersection of human computation and social

computing; Section 4 instantiates the idea proposed

with the GWIDO environment; Section 5 concludes.

2 BACKGROUND

Since the popularization of computational artefacts,

people have worked with them in several interesting

ways. More recently we have also engaged in

communicating through computers. An alternative

way to involve the human in a work process is using

their processing power to solve problems that

computers cannot yet solve. This is the modern

usage of the term “human computation” as coined

by von Ahn (2005). He yet considers that it is

feasible to solve large-scale computational problems

and collect data to teach computers basic human

talents. In fact, the idea is to put human brains

working as processors in a distributed system, each

one performing a small part of a massive

computation (von Ahn, 2009).

Human computation is related to other terms

such as collective intelligence, crowdsourcing, and

social computing but they are not synonymous

(Quinn and Bederson, 2011).

According Howe (2008) crowdsourcing is

defined as the act of conducting traditional human

work with ordinary people. An example is when a

large group of people performs a job responding to

an open call substituting a traditionally designated

agent who would perform that specific job.

On the other hand, social computing is related to

humans in social role where their communication is

mediated by technology (Parameswaran and

Whinston, 2007). Blogs, facebook ©, twiter ©, wikis

are some examples of technologies used to facilitate

the collective action and social interaction online.

Computational problems which are solved by

human computation are occasionally found in

crowdsourcing and social computing applications.

There is an intersection of crowdsourcing and

human computation issues that is shown in Figure 1.

Some applications can be classified in this

intersection such as MonoTrans which provides a

solution for the language translation task (Hu et al.,

2011).

Collective intelligence is presented in Figure 1 as

a superset of social computing, human computation

and crowdsourcing. This term is defined by Malone

et al. (2009) as groups of individuals doing things

collectively that seem intelligent. Some examples

such as Wikipedia have shown a great number of

people collaborating in the same project.

Figure 1: The intersection of Human Computation and

Social Computing can be explored in benefit of interaction

design process (adapted from Quinn and Bederson, 2011).

Although all research areas shown in Figure 1

are relevant and widely studied, this work focuses on

the intersection of Human Computation and Social

Computing. Principles of both areas can contribute

to the interface design process since we can take

advantage of the human ability to solve difficult

problems, associated to the facilities of social

ICEIS2013-15thInternationalConferenceonEnterpriseInformationSystems

Figure 2: Number of papers published in conferences and journals.

networks. The different on-line social networks

available on the web can enable the approximation

of both users and designers of different regions and

cultures.

With regard to Human Computation, a new set of

systems have been developed since 2004 as casual

games to collect annotations from human users; they

are called GWAPs (Game With A Purpose). The

GWAP concept was proposed by von Ahn (2006)

based on Human Computation principles. Problems

solved by humans in GWAP games have two main

assumptions: (1) computers alone are not good at

solving them and (2) they are trivial for humans.

ESP (the name is a joke with Extra Sensorial

Perception) Game was the first GWAP proposed

(von Ahn and Dabish, 2004). The ESP objective is

to label images that are considered a complex task

for computers. In this game, the same image is

presented to two players. Then, they should type

words or phrases which describe the image. Each

player does not know what the other one is typing,

but if both type exactly the same thing, this word or

phrase is a good suggestion for labelling that image.

They will again receive a new image to continue

playing. The players’ goal is to label the largest

number of images in a predefined time, getting

points every time they coincide in the answers.

GWAP is an example of collective intelligence

since this type of game aggregate data from non-

expert players helping in collective decisions that are

similar to opinions from an expert (Chamberlain et

al., 2012).

3 HUMAN COMPUTATION

AND SOCIAL COMPUTING:

A PRELIMINARY SURVEY

Several authors have explored different aspects of

Human Computation and Social Computing in the

last years (von Ahn, 2009); (Parameswaran and

Whinston, 2007); (Quinn and Bederson, 2011);

(Wang et al., 2007). With the objective of assessing

the comprisement of both areas and their

intersection, we conducted a survey on the number

of articles published about each subject in digital

libraries of ACM and IEEE since 2004. A summary

of results obtained in this search is shown in Figure

The survey was conducted considering

expressions such as: "Human Computation", "Social

Computing", "GWAP", "Social Games", "Human

Computation and Social Computing", "GWAP and

Social Computing" in order to evaluate each term

separately and subsequently the association of two

main concepts.

Each expression was searched in the papers full

texts and abstracts, in both ACM and IEEE digital

libraries. The search was conducted in January 7,

2013, being restricted to articles published in

journals or conferences from 2004 to 2012. This

period was chosen considering the modern use of the

term Human Computation that started from the

proposition of the first GWAP called ESP Game

published in 2004.

Results for the terms showed that the amount of

articles about Social Computing is far superior to

that one regarding Human Computation. In part, this

ExploringHumanComputationandSocialComputingtoInformtheDesignProcess

Table 1: GWAPS found in literature from 2004 to 2012.

Main Purpose GWAPs Human Skill

(7) Image tagging

ESP Game (Ahn and Dabbish, 2004) Visual Recognition

Phetch (Ahn et al., 2007) Visual Recognition and Writing

KissKissBan (Ho et al., 2009) Visual Recognition, Reading and Writing

PexAce (Nagy, 2011) Visual Recognition and Writing

Karido (Steinmayr, 2011) Visual Recognition

ARTigo (Bry and Wieser, 2012) Visual Recognition

IdenticalEmotions (Aggarwal, 2012) Visual Recognition and Feelings

(7) Location-based information

Gopher Game (Casey et al., 2007) Reading, Writing and Take Pictures

Eyespy (Bell et al., 2009) Reading and Visual Recognition

Indagator (Goh et al., 2010) Reading, Writing, Walking, Take Pictures

PhotoCity (Tuite et al., 2010, 2011) Reading, Walking, Take Pictures

SPLASH (Goh et al., 2011) Reading, Writing and Visual Recognition

Tsai & Yang game (Tsai and Yang, 2011) Reading, Writing and Take Pictures

Glob (Kothandapani et al., 2012) Reading, Writing and Visual Recognition

(4) Collect common sense facts

Verbosity (Ahn et al., 2007) Reading and Writing

Rapport Game (Kuo et al., 2009) Reading, Writing and Visual Recognition

Virtual Pet Game (Kuo et al., 2009) Reading, Writing and Visual Recognition

Climate Quiz (Scharl et al., 2012) Reading and Writing

(3) Create ranking/

classifications

Matchin (Hacker and Anh, 2009) Visual recognition

Thumbs-Up (Dasdan et al., 2009) Reading and interpretation

Curator (Walsh and Golbeck, 2010) Visual recognition

(3) Natural language processing

OnToGalaxy (Krause et al., 2010) Reading and Writing

Dil Cambazı (Gencer et al, 2012) Reading

Phrase Detectives (Chamberlain et al., 2012) Reading and Interpretation

(3) Mapping users account across

social network

GameMapping (Shehab et al., 2010) Reading

Pearl & Steyvers game (Pearl and Steyvers,

2010)

Reading and Interpretation

GuessWho (Guy et al., 2011) Reading and Writing

(3) Annotating videos

OntoTube (Siorpaes and Hepp, 2008) Watching Videos and Interpretation

Popvideo (Ahn et al, 2008) Watching Videos and Interpretation

Waisda (Oomen et al., 2010) Watching Videos and Interpretation

(3) Creating ontologies or

relationships for semantic web

OntoPronto (Siorpaes and Hepp, 2008) Reading and Interpretation

SpotTheLink (Thaler et al., 2010) Reading and Interpretation

LittleSearchGame (Šimko et al., 2011) Reading and Writing

(2) Locates objects within images

Peekaboom (Ahn et al., 2006) Visual Recognition

P-HOG (Feng et al., 2012) Visual Recognition

(2) Tagging music

Tag-a-Tune (Law & Anh, 2009) Reading, Writing and Listening

Herd It (Barrington et al., 2009) Reading, Writing and Listening

(2) Generate streams of social

annotation

GiveALink Slider (Weng et al., 2011) Reading, Writing and Interpretation

Great Minds Think Alike (Weng et al., 2011) Reading, Writing and Interpretation

(1) Associate images with user

action

GWIDO Image (Romani and Baranauskas,

2009)

Visual Recognition and Interpretation

(1) Visual research and surveys Sketcharoo (Hebecker and Ebbert, 2010) Visual Recognition, Writing and Drawing

(1) Labelling game characters Shadow Shoppe (Islam et al., 2010) Visual Recognition and Interpretation

(1) Image re-targeting for

browsing images

RecognazePicture (Lux et al., 2010) Visual Recognition

(1) Colect personal data

Bake Your Personality10 (Taktamysheva et al.,

2011)

Reading

(1) Mining microblogs for

advice-oriented information

Twiage (Kleek, et al., 2012) Reading and Interpretation

result occurs because the Social Computing area has

already being widely studied since 2004. Thus, to

facilitate the display of results in the graph of Figure

2, we used logarithmic scale. Furthermore, absolute

values were also plotted in the graph. GWAP and

Social Games represent 22% and 10% of the

research developed in Human Computation and

Social Computing, respectively. Both terms are

associated with the games area.

The intersection between Human Computation

and Social Computing is still quite small since the

search engine returned only ninety articles, all

indexed by ACM and IEEE.

Moreover, when we search the terms GWAP and

Social Computing together, few articles were found

ICEIS2013-15thInternationalConferenceonEnterpriseInformationSystems

(nineteen). None of the retrieved articles addresses

both terms on their abstracts. It suggests that these

two issues have not been explored jointly.

Figure 3 shows a new representation of the

survey results to present an accurate idea of the

proportion of articles published in these fields. Thus

we represent each field as a circle proportional to the

number of articles found in the search.

A survey of GWAPs found in literature since

2004 considering the ESP game as the first proposed

GWAP is presented in Table 1. GWAPs were

grouped according to their main purposes.

We consider the support to interaction and

content creation among communities of users as the

main feature of a social computing application.

In this context GWAPs can be designed as a

social computing application since the majority of

them have support of interaction among people. In

addition, by definition all GWAP produces

information. However few GWAPs exploit the

social potential for own benefit or for the benefit of

the community involved. Information generated by

GWAP usually brings benefits linked to their

purpose such as image tagging.

Figure 3: Schematic representation for the proportion of

published articles and the intersection of fields.

Table 1 highlights those GWAPs, which

explicitly promotes the communities’ formation and

generate information with some utility to these

communities. These GWAPs can be classified as

social computing applications such as Indagator,

PhotoCity, SPLASH, Gopher Game, or they make

use of information extracted from social networks

where they are inserted as for example Phrase

Detectives and GameMapping.

4 THE GWIDO ENVIRONMENT

GWIDO Image is a game proposed in 2009 with the

purpose of helping designers to make choices about

interface graphic elements (Romani and

Baranauskas, 2009; 2010). GWIDO Image is a

collaborative and synchronous two player’s GWAP

that is played in the Web at

http://gwido.nied.unicamp.br/gwido. GWIDO is an

output agreement model game. Images and texts are

its inputs provided by designers that represent

possibilities of users’ actions in Graphical User

Interfaces (GUIs). This is one of the games within

the GWIDO environment.

The GWIDO environment is a web social

application where GWIDO games can be developed

with different purposes by developers (Figure 4).

Then these games can be played by any users on the

web (Romani and Baranauskas, 2012).

Figure 4 illustrates a proposed architecture for

the environment where several GWIDOs feed a

common database. Through the GUI, designers

concepts associated to them into the GWIDO

environment. These elements are used in GWIDO

games. After some game rounds, the designer can

collect results verifying the most representative

images for different user profiles in the environment.

Each designer accesses only the information of

elements registered by him. In this model a

researcher can make statistical analysis to verify, for

example, whether there are regional or meaningful

differences between the different user profiles,

enabling a better informed choice of UI elements for

the system.

Figure 4: GWIDO environment architecture.

All data collected during the game can be

accessed by designers who included inputs to the

game. These data can also be shared with the

community of designers. Data gathered during the

ExploringHumanComputationandSocialComputingtoInformtheDesignProcess

game roles are associated with the users’ profiles to

provide accurate information to help designers in

making choices. For example, GWIDO Image

presents inputs (text and candidate images provided

by designers) and instructs the players to select the

image that best represents this text. If both players

select the same image, they get points. All choices

are registered by the game and will be available on

the environment for supporting the designer in

his/her decision process regarding which image to

use in his/her application.

The GWIDO environment is a social web

application, located in the intersection of Human

Computation and Social Computing. GWIDO

incorporates the virtuous cycle of social computing

in which the community uses a service offered by a

computational system, providing information to this

system that also uses this information to improve the

service to offer to the community (Erickson, 2013).

This is a kind of relationship in which both sides

win, what makes social computing such a productive

field.

The GWIDO environment aids at creating this

collaboration cycle between the environment and its

users (designers in this case,); GWIDO also provides

the possibility of making this cycle between

designers and prospective users of the interfaces

plays a GWIDO game, he/she is providing

information to the designers to project new

interfaces which can be used by these own players.

In this context, GWIDO is a socio-technical

environment that may promote a culture of

participation in the design of human computer

interfaces.

5 CONCLUSIONS

When the scope of an application is well-defined

and the set of potential users is limited and

homogenous, designers have been well instrumented

in their practices. However, as the number of users

increases augmenting their differences in terms of

profiles, culture, social context, etc. the choices of

designers become much more difficult. After a

literature review on background work, this paper

presented an effort coming from the intersection of

the human computation and social computing fields,

as instrumental for supporting designers in their

choices of user interface elements. Ongoing work

involves case studies being conducted to evaluate

the effectiveness of this approach, and further work

involves a large scale test of the proposed

environment.

ACKNOWLEDGEMENTS

We thank the Brazilian funding agency CNPQ

(#560044/2010-0) for financial support to EcoWeb

Project, the IC and AFPU for funding the

registration fees and travel.

REFERENCES

Aggarwal, S., 2012. CAPTCHAs with a Purpose. In

Human Computation, AAAI Technical Report WS-12-

08. p. 61-68.

von Ahn, L., 2005. Human Computation. Doctoral Thesis.

UMI Order Number: AAI3205378, CMU.

von Ahn, L., 2006. Games with a Purpose. In Computer,

39, 6, IEEE Computer Society, p. 92-96.

von Ahn, L., 2009. Human Computation. In DAC’09,

ACM.

von Ahn, L. and Dabbish, L. Labeling images with a

computer game. In Proceedings of the SIGCHI

Conference on Human Factors in Computing Systems

ACM Press, New York, 2004, 319–326.

von Ahn, L., Ginosar, S., Kedia, M., and Blum, M.

Improving image search with Phetch. In Proceedings

of the IEEE International Conference on Acoustics,

Speech, and Signal Processing (Honolulu, Apr. 15–

20). IEEE Press, New York, 2007, IV–1209–IV–1212.

von Ahn, L., Kedia, M., and Blum, M. Verbosity: A game

for collecting common-sense knowledge. In

Proceedings of the SIGCHI Conference on Human

Factors in Computing Systems (Montreal, Apr. 22–

27). ACM Press, 2007, 75–78.

von Ahn, L., Liu, R., and Blum, M. Peekaboom: A Game

for locating objects in images. In Proceedings of the

SIGCHI Conference on Human Factors in Computing

Systems. ACM Press, New York, 2006, 55–64.

von Ahn, L., et al./Carnegie Mellon University (2008).

Website GWAP [“Games with a purpose”],

http://www.gwap.com/gwap/, accessed on December

2012.

Barrington, L., Turnbull, D., O’Malley, D., Lanckriet, G.,

2009. User-Centered Design of a Social Game to Tag

Music In KDD-HCOMP’09, ACM. P. 7-10.

Bell, M., Reeves, S., Brown, B., Sherwood, S.,

MacMillan, D., Ferguson, J., Chalmers, M., 2009.

EyeSpy: supporting navigation through play. In

SIGCHI Conference on Human Factors in Computing

Systems (CHI '09). ACM. p. 123-132.

Bry, F., Wieser, C., 2012. Squaring and Scripting the ESP

Game. Technical Report from University of Munich,

Germany, p.1-13

Casey, S., Kirman, B., Rowland, D., 2007. The gopher

game: a social, mobile, locative game with user

generated content and peer review. In Proceedings of

ICEIS2013-15thInternationalConferenceonEnterpriseInformationSystems

the International Conference on Advances in computer

entertainment technology, (ACE '07). ACM, New

York, NY, USA, p. 9-16.

Chamberlain, J., Kruschwitz, U., Poesio, M., 2012.

Motivations for participation in socially networked

collective intelligence systems. In Collective

Intelligence – CI´2012.

Dasdan, A., Drome, C., Kolay, S., 2009. Thumbs-Up: A

Game for Playing to Rank Search Results. In

WWW´2009, p. 1071-1072.

Erickson, T., 2013. Social Computing. In: Soegaard, M.

and Dam, R. F. (eds.). The Encyclopedia of Human-

Computer Interaction, 2nd Ed. Aarhus, Denmark: The

Interaction Design Foundation. Available online at

http://www.interaction-design.org/encyclopedia/social

_computing.html.

Feng, J., Ni, Y., Dong, J., Wang, Z., Yan, S., 2012

Purposive Hidden-Object-Game: Embedding Human

Computation in Popular Game. In IEEE Transactions

on Multimedia, v. 14,n. 5, IEEE. p. 1496-1507.

Gencer, A. E., Güngör, T., Gürer, A., Özsoy, A. S., 2012.

Input-Evaluation: A New Mechanism for Collecting

Data Using Games with a Purpose In IEEE

Symposium on Computers and Communications

(ISCC’12), IEEE Computer Society. p. 239-244.

Goh, D. H.-L., Lee, C. S., Chua, A. Y.-K., 2010. Do

Games Motivate Mobile Content Sharing? In

Chowdhury, G., Khoo, C., Hunter, J. (Eds.): ICADL

2010, LNCS 6102, Springer-Verlag, pp. 61–70.

Goh, D. H.-L., Razikin, K., Chua, A. Y.-K., Lee, C. S.,

Tan, K., 2011. Understanding Location-Based

Information Sharing in a Mobile Human Computation

Game. In IEEE International Conferences on Internet

of Things, and Cyber, Physical and Social Computing,

IEEE, p. 209-216.

Guy, I., Perer, A., Daniel, T., Greenshpan, O., Turbahn, I.,

2011. Guess Who? Enriching the Social Graph

through a Crowdsourcing Game. In CHI 2011, ACM.

p.1373-1382.

Hacker, S., von Ahn, L., 2009. Matchin: Eliciting User

Preferences with an Online Game. In CHI´2009,

ACM. p. 1207-1216.

Hebecker, R., Ebbert, C. 2010. Creation and Validation of

Symbols with Purposeful Games and Online Survey.

In DESIRE ’10. p. 112-120.

Ho, C.-J., Chang, T.-H., Lee, J.-C., Hsu, J. Y.-J., Chen,

K.-T. 2009. KissKissBan: A Competitive Human

Computation Game for Image Annotation In KDD-

HCOMP’09, ACM. p. 21-24.

Howe, J. 2008. Crowdsourcing: why the power of the

crowd is driving the future of business. In

Crown'2008.

Hu, C., Bederson, B. B., Resnik, P., 2011. MonoTrans2: a

new human computation system to support

monolingual translation. In CHI'2011, ACM.

Islam, M. T., Nahiduzzaman, K. M., Why, Y. P., Ashraf,

G., 2010. Learning Character Design from Experts and

Laymen. In International Conference on Cyberworlds,

IEEE. p. 134-141.

Kleek, M. V., Smith, D. A., Stranders, R., Schraefel, M.

2012. Twiage: A Game for Finding Good Advice on

Twitter. In CHI´2012, ACM. p. 889-898.

Kothandapani, S., Goh, D. H.-L., Razikin, K., 2012.

Influence of Playability and Usability in a Mobile

Human Computation Game In 32nd International

Conference on Distributed Computing Systems

Workshops, IEEE Computer Society. p. 75-78.

Krause, M., Takhtamysheva, A., Wittstock, M., Malaka,

R., 2010. Frontiers of a Paradigm – Exploring Human

Computation with Digital Games. In KDD-

HCOMP’10, ACM. p. 22-25.

Kuo, Y., Chiang, K., Chan, C., Lee, J., Wang, R., Shen,

E., Hsu, J. Y., 2009. Community-based Game Design:

Experiments on Social Games for Commonsense Data

Collection. In KDD-HCOMP’09, ACM. p. 15-22.

Law, E., von Ahn, L., 2009. Input-Agreement: A New

Mechanism for Collecting Data Using Human

Computation Games. In CHI´09, ACM. p. 1197-1206.

Lux, M., Böszörmenyi, L., Müller, A., 2010. A Social

Approach to Image Re-targeting Based on an

Interactive Game. In Leitner, G., Hitz, M., Holzinger,

A. (Eds.): USAB 2010, LNCS 6389, Springer-Verlag,

p. 465–470.

Malone, T. W., Laubacher, R., Dellarocas, C. N., 2009.

Harnessing crowds: mapping the genome of collective

intelligence. In MIT Sloan Research Paper 4732-09,

MIT.

Nagy, B., 2011. Acquisition of semantic metadata via an

interactive game. In Bieliková, M., Návrat, P., Barla,

M., Tvarožek, J., Eds. Personalized Web - Science,

Technologies and Engineering: 9th Spring 2011 PeWe

Workshop Proceedings, Slovak University of

Technology, 2011. [Online]. Available:

http://pewe.fiit.stuba.sk/ontoparty-2010- 2011-spring/

Oomen, J., Baltussen, L. B., Limonard, S., Brinkerink, M.,

van Ees, A., Aroyo, L., Vervaart, J., Afsar, K.,

Gligorov, R., 2010. Emerging Practices in the Cultural

Heritage Domain - Social Tagging of Audiovisual

Heritage.

Parameswaran, M., Whinston, A. B., 2007. Social

Computing: an overview. In CAIS'2007, 19, 37 p. 762-

780.

Pearl, L., Steyvers, M., 2010. Identifying Emotions,

Intentions, and Attitudes in Text Using a Game with a

Purpose. In NAACL HLT´2010 Workshop on

Computational Approaches to Analysis and

Generation of Emotion in Text, Association for

Computational Linguistics. p. 71–79.

Quinn, A. J. Bederson, B. B., 2011. Human Computation:

a survey and taxonomy of a growing field. In

CHI'2011, ACM.

Romani, R., Baranauskas, M. C. C., 2009. GWIDO -

Games With Interaction Design Objective. In

International Conference IADIS WWW/Internet,

IADIS. p. 351-358.

Romani, R.; Baranauskas, M. C. C., 2010. GWIDO -

Games With Interaction Design Objective. In IADIS

International Journal on WWW/Internet 8,1, IADIS.

p.1-15.

Romani, R., Baranauskas, M. C. C., 2012. Helping

ExploringHumanComputationandSocialComputingtoInformtheDesignProcess

Designers in Making Choices through Games. In IHC’

2012, SBC. p. 229-238

Scharl, A., Sabou, M., Föls, M., 2012. Climate Quiz – A

Web Application for Eliciting and Validating

Knowledge from Social Networks. In WebMedia’12,

ACM. p. 189-192

Shehab, M., Ko, M. N., Touati, H. 2010. Social Networks

Profile Mapping using Games.

Šimko, J., Bieliková, M., 2011. Games with a Purpose:

User Generated Valid Metadata for Personal Archives.

In Sixth International Workshop on Semantic Media

Adaptation and Personalization, IEEE. p. 45-50

Šimko, J., Tvarožek, M., Bieliková, M., 2011. Little

Search Game: Term Network Acquisition via a

Human Computation Game. In HT’11, ACM. p. 57-71

Siorpaes, K., Hepp, M., 2008. Games with a Purpose for

the Semantic Web. In Intelligent Systems, v.23, n.3,

IEEE Computer Society. p.50-60

Steinmayr, B., Wieser, C., Kneißl, F. Bry, F. 2011.

Karido: A GWAP for Telling Artworks Apart. In 16th

International Conference on Computer Games, IEEE.

p. 193-200

Taktamysheva, A., Krause, M., Smeddinck, J., 2011.

Serious Questionnaires in Playful Social Network

Applications. In J. Anacleto et al. (Eds.): ICEC 2011,

LNCS 6972, IFIP International Federation for

Information Processing. pp. 436–439

Thaler, S., Simperl, E., Siorpaes, K., 2010. SpotTheLink:

A Game for Ontology Alignment

Tsai, T.-C., Yang, T.-R. 2011. GWAP Design for a

Mobile Geo-Tagging System With Confident

Verification. In 14th IEEE International Conference

on Computational Science and Engineering, IEEE. p.

302-307

Tuite, K., Snavely, N., Hsiao, D.-Y., Smith, A. M.,

Popovic, Z. 2010. Reconstructing the World in 3D:

Bringing Games with a Purpose Outdoors. In FDG

2010, ACM. p. 232-239

Tuite, K., Snavely, N., Hsiao, D.-Y., Tabing, N., Popovic,

Z., 2011. PhotoCity: Training Experts at Large-scale

Image Acquisition Through a Competitive Game. In

CHI´2011, ACM. p. 1383-1392

University, N. C. S., 2008. 'The Center for Universal

Design' http://www.ncsu.edu/project/design-

projects/udi/, Retrieved: jan. 2013.

Walsh, G., Golbeck, J., 2010. Curator: A Game with a

Purpose for Collection Recommendation. In CHI

2010, ACM. p. 2079-2082

Wang, F.-Y., Zeng, D., Carley, K. M., Mao, W., 2007.

Social Computing: From Social Informatics to Social

Intelligence In Trends & Controversies, IEEE

Computer Society. p. 79-83

Weng, L., Schifanella, R., Menczer, F., 2011. Design of

Social Games for Collecting Reliable Semantic

Annotations. In 16 th International Conference on

Computer Games, IEEE. p. 185-192

ICEIS2013-15thInternationalConferenceonEnterpriseInformationSystems