CROWDSOURCING AND ITS APPLICATION 
Jelena Mitrović, jmitrovic@gmail.com, University of Belgrade, Faculty of Philology 
Abstract 
The goal of this paper is to point out the importance of crowdsourcing and to present some of the most successful projects that are functioning on the basis of this management model that originated in the business world, but it found its way into the world of culture and science. 
The ways in which crowdsourcing systems function are explored, as well as the technologies they are most commonly based upon. 
Some necessary requirements for the success of crowdsourcing systems are determined and best practice examples described, with extra attention given to the Natural Language Processing (NLP) projects that are carried out via crowdsourcing.  
Keywords 
crowdsourcing, microtasks, MTurk, Natural Language Processing, WordNet 
1. Introduction  
The rise of Web 2.0 technologies (Murugesan 2007) facilitated faster development and improvement of the Digital Humanities paradigm – an interdisciplinary field of research connecting Humanities and Information-Communication technologies (ICT). 
Unlike many other interdisciplinary experiments, the Digital Humanities have a very well-known beginning. 
In 1949, an Italian Jesuit priest, Father Roberto Busa, decided to use computer technologies to make an index verborum of all the words in the works of St Thomas Aquinas, totaling some 11 million words of medieval Latin – 
the Index Thomisticus (Schreibman et. al.2004). 
The term Web 2.0 was coined by Darč Dinucci in an article named “Fragmented future”, in January 1999. 
In the mentioned article, DiNucci described the state of the World Wide Web that was still static and based on mere loading of pages into web browsers; it was just an embryo of the web that was soon to come. 
DiNucci predicted that the web would become a transport mechanism that will allow interactivity (Dinucci 1999). 
Apart from many other innovations, Wеb 2.0 brought along Wikipedia, without which we couldn’t imagine search for information and which is also very important because it gives us a playground for information interchange and knowledge development. 
Social networks like Facebook and Twitter, as well as many other web sites based on blog technologies, open up completely new horizons of cooperation, and if we find the right way to use them to our advantage, they can contribute greatly to the development of many areas of research in the Humanities, be it History, Art, Literature or Linguistics. 
These social networking sites are often used for crowdsourcing projects, and the site Wikipedia is considered to be one of the first crowdsourcing projects on the Internet (Doan et. al. 2011). 
Generally speaking, Web 2.0 is a technological basis upon which crowdsourcing is being developed and upon which it functions (Vuković and Bartolini 2010).  
2. Some definitions of crowdsourcing 
Nowadays, there are many different types of crowdsourcing, and many definitions regarding this model. 
The term crowdsourcing was introduced by Jeff Howe, who called crowdsourcing a business practice that literally means to outsource tasks in a project to a group of people (i.e.crowd) (Howe 2006). 
Howe stressed the importance of this way of distributing work assignments and called it “the mechanism by which talent and knowledge of a group of people is matched to those in need of it.” (Howe 2008). 
Some authors define crowdsourcing as “an online, distributed problem solving and production model alreadž in use by for-profit organizations such as Threadless, iStock...” (Brabham 2008a). 
The same author (Brabham 2008b) gives another definition of crowdsourcing as “a strategic model to attract an interested, motivated crowd of individuals capable of providing solutions superior in quality and quantity to those that even traditional forms of business can. 
Authors (Buecheler et. al. 2010) described crowdsourcing as “a special case of collective intelligence.” 
In paper (Grier 2011) crowdsourcing is described as “a way of using the Internet to employ large numbers of dispersed workers”. 
Another paper (Kazai 2011) describes this type of cooperation as “an open call for contributions from members of the crowd to solve a problem or carry out human intelligence tasks, often in exchange for micro-payments, social recognition, or entertainment value”. 
One of the most elaborate definitions of crowdsourcing is given in (Estellés-Arolas and González-Ladrón-de-Guevara 2012) – ”Crowdsourcing is a type of participative online activity in which an individual, an institution, a non-profit organization, or company proposes to a group of individuals of varying knowledge, heterogeneity, and number, via a flexible open call, the voluntary undertaking of a task. 
The undertaking of the task, of variable complexity and modularity, and in which the crowd should participate bringing their work, money, knowledge and/or experience, always entails mutual benefit. 
The user will receive the satisfaction of a given type of need, be it economic, social recognition, self-esteem, or the development of individual skills, while the crowdsourcer will obtain and utilize to their advantage that what the user has brought to the venture, whose form will depend on the type of activity undertaken”. 
3. Management of crowdsourcing projects 
In the process of planning a crowdsourcing project, one must have answers to many questions. 
The answers to those questions greatly affect successful management of the projects we wish to conduct based on the crowdsourcing model and its paradigm of distributed work. 
In that regard, some of the most important questions will be addressed in the following segments of this paper. 
3.1. What constitutes a crowd? 
It has been shown that different types of crowdsourcing projects require differently structured groups of participants, in order to get the most out of the, so-called, Wisdom of the Crowd. 
Wisdom of the Crowd relates to a phenomenon in which a group of many participants in a project can be much more successful than a few experts. 
Wikipedia is an example of a project involving a large group of participants (that are forming a rather heterogeneous group, in the sense of education and expertise), that is improving the quality of written articles by its Collective Wisdom (Arazy et. al. 2006). 
Other projects that benefit from heterogeneous groups of participants are picture tagging projects or projects involving collecting opinions about new products. 
Still, some projects require involving a homogenous group of participants, that is to say, a group of people with a similar level of education and expertise (Estellés-Arolas and González-Ladrón-de-Guevara 2012), as is the case with creative endeavors (for example, in the case of previously mentioned projects Threadless and iStock). 
3.2. How will the participants contribute? 
The contribution of participants in a crowdsourcing project is affected by the way the tasks are distributed. 
When we are talking about the distribution of tasks to the participants in a crowdsourcing project, we can talk about microtasks and macrotasks. 
Microtasks are carried out when each participant in the crowdsourcing project fulfills some or many small parts of the entire task at hand, in order for all contributions to be merged in the end and to get to the final solution of the problem that the “crowdworkers” were needed to help with. 
On the other hand, a crowdsourcing project is carried out through macrotasks when the entire task is known to potential contributors, so that they can decide which part they want to participate in, based on their knowledge and competencies. 
It has been observed that microtasks are an especially useful model through which tasks are finished faster and more efficiently than through tasks done by employees of a certain company in a workplace environment (Chamberlain et. al. 2012). 
Microtasks are small, well defined tasks that are done by a group of people via certain online platforms. 
The problems that can be resolved through microtasks are the problems that can be easily distributed over a large number of tasks, that is to say, they can be split into a lot of small problems, so that participants in a crowdsourcing project could solve them easily, in a relatively short time, without the need to have some special knowledge or skills pertinent to the problem at hand. 
Microtasks can be used for content labeling and sorting (Maji 2011), specific information retrieval on the internet and generally, for all tasks that require human intelligence but computers are still struggling with.  
Microtasks in any microtasking crowdsourcing project need to be very well defined so that potential participants in that project can have a clear vision of what it is that they have to do to fulfill the requirements. 
Microtasks are usually organized into groups, whether by similarity or according to belonging to the same parts of a project. 
One of the most popular microtasking crowdsourcing platforms is the Amazon Mechanical Turk (MTurk). 
Microtasks on the MTurk platform are called HITs (Human Intelligence Tasks). 
HITs usually have a set of parameters assigned to them, e.g. the number of answers required for each HIT, the time frame in which each HIT should be done, et cetera. 
The type of the “crowd” can also be described, as for some microtasks, participants with a certain level of knowledge are needed, e.g. a certain level of foreign language knowledge for projects that are dealing with improvements in machine translation. 
Amazon Mechanical Turk is, in fact, providing an API – Application Program Interface for web services (this interface defines the ways in which a web service can be used and it sets the rules of usage for that service). 
The Mechanical Turk application program interface can be used so that tasks can be presented through the Amazon Mechanical Turk web site, and those that are fulfilled will be verified and included in other software applications designed for the purposes of a certain crowdsourcing project (Kittur 2008; Vukovic et. al. 2010; Doan et. al. 2011). 
3.3 What is the best way to motivate participants in a crowdsourcing project? 
This question is probably one of the most important questions one needs to consider in the process of organizing a crowdsourcing project. 
How do we motivate participants? 
Authors (Estellés-Arolas and González-Ladrón-de-Guevara 2012) make a connection between the recompense to participants and the Maslow pyramid of needs (Maslow 1943). 
Motivation, apart from the mere financial recompense, can also be a chance to develop one’s creative and other skills; a chance to have fun; to share our knowledge; the love for our community; the love for the task itself; social recognition in our own community or in the environment of social networks. 
In (Chamberlain et. al. 2012) a distinction into three ways to motivate participants in a crowdsourcing project is given, and that is to target personal, social or financial motivation. 
1) Personal motives are mirrored in the willingness of people to be a part of a certain project. 
Generally speaking, personal motives are present whenever the act of participating is a reward in itself and someone just wants to contribute to the cause they feel is worthwhile and important. 
A good example of personal motivation is the Duolingo project, where participants translate parts of the World Wide Web, and what they get in return is the chance to learn foreign languages for free, and in a very structured and professional way (Vesselinov 2012). 
2) Social motives and the social way of rewarding and motivating participants are related to the way of improving the social status of participants in the context of social networks and related to other participants in the crowdsourcing project. 
It has been shown that the system of points and levels is very efficient, as participants tend to finish more tasks in order to obtain a specified number of points and move to the next level. (Von Ahn and Dabbish 2008). 
3) Financial motives, that is to say rewarding participants in a financial way, are used most commonly through Amazon Mechanical Turk and similar platforms. 
For each task that the participant has fulfilled through this platform, the financial reward is defined (often called a micropayment). 
The company Amazon.com, as an owner of MTurk, collects a fee of ten percent on top of what the requesters (so called crowdsourcers) pay to have tasks completed. 
The minimum commission charged through this platform is $0.005 per HIT. 
3.4. Which evaluation methods should be used?  
Amazon Mechanical Turk and other similar platforms are very useful for crowdsourcing projects, but there is always a possibility of misuse and abuse by participants – they sometimes deliberately give wrong answers or they just type in a random set of characters, just to be able to get the financial reward. 
That is why different methods of evaluation are in use, starting with the Redundanč check via models of validation in which many users perform the same task (Oleson et.al. 2011). 
This method helps with determining the correct answer, but it can also help measure the quality of a participant’s work and the said participant’s value in terms of participation in the project. 
Further, the frequently used method is described in (Dawid and Skene1979), where the participants’ reliability check is based on the algorithm (Еxpectation Мaximization – ЕМ) whose result is a set of estimated correct answers for each task, as well as a an error matrix for each participant, which gives the statistical estimate of error making. 
This error matrix can be used for direct quality measuring for each participant. 
A similar approach is given in (Ipeirotis et. al. 2010) where the ЕМ algorithm is used to calculate the quality of contribution for each participant in the project, but it is adapted to the cases when errors are expected, and each participant gets assigned with an error rate, which gives a clear picture of the quality of contributions.  
The evaluation solution that is used successfully in many crowdsourcing projects is setting up a system of “gold tasks” (also called “gold units” or “gold standard” by some authors) that help improve the basic functionality of some crowdsourcing platforms. 
Gold tasks are those tasks for which the answers and solutions are known, or they are trivial, therefore, the accurač of participants’ answers is evaluated based on the answers to those gold tasks. 
These tasks are used to determine whether certain participants can be trusted, because, if they gave correct answers to those tasks for which answers were alreadž known, there is a level of certainty that the other tasks will be done correctly. 
The RECAPTCHA technology that is used in many book digitization projects, as well as in other cultural crowdsourcing projects (Митровић 2013), is based on a similar principle. 
ReCAPTCHA is a free CAPTCHA technology that is facilitating digitization of books, newspapers and old radio shows. 
CAPTCHA (Completely Automated Public Turing Test to Tell Computers and Humans Apart) is a program that is able to decifer if its user is a human or a computer program. 
ReCAPTCHA technology is based on typing two words into a given frame on the web page for the purpose of authentication. 
One of those two words is known to the system, while the other one is a word that was not recognized by the OCR (Optical Character Recognition) technology. 
If users type the control word correctly, the system assumes that their answer for the other word is also correct. 
The same combination of two words is then being offered to other users, and the decision on the correct answer is based on statistical models. 
The end result of this process is an adequately recognized word. 
The use of RECAPTCHA technology can be considered a ubiquitous crowdsourcing project, because all internet users are typing in the words in a RECAPTCHA puzzle (microtasks) for authentication purposes, often not knowing that they are helping digitize books and preserve the world’s cultural heritage, word by word. 
4. Evaluation in NLP projects 
Many complex problems can be resolved successfully through crowdsourcing if the existing crowdsourcing platforms are used in combination with additional management techniques and quality control. 
That is why microtasks can also be used in NLP related projects, usually for different kinds of annotation and validation. 
Moreover, whenever two or more platforms are needed in order to collect and merge participants’ contributions, new methods of evaluation are needed. 
In (Munro et. al. 2010) several crowdsourced NLP projects that are evaluated using the Kappa coefficient are described. 
The Kappa coefficient (Carletta 1996) is used for content analysis in many NLP projects, because it permits comparison of different results obtained from participants and it is used in cases when commonly used evaluation methods (e.g. via gold tasks) are not sufficient.  
In the project named CROWDMAP (Sarasua et. al. 2012), aside from the MTurk platform, the ClowdFlower platform was used as the basis of implementation. 
CrowdFlower acts as an intermediary because it posts microtasks to various platforms (including MTurk, Crowd Guru, Getpaid, Snapvertise). 
The goal of this project is to explore a very important area of NLP – Ontology alignment, using the crowdsourcing method. 
The aim is to improve alreadž existing processes, that are automated, but human input is of great importance to get the right results. 
Evaluation in this project was done through the abovementioned gold tasks, as well as using the Kripendorff’s alpha coefficient. 
The Kripendorff’s alpha coefficient was developed to measure the agreement among participants’ input, the agreement among observers, coders, judges, raters, or measuring instruments drawing distinctions among typically unstructured phenomena or to assign computable values to them, as well as in cases when data are incomplete, e.g. when some participants in a project do not give all answers required for some tasks. 
Kripendorff’s alpha emerged in content analysis but is widely applicable wherever two or more methods of generating data are applied to the same set of objects, units of analysis, or items and the question is how much the resulting data can be trusted to represent something real (Kripendorff 2011). 
Two quality control procedures were used in the project focusing on generation and analysis of a specialized corpus meant to be used for sarcasm and irony identification in texts (Filatova 2012). 
The first kind of quality control procedure was a simple Majority voting – participants were asked to vote for the best solutions given by other participants. 
The second type of quality control procedure was implemented using the abovementioned Kripendorff’s alpha coefficient. 
In Phrase Detectives (Chamberlain et. al. 2012), the social network power of Facebook is used for a crowdsourcing project in which participants annotate texts while playing games. 
The players begin with “training texts” that were set to be a gold standard (gold tasks) – texts alreadž annotated by professional linguists. 
Once a player finishes annotation of texts that were set to be a gold standard, he gets a score, based on the percentage of correctly annotated “gold” texts. 
This project falls into a group of crowdsourcing projects named “Games with a purpose”. 
An important project that is also based on Games with a Purpose is the digitization project of The National Library of Finland, called Digitalkoot, in which video games are used to correct errors made by the Optical Character Recognition software (Митровић et.  al. 2010/2011). 
Some of the very first Games with a purpose were introduced through ESP and Verbosity projects (Von Ahn and Dabbish 2008). 
Transcribe Bentham is also a very important and successful crowdsourcing project. 
It is a participatory project based at University College London. 
Its aim is to engage the public in the online transcription of original and unstudied manuscript papers written by the great philosopher and reformer, Jeremy Bentham (1748-1832). 
Thanks to invaluable help of volunteers, 95 percent of saved manuscripts have been transcribed so far (Causer et.al 2012). 
5. Building Serbian WordNet through crowdsourcing 
WordNet (Fellbaum 1998) is a hierarchical lexical-semantic network of great value in all scientific areas that combine computer technologies with linguistics – Natural Language Processing, Computational Linguistics or a part of Digital Humanities (Biemann and Nygaard 2010; Rumshisky 2011; Biemann 2012). 
In this unique network, lexical concepts are organized into synsets – sets of synonymous words representing a concept. 
Serbian WordNet (SWN) started developing within the scope of BalkaNet (Balkan WordNet project – BWN), as was the case with Bulgarian, Czech, Romanian, Greek and Turkish wordnets (Stamou et.al). 
SWN is based on the structure of the Princeton WordNet (PWN), and it was built following the expand model, in accordance with the rules set by the BalkaNet project – synsets from PWN are copied into SWN and translated, while the hierarchical structure of PWN is preserved.  
After the BalkaNet project ended, in 2004, the development of SWN was leaning on the good will of professors, collaborators and postgraduate students of the Department for Library Science and Information Science of the Faculty of Philology and members of The Group for Language Technologies of the Faculty of Mathematics, University of Belgrade. 
Still, the current version of SWN contains around 20000 synsets, compared to around 117000 synsets in PWN. 
Therefore, the number of synsets in SWN should be increased in a semi-automatic way. 
This could be achieved via crowdsourcing.  
For example, one crowdsourcing project regarding SWN could be based on deciding on the most frequent adjectives in Serbian, compared to adjectives in English that are incorporated in the Princeton WordNet, because PWN is the basis of building our wordnet. 
Pinceton WordNet contains 20479 adjectives, compared to 1488 in Serbian WordNet. 
With significant increase in the number of adjectives, SWN would be much more useful for a very important NLP task – Sentiment analysis, for which wordnets are very suitable, due to their well-developed semantic structure (Torii et. al. 2011). 
The project aiming at deciding on the most frequent adjectives via crowdsourcing would be based on microtasks. 
The tasks should be as simple and easy to solve as possible, so that a lot of people would want to participate. 
Using a simple voting system, we would ask the participants to decide whether a certain English adjective (taken from PWN), that is to say, its translation in Serbian, is relevant. 
The number of tasks each participant would need to fulfill would be limited. 
The same sets of tasks would be offered to many different participants. 
After a certain time period, new sets of tasks would be distributed in order to collect a large number of adjectives that would be added to SWN, and the process would be repeated as many times as needed.  
Answers coming from all participants who didn’t fulfill the requirements of the golden tasks (which are necessary in a project like this one) will be eliminated, while statistical analysis will be performed over the remaining set of data – we will get middle values of YES and NO answers, as the YES answer will have a value of 1 and the NO answer will have a value of 0. 
That way we will get the percentage of value for each given adjective, telling us which adjectives are relevant and in frequent use, according to our participants’ opinions. 
This project would require using the Wisdom of the Crowd of a relatively homogenous group of participants, considering the fact that the participants would need to have a certain level of knowledge regarding both English and Serbian in order for them to bring an informed decision, because the task would be for them to decide whether a given English adjective translated into Serbian would be relevant.              
In the beginning, this project could be rolled out via a Google tool called Google form, which can be used as a part of the Google Drive service. 
Google form facilitates the use of various types of questionnaires that can easily be distributed over social networks, but they can also be sent to e-mail addresses of potential participants in the project. 
The results of the questionnaire are given in the form of a spreadsheet, with time stamps given for each new entry. 
Motivation of participants in a project such as this one would be of personal and social nature – the willingness to help in the process of enhancement of Serbian WordNet, as a valuable resource for processing of Serbian. 
Participants would be enlisted through social networks sites, also aiming at students of English at the faculties of Serbia and students of Serbian at English speaking countries – all potential participants who could contribute to this project. 
Conclusion 
The crowdsourcing model proved to be very successful in cultural and scientific projects, but its importance is prominent in Natural Language Processing, because this field of research is still greatly dependent on human input – human intelligence is crucial for validating automated methods that are implemented with more or less success. 
If crowdsourcing projects are implemented through platforms such is MTurk, participants are rewarded financially, and there is no need to motivate them in any other way. 
In cases when participants are contributing to a project on a voluntary basis, it is necessary to attract their interest and to keep them interested, which is the basis of every successful crowdsourcing project. 
As long as we can find adequate ways to harness human creativity, provided that we pay attention to all aspects of the crowdsourcing project, this model of cooperation can also bring much needed benefits for different domestic projects. 
Serbian WordNet (Krstev et. al. 2004) can be improved and upgraded through crowdsourcing work distribution and some of the evaluation methods discussed in this paper, which is the course of action neccessary for development of tools and resources that are available for Natural Language Processing in the Serbian language. 
