Kick-Off: Coding with the Crowd – Quality Assurance in the Coding Process with Crowd Workers

Standardized content analysis is a core method of communication science and the only major methodical tool that was developed primarily in our discipline. It is claimed that the first content analysis was conducted by Bernard Berelson and Harold D. Lasswell on propaganda in World War II.

With the internet and more textual content to analyse than ever before standardized content analysis has becomme a major interest not only in scientific but also in commercial media research. Due to the large amount of content, automated methods of content analysis are often used today. Nonetheless, the basis of any good machine learning is a meaningful, unbiased, reliable and valid training set. Thus, we need to focus on our core methodical competencies as conventional content analysis remains still very important to generate those training sets.

But how do we built these sample sets? It has become popular to use crowd workers instead of student assistants or personally present coders. This makes sense because coding tasks are very small and can be easily divided. Also, they often are not particularly difficult or need special education. Crow coding is fast and also very inexpensive. But crowd coding is accompanied by a lack of control by the researchers over the coding process. Personal coding training and agreements and questions about the coding process are no longer part of the process, and due to the low pay and the non-permanent working relationship between the crowdworker and the researcher, it is obvious that the coding may have quality deficiencies.

This is where our study „Coding with the Crowd“ comes in: In an experimental design, we want to check whether there are actually quality deficiencies compared to conventional coders. We also want to find out whether gamification elements in online coding can increase motivation and thus improve the quality of coding.
Together with my colleague Anja Dittrich – who is a proven expert in content analysis theory and research – I acquired the project from DGOF Research Funding. We are very grateful for the support that enables us to reward all coders at a fair rate. Especially crowdworkers are often badly paid which causes precarious living conditions. It is our heartfelt concern with this study to point out that these unfortunate circumstances are not acceptable!

I am thrilled about the chance to conduct this study and excited about the results. Of course I will soon report more about the implementation of the project here in the blog.

The project is funded by