Paper Reading #15: TurKit: Human Computation Algorithms on Mechanical Turk

Commentary

See what I have to say about ___'s and ___'s work.

References

Little, G., et al. (2010). Turkit: human computation algorithms on mechanical turk. Proceeding of the Acm conference on user interface software and technology. New York: http://www.acm.org/uist/uist2010/.

Article Summary

In this paper, the researcher describe TurKit, a scripting environment for Amazon's Mechanical Turk (MTurk). MTurk allows people to post Human Intelligence Tasks (HITs) as jobs, for which MTurk Workers (Turkers) may be paid a few cents or more, depending on the difficulty of the task and the amount of time taken to complete the task. Some tasks include adding tags or descriptions to images, organizing images based on their content, or voting on the most descriptive or grammatically correct passage of writing in a set. MTurk provides an API that allows users to interface directly with the job creation system. The TurKit environment helps people who post jobs to automate the task of posting jobs and process iterative jobs as well.

Image courtesy of the above-cited article.

It utilizes the crash-and-rerun programming paradigm, in which a script is run until it crashes or is terminated, and then restarts from the beginning. The key advantage provided by TurKit is that posting jobs costs money, and with this system, posters can be ensured of not reposting tasks that have already been completed, thereby saving money. TurKit also provides for the automation of parallelized tasks as well, a key advantage of MTurk.

Discussion

I'm such a huge fan of automation. Parallelism is an added plus, which I'm hoping is something that will start to catch on here in the not-too-distant future. I also like when system maintainers publish an API so that users with that DIY ethic can tinker around and built something that is incredibly useful, if not to the general population then at least for themselves. In short, I like everything I've heard surrounding MTurk and TurKit.

I think that the thing I'm most excited about with respect to this article is the practicality of TurKit. It takes advantage of all of the best parts about MTurk and makes them more accessible. It's a simple design that is very well executed. Nice work :)