A few weeks ago, I ran my first pilot study on Amazon Mechanical Turk (MTurk).
Essentially, MTurk is a platform that you can use to label data and get participants for studies (using money). There is of course more that can be done here, but from my current understanding, these seem to be the two main uses in academia.
The completion of a survey usually takes the following form:
- A worker / participant on MTurk accepts the work and is presented a link to the survey.
- The worker copies his unique, anonymized workerID into the survey (so that you can reference the data)
- The worker completes the survey and is presented a unique random code at the end
- The worker copies the code into a form on MTurk’s website.
- You match the IDs of workers that completed the survey with IDs in your database and reward workers for their time
You used MTurk before? This sounds familiar? Yes – however, two parts in this chain are rather weak: (1) the manual copying of the workerID and (2) the generation, matching, and copying of the random code at the end. If either of these fails, you have to work out manually if a worker has participated or not.
As you have already guessed, there is a better way to do this (why else would I be writing about this? :D). MTurk offers to host a so called ExternalQuestion, which allows you to embed a custom website via an <iframe>. On top, it passes some meta information, such as the workerID, to the website; all you need to do is read that info and use the value as you see fit. Additionally, it allows the website to submit a form back to MTurk as proof of having completed the work, which we will do at the end of the survey.
In short, we integrate the survey directly into MTurk thereby getting rid of above pitfalls. It roughly follows these steps:
- Create the Survey on the survey tool (in this case Questback)
- Create a small adapter website on GitHub (may not be needed if you have a different survey tool)
- this will accept and forward requests from Mturk
- allow you to create an arbitrary preview of the survey
- pipe the form back to MTurk when the survey is finished
- Add URL parameters to the survey
- forward people to the adapter website upon completion
- Host an external question pointing to the adapter website on GitHub
Small Adapter Website on GitHub
At first, I tried to link Questback and MTurk directly; then I discovered two limitations making this impossible: (1) Questback only accepts URL parameters named “a=..&b=..&c=..” instead of full variable names, and (2) Questback can not post the results of a form; I could only find forwarding via GET.
Hence, I set up a small website hosted on GitHub to do the plumbing between both websites.
Note: If you are working with the sandbox, you have to change the mturk_url appropriately.
This does 3 things, depending on how it is called:
- if the URL contains a parameter called “returnToMTurk” then we assume that the Questback is forwarding to this website via GET. In this case we take the payload (in this case attentionCheckPassed) and forward it to MTurk as a POST request.
- if the URL contains ASSIGNMENT_ID_NOT_AVAILABLE it means that the survey is being previewed. In this case we do nothing and show this website, which will act as the preview of the survey (e.g. display a screenshot of the hit or some other, relevant information to inform people what this task is about). Note that you want to avoid people submitting your survey at this stage, so showing them the raw survey may be counterproductive at this stage.
- Otherwise the website is called from MTurk by a worker who wants to complete the survey; in this case we rename the URL parameters from their actual names into a, b, c, d and forward the request to the Questback survey.
Add URL parameters to the survey
In Questback this can be done in the survey properties > User-defined variables.
As mentioned before, the URL parameters will have names a,b,c, … and can be accessed in the survey tool via #p_0001#, #p_0002#, #p_0003#, … .
Forward People to the GitHub Adapter Upon Completion
This is easily done in the properties section of the final page under Questionnaire editor > Final Page > Properties > Redirect to Survey .
The address should point to the GitHub Adapter and the URL should include three parameters: (1) the assignmentID sent from MTurk, (2)-this is really important– at least one additional parameter to store in MTurk as the result of the task and (3) the “returnToMTurk” parameter used to tell the adapter what to do. An example of a URL could look like this
Note that the assignmentID is set to #p_0001# which is the the first parameter (“a“) passed to the survey from MTurk. In above example attentionCheckPassed is a variable from the survey which we used to determine if participants payed attention or just mindlessly filled out the survey. This is an aggregate of multiple questions and computed by the survey tool upon completion of the survey. This can later be used to automatically accept / reject / ban workers that have completed the assignment.
It is also important to note that MTurk expects the assignmentID and at least one additional parameter to be send through the form’s POST request. For some reason, the additional parameter is mentioned nowhere in the documentation, but, instead, tacitly assumed.
Additionally, the checkbox next to the phrase Automatically add ospe.php3 to URL and Add return ticket have to be disabled. You would use these if you were forwarding / returning to another ESF survey; this isn’t the case here.
Host an External Question Pointing to the GitHub Adapter
All that is left is to actually host a task on MTurk. In this case an External Question.
Unfortunately, this is currently impossible to do through the web UI. Hence, we have to use the API; I decided to do it in Python by adapting a code snippet that I found on the web. It consists of two files: (1) config.py, which stores the credentials, and (2) create_hit.py which creates the actual hit.
That’s it. Running the code will create a HIT that will point to the website on GitHub, which itself points to our survey. The survey will point back to the GitHub website, which will point back to MTurk, going full circle. Neat!
As always, I hope this is useful to some of you and feel free to drop a comment or reach out to me if you have questions.