https://github.com/ahwang16/mturk-templates

A collection of MTurk templates designed to make complex tasks easier for human annotators.
https://github.com/ahwang16/mturk-templates

human-annotation human-evaluation mturk nlp

Last synced: about 6 hours ago
JSON representation

A collection of MTurk templates designed to make complex tasks easier for human annotators.

• Host: GitHub
• URL: https://github.com/ahwang16/mturk-templates
• Owner: ahwang16
• Created: 2023-06-23T20:24:15.000Z (over 1 year ago)
• Default Branch: master
• Last Pushed: 2023-11-11T01:48:56.000Z (about 1 year ago)
• Last Synced: 2024-11-05T08:42:58.612Z (about 2 months ago)
• Topics: human-annotation, human-evaluation, mturk, nlp
• Language: HTML
• Homepage:
• Size: 154 KB
• Stars: 0
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Citation: CITATION.cff

Awesome Lists containing this project

README

        
![MTurk Templates for Complex Human Annotation Tasks](mturk_github.png)

A collection of complex Amazon Mechanical Turk templates I created for my work. These templates are designed to make complex annotations tasks easier for human annotators to evaluate by breaking down each HIT into several subtasks. I will add templates and references here as I work on them.

## Compare Recipes

This MTurk Task asks Turkers to compare two recipes by displaying the entirety of one and providing the other one step at a time. The ingredients list is shown to the far left to help ground the task in background information. Turkers answer questions about each step of the second recipe, followed by questions about both recipes in general. This template could be adapted for any task that involves comparing two long texts, like summarization, style transfer, machine translation, or question answering.



   



Used in [Large Language Models as Sous Chefs: Revising Recipes with GPT-3](https://arxiv.org/abs/2306.13986) (Hwang et al., 2023).

## Use and Citation

Feel free to fork this repository, submit pull requests, or use these templates for your work. If you use any of these templates for your work or inspiration, please cite this repository:

```

@software{Hwang_MTurk_Templates_for_2023,

  author = {Hwang, Alyssa},

  month = jun,

  title = {{MTurk Templates for Complex Human Evaluation Tasks}},

  url = {https://github.com/ahwang16/mturk-templates},

  version = {1.0.0},

  year = {2023}

}

```