{"id":31909653,"url":"https://github.com/allenai/amti","last_synced_at":"2026-03-05T20:24:03.217Z","repository":{"id":46719317,"uuid":"123380305","full_name":"allenai/amti","owner":"allenai","description":"A Mechanical Turk Interface (amti) 🤖","archived":false,"fork":false,"pushed_at":"2024-01-11T17:50:04.000Z","size":113,"stargazers_count":55,"open_issues_count":12,"forks_count":18,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-04-14T07:49:54.268Z","etag":null,"topics":["annotation","cli","command-line-tool","crowdsourcing","mechanical-turk","mturk"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/allenai.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-01T04:05:02.000Z","updated_at":"2024-04-11T10:07:42.000Z","dependencies_parsed_at":"2022-09-13T11:01:38.078Z","dependency_job_id":null,"html_url":"https://github.com/allenai/amti","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/allenai/amti","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Famti","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Famti/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Famti/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Famti/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/allenai","download_url":"https://codeload.github.com/allenai/amti/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Famti/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279016073,"owners_count":26085798,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["annotation","cli","command-line-tool","crowdsourcing","mechanical-turk","mturk"],"created_at":"2025-10-13T16:01:07.762Z","updated_at":"2025-10-13T16:04:40.425Z","avatar_url":"https://github.com/allenai.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"amti\n====\nA Mechanical Turk Interface (amti)\n\n`amti` is a CLI for Mechanical Turk that emphasizes the\nability to quickly *iterate* on and run *reproducible* crowdsourcing\nexperiments.\n\nDesign and deploy HITs to Amazon Mechanical Turk in a way that:\n\n  1. allows HIT definitions to be tracked in version control.\n  2. can manage and generate batches of HITs from JSON data.\n  3. stores the results from HITs in a structured format on disk or in\n     the cloud.\n\nTo get started as a user, see [Installation](#installation) and\n[Quickstart](#quickstart) below. To develop `amti` see\n[Development Setup](#development-setup) and\n[Contributing](#contributing).\n\n\nInstallation\n------------\n`amti` requires Python 3.6. To install `amti`, currently you should just\ninstall from source:\n\n    pip install git+https://github.com/allenai/amti#egg=amti\n\n`amti` is now on the path of whatever environment you installed the\npackage into.\n\nAdditionally, you'll need to make sure that your environment is setup to\nuse Mechanical Turk through [`boto3`][boto3] as described in\n[this tutorial][python-mturk-tutorial].\n\n[boto3]: http://boto3.readthedocs.io/en/latest/reference/services/mturk.html\n[python-mturk-tutorial]: https://blog.mturk.com/tutorial-a-beginners-guide-to-crowdsourcing-ml-training-data-with-python-and-mturk-d8df4bdf2977\n\n\nQuickstart\n----------\nIn this section, we'll walk through an example use case of `amti` to run\na batch of `HTMLQuestion` HITs on Mechanical Turk. If you'd like more\nbackground information beforehand, feel free to jump to the\n[Concepts](#concepts) section.\n\n`amti` comes built-in with two examples, an `HTMLQuestion` and an\n`ExternalQuestion`. The `HTMLQuestion` is a form written in HTML that\nMTurk hosts for you. The `ExternalQuestion` provides an iframe to your\nwebsite that eventually posts back to an MTurk endpoint. Find these\nexamples in [examples directory][examples-directory].\n\nWe'll walk through the `HTMLQuestion` example. All requests are made to\nthe Mechanical Turk sandbox unless you pass the `--live` flag, so feel\nfree to run this example without worrying about posting to the live\nsite.\n\n  1. Create your batch by running:\n\n         amti create-batch examples/html-question/definition examples/html-question/data.jsonl .\n\n     This command creates a batch directory using the definition in\n     `example/html-question/definition` and data in\n     `examples/html-question/data.jsonl`. It saves the batch directory\n     into our current directory (`.`) and uploads HITs for it to MTurk.\n\n     You can see that the batch has been created in the current\n     directory by running `ls`.\n\n  2. Now, check the status of your batch by running:\n\n         amti status-batch batch-*\n\n     Where `batch-*` just needs to match the path to the batch directory\n     that was created.\n\n  3. Find and fill out the example HITs on MTurk in the\n     [worker sandbox][worker-sandbox]. You'll want to search for the\n     HITs using your requester account's user name.\n\n  4. View the status of your now completed HITs with:\n\n         amti status-batch batch-*\n\n  5. (optional) If you want to cancel all the HITs in the batch:\n\n         amti expire-batch batch-*\n\n  6. Since your batch is now ready for review, review it with:\n\n         amti review-batch batch-*\n\n  7. Once you've approved all of your HITs, you can save the results\n     from the batch:\n\n         amti save-batch batch-*\n\n     If you go into the batch directory, you'll now notice that it has a\n     new `results` subdirectory with the information on your HITs and\n     assignments.\n\n  8. Since you've saved your batch, dispose of it from the MTurk site\n     using:\n\n         amti delete-batch batch-*\n\n     This action will delete the batch from MTurk so that it doesn't pop\n     up when you examine your open HITs; however, it leaves your batch\n     directory intact and unchanged.\n\n  9. Lastly, you can extract data from all the assignments you've saved\n     into your batch directory using any of the `extract` commands. To\n     view the available extraction formats, pass the `--help` option:\n\n         $ amti extract --help\n         Usage: amti extract [OPTIONS] COMMAND [ARGS]...\n\n           Extract data from a batch to various formats.\n\n           See the subcommands for extracting batch data into a specific format.\n\n         Options:\n           -h, --help  Show this message and exit.\n\n         Commands:\n           tabular  Extract data from BATCH_DIR to OUTPUT_PATH in a tabular format.\n           xml      Extract XML data from assignments in BATCH_DIR to OUTPUT_DIR.\n\n     The `tabular` command will extract the batch's data into an easy to\n     work with tabular format:\n\n         amti extract tabular batch-* batch-data.jsonl\n\n     For real workflows, it would be a good idea to use the batch id in\n     the name of the output file.\n\nNow you've run a small HIT and have the results in a reproducible\nformat. It's easy to tar up and upload the batch directory to the cloud\nwhere you can store information from many such HITs.\n\nWhen developing your own `HTMLQuestion` HITs, you may want to preview\nthem locally before uploading to Mechanical Turk, with the\n`preview-batch` command:\n\n    amti preview-batch /path/to/definition/directory /path/to/data/file\n\n\n[examples-directory]: ./examples/\n[worker-sandbox]: https://workersandbox.mturk.com/\n\n\nOverview\n--------\n`amti` may be used both as a command line interface for working with\nMechanical Turk as well as a library for scripting on top of Mechanical\nTurk.\n\nFirst, we'll discuss the major [Concepts](#concepts) you'll want to know\nwhen working with `amti`, then we'll describe the\n[CLI](#command-line-interface), and lastly we'll talk about using `amti`\nas a [library](#library).\n\n### Concepts\n\n#### Mechanical Turk Concepts\n\nThe following are the major Mechanical Turk concepts. Most concepts\ncorrespond to a resource endpoint in\n[their ReSTful API][mturk-api]. We've linked each concept to the\nrelevant endpoint's documentation, where available:\n\n  - [**HIT**][hit-doc]: A HIT (Human Intelligence Task) corresponds to a\n    task that a Turker can perform. Usually, a HIT ends up being an HTML\n    form that the Turker can submit. An example HIT could be \"enter\n    three tags for this image\", where an image is also displayed on a\n    web page. **Note** that a HIT is a specific task, not a kind of\n    task. So, image labeling is not a HIT; but rather, labeling a\n    specific image would be a HIT.\n  - **HIT Type**: Because many HITs will be similar, there's a notion of\n    HIT Type which describes a group of HITs. In particular, HIT Types\n    have descriptions of the task, define a reward amount, title for the\n    task, and other properties that are generally shared across multiple\n    HITs.\n  - [**Assignment**][assignment-doc]: Often it's desirable to have a HIT\n    completed multiple times by different people. An assignment is a\n    single opportunity to complete a HIT, and a crowdworker can't take\n    multiple assignments for one HIT. So, an image that should be\n    labeled by 3 people would be posted as one HIT with 3 assignments.\n\n[mturk-api]: https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/Welcome.html\n[hit-doc]: https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/ApiReference_HITDataStructureArticle.html\n[assignment-doc]: https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/ApiReference_AssignmentDataStructureArticle.html\n\n#### `amti` Concepts\n\nMechanical Turk has some features that support creating batches of HITs;\nhowever, they're not particularly well developed and aren't modeled by\nthe API. `amti`'s key concept is that of a *batch*. A *batch* is a\ncollection of HITs generated from some data using a template.\n\n`amti` represents batches as directories with the following structure:\n\n    batch-$BATCHID : root directory for the batch\n    |- README : a text file for developers about the batch\n    |- COMMIT : the commit SHA for the code that generated the batch\n    |- BATCHID : a random UUID for the batch\n    |- definition : files defining the HIT / HIT Type\n    |  |- NOTES : any notes for developers that go along with the task\n    |  |- question.xml.j2 : a jinja2 template for the HITs' question\n    |  |- hittypeproperties.json : properties for the HIT Type\n    |  |- hitproperties.json : properties for the HIT\n    |- data.jsonl : data used to generate each HIT in the batch\n    |- results : results from the HITs on the MTurk site\n    |  |- hit-$ID : results for a single HIT from the batch\n    |  |  |- hit.jsonl : data about the HIT from the MTurk site\n    |  |  |- assignments.jsonl : results from the assignments\n    |  |- ...\n\nTo create a batch, write a batch definition (see the\n[example batch definition][example-batch-definition]), create some data\nin the [JSON Lines][json-lines] format, and then create the batch using\n`amti create-batch`. Use the `-h` option for details. You can find some\nexample data in the [`data.jsonl`][data-jsonl] file.\n\nTo check on the batch's status, use `amti status-batch`. Once the batch\nhas been fully worked by Turkers, you can manually review their work\nwith `amti review-batch`. After approving or rejecting all the HITs in\nthe batch, you can save the batch to disk with `amti\nsave-batch`. Finally, after saving the batch, you can delete all of its\nHITs with `amti delete-batch`. Again, use `-h` for details.\n\n[example-batch-definition]: ./examples/html-question/definition\n[json-lines]: http://jsonlines.org/\n[data-jsonl]: ./examples/html-question/data.jsonl\n\n### Command Line Interface\n\nTo use `amti` as a CLI for Mechanical Turk, [install](#installation)\n`amti` and then call it by typing `amti` at the command line:\n\n    $ amti --help\n    Usage: amti [OPTIONS] COMMAND [ARGS]...\n\n      A Mechanical Turk Interface: a CLI for MTurk.\n\n    Options:\n      -v, --verbose  Set log level to DEBUG.\n      -h, --help     Show this message and exit.\n\n    Commands:\n      associate-qual            Associate workers with a qualification.\n      block-workers             Block workers by WorkerId.\n      create-batch              Create a batch of HITs using DEFINITION_DIR and...\n      create-qualificationtype  Create a Qualification Type using...\n      delete-batch              Delete the batch of HITs defined in BATCH_DIR.\n      disassociate-qual         Disassociate workers with a qualification.\n      expire-batch              Expire all the HITs defined in BATCH_DIR.\n      extract                   Extract data from a batch to various formats.\n      notify-workers            Send notification message to workers.\n      preview-batch             Preview a batch of rendered HITs using...\n      review-batch              Review the batch of HITs defined in BATCH_DIR.\n      save-batch                Save results from the batch of HITs defined in...\n      status-batch              View the status of the batch of HITs defined in...\n      unblock-workers           Unblock workers by WorkerId.\n\nThe CLI is self-documenting and hierarchical, so you should be able to\nfind anything you might need by starting from the top and using the `-h`\noption.\n\n### Library\n\nTo use `amti` as a library, pay attention to the two main subpackages:\n\n  - [`amti/actions`][amti-actions]: Functions implementing all the\n    actions used by `amti`.\n  - [`amti/clis`][amti-clis]: CLIs for many of the actions used by\n    `amti`. All CLI components are implemented using [`click`][click]\n    and so can be reused in other applications.\n\n[amti-actions]: ./amti/actions/\n[amti-clis]: ./amti/clis/\n[click]: http://click.pocoo.org/5/\n\n\nDevelopment Setup\n-----------------\nThis setup guide assumes you have [`pyenv`][pyenv],\n[`pyenv-virtualenv`][pyenv-virtualenv], and [`direnv`][direnv] installed\non your machine.\n\nFrom the root of this repo, create a python environment for `amti` and\ninstall the dependencies:\n\n    pyenv install 3.6.4\n    pyenv virtualenv 3.6.4 amti\n    echo 'amti' \u003e .python-version\n    pip install -r requirements.txt\n\nThen, make sure that you have the proper\n[AWS environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html)\nset in your `.envrc` file for this repo. In particular, you should have values\nfor either:\n\n    AWS_ACCESS_KEY_ID\n    AWS_SECRET_KEY\n    AWS_SECRET_ACCESS_KEY\n\nor\n\n    AWS_PROFILE\n\nThat correspond to your Mechanical Turk account.\n\n[pyenv]: https://github.com/pyenv/pyenv\n[pyenv-virtualenv]: https://github.com/pyenv/pyenv-virtualenv\n[direnv]: https://direnv.net/\n\n\nContributing\n------------\n`amti` is licensed under [Apache 2.0][apache2-license]. Feel free to\nfork the project or do whatever you like under the terms of that\nlicense.\n\nFor inquiries about the project, please file a GitHub issue. If you find\na bug or error in the code, pull requests are strongly preferred.\n\n[apache2-license]: https://www.apache.org/licenses/LICENSE-2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallenai%2Famti","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fallenai%2Famti","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallenai%2Famti/lists"}