{"id":15519914,"url":"https://github.com/ahalterman/multiuser_prodigy","last_synced_at":"2025-04-23T03:51:27.317Z","repository":{"id":49976457,"uuid":"117615454","full_name":"ahalterman/multiuser_prodigy","owner":"ahalterman","description":"Running Prodigy for a team of annotators","archived":false,"fork":false,"pushed_at":"2021-01-08T13:14:11.000Z","size":1359,"stargazers_count":53,"open_issues_count":1,"forks_count":7,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-17T19:17:35.474Z","etag":null,"topics":["prodigy","spacy"],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ahalterman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-01-16T01:36:47.000Z","updated_at":"2023-09-01T20:24:48.000Z","dependencies_parsed_at":"2022-07-31T01:50:03.117Z","dependency_job_id":null,"html_url":"https://github.com/ahalterman/multiuser_prodigy","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahalterman%2Fmultiuser_prodigy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahalterman%2Fmultiuser_prodigy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahalterman%2Fmultiuser_prodigy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahalterman%2Fmultiuser_prodigy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ahalterman","download_url":"https://codeload.github.com/ahalterman/multiuser_prodigy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250366685,"owners_count":21418768,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["prodigy","spacy"],"created_at":"2024-10-02T10:23:37.401Z","updated_at":"2025-04-23T03:51:27.300Z","avatar_url":"https://github.com/ahalterman.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# multiuser_prodigy\n\nThis is a multi-annotator setup for [Prodigy](http://prodi.gy/),\nExplosion AI's data annotation tool, that uses a Mongo DB to allocate\nannotation tasks to annotators working on different Prodigy instances running\non seperate ports. This use case focuses on collecting gold standard\nannotations from a team of annotators using Prodigy, rather than on the active\nlearning, single-annotator setup that Prodigy is primarily intended for.\n\nThere are a few examples of annotation interfaces in the repo, including code\nfor annotators working on training an NER model or doing sentence\nclassification with document context.  Each annotator works on the Prodigy/port\nassigned to them, and a new `DBStream` class handles pulling the examples from\nProdigy that are assigned to each worker.\n\nI've used this setup for three major annotation projects now, but you'll need\nto modify the code to get it working for your project as well.\n\n## Mongo database\n\nAll tasks are stored in a Mongo DB, which allows different logic for how tasks\nare assigned to annotators. For instance, examples can go out to annotators\nuntil three annotations are collected, examples could go to two predetermined\nannotators from the wider pool, or annotations can be automatically resubmitted\nto a third annotator if the first two annotations disagree.\n\nYou can start a Mongo DB in a Docker container:\n\n```\nsudo docker run -d -p 127.0.0.1:27017:27017 -v /home/andy/MIT/multiuser_prodigy/db:/data/db  mongo\n```\n\nTo load a list of tasks into the database:\n\n```\npython mongo_load.py -i assault_not_assault.jsonl -c \"assault_gsr\"\n```\n\nwhere `-i` is a JSONL file of tasks and `-c` specifies the collection name to\nload them into.\n\n\"seen\" : {\"$in\" : [0,1]}},\n            {\"coders\"\n\n## Running\n\nYou'll need to modify the code of `multiuser_db.py` to access the right\ncollection, set the names/ports of annotators, and the desired interface (NER,\nclassification, etc).\n\nThen you should launch the processes either in a `screen` or in the background:\n\n```\npython multiuser_db.py\n```\n\n## Analysis\n\n![](streamlit_dashboard.png)\n\nYou can use Streamlit to set up a dashboard so annotators can check their\nprogress. This one pulls results from the Mongo DB, but you could also call the\nProdigy DB and show results from there.\n\n\nA more complicated analysis dashboard setup is in \n`Report.Rmd`. This RMarkdown file reads in a CSV of coding information and\ngenerates figures in an HTML page that can be served from the annotation\nserver. To record information about how long each task takes, add something\nlike `eg['time_loaded'] = datetime.now().isoformat()` to your stream code and\nsomething like `eg['time_returned'] = datetime.now().isoformat()` to your\nupdate code. `report_maker.py` exports the DB to CSV and knits the RMarkdown on\nthat CSV.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahalterman%2Fmultiuser_prodigy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fahalterman%2Fmultiuser_prodigy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahalterman%2Fmultiuser_prodigy/lists"}