{"id":43916887,"url":"https://github.com/zooniverse/hamlet","last_synced_at":"2026-02-06T21:34:56.416Z","repository":{"id":36373444,"uuid":"184289888","full_name":"zooniverse/hamlet","owner":"zooniverse","description":"Produces subject set and classification exports suitable for AutoML","archived":false,"fork":false,"pushed_at":"2024-07-24T10:10:03.000Z","size":1073,"stargazers_count":5,"open_issues_count":17,"forks_count":0,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-07-24T11:28:48.746Z","etag":null,"topics":["automl","django","machine-learning","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zooniverse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-30T15:46:40.000Z","updated_at":"2024-07-24T10:10:05.000Z","dependencies_parsed_at":"2024-07-24T11:28:38.657Z","dependency_job_id":null,"html_url":"https://github.com/zooniverse/hamlet","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/zooniverse/hamlet","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zooniverse%2Fhamlet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zooniverse%2Fhamlet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zooniverse%2Fhamlet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zooniverse%2Fhamlet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zooniverse","download_url":"https://codeload.github.com/zooniverse/hamlet/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zooniverse%2Fhamlet/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29177554,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-06T20:14:21.878Z","status":"ssl_error","status_checked_at":"2026-02-06T20:14:21.443Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automl","django","machine-learning","python"],"created_at":"2026-02-06T21:34:56.327Z","updated_at":"2026-02-06T21:34:56.396Z","avatar_url":"https://github.com/zooniverse.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hosted AutoML Export Transformer (Hamlet)\n\nHamlet is a website that connects Zooniverse data from Panoptes with external data services, e,g. sending a camera trap project's animal-filled photos to an animal-identifying machine-learning system.\n\n## Auto ML Export\n\n## Subject Assistant\n\nHamlet has an export feature that ties into the Zooniverse Machine Learning Subject Assistant, [(app)](https://subject-assistant.zooniverse.org/) [(source)](https://github.com/zooniverse/zoo-ml-subject-assistant) which lets project owners/researchers submit their camera trap photos to an external Machine Learning (ML) service, which in turn finds animals in those images.\n\n### User Story\n\nThe user story is as follows:\n- Users start at the Subject Assistant app.\n- Users are directed to Hamlet, where they choose a Subject Set to export to the external ML Service.\n- Hamlet performs the export feature, and provides users with a link back to the Subject Assistant with an \"ML Task ID\" - e.g. `https://subject-assistant.zooniverse.org/#/tasks/6378`\n- Users click that link, and process the ML-tagged photos on the Subject Assistant app.\n\n### External Dependencies\n\nThe Subject Assistant requires the following external systems:\n\n- Machine Learning Service - in this case, powered by Microsoft.\n- an Azure Storage Container - works in conjunction with the ML Service, which requires \"subject manifest\" files to be stored on Azure.\n\nAs of late 2022, these services are maintained by the Zooniverse team.\n\n### Environmental Variables\n\nThe Subject Assistant feature requires the following ENV variables defined:\n\n- `SUBJECT_ASSISTANT_AZURE_ACCOUNT_NAME`\n- `SUBJECT_ASSISTANT_AZURE_ACCOUNT_KEY`\n- `SUBJECT_ASSISTANT_AZURE_CONTAINER_NAME`\n- `SUBJECT_ASSISTANT_ML_SERVICE_CALLER_ID` - provided by our friends in Microsoft who run the ML Service.\n- `SUBJECT_ASSISTANT_ML_SERVICE_URL` - ditto\n\nOptionally, the following ENV variables can be defined:\n\n- `SUBJECT_ASSISTANT_EXTERNAL_URL` - defaults to `http://subject-assistant.zooniverse.org/#/tasks/`\n\n### Mechanics: Django Pages/Views\n\nThe ML Subject Assistant feature in Hamlet has two views:\n\n- `GET /subject-assistant/\u003cint:project_id\u003e/` - lists all the Subject Sets for a Project, along with their \"ML export\" status and (if the export is successful) a link back to the Subject Assistant app.\n- `POST /subject-assistant/\u003cint:project_id\u003e/subject-sets/\u003cint:subject_set_id\u003e/` - performs the ML Export action for a given Subject Set, then redirects users back to the listing page.\n\n### Mechanics: Database Model\n\nThe `MLSubjectAssistantExport` table has the following fields:\n\n- subject_set_id - the ID of the Zooniverse Subject Set that was exported to the external ML Service\n- json - the \"subject manifest\" file, in JSON format, created from all the Subjects of the Subject Set. The format is specific to the ML Service.\n- azure_url - the URL of the \"subject manifest\" file that was uploaded to an external Azure storage container. (See Mechanics: ML Export Action for why)\n- ml_task_uuid - the task request ID or \"job ID\" for the ML Export action. This is generated by the external ML Service.\n\n### Mechanics: ML Export Action\n\nMechanically, the ML Subject Assistant's \"export to Microsoft\" action performs the following:\n\n1. get all the Subjects for a given Subject Set (pulling from Panoptes)\n2. create a JSON file - the \"subject manifest\" - that describes the Subjects to be exported, in a format specified by the external ML Service.\n3. upload the JSON file to an external Azure storage container (reason: the current external ML Service only reads subject manifest files from Azure), then create a \"shareable URL\" to that JSON file. (Clarification: Azure uses a SAS or Shared Access Signature tokens to create shareable URLs with limited lifespans.)\n4. Submit the shareable URL to the ML Service, and get the \"job ID\" it returns.\n\nThe Job ID plus the known Subject Assistant app URL is all that's required to construct a \"return URL\" for the user.\n\n## Development\n\nUse docker \u0026 docker-compose to setup a development env.\n\n1. Run `docker-compose build` to build the app container.\n2. Run the tests `docker-compose run -T --rm app bundle exec pytest --cov=hamlet`\n\nAlternatively you can use docker \u0026 compose to run an interactive bash shell for development and testing\n\n1. Run `docker-compose run --service-ports --rm app bash` to start the containers\n2. Run `pytest --cov=hamlet` to run the test suite in that shell (sadly this system has no tests :sadpanda:)\n3. Or `./start_server.sh` to run the server (see Pipfile)\n\n### Troubleshooting\n\n**I can't login on local development**\n\nProblem:\n- You're able to run `docker-compose build ; docker-compose up`, and you can view Hamlet on local development on `http://localhost:8080`\n- However, when you click on the \"Login with Zooniverse\" button and provide your details on the Panoptes login page, you\n\nAnalysis:\n- It's likely that your instance of Hamlet is missing the `PANOPTES_APPLICATION_ID` and `PANOPTES_SECRET` environment variables.\n- These env vars are required to tell Panoptes _which oAuth application_ you're logging into.\n\nSolution:\n- Go to [Panoptes's oAuth applications list,](https://panoptes.zooniverse.org/oauth/applications) find the Hamlet app, and copy the Application ID and Secret\n- Add these to your local development Docker's environment variables, as `PANOPTES_APPLICATION_ID` and `PANOPTES_SECRET`\n  - This can be done easily by creating a `.env` file in the root folder of your `hamlet` repo.\n\nRelated issue: [479](https://github.com/zooniverse/hamlet/issues/479)\n\n**The database won't start on local development**\n\nProblem:\n- When you run `docker-compose build ; docker-compose up`, you notice that the PostgreSQL database isn't running. \n- There's probably a few error message in the console: `app_1` will continuously complain that it's trying (and failing) to find the PostgreSQL database, while `postgres_1` might say something about \"can't initialise due to incompatible database\".\n\nAnalysis:\n- It's possible that your existing local PostgreSQL database (i.e. the `/postgres_data` folder) was built on an older version of PostgreSQL, and recent updates to Hamlet have upgraded the PostgreSQL that Hamlet uses, causing an incompatibility.\n\nSolution:\n- Check if you have an existing `/postgres_data` folder in your local `hamlet` repo.\n- If yes, delete it. The next time you start Hamlet, the database will be rebuilt with the latest version. \n\n### Useful application scripts\n\n- console: `python manage.py shell`\n- create_local_db: `createdb -U halmet -O hamlet hamlet`\n- drop_local_db: `dropdb -U hamlet hamlet`\n- makemigrations: `python manage.py makemigrations`\n- migrate `python manage.py migrate`\n- server: `bash -e ./start_server.sh`\n- tests: `pytest --cov=hamlet`\n- tree: `bash -c 'find . | grep -v git | grep -v cache'`\n- worker: `bash -c ./start_worker.sh`\n\n### Updating a package with peotry\n\n- `poetry update django`\n\nSee [Poetry docs](https://python-poetry.org/docs/basic-usage/#installing-dependencies) for more details\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzooniverse%2Fhamlet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzooniverse%2Fhamlet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzooniverse%2Fhamlet/lists"}