{"id":28529399,"url":"https://github.com/mlcommons/modelplane","last_synced_at":"2025-10-24T13:04:59.439Z","repository":{"id":294537369,"uuid":"987297626","full_name":"mlcommons/modelplane","owner":"mlcommons","description":null,"archived":false,"fork":false,"pushed_at":"2025-10-01T14:50:45.000Z","size":1455,"stargazers_count":1,"open_issues_count":14,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-10-01T16:35:10.941Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mlcommons.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-20T21:46:49.000Z","updated_at":"2025-09-04T14:23:25.000Z","dependencies_parsed_at":"2025-06-04T12:55:46.411Z","dependency_job_id":"17b6a6fe-953b-475e-ac13-f5887b449691","html_url":"https://github.com/mlcommons/modelplane","commit_stats":null,"previous_names":["mlcommons/modelplane"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mlcommons/modelplane","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fmodelplane","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fmodelplane/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fmodelplane/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fmodelplane/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mlcommons","download_url":"https://codeload.github.com/mlcommons/modelplane/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fmodelplane/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280799423,"owners_count":26393089,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-24T02:00:06.418Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-09T14:10:34.657Z","updated_at":"2025-10-24T13:04:59.427Z","avatar_url":"https://github.com/mlcommons.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# modelplane - an AI evaluator development platform\n\n## ⚠️ Content warning\n\nThe sample datasets provided in the [`flightpaths/data`](https://github.com/mlcommons/modelplane/tree/main/flightpaths/data) \ndirectory are a truncated version of the datasets provided [here](https://github.com/mlcommons/ailuminate).\nThese data come with the following warning:\n\n\u003eThis dataset was created to elicit hazardous responses. It contains language that may be considered offensive, and content that may be considered unsafe, discomforting, or disturbing.\n\u003eConsider carefully whether you need to view the prompts and responses, limit exposure to what's necessary, take regular breaks, and stop if you feel uncomfortable.\n\u003eFor more information on the risks, see [this literature review](https://www.zevohealth.com/wp-content/uploads/2024/07/lit_review_IN-1.pdf) on vicarious trauma.\n\n## Quickstart\n\nYou must have a docker engine installed on your system. The given\ndocker-compose.yaml file has definitions for running the following services\nlocally:\n\n* mlflow tracking server + postgres\n* jupyter\n\nFirst, clone this repo:\n```bash\ngit clone https://github.com/mlcommons/modelplane.git\ncd modelplane\n```\n\nIf you plan to share notebooks, clone \n[modelplane-flights](https://github.com/mlcommons/modelplane-flights) as well. Both `modelplane`\nand `modelplane-flights` should be in the same directory.\n\nFinally, set up secrets for accessing SUTs, as needed in \n`modelplane/flightpaths/config/secrets.toml`. See [modelbench](https://github.com/mlcommons/modelbench) for more details.\n\n\n### Running jupyter locally against the MLCommons mlflow server.\n\n1. Ensure you have access to the MLCommons mlflow tracking \nand artifact server. If not, email \n[airr-engineering@mlcommons.org](mailto:airr-engineering@mlcommons.org)\nfor access.\n1. Modify `.env.jupyteronly` to include  your credentials for the\nMLFlow server (`MLFLOW_TRACKING_USERNAME` /\n`MLFLOW_TRACKING_PASSWORD`).\n    * Alternatively, put the credentials in `~/.mlflow/credentials` as described [here](https://mlflow.org/docs/latest/ml/auth/#credentials-file).\n1. To access `modelbench-private` code (assuming you have \naccess), you must also set `USE_MODELBENCH_PRIVATE=true` in `.env.jupyteronly`. This will forward your ssh agent to the container\nallowing it to load the private repository to build the image.\n1. Start jupyter with `./start_jupyter.sh`. (You can add the\n`-d` flag to start in the background.)\n\n### Running jupyter and mlflow locally.\n\n1. Adjust the `.env` file as needed. The committed `.env` / \n`docker-compose.yaml` will bring up mlflow, postgres, jupyter, and set up mlflow to use a local disk for artifact storage.\n1. Start services with `./start_services.sh`. (You can add the\n`-d` flag to start in the background.)\n\n    * If you are using the cli only, and not using jupyter, you must pass the `--no-jupyter` option: \n    `./start_services.sh -d`\n\n## Getting started in JupyterLab.\n\n1. Visit the [Jupyter Server](http://localhost:8888/lab?token=changeme). The\n   token is configured in the .env file. You shouldn't need to enter it \n   more than once (until the server is restarted). You can get started with\n   the template notebook or create a new one.\n1. You should see the `flights` directory, which leads to the\n`modelplane-flights` repository. Create a user directory\nfor yourself (`flights/users/{username}`) and either\ncopy an existing flightpath there or create a notebook from\nscratch.\n    * You can manage branches and commits for \n    `modelplane-flights` directly from jupyter.\n\n## Caching\n\nAnnotator and SUT responses will be cached (locally) unless you pass the\n`disable_cache` flag to the appropriate calls.\n\n## CLI\n\nYou can also interact with modelplane via CLI. Run `poetry run modelplane --help`\nfor more details.\n\n*Important:* You must set the `MLFLOW_TRACKING_URI` environmental variable.\nFor example, if you've brought up MLFlow using the fully local docker compose process above,\nyou could run:\n```\nMLFLOW_TRACKING_URI=http://localhost:8080 poetry run modelplane get-sut-responses --sut_id {sut_id} --prompts tests/data/prompts.csv --experiment expname\n```\nAfter running the command, you'd see the `run_id` in the output from mlflow, \nor you can get the `run_id` via the MLFlow UI.\n\n### Basic Annotations\n```\nMLFLOW_TRACKING_URI=http://localhost:8080 poetry run modelplane annotate --annotator_id {annotator_id} --experiment expname --response_run_id {run_id}\n```\n\n### Custom Ensembles\n```\nMLFLOW_TRACKING_URI=http://localhost:8080 poetry run modelplane annotate --annotator_id {annotator_id1} --annotator_id {annotator_id2} --ensemble_strategy {ensemble_strategy} --experiment expname --response_file path/to/response.csv\n```\n\n### Private Ensemble\nIf you have access to the private ensemble, you can install with the needed extras\n```\npoetry install --extras modelbench-private\n```\nAnd then run annotations with:\n```\nMLFLOW_TRACKING_URI=http://localhost:8080 poetry run modelplane annotate --ensemble_id official --experiment expname --response_run_id {run_id}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlcommons%2Fmodelplane","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmlcommons%2Fmodelplane","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlcommons%2Fmodelplane/lists"}