{"id":50850348,"url":"https://github.com/mlcommons/endpoints-submission-cli","last_synced_at":"2026-06-14T13:31:56.055Z","repository":{"id":362720565,"uuid":"1243750864","full_name":"mlcommons/endpoints-submission-cli","owner":"mlcommons","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-12T17:56:02.000Z","size":463,"stargazers_count":1,"open_issues_count":7,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-12T19:22:30.386Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mlcommons.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-19T16:20:14.000Z","updated_at":"2026-06-11T02:33:59.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mlcommons/endpoints-submission-cli","commit_stats":null,"previous_names":["mlcommons/endpoints-submission-cli"],"tags_count":13,"template":false,"template_full_name":"mlcommons/template","purl":"pkg:github/mlcommons/endpoints-submission-cli","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fendpoints-submission-cli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fendpoints-submission-cli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fendpoints-submission-cli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fendpoints-submission-cli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mlcommons","download_url":"https://codeload.github.com/mlcommons/endpoints-submission-cli/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fendpoints-submission-cli/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34323995,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-14T02:00:07.365Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-14T13:31:55.501Z","updated_at":"2026-06-14T13:31:56.049Z","avatar_url":"https://github.com/mlcommons.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MLCommons Endpoints Submission Tools\n\nA Python package with two tools for managing MLPerf Endpoints benchmark submissions:\n\n- **`endpoints-submission-cli`** — registers benchmark runs, assembles submission packages, runs compliance checks, and opens GitHub pull requests via the PRISM API.\n- **`submission-checker`** — validates a submission folder against the §9.1 automated compliance rules before or after upload.\n\n---\n\n## Installation\n\n**With pip:**\n\n```bash\npip install endpoints-submission-cli\n```\n\n**From source (editable):**\n\n```bash\npip install -e \".[dev]\"\n```\n\n**With [uv](https://github.com/astral-sh/uv):**\n\n```bash\nuv sync --extra dev\n```\n\n---\n\n# endpoints-submission-cli\n\n## Requirements\n\n- Python 3.10 or later\n- [`gh` CLI](https://cli.github.com/) — required for creating, updating, and withdrawing submissions\n\n## Authentication\n\nEvery command requires a PRISM API token in `mlc_…` format. Supply it as an env var or pass `--token` per command:\n\n```bash\n# Persistent (add to shell profile)\nexport PRISM_USER_API_TOKEN=mlc_your_token_here\n\n# Per-command override\nendpoints-submission-cli runs list --token mlc_your_token_here\n```\n\nSubmission commands that create or update GitHub pull requests also require the `gh` CLI:\n\n```bash\ngh auth login\n```\n\n## Configuration\n\n| Environment variable | Default | Description |\n|---|---|---|\n| `PRISM_USER_API_TOKEN` | — | API key. Required unless `--token` is passed. |\n| `MLPERF_SUBMISSION_REPO` | `MLCommons-Systems/test-endpoints-submission-repo` | Target GitHub repository for submission PRs (`owner/repo`). |\n\nAdd to your shell profile for a persistent setup:\n\n```bash\nexport PRISM_USER_API_TOKEN=mlc_your_token_here\nexport MLPERF_SUBMISSION_REPO=MLCommons-Systems/endpoints-submission-repo\n```\n\n## Quick start\n\n```bash\n# 1. Verify connectivity\nendpoints-submission-cli runs list\n\n# 2. Register a benchmark run from a local result folder\nendpoints-submission-cli runs create --path /results/llama3_h100_c4\n# → Run created: d5d9873e-5eca-4f8d-a487-4be1cb8b440c\nRUN_ID=d5d9873e-5eca-4f8d-a487-4be1cb8b440c\n\n# 3. Create a submission (assembles, checks, uploads, opens PR)\nendpoints-submission-cli submissions create \\\n  --division standardized \\\n  --availability available \\\n  --run-ids $RUN_ID\n# → Submission created: a1b2c3d4-…\n# → PR: https://github.com/MLCommons-Systems/…/pull/42\nSUB_ID=a1b2c3d4-e5f6-7890-abcd-ef1234567890\n\n# 4. Add another run later\nendpoints-submission-cli submissions add-run \\\n  --submission-id $SUB_ID \\\n  --run-id \u003cnew-run-id\u003e\n\n# 5. Withdraw if needed\nendpoints-submission-cli submissions withdraw --submission-id $SUB_ID\n```\n\n## Command reference\n\n```\nendpoints-submission-cli\n├── runs\n│   ├── list        List all runs\n│   ├── create      Register a run from a local folder\n│   ├── get         Fetch run details\n│   ├── delete      Delete a run and its archive\n│   ├── pin         Pin a run (prevent expiry)\n│   └── unpin       Restore normal expiry\n└── submissions\n    ├── list        List all submissions\n    ├── create      Create a submission from runs (full pipeline)\n    ├── get         Fetch submission details\n    ├── update      Update run list or metadata\n    ├── withdraw    Withdraw a submission\n    ├── add-run     Add a run to an existing submission\n    └── remove-run  Remove a run from a submission\n```\n\nUse `--help` on any command for full flag details:\n\n```bash\nendpoints-submission-cli submissions create --help\n```\n\n---\n\n# submission-checker\n\nCLI tool for validating MLPerf Endpoints submissions against the §9.1 automated compliance checks.\n\n## Usage\n\n### Check a submission\n\n```bash\nsubmission-checker check /path/to/submission\n```\n\nThe tool expects the submission root to contain `systems/` and `pareto/` subdirectories as specified in §8.1.\n\n**Options:**\n\n| Flag | Description |\n|------|-------------|\n| `--strict` | Treat warnings as errors (exit 1 on any warning) |\n| `--quiet` / `-q` | Suppress INFO-level passing checks |\n| `--output FILE` / `-o FILE` | Write full results as JSON to *FILE* |\n\n**Exit codes:** `0` = all checks passed, `1` = one or more errors (or warnings with `--strict`).\n\n### Show region boundaries\n\n```bash\nsubmission-checker regions --max-concurrency 1024\n```\n\nPrints the concurrency ranges for each region given a declared Maximum Supported Concurrency *M* (§5.5).\n\n## Required Files in submission structure\n\n```\n\u003corg\u003e/\n├── systems/\n│   └── \u003csystem_desc_id\u003e.json         # §8.2 — hardware + software description\n└── pareto/\n    └── \u003csystem_desc_id\u003e/\n        └── \u003cbenchmark_model\u003e/\n            ├── points/\n            │   └── point_\u003cN\u003e.yaml    # §8.3 — one config per measurement point\n            ├── results/\n            │   └── point_\u003cN\u003e/\n            │       ├── mlperf_endpoints_log_summary.json\n            │       └── mlperf_endpoints_log_detail.json\n            └── accuracy/\n                ├── accuracy.txt\n                └── accuracy_result.json\n```\n\n## What gets checked\n\n| Rule | Spec | Description |\n|------|------|-------------|\n| `path-exists` | §1 | Submission root directory exists |\n| `required-dir` | §1 | `systems/` and `pareto/` present |\n| `system-description-present` | §1 | At least one `*.json` file found in `systems/` |\n| `system-description-valid` | §1 | `systems/*.json` parses against schema |\n| `src-dir` | §1 | `src/` present for Standardized submissions |\n| `pareto-dir-exists` | §1 | `pareto/\u003csystem_id\u003e/` directory exists |\n| `benchmark-model-dir` | §1 | At least one benchmark-model directory in `pareto/\u003csystem_id\u003e/` |\n| `pareto-subdir` | §1 | `points/`, `results/`, `accuracy/` present |\n| `measurement-points-present` | §1 | At least one `point_*.yaml` found |\n| `point-config-valid` | §1 | YAML parses against `PointConfig` schema |\n| `point-filename-concurrency` | §1 | Filename concurrency matches declared value |\n| `result-file-present` | §1 | Result summary log exists for each point config |\n| `result-detail-present` | §1 | Result detail log exists for each point config |\n| `result-file-valid` | §1 | Result summary log parses against `PointSummary` schema |\n| `point-count` | §2, §8 | 7–32 measurement points |\n| `point-cap` | §2, §8 | Point count does not exceed 32 |\n| `low-latency-coverage` | §3 | At least one point in Low Latency region |\n| `low-throughput-coverage` | §4 | At least one point in Low Throughput region |\n| `med-throughput-coverage` | §5 | At least one point in Medium Throughput region |\n| `high-throughput-coverage` | §6 | At least one point in High Throughput region |\n| `max-concurrency-declared` | §7 | `max_supported_concurrency` field present |\n| `region-computation` | §7 | *M* \u003e 32 (required for region formula) |\n| `concurrency-in-range` | §9 | Concurrency within region bounds (incl. 10% margin) |\n| `load-pattern` | §10 | `load_pattern` is `concurrency` with a positive concurrency level |\n| `point-duration` | §11 | Point meets per-region minimum duration |\n| `min-query-count` | §12 | `n_samples_completed` meets dataset-specific minimum (§6.4) |\n| `streaming-config` | §13 | `stream_all_chunks` is `True` |\n| `metric-consistency-duration` | §14 | `duration_ns` \u003e 0 |\n| `metric-consistency-accounting` | §14 | `completed + failed == issued` |\n| `metric-consistency-output-tokens` | §14 | `total_output_tokens` ≥ 0 |\n| `metric-consistency-system-tps` | §9.1 | Stored `system_tps` consistent with derived value |\n| `metric-consistency-tps-per-user` | §9.1 | Stored `tps_per_user` consistent with `system_tps / concurrency` |\n| `accuracy-file` | §15 | `accuracy.txt` and `accuracy_result.json` present |\n| `accuracy-valid` | §15 | `accuracy_result.json` parses correctly |\n| `accuracy-consistency` | §15 | `passed` flag consistent with `score \u003e= quality_target` |\n| `accuracy-gate` | §15 | Score ≥ quality target |\n| `config-consistency-dataset` | §16 | All points use the same dataset |\n| `config-consistency-model` | §16 | Directory name matches `benchmark_model` |\n| `region-declared` | §8.3 | Declared `region` field (if present) is valid and matches computed region |\n\n## Programmatic API\n\n```python\nfrom submission_checker import SubmissionChecker, Report\n\nchecker = SubmissionChecker(Path(\"/submissions/acme_corp\"))\nreport = checker.run()\n\nif report.passed:\n    print(\"All checks passed\")\nelse:\n    for result in report.errors:\n        print(f\"[{result.rule}] {result.message}\")\n```\n\nThe `Report` object also exposes `report.warnings` and serialises cleanly via `report.model_dump_json()`.\n\n---\n\n## Development\n\n```bash\nuv run pytest                          # run all tests\nuv run pytest --no-cov -x             # fast fail on first error\nuv run ruff check src/ tests/          # lint\nuv run ruff format src/ tests/         # auto-format\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlcommons%2Fendpoints-submission-cli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmlcommons%2Fendpoints-submission-cli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlcommons%2Fendpoints-submission-cli/lists"}