{"id":50897772,"url":"https://github.com/EndoTheDev/OMeter","last_synced_at":"2026-06-25T00:00:41.482Z","repository":{"id":353781916,"uuid":"1220125073","full_name":"EndoTheDev/OMeter","owner":"EndoTheDev","description":"Benchmark and compare Ollama models across local and cloud endpoints with rich, sortable tables.","archived":false,"fork":false,"pushed_at":"2026-06-19T15:20:30.000Z","size":1074,"stargazers_count":17,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-19T16:14:22.839Z","etag":null,"topics":["benchmark","cli","ollama","performance","python","rich"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EndoTheDev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-04-24T15:13:55.000Z","updated_at":"2026-06-19T15:20:34.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/EndoTheDev/OMeter","commit_stats":null,"previous_names":["endothedev/ollamameter","endothedev/ometer"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/EndoTheDev/OMeter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EndoTheDev%2FOMeter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EndoTheDev%2FOMeter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EndoTheDev%2FOMeter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EndoTheDev%2FOMeter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EndoTheDev","download_url":"https://codeload.github.com/EndoTheDev/OMeter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EndoTheDev%2FOMeter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34753781,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-24T02:00:07.484Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","cli","ollama","performance","python","rich"],"created_at":"2026-06-16T01:31:30.079Z","updated_at":"2026-06-25T00:00:41.476Z","avatar_url":"https://github.com/EndoTheDev.png","language":"Python","funding_links":[],"categories":["cli","Model Evaluation and Testing Tools"],"sub_categories":[],"readme":"# OMeter\n\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/python-3.14+-blue.svg?style=flat-square\" alt=\"Python 3.14+\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/license-MIT-green.svg?style=flat-square\" alt=\"MIT License\"\u003e\n\u003c/p\u003e\n\nBenchmark and compare Ollama models across local and cloud endpoints with rich, sortable tables.\n\n## Features\n\n- 🌐 **Live dashboard** — auto-published benchmark trends at [EndoTheDev.github.io/OMeter](https://EndoTheDev.github.io/OMeter/) with filtering, sorting, and charts\n- 📋 **List models** from local and cloud Ollama endpoints\n- 📊 **Rich tables** with sorting by name, size, context length, modification date, TTFT, or TPS\n- 🔃 **Reverse sort** with `--reverse`\n- ⏱️ **Benchmark** time-to-first-token (TTFT) and tokens-per-second (TPS)\n- 🔍 **Model filtering** by exact name or family match (e.g. `llama3` matches `llama3:latest`)\n- 📤 **Export results** to JSON or CSV (stdout or file)\n- 🧪 **Multi-prompt averaging** — 3 prompts per model for robust stats (or use `--prompts` for custom prompts)\n- 🧬 **Embedding model support** — automatically uses `/api/embed` for local embedding models\n- 🎨 **Beautiful CLI** powered by `rich` + `InquirerPy`\n- 📜 **Benchmark history** — runs are auto-saved to a local SQLite database and merged into the public dashboard history; view past results with `--history`\n- 📈 **Performance trends** — arrows (↑↓→) automatically appear inline next to TTFT/TPS values when historical data is available\n\n## Preview\n\n\u003cdetails name=\"screenshots\" open\u003e\n\u003csummary\u003e\u003cstrong\u003eCloud model listing\u003c/strong\u003e — \u003ccode\u003eometer --cloud\u003c/code\u003e\u003c/summary\u003e\n\u003cimg src=\"assets/cloud.png\" alt=\"Cloud models\"\u003e\n\u003c/details\u003e\n\n\u003cdetails name=\"screenshots\"\u003e\n\u003csummary\u003e\u003cstrong\u003eLocal model listing\u003c/strong\u003e — \u003ccode\u003eometer --local\u003c/code\u003e\u003c/summary\u003e\n\u003cimg src=\"assets/local.png\" alt=\"Local models\"\u003e\n\u003c/details\u003e\n\n\u003cdetails name=\"screenshots\"\u003e\n\u003csummary\u003e\u003cstrong\u003eBenchmark with per-run breakdown\u003c/strong\u003e — \u003ccode\u003eometer --local --ttft --tps --verbose --runs 2 --parallel 1\u003c/code\u003e\u003c/summary\u003e\n\u003cimg src=\"assets/local-ttft-tps-verbose-runs-parallel.png\" alt=\"Benchmark with breakdown\"\u003e\n\u003c/details\u003e\n\n## Installation\n\n### Install as a uv tool (recommended)\n\nFrom the project directory:\n\n```bash\nuv tool install .\n```\n\nOr install directly from GitHub:\n\n```bash\nuv tool install git+https://github.com/EndoTheDev/OMeter.git\n```\n\nThis installs `ometer` and `ometer` globally, so you can run them from anywhere.\n\n**Update:**\n\n```bash\nuv tool install --upgrade ometer\n```\n\n**Uninstall:**\n\n```bash\nuv tool uninstall ometer\n```\n\n### Install into a project\n\n```bash\nuv add ometer\n```\n\nOr via pip:\n\n```bash\npip install ometer\n```\n\n## Usage\n\nShow the version:\n\n```bash\nometer --version\n```\n\nList models with an **interactive menu**:\n\n```bash\nometer\n```\n\nList **local** models only:\n\n```bash\nometer --local\n```\n\nList **cloud** models only:\n\n```bash\nometer --cloud\n```\n\nList **both** local and cloud models:\n\n```bash\nometer --local --cloud\n```\n\nBenchmark **time-to-first-token** and **tokens-per-second**:\n\n```bash\nometer --cloud --ttft --tps\n```\n\nBenchmark models in **parallel** for faster results (default is 1 — max 10):\n\n```bash\nometer --cloud --ttft --tps --parallel 4\n```\n\nShow **per-run breakdown** in the table:\n\n```bash\nometer --cloud --ttft --tps --verbose\n```\n\nRun with **fewer benchmark prompts** for faster results (default is 3 — max 3):\n\n```bash\nometer --cloud --ttft --tps --verbose --runs 1\nometer --cloud --ttft --tps --verbose --runs 2\n```\n\nUse **custom benchmark prompts** instead of the built-in defaults (overrides `--runs`):\n\n```bash\nometer --local --ttft --tps --prompts \"why is the ocean salty?\"\nometer --local --ttft --tps --prompts prompts.txt\n```\n\nPass a filename to read one prompt per line (skips blank lines, strips whitespace).\n\nFilter to **specific models** (exact name or family match, accepts multiple names):\n\n```bash\nometer --model llama3 --ttft --tps\nometer --local --model llama3.2:3b llama3.3:8b --ttft --tps\n```\n\nSort results by **model size** (largest first) or **name** (A–Z):\n\n```bash\nometer --cloud --sort size\nometer --cloud --sort name\n```\n\nSort by **context length** (largest first) or **modification date** (newest first):\n\n```bash\nometer --cloud --sort ctx\nometer --local --sort modified\n```\n\nSort by **benchmark metrics** — TTFT (lowest/best first) and TPS (highest/best first):\n\n```bash\nometer --cloud --ttft --tps --sort ttft\nometer --cloud --ttft --tps --sort tps\n```\n\n**Reverse** any sort order (worst first, Z–A, oldest first):\n\n```bash\nometer --cloud --sort name --reverse\nometer --cloud --ttft --tps --sort tps --reverse\n```\n\nExport results as **JSON** (to stdout or a file):\n\n```bash\nometer --cloud --ttft --tps --json\nometer --cloud --ttft --tps --json results.json\n```\n\nExport results as **CSV** (to stdout or a file):\n\n```bash\nometer --local --ttft --tps --csv\nometer --local --ttft --tps --csv results.csv\n```\n\nView **benchmark history** (latest run per model):\n\n```bash\nometer --history\n```\n\nShow all historical runs with full details:\n\n```bash\nometer --history --verbose\n```\n\nFilter history to specific models:\n\n```bash\nometer --history --model llama3\n```\n\nExport history as **JSON** or **CSV**:\n\n```bash\nometer --history --json\nometer --history --csv history.csv\n```\n\nPerformance trend arrows (↑ improved, ↓ degraded, → stable within 5%) appear inline next to TTFT and TPS values automatically. No flag needed.\n\nSee all options:\n\n```bash\nometer --help\n```\n\n## Web Dashboard\n\nBenchmark data is automatically merged into the live dashboard after each\nscheduled GitHub Actions run:\n\n**[EndoTheDev.github.io/OMeter](https://EndoTheDev.github.io/OMeter/)**\n\nThe dashboard supports filtering by capability, context window, parameter\nsize, and model name, plus sorting and time-series charts once multiple runs\nhave been collected.\n\n## Environment Variables\n\nOMeter looks for a `.env` file in this order, using the **first one found**:\n\n1. **`./.env`** — current working directory (project-specific)\n2. **`~/.env`** — home directory (global fallback)\n3. **`~/.config/ometer/.env`** — dedicated config directory (recommended for global installs)\n\nCreate the config directory and file:\n\n```bash\nmkdir -p ~/.config/ometer\ncat \u003e ~/.config/ometer/.env \u003c\u003c 'EOF'\nOLLAMA_CLOUD_BASE_URL=https://ollama.com\nOLLAMA_CLOUD_API_KEY=your_api_key_here\nOLLAMA_LOCAL_BASE_URL=http://localhost:11434\n\n# Number of benchmark prompts per model (1–3, default 3). Ignored when --prompts is used.\nOMETER_RUNS=3\n\n# Number of models benchmarked in parallel (default 1, max 10)\nOMETER_PARALLEL=1\nEOF\n```\n\nThe cloud API key is **only needed for benchmarking cloud models**.\n\nBenchmark results are **auto-saved** to a local SQLite database. The database path can be overridden:\n\n```bash\nexport OMETER_HISTORY_DB=/custom/path/history.db\n```\n\nBy default it lives at `~/.local/share/ometer/ometer_history.db`.\n\nOMeter has six modules that handle distinct concerns:\n\n```txt\nUser ──► cli.py ──► config.py ──► api.py ──► display.py\n           │             │            │            │\n      arg parsing    .env load    HTTP calls    rich tables\n      mode resolve   validate     benchmark     color thresholds\n      interactive     clamp       stream        live updates\n      export             │            │            │\n         │               │            │        history.py\n         │               │            │            │\n     export.py           │            │       SQLite DB\n         │                              │\n    JSON/CSV output                auto-save + trend\n```\n\n- **cli.py** — Entry point, argument parsing, interactive model selection, export dispatch\n- **config.py** — Hierarchical `.env` loading, settings validation and clamping\n- **api.py** — HTTP communication with Ollama, TTFT/TPS measurement\n- **display.py** — Rich terminal UI, live table updates, percentile-based color coding\n- **export.py** — JSON/CSV export formatting and file output\n- **history.py** — SQLite-backed benchmark persistence, trend computation, history queries\n\nFor detailed documentation, see the [docs](docs/) directory:\n\n- [Architecture](docs/architecture.md) — Module decomposition, request lifecycle, data entities\n- [Benchmarking Pipeline](docs/benchmarking.md) — TTFT/TPS methodology, concurrency, color thresholds\n- [Configuration](docs/configuration.md) — Environment variables, CLI flags, loading order\n- [API Reference](docs/api-reference.md) — Ollama endpoints, function reference, BenchmarkResult\n- [Development](docs/development.md) — Dev setup, running tests, project structure, conventions\n\n## License\n\nMIT License — see [LICENSE](LICENSE) for details.\n\n---\n\nMade for you with vibes by [Endo](https://github.com/EndoTheDev)🎵 \u0026 [Kimi](https://ollama.com/library/kimi-k2.7-code) \u0026 [Hermes](https://github.com/nousresearch/hermes-agent) \u0026 [Ollama](https://github.com/ollama/ollama)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEndoTheDev%2FOMeter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FEndoTheDev%2FOMeter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEndoTheDev%2FOMeter/lists"}