{"id":47001947,"url":"https://github.com/bluet/arguslm","last_synced_at":"2026-03-11T19:18:39.476Z","repository":{"id":338535819,"uuid":"1149166874","full_name":"bluet/arguslm","owner":"bluet","description":"The hundred-eyed watcher for your LLM providers. Monitor uptime, TTFT, TPS, and latency across OpenAI, Anthropic, Azure, Bedrock, Ollama, LM Studio, and 100+ providers through a single dashboard. Benchmark, compare, and get alerts — all self-hosted.","archived":false,"fork":false,"pushed_at":"2026-02-15T02:31:19.000Z","size":1600,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-02-15T08:43:43.333Z","etag":null,"topics":["ai-ops","anthropic","dashboard","fastapi","litellm","llm","llm-benchmark","llm-monitoring","llm-ops","lm-studio","mlops","monitoring","observability","ollama","openai","performance-monitoring","python","react","self-hosted","typescript"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bluet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-02-03T19:51:59.000Z","updated_at":"2026-02-15T02:31:22.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/bluet/arguslm","commit_stats":null,"previous_names":["bluet/arguslm"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/bluet/arguslm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluet%2Farguslm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluet%2Farguslm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluet%2Farguslm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluet%2Farguslm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bluet","download_url":"https://codeload.github.com/bluet/arguslm/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluet%2Farguslm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30395208,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-11T18:46:22.935Z","status":"ssl_error","status_checked_at":"2026-03-11T18:46:17.045Z","response_time":84,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-ops","anthropic","dashboard","fastapi","litellm","llm","llm-benchmark","llm-monitoring","llm-ops","lm-studio","mlops","monitoring","observability","ollama","openai","performance-monitoring","python","react","self-hosted","typescript"],"created_at":"2026-03-11T19:18:38.462Z","updated_at":"2026-03-11T19:18:39.471Z","avatar_url":"https://github.com/bluet.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ArgusLM — Open-Source LLM Monitoring \u0026 Benchmarking\n\n[![PyPI](https://img.shields.io/pypi/v/arguslm?style=flat-square)](https://pypi.org/project/arguslm/)\n[![CI](https://github.com/bluet/arguslm/actions/workflows/ci.yml/badge.svg)](https://github.com/bluet/arguslm/actions/workflows/ci.yml)\n[![License](https://img.shields.io/github/license/bluet/arguslm?style=flat-square)](LICENSE)\n[![GitHub Stars](https://img.shields.io/github/stars/bluet/arguslm?style=social)](https://github.com/bluet/arguslm/stargazers)\n[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue?style=flat-square)](https://www.python.org/downloads/)\n[![Docker Hub](https://img.shields.io/docker/v/bluet/arguslm?label=docker\u0026style=flat-square)](https://hub.docker.com/r/bluet/arguslm)\n\n\u003e Know exactly which LLM providers are up, which are fastest,\n\u003e and which are degrading — before your users notice.\n\n![ArgusLM Dashboard Overview](docs/images/dashboard-overview.png)\n\n## The Problem\n\nModern AI architectures use dozens of LLM providers across services — OpenAI, Anthropic, Bedrock, Vertex, local Ollama, custom endpoints — each with different availability, latency, and throughput characteristics. When providers fail or slow down, you find out from support tickets, not monitoring dashboards. Existing tools are either SaaS-only (expensive, locked-in), infrastructure-focused (can't probe LLM APIs), or require complex instrumentation (changes your code).\n\n## Why ArgusLM?\n\n| Aspect | Datadog / Langfuse | Prometheus | LLM Overwatch | **ArgusLM** |\n|--------|-------------------|-----------|---------------|-------------|\n| Deployment | SaaS-only | Self-hosted | SaaS-only | Self-hosted |\n| Local Models | ❌ No | ❌ No | ❌ No | ✅ Ollama, LM Studio, local APIs |\n| Probing vs Tracing | Tracing only | Infrastructure only | Probing only | Synthetic probing |\n| Metrics | Request-level | Node-level | Response time | TTFT, TPS, latency, uptime |\n| Pricing | $$$$ | Free | $$$ | ✅ Free \u0026 Open-Source |\n| Extensible | Limited | Limited | No | ✅ Full Python SDK + HTTP API |\n\n**What makes ArgusLM unique:** The only open-source tool that actively probes any LLM provider (including local Ollama/LM Studio) for real uptime, Time to First Token (TTFT), Tokens per Second (TPS), and latency — with a unified Python SDK for custom automation.\n\n## Use Cases\n\n**ArgusLM is for you if:**\n\n- **You're building production AI systems** — Monitor uptime and performance of multiple LLM providers in real-time, detect degradations before users do.\n- **You run self-hosted LLM deployments** — Track local Ollama/LM Studio availability and response metrics alongside cloud providers in one dashboard.\n- **You provider LLM-based services** — Know exactly which provider to route traffic to based on real performance data, not assumptions or marketing claims.\n- **You need automated benchmarking** — Run scheduled comparisons between models (GPT-4 vs Claude vs local Llama) to optimize costs and quality.\n- **You must keep costs private** — Self-hosted, no SaaS lock-in, full control over your observability data.\n\n---\n\n## Quick Start\n\nDeploy ArgusLM in under a minute:\n\n```bash\ngit clone https://github.com/bluet/arguslm.git \u0026\u0026 cd arguslm\ncp .env.example .env\n\n# Generate secrets (requires cryptography package, or use the Docker one-liner in .env.example)\npython3 scripts/generate-secrets.py \u003e\u003e .env\n\ndocker compose up -d\n```\n\n**Dashboard**: [http://localhost:3000](http://localhost:3000)\n**API Documentation**: [http://localhost:8000/docs](http://localhost:8000/docs)\n\n---\n\n## Features\n\n| Category | Capabilities |\n| :--- | :--- |\n| Monitoring | Automated uptime checks, real-time status tracking, and configurable availability intervals. |\n| Benchmarking | Parallel multi-model testing with deep metrics for TTFT, TPS, and total latency. |\n| Visualization | Live performance charts, historical trends, and side-by-side model comparisons. |\n| Alerting | Proactive downtime detection and performance degradation notifications. |\n| Integration | 90+ providers via LiteLLM (16 tested, all others auto-discovered from LiteLLM catalog). |\n\n---\n\n## Architecture\n\nArgusLM is built for scale and reliability, leveraging a modern asynchronous stack.\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│                         ArgusLM                                 │\n├─────────────────────────────────────────────────────────────────┤\n│  Frontend (React + Vite)           Backend (FastAPI)            │\n│  ┌─────────────────────┐           ┌──────────────────────┐    │\n│  │ Dashboard           │◄─────────►│ REST API + WebSocket │    │\n│  │ Benchmarks          │           │ Background Scheduler │    │\n│  │ Monitoring          │           │ Alert Engine         │    │\n│  │ Providers           │           └──────────┬───────────┘    │\n│  └─────────────────────┘                      │                 │\n│                                               ▼                 │\n│                              ┌─────────────────────────────┐   │\n│                              │  LiteLLM Abstraction Layer  │   │\n│                              └─────────────┬───────────────┘   │\n│                                            │                    │\n└────────────────────────────────────────────┼────────────────────┘\n                                             ▼\n              ┌──────────────────────────────────────────────────┐\n              │                  LLM Providers                   │\n              │  OpenAI │ Anthropic │ Bedrock │ Vertex │ Azure   │\n              │  Ollama │ LM Studio │ xAI │ DeepSeek │ 90+      │\n              └──────────────────────────────────────────────────┘\n```\n\n---\n\n## Usage Examples\n\n### Trigger Monitoring (HTTP API)\n\n```bash\n# Trigger a manual monitoring run\ncurl -X POST http://localhost:8000/api/v1/monitoring/run\n\n# Get current monitoring configuration\ncurl http://localhost:8000/api/v1/monitoring/config\n\n# Get uptime history for all providers (last 100 checks)\ncurl \"http://localhost:8000/api/v1/monitoring/uptime?limit=100\"\n```\n\n### Run Benchmarks (HTTP API)\n\n```bash\n# Start benchmark for specific models\ncurl -X POST http://localhost:8000/api/v1/benchmarks \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model_ids\": [\"uuid-1\", \"uuid-2\"],\n    \"prompt_pack\": \"health_check\",\n    \"max_tokens\": 100,\n    \"num_runs\": 5\n  }'\n\n# List all benchmarks\ncurl http://localhost:8000/api/v1/benchmarks\n\n# Get results for specific benchmark run\ncurl http://localhost:8000/api/v1/benchmarks/{run_id}/results\n```\n\n### Python SDK\n\n```bash\npip install arguslm\n```\n\n```python\nfrom arguslm import ArgusLMClient\nfrom arguslm.schemas import BenchmarkCreate\n\nwith ArgusLMClient(base_url=\"http://localhost:8000\") as client:\n    # Check provider uptime\n    uptime = client.get_uptime_history(limit=10)\n    for check in uptime.items:\n        print(f\"{check.model_name}: {check.status} ({check.ttft_ms}ms TTFT)\")\n\n    # Run a benchmark\n    benchmark = client.start_benchmark(BenchmarkCreate(\n        model_ids=[\"uuid-1\", \"uuid-2\"],\n        prompt_pack=\"shakespeare\",\n        num_runs=3,\n    ))\n    print(f\"Benchmark started: {benchmark.id}\")\n```\n\nAsync support:\n\n```python\nfrom arguslm import AsyncArgusLMClient\n\nasync with AsyncArgusLMClient() as client:\n    config = await client.get_monitoring_config()\n    providers = await client.list_providers()\n```\n\n---\n\n## Key Metrics\n\nArgusLM tracks the metrics that define real-world LLM performance:\n\n- **Time to First Token (TTFT)**: Measure user-perceived responsiveness and cold-start latency.\n- **Tokens per Second (TPS)**: Evaluate sustained streaming throughput independent of initial latency.\n- **End-to-End Latency**: Track total request duration for non-streaming workloads.\n- **Availability**: Monitor uptime and reliability trends with granular failure analysis.\n\n---\n\n\u003cdetails\u003e\n\u003csummary\u003eDashboard Screenshots\u003c/summary\u003e\n\n![Performance Trends](docs/images/dashboard-performance.png)\n*Real-time tracking of latency and throughput trends across all configured providers.*\n\n![Model Comparison](docs/images/dashboard-comparison.png)\n*Side-by-side performance comparison to identify the most efficient models for your workload.*\n\n![Monitoring Configuration](docs/images/monitoring.png)\n*Configure granular monitoring intervals and thresholds for each provider.*\n\n![Benchmark Runner](docs/images/benchmarks.png)\n*Execute standardized benchmark suites to validate provider performance under load.*\n\n\u003c/details\u003e\n\n---\n\n## Configuration\n\n| Variable | Description | Default |\n| :--- | :--- | :--- |\n| `DATABASE_URL` | PostgreSQL connection string | `postgresql+asyncpg://...` |\n| `SECRET_KEY` | Session encryption key | *required* |\n| `ENCRYPTION_KEY` | Credential encryption (Fernet) | *required* |\n\nDetailed setup instructions are available in the [Configuration Guide](docs/CONFIGURATION.md).\n\n---\n\n## Local Development\n\n### Backend\n```bash\npip install -e \".[server]\"\nalembic upgrade head\nuvicorn arguslm.server.main:app --reload\n```\n\n### Frontend\n```bash\ncd frontend\nnpm install\nnpm run dev\n```\n\n---\n\n## Tech Stack\n\n| Layer | Technology |\n| :--- | :--- |\n| Backend | FastAPI, Python 3.11+, SQLAlchemy, Alembic |\n| Frontend | React 18, TypeScript, Vite, Tailwind CSS, Recharts |\n| Database | PostgreSQL (Production) / SQLite (Development) |\n| Abstraction | LiteLLM |\n\n---\n\n## Installation\n\n```bash\n# SDK only (lightweight — for querying an ArgusLM instance)\npip install arguslm\n\n# Full server (for self-hosted deployment without Docker)\npip install arguslm[server]\n```\n\n---\n\n## Documentation\n\n- [Architecture Overview](docs/architecture.md)\n- [Python SDK Guide](docs/sdk-guide.md)\n- [REST API Reference](docs/api-reference.md)\n- [Configuration Guide](docs/CONFIGURATION.md)\n- [Troubleshooting](docs/TROUBLESHOOTING.md)\n- [Comparison with Alternatives](docs/comparison.md)\n- [Interactive API Docs](http://localhost:8000/docs) (Swagger UI, available when server is running)\n\n---\n\n## Contributing\n\nWe welcome contributions from the community. Please review our [Contributing Guidelines](CONTRIBUTING.md) before submitting a Pull Request.\n\n---\n\n## Author\n\n**Matthew (BlueT) Lien**\n- Twitter: [@BlueT](https://twitter.com/BlueT)\n- LinkedIn: [bluet](https://www.linkedin.com/in/bluet/)\n- GitHub: [@BlueT](https://github.com/bluet)\n\n---\n\n## License\n\nArgusLM is released under the [Apache License 2.0](LICENSE).\n\n---\n\n*Named after Argus Panoptes, the all-seeing giant of Greek mythology.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbluet%2Farguslm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbluet%2Farguslm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbluet%2Farguslm/lists"}