{"id":50750181,"url":"https://github.com/felipemchdev/arbiter","last_synced_at":"2026-06-11T01:00:38.357Z","repository":{"id":362204544,"uuid":"1257779644","full_name":"felipemchdev/arbiter","owner":"felipemchdev","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-03T05:58:13.000Z","size":45,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-03T06:22:49.186Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/felipemchdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-03T02:12:23.000Z","updated_at":"2026-06-03T05:58:17.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/felipemchdev/arbiter","commit_stats":null,"previous_names":["felipemchdev/arbiter"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/felipemchdev/arbiter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Farbiter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Farbiter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Farbiter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Farbiter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/felipemchdev","download_url":"https://codeload.github.com/felipemchdev/arbiter/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Farbiter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34177444,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-10T02:00:07.152Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-11T01:00:22.527Z","updated_at":"2026-06-11T01:00:38.348Z","avatar_url":"https://github.com/felipemchdev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Arbiter\n\n[![CI](https://github.com/felipemchdev/arbiter/actions/workflows/deploy.yml/badge.svg)](https://github.com/felipemchdev/arbiter/actions/workflows/deploy.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)\n\n\n[Português](#pt-br) · [English](#en)\n\n---\n\n## PT-BR\n\n### Arquitetura\n\n```\n                   POST /api/v1/runs/ingest\n┌──────────────┐  ──────────────────────────────►  ┌──────────────┐\n│  Airflow     │                                   │              │\n│  Azure Func  │                                   │  arbiter-api │\n│  Python      │                                   │  FastAPI     │\n│  Cron Job    │                                   │  :8000       │\n└──────────────┘                                   └──────┬───────┘\n                                                          │\n                          ┌───────────────────────────────┼───────────────────────────────┐\n                          │                               │                               │\n                          ▼                               ▼                               ▼\n                   ┌──────────────┐              ┌──────────────┐              ┌──────────────────┐\n                   │  SQLite /    │              │    Redis     │              │  Celery Worker   │\n                   │  PostgreSQL  │              │  broker +    │              │  + Beat          │\n                   │  pipelines   │              │  result      │              │  check_stale     │\n                   │  runs        │              │  backend     │              │  a cada 10 min   │\n                   │  tasks       │              └──────────────┘              └──────────────────┘\n                   │  alerts      │\n                   └──────┬───────┘\n                          │\n                          │  GET /api/v1/*\n                          ▼\n                   ┌──────────────────┐\n                   │  arbiter-        │\n                   │  dashboard       │\n                   │  Next.js :3000   │\n                   └──────────────────┘\n```\n\n### Componentes\n\n| Servico             | Porta | Descricao                                                                                  |\n| ------------------- | ----- | ------------------------------------------------------------------------------------------ |\n| `arbiter-api`       | 8000  | API REST - ingestao de runs, metricas, autenticacao JWT + API keys                         |\n| `arbiter-dashboard` | 3000  | Frontend Next.js - DAG interativo (ReactFlow), metricas, timeline, alertas, tema dark/light |\n| `arbiter-worker`    | -     | Celery worker + beat - executa `check_stale_pipelines` a cada 10 min                       |\n| `arbiter-redis`     | 6379  | Redis 7 - broker e result backend do Celery                                                |\n| `sqlite / postgres`  | 5432* | SQLite (default local) ou PostgreSQL 15 (Docker/producao) - fonte unica de verdade          |\n\n\u003e *Porta 5432 apenas quando usando PostgreSQL via Docker\n\n### Fluxo de dados\n\n1. Um sistema externo (Airflow, Azure Function, script Python, cron job) faz `POST` com dados da run\n2. A API persiste pipeline, run, tasks e DAG no banco (SQLite ou PostgreSQL) em uma transacao atomica\n3. No mesmo request, processa o evento inline: calcula `duration_ms`, atualiza `last_run_status`, cria alerta de `failure` se necessario\n4. O dashboard consulta a API e renderiza tudo em tempo real\n5. A cada 10 min, o Celery Beat dispara `check_stale_pipelines` - varre pipelines sem runs em 24h e cria alertas `no_run`\n\n### Quick Start\n\n```sh\ngit clone https://github.com/felipemchdev/arbiter.git\ncd arbiter\ncp .env.example .env\ndocker compose up -d\n\n# Dashboard:  http://localhost:3000 (login: admin@arbiter / arbiter26@)\n# API Docs:   http://localhost:8000/docs\n# API Health: http://localhost:8000/api/v1/health\n```\n\n### Autenticacao\n\nA API aceita dois metodos simultaneos:\n\n| Metodo      | Uso                    | Fluxo                                                                 |\n| ----------- | ---------------------- | --------------------------------------------------------------------- |\n| **JWT**     | Dashboard (usuarios)   | Login -\u003e access token (15 min) + refresh token com rotacao automatica |\n| **API Key** | Coletor / SDK / CI/CD  | Header `X-API-Key: arb_live_...` ou `arb_test_...`                    |\n\nRefresh tokens usam formato `jti.raw_token` - o `jti` permite lookup O(1) no banco.\nO dashboard renova o access token automaticamente e revoga todos os refresh tokens no logout.\n\nAPI Keys sao gerenciadas via dashboard (`/dashboard/api-keys`) ou API (`POST /api/v1/api-keys`).\nPrefixos: `arb_live_` (producao) e `arb_test_` (desenvolvimento/teste).\n\n### Credenciais padrao\n\n| Usuario          | Senha        | Role   |\n| ---------------- | ------------ | ------ |\n| `admin@arbiter`  | `arbiter26@` | owner  |\n| `viewer@arbiter` | `arbiter26@` | viewer |\n\nA seed de desenvolvimento gera uma API key e exibe no log: `docker compose logs api | grep \"API Key\"`\n\n### Enviando uma Run\n\n```sh\ncurl -X POST http://localhost:8000/api/v1/runs/ingest \\\n  -H \"X-API-Key: arb_live_xxx\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"dags\": [{\n      \"dag_id\": \"demo\",\n      \"name\": \"demo_pipeline\",\n      \"runs\": [{\n        \"run_id\": \"manual__2026-01-01\",\n        \"status\": \"success\",\n        \"started_at\": \"2026-01-01T00:00:00Z\",\n        \"finished_at\": \"2026-01-01T00:05:00Z\",\n        \"duration_ms\": 300000\n      }],\n      \"nodes\": [\n        {\"id\": \"extract\", \"label\": \"Extract\"},\n        {\"id\": \"transform\", \"label\": \"Transform\"},\n        {\"id\": \"load\", \"label\": \"Load\"}\n      ],\n      \"edges\": [\n        {\"source\": \"extract\", \"target\": \"transform\"},\n        {\"source\": \"transform\", \"target\": \"load\"}\n      ]\n    }]\n  }'\n```\n\n### Desenvolvimento\n\n```sh\n# API\ncd arbiter-api\npip install -r requirements.txt\nuvicorn app.main:app --reload\n\n# Dashboard\ncd arbiter-dashboard\nnpm install\nnpm run dev\n\n# Worker\ncd arbiter-api\ncelery -A app.workers.celery_app worker --loglevel=info\n```\n\n---\n\n## EN\n\nArbiter is a lightweight observability layer for data pipelines. It does **not** orchestrate, schedule, or execute anything. It receives run events via HTTP and displays DAGs, metrics, task timelines, and alerts.\n\n### Architecture\n\n```\n                   POST /api/v1/runs/ingest\n┌──────────────┐  ──────────────────────────────►  ┌──────────────┐\n│  Airflow     │                                   │              │\n│  Azure Func  │                                   │  arbiter-api │\n│  Python      │                                   │  FastAPI     │\n│  Cron Job    │                                   │  :8000       │\n└──────────────┘                                   └──────┬───────┘\n                                                          │\n                          ┌───────────────────────────────┼───────────────────────────────┐\n                          │                               │                               │\n                          ▼                               ▼                               ▼\n                   ┌──────────────┐              ┌──────────────┐              ┌──────────────────┐\n                   │  SQLite /    │              │    Redis     │              │  Celery Worker   │\n                   │  PostgreSQL  │              │  broker +    │              │  + Beat          │\n                   │  pipelines   │              │  result      │              │  check_stale     │\n                   │  runs        │              │  backend     │              │  every 10 min    │\n                   │  tasks       │              └──────────────┘              └──────────────────┘\n                   │  alerts      │\n                   └──────┬───────┘\n                          │\n                          │  GET /api/v1/*\n                          ▼\n                   ┌──────────────────┐\n                   │  arbiter-        │\n                   │  dashboard       │\n                   │  Next.js :3000   │\n                   └──────────────────┘\n```\n\n### Components\n\n| Service             | Port | Description                                                                                 |\n| ------------------- | ---- | ------------------------------------------------------------------------------------------- |\n| `arbiter-api`       | 8000 | Core REST API - run ingestion, metrics, JWT + API key auth                                  |\n| `arbiter-dashboard` | 3000 | Next.js frontend - interactive DAG (ReactFlow), metrics, timeline, alerts, dark/light theme |\n| `arbiter-worker`    | -    | Celery worker + beat - runs `check_stale_pipelines` every 10 min                            |\n| `arbiter-redis`     | 6379 | Redis 7 - Celery broker and result backend                                                  |\n| `sqlite / postgres`  | 5432* | SQLite (default local) or PostgreSQL 15 (Docker/production) - single source of truth        |\n\n\u003e *Port 5432 only when using PostgreSQL via Docker\n\n### Data flow\n\n1. An external system (Airflow DAG, Azure Function, Python script, cron job) POSTs run data to the API\n2. The API persists pipeline, run, tasks, and DAG definition in the database (SQLite or PostgreSQL) in a single atomic transaction\n3. In the same request, it processes the event inline: computes `duration_ms`, updates `last_run_status`, creates a `failure` alert if needed\n4. The dashboard queries the API and renders everything in real time\n5. Every 10 min, Celery Beat triggers `check_stale_pipelines` - scans for pipelines with no runs in 24h and creates `no_run` alerts\n\n### Quick Start\n\n```sh\ngit clone https://github.com/felipemchdev/arbiter.git\ncd arbiter\ncp .env.example .env\ndocker compose up -d\n\n# Dashboard:  http://localhost:3000  (login: admin@arbiter / arbiter26@)\n# API Docs:   http://localhost:8000/docs\n# API Health: http://localhost:8000/api/v1/health\n```\n\n### Authentication\n\nThe API accepts two authentication methods simultaneously:\n\n| Method      | Use                     | Flow                                                                   |\n| ----------- | ----------------------- | ---------------------------------------------------------------------- |\n| **JWT**     | Dashboard (human users) | Login -\u003e access token (15 min) + refresh token with automatic rotation |\n| **API Key** | Collector / SDK / CI/CD | `X-API-Key: arb_live_...` or `arb_test_...` header                     |\n\nRefresh tokens use `jti.raw_token` format - the `jti` enables O(1) database lookup.\nThe dashboard auto-refreshes the access token and revokes all refresh tokens on logout.\n\nAPI Keys are managed via dashboard (`/dashboard/api-keys`) or API (`POST /api/v1/api-keys`).\nPrefixes: `arb_live_` (production) and `arb_test_` (development/testing).\n\n### Default Credentials\n\n| User             | Password     | Role   |\n| ---------------- | ------------ | ------ |\n| `admin@arbiter`  | `arbiter26@` | owner  |\n| `viewer@arbiter` | `arbiter26@` | viewer |\n\nThe dev seed generates an API key and prints it to the log: `docker compose logs api | grep \"API Key\"`\n\n### Ingesting a Run\n\n```sh\ncurl -X POST http://localhost:8000/api/v1/runs/ingest \\\n  -H \"X-API-Key: arb_live_xxx\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"dags\": [{\n      \"dag_id\": \"demo\",\n      \"name\": \"demo_pipeline\",\n      \"runs\": [{\n        \"run_id\": \"manual__2026-01-01\",\n        \"status\": \"success\",\n        \"started_at\": \"2026-01-01T00:00:00Z\",\n        \"finished_at\": \"2026-01-01T00:05:00Z\",\n        \"duration_ms\": 300000\n      }],\n      \"nodes\": [\n        {\"id\": \"extract\", \"label\": \"Extract\"},\n        {\"id\": \"transform\", \"label\": \"Transform\"},\n        {\"id\": \"load\", \"label\": \"Load\"}\n      ],\n      \"edges\": [\n        {\"source\": \"extract\", \"target\": \"transform\"},\n        {\"source\": \"transform\", \"target\": \"load\"}\n      ]\n    }]\n  }'\n```\n\n### Development\n\n```sh\n# API\ncd arbiter-api\npip install -r requirements.txt\nuvicorn app.main:app --reload\n\n# Dashboard\ncd arbiter-dashboard\nnpm install\nnpm run dev\n\n# Worker\ncd arbiter-api\ncelery -A app.workers.celery_app worker --loglevel=info\n```\n\n## License\n\nMIT © 2026 Felipe Machado\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffelipemchdev%2Farbiter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffelipemchdev%2Farbiter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffelipemchdev%2Farbiter/lists"}