{"id":31284879,"url":"https://github.com/alleyesonmypon/opslog","last_synced_at":"2026-05-02T23:36:26.024Z","repository":{"id":310820477,"uuid":"1040996186","full_name":"AllEyesOnMyPon/OpsLog","owner":"AllEyesOnMyPon","description":"Lightweight logging gateway prototype built with FastAPI. Accepts structured log messages over HTTP and provides a foundation for further DevOps observability experiments.","archived":false,"fork":false,"pushed_at":"2025-09-17T17:40:46.000Z","size":5397,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-24T07:39:04.794Z","etag":null,"topics":["alertmanager","c4-model","devops","fastapi","grafana","hmac-authentication","logging","loki","monitoring","observability","orchestration","prometheus","python","rate-limiting","slo"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AllEyesOnMyPon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/roadmap.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-19T20:27:15.000Z","updated_at":"2025-09-17T17:39:25.000Z","dependencies_parsed_at":"2025-08-20T13:17:42.184Z","dependency_job_id":"e000a1be-7ce0-459e-95d7-74e62836a114","html_url":"https://github.com/AllEyesOnMyPon/OpsLog","commit_stats":null,"previous_names":["alleyesonmypon/opslog"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/AllEyesOnMyPon/OpsLog","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AllEyesOnMyPon%2FOpsLog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AllEyesOnMyPon%2FOpsLog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AllEyesOnMyPon%2FOpsLog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AllEyesOnMyPon%2FOpsLog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AllEyesOnMyPon","download_url":"https://codeload.github.com/AllEyesOnMyPon/OpsLog/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AllEyesOnMyPon%2FOpsLog/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279006110,"owners_count":26084027,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alertmanager","c4-model","devops","fastapi","grafana","hmac-authentication","logging","loki","monitoring","observability","orchestration","prometheus","python","rate-limiting","slo"],"created_at":"2025-09-24T07:34:47.558Z","updated_at":"2025-10-11T03:39:11.572Z","avatar_url":"https://github.com/AllEyesOnMyPon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LogOps\r\n\r\n![Python](https://img.shields.io/badge/Python-3.11%2B-blue?logo=python\u0026logoColor=white)\r\n![FastAPI](https://img.shields.io/badge/FastAPI-0.111%2B-009688?logo=fastapi)\r\n![Docker](https://img.shields.io/badge/Docker-ready-2496ED?logo=docker\u0026logoColor=white)\r\n![Grafana](https://img.shields.io/badge/Grafana-dashboard-F46800?logo=grafana\u0026logoColor=white)\r\n![Prometheus](https://img.shields.io/badge/Prometheus-alerts-E6522C?logo=prometheus\u0026logoColor=white)\r\n![License](https://img.shields.io/badge/license-MIT-green)\r\n![version](https://img.shields.io/badge/version-v0.5-blue.svg)\r\n\r\nSystem do **emisji, zbierania i obserwowalności logów** w środowisku deweloperskim.\r\n\r\n**Przepływ:** emitery → (opcjonalnie) **AuthGW** (HMAC/RL/backpressure) → **IngestGW** (normalizacja) → **Core** (przyjęcie/sink/metryki) → Promtail/Loki → Grafana/Prometheus/Alertmanager.\r\n\r\n---\r\n\r\n## Quickstart (TL;DR)\r\n\r\n### Opcja A — wszystko jednym poleceniem (demo)\r\n```bash\r\nmake demo\r\n```\r\nTo uruchomi stack observability (Loki/Promtail/Prometheus/Grafana/Alertmanager), wystartuje **Core**, **IngestGW** (forward do Core) i (opcjonalnie) **AuthGW**, a następnie odpali przykładowy scenariusz ruchu.\r\n\r\n### Opcja B — ręcznie, krok po kroku\r\n```bash\r\n# 1) Observability stack\r\ndocker compose -f infra/docker/observability/docker-compose.yml up -d\r\n\r\n# 2) Serwisy (dev)\r\nuvicorn services.core.app:app     --host 0.0.0.0 --port 8095 --reload \u0026\r\nuvicorn services.ingestgw.app:app --host 0.0.0.0 --port 8080 --reload \u0026\r\nuvicorn services.authgw.app:app   --host 0.0.0.0 --port 8081 --reload \u0026   # opcjonalnie\r\n\r\n# 3) Prosty ruch z emitera (CSV)\r\npython emitters/csv.py -n 10 --partial-ratio 0.2\r\n\r\n# 4) Logi w Loki\r\ncurl -G \"http://localhost:3100/loki/api/v1/query\" \\\r\n  --data-urlencode 'query={job=\"logops-ndjson\",app=\"logops\"}'\r\n```\r\n\r\nGrafana: \u003chttp://localhost:3000\u003e (admin / admin) • Prometheus: \u003chttp://localhost:9090\u003e • Alertmanager: \u003chttp://localhost:9093\u003e\r\n\r\n---\r\n\r\n## Najważniejsze funkcje\r\n\r\n- **Emitery**: CSV / JSON / minimal / noise / syslog + gotowe **scenariusze** (`scenarios/*.yaml`).\r\n- **Auth Gateway**: HMAC (`X-Api-Key`, `X-Timestamp`, `X-Content-SHA256`, `X-Signature`, `X-Nonce`), **rate limit** (token bucket), **backpressure**, retry + **circuit breaker**.\r\n- **Ingest Gateway**: normalizacja (ts/level/PII), metryki, **forward do Core**.\r\n- **Core**: przyjęcie `/v1/logs`, metryki (`*_accepted_total`, `*_request_latency_seconds`, `*_rejected_total`), opcjonalny **NDJSON sink**.\r\n- **Observability**: Promtail/Loki + Prometheus/Alertmanager + Grafana (dashboard SLO/p95).\r\n- **Narzędzia**: `tools/run_scenario.py`, `tools/orch_cli.py`, `tools/sign_hmac.py`, `tools/hmac_curl.sh`, `tools/verify_hmac_against_signer.py`, `tools/housekeeping.py`.\r\n\r\n---\r\n\r\n## Endpointy (dev)\r\n\r\n### Core\r\n- `GET /healthz` — zdrowie\r\n- `GET /metrics` — metryki Prometheus\r\n- `POST /v1/logs` — przyjęcie logów (JSON/NDJSON/CSV), opcjonalny zapis do `data/core/` (gdy włączony sink)\r\n\r\n### IngestGW\r\n- `GET /healthz`, `GET /metrics`\r\n- `POST /v1/logs` — normalizacja → **forward do Core** (adres z `CORE_URL`)\r\n\r\n### AuthGW (opcjonalnie przed Ingest)\r\n- `GET /healthz`, `GET /metrics`\r\n- `POST /ingest` — HMAC/RL/backpressure → **forward do Ingest** (adres z `forward.url`)\r\n\r\n---\r\n\r\n## Scenariusze i orkiestracja\r\n\r\n- Szybko uruchom gotowe profile:\r\n  ```bash\r\n  make scenario-default\r\n  make scenario-spike\r\n  make scenario-high-errors\r\n  ```\r\n- Runner (ticki, rampy, jitter, seed, JSONL z metadanymi):\r\n  ```bash\r\n  python tools/run_scenario.py -s scenarios/burst-then-ramp.yaml --log-file logs/scenario-default.jsonl\r\n  ```\r\n- Orchestrator CLI (HTTP do usługi orchestratora):\r\n  ```bash\r\n  python tools/orch_cli.py list\r\n  python tools/orch_cli.py start --yaml-path scenarios/spike.yaml --debug\r\n  python tools/orch_cli.py stop \u003cSCENARIO_ID\u003e\r\n  ```\r\n\r\n---\r\n\r\n## HMAC — narzędzia\r\n\r\n- Generowanie nagłówków (kanonikalizacja **PATH-only**, bez query):\r\n  ```bash\r\n  python tools/sign_hmac.py demo-pub-1 demo-priv-1 POST 'http://127.0.0.1:8081/ingest' '{\"msg\":\"hello\"}' --nonce\r\n  ```\r\n- Wygodny wrapper:\r\n  ```bash\r\n  tools/hmac_curl.sh --nonce -d '{\"msg\":\"hello\"}' -- -i\r\n  ```\r\n- Offline weryfikacja podpisu (nagłówki z `stdin`, sekret z `LOGOPS_SECRET`):\r\n  ```bash\r\n  tools/hmac_curl.sh --nonce -d '{\"msg\":\"hello\"}' --echo-headers \\\r\n  | LOGOPS_SECRET=demo-priv-1 python tools/verify_hmac_against_signer.py \\\r\n      --url http://127.0.0.1:8081/ingest --method POST --body-file \u003c(printf '{\"msg\":\"hello\"}')\r\n  ```\r\n\r\n---\r\n\r\n## Struktura repo (skrót)\r\n\r\n```\r\n.\r\n├── docs/                       # dokumentacja (services/, tools/, releases/, infra, observability)\r\n├── services/\r\n│   ├── core/                   # Core FastAPI (app.py)\r\n│   ├── ingestgw/               # Ingest FastAPI (app.py, normalize/metrics/parsers)\r\n│   ├── authgw/                 # AuthGW (HMAC/RL/backpressure/retry+CB)\r\n│   └── orchestrator/           # API do zarządzania scenariuszami\r\n├── emitters/                   # emitery: csv/json/minimal/noise/syslog\r\n├── tools/                      # run_scenario, orch_cli, HMAC tools, housekeeping\r\n├── infra/docker/observability/ # docker-compose + loki/promtail/prometheus/grafana/alertmanager\r\n├── scenarios/                  # gotowe scenariusze\r\n├── data/                       # NDJSON dla ingest/core, wyniki scenariuszy\r\n└── logs/ \u0026 run/                # logi i pid-y uruchomionych procesów\r\n```\r\n\r\n---\r\n\r\n## Dokumentacja szczegółowa\r\n\r\n- **Overview:** `docs/overview.md`\r\n- **Observability:** `docs/observability.md`\r\n- **Infra (Compose):** `docs/infra.md`\r\n- **Env (.env):** `docs/env.md`\r\n\r\n**Serwisy**\r\n- **Core:** `docs/services/core/core_app_readme.md`\r\n- **IngestGW:** `docs/services/ingestGW/ingestGW_app_readme.md`\r\n- **AuthGW:** `docs/services/authGW/authGW_app_readme.md`\r\n\r\n**Tools**\r\n- **Scenariusze (runner):** `docs/tools/run_scenario.md`\r\n- **Orch CLI:** `docs/tools/orch_cli_readme.md`\r\n- **HMAC signer:** `docs/tools/sign_hmac.md`\r\n- **HMAC curl wrapper:** `docs/tools/hmac_curl.md`\r\n- **HMAC verify:** `docs/tools/verify_hmac_against_signer_readme.md`\r\n- **Housekeeping:** `docs/tools/housekeeping.md`\r\n\r\n**Releases**\r\n- `docs/releases/v0.5.md` (najnowsze)\r\n- `docs/releases/v0.4.md`, `v0.3.md`, `v0.2.md`, `v0.1.md`\r\n\r\n---\r\n\r\n## Testy \u0026 jakość\r\n\r\n```bash\r\n# testy\r\npytest -q\r\n\r\n# format/lint (pre-commit)\r\npre-commit run --all-files\r\n```\r\n\r\n---\r\n\r\n## Licencja\r\n\r\nMIT\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falleyesonmypon%2Fopslog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falleyesonmypon%2Fopslog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falleyesonmypon%2Fopslog/lists"}