{"id":31079552,"url":"https://github.com/usekarma/adage-fabric","last_synced_at":"2026-05-03T20:35:21.516Z","repository":{"id":314643116,"uuid":"1056182346","full_name":"usekarma/adage-fabric","owner":"usekarma","description":"A universal, governance-first pattern for unifying event streams with Kafka, ClickHouse, and Grafana.","archived":false,"fork":false,"pushed_at":"2025-09-13T18:24:18.000Z","size":12,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-13T20:32:36.636Z","etag":null,"topics":["adage","clickhouse","event-streaming","fabric","governance","grafana","kafka","observability","sre"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/usekarma.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-13T15:02:03.000Z","updated_at":"2025-09-13T18:50:21.000Z","dependencies_parsed_at":"2025-09-13T20:32:41.247Z","dependency_job_id":"d77b20a2-1779-4c61-82ad-49e6d58b3b08","html_url":"https://github.com/usekarma/adage-fabric","commit_stats":null,"previous_names":["usekarma/adage-fabric"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/usekarma/adage-fabric","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usekarma%2Fadage-fabric","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usekarma%2Fadage-fabric/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usekarma%2Fadage-fabric/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usekarma%2Fadage-fabric/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/usekarma","download_url":"https://codeload.github.com/usekarma/adage-fabric/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usekarma%2Fadage-fabric/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32584219,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T06:36:36.687Z","status":"ssl_error","status_checked_at":"2026-05-03T06:36:09.306Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adage","clickhouse","event-streaming","fabric","governance","grafana","kafka","observability","sre"],"created_at":"2025-09-16T10:24:56.192Z","updated_at":"2026-05-03T20:35:21.317Z","avatar_url":"https://github.com/usekarma.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fabric\n\n**Fabric** is a universal, governance‑first pattern for unifying event streams.  \nIt runs on **Kafka (Redpanda) + ClickHouse + Grafana** and provides a consistent path:\n\n```\nraw → parsed → facts → views\n```\n\n## Why Fabric?\n\nModern systems emit countless signals — CDC events, alerts, tickets, commits, logs. Each siloed stream makes sense in isolation, but operators need a single fabric to see the whole picture.\n\nFabric gives you:\n\n- **Simplicity:** Only Kafka, ClickHouse, and Grafana required.\n- **Governance:** TTLs, cost bounds, and role separation baked in.\n- **Extensibility:** Add new sources by defining raw → parsed mappings.\n- **Observability:** Grafana dashboards on top of facts and views.\n- **ML‑ready:** Expose stable views for downstream enrichment and modeling.\n\n## Quickstart\n\n```bash\ngit clone https://github.com/usekarma/adage-fabric.git\ncd adage-fabric\n\n# 1) Start core services (Redpanda, ClickHouse, Grafana)\nmake up\n\n# 2) Bootstrap shared tables, lanes, and views (schema only)\nmake sql.bootstrap\n```\n\nThis starts:\n\n- Redpanda (Kafka API) on :9092  \n- ClickHouse on :8123 (HTTP) and :9000 (native)  \n- Grafana on :3000 (admin/admin, with starter dashboards)\n\nFeed sample fixtures into Kafka and query in ClickHouse or view dashboards at \u003chttp://localhost:3000\u003e.\n\n### Optional add‑ons (profiles)\n\nServices are opt‑in via **Compose profiles** so they never auto‑start during `make up` / `make bootstrap`:\n\n- **Kafka UI** (profile: `ui`): `make kafka-ui` or `docker compose --profile ui up -d kafka-ui` → \u003chttp://localhost:8080\u003e\n- **Flink** (profile: `flink`): `docker compose --profile flink up -d`\n- **Kafka Connect (Debezium)** (profile: `connect`): `docker compose --profile connect up -d`\n- **MongoDB (replica set for CDC)** (profile: `mongo`): `docker compose --profile mongo up -d`\n\nIf you want *everything* (core + UI) at once:\n```bash\nmake up.all\n```\n\n### Useful commands\n\n```bash\nmake help        # show commands and descriptions\nmake down        # stop containers\nmake restart     # restart core stack\nmake logs        # stream container logs\nmake clean       # stop + remove volumes (fresh state)\n\n# Kafka helpers\nmake feed FILE=fixtures/case_orders_happy.jsonl TOPIC=adage.demo.mongodb.cdc.v1\nmake topic.ensure TOPIC=adage.demo.mongodb.cdc.v1\nmake topic.create TOPIC=adage.demo.mongodb.cdc.v1\n\n# Lanes (DDL)\ncd sql\nmake lane SOURCE=mongodb STREAM=cdc ORG=adage DOMAIN=demo VERSION=1\nmake lanes       # apply all rendered lanes/*.sql\n```\n\n## How it works (Architecture)\n\n1) **Raw**\n   - Ingest directly from Kafka (Redpanda) using a ClickHouse **Kafka Engine** table.\n   - Short TTL (e.g., 3 days). Append‑only; no expensive transforms here.\n   - Responsibility: get bytes in reliably.\n\n2) **Parsed**\n   - Materialized View (MV) from `raw` extracts JSON once and writes to a typed **MergeTree** table.\n   - Longer TTL (e.g., 90 days). Stable column types; add light denorm if helpful.\n   - Responsibility: make events queryable and inexpensive to scan.\n\n3) **Facts**\n   - Domain rollups (counts, rates, latencies) with clear grains (minute/hour/day).\n   - Prefer additive engines (e.g., `SummingMergeTree`) or snapshots with explicit grain keys.\n   - Responsibility: cheap dashboards and SLO views.\n\n4) **Views**\n   - Semantic SELECTs (e.g., `vw_events`, `vw_alerts`, `vw_changes`) joining multiple sources.\n   - This layer is what Grafana and downstream ML read from; treat as your contract.\n   - Responsibility: stable API for external consumption.\n\n\u003e TL;DR: Kafka Engine ➜ MV ➜ MergeTree ➜ Facts ➜ Views\n\n## Querying ClickHouse\n\n**Interactive client inside the container:**\n```bash\ndocker compose exec -it clickhouse clickhouse-client\n```\nThen run SQL:\n```sql\nSHOW DATABASES;\nUSE default;\nSHOW TABLES;\nSELECT * FROM parsed_mongodb_cdc LIMIT 10;\n```\n\n**One‑liner from host:**\n```bash\ndocker compose exec -i clickhouse clickhouse-client -q \"SHOW TABLES FROM default\"\n```\n\n**Web UI:**  \nOpen \u003chttp://localhost:8123/play\u003e for a simple SQL console.\n\n**Grafana:**  \nGrafana (\u003chttp://localhost:3000\u003e) is pre‑configured to use ClickHouse as a datasource. Dashboards query tables like `fact_events_minute_all` and `fact_latency_minute_all`.\n\n## Testing\n\nYou can run the entire matrix or a single case.\n\n**Run the whole matrix** (defaults to `tests/test_matrix.txt`):  \n```bash\nmake test.all\n```\nOverride the matrix file:  \n```bash\nmake test.all MATRIX=tests/smoke.txt\n```\n\n**Run one named case** (convenience target):  \n```bash\nmake test.orders\n```\n\n**Or call the runner directly:**  \n```bash\n./scripts/run_case.sh case_orders_happy\n```\n\n\u003e The matrix format is one case name per line; blank lines and `#` comments are ignored. Example:\n\u003e ```\n\u003e # tests/test_matrix.txt\n\u003e case_orders_happy\n\u003e # case_orders_delete   (uncomment when fixture is ready)\n\u003e ```\n\n**Generate expected snapshots** from current DB state (example for parsed):  \n```bash\n./scripts/query.sh \"\n  SELECT ts_event, source, event_type, ns, event_id, severity, op, status\n  FROM parsed_mongodb_cdc\n  ORDER BY ts_event, event_id\n  FORMAT CSV\" \u003e expected/case_orders_happy.parsed.csv\n```\nRepeat for facts, etc. Future runs will diff against these CSVs.\n\n\n**Generate expected snapshots** with Make targets:\n\n```bash\n# parsed snapshot\nmake snap.parsed CASE=case_orders_happy\n\n# facts snapshot (defaults to fact_events_minute_all)\nmake snap.facts CASE=case_orders_happy\n\n# override fact table/columns\nmake snap.facts CASE=case_orders_happy FACT_TABLE=fact_latency_minute_all FACT_COLUMNS='t_min, p50_ms, p95_ms, p99_ms'\n\n# custom query snapshot (you supply SQL that ends with FORMAT CSV)\nmake snap.custom CASE=case_orders_happy NAME=my_view \\\\\n  SQL='SELECT * FROM vw_events ORDER BY ts_event FORMAT CSV'\n\n# do parsed + default facts in one shot\nmake snap.all CASE=case_orders_happy\n```\n\n\n## Repo Structure\n\n```\nadage-fabric/\n  Makefile                 # root: compose orchestration + sql bootstrap\n  docker-compose.yml       # core services + optional services behind profiles\n  provisioning/            # environment provisioning assets (e.g., dashboards, configs)\n  dashboards/              # Grafana dashboards\n  sql/                     # schema Makefile + DDL templates\n    Makefile               # schema only (no docker compose)\n    templates/             # lane template (envsubst)\n    lanes/                 # rendered lane SQL\n    shared/                # global rollups and views\n  connectors/              # (optional) Kafka Connect configs (e.g., mongodb-cdc.json)\n  fixtures/                # JSONL input files (test cases)\n  expected/                # expected CSV snapshots\n  scripts/                 # feed, query, reset\n  tests/                   # test matrix and helpers\n  docs/                    # design notes\n```\n\n## Documentation\n\n- `docs/fact_template.md` — checklist for designing facts.  \n- `docs/join_strategy.md` — where to join (CH, Grafana, ML).  \n\n## Roadmap\n\n- [ ] Add sample source mappings (Mongo CDC, Jira, Splunk).  \n- [ ] Provide canonical dims (`service`, `ticket_id`, `env`).  \n- [ ] Ship async enrichment slot (entropy, anomalies, embeddings).  \n- [ ] Expand starter Grafana dashboards.  \n- [ ] Tag `v0.1` once schema and dashboards are stable for demo.\n\n## Optional niceties\n\n- Parametrize the topic per case (`adage.demo.${CASE}.v1`) to avoid consumer‑offset interference (or run `ALTER TABLE kafka_mongodb_cdc MODIFY SETTING kafka_group_name='fabric_demo_$$RANDOM'` before each case).\n- If you later add async enrichment, add expected CSVs for the enriched view too.\n\n## Suggestions to polish / demo improvements\n\n| Area | Suggestion |\n|---|---|\n| **Examples** | Add more fixture cases: e.g., delete events, late arrivals, out-of-order timestamps — to stress parsing logic in obvious ways. |\n| **Dashboard defaults** | Ship Grafana dashboards that display something out-of-the-box (e.g., events per minute from facts). Provide a sample fixture so panels aren’t blank on first run. |\n| **Version tagging / releases** | Tag `v0.1` when reaching a minimum usable demo state. |\n| **Readme front matter** | Add a screenshot (Grafana panel) once populated. Consider an FAQ clarifying *what Fabric is* vs *what it isn’t* (demo vs production). |\n\n## License\n\nApache 2.0 — free to use, adapt, and extend.  \nAuthored by [usekarma](https://github.com/usekarma).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fusekarma%2Fadage-fabric","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fusekarma%2Fadage-fabric","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fusekarma%2Fadage-fabric/lists"}