{"id":51441265,"url":"https://github.com/mirlan-irokez/rawbbit","last_synced_at":"2026-07-24T04:01:24.651Z","repository":{"id":360015008,"uuid":"1229145922","full_name":"mirlan-irokez/rawbbit","owner":"mirlan-irokez","description":null,"archived":false,"fork":false,"pushed_at":"2026-07-07T17:44:54.000Z","size":1301,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-07-07T19:14:48.560Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://rawbbit.one/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mirlan-irokez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-04T18:43:26.000Z","updated_at":"2026-07-07T17:45:09.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mirlan-irokez/rawbbit","commit_stats":null,"previous_names":["mirlan-irokez/dataqueryevent","mirlan-irokez/rawbbit"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mirlan-irokez/rawbbit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirlan-irokez%2Frawbbit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirlan-irokez%2Frawbbit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirlan-irokez%2Frawbbit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirlan-irokez%2Frawbbit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mirlan-irokez","download_url":"https://codeload.github.com/mirlan-irokez/rawbbit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirlan-irokez%2Frawbbit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35826032,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-24T02:00:07.870Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-05T12:00:28.888Z","updated_at":"2026-07-24T04:01:24.641Z","avatar_url":"https://github.com/mirlan-irokez.png","language":"Python","funding_links":[],"categories":["Data Ingestion"],"sub_categories":[],"readme":"# Rawbbit\n\nRawbbit is a self-hosted event telemetry pipeline for mobile and web games.\nIt helps small teams collect player events, sessions, funnels, retention signals, economy events, monetization events, and backend game-service events while keeping ownership of the raw data.\n\nIt was created for teams that want game analytics without making a proprietary vendor the center of their telemetry stack.\nThe same pipeline also works for web apps, mobile apps, browser surfaces, and other event-producing products, but the OSS docs are written first for game developers operating their own stack.\n\nGame clients, web builds, mobile apps, and backend services send batched events over HTTP.\nRawbbit validates and enriches those events in the collector, buffers them through [NATS JetStream](https://nats.io/), writes partitioned Parquet files to S3-compatible object storage such as [SeaweedFS](https://github.com/seaweedfs/seaweedfs), loads the raw layer into ClickHouse, and exposes the ClickHouse-backed data through MCP, Metabase, agents, and direct SQL access.\n\n![Rawbbit](./assets/Rawbbit.png)\n\nMain Rawbbit path:\n\n```text\nProducer -\u003e Collector API -\u003e NATS JetStream -\u003e Raw Writer\n  -\u003e S3-compatible object storage / SeaweedFS Parquet\n  -\u003e ClickHouse loader cron job\n  -\u003e ClickHouse\n  -\u003e MCP / Metabase / agents / users\n```\n\nOptional BigQuery path:\n\n```text\nProducer -\u003e Collector API -\u003e NATS JetStream -\u003e Raw Writer\n  -\u003e object storage Parquet\n  -\u003e BigQuery external table\n  -\u003e SQLMesh base model\n```\n\nStorage note:\n- SeaweedFS/S3-compatible storage is the preferred OSS raw-storage path\n- GCS remains supported\n- the BigQuery external-table path remains a supported optional downstream path, not the main architecture\n\nProduction-oriented OSS deployment is split into two provider-neutral VM\nguides:\n\n- VM one runs ingestion and raw object storage: Caddy, NATS JetStream,\n  collector-api, raw-writer, and SeaweedFS\n- VM two runs analytics and access surfaces: ClickHouse, the Rawbbit MCP\n  server, Metabase, and Postgres for Metabase state\n\nThe boundary between the VMs is the raw Parquet layer in S3-compatible object\nstorage.\n\n## Table of contents\n\n- [What is included](#what-is-included)\n- [Architecture](#architecture)\n- [Quickstart](#quickstart)\n- [Published images](#published-images)\n- [Configuration](#configuration)\n- [Repository layout](#repository-layout)\n- [Project status](#project-status)\n- [Documentation](#documentation)\n- [License](#license)\n\n## What is included\n\nThis repository contains the public game-telemetry ingestion-to-analytics path:\n\n- `backend/collector-api` — HTTP ingestion service for player, gameplay, product, and server events\n- `backend/raw-writer` — JetStream consumer that writes partitioned Parquet files\n- `backend/deploy/` — Docker Compose and environment scaffolding for local or simple self-hosted setups\n- `clickhouse/` — main analytical/query path for loading and querying raw game telemetry in ClickHouse\n- `mcp-server/` — Rawbbit MCP server, MCP client setup, and combined MCP + Metabase deployment guide\n- AI agents and MCP clients such as Codex, OpenCode, and OpenClaw can connect to the Rawbbit MCP server endpoint for read-only analytics exploration\n- `metabase/` — optional standalone Metabase deployment guide\n- `sqlmesh_project/` — optional [SQLMesh](https://sqlmesh.readthedocs.io/en/stable/) project for the BigQuery external-table path\n\n## Architecture\n\nThe system is built around a few explicit boundaries:\n\n- game clients, web builds, mobile apps, and backend services produce event batches\n- the collector accepts and validates player and gameplay events\n- [NATS JetStream](https://nats.io/) separates request handling from storage writes\n- the raw writer lands durable Parquet files in object storage\n- raw Parquet is the system-of-record boundary for downstream telemetry work\n- ClickHouse is the main analytical database and serving path for retention, funnel, session, and event analysis\n- MCP, Metabase, agents, and direct SQL are consumption surfaces on top of ClickHouse\n- downstream modeling can evolve without changing the ingestion contract\n\nFor the deeper architecture note, see [`docs/architecture.md`](docs/architecture.md).\n\n## Quickstart\n\nFor the recommended provider-neutral ingestion VM deployment guide, see\n[`quickstart/vm_rawbbit_one/README.md`](quickstart/vm_rawbbit_one/README.md).\nIt deploys Caddy, NATS JetStream, collector-api, raw-writer, and SeaweedFS with\npersistent host storage.\n\nFor the matching analytics VM deployment guide, see\n[`quickstart/vm_rawbbit_two/README.md`](quickstart/vm_rawbbit_two/README.md).\nIt deploys ClickHouse, the Rawbbit MCP server, Metabase, and Postgres for\nMetabase state. It reads Parquet from the VM-one S3-compatible endpoint and\nloads it into ClickHouse.\n\nFor local validation and development, keep using [`docs/quickstart.md`](docs/quickstart.md).\n\nThe shortest path to a working local setup is:\n\n1. copy `backend/deploy/.env.example` to `backend/deploy/.env`\n2. set API keys, object-storage bucket, and credentials\n3. start the stack with Docker Compose\n4. send a test player event to `POST /v1/events:batch`\n5. verify that Parquet files land in object storage\n\nFor the full local walkthrough, see [`docs/quickstart.md`](docs/quickstart.md).\n\n## Published images\n\nRawbbit publishes public service images to GHCR:\n\n- `ghcr.io/mirlan-irokez/rawbbit-collector-api:0.1.7`\n- `ghcr.io/mirlan-irokez/rawbbit-raw-writer:0.1.8`\n- `ghcr.io/mirlan-irokez/rawbbit-mcp-server:0.0.2`\n\nThe same images also have `latest` tags for convenience. Prefer pinned version tags for deployments.\n\nYou can run the stack in two ways:\n\n- build from source with Docker Compose while developing or validating local changes\n- pull the public GHCR images for repeatable OSS deployments\n\nSecrets are not baked into these images. Provide API keys, storage credentials, ClickHouse passwords, and MCP bearer tokens through private `.env` files or secret management.\n\n## Configuration\n\nThe canonical environment-variable reference is `backend/deploy/.env.example`.\n\nImportant configuration groups:\n\n- NATS and stream settings\n- collector API limits, API keys, CORS settings, and optional GeoIP-related attribution requirements\n- raw-writer batching and ACK behavior\n- object-storage bucket, prefix, and credentials\n\nFor the grouped configuration guide, see [`docs/configuration.md`](docs/configuration.md).\n\n## Repository layout\n\n```text\nbackend/\n  collector-api/   HTTP ingestion service\n  raw-writer/      Parquet landing worker\n  deploy/          Local and self-hosted runtime scaffolding\nsqlmesh_project/   Optional BigQuery SQLMesh starter model\nclickhouse/        Main ClickHouse query/loading path\nmcp-server/        Rawbbit MCP server and optional Metabase deploy path\nmetabase/          Metabase OSS ver. deploy instructions\nquickstart/        Provider-neutral VM deployment guides\ndocs/              OSS documentation\n```\n\nComponent reference notes:\n\n- [`backend/README.md`](backend/README.md)\n- [`backend/collector-api/README.md`](backend/collector-api/README.md)\n- [`backend/raw-writer/README.md`](backend/raw-writer/README.md)\n- [`backend/deploy` runtime notes](backend/deploy/README.md)\n- [`clickhouse/README.md`](clickhouse/README.md)\n- [`mcp-server/README.md`](mcp-server/README.md)\n- [`metabase/README.md`](metabase/README.md)\n\n## Project status\n\nCurrent maturity:\n\n- player event ingestion path is implemented\n- raw Parquet landing path is implemented\n- raw storage backend selection is implemented for both GCS and S3-compatible targets\n- SeaweedFS/S3-compatible storage is the preferred OSS raw-storage path\n- ClickHouse loading from raw Parquet is the main analytical/query path for player telemetry\n- Rawbbit MCP server can expose a read-only analytical tool surface over a configured Rawbbit ClickHouse events table\n- AI agents and MCP clients can use that MCP surface without direct access to the ingestion runtime\n- Metabase can be deployed separately or together with the Rawbbit MCP server package\n- BigQuery external-table querying is supported as an optional path\n- [SQLMesh](https://sqlmesh.readthedocs.io/en/stable/) is included as an optional starter layer for the BigQuery path\n\nThe current release is intentionally narrow: it focuses on reliable player-event ingestion, durable raw storage, ClickHouse-backed analytics, and agent/BI access on top of that serving layer.\n\nThe included [SQLMesh](https://sqlmesh.readthedocs.io/en/stable/) model is intentionally small. It reads from the BigQuery external table over the raw Parquet layer and serves as an optional BigQuery starter path rather than the central Rawbbit modeling layer.\n\n## Documentation\n\n- [`docs/architecture.md`](docs/architecture.md)\n- [`quickstart/vm_rawbbit_one/README.md`](quickstart/vm_rawbbit_one/README.md)\n- [`quickstart/vm_rawbbit_two/README.md`](quickstart/vm_rawbbit_two/README.md)\n- [`docs/quickstart.md`](docs/quickstart.md)\n- [`docs/configuration.md`](docs/configuration.md)\n- [`clickhouse/README.md`](clickhouse/README.md)\n- [`mcp-server/README.md`](mcp-server/README.md) — includes Codex, OpenCode, and OpenClaw MCP client examples\n- [`metabase/README.md`](metabase/README.md)\n\n## License\n\nThis project is released under the Apache License 2.0. See [`LICENSE`](LICENSE).\n\n---\n\n## Inspired by\n\n- [awesome-data-engineering](https://github.com/igorbarinov/awesome-data-engineering) — for the broader data engineering ecosystem\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmirlan-irokez%2Frawbbit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmirlan-irokez%2Frawbbit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmirlan-irokez%2Frawbbit/lists"}