{"id":49088579,"url":"https://github.com/enoughdrama/cross-services-analytics","last_synced_at":"2026-04-20T17:11:16.168Z","repository":{"id":351845477,"uuid":"1211922928","full_name":"enoughdrama/cross-services-analytics","owner":"enoughdrama","description":"Unified subscription, entitlement \u0026 analytics platform across payment rails (Stripe, PayPal, Apple, Google, custom). Node 22 · TypeScript strict · Fastify · Mongo · NATS JetStream · ClickHouse · Next.js.","archived":false,"fork":false,"pushed_at":"2026-04-16T19:06:01.000Z","size":2255,"stargazers_count":0,"open_issues_count":15,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-16T19:33:11.144Z","etag":null,"topics":["analytics","clickhouse","event-sourcing","fastify","microservices","mongodb","nats-jetstream","nextjs","nodejs","paypal","stripe","subscriptions","typescript"],"latest_commit_sha":null,"homepage":"https://github.com/enoughdrama/cross-services-analytics","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/enoughdrama.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["enoughdrama"]}},"created_at":"2026-04-15T22:17:05.000Z","updated_at":"2026-04-16T19:06:05.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/enoughdrama/cross-services-analytics","commit_stats":null,"previous_names":["enoughdrama/cross-services-analytics"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/enoughdrama/cross-services-analytics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/enoughdrama%2Fcross-services-analytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/enoughdrama%2Fcross-services-analytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/enoughdrama%2Fcross-services-analytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/enoughdrama%2Fcross-services-analytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/enoughdrama","download_url":"https://codeload.github.com/enoughdrama/cross-services-analytics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/enoughdrama%2Fcross-services-analytics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32056996,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T11:35:06.609Z","status":"ssl_error","status_checked_at":"2026-04-20T11:34:48.899Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","clickhouse","event-sourcing","fastify","microservices","mongodb","nats-jetstream","nextjs","nodejs","paypal","stripe","subscriptions","typescript"],"created_at":"2026-04-20T17:11:15.399Z","updated_at":"2026-04-20T17:11:16.163Z","avatar_url":"https://github.com/enoughdrama.png","language":"TypeScript","funding_links":["https://github.com/sponsors/enoughdrama"],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# cross-services-analytics\n\n**Unified subscription, entitlement \u0026 analytics platform across payment rails.**\n\nAggregates subscription and payment events from Stripe, PayPal, Apple, Google Play and custom rails into a single entitlement model, exposes a cross-platform read API for client apps, and ships with an Adapty-class analytics dashboard.\n\n[![CI](https://github.com/enoughdrama/cross-services-analytics/actions/workflows/ci.yml/badge.svg)](https://github.com/enoughdrama/cross-services-analytics/actions/workflows/ci.yml)\n[![CodeQL](https://github.com/enoughdrama/cross-services-analytics/actions/workflows/codeql.yml/badge.svg)](https://github.com/enoughdrama/cross-services-analytics/actions/workflows/codeql.yml)\n[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)\n[![Node](https://img.shields.io/badge/node-22_LTS-43853d?logo=node.js\u0026logoColor=white)](./.nvmrc)\n[![TypeScript](https://img.shields.io/badge/TypeScript-strict-3178c6?logo=typescript\u0026logoColor=white)](./tsconfig.base.json)\n[![pnpm](https://img.shields.io/badge/pnpm-9-f69220?logo=pnpm\u0026logoColor=white)](./pnpm-workspace.yaml)\n\n\u003c/div\u003e\n\n---\n\n## Table of contents\n\n- [Overview](#overview)\n- [Architecture](#architecture)\n- [Tech stack](#tech-stack)\n- [Repository layout](#repository-layout)\n- [Getting started](#getting-started)\n- [Development workflow](#development-workflow)\n- [Testing](#testing)\n- [Deployment](#deployment)\n- [Roadmap](#roadmap)\n- [Contributing](#contributing)\n- [Security](#security)\n- [License](#license)\n\n## Overview\n\n`cross-services-analytics` (internal codename **subController**, package scope `@sc/*`) is a microservice platform built on Node.js 22 + TypeScript that ingests payment-rail webhooks, normalizes them into a provider-agnostic domain model, and projects the result into two complementary stores:\n\n- **MongoDB** — source-of-truth for subscriptions, entitlements and ledger entries.\n- **ClickHouse** — columnar store for event-level analytics, cohorts, funnels, retention and revenue metrics.\n\nThe platform is a **single-tenant, owner-operated** system — not a SaaS. It is optimised for maximum-quality engineering over feature breadth: strict TypeScript, pure domain layer, idempotent ingestion end-to-end, transactional outbox, append-only event log, and fully replayable analytics.\n\n## Architecture\n\n```\n                     ┌──────────────────────────────────────────────┐\n                     │               Payment rails                  │\n                     │  Stripe · PayPal · Apple · Google · custom   │\n                     └──────────────────────┬───────────────────────┘\n                                            │ webhooks\n                                            ▼\n                          ┌──────────────────────────────┐\n                          │       ingress-gateway        │   signature verify\n                          │   (Fastify, idempotent)      │   + Redis SET NX\n                          └──────────────┬───────────────┘\n                                         │ raw.\u003cprovider\u003e.* (append-only)\n                                         ▼\n                         ┌────────────────────────────────┐\n                         │  NATS JetStream (event bus)    │\n                         └───┬────────────────────────┬───┘\n                             │                        │\n                             ▼                        ▼\n               ┌─────────────────────┐      ┌─────────────────────┐\n               │ normalizer-\u003crail\u003e   │      │   analytics-svc     │\n               │  raw.* → domain.*   │      │  domain.* → CH      │\n               └──────────┬──────────┘      └──────────┬──────────┘\n                          │ domain.*                   │\n                          ▼                            ▼\n               ┌─────────────────────┐      ┌─────────────────────┐\n               │  entitlement-svc    │      │     ClickHouse      │\n               │  (sole writer of    │      │ events_raw + typed  │\n               │   Mongo state)      │      │  materialized views │\n               └──────────┬──────────┘      └──────────┬──────────┘\n                          │                            │\n                          ▼                            ▼\n               ┌─────────────────────┐      ┌─────────────────────┐\n               │   client apps       │      │      admin-api      │\n               │ (public read API)   │      │    (BFF for UI)     │\n               └─────────────────────┘      └──────────┬──────────┘\n                                                       │\n                                                       ▼\n                                            ┌─────────────────────┐\n                                            │     admin-web       │\n                                            │ Next.js · shadcn/ui │\n                                            │      · Tremor       │\n                                            └─────────────────────┘\n```\n\n### Core invariants\n\n| # | Invariant |\n|---|-----------|\n| 1 | `entitlement-svc` is the **only writer** of `subscriptions`, `entitlements`, `ledger_entries`. |\n| 2 | **End-to-end idempotency** — Redis `SET NX` on ingress + JetStream `Nats-Msg-Id` dedup + Mongo `processed_events` unique index. |\n| 3 | **Transactional outbox** — `domain.entitlement.changed` is written in the same Mongo transaction as the state change. |\n| 4 | **Pure domain layer** (`@sc/domain`) has zero I/O dependencies and is 100 % unit-tested. |\n| 5 | The `raw.*` stream in NATS is **append-only forever** — full replay is always possible if normalization logic changes. |\n| 6 | Events are **versioned by subject prefix** (`domain.subscription.renewed.v1`); new versions are additive. |\n\n### Provider adapter contract\n\nNew payment rails are added by implementing three interfaces exported from `@sc/providers`:\n\n```ts\ninterface IProviderIngress     { verify(req): Promise\u003cRawEnvelope\u003e; }\ninterface IProviderNormalizer  { normalize(raw): Promise\u003cDomainEvent[]\u003e; }\ninterface IProviderActions     { cancel(sub): Promise\u003cvoid\u003e; refund(...): Promise\u003cvoid\u003e; }\n```\n\nStripe and PayPal ship as full implementations. Apple, Google Play and RU acquiring are stubs with full JSDoc contracts and `NotImplementedError` — registering a new rail requires no core changes.\n\n## Tech stack\n\n| Layer | Choice |\n|---|---|\n| Runtime | Node.js 22 LTS |\n| Language | TypeScript (strict, `exactOptionalPropertyTypes`, no implicit any, no unchecked indexed access) |\n| HTTP | Fastify 5 + Zod |\n| Source of truth | MongoDB 7 (3-node replica set, transactions) |\n| Event bus / store | NATS JetStream |\n| Analytics store | ClickHouse |\n| Cache / locks / queues | Redis 7 + BullMQ |\n| Frontend | Next.js 15 · shadcn/ui · Tremor |\n| Workspace | pnpm 9 + Turborepo |\n| Tests | Vitest + Testcontainers |\n| Observability | OpenTelemetry → Grafana + Prometheus + Loki + Tempo |\n| Deploy | Docker Compose + Caddy + Let's Encrypt |\n| Secrets | SOPS + age |\n\n## Repository layout\n\n```\napps/\n  ingress-gateway/      # webhook receive, signature verify, raw.* publish\n  normalizer-stripe/    # raw.stripe.* → domain.*\n  normalizer-paypal/    # raw.paypal.* → domain.*\n  normalizer-client/    # raw.client.* → domain.* (first-party SDK events)\n  entitlement-svc/      # sole Mongo writer; public client-app read API\n  analytics-svc/        # domain.* → ClickHouse projections\n  admin-api/            # BFF for the dashboard\n  admin-web/            # Next.js admin dashboard\n\npackages/\n  contracts/            # Zod schemas for HTTP endpoints\n  events/               # Zod schemas for NATS subjects (versioned)\n  domain/               # pure domain — no I/O\n  mongo/                # shared driver wrapper + typed repositories\n  nats/                 # JetStream helpers and durable consumers\n  providers/            # IProvider* contracts + registry\n  observability/        # OTel bootstrap\n\ndocs/                   # hand-written operator docs\ninfra/                  # dev-infra config (Mongo/NATS/Redis/CH/Grafana)\ndocker/                 # base Dockerfiles\nscripts/                # dev/ops shell scripts\n```\n\n## Getting started\n\n### Prerequisites\n\n- **Node.js 22 LTS** (`.nvmrc` / `.node-version` pin)\n- **pnpm 9** (`corepack enable`)\n- **Docker** + Compose v2\n- **Stripe CLI** (optional, for local webhook forwarding)\n\n### 1. Install\n\n```bash\ncorepack enable\npnpm install\n```\n\n### 2. Configure credentials\n\nThe dev-environment file `.env.dev` is gitignored. Populate it with test-scope credentials:\n\n```bash\ncp .env.dev.example .env.dev   # fill in values\n```\n\nRequired keys (see inline comments in the example):\n\n- `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET`, `STRIPE_PRICE_REGULAR`, `STRIPE_PRICE_PREMIUM`\n- `PAYPAL_CLIENT_ID`, `PAYPAL_CLIENT_SECRET`, `PAYPAL_MODE`\n- `ADMIN_API_KEY`, `ADMIN_SESSION_SECRET`\n\n### 3. Provider config maps\n\n```bash\ncp apps/ingress-gateway/config/apps.example.json               apps/ingress-gateway/config/apps.json\ncp apps/normalizer-stripe/config/stripe-products.example.json  apps/normalizer-stripe/config/stripe-products.json\ncp apps/normalizer-paypal/config/paypal-products.example.json  apps/normalizer-paypal/config/paypal-products.json\n```\n\n### 4. Bring up dev infrastructure\n\n```bash\npnpm infra:up        # Mongo RS · NATS · Redis Stack · ClickHouse · Grafana\npnpm infra:check     # sanity-checks every component\n```\n\n### 5. Seed an admin user\n\n```bash\npnpm --filter @sc/admin-api seed:admin admin@local admin123\n```\n\n### 6. Run all services\n\n```bash\nbash scripts/run-dev-all.sh\n```\n\n### 7. Open the dashboard\n\nVisit **[http://localhost:3020](http://localhost:3020)** and log in as `admin@local` / `admin123`.\n\nForward Stripe webhooks in a separate shell:\n\n```bash\nstripe listen --forward-to localhost:3010/webhooks/stripe\n```\n\n## Development workflow\n\n| Command | Purpose |\n|---|---|\n| `pnpm dev`            | Run every service with `tsx watch` |\n| `pnpm build`          | Build every package and app (Turborepo) |\n| `pnpm lint`           | ESLint across the workspace |\n| `pnpm typecheck`      | Strict TypeScript compile check |\n| `pnpm test`           | Vitest (unit + integration via Testcontainers) |\n| `pnpm coverage`       | Vitest coverage, per-package |\n| `pnpm format`         | Prettier write |\n| `pnpm infra:up`       | Start dev infra containers |\n| `pnpm infra:down`     | Stop \u0026 remove dev infra containers |\n| `pnpm infra:logs`     | Tail dev infra logs |\n| `pnpm infra:check`    | Health-check every dev infra component |\n\n## Testing\n\n- **Unit tests** live next to the code and cover the pure domain layer exhaustively.\n- **Integration tests** use Testcontainers to boot throw-away Mongo / NATS / Redis / ClickHouse instances.\n- Coverage thresholds (enforced in CI): **90 % lines** on `@sc/domain`, **85 % lines** on every service.\n- Two-`tsconfig` pattern per app — `tsconfig.json` for source, `tsconfig.test.json` with relaxed settings for tests.\n\n## Deployment\n\nProduction is **Docker Compose + Caddy** on a single-tenant home server. See [`docker-compose.prod.yml`](./docker-compose.prod.yml) for the service topology.\n\nSecrets are managed with **SOPS + age**; encrypted files live under `secrets/` and decrypted copies are gitignored.\n\n## Roadmap\n\nThe platform ships in four analytics-uplift phases on top of the core MVP:\n\n| Phase | Scope | Status |\n|---|---|---|\n| **MVP**     | Ingress · normalizers · entitlement-svc · basic analytics · admin UI                         | shipped |\n| **Phase 1** | Enriched client-event pipeline · envelope v2 · cohorts · funnels · retention · typed CH MVs  | shipped |\n| Phase 2     | Placements + A/B paywalls                                                                    | planned |\n| Phase 3     | Predicted LTV                                                                                | planned |\n| Phase 4     | Outgoing integrations + raw-data export                                                      | planned |\n\n## Contributing\n\nSee [CONTRIBUTING.md](./CONTRIBUTING.md). Short version:\n\n1. Create a branch off `main`.\n2. Follow the conventions in the file — strict TS, Zod at every boundary, pure domain, no direct cross-service DB access.\n3. `pnpm lint \u0026\u0026 pnpm typecheck \u0026\u0026 pnpm test \u0026\u0026 pnpm build` must pass locally.\n4. Open a PR using the template. CI must be green; coverage thresholds must hold.\n\n## Security\n\nPlease read [SECURITY.md](./SECURITY.md) before reporting a vulnerability. Do **not** open a public issue for security problems.\n\n## License\n\nReleased under the [MIT License](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fenoughdrama%2Fcross-services-analytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fenoughdrama%2Fcross-services-analytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fenoughdrama%2Fcross-services-analytics/lists"}