https://github.com/enoughdrama/cross-services-analytics
Unified subscription, entitlement & analytics platform across payment rails (Stripe, PayPal, Apple, Google, custom). Node 22 · TypeScript strict · Fastify · Mongo · NATS JetStream · ClickHouse · Next.js.
https://github.com/enoughdrama/cross-services-analytics
analytics clickhouse event-sourcing fastify microservices mongodb nats-jetstream nextjs nodejs paypal stripe subscriptions typescript
Last synced: 2 months ago
JSON representation
Unified subscription, entitlement & analytics platform across payment rails (Stripe, PayPal, Apple, Google, custom). Node 22 · TypeScript strict · Fastify · Mongo · NATS JetStream · ClickHouse · Next.js.
- Host: GitHub
- URL: https://github.com/enoughdrama/cross-services-analytics
- Owner: enoughdrama
- License: mit
- Created: 2026-04-15T22:17:05.000Z (2 months ago)
- Default Branch: main
- Last Pushed: 2026-04-16T19:06:01.000Z (2 months ago)
- Last Synced: 2026-04-16T19:33:11.144Z (2 months ago)
- Topics: analytics, clickhouse, event-sourcing, fastify, microservices, mongodb, nats-jetstream, nextjs, nodejs, paypal, stripe, subscriptions, typescript
- Language: TypeScript
- Homepage: https://github.com/enoughdrama/cross-services-analytics
- Size: 2.15 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 15
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
README
# cross-services-analytics
**Unified subscription, entitlement & analytics platform across payment rails.**
Aggregates subscription and payment events from Stripe, PayPal, Apple, Google Play and custom rails into a single entitlement model, exposes a cross-platform read API for client apps, and ships with an Adapty-class analytics dashboard.
[](https://github.com/enoughdrama/cross-services-analytics/actions/workflows/ci.yml)
[](https://github.com/enoughdrama/cross-services-analytics/actions/workflows/codeql.yml)
[](./LICENSE)
[](./.nvmrc)
[](./tsconfig.base.json)
[](./pnpm-workspace.yaml)
---
## Table of contents
- [Overview](#overview)
- [Architecture](#architecture)
- [Tech stack](#tech-stack)
- [Repository layout](#repository-layout)
- [Getting started](#getting-started)
- [Development workflow](#development-workflow)
- [Testing](#testing)
- [Deployment](#deployment)
- [Roadmap](#roadmap)
- [Contributing](#contributing)
- [Security](#security)
- [License](#license)
## Overview
`cross-services-analytics` (internal codename **subController**, package scope `@sc/*`) is a microservice platform built on Node.js 22 + TypeScript that ingests payment-rail webhooks, normalizes them into a provider-agnostic domain model, and projects the result into two complementary stores:
- **MongoDB** — source-of-truth for subscriptions, entitlements and ledger entries.
- **ClickHouse** — columnar store for event-level analytics, cohorts, funnels, retention and revenue metrics.
The platform is a **single-tenant, owner-operated** system — not a SaaS. It is optimised for maximum-quality engineering over feature breadth: strict TypeScript, pure domain layer, idempotent ingestion end-to-end, transactional outbox, append-only event log, and fully replayable analytics.
## Architecture
```
┌──────────────────────────────────────────────┐
│ Payment rails │
│ Stripe · PayPal · Apple · Google · custom │
└──────────────────────┬───────────────────────┘
│ webhooks
▼
┌──────────────────────────────┐
│ ingress-gateway │ signature verify
│ (Fastify, idempotent) │ + Redis SET NX
└──────────────┬───────────────┘
│ raw..* (append-only)
▼
┌────────────────────────────────┐
│ NATS JetStream (event bus) │
└───┬────────────────────────┬───┘
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ normalizer- │ │ analytics-svc │
│ raw.* → domain.* │ │ domain.* → CH │
└──────────┬──────────┘ └──────────┬──────────┘
│ domain.* │
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ entitlement-svc │ │ ClickHouse │
│ (sole writer of │ │ events_raw + typed │
│ Mongo state) │ │ materialized views │
└──────────┬──────────┘ └──────────┬──────────┘
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ client apps │ │ admin-api │
│ (public read API) │ │ (BFF for UI) │
└─────────────────────┘ └──────────┬──────────┘
│
▼
┌─────────────────────┐
│ admin-web │
│ Next.js · shadcn/ui │
│ · Tremor │
└─────────────────────┘
```
### Core invariants
| # | Invariant |
|---|-----------|
| 1 | `entitlement-svc` is the **only writer** of `subscriptions`, `entitlements`, `ledger_entries`. |
| 2 | **End-to-end idempotency** — Redis `SET NX` on ingress + JetStream `Nats-Msg-Id` dedup + Mongo `processed_events` unique index. |
| 3 | **Transactional outbox** — `domain.entitlement.changed` is written in the same Mongo transaction as the state change. |
| 4 | **Pure domain layer** (`@sc/domain`) has zero I/O dependencies and is 100 % unit-tested. |
| 5 | The `raw.*` stream in NATS is **append-only forever** — full replay is always possible if normalization logic changes. |
| 6 | Events are **versioned by subject prefix** (`domain.subscription.renewed.v1`); new versions are additive. |
### Provider adapter contract
New payment rails are added by implementing three interfaces exported from `@sc/providers`:
```ts
interface IProviderIngress { verify(req): Promise; }
interface IProviderNormalizer { normalize(raw): Promise; }
interface IProviderActions { cancel(sub): Promise; refund(...): Promise; }
```
Stripe and PayPal ship as full implementations. Apple, Google Play and RU acquiring are stubs with full JSDoc contracts and `NotImplementedError` — registering a new rail requires no core changes.
## Tech stack
| Layer | Choice |
|---|---|
| Runtime | Node.js 22 LTS |
| Language | TypeScript (strict, `exactOptionalPropertyTypes`, no implicit any, no unchecked indexed access) |
| HTTP | Fastify 5 + Zod |
| Source of truth | MongoDB 7 (3-node replica set, transactions) |
| Event bus / store | NATS JetStream |
| Analytics store | ClickHouse |
| Cache / locks / queues | Redis 7 + BullMQ |
| Frontend | Next.js 15 · shadcn/ui · Tremor |
| Workspace | pnpm 9 + Turborepo |
| Tests | Vitest + Testcontainers |
| Observability | OpenTelemetry → Grafana + Prometheus + Loki + Tempo |
| Deploy | Docker Compose + Caddy + Let's Encrypt |
| Secrets | SOPS + age |
## Repository layout
```
apps/
ingress-gateway/ # webhook receive, signature verify, raw.* publish
normalizer-stripe/ # raw.stripe.* → domain.*
normalizer-paypal/ # raw.paypal.* → domain.*
normalizer-client/ # raw.client.* → domain.* (first-party SDK events)
entitlement-svc/ # sole Mongo writer; public client-app read API
analytics-svc/ # domain.* → ClickHouse projections
admin-api/ # BFF for the dashboard
admin-web/ # Next.js admin dashboard
packages/
contracts/ # Zod schemas for HTTP endpoints
events/ # Zod schemas for NATS subjects (versioned)
domain/ # pure domain — no I/O
mongo/ # shared driver wrapper + typed repositories
nats/ # JetStream helpers and durable consumers
providers/ # IProvider* contracts + registry
observability/ # OTel bootstrap
docs/ # hand-written operator docs
infra/ # dev-infra config (Mongo/NATS/Redis/CH/Grafana)
docker/ # base Dockerfiles
scripts/ # dev/ops shell scripts
```
## Getting started
### Prerequisites
- **Node.js 22 LTS** (`.nvmrc` / `.node-version` pin)
- **pnpm 9** (`corepack enable`)
- **Docker** + Compose v2
- **Stripe CLI** (optional, for local webhook forwarding)
### 1. Install
```bash
corepack enable
pnpm install
```
### 2. Configure credentials
The dev-environment file `.env.dev` is gitignored. Populate it with test-scope credentials:
```bash
cp .env.dev.example .env.dev # fill in values
```
Required keys (see inline comments in the example):
- `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET`, `STRIPE_PRICE_REGULAR`, `STRIPE_PRICE_PREMIUM`
- `PAYPAL_CLIENT_ID`, `PAYPAL_CLIENT_SECRET`, `PAYPAL_MODE`
- `ADMIN_API_KEY`, `ADMIN_SESSION_SECRET`
### 3. Provider config maps
```bash
cp apps/ingress-gateway/config/apps.example.json apps/ingress-gateway/config/apps.json
cp apps/normalizer-stripe/config/stripe-products.example.json apps/normalizer-stripe/config/stripe-products.json
cp apps/normalizer-paypal/config/paypal-products.example.json apps/normalizer-paypal/config/paypal-products.json
```
### 4. Bring up dev infrastructure
```bash
pnpm infra:up # Mongo RS · NATS · Redis Stack · ClickHouse · Grafana
pnpm infra:check # sanity-checks every component
```
### 5. Seed an admin user
```bash
pnpm --filter @sc/admin-api seed:admin admin@local admin123
```
### 6. Run all services
```bash
bash scripts/run-dev-all.sh
```
### 7. Open the dashboard
Visit **[http://localhost:3020](http://localhost:3020)** and log in as `admin@local` / `admin123`.
Forward Stripe webhooks in a separate shell:
```bash
stripe listen --forward-to localhost:3010/webhooks/stripe
```
## Development workflow
| Command | Purpose |
|---|---|
| `pnpm dev` | Run every service with `tsx watch` |
| `pnpm build` | Build every package and app (Turborepo) |
| `pnpm lint` | ESLint across the workspace |
| `pnpm typecheck` | Strict TypeScript compile check |
| `pnpm test` | Vitest (unit + integration via Testcontainers) |
| `pnpm coverage` | Vitest coverage, per-package |
| `pnpm format` | Prettier write |
| `pnpm infra:up` | Start dev infra containers |
| `pnpm infra:down` | Stop & remove dev infra containers |
| `pnpm infra:logs` | Tail dev infra logs |
| `pnpm infra:check` | Health-check every dev infra component |
## Testing
- **Unit tests** live next to the code and cover the pure domain layer exhaustively.
- **Integration tests** use Testcontainers to boot throw-away Mongo / NATS / Redis / ClickHouse instances.
- Coverage thresholds (enforced in CI): **90 % lines** on `@sc/domain`, **85 % lines** on every service.
- Two-`tsconfig` pattern per app — `tsconfig.json` for source, `tsconfig.test.json` with relaxed settings for tests.
## Deployment
Production is **Docker Compose + Caddy** on a single-tenant home server. See [`docker-compose.prod.yml`](./docker-compose.prod.yml) for the service topology.
Secrets are managed with **SOPS + age**; encrypted files live under `secrets/` and decrypted copies are gitignored.
## Roadmap
The platform ships in four analytics-uplift phases on top of the core MVP:
| Phase | Scope | Status |
|---|---|---|
| **MVP** | Ingress · normalizers · entitlement-svc · basic analytics · admin UI | shipped |
| **Phase 1** | Enriched client-event pipeline · envelope v2 · cohorts · funnels · retention · typed CH MVs | shipped |
| Phase 2 | Placements + A/B paywalls | planned |
| Phase 3 | Predicted LTV | planned |
| Phase 4 | Outgoing integrations + raw-data export | planned |
## Contributing
See [CONTRIBUTING.md](./CONTRIBUTING.md). Short version:
1. Create a branch off `main`.
2. Follow the conventions in the file — strict TS, Zod at every boundary, pure domain, no direct cross-service DB access.
3. `pnpm lint && pnpm typecheck && pnpm test && pnpm build` must pass locally.
4. Open a PR using the template. CI must be green; coverage thresholds must hold.
## Security
Please read [SECURITY.md](./SECURITY.md) before reporting a vulnerability. Do **not** open a public issue for security problems.
## License
Released under the [MIT License](./LICENSE).