https://github.com/rafaesc/cloud-ledger
https://github.com/rafaesc/cloud-ledger
aws awsxray cicd devcontainer docker dynamodb foci java lambda python spring-boot sqs terraform
Last synced: 5 days ago
JSON representation
- Host: GitHub
- URL: https://github.com/rafaesc/cloud-ledger
- Owner: rafaesc
- Created: 2026-06-11T18:18:01.000Z (about 2 months ago)
- Default Branch: master
- Last Pushed: 2026-07-01T16:01:54.000Z (about 1 month ago)
- Last Synced: 2026-07-01T17:29:10.088Z (about 1 month ago)
- Topics: aws, awsxray, cicd, devcontainer, docker, dynamodb, foci, java, lambda, python, spring-boot, sqs, terraform
- Language: Java
- Homepage:
- Size: 2 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# CloudLedger
**An audit-grade, event-sourced financial ledger API on AWS — the primitive layer payment systems are built on.**
[](https://github.com/rafaesc/cloud-ledger/actions/workflows/e2e.yml)
[](https://github.com/rafaesc/cloud-ledger/actions/workflows/deploy-prod.yml)
## What it is
A financial ledger must be tamper-proof, auditable, and correct under concurrent writes. CloudLedger
solves this with an **append-only Aurora event store** — balances are never updated in place; every
deposit, withdrawal, and transfer is an immutable fact (`MoneyDeposited`, `TransferDebited`, …)
replayed on read, so every state an account was ever in is reconstructable. Concurrent writes are
coordinated by an **optimistic-lock fence** (`WHERE version = :expected`) that surfaces contention as
a `409 Conflict` and a retry — never silent data loss.
On right-sized dev infrastructure it sustains **p99 < 200 ms up to ~150 transfers/s** before the tail
knee, with reads served sub-10 ms from a Redis write-through cache (DynamoDB fallback). Full latency
curve and load evidence: [ADR-004](docs/adrs/ADR-004-right-size-before-tuning.md). All infrastructure
is **Terraform** — S3 backend + native lock, per-service KMS CMKs, `local`/`prod` environments.
## Architecture
Commands append events to **Aurora** (atomic with the optimistic-lock fence) → the new balance is
**write-through** to **Redis** → events publish to **SQS** post-commit (transactional-outbox fallback
if SQS is down) → a **Lambda projector** builds the **DynamoDB** read model. Reads hit Redis (sub-ms)
with DynamoDB fallback; Aurora is never on the read path. Collector-less **OpenTelemetry → X-Ray**
traces every hop, propagated across the SQS boundary.
*Aurora + Redis in private subnets; ECS in public subnets reaching AWS via VPC endpoints (no NAT).*
## Demo
Recorded on live AWS (a temporary deploy, since torn down for cost). **Videos in progress — links
coming soon:**
- ▶️ **Concurrent conflict** — two simultaneous transfers → one `409` → retry succeeds, no double-spend
- ▶️ **Projection rebuild** — delete the DynamoDB read model, watch it reconstruct 0→N from the Aurora event log
- ▶️ **Breakpoint under load** — the CloudWatch dashboard saturating as k6 climbs past the knee
Transfer latency vs. arrival rate (prod, 2 vCPU) — flat ~87 ms to 150 rps, p99 knee ~200 rps,
collapse ~300 rps ([ADR-004](docs/adrs/ADR-004-right-size-before-tuning.md)).
## Demo scenarios
Assumes the local stack from [Local development](#local-development) is already up
(`docker compose up -d` + `local-bootstrap.sh` + set variables, API healthy on `http://localhost`). Get a token and
open an account once, then run any of the three scenarios below against it:
```bash
TOKEN=$(curl -s -X POST "$TOKEN_URL" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials&client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&scope=https://api.getcloudledger.com/write%20https://api.getcloudledger.com/read" \
| jq -r '.access_token')
ACCOUNT_ID=$(uuidgen)
curl -s -X POST http://localhost/v1/accounts -H "Authorization: Bearer $TOKEN" \
-H "Idempotency-Key: $(uuidgen)" -H "Content-Type: application/json" \
-d "{\"accountId\":\"$ACCOUNT_ID\",\"currency\":\"USD\"}"
```
**1. Idempotency** — replay a write with the same `Idempotency-Key` → the original `201`, no second event:
```bash
KEY=$(uuidgen)
curl -s -o /dev/null -w '%{http_code}\n' -X POST http://localhost/v1/accounts/$ACCOUNT_ID/deposits \
-H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $KEY" -H "Content-Type: application/json" \
-d '{"amount":100}'
# replay with the SAME key → identical 201, exactly one event in Aurora
curl -s -o /dev/null -w '%{http_code}\n' -X POST http://localhost/v1/accounts/$ACCOUNT_ID/deposits \
-H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $KEY" -H "Content-Type: application/json" \
-d '{"amount":100}'
```
**2. Concurrent conflict** — two same-version writes race on the same account → one `201`, one
`409 {"error":"version_conflict"}` → re-read + retry succeeds. Money neither lost nor created.
Plain curl `&` backgrounding races the two requests but doesn't guarantee they land on the exact
same in-memory version — for a real, repeatable proof of the `409` (fired at true concurrency via
`http.batch`) run [`k6/scenarios/conflict.js`](k6/scenarios/conflict.js) instead:
```bash
curl -s -o /dev/null -w '%{http_code}\n' -X POST http://localhost/v1/accounts/$ACCOUNT_ID/withdrawals \
-H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" -H "Content-Type: application/json" \
-d '{"amount":10}' &
curl -s -o /dev/null -w '%{http_code}\n' -X POST http://localhost/v1/accounts/$ACCOUNT_ID/withdrawals \
-H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" -H "Content-Type: application/json" \
-d '{"amount":10}' &
wait
```
**3. CQRS projection** — the command returns on Aurora commit + Redis write-through (read-your-writes
immediately); the projector catches DynamoDB up within ~seconds. `GET /transactions` reflects it once
projected — no Aurora replay on the read path:
```bash
curl -s http://localhost/v1/accounts/$ACCOUNT_ID/balance -H "Authorization: Bearer $TOKEN" | jq # Redis write-through — fresh immediately
curl -s http://localhost/v1/accounts/$ACCOUNT_ID/transactions -H "Authorization: Bearer $TOKEN" | jq # DynamoDB TXNS# — populated once the projector catches up
```
## Tech stack
| Layer | Technology |
|---|---|
| API language / framework | Java 21 · Spring Boot 4.1 (hexagonal ports & adapters) |
| Lambda language | Python 3.12 (`uv`-managed) — outbox-poller, projector, cleanup |
| Event store | Aurora PostgreSQL — ACID: the optimistic-lock fence + event insert commit in one txn |
| Async fan-out | SQS-first (KMS-encrypted, DLQ) + transactional-outbox fallback → Python projector λ |
| Read model | DynamoDB single-table + GSI · ElastiCache Redis (write-through, 30-min TTL, circuit breaker) |
| Auth | Cognito M2M (Client Credentials JWT); a separate `api/admin` scope for operator endpoints |
| IaC | Terraform — S3 backend + native lock, per-service KMS CMKs, `local`/`prod` envs |
| Observability | Collector-less OpenTelemetry (ADOT, SigV4) → X-Ray Transaction Search · CloudWatch dashboard |
| Load & verification | k6 — smoke/conflict CI gates + load/read-balance/breakpoint sandbox runs |
| CI/CD | GitHub Actions → ECR → rolling ECS deploy → k6 smoke + conflict gates |
| Local dev | Docker Compose + Floci (LocalStack-compatible) · Testcontainers · pytest e2e |
## Key engineering decisions
- **Optimistic locking via a DB unique constraint — not `SERIALIZABLE`, not row locks** — a version
race loses atomically on `UNIQUE (aggregate_id, version)` and surfaces as a retryable `409`
instead of opaque serialization retries or hot-account queuing; proven under a real race by the
k6 conflict gate ([ADR-001](docs/adrs/ADR-001-optimistic-locking-db-constraint.md))
- **SQS-first publish, transactional outbox demoted to fallback** — the classic outbox makes every
event pay the relay hop; publishing post-commit and writing outbox rows only when SQS fails keeps
projection lag at publish latency while the event store remains the recovery path
([ADR-002](docs/adrs/ADR-002-sqs-first-outbox-fallback.md))
- **Command-path Redis write-through over cache-aside** — the write path already holds the
authoritative balance at commit, so writing the snapshot through gives read-your-writes without
waiting on the async projector ([ADR-003](docs/adrs/ADR-003-command-path-redis-write-through.md))
- **Right-size before tuning** — an X-Ray trace showed 2 ms queries inside a 3.5 s p99: contention
(10-connection pool on 0.5 vCPU), not slow work; 4× CPU + a fixed pool of 20 tripled the knee,
with virtual threads deliberately held as a separate experiment
([ADR-004](docs/adrs/ADR-004-right-size-before-tuning.md))
- **Collector-less OpenTelemetry → X-Ray** — ADOT SigV4-signs OTLP itself, so no collector sidecar
runs anywhere; trace context is persisted on outbox rows so the projector joins the *original*
trace across the SQS hop ([ADR-005](docs/adrs/ADR-005-collectorless-otel-to-xray.md))
For the longer-form walkthrough — how these five decisions connect, including the trace that led
from ADR-005 to the ADR-004 performance fix — see
[`docs/architecture-narrative.md`](docs/architecture-narrative.md).
## Local development
**Prerequisites:** Java 21 · Docker · Terraform · Python 3.12 + `uv` · k6
```bash
git clone https://github.com/rafaesc/cloud-ledger && cd cloud-ledger
docker compose up -d # Floci: SQS, DynamoDB, Lambda, RDS, Cognito
bash terraform/scripts/local-bootstrap.sh # apply Terraform + migrations + build/push + boot API + Lambda images
# Pull M2M creds from the local Terraform state (-chdir works from the repo root)
export CLIENT_ID=$(terraform -chdir=terraform/envs/local output -raw cognito_client_id)
export CLIENT_SECRET=$(terraform -chdir=terraform/envs/local output -json cognito_client_secret | jq -r .)
export BASE_URL=http://localhost # Floci ALB on :80 — not the .elb.localhost output
export TOKEN_URL="http://localhost:4566/cognito-idp/oauth2/token"
cd api && ./gradlew test # unit + Testcontainers
cd lambdas && uv run pytest # Lambda tests (moto)
cd e2e && uv run pytest -v # full-stack e2e against the running API (http://localhost)
```
`local-bootstrap.sh` also builds and pushes the API image, and Floci runs it as an ECS task behind
an ALB — no separate `./gradlew bootRun` needed. The API is live at `http://localhost` (port 80,
not `:8080`); confirm with `curl http://localhost/actuator/health`, the same check
[`e2e.yml`](.github/workflows/e2e.yml) polls before running the e2e suite against it.
### Running the k6 suite
Requires the stack from the steps above already running (`docker compose up -d` +
`local-bootstrap.sh`) — `CLIENT_ID`/`CLIENT_SECRET` come from the Cognito pool Terraform just
created, so nothing here works standalone:
```bash
curl -s -o /dev/null -w '%{http_code}\n' http://localhost/actuator/health # expect 200 before continuing
BASE_URL=$BASE_URL PROJECTION_LAG_MS=60000 k6 run k6/scenarios/smoke.js # CI gate: 1 VU x 10 transfers E2E
BASE_URL=$BASE_URL PROJECTION_LAG_MS=60000 k6 run k6/scenarios/conflict.js # correctness demo: one 201 + N-1 409s, no double-spend
```
More details: [`k6/README.md`](k6/README.md).