{"id":51531644,"url":"https://github.com/keploy/orderflow","last_synced_at":"2026-07-09T03:01:14.167Z","repository":{"id":353072658,"uuid":"1217017014","full_name":"keploy/orderflow","owner":"keploy","description":null,"archived":false,"fork":false,"pushed_at":"2026-04-30T20:27:12.000Z","size":94,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-20T22:00:05.549Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/keploy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["keploy"]}},"created_at":"2026-04-21T13:12:43.000Z","updated_at":"2026-04-30T19:07:09.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/keploy/orderflow","commit_stats":null,"previous_names":["keploy/orderflow"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/keploy/orderflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keploy%2Forderflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keploy%2Forderflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keploy%2Forderflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keploy%2Forderflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/keploy","download_url":"https://codeload.github.com/keploy/orderflow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keploy%2Forderflow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35284881,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-09T02:00:07.329Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-09T03:01:13.266Z","updated_at":"2026-07-09T03:01:14.156Z","avatar_url":"https://github.com/keploy.png","language":"HTML","funding_links":["https://github.com/sponsors/keploy"],"categories":[],"sub_categories":[],"readme":"# OrderFlow — Real-time Order Processing Pipeline\n\nA production-grade Go application demonstrating a real-time e-commerce order processing system with:\n\n- **Producer API** → writes to sharded Postgres + S3 + publishes to Kafka\n- **Consumer Service** → reads from Kafka only (no S3/Postgres access by design)\n- **Frontend Dashboard** → real-time order placement and monitoring\n\n---\n\n## Architecture\n\n```\nFrontend (index.html)\n       │\n       │  POST /api/orders\n       ▼\n┌─────────────────┐\n│  Producer API   │  (Go HTTP Server, :8080)\n│  (main.go)      │\n└────┬────┬───────┘\n     │    │    │\n     │    │    │ 1. INSERT\n     │    │    ▼\n     │    │  ┌─────────────────┐  ┌─────────────────┐\n     │    │  │ Postgres Shard1 │  │ Postgres Shard2 │\n     │    │  │ Users A-M :5432 │  │ Users N-Z :5433 │\n     │    │  └─────────────────┘  └─────────────────┘\n     │    │\n     │    │ 2. PUT (receipt .txt)\n     │    ▼\n     │  ┌──────────────────────┐\n     │  │  S3 (LocalStack)     │\n     │  │  bucket:order-receipts│\n     │  │  port :4566          │\n     │  └──────────────────────┘\n     │\n     │ 3. PUBLISH (order.created event)\n     ▼\n┌─────────────────┐     CONSUME     ┌─────────────────┐\n│  Kafka          │ ──────────────► │  Consumer       │\n│  topic: orders  │                 │  (Go Service)   │\n│  port: :9092    │                 │  Kafka only     │\n└─────────────────┘                 │  ❌ No S3/Postgres│\n                                    └─────────────────┘\n```\n\n### Sharding Strategy\n\nOrders are sharded by the first letter of `user_id`:\n- **Shard 1** (port 5432): Users whose ID starts with **A–M**\n- **Shard 2** (port 5433): Users whose ID starts with **N–Z**\n\n---\n\n## Folder Structure\n\n```\norderflow/\n├── docker-compose.yml          # All services orchestration\n├── frontend/\n│   └── index.html              # Dashboard UI\n├── producer/                   # Producer API (Go)\n│   ├── main.go                 # Entry point, HTTP server\n│   ├── Dockerfile\n│   ├── go.mod\n│   ├── config/\n│   │   └── config.go           # ENV-based config\n│   ├── models/\n│   │   └── order.go            # Order structs + Kafka event\n│   ├── handlers/\n│   │   └── order.go            # HTTP handlers (create, list)\n│   ├── kafka/\n│   │   └── producer.go         # Kafka producer client\n│   └── storage/\n│       ├── postgres.go         # Sharded Postgres (shard1 + shard2)\n│       └── s3.go               # S3 receipt upload\n├── consumer/                   # Consumer Service (Go)\n│   ├── main.go                 # Kafka consumer loop\n│   ├── Dockerfile\n│   ├── go.mod\n│   ├── config/\n│   │   └── config.go\n│   └── handlers/\n│       └── processor.go        # Event processor (Kafka only, no S3/PG)\n└── scripts/\n    ├── init_shard1.sql         # Postgres schema for shard 1\n    ├── init_shard2.sql         # Postgres schema for shard 2\n    └── init_localstack.sh      # Creates S3 bucket on startup\n```\n\n---\n\n## Prerequisites\n\n- **Docker** and **Docker Compose** installed\n- **Go 1.21+** (only for local dev without Docker)\n- **librdkafka** (only for local dev — `brew install librdkafka` on Mac)\n\n---\n\n## Running with Docker Compose (Recommended)\n\n### Step 1 — Clone and navigate\n```bash\ncd orderflow\n```\n\n### Step 2 — Make the LocalStack init script executable\n```bash\nchmod +x scripts/init_localstack.sh\n```\n\n### Step 3 — Start all services\n```bash\ndocker-compose up --build\n```\n\nWait ~30 seconds for all services to initialize. You'll see:\n```\nproducer  | ✓ Connected to sharded Postgres (shard1: A-M, shard2: N-Z)\nproducer  | ✓ S3 client initialized (bucket: order-receipts)\nproducer  | ✓ Kafka producer connected to kafka:9092\nproducer  | ✓ Producer API listening on :8080\nconsumer  | ✓ Subscribed to topic 'orders' with group 'order-consumer-group'\nconsumer  | ✓ Waiting for messages...\n```\n\n### Step 4 — Open the Frontend\nOpen `frontend/index.html` in your browser (just double-click the file), or:\n```bash\nopen frontend/index.html   # Mac\nxdg-open frontend/index.html  # Linux\n```\n\n---\n\n## Running Locally (Without Docker)\n\n### Start dependencies only\n```bash\ndocker-compose up zookeeper kafka postgres_shard1 postgres_shard2 localstack\n```\n\n### Run Producer\n```bash\ncd producer\ngo mod tidy\nexport PG_SHARD1_DSN=\"postgres://orderuser:orderpass@localhost:5432/orders_shard1?sslmode=disable\"\nexport PG_SHARD2_DSN=\"postgres://orderuser:orderpass@localhost:5433/orders_shard2?sslmode=disable\"\nexport S3_ENDPOINT=\"http://localhost:4566\"\nexport KAFKA_BROKERS=\"localhost:9092\"\nCGO_ENABLED=1 go run main.go\n```\n\n### Run Consumer (separate terminal)\n```bash\ncd consumer\ngo mod tidy\nexport KAFKA_BROKERS=\"localhost:9092\"\nCGO_ENABLED=1 go run main.go\n```\n\n---\n\n## API Reference\n\n### Health Check\n```bash\ncurl http://localhost:8080/health\n```\n\n### Create Order\n```bash\ncurl -X POST http://localhost:8082/api/orders \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"user_id\": \"alice\",\n    \"product_name\": \"Mechanical Keyboard\",\n    \"quantity\": 2,\n    \"price\": 149.99\n  }'\n```\n\n**Response:**\n```json\n{\n  \"order\": {\n    \"id\": \"uuid-here\",\n    \"user_id\": \"alice\",\n    \"product_name\": \"Mechanical Keyboard\",\n    \"quantity\": 2,\n    \"price\": 149.99,\n    \"status\": \"created\"\n  },\n  \"shard\": \"shard1\",\n  \"shard_info\": \"shard1 (A-M)\",\n  \"receipt\": \"receipts/alice/uuid-here.txt\"\n}\n```\n\n### List All Orders\n```bash\ncurl http://localhost:8080/api/orders\n```\n\n### List Orders by User\n```bash\ncurl \"http://localhost:8080/api/orders?user_id=alice\"\n```\n\n---\n\n## Test Sharding\n\n```bash\n# This goes to Shard 1 (A-M)\ncurl -X POST http://localhost:8080/api/orders \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"user_id\":\"alice\",\"product_name\":\"Monitor\",\"quantity\":1,\"price\":399.99}'\n\n# This goes to Shard 2 (N-Z)\ncurl -X POST http://localhost:8080/api/orders \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"user_id\":\"zara\",\"product_name\":\"Keyboard\",\"quantity\":2,\"price\":99.99}'\n\n# Verify directly in Postgres\ndocker exec -it orderflow-postgres_shard1-1 psql -U orderuser -d orders_shard1 -c \"SELECT id, user_id, product_name FROM orders;\"\ndocker exec -it orderflow-postgres_shard2-1 psql -U orderuser -d orders_shard2 -c \"SELECT id, user_id, product_name FROM orders;\"\n```\n\n## Verify S3 Receipts\n```bash\naws --endpoint-url=http://localhost:4566 s3 ls s3://order-receipts/receipts/ --recursive\n```\n\n## Watch Kafka Events\n```bash\ndocker exec -it orderflow-kafka-1 kafka-console-consumer \\\n  --bootstrap-server localhost:9092 \\\n  --topic orders \\\n  --from-beginning\n```\n\n---\n\n## Environment Variables\n\n### Producer\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `KAFKA_BROKERS` | `localhost:9092` | Kafka broker addresses |\n| `PG_SHARD1_DSN` | local shard1 | Postgres DSN for shard 1 |\n| `PG_SHARD2_DSN` | local shard2 | Postgres DSN for shard 2 |\n| `S3_ENDPOINT` | `http://localhost:4566` | S3/LocalStack endpoint |\n| `S3_BUCKET` | `order-receipts` | S3 bucket name |\n| `PORT` | `8080` | HTTP server port |\n\n### Consumer\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `KAFKA_BROKERS` | `localhost:9092` | Kafka broker addresses |\n| `KAFKA_GROUP_ID` | `order-consumer-group` | Consumer group ID |\n| `KAFKA_TOPIC` | `orders` | Topic to consume |\n\n---\n\n## Stop Everything\n```bash\ndocker-compose down -v\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeploy%2Forderflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkeploy%2Forderflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeploy%2Forderflow/lists"}