{"id":51684056,"url":"https://github.com/pach24/recon-engine","last_synced_at":"2026-07-15T17:35:37.024Z","repository":{"id":368669897,"uuid":"1286284517","full_name":"pach24/recon-engine","owner":"pach24","description":"Automates reconciliation of transactions across multiple data sources and flags discrepancies in real time. Built with Java 17, Spring Boot, Kafka \u0026 PostgreSQL/Oracle.","archived":false,"fork":false,"pushed_at":"2026-07-01T16:31:46.000Z","size":13,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-01T18:11:39.324Z","etag":null,"topics":["aws","docker","event-driven","java","kafka","microservice","postgresql","spring-boot"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pach24.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-01T16:12:01.000Z","updated_at":"2026-07-01T16:32:26.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/pach24/recon-engine","commit_stats":null,"previous_names":["pach24/recon-engine"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/pach24/recon-engine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pach24%2Frecon-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pach24%2Frecon-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pach24%2Frecon-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pach24%2Frecon-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pach24","download_url":"https://codeload.github.com/pach24/recon-engine/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pach24%2Frecon-engine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35515478,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-15T02:00:06.706Z","response_time":131,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","docker","event-driven","java","kafka","microservice","postgresql","spring-boot"],"created_at":"2026-07-15T17:35:35.687Z","updated_at":"2026-07-15T17:35:37.019Z","avatar_url":"https://github.com/pach24.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reconciliation Platform — Automated Multi-Source Transaction Matching\n\n\u003e Automatically detect and surface discrepancies between financial data sources — eliminating thousands of manual reconciliation hours.\n\n---\n\n## The Problem\n\nLarge enterprises (banking, insurance, retail) receive the same business data from multiple systems that rarely agree: a payment appears in the core banking system but is missing from the accounting ledger; an invoice is recorded differently in the ERP and the warehouse system. Reconciling these discrepancies manually is expensive, error-prone, and slow — often taking entire teams days to find mismatches that a rules engine can catch in seconds.\n\nThis platform ingests data from heterogeneous sources, normalises it, applies configurable matching rules, and emits real-time alerts and reports on every discrepancy found.\n\n---\n\n## Architecture\n\n```mermaid\nflowchart LR\n    subgraph Sources\n        CSV[CSV Files]\n        API[External APIs]\n        ORA[(Oracle 23c\\nlegacy batch export)]\n    end\n\n    subgraph ingestion-service\n        ING[Normalise \u0026 publish]\n        POLL[Legacy poller]\n    end\n\n    KAFKA[(Kafka\\nKRaft)]\n\n    subgraph reconciliation-engine\n        REC[Match records\\nApply rules]\n    end\n\n    PG[(PostgreSQL 16\\nreconciliation store)]\n\n    subgraph Consumers\n        ALERT[alert-service\\nNotifications]\n        REPORT[reporting-service\\nStatus reports]\n    end\n\n    CSV --\u003e ING\n    API --\u003e ING\n    ORA --\u003e POLL\n    POLL --\u003e KAFKA\n    ING --\u003e KAFKA\n    KAFKA --\u003e REC\n    REC --\u003e PG\n    REC --\u003e KAFKA\n    KAFKA --\u003e ALERT\n    KAFKA --\u003e REPORT\n```\n\n---\n\n## Tech Stack\n\n| Layer | Technology | Why |\n|---|---|---|\n| Language | Java 17 (LTS) | Enterprise standard; long-term support |\n| Framework | Spring Boot 3.x | Production-grade microservices with minimal boilerplate |\n| Build | Maven multi-module | Centralised version management; familiar in banking/legacy orgs |\n| Messaging | Apache Kafka (KRaft) | High-throughput event streaming; KRaft removes Zookeeper complexity |\n| DB — new system | PostgreSQL 16 | Reliable, open-source RDBMS with strong JSON support |\n| DB — legacy system | Oracle Free 23c | A `LegacyOraclePoller` in ingestion-service reads unprocessed rows from `legacy_transactions` and republishes them as `sourceSystem=LEGACY_ORACLE` transactions, feeding the same reconciliation pipeline |\n| Containers | Docker (multi-stage builds) | Minimal production images; reproducible environments |\n| Orchestration | Docker Compose | Single-command local setup |\n| Integration tests | Testcontainers | Tests run against real database engines, not mocks |\n| Cloud target | AWS free tier (EC2 + RDS + ECR) | Zero-cost demo deployment; documented migration path to OpenShift |\n\n---\n\n## Prerequisites\n\n- **Docker** \u003e= 24 and **Docker Compose** \u003e= 2.24\n- **JDK 17** (for local builds outside Docker)\n- **Maven** \u003e= 3.9 (or use the Maven Wrapper once added)\n\n---\n\n## Getting Started\n\n```bash\n# 1. Clone the repo\ngit clone https://github.com/pach24\ncd recon-engine\n\n# 2. Copy and review environment variables\ncp .env.example .env\n\n# 3. Build images and start all services\ndocker-compose up --build\n\n# 4. Verify all health checks (in a second terminal)\ncurl http://localhost:8081/health   # ingestion-service\ncurl http://localhost:8082/health   # reconciliation-engine\ncurl http://localhost:8083/health   # alert-service\ncurl http://localhost:8084/health   # reporting-service\n```\n\nAll four endpoints should return:\n```json\n{\"service\":\"\u003cservice-name\u003e\",\"status\":\"UP\"}\n```\n\nSpring Actuator is also available at `/actuator/health` on each port.\n\n---\n\n## Repository Structure\n\n```\nrecon-engine/\n├── .env.example                     # Environment variable template\n├── docker-compose.yml               # Full local stack\n├── pom.xml                          # Parent POM (packaging: pom)\n├── docs/\n│   └── architecture.md              # Detailed design decisions + diagrams\n├── ingestion-service/               # Port 8081 -- ingest \u0026 normalise sources\n│   ├── pom.xml\n│   ├── Dockerfile\n│   └── src/main/\n│       ├── java/com/recon/ingestion/\n│       └── resources/application.yml\n├── reconciliation-engine/           # Port 8082 -- match records, detect gaps\n├── alert-service/                   # Port 8083 -- emit notifications\n└── reporting-service/               # Port 8084 -- status reports \u0026 dashboards\n```\n\n---\n\n## Endpoints\n\n| Service | Port | Endpoint | Description |\n|---|---|---|---|\n| ingestion-service | 8081 | `POST /transactions` | Ingest a single transaction (JSON), publishes to `raw.transactions` |\n| ingestion-service | 8081 | `POST /transactions/csv` | Bulk-ingest a CSV export (multipart `file`); valid rows publish to `raw.transactions`, malformed rows are reported per-row |\n| ingestion-service | 8081 | `GET /health` | Custom health response |\n| ingestion-service | 8081 | `GET /actuator/health` | Spring Actuator health |\n| reconciliation-engine | 8082 | `GET /health` | Custom health response |\n| reconciliation-engine | 8082 | `GET /actuator/health` | Spring Actuator health |\n| alert-service | 8083 | `GET /health` | Custom health response |\n| alert-service | 8083 | `GET /actuator/health` | Spring Actuator health |\n| reporting-service | 8084 | `GET /health` | Custom health response |\n| reporting-service | 8084 | `GET /actuator/health` | Spring Actuator health |\n\n### Bulk CSV ingestion\n\n`POST /transactions/csv` accepts a multipart upload. The first line is a header; columns\nare matched by name (order-independent). Required: `transactionId`, `accountId`, `amount`,\n`currency`, `occurredAt` (ISO-8601 instant). `sourceSystem` is optional and defaults to\n`CSV_IMPORT` (`app.csv.default-source-system`).\n\n```bash\ncurl -F \"file=@transactions.csv\" http://localhost:8081/transactions/csv\n```\n\n```csv\ntransactionId,accountId,amount,currency,occurredAt\nTXN-1,ACC-1,100.50,USD,2026-07-06T10:00:00Z\n```\n\nValid rows are published to `raw.transactions`; malformed rows are skipped and returned\nin the response so one bad line never fails the whole upload:\n\n```json\n{\"received\":2,\"published\":1,\"errors\":[{\"line\":3,\"message\":\"amount must be positive: '-5.00'\"}]}\n```\n\n---\n\n## Running Tests\n\nIntegration tests use **Testcontainers** to spin up real PostgreSQL and Oracle instances per test suite — no mocks, no shared state.\n\n```bash\n# Unit + integration tests for all modules\nmvn verify\n\n# Single module\nmvn verify -pl ingestion-service\n```\n\n\u003e Testcontainers requires Docker to be running. CI pipelines run these tests against a Docker-in-Docker (DinD) executor.\n\n---\n\n## Cloud Deployment\n\n### AWS Free Tier (demo)\n\n| Component | AWS Service | Notes |\n|---|---|---|\n| Microservices | EC2 t2.micro | Package each service as a Docker image |\n| Container registry | ECR | Push images from CI, pull on EC2 |\n| PostgreSQL | RDS db.t3.micro | Free tier: 20 GB storage |\n| Kafka | MSK Serverless or self-hosted on EC2 | MSK Serverless has no free tier; self-hosted EC2 is free-tier-compatible |\n| Oracle | EC2 + gvenzl/oracle-free | RDS Oracle has no free tier |\n\nDeployment pipeline: GitHub Actions -\u003e build \u0026 push to ECR -\u003e SSH deploy on EC2 via `docker-compose pull \u0026\u0026 docker-compose up -d`.\n\n### OpenShift (production)\n\nEach service ships as a container image and maps cleanly to an OpenShift `Deployment`. Kafka can be managed by the **Strimzi** operator; databases connect via `Service` objects pointing to managed RDS/Oracle instances. Health endpoints integrate directly with OpenShift liveness and readiness probes.\n\n---\n\n## Design Decisions\n\nSee [`docs/architecture.md`](docs/architecture.md) for the full rationale. Quick summary:\n\n- **Kafka over RabbitMQ** — event log retention lets the reconciliation engine replay history when rules change; RabbitMQ deletes messages on ACK.\n- **Two databases** — PostgreSQL is the reconciliation store; Oracle Free 23c is a real second source, not just provisioned infrastructure. ingestion-service's `LegacyOraclePoller` reads legacy transactions out of Oracle and feeds them into the same matching pipeline as the JSON API, mirroring how banking legacy systems export via batch tables rather than APIs.\n- **Microservices over monolith** — each service can be scaled, deployed, and failed independently; reconciliation workloads spike at month-end while alerting is constant.\n- **Maven multi-module** — single `mvn verify` builds and tests the entire platform; version conflicts are caught at compile time, not at runtime.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpach24%2Frecon-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpach24%2Frecon-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpach24%2Frecon-engine/lists"}