{"id":13713854,"url":"https://github.com/marselester/distributed-payment","last_synced_at":"2026-03-06T12:33:04.550Z","repository":{"id":65542851,"uuid":"137072053","full_name":"marselester/distributed-payment","owner":"marselester","description":"Demo execution of a payment transaction without an atomic commit across 3 partitions.","archived":false,"fork":false,"pushed_at":"2018-09-28T13:02:25.000Z","size":19,"stargazers_count":35,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-11T01:36:30.090Z","etag":null,"topics":["distributed-systems","go","kafka","rocksdb"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marselester.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-12T12:59:12.000Z","updated_at":"2025-04-03T02:07:37.000Z","dependencies_parsed_at":"2023-01-28T12:25:10.158Z","dependency_job_id":null,"html_url":"https://github.com/marselester/distributed-payment","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/marselester/distributed-payment","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marselester%2Fdistributed-payment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marselester%2Fdistributed-payment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marselester%2Fdistributed-payment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marselester%2Fdistributed-payment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marselester","download_url":"https://codeload.github.com/marselester/distributed-payment/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marselester%2Fdistributed-payment/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30176264,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T11:48:51.886Z","status":"ssl_error","status_checked_at":"2026-03-06T11:48:51.460Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-systems","go","kafka","rocksdb"],"created_at":"2024-08-02T23:01:46.065Z","updated_at":"2026-03-06T12:33:04.383Z","avatar_url":"https://github.com/marselester.png","language":"Go","funding_links":[],"categories":["Repositories"],"sub_categories":[],"readme":"# Distributed Payment\n\n[![Documentation](https://godoc.org/github.com/marselester/distributed-payment?status.svg)](https://godoc.org/github.com/marselester/distributed-payment)\n[![Go Report Card](https://goreportcard.com/badge/github.com/marselester/distributed-payment)](https://goreportcard.com/report/github.com/marselester/distributed-payment)\nAlso have a look at [distributed signup](https://github.com/marselester/distributed-signup).\n\nThis project demonstrates execution of a payment transaction without an atomic commit across 3 partitions\n(a primer from \"Designing Data-Intensive Applications\" book):\n\n1. Alice wants to send $0.5 to Bob: the intent is stored in 💬 partition.\n2. Alice's -$0.5 outgoing payment is created in 👩 partition.\n3. Bob's +$0.5 incoming payment is persisted in 👨🏻 partition.\n\nThe idea is to write a money transfer request into `wallet.transfer_request` Kafka topic\nwhich is partitioned by request ID (some unique ID generated by Alice).\nHence all requests with the same ID will be stored in the same Kafka partition 💬 based on\n[consistent hashing algorithm](http://medium.com/@dgryski/consistent-hashing-algorithmic-tradeoffs-ef6b8e2fcae8).\nFor example, `{from: Alice, amount: 0.5, to: Bob, request_id: a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11}` message is written\nto `hash('a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11') % partitions_count` partition 💬.\nLet's have two partitions `partitions_count=2` for each Kafka topic for simplicity.\n\nA **transfer-server** instance appends transfer requests into `wallet.transfer_request` topic.\n\nEach **paymentd** instance (two in our case) sequentially reads Kafka messages from its own partition of `wallet.transfer_request`\nand creates two payment instructions in `wallet.payment` topic:\n\n- `{account: Alice, direction: outgoing, amount: 0.5, request_id: a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11}` message\n  goes into 👩 partition based on `hash('Alice') % 2`.\n- `{account: Bob, direction: incoming, amount: 0.5, request_id: a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11}` message\n  goes into 👨🏻 partition based on `hash('Bob') % 2`.\n\nThere might be duplicate credit/debit instructions when a process crashes and restarts.\n\nEach **accountantd** instance sequentially reads Kafka messages from its own partition of `wallet.payment` topic,\ndeduplicates messages by request ID, and applies the changes to the balances. For example, the accountant №1\nhas read the following messages:\n\n- `{account: Alice, direction: outgoing, amount: 0.5, request_id: a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11}`\n- `{account: John, direction: incoming, amount: 99, request_id: 6ba7b810-9dad-11d1-80b4-00c04fd430c8}`\n- `{account: Alice, direction: outgoing, amount: 0.5, request_id: a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11}`\n\nAlice's account must be deducted only once. The accountant №2 skipped a duplicate and credited Bob $0.5:\n\n- `{account: Bob, direction: incoming, amount: 0.5, request_id: a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11}`\n- `{account: Bob, direction: incoming, amount: 0.5, request_id: a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11}`\n\nNote, `request_id` is generated by a client who sends money (Alice).\nRequest IDs are kept for a certain duration (until a message ages out) or limited by storage size.\nSegment shared how they leverage RocksDB in\n[Delivering Billions of Messages Exactly Once](https://segment.com/blog/exactly-once-delivery/):\n\n\u003e If the dedupe worker crashes for any reason or encounters an error from Kafka,\n\u003e when it re-starts it will first consult the \"source of truth\" for whether an event was published: the output topic.\n\u003e If a message was found in the output topic, but not RocksDB (or vice-versa)\n\u003e the dedupe worker will make the necessary repairs to keep the database and RocksDB in-sync.\n\u003e In essence, we're using the output topic as both our write-ahead-log, and our end source of truth,\n\u003e with RocksDB checkpointing and verifying it.\n\nThat inspired me to try RocksDB as a deduplication storage as well.\n\n## Get Started\n\nWe need Kafka which will have `wallet.transfer_request` and `wallet.payment` topics with 2 partitions and 1 replica.\nDocker Compose will take care of that. The only caveat is that you should set `KAFKA_ADVERTISED_HOST_NAME`.\n\n```sh\n$ cd ./docker/\n$ KAFKA_ADVERTISED_HOST_NAME=$(ipconfig getifaddr en0) docker-compose up\n```\n\nInstall dependencies using dep package manager and build all commands.\nNote, you need to install RocksDB first (assuming you're on Mac).\n\n```sh\n$ brew install rocksdb\n$ dep ensure\n$ make build\n```\n\nRun a **transfer-server** to validate transfer requests and persist them in `wallet.transfer_request` topic\npartitioned by request ID.\n\n```sh\n$ ./transfer-server\n```\n\nSend a money transfer request:\n\n```sh\n$ curl -i -X POST -d '{\"from\": \"Alice\", \"to\": \"Bob\", \"amount\": \"0.5\", \"request_id\": \"a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11\"}' \\\n    http://localhost:8000/api/v1/transfers\nHTTP/1.1 201 Created\nContent-Type: application/json\nContent-Length: 96\n\n{\"request_id\":\"a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11\",\"from\":\"Alice\",\"amount\":\"0.50\",\"to\":\"Bob\"}\n```\n\n## Stream Processors\n\nSince we have a transfer request in Kafka, we can run two **paymentd** processes for each partition\nto create corresponding payments.\n\n```sh\n$ ./paymentd -partition=0\n$ ./paymentd -partition=1\n1:0 a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11 Alice -$0.50\n0:0 a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11 Bob +$0.50\n```\n\nPayments are printed in `partition_id:offset request_id account amount` format.\nAs you can see:\n\n- a transfer has been stored in partition 1 (no output from `./paymentd -partition=0`),\n- Alice's outgoing payment was stored in partition 1,\n- Bob's incoming payment landed at partition 0.\n\nPayment instructions end up in `wallet.payment` topic's partitions. Let's process them, so Alice's and Bob's balances are updated:\n\n```sh\n$ ./accountantd -partition=0\nBob balance: 0.50 USD\n$ ./accountantd -partition=1\nAlice balance: -0.50 USD\n```\n\nTry sending a duplicate request and see if balances stay the same.\n\n## Future Work\n\n- Validate sender's balance before creating a transfer.\n- It will be interesting to check invariants by [DInv](https://bitbucket.org/bestchai/dinv/), [TLA+](https://en.wikipedia.org/wiki/TLA%2B).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarselester%2Fdistributed-payment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarselester%2Fdistributed-payment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarselester%2Fdistributed-payment/lists"}