{"id":37902072,"url":"https://github.com/dinorain/depositaja","last_synced_at":"2026-01-16T17:05:19.228Z","repository":{"id":37963788,"uuid":"502563986","full_name":"dinorain/depositaja","owner":"dinorain","description":"Depositaja - A fictitious service where user can deposit into their wallet and fetch their current balance using Event-driven architecture design using Apache Kafka.","archived":false,"fork":false,"pushed_at":"2022-08-03T17:04:53.000Z","size":180,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-06-21T01:41:37.874Z","etag":null,"topics":["apache-kafka","concurrency","event-driven","go","golang","hacktoberfest","kafka","message-broker","message-queue","protobuf","protocol-buffers","stream-processing"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dinorain.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-12T08:49:56.000Z","updated_at":"2024-01-21T08:33:34.000Z","dependencies_parsed_at":"2022-08-27T08:20:17.155Z","dependency_job_id":null,"html_url":"https://github.com/dinorain/depositaja","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dinorain/depositaja","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dinorain%2Fdepositaja","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dinorain%2Fdepositaja/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dinorain%2Fdepositaja/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dinorain%2Fdepositaja/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dinorain","download_url":"https://codeload.github.com/dinorain/depositaja/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dinorain%2Fdepositaja/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28480081,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","concurrency","event-driven","go","golang","hacktoberfest","kafka","message-broker","message-queue","protobuf","protocol-buffers","stream-processing"],"created_at":"2026-01-16T17:05:19.162Z","updated_at":"2026-01-16T17:05:19.215Z","avatar_url":"https://github.com/dinorain.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Depositaja: Wallet deposit service using Event-driven architecture\n\nIn this project, I build a fictitious wallet deposit service. Our main goal is to explore implementation of event-driven architecture using Kafka as the message broker.\nSeveral assumptions were made to pertain simplicity of the project, such as no authentication middleware, users cannot withdraw, and other out-of-topic features.\n\nThe service offers only two endpoints:\n\n1. `localhost:8080/deposit` HTTP POST endpoint for user to deposit money. This endpoint takes JSON data containing the *wallet_id*, and the *amount* of the deposit and emits it to Kafka.\n2. `localhost:8080/check/{wallet_id}` HTTP GET endpoint to get the balance of the wallet, also a flag whether the wallet has ever done one or more deposits with amounts more than 10,000 within a single 2-minute window (rolling-period). The *wallet_id* in the URLs refers to the wallet id.\n\nIn building this, we will follow the following architecture requirements:\n\n1. Use [Goka](https://github.com/lovoo/goka), a stream processing library for [Apache Kafka](https://kafka.apache.org/) written in Go.\n2. Use [protobuf](https://developers.google.com/protocol-buffers/docs/gotutorial) when encoding/decoding payload to/from Kafka broker.\n3. Use Goka's Local storage mechanism.\n\nTo run the project, you can follow [this](https://github.com/dinorain/depositaja#running-the-project).\n\n## Basic components and features\n\nGoka provides three components to build systems: emitters, processors, and views.\nThe following figure depicts the design used in this project, using those three components together with Kafka and the endpoints.\n\n![Architecture](figs/arch20220613-1.jpg)\n\n### Deposit endpoint\n\nThe main message type we will be dealing with is the [`depositRequest`](service/service.go#L19) type:\n\n```go\ntype depositRequest struct {\n\tWalletID string  `json:\"wallet_id\"`\n\tAmount   float64 `json:\"amount\"`\n}\n```\n\nIf Dustin wants to deposit an amount of money, he would send a request to the deposit endpoint with the wallet id and the amount of the money.\nFor example:\n\n```sh\ncurl -X POST                                                   \\\n    -d '{\"wallet_id\": \"0x1a3565a67721b6ab46fB11d5CF33A72D871aEbA3\", \"amount\": \"2000\"}' \\\n    http://localhost:8080/deposit\n```\n\nThe deposit handler parses request message type.\nAfterwards, it emits the message into the `DepositStream` topic using the receiver *wallet_id* as key:\n\n```go\nfunc deposit(emitter *goka.Emitter, stream goka.Stream) func(w http.ResponseWriter, r *http.Request) {\n\treturn func(w http.ResponseWriter, r *http.Request) {\n\t\t...\n\t}\n}\n```\n\nWe then configure the emitter to emit into the `DepositStream` topic by default and to use `DepositCodec` to encode the message.\nThis is how the emitter is [created](service/service.go#L55):\n```go\nemitter, err := goka.NewEmitter(brokers, stream, new(depositaja.DepositCodec))\nrouter.HandleFunc(\"/deposit\", deposit(emitter)).Methods(\"POST\")\n```\n\nNote I am ignoring errors in this document for the sake of readability.\nThe complete example in the repository handles them, though.\n\n### Collecting messages with `Context.Value()` and `Context.SetValue()`\n\nDefine the *balance table* to contain the deposit history stated in each wallet.\nThe *collector processor* keeps the table up-to-date by consuming `DepositStream`.\nDefine the collector callback as [follows](collector/collector.go#L17):\n\n```go\n// collect callback is called for every message from DepositStream.\n// ctx allows access to collector table and msg is the input message.\nfunc collect(ctx goka.Context, msg interface{}) {\n\tml := \u0026pb.DepositHistory{}\n\tif v := ctx.Value(); v != nil {\n\t\tml = v.(*pb.DepositHistory)\n\t}\n\n\tm := msg.(*pb.Deposit)\n\n\tml.WalletId = m.WalletId\n\tml.Deposits = append(ml.Deposits, m)\n\n\tctx.SetValue(ml)\n}\n```\n\nThe `ctx` is scoped with the key of the input message -- remember we used the receiver as key in the emitter.\nWith `ctx.Value()` we fetch the table value for that key.\nFinally, we store the value back in the table with `ctx.SetValue()`.\n\n\nTo create the processor, we need to define the group input stream and table's persistence:\n```go\ng := goka.DefineGroup(goka.Group(\"balance\"),\n\t// the group table (\"balance-table\") persists deposit lists\n\tgoka.Persist(new(depositaja.DepositListCodec)),\n\t// input stream is DepositStream with DepositListCodec and collect callback\n\tgoka.Input(depositaja.DepositStream, new(depositaja.DepositCodec), collect),\n)\np, _ := goka.NewProcessor(brokers, g)\n```\n\n### Check endpoint\n\nWhen Dustin wants to check his wallet balance, he requests that from the check endpoint.\nFor example:\n```\n curl localhost:8080/check/0x1a3565a67721b6ab46fB11d5CF33A72D871aEbA3\n{\n\t\"wallet_id\": \"0x1a3565a67721b6ab46fB11d5CF33A72D871aEbA3\",\n\t\"balance\": 2000,\n\t\"above_threshold\": false\n}\n```\n\nThe handler employs a view on `collector.Table` to retrieve the messages for Dustin.\nIt gets the *wallet_id* from the URL and tries to get the value from the view.\nIf no value is available, the user has received no messages yet.\nOtherwise, the handler loops over the messages, calculate the balance, check if the wallet deposit is above the threshold, and formats the output.\n\n```go\nfunc check(view *goka.View, flaggerView *goka.View) func(w http.ResponseWriter, r *http.Request) {\n\treturn func(w http.ResponseWriter, r *http.Request) {\n\t\t...\n\t}\n}\n```\n\nWhen creating the view, it is configured to watch the `collector.Table` and use `depositaja.DepositListCodec` to decode table values. Moreover, we also configure it to watch the `flagger.Table` for reading above-threshold flag purpose.\n\n```go\nview, _ := goka.NewView(\n\tbrokers,\n\tcollector.Table,\n\tnew(depositaja.DepositListCodec),\n)\nflaggerView, _ := goka.NewView(\n    brokers, \n    flagger.Table, \n    new(flagger.FlagValueCodec),\n)\nrouter.HandleFunc(\"/check/{wallet_id}\", check(view, flaggerView)).Methods(\"GET\")\n```\n\n`DepositListCodec` simply encodes and decodes the message of `DepositList` into and from \nprotocol buffer data.\n\n### Flagging wallets\n\nWe would need to flag wallets that have ever done one or more deposits with amounts more than 10,000 within a single 2-minute window (rolling-period).\n\nFor that, create a flagger processor, which keeps a table of wallets that have been flagged.\nThe flagger processor consumes from `flagger.Stream` and stores a `FlagValue` in the `flagger.Table`:\n\n```go\nfunc flag(ctx goka.Context, msg interface{}) {\n\t...\n}\n```\n\nTo add or remove a wallet manually from the flagger table, we can use the command line tool cmd/flag-wallet:\n\n```sh\ngo run cmd/flag-wallet/main.go -wallet 0x1a3565a67721b6ab46fB11d5CF33A72D871aEbA3 # use -remove to remove the flag\n```\n\n### Automatic above-threshold detection\n\nIn this project, we give a flag if the wallet has ever done one or more deposits with amounts more than 10,000 within a single 2-minute window (rolling-period).\nSo, if we can detect wallets that fulfill that property, we can flag them.\n\nWe want to build a detector processor that counts total amount of deposits within a single 2-minute window (rolling-period) and issues a `FlagEvent` if the amount exceeds a threshold.\nThe detector table should keep the following value for each user.  \n```go\ntype Counter struct {\n\tstate         protoimpl.MessageState\n\tsizeCache     protoimpl.SizeCache\n\tunknownFields protoimpl.UnknownFields\n\n\tRollingPeriodStartUnix int64   `protobuf:\"varint,1,opt,name=rolling_period_start_unix,json=rollingPeriodStartUnix,proto3\" json:\"rolling_period_start_unix,omitempty\"`\n\tReceived               float64 `protobuf:\"fixed64,2,opt,name=received,proto3\" json:\"received,omitempty\"`\n}\n```\n\nThe Counter struct is one of the protocol buffer message's structs in [pb](pb) generated by invoking the command below. Find more [here](https://developers.google.com/protocol-buffers/docs/gotutorial).\n```go\nprotoc -I=proto --go_out=proto proto/detector.proto\n```\n\nWhenever update happens in the table value, it should check whether the wallet gets a flag.\nIf the amount of deposit sent is more than `maxAmount` within a single 2-minute window (rolling-period), we add a flag the wallet and issue a `FlagEvent`.\n\n```go\nfunc detectSpammer(ctx goka.Context, c *pb.Counter) bool {\n\treturn c.Received \u003e= maxAmount \u0026\u0026 c.RollingPeriodStartUnix != 0\n}\n```\n\n### Counting deposit amounts\n\nNow, we defined an approach to detect if a wallet is above threshold, but we have to keep the values in the group table updated.\nWe define the group graph in parts.\nHere is the callback for `DepositStream`:\n\n```go\ninput := goka.Input(depositaja.DepositStream, new(depositaja.DepositCodec), func(ctx goka.Context, msg interface{}) {\n\t...\n}\nfunc getValue(ctx goka.Context) *pb.Counter {\n\tif v := ctx.Value(); v != nil {\n\t\treturn v.(*pb.Counter)\n\t}\n\treturn \u0026pb.Counter{}\n}\n```\n\nFor every message received from `DepositStream`, we first get the value for the key or create a new `Counter` protocol buffer object.\n`DepositStream` has the sender as key, so we add amount `c.Received` and store back in the group table with `ctx.SetValue()`.\nNext, we call `detectSpammer(ctx, c)`, which will check whether sent rate is higher than a threshold.\n\nNext, we check whether the wallet goes above the threshold with the following function.\n\n### Group graph\nFinally, we define the complete group as follows:\n```go\ng := goka.DefineGroup(goka.Group(\"threshold\"),\n\tinput,\n\tgoka.Output(flagger.Stream, new(flagger.FlagEventCodec)),\n\tgoka.Persist(new(CounterCodec)),\n)\np, _ := goka.NewProcessor(brokers, g)\n```\n\n### Recap\n\nAt this point, let's make a short recap. So far we have created:\n\n- a [service](service/service.go) with deposit and check endpoints;\n- a [collector processor](collector/collector.go) to collect deposit sent to wallet;\n- a [flagger processor](flagger/flagger.go) to keep a table tracking flagged wallets;\n- a [detector processor](detector/detector.go) to automatically flag wallets with deposits above threshold;\n- a [flag-wallet tool](cmd/flag-wallet) to add/remove flags to/from wallets.\n\n### Running the project\n\nIn this project, we can put the endpoint handlers and, consequently, emitter and view in the same Go program.\nIn another Go program, we start the collector processor. This will allows us to start, stop, and scale them independently.\n\nBefore starting any Go program, run `make start` to start Docker containers for ZooKeeper and Kafka.\n\nMake sure you [create](https://docs.confluent.io/5.5.4/quickstart/cos-docker-quickstart.html#step-2-create-ak-topics) the required Kafka topics.\n\n```sh\nmake dev # to start endpoint handlers, emitter and view\n```\n\nIn another terminal, start the processor:\n\n```sh\nmake processor # start collector processor, detector processor, and flagger processor\n```\n\nor \n\n```sh\ngo run cmd/processor/main.go -collector -flagger -detector\n```\n\nInternally the Go Program will start three Goka processors.\nAlternatively, you can run the processors individually by starting the program multiple times with the respective flags.\n\nAfter you started both Go programs, you can use `curl` to check Dustin's wallet:\n\n```sh\ncurl localhost:8080/check/0x1a3565a67721b6ab46fB11d5CF33A72D871aEbA3\n```\n\nor open [http://localhost:8080/check/0x1a3565a67721b6ab46fB11d5CF33A72D871aEbA3](http://localhost:8080/check/0x1a3565a67721b6ab46fB11d5CF33A72D871aEbA3) in the browser.\n\nYou can deposit money to Dustin's wallet using `curl`, for example,\n\n```sh\ncurl -X POST                                                   \\\n    -d '{\"wallet_id\": \"0x1a3565a67721b6ab46fB11d5CF33A72D871aEbA3\", \"amount\": \"999999999\"}' \\\n    http://localhost:8080/deposit\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdinorain%2Fdepositaja","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdinorain%2Fdepositaja","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdinorain%2Fdepositaja/lists"}