{"id":19506811,"url":"https://github.com/calvinlfer/akka-streams-custom-stream-processing-examples","last_synced_at":"2025-06-23T20:05:14.980Z","repository":{"id":68736711,"uuid":"80977495","full_name":"calvinlfer/Akka-Streams-custom-stream-processing-examples","owner":"calvinlfer","description":"Demos of how to do custom stream processing using the Akka Streams GraphStages API","archived":false,"fork":false,"pushed_at":"2017-02-05T21:23:20.000Z","size":22,"stargazers_count":13,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-23T06:17:41.314Z","etag":null,"topics":["akka","akka-streams","scala","stream-processing"],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/calvinlfer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-02-05T07:42:46.000Z","updated_at":"2019-12-06T20:05:14.000Z","dependencies_parsed_at":"2023-03-14T21:45:15.943Z","dependency_job_id":null,"html_url":"https://github.com/calvinlfer/Akka-Streams-custom-stream-processing-examples","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/calvinlfer/Akka-Streams-custom-stream-processing-examples","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2FAkka-Streams-custom-stream-processing-examples","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2FAkka-Streams-custom-stream-processing-examples/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2FAkka-Streams-custom-stream-processing-examples/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2FAkka-Streams-custom-stream-processing-examples/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/calvinlfer","download_url":"https://codeload.github.com/calvinlfer/Akka-Streams-custom-stream-processing-examples/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/calvinlfer%2FAkka-Streams-custom-stream-processing-examples/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261548718,"owners_count":23175492,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["akka","akka-streams","scala","stream-processing"],"created_at":"2024-11-10T22:38:29.915Z","updated_at":"2025-06-23T20:05:14.972Z","avatar_url":"https://github.com/calvinlfer.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Custom Stream Processing in Akka Streams with GraphStages\nDemos of how to do custom stream processing using the Akka Streams GraphStages API\n\n## Dealing with asynchronous channels\nYou would use this kind of custom stream processing when you have to deal with integrating with external parties like \nAmazon's SQS, polling an HTTP endpoint (take a look at timers if you are getting throttled), etc.\n\nFrom the [documentation](http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html?_ga=1.95210907.668506683.1483746547#Using_asynchronous_side-channels)\n\n*In order to receive asynchronous events that are not arriving as stream elements (for example a completion of a future \nor a callback from a 3rd party API) one must acquire a `AsyncCallback` by calling `getAsyncCallback()` from the stage \nlogic. The method `getAsyncCallback` takes as a parameter a callback that will be called once the asynchronous event \nfires. \u003cstrong\u003eIt is important to not call the callback directly\u003c/strong\u003e, instead, the external API must call the `invoke(event)` \nmethod on the returned `AsyncCallback`. The execution engine will take care of calling the provided callback in a \n\u003cstrong\u003ethread-safe\u003c/strong\u003e way. The callback can safely access the state of the `GraphStageLogic` implementation.*\n\nThe documentation provides an example but here's something slightly more complex.\n\n### Problem statement\nCreate a Random Numbers `Source` where in order to get a random number, you have to poll an API (we pretend to do this) \nand the result is in a `Future`. As an added twist, the API could fail and we want to retry after X seconds. \n\n### Solution\nThe documentation says you need an `AsyncCallback`. When your asynchronous call gets a result, it needs to call `invoke` \nwith the result on this `AsyncCallback`. They also mention that the `AsyncCallback` takes a function as a parameter. \nIt will call the function with the result of the asynchronous call in a thread safe way. In our code, we will call this \nthe target handler. They also recommend that you set all this up in the `preStart` hook.\n\n![image](https://cloud.githubusercontent.com/assets/14280155/22624572/10a9ed9a-eb4e-11e6-9340-329dd623288f.png)\n\nHere `bufferMessageAndEmulatePull` is our target handler. As you can see, when the target handler gets a message, it \nadds it to the buffer and emulates a pull as if the downstream is calling. Your first reaction is correct in saying that \nit is unsafe to do this which is why in the handler we will check whether we were truly called using the query API to do \nso. Doing this actually allows us to write less code since the checks are in a single area.\n\nWe have setup our `AsyncHandler` so that when it is called with the results, it calls our target handler with the results \nin a thread safe way.\n\nThe last thing that is left to do is actually make the asynchronous call and when the call completes to call `invoke` on \nour `AsyncHandler` so we can get the results safely.\n\n![image](https://cloud.githubusercontent.com/assets/14280155/22624583/4ae4f22a-eb4e-11e6-936e-1dcb61e6bd86.png)\n\nLet's walk through those functions\n\n![image](https://cloud.githubusercontent.com/assets/14280155/22624586/5a7acb9c-eb4e-11e6-91d5-7d6fc1c442ba.png)\n\nI said our asynchronous call could fail by throwing an exception (:cry:) or it produces a result. Here is a \nsynchronous call that produces numbers and we wrap it in a `Future` to make it asynchronous.\n\nSo where are we going to call `invoke` then? \n\n![image](https://cloud.githubusercontent.com/assets/14280155/22624738/09a36ee0-eb53-11e6-936f-0925009092d9.png)\n\nHere's where we listen for completion of the asynchronous call, get the result and call `invoke`. Remember I also said \nI wanted to retry after X seconds (2 seconds for now), so we do that here.\n\nWith all this in place we are ready to write our `OutHandler`.\n\n![image](https://cloud.githubusercontent.com/assets/14280155/22624733/d70ba506-eb52-11e6-8eda-abae4d06fadf.png)\n\nAs you can see, when we are pulled we have to first check whether we are truly pulled from the downstream or whether we \nare artificially called from `bufferMessageAndEmulatePull`. So we use the query `isAvailable` on the `outlet` port to \ncheck.\n\n- If we are truly pulled then we take the first element off the buffer and send it downstream using `push`. \n\n- If we have run out of elements in the buffer then we make our asynchronous call which involves taking the results and \nobtaining them in a thread safe way.\n\nYou can find the entire picture [here](https://github.com/calvinlfer/Akka-Streams-custom-stream-processing-examples/blob/master/src/main/scala/com/calvin/streamy/SideChannelSource.scala)\n\nYou can find it in action [here](https://github.com/calvinlfer/Akka-Streams-custom-stream-processing-examples/blob/master/src/main/scala/com/calvin/streamy/Example.scala#L135)\n\nThe example will keep producing random numbers and handle retries internally in the midst of failures that could occur.\n\n#### Credits\n- [Akka documentation](http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html?_ga=1.95210907.668506683.1483746547#Completion)\n- [Akka Streams SQS](https://github.com/s12v/akka-stream-sqs)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcalvinlfer%2Fakka-streams-custom-stream-processing-examples","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcalvinlfer%2Fakka-streams-custom-stream-processing-examples","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcalvinlfer%2Fakka-streams-custom-stream-processing-examples/lists"}