{"id":15285977,"url":"https://github.com/ymtszw/elm-broker","last_synced_at":"2025-03-23T21:13:49.650Z","repository":{"id":62419675,"uuid":"142188545","full_name":"ymtszw/elm-broker","owner":"ymtszw","description":"Data stream buffer for Elm application, inspired by Apache Kafka","archived":false,"fork":false,"pushed_at":"2018-11-12T13:19:34.000Z","size":74,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-14T18:22:48.642Z","etag":null,"topics":["broker","buffer","elm","elm-lang","kafka","stream-processing"],"latest_commit_sha":null,"homepage":"https://package.elm-lang.org/packages/ymtszw/elm-broker/latest/","language":"Elm","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ymtszw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-24T17:00:37.000Z","updated_at":"2021-12-09T16:44:18.000Z","dependencies_parsed_at":"2022-11-01T17:00:34.767Z","dependency_job_id":null,"html_url":"https://github.com/ymtszw/elm-broker","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ymtszw%2Felm-broker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ymtszw%2Felm-broker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ymtszw%2Felm-broker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ymtszw%2Felm-broker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ymtszw","download_url":"https://codeload.github.com/ymtszw/elm-broker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245168912,"owners_count":20571804,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["broker","buffer","elm","elm-lang","kafka","stream-processing"],"created_at":"2024-09-30T15:08:53.592Z","updated_at":"2025-03-23T21:13:49.630Z","avatar_url":"https://github.com/ymtszw.png","language":"Elm","funding_links":[],"categories":[],"sub_categories":[],"readme":"# elm-broker\n\n[![CircleCI](https://circleci.com/gh/ymtszw/elm-broker/tree/master.svg?style=svg)](https://circleci.com/gh/ymtszw/elm-broker/tree/master)\n\nData stream buffer for Elm application, inspired by [Apache Kafka](https://kafka.apache.org/).\n\n## What is this?\n\n- `Broker` is essentially a [circular buffer](https://www.wikiwand.com/en/Circular_buffer), internally using `Array`\n- Read pointers (`Offset`) are exposed to clients, allowing \"pull\"-style data consumption, just as in Kafka\n- Insert(`append`), `read`, and `update` all take _O(1)_\n- A buffer is made of multiple `Segment`s. Buffer size (= number of `Segment`s and size of each `Segment`) can be configured\n- When the whole buffer is filled up, a new \"cycle\" begins and old `Segment`s are evicted one by one\n\n## Expected Usage\n\n- A `Broker` accepts incoming data stream. \u0026dagger;\n- Several **consumers** reads (\"pulls\") data from the `Broker` individually, while maintaining each `Offset` as their internal states.\n- Consumers perform arbitrary operations against acquired data, then read `Broker` again after previous `Offset`. Rinse and repeat.\n\n\u0026dagger;\nIt is possible to have multiple `Broker`s in your application for different purposes,\nhowever you must be careful so that you do not mix up `Offset`s produced from one `Broker` to ones from others.\nSince `Offset`s are only valid for their generating `Broker`.\nWrapping `Offset`s in phantom types is a possible technique to enforce this restriction.\n\n## Remarks\n\n- Technically, it can also perform _O(1)_ `delete`, but it is still unclear whether we want `delete` API\n    - Original Kafka now [supports this as an admin command](https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/DeleteRecordsCommand.scala)\n- There are several major features I am interested in:\n    - More sophisticated/efficient dump and reload. Current implementation is plain old `encode` and `decoder` pair,\n      which is potentially inefficient for big-capacity `Broker`s.\n    - Bulk append and bulk read\n    - Callback mechanism around `Segment` eviction\n\n## Development\n\nInstall Elm Platform.\n\n```sh\n$ elm make\n$ elm-test                    # full test\n$ elm-test tests/MainTest.elm # only light-weight tests\n```\n\n## License\n\nCopyright \u0026copy; 2018, [Yu Matsuzawa](https://github.com/ymtszw)\n\nBSD-3-Clause\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fymtszw%2Felm-broker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fymtszw%2Felm-broker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fymtszw%2Felm-broker/lists"}