{"id":17964820,"url":"https://github.com/adroll/exmld","last_synced_at":"2025-03-25T06:31:05.622Z","repository":{"id":28231612,"uuid":"115046957","full_name":"AdRoll/exmld","owner":"AdRoll","description":"elixir interface to kinesis client library via MultiLangDaemon","archived":false,"fork":false,"pushed_at":"2023-07-28T00:52:06.000Z","size":205,"stargazers_count":44,"open_issues_count":5,"forks_count":7,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-03-19T09:21:08.912Z","etag":null,"topics":["elixir","erlang","hacktoberfest","pipeline"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AdRoll.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-21T21:04:30.000Z","updated_at":"2024-09-09T20:58:57.000Z","dependencies_parsed_at":"2022-08-07T13:15:31.887Z","dependency_job_id":null,"html_url":"https://github.com/AdRoll/exmld","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdRoll%2Fexmld","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdRoll%2Fexmld/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdRoll%2Fexmld/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdRoll%2Fexmld/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AdRoll","download_url":"https://codeload.github.com/AdRoll/exmld/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245413645,"owners_count":20611351,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","erlang","hacktoberfest","pipeline"],"created_at":"2024-10-29T12:08:53.299Z","updated_at":"2025-03-25T06:31:05.323Z","avatar_url":"https://github.com/AdRoll.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# exmld\n\nThis application allows Kinesis and DynamoDB streams to be processed using Elixir or\nErlang (by way of the KCL MultiLangDaemon).  It's particularly useful when aggregate\nrecords are being used and items can be processed in approximate order (as opposed to\nstrict order within each shard), but that isn't a requirement.\n\nUsing [erlmld](https://github.com/AdRoll/erlmld), a normal Erlang Kinesis processing\napplication looks like this:\n\n![Erlang - MultiLangDaemon processing](img/erlang-mld-workers.png)\n\nUsing this Elixir library (which uses erlmld), a processing application looks like this:\n\n![Elixir - MultiLangDaemon processing](img/elixir-mld-pipeline.png)\n\nThis is done using the [Flow](https://hexdocs.pm/flow/Flow.html) framework to set up a\nMapReduce-style processing pipeline within a single BEAM node.\n\nBy virtue of using the KCL, processing applications can horizontally scale across a group\nof ([homogenous](https://github.com/awslabs/amazon-kinesis-client/issues/103)) worker\ninstances.\n\nUnlike most applications using the KCL's MultiLangDaemon, an Erlang or Elixir processing\napplication using this library can easily make full use of a worker's processing power\n(even if the stream contains a single shard) due to use of the Flow framework.\n\n# Examples\n\nSee:\n\n  1. [example erlang processor](examples/erlang_processor/)\n  2. [example elixir processor](examples/elixir_processor/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadroll%2Fexmld","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadroll%2Fexmld","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadroll%2Fexmld/lists"}