https://github.com/adroll/exmld

elixir interface to kinesis client library via MultiLangDaemon
https://github.com/adroll/exmld

elixir erlang hacktoberfest pipeline

Last synced: about 2 months ago
JSON representation

elixir interface to kinesis client library via MultiLangDaemon

• Host: GitHub
• URL: https://github.com/adroll/exmld
• Owner: AdRoll
• License: bsd-3-clause
• Created: 2017-12-21T21:04:30.000Z (about 7 years ago)
• Default Branch: main
• Last Pushed: 2023-07-28T00:52:06.000Z (over 1 year ago)
• Last Synced: 2024-10-29T08:42:16.839Z (about 2 months ago)
• Topics: elixir, erlang, hacktoberfest, pipeline
• Language: Elixir
• Homepage:
• Size: 200 KB
• Stars: 44
• Watchers: 19
• Forks: 7
• Open Issues: 5
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# exmld

This application allows Kinesis and DynamoDB streams to be processed using Elixir or

Erlang (by way of the KCL MultiLangDaemon).  It's particularly useful when aggregate

records are being used and items can be processed in approximate order (as opposed to

strict order within each shard), but that isn't a requirement.

Using [erlmld](https://github.com/AdRoll/erlmld), a normal Erlang Kinesis processing

application looks like this:

![Erlang - MultiLangDaemon processing](img/erlang-mld-workers.png)

Using this Elixir library (which uses erlmld), a processing application looks like this:

![Elixir - MultiLangDaemon processing](img/elixir-mld-pipeline.png)

This is done using the [Flow](https://hexdocs.pm/flow/Flow.html) framework to set up a

MapReduce-style processing pipeline within a single BEAM node.

By virtue of using the KCL, processing applications can horizontally scale across a group

of ([homogenous](https://github.com/awslabs/amazon-kinesis-client/issues/103)) worker

instances.

Unlike most applications using the KCL's MultiLangDaemon, an Erlang or Elixir processing

application using this library can easily make full use of a worker's processing power

(even if the stream contains a single shard) due to use of the Flow framework.

# Examples

See:

  1. [example erlang processor](examples/erlang_processor/)

  2. [example elixir processor](examples/elixir_processor/)