https://github.com/adroll/erlmld

erlang interface to kinesis client library via MultiLangDaemon
https://github.com/adroll/erlmld

erlang hacktoberfest java stream

Last synced: 1 day ago
JSON representation

erlang interface to kinesis client library via MultiLangDaemon

• Host: GitHub
• URL: https://github.com/adroll/erlmld
• Owner: AdRoll
• License: bsd-3-clause
• Created: 2017-03-31T00:26:26.000Z (over 7 years ago)
• Default Branch: main
• Last Pushed: 2024-07-01T00:09:58.000Z (5 months ago)
• Last Synced: 2024-11-13T00:04:12.788Z (1 day ago)
• Topics: erlang, hacktoberfest, java, stream
• Language: Erlang
• Homepage:
• Size: 168 KB
• Stars: 39
• Watchers: 16
• Forks: 9
• Open Issues: 8
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# erlmld

This application allows Kinesis and DynamoDB streams to be processed using Erlang (by way

of the KCL MultiLangDaemon).

## Erlang - MultiLangDaemon concept

The [Kinesis Client Library](https://github.com/awslabs/amazon-kinesis-client) abstracts

away the distribution of work among a group of workers which are processing data found on

a Kinesis stream, including lease management and the assignment of specific workers to

specific shards.

A component of the KCL is the MultiLangDaemon, which allows non-Java applications to be

more easily integrated with the KCL by way of a simple JSON protocol: it launches one

subprocess per owned shard and communicates on stdin/stdout.

AdRoll needed the ability to perform Kinesis data processing in an Erlang system, and thus

this project was born.  We launch one `erlmld` supervision tree per Kinesis/DynamoDB

stream being processed in an Erlang node.  This results in an instance of the

MultiLangDaemon being launched as a port program for each stream being processed.

Each MLD process in turn launches one subprocess for each owned shard; these are simply

netcat or socat processes which map I/O back to the Erlang node via TCP, allowing us to do

all data processing in Erlang:

![Erlang - MultiLangDaemon concept](img/beam-mld.png)

A `gen_statem` worker implements the MLD protocol on the Erlang side; one of these runs in

a process for each shard owned by the current worker.

A [MultiLangDaemon

adapter](https://github.com/awslabs/dynamodb-streams-kinesis-adapter/blob/master/src/main/java/com/amazonaws/services/dynamodbv2/streamsadapter/StreamsMultiLangDaemon.java)

supports DynamoDB streams using the same MLD protocol.

AdRoll uses this to implement a high-performance global cache population system which

supports a real-time bidding platform also implemented in Erlang.