{"id":15176418,"url":"https://github.com/adroll/erlmld","last_synced_at":"2026-03-16T15:31:27.377Z","repository":{"id":28231625,"uuid":"86761416","full_name":"AdRoll/erlmld","owner":"AdRoll","description":"erlang interface to kinesis client library via MultiLangDaemon","archived":false,"fork":false,"pushed_at":"2024-07-01T00:09:58.000Z","size":172,"stargazers_count":39,"open_issues_count":8,"forks_count":9,"subscribers_count":15,"default_branch":"main","last_synced_at":"2024-12-22T03:06:54.090Z","etag":null,"topics":["erlang","hacktoberfest","java","stream"],"latest_commit_sha":null,"homepage":"","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AdRoll.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2017-03-31T00:26:26.000Z","updated_at":"2023-12-12T03:13:07.000Z","dependencies_parsed_at":"2024-02-14T10:27:36.699Z","dependency_job_id":"a3aaf550-7226-40e1-9f56-148052e5a6ed","html_url":"https://github.com/AdRoll/erlmld","commit_stats":{"total_commits":75,"total_committers":12,"mean_commits":6.25,"dds":0.6533333333333333,"last_synced_commit":"2bfe050aa1db486d7539bcbbbf22e29ca97f36ab"},"previous_names":[],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdRoll%2Ferlmld","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdRoll%2Ferlmld/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdRoll%2Ferlmld/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdRoll%2Ferlmld/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AdRoll","download_url":"https://codeload.github.com/AdRoll/erlmld/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231699952,"owners_count":18413121,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["erlang","hacktoberfest","java","stream"],"created_at":"2024-09-27T13:04:08.219Z","updated_at":"2026-03-16T15:31:27.323Z","avatar_url":"https://github.com/AdRoll.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"# erlmld\n\nThis application allows Kinesis and DynamoDB streams to be processed using Erlang (by way\nof the KCL MultiLangDaemon).\n\n\n## Erlang - MultiLangDaemon concept\n\nThe [Kinesis Client Library](https://github.com/awslabs/amazon-kinesis-client) abstracts\naway the distribution of work among a group of workers which are processing data found on\na Kinesis stream, including lease management and the assignment of specific workers to\nspecific shards.\n\nA component of the KCL is the MultiLangDaemon, which allows non-Java applications to be\nmore easily integrated with the KCL by way of a simple JSON protocol: it launches one\nsubprocess per owned shard and communicates on stdin/stdout.\n\nAdRoll needed the ability to perform Kinesis data processing in an Erlang system, and thus\nthis project was born.  We launch one `erlmld` supervision tree per Kinesis/DynamoDB\nstream being processed in an Erlang node.  This results in an instance of the\nMultiLangDaemon being launched as a port program for each stream being processed.\n\nEach MLD process in turn launches one subprocess for each owned shard; these are simply\nnetcat or socat processes which map I/O back to the Erlang node via TCP, allowing us to do\nall data processing in Erlang:\n\n![Erlang - MultiLangDaemon concept](img/beam-mld.png)\n\nA `gen_statem` worker implements the MLD protocol on the Erlang side; one of these runs in\na process for each shard owned by the current worker.\n\nA [MultiLangDaemon\nadapter](https://github.com/awslabs/dynamodb-streams-kinesis-adapter/blob/master/src/main/java/com/amazonaws/services/dynamodbv2/streamsadapter/StreamsMultiLangDaemon.java)\nsupports DynamoDB streams using the same MLD protocol.\n\nAdRoll uses this to implement a high-performance global cache population system which\nsupports a real-time bidding platform also implemented in Erlang.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadroll%2Ferlmld","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadroll%2Ferlmld","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadroll%2Ferlmld/lists"}