Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/BigThinkcode/ex_aws_msk_iam_auth
AWS_MSK_IAM Authentication Plugin for Broadway Kafka
https://github.com/BigThinkcode/ex_aws_msk_iam_auth
Last synced: 3 months ago
JSON representation
AWS_MSK_IAM Authentication Plugin for Broadway Kafka
- Host: GitHub
- URL: https://github.com/BigThinkcode/ex_aws_msk_iam_auth
- Owner: BigThinkcode
- License: mit
- Created: 2022-10-24T09:35:16.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2023-06-12T14:24:34.000Z (over 1 year ago)
- Last Synced: 2024-04-22T13:33:25.264Z (7 months ago)
- Language: Elixir
- Size: 53.7 KB
- Stars: 4
- Watchers: 6
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-elixir - ex_aws_msk_iam_auth - AWS Managed Streaming for Apache Kafka (MSK) IAM Authentication plugin for Broadway Kafka. (Authentication)
README
# ex_aws_msk_iam_auth
`ex_aws_msk_iam_auth` is an authentication plugin for broadway_kafka. It enables Broadway Kafka clients to authenticate with Amazon's Managed Streaming for Apache Kafka(Amazon MSK) via [AWS_MSK_IAM](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html) SASL mechanism.
## Installation
Add the following dependency to your `mix.exs`
```elixir
def deps do
[
{:ex_aws_msk_iam_auth, git: "https://github.com/BigThinkcode/ex_aws_msk_iam_auth"}
]
end
```
## Usage
Broadway Kafka supports connecting to Kafka broker via SASL authentication. The following sample configuration shows how `ex_aws_msk_iam_auth` plugin can be used with it.
Ref: https://hexdocs.pm/broadway_kafka/BroadwayKafka.Producer.html#module-client-config-options
```elixir
client_config: [
sasl:
{
:callback,
ExAwsMskIamAuth,
{:AWS_MSK_IAM, "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"}
},
ssl: true
]
```
## Background
### Broadway Kafka and brod
[Broadway Kafka](https://github.com/dashbitco/broadway_kafka) is a Kafka Connector for [Broadway](https://github.com/dashbitco/broadway) - an Elixir library to build concurrent, multi-stage data ingestion/processing pipelines with Elixir.
Broadway Kafka is an amalgamation of awesome features from Broadway with Kafka as a producer. Internally, it uses [brod](https://github.com/kafka4beam/brod) as its Kafka client acting as a wrapper. Brod supports `SASL PLAIN`, `SCRAM-SHA-256` and `SCRAM-SHA-512` authentication mechanisms out of the box and also offers extension points to support custom [authentication plugins](https://github.com/kafka4beam/brod#authentication-support).
### AWS MSK Authentication Mechanisms
MSK supports two variants - MSK Fully Managed and MSK Serverless. In both the variants, Kafka service can be protected via SASL, in particular, AWS's custom SASL mechanism AWS_MSK_IAM(https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html). At the time of writing this library, MSK's Serverless variant's only supported authentication was AWS_MSK_IAM SASL mechanism.
### Solution Diagram
### Implementation
This library takes inspiration from its Java counterpart [aws-msk-iam-auth](https://github.com/aws/aws-msk-iam-auth)
### Relevant Issues/PRs
1. https://github.com/dashbitco/broadway_kafka/issues/82
2. https://github.com/dashbitco/broadway_kafka/pull/85
3. https://github.com/aws-beam/aws_signature/issues/14