{"id":27646274,"url":"https://github.com/znsio/aws-lambda-kafka-with-localstack","last_synced_at":"2025-04-24T01:17:56.645Z","repository":{"id":276364004,"uuid":"928449365","full_name":"znsio/aws-lambda-kafka-with-localstack","owner":"znsio","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-10T13:28:46.000Z","size":693,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-04-24T01:17:53.107Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/znsio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-06T16:56:32.000Z","updated_at":"2025-02-10T13:28:50.000Z","dependencies_parsed_at":"2025-02-07T19:42:47.297Z","dependency_job_id":null,"html_url":"https://github.com/znsio/aws-lambda-kafka-with-localstack","commit_stats":null,"previous_names":["znsio/aws-lambda-kafka-with-localstack"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/znsio%2Faws-lambda-kafka-with-localstack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/znsio%2Faws-lambda-kafka-with-localstack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/znsio%2Faws-lambda-kafka-with-localstack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/znsio%2Faws-lambda-kafka-with-localstack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/znsio","download_url":"https://codeload.github.com/znsio/aws-lambda-kafka-with-localstack/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250540887,"owners_count":21447428,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-24T01:17:56.129Z","updated_at":"2025-04-24T01:17:56.630Z","avatar_url":"https://github.com/znsio.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Run Specmatic Kafka Contract Test on AWS Lambda with Amazon MSK on LocalStack using AsyncAPI 3.0\n\nThis project demonstrates below aspects\n* Setting up **AWS Lambda** to consume messages from **Amazon MSK (Kafka)**, all running locally using **LocalStack**.\n* **Contract Test** the Lambda based on [AsyncAPI 3.0](https://www.asyncapi.com/docs/reference/specification/v3.0.0) spec using **Specmatic Kafka Support** (#NOCODE #LOWCODE approach)\n\n## 🚀 Prerequisites\n\n### **1. Install Docker Desktop and AWS CLI**\n\nPlease make sure you have Docker Desktop installed on your machine.\n\nIf you don’t have the AWS CLI installed, install it from:\n[AWS CLI installation](https://docs.aws.amazon.com/cli/v1/userguide/install-macos.html)\n\n### **2. Install LocalStack**\nYou can install LocalStack via pip:\n```sh\nbrew install localstack\n```\nSignup with localstack to get an Auth Token (choose an appropriate license - Example: Trial or Hobby licence).\n\n### **3. Create a Fake AWS Profile for LocalStack**\nSince LocalStack is a **mock AWS environment**, configure a fake profile:\n```sh\naws configure --profile localstack\n```\n- **AWS Access Key:** `test`\n- **AWS Secret Access Key:** `test`\n- **Region:** `us-east-1`\n- **Output Format:** `json`\n\n## 🛠️ **Set Up Kafka \u0026 Lambda in LocalStack**\n\n### **1️⃣ Start LocalStack with persistence enabled**\n```sh\nlocalstack auth set-token \u003cyour-auth-token\u003e\nLOCALSTACK_PERSISTENCE=1 localstack start\n```\n\n#### Troubleshooting `vmnetd` issues with Docker on MacOS\n\nPlease refer to [GitHub comment](https://github.com/docker/for-mac/issues/6677#issuecomment-1593787335).\n\n## 🚀 Setting up Kafka cluster\n### **2️⃣ Create an Amazon Kafka MSK Cluster**\n```sh\naws kafka create-cluster \\\n    --cluster-name my-kafka-cluster \\\n    --kafka-version 2.8.1 \\\n    --broker-node-group-info file://broker-config.json \\\n    --number-of-broker-nodes 1 \\\n    --region us-east-1 \\\n    --profile localstack \\\n    --endpoint-url=http://localhost:4566\n```\n\n### **3️⃣ Verify the Kafka Cluster**\n```sh\naws kafka list-clusters \\\n    --region us-east-1 \\\n    --profile localstack \\\n    --endpoint-url=http://localhost:4566\n```\n\nPlease save the value of the **ClusterArn** field in the response.\nYou will need to use this in some of the next steps where you see `\u003cYOUR_CLUSTER_ARN\u003e`.\n\n### **4️⃣ Get Kafka Bootstrap Brokers**\n```sh\naws kafka get-bootstrap-brokers \\\n    --cluster-arn \"\u003cYOUR_CLUSTER_ARN\u003e\" \\\n    --profile localstack \\\n    --endpoint-url=http://localhost:4566\n```\n**Example Response:**\n```json\n{\n    \"BootstrapBrokerString\": \"localhost.localstack.cloud:4511\"\n}\n```\n\n### **5️⃣ Create Kafka Topics**\n\n**Pre-requisite:** Install Kafka on your local machine to use the `kafka-topics.sh` command.\n\n```sh\nkafka-topics --create \\\n    --bootstrap-server localhost.localstack.cloud:4511 \\\n    --replication-factor 1 \\\n    --partitions 1 \\\n    --topic io.specmatic.json.request\n```\n```sh\nkafka-topics --create \\\n    --bootstrap-server localhost.localstack.cloud:4511 \\\n    --replication-factor 1 \\\n    --partitions 1 \\\n    --topic io.specmatic.json.reply\n```\n\n### **6️⃣ Deploy AWS Lambda**\n\n**Pre-requisite:** Use JDK 17, for example if you are using jenv, please run: `jenv local 17`\n\nFrom the project root folder:\n\nBuild the project and create a fat jar \n```shell\n./gradlew clean shadowJar\n```\n\nDeploy the fat jar as a lambda function\n```sh\naws lambda create-function \\\n    --function-name LambdaToKafka \\\n    --runtime java17 \\\n    --role arn:aws:iam::000000000000:role/lambda-role \\\n    --handler com.example.XsdMessageHandler \\\n    --zip-file fileb://build/libs/aws-lambda-kafka.jar \\\n    --timeout 30 \\\n    --memory-size 512 \\\n    --region us-east-1 \\\n    --profile localstack \\\n    --endpoint-url=http://localhost:4566\n```\n\n`Use `q` to quit.`\n\n### **7️⃣ Verify Lambda Deployment**\n```sh\naws lambda list-functions --profile localstack --endpoint-url=http://localhost:4566\n```\n\nSearch for the function `LambdaToKafka` using `/` and `q` to quit.\n\n### **8️⃣ Create Event Source Mapping for Kafka**\n```sh\naws lambda create-event-source-mapping \\\n    --function-name LambdaToKafka \\\n    --event-source-arn \"\u003cYOUR_CLUSTER_ARN\u003e\" \\\n    --topics \"io.specmatic.json.request\" \\\n    --starting-position LATEST \\\n    --region us-east-1 \\\n    --profile localstack \\\n    --endpoint-url=http://localhost:4566\n```\n\n`Use `q` to quit.`\n\n## Testing the Lambda function:\n\n### Publish a message on the **io.specmatic.json.request** topic:\n\n```shell\nkafka-console-producer --broker-list localhost:4511 --topic io.specmatic.json.request\n```\n\nCopy and paste the following json object and press enter:\n```json\n{\"id\": 1, \"xsd\": \"xsd 1\"}\n```\n\nPress `Ctrl+D`.\n\n### Verify message on the **io.specmatic.json.reply** topic:\n```shell\nkafka-console-consumer --bootstrap-server localhost:4511 --topic io.specmatic.json.reply --from-beginning\n```\n\nYou should see the following message :\n```json\n{\"id\": 1, \"json\": \"Converted from XSD\"}\n```\n\nIf you don't see this message, check the logs for your lambda function:\n```shell\naws logs tail /aws/lambda/LambdaToKafka --follow \\\n    --profile localstack \\\n    --endpoint-url=http://localhost:4566\n```\n\n## **Run Specmatic Kafka Contract Tests using AsyncAPI spec**\n\nThis step now uses **Specmatic Kafka Support** to leverage **AsyncAPI 3.0 spec** to contract test the above Lambda setup.\nThe **AsyncAPI 3.0 spec** models the Event Driven Architecture, the topics and the schema of messages sent / received on those topics.\n\nPlease keep your Local Stack running for this next step.\n\n```shell\n  ./gradlew test\n```\n\nYou should now see the interactive Specmatic Kafka HTML test report here - [`build/reports/index.html`](build/reports/index.html).\nThe report has drill down details on the messages sent and received on the Kafka topics and if the messages are as per the schema in **AsyncAPI spec**.\n\n## Shutdown LocalStack\n\n```shell\nlocalstack stop\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fznsio%2Faws-lambda-kafka-with-localstack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fznsio%2Faws-lambda-kafka-with-localstack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fznsio%2Faws-lambda-kafka-with-localstack/lists"}