{"id":18462702,"url":"https://github.com/extism/extism-kafka-consumer","last_synced_at":"2025-07-09T16:09:54.305Z","repository":{"id":139443771,"uuid":"593385127","full_name":"extism/extism-kafka-consumer","owner":"extism","description":"Extend Kafka Consumers with Extism","archived":false,"fork":false,"pushed_at":"2023-03-15T20:38:20.000Z","size":23,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-28T12:19:52.843Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/extism.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-25T21:48:20.000Z","updated_at":"2024-05-25T08:39:52.000Z","dependencies_parsed_at":null,"dependency_job_id":"af6f7b6a-16e1-4574-bb0a-6b819de691fc","html_url":"https://github.com/extism/extism-kafka-consumer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/extism/extism-kafka-consumer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/extism%2Fextism-kafka-consumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/extism%2Fextism-kafka-consumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/extism%2Fextism-kafka-consumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/extism%2Fextism-kafka-consumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/extism","download_url":"https://codeload.github.com/extism/extism-kafka-consumer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/extism%2Fextism-kafka-consumer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264492775,"owners_count":23617049,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T09:04:08.537Z","updated_at":"2025-07-09T16:09:53.769Z","avatar_url":"https://github.com/extism.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Extism Kafka Consumer\n\n\u003e **Note**: This is not currently meant to be a finished product itself but rather an experiment in\n\u003e  how you could allow customers to extend your Kafka cluster. \n\nOne interesting use case for Extism is to allow users to respond to events in your infrastructure. Something like\nwebhooks but, the compute is inside your infrastructure instead remotely in your customer's.\n\nMost companies have embraced Kafka as a tool to build these kind of event based architectures and they are running\ntheir own clusters or using hosted services such as [Confluent.io](https://www.confluent.io/).\n\nThis repo explores the idea of extending parts of the Kafka ecosystem with Extism plug-ins. Specifically this\nallows you to run an Extism plug-in as a [Kafka Consumer](https://kafka.apache.org/25/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html).\nYou could also do the same thing with the [Producer](https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html) or any other parts of the Kafka ecosystem.\n\nThis is also to experiment and improve upon the [Host Functions](https://extism.org/docs/concepts/host-functions/) interface. We need the plugins to be able to invoke some of the Kafka Consumer API (e.g. `Consumer#Commit()`).\n\n## Using\n\nYou need docker, rust, go, and extism installed.\n\n```bash\ngit clone git@github.com/extism/extism-kafka-consumer.git\ncd extism-kafka-consumer\n\n# build the rust plugin:\ncd plugin/ \u0026\u0026 cargo build --target wasm32-unknown-unknown --release \u0026\u0026 cd ..\n\n# build the consumer\ngo build\n\n# spin up kafka using docker compose\ndocker compose up -d\n\n# create a topic called weather:\ndocker compose exec broker \\\n    kafka-topics --create \\\n        --topic weather \\\n        --bootstrap-server localhost:9092 \\\n        --replication-factor 1 \\\n        --partitions 1\n\n# run the consumer\n./kafka-consumer\n```\n\nIn another tab, open a producer and feed some events into it:\n\n```bash\ndocker compose exec broker bash\nkafka-console-producer --topic weather --broker-list broker:9092\n\u003e72 \n\u003e73\n\u003e74\n\u003e\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fextism%2Fextism-kafka-consumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fextism%2Fextism-kafka-consumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fextism%2Fextism-kafka-consumer/lists"}