{"id":19968369,"url":"https://github.com/floydspace/effect-kafka","last_synced_at":"2025-05-04T01:30:24.227Z","repository":{"id":258718867,"uuid":"874476317","full_name":"floydspace/effect-kafka","owner":"floydspace","description":"📨 Effectful Kafka","archived":false,"fork":false,"pushed_at":"2025-02-04T22:30:10.000Z","size":658,"stargazers_count":14,"open_issues_count":4,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-29T07:26:18.860Z","etag":null,"topics":["confluent-kafka","ecosystem","effect","effect-ts","kafka","kafka-client","kafka-consumer","kafka-producer","kafkajs","typescript"],"latest_commit_sha":null,"homepage":"https://floydspace.github.io/effect-kafka/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/floydspace.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"floydspace"}},"created_at":"2024-10-17T22:36:16.000Z","updated_at":"2025-04-25T09:27:30.000Z","dependencies_parsed_at":"2024-11-06T10:35:12.334Z","dependency_job_id":"8b15e6b4-e56d-41df-a992-46b0d10937e5","html_url":"https://github.com/floydspace/effect-kafka","commit_stats":{"total_commits":43,"total_committers":3,"mean_commits":"14.333333333333334","dds":"0.13953488372093026","last_synced_commit":"975f8b0c584daf6b9ae26ee1aff67c74103345e6"},"previous_names":["floydspace/effect-kafka"],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/floydspace%2Feffect-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/floydspace%2Feffect-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/floydspace%2Feffect-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/floydspace%2Feffect-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/floydspace","download_url":"https://codeload.github.com/floydspace/effect-kafka/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252224433,"owners_count":21714394,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["confluent-kafka","ecosystem","effect","effect-ts","kafka","kafka-client","kafka-consumer","kafka-producer","kafkajs","typescript"],"created_at":"2024-11-13T02:45:36.676Z","updated_at":"2025-05-04T01:30:24.208Z","avatar_url":"https://github.com/floydspace.png","language":"TypeScript","funding_links":["https://github.com/sponsors/floydspace"],"categories":[],"sub_categories":[],"readme":"# Effect Kafka\n\n[`effect-kafka`](https://github.com/floydspace/effect-kafka) is a Kafka client for [Effect](https://github.com/Effect-TS/effect). It provides a purely functional interface to the Kafka client and integrates effortlessly with Effect ecosystem.\n\n[![npm version](https://img.shields.io/npm/v/effect-kafka?color=brightgreen\u0026label=npm%20package)](https://www.npmjs.com/package/effect-kafka)\n[![npm downloads](https://img.shields.io/npm/dm/effect-kafka)](https://www.npmjs.com/package/effect-kafka)\n[![build status](https://img.shields.io/github/actions/workflow/status/floydspace/effect-kafka/release.yml?branch=main)](https://github.com/floydspace/effect-kafka/actions)\n[![wakatime](https://wakatime.com/badge/github/floydspace/effect-kafka.svg)](https://wakatime.com/badge/github/floydspace/effect-kafka)\n\nEffect Docs: https://www.effect.website\u003cbr\u003e\nEffect Reference: https://effect-ts.github.io/effect\u003cbr\u003e\nEffect Kafka Reference: https://floydspace.github.io/effect-kafka\n\n# Installation\n\nChoose your preferred package manager and run one of the following commands in your terminal:\n\n- **Using npm:**\n\n  ```sh\n  npm install effect-kafka\n  ```\n\n- **Using pnpm:**\n\n  ```sh\n  pnpm add effect-kafka\n  ```\n\n- **Using yarn:**\n  ```sh\n  yarn add effect-kafka\n  ```\n\nNext install one of kafka engine packages:\n- [KafkaJS](https://github.com/tulios/kafkajs?tab=readme-ov-file#-getting-started) - Fully JavaScript implementation.\n- [@confluentinc/kafka-javascript](https://github.com/confluentinc/confluent-kafka-javascript?tab=readme-ov-file#requirements) - JavaScript interface for C++ librdkafka implementation, which is more performant, but requires native bindings.\n\n_**Note:** You can use any of the above Kafka engine packages, depending on your preference._\n\n# Usage\n\nLet's write a simple Kafka producer and consumer using `effect-kafka`. Before everything, we need a running instance of Kafka. We can do that by saving the following docker-compose script in the `docker-compose.yml` file and run `docker-compose up`:\n\n```yaml\nversion: '2'\nservices:\n  zookeeper:\n    image: confluentinc/cp-zookeeper:latest\n    environment:\n      ZOOKEEPER_CLIENT_PORT: 2181\n      ZOOKEEPER_TICK_TIME: 2000\n    ports:\n      - 22181:2181\n  \n  kafka:\n    image: confluentinc/cp-kafka:latest\n    depends_on:\n      - zookeeper\n    ports:\n      - 29092:29092\n    environment:\n      KAFKA_BROKER_ID: 1\n      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092\n      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT\n      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT\n      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\n```\n\nNow, we can run our `effect-kafka` application:\n```typescript\nimport { NodeRuntime } from \"@effect/platform-node\";\nimport { Console, Effect, Layer, Random, Schedule, Stream } from \"effect\";\nimport { Consumer, Producer } from \"effect-kafka\";\nimport { ConfluentKafkaJS } from \"effect-kafka/ConfluentKafka\";\n\nconst producer = Stream.repeatEffect(Random.nextInt).pipe(\n  Stream.schedule(Schedule.fixed(\"2 seconds\")),\n  Stream.flatMap((random) =\u003e\n    Producer.send({\n      topic: \"random\",\n      messages: [{ key: String(random % 4), value: random.toString() }],\n    }),\n  ),\n);\n\nconst consumer = Consumer.serveStream(\"random\").pipe(\n  Stream.tap((record) =\u003e Console.log(record.value?.toString()))\n);\n\nconst program = Stream.merge(producer, consumer).pipe(Stream.runDrain);\n\nconst ProducerLive = Producer.layer({ allowAutoTopicCreation: true });\nconst ConsumerLive = Consumer.layer({ groupId: \"group\" });\n\nconst KafkaLive = ConfluentKafkaJS.layer({ brokers: [\"localhost:29092\"] });\nconst MainLive = program.pipe(\n  Effect.provide(Layer.merge(ProducerLive, ConsumerLive)),\n  Effect.provide(KafkaLive)\n);\n\nNodeRuntime.runMain(MainLive);\n```\n\nSee more examples in the [examples](./examples) directory.\n\n# Roadmap\n\n- [x] Consumer\n- [x] Producer\n- [x] Consumer Streams\n- [ ] Producer Streams\n- [x] Acknowledge management\n- [ ] Transactions\n\n# Contributing Guidelines\n\nThank you for considering contributing to our project! Here are some guidelines to help you get started:\n\n## Reporting Bugs\n\nIf you have found a bug, please open an issue on our [issue tracker](https://github.com/floydspace/effect-kafka/issues) and provide as much detail as possible. This should include:\n\n- A clear and concise description of the problem\n- Steps to reproduce the problem\n- The expected behavior\n- The actual behavior\n- Any relevant error messages or logs\n\n## Suggesting Enhancements\n\nIf you have an idea for an enhancement or a new feature, please open an issue on our [issue tracker](https://github.com/floydspace/effect-kafka/issues) and provide as much detail as possible. This should include:\n\n- A clear and concise description of the enhancement or feature\n- Any potential benefits or use cases\n- Any potential drawbacks or trade-offs\n\n## Pull Requests\n\nWe welcome contributions via pull requests! Here are some guidelines to help you get started:\n\n1. Fork the repository and clone it to your local machine.\n2. Create a new branch for your changes: `git checkout -b my-new-feature`\n3. Install dependencies: `pnpm install` (`pnpm@9.x`, using `corepack`)\n    - if you introduce new dependencies, please use `.projenrc.ts` to add them\n    - then run `pnpm default` to update the project\n4. Make your changes and add tests if applicable.\n5. Run the tests: `pnpm test`\n6. Commit your changes: `git commit -am 'Add some feature'`\n7. Push your changes to your fork: `git push origin my-new-feature`\n8. Open a pull request against our `main` branch.\n\n### Pull Request Guidelines\n\n- Please make sure your changes are consistent with the project's existing style and conventions.\n- Please write clear commit messages and include a summary of your changes in the pull request description.\n- Please make sure all tests pass and add new tests as necessary.\n- If your change requires documentation, please update the relevant documentation.\n- Please be patient! We will do our best to review your pull request as soon as possible.\n\n## License\n\nBy contributing to this project, you agree that your contributions will be licensed under the project's [MIT License](./LICENSE).\n\n# Sponsors\n\nWe are grateful to the following sponsors for supporting this project:\n\n\u003cdiv style=\"display: flex\"\u003e\n    \u003cdiv style=\"display: flex; justify-content: center; flex-direction: column; align-items: center\"\u003e\n        \u003ca href=\"https://github.com/superwall\"\u003e\n        \u003cimg title=\"@superwall\" src=\"https://avatars.githubusercontent.com/u/88794805?s=200\u0026amp;v=4\" width=\"75\" height=\"75\" alt=\"@superwall\"\u003e\n        \u003c/a\u003e\n        \u003cdiv\u003eSuperwall\u003c/div\u003e\n    \u003c/div\u003e\n\u003c/div\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffloydspace%2Feffect-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffloydspace%2Feffect-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffloydspace%2Feffect-kafka/lists"}