{"id":41830387,"url":"https://github.com/cafjs/caf_kafka","last_synced_at":"2026-01-25T08:36:35.071Z","repository":{"id":64036905,"uuid":"522758441","full_name":"cafjs/caf_kafka","owner":"cafjs","description":"A caf.js plugin to build a Kafka gateway","archived":false,"fork":false,"pushed_at":"2023-04-19T00:32:40.000Z","size":52,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-02T06:46:01.306Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cafjs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE-2.0.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-08-09T01:12:29.000Z","updated_at":"2022-08-09T01:12:57.000Z","dependencies_parsed_at":"2023-01-14T19:46:17.052Z","dependency_job_id":null,"html_url":"https://github.com/cafjs/caf_kafka","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cafjs/caf_kafka","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cafjs%2Fcaf_kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cafjs%2Fcaf_kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cafjs%2Fcaf_kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cafjs%2Fcaf_kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cafjs","download_url":"https://codeload.github.com/cafjs/caf_kafka/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cafjs%2Fcaf_kafka/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28749600,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-25T08:31:04.260Z","status":"ssl_error","status_checked_at":"2026-01-25T08:30:28.859Z","response_time":113,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-25T08:36:34.411Z","updated_at":"2026-01-25T08:36:35.064Z","avatar_url":"https://github.com/cafjs.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Caf.js\n\nCo-design cloud assistants with your web app and IoT devices.\n\nSee https://www.cafjs.com\n\n## Caf.js library for processing Kafka streams\n\n[![Build Status](https://github.com/cafjs/caf_kafka/actions/workflows/push.yml/badge.svg)](https://github.com/cafjs/caf_kafka/actions/workflows/push.yml)\n\nThe Kafka Streams API https://docs.confluent.io/platform/current/streams/index.html provides \"exactly-once\" processing of Kafka events, but requires that the input and output data are stored in a Kafka cluster.\n\nCan we also provide \"exactly-once\" event processing when we need interactions with the external world?\n\nFor example, calling Web APIs of third-party providers, or micro-services hosted in a different cloud, or triggering actions on connected IoT devices, or notifying clients...\n\nThis Caf.js library helps you to build a reliable Kafka Gateway to do just that.\n\nIt is typically combined with the standard Kafka Streams API, providing a last  stage of the processing pipeline:\n\n    \u003cTopicA\u003e -\u003e \u003cKafka Streams API\u003e -\u003e \u003cTopicB\u003e -\u003e \u003cKafka/Caf.js Gateway\u003e -\u003e \u003cWeb APIs/Microservice APIs/IoT device APIs/...\u003e\n\nand it can also poll external services/devices regularly, i.e., using Caf.js autonomous agents, to create events for a new Kafka topic:\n\n    \u003cExternal services/devices\u003e -\u003e \u003cKafka/Caf.js Gateway\u003e -\u003e \u003cTopicA\u003e -\u003e \u003cKafka Streams API\u003e -\u003e \u003cTopicB\u003e\n\nA challenge is to provide a reasonable throughput while respecting event ordering. The strategy of the Kafka Streams API, one logical thread per partition, is too slow when one Web API call can take 100 msec of latency or more...\n\nLuckily, only events with the same key need to be processed in order, and we can exploit concurrency between keys in the same partition to increase throughput.\n\nBut then events can be processed out of order, making \"exactly-once\" event processing much more complicated...\n\n### Our Solution\n\nWe associate a Kafka partition with one Caf.js process, and then the `caf_kafka` plugin will consume events in that partition, dispatching them to local CAs, i.e., transactional actors called Cloud Assistants, based on the event key.\n\nCAs never block the node.js loop, and now we can process concurrently thousands of events if they have different keys.\n\nCAs remember the offset of the last processed event in internal (checkpointed) state, filtering duplicated events.\n\nThey also serialize requests with fencing, and CAs internal state is always externally consistent after recovering from a failure.\n\nMoreover, external actions are mediated by transactional plugins, which delay them until a request commits, and also guarantee that recovery actions are idempotent.\n\nThis means that, with the right plugins, we can extend exactly-once semantics to the external world without needing distributed transactions.\n\nSee https://www.cafjslabs.com/orchestration for details.\n\n## API\n\nSee {@link module:caf_kafka/proxy_kafka}\n\n## Configuration\n\n### framework.json\n\nSee {@link module:caf_kafka/plug_kafka}\n\n### ca.json\n\nSee {@link module:caf_kafka/plug_ca_kafka}\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcafjs%2Fcaf_kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcafjs%2Fcaf_kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcafjs%2Fcaf_kafka/lists"}