{"id":17888229,"url":"https://github.com/poga/hyperkafka","last_synced_at":"2025-03-22T19:33:48.697Z","repository":{"id":137993846,"uuid":"81452135","full_name":"poga/hyperkafka","owner":"poga","description":"interplanetary messaging platform","archived":false,"fork":false,"pushed_at":"2018-07-06T14:04:26.000Z","size":44,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-01T18:03:24.063Z","etag":null,"topics":["decentralized","hyperdrive","interplanetary","javascript","kafka"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/poga.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-02-09T13:19:12.000Z","updated_at":"2021-09-12T19:07:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"c1c0703a-9db0-4d56-80fb-c6c7b6848bcb","html_url":"https://github.com/poga/hyperkafka","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poga%2Fhyperkafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poga%2Fhyperkafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poga%2Fhyperkafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poga%2Fhyperkafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/poga","download_url":"https://codeload.github.com/poga/hyperkafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244236061,"owners_count":20420753,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["decentralized","hyperdrive","interplanetary","javascript","kafka"],"created_at":"2024-10-28T13:36:52.648Z","updated_at":"2025-03-22T19:33:48.685Z","avatar_url":"https://github.com/poga.png","language":"JavaScript","readme":"# hyperkafka\n\n![stability-experimental](https://img.shields.io/badge/stability-experimental-orange.svg?style=flat-square)\n[![NPM Version](https://img.shields.io/npm/v/hyperkafka.svg?style=flat-square)](https://www.npmjs.com/package/hyperkafka)\n[![JavaScript Style Guide](https://img.shields.io/badge/code%20style-standard-brightgreen.svg?style=flat-square)](http://standardjs.com/)\n\nA decentralized messaging platform.\n\n`npm i hyperkafka`\n\n## Usage\n\n```js\n// producer\nvar drive = hyperdrive(memdb())\nvar archive = drive.createArchive()\n\nvar producer = hyperkafka.Producer(archive)\nproducer.write('topic', 'hello')\n\n\n// somewhere, another machine\nvar archive2 = drive.createArchive(archive.key)\nvar consumer = hyperkafka.Consumer(archive2)\nconsumer.get('topic', 0, function (err, msg) {\n  // msg === {offset: 0, ts: timestamp, payload: buffer('hello')}\n})\n\n// create a readable stream\nvar rs = consumer.createReadStream('topic')\nrs.on('data', msg =\u003e { })\n```\n\n## API\n\n### Producer\n\n#### `p = new Producer(archive, opts)`\n\nCreate a new `Producer` with a [hyperdrive](https://github.com/mafintosh/hyperdrive) archive.\n\nOption `opts` includes:\n\n* segmentSize: the file size limit of a single segment.\n* linger: the delay before flushing buffer into hyperdrive\n\n#### `p.write(topic, message, [timestamp])`\n\nAppend a message to a topic. `message` can be `Buffer`, `String`, or object.\n\nYou can specify timestamp for the message. The timestamp should be a `uint64be` buffer because there's no int64 support in javascript. By default it will be current time (`Date.now()`).\n\n#### `p.topics(cb(err, topics))`\n\nReturns an array of topics\n\n#### `p.on('flush', cb(segmentSize, flushedTopci))`\n\nProducer will emit `flush` event when a segment is flushed into hyperdrive.\n\n### Consumer\n\n#### `c = new Consumer(archive)`\n\nCreate a new `Consumer` with a [hyperdrive](https://github.com/mafintosh/hyperdrive) archive.\n\n#### `c.get(topic, offset, cb(err, message))`\n\nGet a message on a topic with specified offset. Returns error if there's no message with given offset.\n\n#### `c.createReadStream(topic, start)`\n\nGet a readable stream with all messages in a topic after given offset.\n\n#### `c.topics(cb(err, topics))`\n\nReturns an array of topics\n\n\n## FAQ\n\n#### Why?\n\nEvent-sourcing is a popular pattern for building large-scale high-performance system. It's also a simple and extensible software architecture.\n\nBy utilizing the well-designed Kafka's API, we can make developing P2P app simpler and easier to reason.\n\n#### This is just append-only log! Why not just use hypercore?\n\nHyperdrive's indexing allows us to query and insert data efficiently.\n\nAlso, If we simply push every single message as a seperated block into hypercore. The overhead is big: 144 bytes per message for 100000 messages. Therefore we need batching.\n\n#### This is basic messaging! How about MQTT/RabbitMQ/ZeroMQ...or any other message queue?\n\nMessage queues usually don't preserve message history. I want a messaging platform which can automatically archive all message in a decentralized environment.\n\n#### Should I replace my apache kafka cluster with hyperkafka?\n\nNo. They have different goal and use cases. hyperkafka use the name because its internal storage is inspired by apache kafka, also it sounds cool.\n\nIf you want a private, highly-scalable messaging platform, use apache kafka.\n\nIf you want to store and share public realtime data, try hyperkafka. You might like it.\n\n## License\n\nThe MIT License\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpoga%2Fhyperkafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpoga%2Fhyperkafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpoga%2Fhyperkafka/lists"}