{"id":20660609,"url":"https://github.com/gamaops/hfx-bus","last_synced_at":"2025-10-12T15:52:00.232Z","repository":{"id":34504027,"uuid":"179510124","full_name":"gamaops/hfx-bus","owner":"gamaops","description":"Redis backed high frequency exchange bus","archived":false,"fork":false,"pushed_at":"2022-12-30T17:25:19.000Z","size":518,"stargazers_count":24,"open_issues_count":9,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-15T03:01:46.053Z","etag":null,"topics":["broker","bus","cqrs","event-sourcing","microservices","pubsub","redis","soa"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gamaops.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-04-04T14:08:01.000Z","updated_at":"2024-01-12T17:57:10.000Z","dependencies_parsed_at":"2023-01-15T07:29:26.853Z","dependency_job_id":null,"html_url":"https://github.com/gamaops/hfx-bus","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gamaops%2Fhfx-bus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gamaops%2Fhfx-bus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gamaops%2Fhfx-bus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gamaops%2Fhfx-bus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gamaops","download_url":"https://codeload.github.com/gamaops/hfx-bus/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249071982,"owners_count":21208097,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["broker","bus","cqrs","event-sourcing","microservices","pubsub","redis","soa"],"created_at":"2024-11-16T19:05:33.195Z","updated_at":"2025-10-12T15:51:55.191Z","avatar_url":"https://github.com/gamaops.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HFXBus\n\n**HFXBus** is a bus implementation for NodeJS backed by Redis Streams and PubSub.\n\n* Focused on performance for asynchronous communication between endpoints\n* Replaces Kafka, RabbitMQ and other brokers\n* Support Redis Standalone or Cluster\n* Support for **correct** message acknowledgement, resolution and rejection states\n* Message's payload can be raw Buffer or JSON\n* Claiming logic to retry and collect stalled out messages\n* Architecture and payload agnostic\n* Limit the number of parallel processing done by your microservices\n* Supports client side partitioning to achieve Redis HA\n* Supports distributed routing to distribute streams across nodes\n* Unit and E2E tested\n\nIt's simple and effective to achieve high performance event-sourcing environment and microservice communication.\n\n```bash\nnpm install --save hfxbus\n```\n\n----------------------\n\n## Upgrading\n\nThis project was rewritten in Typescript on v2, if you're running v1 and need reference please visit the branch v1. **The v2+ still defined as RC and will only achieve GA when we finish the tests for HFXBus v2.**\n\n----------------------\n\n## How it works\n\nHFXBus uses [Redis Streams](https://redis.io/topics/streams-intro) to enqueue messages and groups to consume messages, but these streams only controls the flow of messages to make the processing lighweight in networking and memory/CPU aspects. All payload is stored as regular Redis keys and it's up to your endpoints decide which keys need to be loaded/created.\n\n[Redis PubSub](https://redis.io/topics/pubsub) is used to emit events happening to messages like when a message is **consumed**, so your endpoints have feedback about messages events.\n\nAnd finally, with [XTRIM](https://redis.io/commands/xtrim) you can keep your Redis server memory utilization low and with [XCLAIM](https://redis.io/commands/xclaim) improve your (micro)services redundancy/resilience. We implemented the command XRETRY using Lua Scripting to achieve a reliable way to retry stalled out messages.\n\n----------------------\n\n## Client side partitioning\n\nHFXBus provides client side partitioning through the method **ConnectionManager.nodes()**, but you need to be aware of the following points:\n\n* This partitioning is efficient for partitioning job's payload.\n* Consumers group can't read from streams spread through multiple nodes, so you'll need to make them as static routing using the **route** parameter of consumers and the **staticRoutes** of connection parameters.\n* Always use the **sequence** parameter to specify the sequence of nodes when partitioning the data.\n* Producer's listen to Pub/Sub from a single connection only if the stream name is not a pattern, otherwise it'll listen to all nodes.\n\nThis feature was designed to work with the following architecture:\n\n![client side partitioning](https://raw.githubusercontent.com/gamaops/hfx-bus/master/doc/images/client-side-partitioning.png)\n\n## Distributed routing\n\nTo really scale Redis horizontally using the architecture above HFXBus provides a routing method named \"distributed routing\":\n\n* Consumers will try to acquire messages from all connections using a round-robin algorithm to distribute the workload.\n* Producers will not more send messages using the specified **route**, instead they will use the job's ID as route.\n* You don't need to specify static routes.\n\nBut there are tradeoffs with this method:\n\n* Stream messages IDs can be repeated across nodes.\n* The number of connections done by each consumer increases.\n* Currently HFXBus doesn't auto eject failing connections (maybe you want to open a PR for this feature?).\n\n----------------------\n\n## Quick Start\n\nFirst, setup a Redis running at `127.0.0.1:6379` (you can use [docker](https://hub.docker.com/_/redis)). And then create a **consumer.ts** file with the following content:\n\n```typescript\nimport { ConnectionManager, Consumer } from 'hfxbus';\n\nconst connection = ConnectionManager.standalone({\n  port: 6379,\n  host: '127.0.0.1'\n});\n\nconst consumer = new Consumer(connection, { group: 'worldConcat' });\n\nconsumer.process({\n  stream: 'concat',\n  processor: async (job) =\u003e {\n    \n    console.log(`Received job: ${job.id}`);\n\n    const {\n      inbound\n    } = await job.get('inbound', false).del('inbound').pull();\n\n    console.log(`Received inbound: ${inbound}`);\n\n    await job.set('outbound', `${inbound} world!`).push();\n\n    console.log('Job consumed');\n\n  }\n});\n\nconsumer.play().then(() =\u003e {\n  console.log(`Consumer is waiting for jobs (consumer id is ${consumer.id})`);\n}).catch((error) =\u003e console.error(error));\n\n```\n\nAnd another file as **producer.ts**:\n\n```javascript\nimport { ConnectionManager, Producer } from 'hfxbus';\n\nconst connection = ConnectionManager.standalone({\n  port: 6379,\n  host: '127.0.0.1'\n});\n\nconst producer = new Producer(connection);\n\nconst execute = async () =\u003e {\n  \n  await producer.listen();\n\n  console.log(`Producer is listening for messages (producer id is ${producer.id})`);\n\n  const job = producer.job();\n\n  console.log(`Created job: ${job.id}`);\n\n  await job.set('inbound', 'Hello').push();\n\n  await producer.send({\n    stream: 'concat',\n    waitFor: [\n      'worldConcat'\n    ],\n    job\n  });\n\n  console.log(`Sent job: ${job.id}`);\n  \n  await job.finished();\n\n  console.log(`Finished job: ${job.id}`);\n\n  const {\n    outbound\n  } = await job.get('outbound', false).del('outbound').pull();\n\n  console.log(`Outbound is: ${outbound}`);\n\n}\n\nexecute().catch((error) =\u003e console.error(error));\n```\n\nRemember to start **consumer.ts** before **producer.ts** as by default consumer will receive only new jobs, you can change this behavior, take a look at the API Documentation.\n\n----------------------\n\n## API Documentation\n\nYour can learn more about HFXBus API [clicking here](https://github.com/gamaops/hfx-bus/blob/master/API.md).\n\n----------------------\n\n## Related Projects\n\n* [HFXWorker](https://github.com/gamaops/hfx-worker) - A worker pool using NodeJS worker threads.\n* [HFXEventStash](https://github.com/gamaops/hfx-eventstash) - A high performance event store to persist commands (CQRS).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgamaops%2Fhfx-bus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgamaops%2Fhfx-bus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgamaops%2Fhfx-bus/lists"}