{"id":23274025,"url":"https://github.com/sparsh-kumar/redis-message-queue","last_synced_at":"2026-05-03T09:38:42.676Z","repository":{"id":268768078,"uuid":"905395223","full_name":"Sparsh-Kumar/redis-message-queue","owner":"Sparsh-Kumar","description":"Implementation of message queue using Redis as message broker from scratch.","archived":false,"fork":false,"pushed_at":"2025-01-05T15:27:36.000Z","size":341,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-12T16:26:06.012Z","etag":null,"topics":["backend-architecture","gulp","message-queue","node-js","redis","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Sparsh-Kumar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-18T18:30:44.000Z","updated_at":"2025-01-05T15:27:39.000Z","dependencies_parsed_at":"2024-12-18T20:22:58.378Z","dependency_job_id":"bd8a93ad-6bf1-4b28-9da5-29b49dd88ac7","html_url":"https://github.com/Sparsh-Kumar/redis-message-queue","commit_stats":{"total_commits":17,"total_committers":1,"mean_commits":17.0,"dds":0.0,"last_synced_commit":"1c63d8a672a755152066b423a9003713ac7940b7"},"previous_names":["sparsh-kumar/redis-message-queue"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sparsh-Kumar%2Fredis-message-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sparsh-Kumar%2Fredis-message-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sparsh-Kumar%2Fredis-message-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sparsh-Kumar%2Fredis-message-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Sparsh-Kumar","download_url":"https://codeload.github.com/Sparsh-Kumar/redis-message-queue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247471385,"owners_count":20944154,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend-architecture","gulp","message-queue","node-js","redis","typescript"],"created_at":"2024-12-19T20:11:33.435Z","updated_at":"2026-05-03T09:38:42.637Z","avatar_url":"https://github.com/Sparsh-Kumar.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Overview\r\n\r\n## What is a Redis Message Queue?\r\n\r\nThe Redis Message Queue is a straightforward implementation of message queues, utilizing Redis streams as the message broker.\r\n\r\n## Development Set Up\r\n- Clone the repository using ```git clone https://github.com/Sparsh-Kumar/redis-message-queue```\r\n- Go into cloned directory using ```cd {{cloned_path}}```\r\n- Do ```npm install```\r\n- Create a ```.env``` file in the root directory of the project.\r\n- Take reference from ```.env.sample``` file which environment variables are required.\r\n\r\n## Usage\r\n\r\n## Message Producer Sample Code\r\n\r\n```javascript\r\n// This is how to produce the messages;\r\nasync function produceMessage() {\r\n\r\n  // Creating \u0026 Initializing the Redis Message Queue\r\n  const redisMessageQueue = new RedisMessageQueue({\r\n    redisMessageQueueName: 'msgq',\r\n    redisUrl: '',\r\n    redisPort: 6379,\r\n    failOverQueueHandling: false,\r\n  });\r\n  await redisMessageQueue.initializeQueue();\r\n\r\n  // Produce the messages indefinitely\r\n  let idx = 1;\r\n  setInterval(async () =\u003e {\r\n    await redisMessageQueue.produce({ data: idx });\r\n    idx += 1;\r\n  }, 1000);\r\n}\r\n\r\nasync function main() {\r\n  await produceMessage();\r\n}\r\n\r\nmain()\r\n```\r\n\r\n## Message Consumer (Blocking Mode) Sample Code\r\n\r\n```javascript\r\n// This is how to consume the messages;\r\nasync function consumeMessage() {\r\n\r\n  // Creating \u0026 Initializing the Redis Message Queue\r\n  const redisMessageQueue = new RedisMessageQueue({\r\n    redisMessageQueueName: 'msgq',\r\n    redisUrl: '',\r\n    redisPort: 6379,\r\n    failOverQueueHandling: false,\r\n  });\r\n  await redisMessageQueue.initializeQueue();\r\n\r\n  // Consume the messages indefinitely in blocking mode.\r\n  redisMessageQueue.consume({\r\n    consumerName: 'qconsumer',\r\n    callback: function(value) {\r\n      console.log(`The value is ${value}`);\r\n    },\r\n    count: 2,\r\n  })\r\n}\r\n\r\nasync function main() {\r\n  await consumeMessage();\r\n}\r\n\r\nmain()\r\n```\r\n\r\n## Why Choose Redis Streams?\r\n\r\nWhen selecting a message queue, several factors must be considered, including:\r\n\r\n- **Propagation:** Ensuring messages are disseminated effectively.\r\n- **Delivery:** Guaranteeing reliable message delivery.\r\n- **Persistence:** Maintaining message durability for fault tolerance.\r\n- **Consumer Groups:** While implemented but not currently utilized, Consumer Groups can enable event-driven microservice architectures alongside message queue functionality.\r\n\r\n## Propagation\r\n\r\nPropagation refers to how messages are transferred by the message queue. There are two types of propagation:  \r\n\r\n1. **One-to-One Propagation**  \r\n   In this type, the producer sends a message to the queue, and the message is received by only one consumer. This approach ensures that each message is processed by a single recipient.\r\n\r\n2. **One-to-Many Propagation (Fan-out)**  \r\n   In this type, a single message from the producer can be delivered to multiple consumers. Although the producer sends only one message, it is propagated to many receivers. This behavior is commonly referred to as \"fan-out.\"  \r\n\r\nThe distinction between these two types lies in how the message is distributed, which depends on the use case and the underlying architecture of the system.\r\n\r\n![Message Propagation Methods](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F5fwrwguwgirwbrava543.png \"Message Propagation Methods\")\r\n\r\n\r\n## Delivery\r\n\r\nDelivery is a critical aspect of message queuing systems, and most systems provide specific delivery guarantees. The three most common delivery guarantees are:\r\n\r\n1. **At-Most-Once**  \r\n   This guarantee ensures that a message is delivered at most one time. It is relatively easy to achieve and is supported by all queuing systems. In this case, the consumer may either receive the message or not receive it at all. This behavior can occur under several circumstances:  \r\n   - The message is lost due to network issues.  \r\n   - The consumer receives the message but fails to process it properly, such as when it crashes.  \r\n   If the message is lost, it cannot be retrieved again, making this the least reliable guarantee.\r\n\r\n2. **At-Least-Once**  \r\n   This guarantee ensures that every message is processed, but it may result in a message being processed multiple times. Well-known systems such as RabbitMQ and Kafka often provide this level of guarantee. For example, if a consumer does not acknowledge the queue that the message has been processed, the queue will resend the message. While this approach ensures message delivery, it requires additional handling to manage duplicate messages.\r\n\r\n3. **Exactly-Once**  \r\n   This is the strictest and most reliable delivery guarantee. It ensures that each message is processed exactly once. However, achieving this guarantee is challenging and often comes with a trade-off in performance. Even popular systems like RabbitMQ struggle to implement this guarantee consistently. Kafka, on the other hand, can achieve exactly-once delivery with proper configuration and usage, though it requires sacrificing some performance.  \r\n\r\nEach delivery guarantee serves different use cases, and the choice depends on the application's requirements and tolerance for duplication or loss.\r\n\r\n## Persistence  \r\n\r\nPersistence refers to whether a message remains in the system after being sent. There are three main types of persistence:  \r\n\r\n1. **In-Memory**  \r\n   In this type, messages are stored in memory only, making access and processing extremely fast. However, the drawback is that messages are lost if the system crashes or restarts, as there is no backup on disk.\r\n\r\n2. **In-Disk**  \r\n   Here, messages are written directly to disk, ensuring durability even in the event of a system failure. Contrary to common belief, persisting messages on disk does not always result in slower performance. The implementation plays a critical role. For instance:  \r\n   - **Kafka** utilizes a Log-Structured Merge Tree (LSM-tree) to achieve high throughput, offering better performance than systems like RabbitMQ that primarily rely on memory.  \r\n   - Similarly, **Cassandra**, known for its rapid write speeds, also uses LSM-tree for efficient disk operations.\r\n\r\n3. **Hybrid**  \r\n   The hybrid approach combines the strengths of in-memory and in-disk persistence. To optimize write performance, messages are initially written to memory and then flushed to disk. **RabbitMQ** is a well-known example of a hybrid system, as it writes messages to memory first for speed but eventually stores them on disk for durability. Notably, RabbitMQ can also be configured to operate entirely as an in-disk system if required.  \r\n\r\nThe choice of persistence type depends on the system's requirements for performance, durability, and fault tolerance, as well as the specific implementation of the queuing system.\r\n\r\n## Consumer Groups  \r\n\r\nIn my opinion, consumer groups are the most crucial feature of a queuing system. Processing messages often takes time, which necessitates the use of multiple consumers to handle them, enabling what is commonly referred to as **scale-out**.  \r\n\r\nIn scenarios involving consumer groups, the targets for both **one-to-one** and **one-to-many** propagation change from a single consumer to a group of consumers.  \r\n\r\n- **One-to-One with Consumer Groups:** A single message is consumed by only one member of the group. This ensures load balancing, as each message is processed by a different consumer within the group.  \r\n- **One-to-Many with Consumer Groups:** All members of the group receive the same message, enabling parallel processing of the same data by multiple consumers.  \r\n\r\nThis approach allows systems to distribute workloads efficiently and scale horizontally, ensuring higher throughput and better resource utilization.\r\n\r\n![Consumer Groups](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ftqt3ba1af71mniqcmdf3.png \"Consumer Groups\")\r\n\r\n## Redis Queue\r\n\r\nAfter talking about the properties in a queuing system, let's talk about how Redis be a message queue. There are 3 ways to do it,\r\n\r\n- Pub/Sub\r\n- List\r\n- Stream\r\n  \r\nWe will introduce one by one, and then give a comprehensive summary.\r\n\r\n## Pub/Sub  \r\n\r\n![Pub Sub](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fujad5nqivupxwkhsf6x6.png \"Pub Sub\")\r\n\r\n**Pub/Sub** (Publish/Subscribe) is a well-known messaging pattern for notification systems. This feature emerged almost simultaneously with Redis. In this model:  \r\n- Consumers **SUBSCRIBE** to a topic (a key in Redis).  \r\n- Producers **PUBLISH** messages to the same topic.  \r\n- The subscribed consumers receive the data in real-time.  \r\n\r\nAs a traditional Pub/Sub feature, Redis Pub/Sub also supports **fan-out**, allowing a single message to be delivered to multiple consumers. Additionally, a certain degree of message routing can be achieved using the **PSUBSCRIBE** command, which enables pattern-based subscriptions.  \r\n\r\n## Limitations of Redis Pub/Sub  \r\nDespite its simplicity, Redis Pub/Sub is not widely adopted for most use cases due to its inherent limitations:  \r\n1. **At-Most-Once Delivery:**  \r\n   - Messages are delivered only if the consumer is active and ready to receive them at the time of publishing.  \r\n   - If the consumer is offline or unavailable, the message is lost.  \r\n\r\n2. **Lack of Persistence:**  \r\n   - Redis Pub/Sub does not persist messages.  \r\n   - All messages are ephemeral and are lost if Redis shuts down or restarts.  \r\n\r\n3. **No Consumer Groups:**  \r\n   - Redis Pub/Sub does not support consumer groups, which limits its scalability for processing workloads across multiple consumers.  \r\n\r\n### Summary of Redis Pub/Sub Features:  \r\n- **Propagation:** Supports both **1-to-1** and **1-to-many** communication.  \r\n- **Delivery Guarantee:** At-most-once.  \r\n- **Persistence:** None (messages are not stored).  \r\n- **Consumer Groups:** Not supported.  \r\n\r\nThese limitations make Redis Pub/Sub suitable for real-time, transient notification systems but less ideal for scenarios requiring message durability or robust delivery guarantees.\r\n\r\n## List  \r\n\r\n![List](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F8a7czjtd4ikvktir13oc.png \"List\")\r\n\r\nThe **List** data structure in Redis is a versatile tool that can easily implement a FIFO (First-In-First-Out) queue. The key feature of using Redis Lists for queuing is the ability to use the `BLPOP` command, which allows consumers to wait for messages in a **blocking mode**. To prevent indefinite blocking, it is recommended to set a timeout when using `BLPOP`.  \r\n\r\n## Key Features:  \r\n1. **Implicit Consumer Group Formation:**  \r\n   - If multiple consumers simultaneously use `BLPOP` to wait for messages on the same list, they effectively form a consumer group without any explicit configuration.  \r\n   - Each message is delivered to only one consumer in the group, achieving a load-balancing effect.  \r\n\r\n2. **No Fan-Out Capability:**  \r\n   - Redis Lists do not support message fan-out.  \r\n   - Once a message is retrieved by one consumer via `BLPOP`, it is no longer available to others, even if the message is lost by that consumer.  \r\n\r\n3. **Message Persistence:**  \r\n   - Messages in a Redis List are persisted in memory.  \r\n   - If **Append-Only File (AOF)** or **Redis Data Backup (RDB)** is enabled, messages can also be backed up to disk.  \r\n   - However, this approach is not entirely reliable for true data persistence, as backups are dependent on periodic snapshots or append operations.  \r\n\r\n## Summary of Redis List Features:  \r\n- **Propagation:** Supports **1-to-1** communication but not **1-to-many**.  \r\n- **Delivery Guarantee:** At-most-once (messages are lost if a consumer fails).  \r\n- **Persistence:** In-memory with optional disk backups using AOF or RDB.  \r\n- **Consumer Groups:** Supported implicitly when multiple consumers use `BLPOP` on the same list.  \r\n\r\nRedis Lists provide a simple and effective way to implement a queue, especially for systems that prioritize simplicity and lightweight operations. However, the lack of fan-out and stronger persistence guarantees may limit their use in more complex messaging scenarios.\r\n\r\n## Stream  \r\n\r\nAfter discussing **Pub/Sub** and **List**, it's evident that neither fully addresses all messaging system requirements due to their respective limitations. Recognizing these shortcomings, **Stream** was introduced in Redis 5.0 to provide a more robust solution.  \r\n\r\nStreams are a more advanced data structure designed to address the issues in **Pub/Sub** and **List**, offering enhanced functionality and reliability.  \r\n\r\n## Key Features and Benefits of Redis Streams:  \r\n\r\n1. **1-to-1 and 1-to-Many Propagation:**  \r\n   - Streams handle both **1-to-1** and **1-to-many** communication scenarios effectively.  \r\n\r\n2. **At-Least-Once Delivery:**  \r\n   - Streams support **at-least-once** delivery, ensuring that every message is processed, even if it means resending the message if the consumer does not acknowledge it.  \r\n\r\n3. **Persistence:**  \r\n   - Messages in Streams are persisted in memory.  \r\n   - With **Append-Only File (AOF)** or **Redis Data Backup (RDB)** enabled, messages can also be backed up to disk, providing a degree of durability.  \r\n\r\n4. **Consumer Groups:**  \r\n   - Streams natively support **consumer groups**, allowing multiple consumers to collaborate efficiently in processing messages.  \r\n   - This feature makes it easy to scale out processing workloads without requiring complex configurations.  \r\n\r\n![Stream](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Filb8q8g5dzon3k9ys1uh.png \"Stream\")\r\n\r\n## Why Choose Streams?  \r\nStreams solve the issues present in **Pub/Sub** and **List** by offering features like **at-least-once delivery** and native support for **consumer groups**. These enhancements make Streams a powerful and reliable tool for building messaging systems.  \r\n\r\n## Summary of Redis Stream Features:  \r\n- **Propagation:** Supports both **1-to-1** and **1-to-many** scenarios.  \r\n- **Delivery Guarantee:** At-least-once.  \r\n- **Persistence:** In-memory with optional disk backups via AOF or RDB.  \r\n- **Consumer Groups:** Fully supported.  \r\n\r\nRedis Streams address the limitations of previous methods and provide a more feature-rich solution for modern messaging systems.\r\n\r\n## Stream and Consumer Groups  \r\n\r\nIn addition to supporting **one-to-one** mapping, Redis Streams also natively support **consumer groups**, enabling efficient workload distribution among multiple consumers.  \r\n\r\n## Consumer Group Functionality  \r\n\r\nWith consumer groups, multiple consumers can collaborate to process messages from the same stream. Each consumer in the group is assigned specific messages, ensuring no duplication of work.  \r\n\r\n![Streams Consumer Groups](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Febx3ukbbc59u2yrhshl6.png \"Streams Consumer Groups\")\r\n\r\n## Achieving At-Least-Once Delivery  \r\n\r\nTo ensure the **at-least-once** delivery guarantee, Redis Streams require consumers to explicitly acknowledge messages after processing them. This is done using the `XACK` command. If a message is not acknowledged, Redis can reassign it to another consumer, ensuring that the message is eventually processed.  \r\n\r\n## Advantages of Stream with Consumer Groups  \r\n- **Scalability:** Multiple consumers can process messages concurrently.  \r\n- **Reliability:** Messages are not lost, as unacknowledged messages can be reassigned.  \r\n- **Efficiency:** Ensures proper load balancing across the group.  \r\n\r\nRedis Streams, with their consumer group support and acknowledgment mechanisms, provide a robust and reliable foundation for scalable and fault-tolerant messaging systems.\r\n\r\n## Contributing\r\n\r\nContributions are welcome! Feel free to open issues or pull requests for improvements or bug fixes.\r\n\r\n## License\r\n\r\nThis project is licensed under the MIT License - see the [LICENSE](https://github.com/Sparsh-Kumar/backtesting.py/blob/main/LICENSE) file for details\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsparsh-kumar%2Fredis-message-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsparsh-kumar%2Fredis-message-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsparsh-kumar%2Fredis-message-queue/lists"}