{"id":15583440,"url":"https://github.com/davidmarquis/redisq","last_synced_at":"2025-04-24T03:44:56.433Z","repository":{"id":138625958,"uuid":"73480784","full_name":"davidmarquis/redisq","owner":"davidmarquis","description":"RedisQ - Java implementation of a reliable Pub/Sub message queue based on Redis","archived":false,"fork":false,"pushed_at":"2017-09-04T16:33:36.000Z","size":57,"stargazers_count":107,"open_issues_count":0,"forks_count":37,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-04-24T03:44:46.796Z","etag":null,"topics":["consumer","java","producer","queue","redis","reliable"],"latest_commit_sha":null,"homepage":"http://blog.radiant3.ca/2013/01/03/reliable-delivery-message-queues-with-redis/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidmarquis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-11-11T13:41:59.000Z","updated_at":"2025-03-18T00:19:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"d35286b1-31ec-4f10-8f1d-fa6741aadedf","html_url":"https://github.com/davidmarquis/redisq","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmarquis%2Fredisq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmarquis%2Fredisq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmarquis%2Fredisq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidmarquis%2Fredisq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidmarquis","download_url":"https://codeload.github.com/davidmarquis/redisq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250559813,"owners_count":21450168,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consumer","java","producer","queue","redis","reliable"],"created_at":"2024-10-02T20:08:19.594Z","updated_at":"2025-04-24T03:44:56.408Z","avatar_url":"https://github.com/davidmarquis.png","language":"Java","funding_links":[],"categories":["进程间通信"],"sub_categories":["Spring Cloud框架"],"readme":"RedisQ\n======\n\nA Java library for Reliable Delivery Pub/Sub over Redis\n\nWhat is this?\n-------------\n\nRedisQ is a Java implementation of a distributed message queue that uses Redis as a backend. It has the following features:\n\n - **Multiple consumers per message queue**: Each queue can have multiple consumer clients sending messages at their own rate.\n - **Single or multi-threaded consumers**: Each consumer can be either single threaded or multi threaded.\n - **Distributed processing**: Multiple clients/processes/nodes can consume messages from a queue in parallel.\n - **Reliable delivery**: Each consumer on a queue will receive each message even if they are temporarily offline.\n - **Pluggable queue/dequeue algorithms**: By default, FIFO is used, but this is pluggable (see below).\n - **Optional sequential delivery**: Consumers can be configured to use a locking mechanism on a queue to make sure each message is delivered in order.\n - **Configurable payload serialization**: Out of the box, JAXB XML, JSON (using Google's GSON) and String serializers are available.\n - **Optional pluggable retry strategies**: By default, consumers do not retry consumption upon error. A pluggable mechanism exists to enable retry schemes when consuming messages.\n - **High performance**: Hey, it's Redis!\n\nWhy not use Redis Pub/Sub?\n--------------------------\n\nThis implementation does not use the Pub/Sub functionality offered by Redis. \n\nWhy not use the Pub/Sub semantics supported out-of-the-box? The reason is two fold:\n\n - What Redis offers with Pub/Sub is a listener model, where each subscriber receives each messages when it is listening, but won't receive them when not connected. We want every consumer to eventually receive all messages, independently of their online or offline status.\n - In a clustered environment where you have multiple instances of your consumer component (application) running at the same time, each instance would receive each message produced on the channel. This library makes sure any given message is consumed once per logical consumer, even when multiple instances of this component are running.\n \nHence the name \"Reliable Delivery\", because we want to make sure every logical consumer eventually receives all messages produced on a queue once and only once, even when those consumers are not connected - i.e. due to a deployment, a restart or a application failure/crash.\n\nHigh level concepts\n-------------------\n\n#### Message queue\n\nThe core concept of RedisQ is the queue itself. A queue has a name, and that's pretty much it. Messages can be published and consumed from a MessageQueue. The `MessageQueue` interface also provides some read only  meta information, i.e. getting the list of registered consumers on the queue, the number of messages in the queue for each consumer, etc.\n\n#### Message\n\nA `Message` is the entity that gets published and consumed on the queue. A `Message` instance provides some meta information about your actual message, along with its _payload_. The payload is the actual content that your application publishes and consumes. Within Redis, each individual message is stored as a Hash containing all of the message attributes. Each attribute in the hash is stored as strings, including the payload. For this reason, a (configurable) serialization mechanism exists. More on that later.\n\n#### Message producer\n\nThe `MessageProducer` is the side of the system that publishes messages on a queue for consumption by consumers. Multiple producers can exist for the same logical queue.\n\n#### Message consumer\n\nA message consumer will consume messages from the queue and pass them out to your application using the `MessageListener` that you define.\n\nYou can define an ID for each logical application consuming messages on a queue, and messages submitted to a queue will be distributed independently to each logical consumer. This allows for per-consumer reliable delivery of messages. In practice, a separate Redis List is created and managed for each registered consumer ID as their own queues.\n\nMultiple application instances (processes) can be defined using the same consumer ID for distributed processing of messages - effectively enabling reliable clustering on your application.\n\nUsing consumer IDs is optional. If not defined, a default consumer ID is used (`default`) on both the producer and the consumer side. This is sufficient for simple cases where you're using a single Redis server in a single logical application.\n\nThe class that is used for defining a message consumer is conveniently called `MessageConsumer`.\n\n#### Message listener\n\nOn the consumer side, the `MessageListener` interface represents the link between the queue and your application. Your application must implement the `MessageListener` interface in order to actually consume messages. This interface defines a single `onMessage(Message\u003cT\u003e message)` method that gets called when there's a message available for consumption. This interface is generically typed and the type you define in your implementation actually gets passed as a hint to your configured `PayloadSerializer`. \n\nConfiguring payload serialization\n---------------------------------\n\nBy default, JAXB is used to serialize message payloads that you publish through the `MessageProducer` interface (producer-side) and that you consume through the `MessageListener` interface (consumer side).\n\nTo change this default implementation, you  need to define a bean that implements the `PayloadSerializer` interface in your Spring context.\n\nA few serializers are available out-of-the-box:\n\n - **JaxbPayloadSerializer**: Uses JAXB to serialize your payload objects. Your payload objects *must* be annotated\n using JAXB annotations (such as `@XmlRootElement` and the like). This serializer supports inheritance in your payload\n objects.\n - **GsonPayloadSerializer**: Uses Google's GSON library to serialize your payload objects. This serializer does *not*\n support inheritance in your payload objects, but provides a simple and effective way to serialize simple message objects (note: when using GSON as your serializer, the GSON library must be explictly provided by your project as a dependency).\n - **StringPayloadSerializer**: Expects your payloads to be strings, formatted as needed. Calls `toString()` on your payload objects to serialize them, and the deserialization operation is a pass-through.\n \nAdding a payload serializer is as simple as implementing an interface with a `serialize` and a `deserialize` functions.\n\nMaven dependency\n----------------\n\nThis artifact is published on Maven Central since version 2.0.0:\n\n``` xml\n    \u003cdependency\u003e\n        \u003cgroupId\u003ecom.github.davidmarquis\u003c/groupId\u003e\n        \u003cartifactId\u003eredisq\u003c/artifactId\u003e\n        \u003cversion\u003e2.0.0\u003c/version\u003e\n    \u003c/dependency\u003e\n```\n\nUsage with Spring\n-----------------\n\nNote: the examples below assume you're using Spring's autowiring features.\n\nFirst declare the base beans for Redis connectivity:\n\n``` xml\n    \u003cbean id=\"jedisConnectionFactory\" class=\"org.springframework.data.redis.connection.jedis.JedisConnectionFactory\"\u003e\n        \u003cproperty name=\"hostName\" value=\"localhost\"/\u003e\n        \u003cproperty name=\"port\" value=\"6379\"/\u003e\n    \u003c/bean\u003e\n\n    \u003cbean id=\"redisTemplate\" class=\"org.springframework.data.redis.core.RedisTemplate\"\u003e\n        \u003cproperty name=\"connectionFactory\" ref=\"jedisConnectionFactory\"/\u003e\n        \u003cproperty name=\"keySerializer\"\u003e\n            \u003cbean class=\"org.springframework.data.redis.serializer.StringRedisSerializer\"/\u003e\n        \u003c/property\u003e\n    \u003c/bean\u003e\n\n    \u003cbean id=\"redisOps\" class=\"com.github.davidmarquis.redisq.persistence.RedisOps\" /\u003e\n```\n\nThen declare each queue as a bean of type RedisMessageQueue:\n\n``` xml\n    \u003cbean id=\"myQueue\" class=\"com.github.davidmarquis.redisq.RedisMessageQueue\"\u003e\n        \u003cproperty name=\"queueName\" value=\"my.queue\"/\u003e\n    \u003c/bean\u003e\n```\n\nOnce your queue bean is created, you need to attach a Producer:\n\n``` xml\n    \u003cbean id=\"messageProducer\" class=\"com.github.davidmarquis.redisq.producer.DefaultMessageProducer\"\u003e\n        \u003cproperty name=\"queue\" ref=\"myQueue\"/\u003e\n    \u003c/bean\u003e\n```\n\nand/or a Consumer:\n\n``` xml\n    \u003cbean id=\"messageListener\" class=\"...\"/\u003e\n\n    \u003cbean id=\"messageConsumer\" class=\"com.github.davidmarquis.redisq.consumer.MessageConsumer\"\u003e\n        \u003cproperty name=\"queue\" ref=\"myQueue\" /\u003e\n        \u003cproperty name=\"consumerId\" value=\"someConsumerId\" /\u003e\n        \u003cproperty name=\"messageListener\" ref=\"messageListener\"/\u003e\n    \u003c/bean\u003e\n```\n\nUsually, the Producer and Consumer beans will reside in distinct application and processes, but nothing prevents you from having both a Producer and a Consumer within the same application.\n\nUsing multi-threading on consumers\n----------------------------------\n\nBy default, consumers are using a threading strategy that uses a single thread. Multi-threading is easily configurable using Spring:\n\n``` xml\n    \u003cbean id=\"messageConsumer\" class=\"com.github.davidmarquis.redisq.consumer.MessageConsumer\"\u003e\n        \u003cproperty name=\"queue\" ref=\"myQueue\" /\u003e\n        \u003cproperty name=\"consumerId\" value=\"someConsumerId\" /\u003e\n        \u003cproperty name=\"messageListener\" ref=\"messageListener\"/\u003e\n        \u003cproperty name=\"threadingStrategy\"\u003e\n            \u003cbean class=\"com.github.davidmarquis.redisq.consumer.MultiThreadingStrategy\"\u003e\n                \u003cconstructor-arg name=\"numThreads\" value=\"4\"/\u003e\n            \u003c/bean\u003e\n        \u003c/property\u003e\n    \u003c/bean\u003e\n```\n\nDo note that in both cases (both single and multi-threaded strategies), a minimum of 1 separate thread will always be created by RedisQ for consuming messages.\n\nPublishing messages to a queue\n------------------------------\n\nSample code (once your Spring beans are properly setup as detailed above):\n\n``` java\n    @Autowired\n    private MessageProducer queue;\n\n    ...\n\n    queue.create(new SomePayload(\"with some data\")).submit();\n```\n\nConsuming messages from a queue\n-------------------------------\n\n``` java\n    public class SomePayloadListener implements MessageListener\u003cSomePayload\u003e {\n\n        public void onMessage(Message\u003cSomePayload\u003e message) {\n            SomePayload payload = message.getPayload();\n\n            // do your stuff with the payload...\n        }\n    }\n```\n\nManually starting up consumers\n-------------------------------\n\nBy default, instances of `MessageConsumer` will automatically start consuming messages from their queue when the application starts up. If you want to manually control when the consumers start, set `autoStartConsumers` to `false`  on your consumer instances:\n\n``` xml\n    \u003cbean id=\"messageConsumer\" class=\"com.github.davidmarquis.redisq.consumer.MessageConsumer\"\u003e\n        \u003cproperty name=\"queue\" ref=\"myQueue\" /\u003e\n        \u003cproperty name=\"consumerId\" value=\"someConsumerId\" /\u003e\n        \u003cproperty name=\"messageListener\" ref=\"messageListener\"/\u003e\n        \u003cproperty name=\"autoStartConsumers\" value=\"false\"/\u003e\n    \u003c/bean\u003e\n```\n\nEnabling retries for failed messages\n-------------------------------------\n\nRedisQ does not retry message consumptions when an exception arises during message consumption. You must configure a retry strategy on your consumers in order to enable retries. \n\nMoreover, __your code must explicitly throw a `RetryableMessageException`__ to inform RedisQ that a known consumer error has been identified, that this error is recoverable and thus can be retried.\n\n``` xml\n    \u003cbean id=\"messageConsumer\" class=\"com.github.davidmarquis.redisq.consumer.MessageConsumer\"\u003e\n        \u003cproperty name=\"queue\" ref=\"myQueue\" /\u003e\n        \u003cproperty name=\"consumerId\" value=\"someConsumerId\" /\u003e\n        \u003cproperty name=\"messageListener\" ref=\"messageListener\"/\u003e\n        \u003cproperty name=\"retryStrategy\"\u003e\n            \u003cbean class=\"com.github.davidmarquis.redisq.consumer.retry.MaxRetriesStrategy\"\u003e\n                \u003cconstructor-arg name=\"maxRetries\" value=\"2\"/\u003e\n            \u003c/bean\u003e\n        \u003c/property\u003e\n    \u003c/bean\u003e\n```\n\nTwo `MessageRetryStrategy` implementations are provided out-of-the-box:\n\n- **`NoRetryStrategy`**: (default) Does not attempt any retry of messages that failed.\n- **`MaxRetriesStrategy`**: Will retry message consumption when a `RetryableMessageException` is raised, up to a configurable maximum of times.\n\nChanging the queue/dequeue algorithm for a queue\n------------------------------------------------\n\nBy default, each queue is configured to produce and consume messages as FIFO (First In First Out), but this mechanism can be changed using the `queueDequeueStrategy` attribute on class `RedisMessageQueue`.\n\n``` xml\n    \u003cbean id=\"myQueue\" class=\"com.github.davidmarquis.redisq.RedisMessageQueue\"\u003e\n        \u003cproperty name=\"queueName\" value=\"my.queue\"/\u003e\n        \u003cproperty name=\"queueDequeueStrategy\"\u003e\n            \u003cbean class=\"com.github.davidmarquis.redisq.queuing.FIFOQueueDequeueStrategy\"/\u003e\n        \u003c/property\u003e\n    \u003c/bean\u003e\n```\n\nImplementations bundled in the library (in package `com.github.davidmarquis.redisq.queuing`):\n\n- **FIFOQueueDequeueStrategy**: (default) Messages are submitted to the tail of a Redis List, and are consumed from the head.\n- **RandomQueueDequeueStrategy**: Messages are submitted in a Redis Set, then consumed in random order from that set. To prevent the need for polling, a separate supporting Redis List is used to notify consumers of new items in the Set.\n\nPerformance tip: Disabling \"multi-consumer\" (fan-out) mode\n---------------------------------------------------------\n\nBy default, RedisQ producers will publish messages to all registered consumers (fan-out). If your application's design does not require multiple consumers for a given queue, then you should switch to the \"single\" consumer mode, this will slightly improve performance for producing each message as it removes the need for a lookup that is otherwise required when multiple consumers are used.\n\n``` xml\n    \u003cbean id=\"messageProducer\" class=\"com.github.davidmarquis.redisq.producer.DefaultMessageProducer\"\u003e\n        \u003cproperty name=\"queue\" ref=\"myQueue\"/\u003e\n        \u003cproperty name=\"submissionStrategy\"\u003e\n            \u003cbean class=\"com.github.davidmarquis.redisq.producer.SingleConsumerSubmissionStrategy\"/\u003e\n        \u003c/property\u003e\n    \u003c/bean\u003e\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidmarquis%2Fredisq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidmarquis%2Fredisq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidmarquis%2Fredisq/lists"}