{"id":23328809,"url":"https://github.com/rabbitmq-community/rstream","last_synced_at":"2025-04-13T00:46:47.691Z","repository":{"id":40382547,"uuid":"395460515","full_name":"rabbitmq-community/rstream","owner":"rabbitmq-community","description":"A Python asyncio-based client for RabbitMQ Streams","archived":false,"fork":false,"pushed_at":"2024-12-10T12:45:11.000Z","size":429,"stargazers_count":87,"open_issues_count":12,"forks_count":13,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-12-21T02:56:29.781Z","etag":null,"topics":["asyncio","python","rabbitmq","rabbitmq-client","rabbitmq-streams"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rabbitmq-community.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-12T22:41:48.000Z","updated_at":"2024-12-14T02:15:45.000Z","dependencies_parsed_at":"2024-02-13T09:27:11.465Z","dependency_job_id":"c9013f72-692e-4976-b9b8-1d3d5e8e1821","html_url":"https://github.com/rabbitmq-community/rstream","commit_stats":{"total_commits":91,"total_committers":6,"mean_commits":"15.166666666666666","dds":0.5604395604395604,"last_synced_commit":"6038781f26e260f9625b2abe077bf0a5881a5b2d"},"previous_names":["rabbitmq-community/rstream","qweeze/rstream"],"tags_count":35,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rabbitmq-community%2Frstream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rabbitmq-community%2Frstream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rabbitmq-community%2Frstream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rabbitmq-community%2Frstream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rabbitmq-community","download_url":"https://codeload.github.com/rabbitmq-community/rstream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248650434,"owners_count":21139672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asyncio","python","rabbitmq","rabbitmq-client","rabbitmq-streams"],"created_at":"2024-12-20T21:11:23.452Z","updated_at":"2025-04-13T00:46:47.661Z","avatar_url":"https://github.com/rabbitmq-community.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RabbitMQ Stream Python Client\n\nA Python asyncio-based client for [RabbitMQ Streams](https://www.rabbitmq.com/stream.html)\n\n\nThe RabbitMQ stream plug-in is required. See the [documentation](https://www.rabbitmq.com/stream.html#enabling-plugin) for enabling it.\n\n\n# Table of Contents\n\n\n- [Installation](#installation)\n- [Examples](#examples)\n- [Client Codecs](#client-codecs)\n    * [AMQP 1.0 codec  vs Binary](#amqp-10-codec--vs-binary)\n- [Publishing messages](#publishing-messages)\n    * [Publishing with confirmation](#publishing-with-confirmation)\n- [Sub-Entry Batching and Compression](#sub-entry-batching-and-compression)\n- [Deduplication](#deduplication)\n- [Consuming messages](#consuming-messages)\n    * [Server-side offset tracking](#server-side-offset-tracking)\n- [Super Streams](#superstreams)\n- [Single Active Consumer](#single-active-consumer)\n- [Connecting with SSL](#connecting-with-ssl)\n- [Sasl Mechanisms](#sasl-mechanisms)\n- [Managing disconnections](#managing-disconnections)\n\t* [Reconnect](#reconnect)\n- [Load Balancer](#load-balancer)\n- [Client Performances](#client-performances)\n   * [Test case](#test-case)\n- [Build and Test](#build-and-test)\n- [Project Notes](#project-notes)\n\n\n## Installation\n\nThe RabbitMQ stream plug-in is required. See the [documentation](https://www.rabbitmq.com/stream.html#enabling-plugin) for enabling it.\n\n\nThe client is distributed via [`PIP`](https://pypi.org/project/rstream/):\n```bash\n pip install rstream\n```\n\n## Examples\n\n[Here](https://github.com/qweeze/rstream/blob/master/docs/examples/) you can find different examples.\n\n\n## Client Codecs\nBefore start using the client is important to read this section.\nThe client supports two codecs to store the messages to the server:\n - `AMQP 1.0`\n - `Binary`\n\nBy default you should use `AMQP 1.0` codec:\n```python\n   amqp_message = AMQPMessage(\n    body=bytes(\"hello: {}\".format(i), \"utf-8\"),\n  )\n```\n \n#### AMQP 1.0 codec  vs Binary\n\nYou need to use the `AMQP 1.0` codec to exchange messages with other stream clients like\n[Java](https://github.com/rabbitmq/rabbitmq-stream-java-client), [.NET](https://github.com/rabbitmq/rabbitmq-stream-dotnet-client), [Rust](https://github.com/rabbitmq/rabbitmq-stream-rust-client), [Go](https://github.com/rabbitmq/rabbitmq-stream-go-client) or if you want to use the `AMQP 0.9.1` clients. \n\nYou can use the `Binary` version if you need to exchange messages from Python to Python. \n\n\u003cb\u003eNote\u003c/b\u003e: The messages stored in `Binary` are not compatible with the other clients and with AMQP 0.9.1 clients. \u003cbr /\u003e \nOnce the messages are stored to the server, you can't change them. \n\nRead also the [Client Performances](#client-performances) section \n\n\n## Publishing messages\n\nYou can publish messages with four different methods:\n\n* `send`: asynchronous, messages are automatically buffered internally and sent at once after a timeout expires.\n* `send_batch`: synchronous, the user buffers the messages and sends them. This is the fastest publishing method.\n* `send_wait`: synchronous, the caller wait till the message is confirmed. This is the slowest publishing method.\n* `send_sub_entry`: asynchronous. See [Sub-entry batching and compression](#sub-entry-batching-and-compression).\n\nOn the [examples](https://github.com/qweeze/rstream/blob/master/docs/examples/) directory you can find diffent way to send the messages:\n- [producer using send](https://github.com/qweeze/rstream/blob/master/docs/examples/basic_producers/producer_send.py)\n- [producer using send_wait](https://github.com/qweeze/rstream/blob/master/docs/examples/basic_producers/producer_send_wait.py)\n- [producer using send_batch](https://github.com/qweeze/rstream/blob/master/docs/examples/basic_producers/producer_send_batch.py)\n- [producer using sub_entry_batch](https://github.com/qweeze/rstream/blob/master/docs/examples/sub_entry_batch/producer_sub_entry_batch.py)\n\n\n### Publishing with confirmation\n\nThe Send method takes as parameter an handle function that will be called asynchronously when the message sent will be notified from the server to have been published.\n\nExample:\n- [producer using send and handling confirmation](https://github.com/qweeze/rstream/blob/master/docs/examples/producers_with_confirmations/send_with_confirmation.py)\n- [producer using send_batch and handling confirmation](https://github.com/qweeze/rstream/blob/master/docs/examples/producers_with_confirmations/send_batch_with_confirmation.py)\n\nWith `send_wait` instead will wait until the confirmation from the server is received.\n\n\n## Sub-Entry Batching and Compression\nRabbitMQ Stream provides a special mode to publish, store, and dispatch messages: sub-entry batching. This mode increases throughput at the cost of increased latency and potential duplicated messages even when deduplication is enabled. It also allows using compression to reduce bandwidth and storage if messages are reasonably similar, at the cost of increasing CPU usage on the client side.\n\nSub-entry batching consists in squeezing several messages – a batch – in the slot that is usually used for one message. This means outbound messages are not only batched in publishing frames, but in sub-entries as well.\n\n```python\n\n  # sending with compression\n   await producer.send_sub_entry(\n        STREAM, compression_type=CompressionType.Gzip, sub_entry_messages=messages\n   )\n```\n[Full example producer using sub-entry batch](https://github.com/qweeze/rstream/blob/master/docs/examples/sub_entry_batch/producer_sub_entry_batch.py)\n\nConsumer side is automatic, so no need configurations. \n\nThe client is shipped with No Compression (`CompressionType.No`) and Gzip Compression (`CompressionType.Gzip`) the other compressions (`Snappy`, `Lz4`, `Zstd`) can be used implementing the `ICompressionCodec` class. \n\n\n\n## Deduplication\n\nRabbitMQ Stream can detect and filter out duplicated messages, based on 2 client-side elements: the producer name and the message publishing ID.\nAll the producer methods to send messages (send, send_batch, send_wait) takes a publisher_name parameter while the message publishing id can be set in the AMQP message.\n\nExample:\n- [producer with deduplication](https://github.com/qweeze/rstream/blob/master/docs/examples/deduplication/producer_ded.py)\n\n## Consuming messages\n\nSee [consumer examples](https://github.com/qweeze/rstream/blob/master/docs/examples/basic_consumers)  for basic consumer and consumers with different offsets.\n\n\n### Server-side offset tracking\n\nRabbitMQ Streams provides server-side offset tracking for consumers. This features allows a consuming application to restart consuming where it left off in a previous run.\nYou can use the store_offset (to store an offset in the server) and query_offset (to query it) methods of the consumer class like in this example:\n- [server side offset tracking](https://github.com/qweeze/rstream/blob/master/docs/examples/manual_server_offset_tracking/consumer.py)\n\n\n## Superstreams\n\nA super stream is a logical stream made of individual, regular streams. It is a way to scale out publishing and consuming with RabbitMQ Streams: a large logical stream is divided into partition streams, splitting up the storage and the traffic on several cluster nodes.\n\nSee the [blog post](https://blog.rabbitmq.com/posts/2022/07/rabbitmq-3-11-feature-preview-super-streams/) for more info.\n\nYou can use `superstream_producer` and `superstream_consumer` classes which internally uses producers and consumers to operate on the componsing streams.\n\n\nSee the [Super Stream example](https://github.com/qweeze/rstream/tree/master/docs/examples/super_stream)\n\n### Single Active Consumer\n\nSingle active consumer provides exclusive consumption and consumption continuity on a stream. \u003cbr /\u003e \nSee the [blog post](https://blog.rabbitmq.com/posts/2022/07/rabbitmq-3-11-feature-preview-single-active-consumer-for-streams) for more info.\nSee examples in: \n\nSee the [single active consumer example](https://github.com/qweeze/rstream/blob/master/docs/examples/single_active_consumer/)\n\n### Filtering\n\nFiltering is a new streaming feature enabled from RabbitMQ 3.13 based on Bloom filter.\nRabbitMQ Stream provides a server-side filtering feature that avoids reading all the messages of a stream and filtering \nonly on the client side. This helps to save network bandwidth when a consuming application needs only a subset of \nmessages.\n\nhttps://rabbitmq.github.io/rabbitmq-stream-java-client/stable/htmlsingle/#filtering \n\nSee the [filtering examples](https://github.com/qweeze/rstream/blob/master/docs/examples/filtering/)\n\n### Connecting with SSL\n\nYou can enable ssl/tls.\nSee example here:\n[tls example](https://github.com/qweeze/rstream/blob/master/docs/examples/tls/producer.py)\n\n### Sasl Mechanisms\n\nYou can use the following sasl mechanisms:\n- PLAIN\n- EXTERNAL\n\nThe client uses `PLAIN` mechanism by default.\n\nThe `EXTERNAL` mechanism is used to authenticate a user based on a certificate presented by the client.\nExample:\n```python\n    ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)\n    # put the root certificate of the ca\n    ssl_context.load_verify_locations(\"certs/ca_certificate.pem\")\n    ssl_context.load_cert_chain(\n        \"certs/client_HOSTNAME_certificate.pem\",\n        \"certs/client_HOSTNAME_key.pem\",\n    )\n\n    async with Producer(\n        \"HOSTNAME\",\n        username=\"not_important\",\n        password=\"not_important\",\n        port=5551,\n        ssl_context=ssl_context,\n        sasl_configuration_mechanism=SlasMechanism.MechanismExternal ## \u003c--- here EXTERNAL configuration\n```\nThe plugin `rabbitmq_auth_mechanism_ssl` needs to be enabled on the server side, and `ssl_options.fail_if_no_peer_cert` needs to set to `true`\nconfig example:\n```\nauth_mechanisms.3 = PLAIN\nauth_mechanisms.2 = AMQPLAIN\nauth_mechanisms.1 = EXTERNAL\n\nssl_options.cacertfile = certs/ca_certificate.pem\nssl_options.certfile = certs/server_certificate.pem\nssl_options.keyfile = certs/server_key.pem\nlisteners.ssl.default = 5671\nstream.listeners.ssl.default = 5551\nssl_options.verify               = verify_peer\nssl_options.fail_if_no_peer_cert = true\n```\n\n### Managing disconnections\n\nThe client supports auto-reconnect just for Producer and SuperstreamProducer at the moment.\n\nWhen the TCP connection is disconnected unexpectedly, the Producer and the SuperstreamProducer will try to automatically\nreconnect while in case of the Consumer/SuperstreamConsumer the client raises an event that needs to be managed:\n\n```python\nasync def on_connection_closed(disconnection_info: DisconnectionErrorInfo) -\u003e None:\n    print(\n        \"connection has been closed from stream: \"\n        + str(disconnection_info.streams)\n        + \" for reason: \"\n        + str(disconnection_info.reason)\n    )\n\nconsumer = Consumer(\n..        \non_close_handler=on_connection_closed,\n)\n```\n\n### Reconnect\nWhen the `on_close_handler` event is raised, you can close the Consumers by doing a correct clean-up or try reconnecting using the reconnect stream.\n\nExample:\n```python\n async def on_connection_closed(disconnection_info: OnClosedErrorInfo) -\u003e None:\n        print(\n            \"connection has been closed from stream: \"\n            + str(disconnection_info.streams)\n            + \" for reason: \"\n            + str(disconnection_info.reason)\n        )\n\n        for stream in disconnection_info.streams:\n            print(\"reconnecting stream: \" + stream)\n            await producer.reconnect_stream(stream)\n```\n\nPlease take a look at the complete reliable client example [here](https://github.com/qweeze/rstream/blob/master/docs/examples/reliable_client/)\n\n### Metadata Update\n\nIf the streams topology changes (ex:Stream deleted or add/remove follower), The server removes the producers and consumers \nlinked to the stream and then it sends the Metadata update event.\nthe behaviour is similar to what we have for disconnections. In case of the Producer/Superstream Producer \nthe Client will try to automatically reconnect while the Consumer needs to manage the on_close_handler event.\n\nPlease take a look at the complete reliable client example [here](https://github.com/qweeze/rstream/blob/master/docs/examples/reliable_client/)\n\n## Load Balancer\n\nIn order to handle load balancers, you can use the `load_balancer_mode` parameter for producers and consumers. This will always attempt to create a connection via the load balancer, discarding connections that are inappropriate for the client type.\n\nProducers must connect to the leader node, while consumers can connect to any, prioritizing replicas if available.\n\n\n## Client Performances\n\nThe RabbitMQ Stream queues can handle high throughput. Currently, the client cannot reach the maximum throughput the server can handle. \n\nWe found some bottlenecks; one of them is the current AMQP 1.0 marshal and unmarshal message format. \n\nThis one:\n```python\n for i in range(1_000_000):\n            amqp_message = AMQPMessage(\n                body=bytes(\"hello: {}\".format(i), \"utf-8\"),\n            )\n            # send is asynchronous\n            await producer.send(stream=STREAM, message=amqp_message)\n```\n\nis more or less ~55% slower than:\n```python\n for i in range(1_000_000):\n            # send is asynchronous\n            await producer.send(stream=STREAM, message=b\"hello\")\n```\n\nYou can use the `batch_send` to test the performances.\n\nWe are evaluating rewriting the `AMQP 1.0 codec` optimized for the stream use case.\n\n### Test case\n- Linux Ubuntu 4 cores and 8 GB of Ram\n- RabbitMQ installed to the server \n\n- Send batch with AMQP 1.0 codec:\n```python\n$  python3 docs/examples/basic_producers/producer_send_batch.py\nSent 1.000.000 messages in 9.3218 seconds. 107.275,5970 messages per second\n```\n\n- Send batch with binary codec:\n```python\n$ python3 docs/examples/basic_producers/producer_send_batch_binary.py\nSent 1.000.000 messages in 2.9930 seconds. 334.116,5639 messages per second\n```\n\n\n## Build and Test\n\nTo run the tests, you need to have a running RabbitMQ Stream server. \nYou can use the docker official image.\n\nRun the server with the following command:\n```bash\ndocker run -it --rm --name rabbitmq -p 5552:5552 -p 5672:5672 -p 15672:15672 \\\n    -e RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS='-rabbitmq_stream advertised_host localhost' \\\n    rabbitmq:3.13.1-management\n```\n\nenable the plugin:\n```bash\ndocker exec rabbitmq rabbitmq-plugins enable rabbitmq_stream rabbitmq_stream_management rabbitmq_amqp1_0\n```\n\nand run the tests:\n```bash\n poetry run pytest\n```\n\n\n## Project Notes\nThe project is in development and stabilization phase. Features and API are subject to change, but breaking changes will be kept to a minimum. \u003c/br\u003e\nAny feedback or contribution is welcome\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frabbitmq-community%2Frstream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frabbitmq-community%2Frstream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frabbitmq-community%2Frstream/lists"}