{"id":17929651,"url":"https://github.com/thrau/pymq","last_synced_at":"2025-03-24T04:31:05.613Z","repository":{"id":35074384,"uuid":"203297485","full_name":"thrau/pymq","owner":"thrau","description":"A simple message-oriented middleware library built for Python IPC across machine boundaries","archived":false,"fork":false,"pushed_at":"2022-07-17T19:52:24.000Z","size":137,"stargazers_count":23,"open_issues_count":5,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-18T04:09:14.587Z","etag":null,"topics":["message-oriented-middleware","pubsub","python","queues","redis","rpc"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thrau.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-20T04:06:51.000Z","updated_at":"2024-12-14T09:16:42.000Z","dependencies_parsed_at":"2022-08-06T09:16:27.654Z","dependency_job_id":null,"html_url":"https://github.com/thrau/pymq","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thrau%2Fpymq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thrau%2Fpymq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thrau%2Fpymq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thrau%2Fpymq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thrau","download_url":"https://codeload.github.com/thrau/pymq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245210953,"owners_count":20578321,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["message-oriented-middleware","pubsub","python","queues","redis","rpc"],"created_at":"2024-10-28T21:09:57.553Z","updated_at":"2025-03-24T04:31:05.332Z","avatar_url":"https://github.com/thrau.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"PyMQ\n====\n\n[![Build Status](https://github.com/thrau/pymq/actions/workflows/test.yml/badge.svg)](https://github.com/thrau/pymq/actions/workflows/test.yml)\n[![PyPI Version](https://badge.fury.io/py/pymq.svg)](https://badge.fury.io/py/pymq)\n[![PyPI - Downloads](https://img.shields.io/pypi/dm/pymq)](https://pypistats.org/packages/pymq)\n[![PyPI License](https://img.shields.io/pypi/l/pymq.svg)](https://img.shields.io/pypi/l/pymq.svg)\n[![Coverage Status](https://coveralls.io/repos/github/thrau/pymq/badge.svg?branch=master)](https://coveralls.io/github/thrau/pymq?branch=master)\n[![Codestyle](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\nPyMQ is a simple message-oriented middleware library for implementing Python IPC across machine boundaries. The API\nenables different styles of remoting via Pub/Sub, Queues, and synchronous RPC.\n\nWith PyMQ, developers can integrate Python applications running on different machines in a loosely coupled way over\nexisting transport mechanisms.\nPyMQ currently provides\na Redis backend,\nan AWS backend using SQS and SNS,\na POSIX IPC backend for single-machine IPC, and\nan in-memory backend for testing.\nThe API is extensible and other transports can be plugged in.\n\nUsing PyMQ\n----------\n\n### Install\n\nStarting from pymq 0.5, all providers are separated into setuptools extras.\nIf you want to use the redis backend, then run\n\n    pip install pymq[redis]\n\nAvailable providers:\n\n* `pymq[redis]` Redis provider\n* `pymq[aws]` AWS provider using SNS and SQS\n* `pymq[ipc]` Linux IPC provider\n* `pymq[full]` install all providers\n\n### Initialize PyMQ\n\nThe core module manages a global eventbus instance that provides the remoting primitives. The default Redis\nimplementation uses an event loop over a pubsub object. The global eventbus is initialized via `pymq.init` and by\npassing a provider factory.\n\n```python\nimport pymq\nfrom pymq.provider.redis import RedisConfig\n\n# starts a new thread with a Redis event loop\npymq.init(RedisConfig())\n\n# main application control loop\n\npymq.shutdown()\n```\nThis will create an eventbus instance on top of a local Redis server.\n\n### Pub/Sub\n\nPub/Sub allows asynchronous event-based communication. Event classes are used to transport state and identify channels.\n\n```python\nimport pymq\n\n# common code\nclass MyEvent:\n    pass\n\n# subscribe code\n@pymq.subscriber\ndef on_event(event: MyEvent):\n    print('event received')\n\n# publisher code\npymq.publish(MyEvent())\n```\n\n### Queues\n\nQueues are straight forward, as they are compatible to the Python `queue.Queue` specification.\n\n```python\nimport pymq\n\nqueue = pymq.queue('my_queue') \nqueue.put('obj')\nprint(queue.get()) # outputs 'obj'\n```\n\n### RPC\n\nServer code\n\n```python\nimport pymq\n\n@pymq.remote('product_remote')\ndef product(a: int, b: int) -\u003e int: # pymq relies on type hints for marshalling\n    return a * b\n```\n\nClient code\n```python\nimport pymq\n\nproduct = pymq.stub('product_remote')\nproduct(2, 4) # 8\n```\n\nWith a shared code-base, methods can also be exposed and called by passing the callable. For example,\n```python\nimport pymq\n\n# common code\nclass Remote:\n    def echo(self, param) -\u003e None:\n        return 'echo: ' + param\n\n# server\nobj = Remote()\npymq.expose(obj.echo)\n\n# client\necho = pymq.stub(Remote.echo)\necho('pymq') # \"echo: pymq\"\n```\n\nIf there are multiple providers of the same object, then a stub can be initialized with `multi=True` to get a list of\nresults. It may be useful to use a timeout in this case.\n\n```python\nremote = pymq.stub('remote_method', multi=True, timeout=2)\n\nresult = remote() # result will be a list containing the results of all invocations of available remote objects\n```\n\nProviders\n---------\n\n* `SimpleEventBus` used for testing and rudimentary single-thread dispatching\n* `RedisEventBus` works across network and process boundaries but requires a running redis instance\n* `AwsEventBus` uses AWS Simple Notification Service (SNS) and Simple Queuing Service (SQS)\n* `IpcEventBus` uses `posix_ipc` message queues as event loops and maintains a tree of topic subscriptions in\n  `/run/shm`. Useful for eventing across process boundaries without an additional server component.\n\nCompatibility\n-------------\n\nPython 3.8+\n\nKnown Limitations\n-----------------\n\n* JSON serialization relies heavily on type hints. Sending complex types without type hints will cause type errors.\n* There is currently no support for polymorphism with JSON serialization\n* Pattern-based topic matching does not work for the in-memory eventbus or the IPC event bus\n* You can only have a limited number of Queues when using the IPC provider, as the kernel limits the number of file\n  descriptors per process\n* Subscriptions by foreign components to RPC channels will cause issues in multi-call scenarios\n* Using the `pymq` singleton in multiprocessing scenarios may not work as expected because the module holds a Thread in\n  a global variable. A workaround is to re-start the bus by calling `shutdown()` and `init()` in the forked Process.\n* IPC provider only works for Linux\n* Multi-RPC does not work for the AWS provider\n* AWS provider cannot automatically delete all queues and topics.\n\nBackground\n----------\n\nOriginally part of the [Symmetry](https://dsg.tuwien.ac.at/team/trausch/pub/sec19-demo-trausch.pdf) project, was extracted as a standalone\nlibrary.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthrau%2Fpymq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthrau%2Fpymq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthrau%2Fpymq/lists"}