{"id":35812152,"url":"https://github.com/danionella/dejaq","last_synced_at":"2026-01-26T09:04:04.230Z","repository":{"id":255441789,"uuid":"851458684","full_name":"danionella/dejaq","owner":"danionella","description":"DejaQueue – a fast alternative to multiprocessing.Queue","archived":false,"fork":false,"pushed_at":"2026-01-23T16:42:21.000Z","size":233,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-01-24T06:43:54.531Z","etag":null,"topics":["multiprocessing","numpy-arrays","parallel","queues","shared-memory"],"latest_commit_sha":null,"homepage":"https://danionella.github.io/dejaq/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danionella.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-09-03T06:18:48.000Z","updated_at":"2026-01-23T16:42:25.000Z","dependencies_parsed_at":"2025-02-04T15:28:16.566Z","dependency_job_id":"023a4766-43d2-4313-9850-0c8e5a81a9fc","html_url":"https://github.com/danionella/dejaq","commit_stats":null,"previous_names":["danionella/dejaq"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/danionella/dejaq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danionella%2Fdejaq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danionella%2Fdejaq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danionella%2Fdejaq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danionella%2Fdejaq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danionella","download_url":"https://codeload.github.com/danionella/dejaq/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danionella%2Fdejaq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28771542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T08:38:24.014Z","status":"ssl_error","status_checked_at":"2026-01-26T08:38:22.080Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["multiprocessing","numpy-arrays","parallel","queues","shared-memory"],"created_at":"2026-01-07T15:13:53.706Z","updated_at":"2026-01-26T09:04:04.224Z","avatar_url":"https://github.com/danionella.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Python Version](https://img.shields.io/badge/python-3.8+-blue)\n[![PyPI - Version](https://img.shields.io/pypi/v/dejaq)](https://pypi.org/project/dejaq/)\n[![Conda Version](https://img.shields.io/conda/v/conda-forge/dejaq)](https://anaconda.org/conda-forge/dejaq)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n![GitHub last commit](https://img.shields.io/github/last-commit/danionella/dejaq)\n\n# Déjà Queue\n\nA fast alternative to `multiprocessing.Queue`. Faster, because it takes advantage of a shared memory ring buffer (rather than slow pipes) and [pickle protocol 5 out-of-band data](https://peps.python.org/pep-0574/) to minimize copies. [`dejaq.DejaQueue`](#dejaqdejaqueue) supports any type of [picklable](https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled) Python object, including numpy arrays or nested dictionaries with mixed content.\n\n\u003cimg src=\"https://github.com/user-attachments/assets/837ca980-b225-45b7-8bf5-f0ce6cf7083c\" width=\"100%\"\u003e\n\nThe speed advantage of `DejaQueue` becomes substantial for items of \u003e 1 MB size. It enables efficient inter-job communication in big-data processing pipelines using [`dejaq.Actor`](#dejaqactor-and-actordecorator) or [`dejaq.stream`](#dejaqstream---building-data-pipelines).\n\n### Features:\n- Fast, low-latency, high-throughput inter-process communication\n- Supports any picklable Python object, including numpy arrays and nested dictionaries\n- Zero-copy data transfer with pickle protocol 5 out-of-band data\n- Picklable queue instances (queue object itself can be passed between processes)\n- Peekable (non-destructive read)\n- Actor class for remote method calls and attribute access in a separate process (see [dejaq.Actor](#dejaqactor-and-actordecorator))\n\nAuto-generated (minimal) API documentation: https://danionella.github.io/dejaq\n\n\n## Installation\n- `conda install conda-forge::dejaq `\n\n- or, if you prefer pip: `pip install dejaq`\n\n- for development, clone this repository, navigate to the root directory and type `pip install -e .`\n\n## Examples\n### dejaq.DejaQueue\n```python\nimport numpy as np\nfrom multiprocessing import Process\nfrom dejaq import DejaQueue\n\ndef produce(queue):\n    for i in range(10):\n        arr = np.random.randn(100,200,300)\n        data = dict(array=arr, i=i)\n        queue.put(data)\n        print(f'produced {type(arr)} {arr.shape} {arr.dtype}; meta: {i}; hash: {hash(arr.tobytes())}\\n', flush=True)\n\ndef consume(queue, pid):\n    while True:\n        data = queue.get()\n        array, i = data['array'], data['i']\n        print(f'consumer {pid} consumed {type(array)} {array.shape} {array.dtype}; index: {i}; hash: {hash(array.tobytes())}\\n', flush=True)\n\nqueue = DejaQueue(buffer_bytes=100e6)\nproducer = Process(target=produce, args=(queue,))\nconsumers = [Process(target=consume, args=(queue, pid)) for pid in range(3)]\nfor c in consumers:\n    c.start()\nproducer.start()\n```\n\n## dejaq.Actor and ActorDecorator\n\n`dejaq.Actor` allows you to run a class instance in a separate process and call its methods or access its attributes remotely, as if it were local. This is useful for isolating heavy computations, stateful services, or legacy code in a separate process, while keeping a simple Pythonic interface.\n\n\n### Example: Using `Actor` directly\n\n```python\nfrom dejaq import Actor\n\nclass Counter:\n    def __init__(self, start=0):\n        self.value = start\n    def increment(self, n=1):\n        self.value += n\n        return self.value\n    def get(self):\n        return self.value\n\n# Start the actor in a separate process\ncounter = Actor(Counter, start=10)\n\nprint(counter.get())         # 10\nprint(counter.increment())   # 11\nprint(counter.increment(5))  # 16\nprint(counter.get())         # 16\n\ncounter.close()  # Clean up the process\n```\n\n### Example: Using `ActorDecorator`\n\n```python\nfrom dejaq import ActorDecorator\n\n@ActorDecorator\nclass Greeter:\n    def __init__(self, name):\n        self.name = name\n    def greet(self):\n        return f\"Hello, {self.name}!\"\n\ngreeter = Greeter(\"Alice\")\nprint(greeter.greet())  # \"Hello, Alice!\"\ngreeter.close()\n```\n\n### Features\n\n- **Remote method calls:** Call methods as if the object was local.\n- **Remote attribute access:** Get/set attributes of the remote object.\n- **Async support:** Call `method_async()` to get a `Future` for non-blocking calls.\n- **Tab completion:** Works in Jupyter and most IDEs.\n\n\n\n## dejaq.Parallel\nThe following examples show how to use `dejaq.Parallel` to parallelize a function or a class, and how to create job pipelines.\n\nHere we execute a function and map iterable inputs across 10 workers. To enable pipelining, the results of each stage are provided as iterable generator. Use `.run()` (or `.compute()` for backwards compatibility) to get the final result. Results are always ordered.\n\n```python\nfrom time import sleep\nfrom dejaq import Parallel\n\ndef slow_function(arg):\n    sleep(1.0)\n    return arg + 5\n\ninput_iterable = range(100)\nslow_function = Parallel(n_workers=10)(slow_function)\nstage = slow_function(input_iterable)\nresult = stage.run() # or list(stage)\n# or shorter: \nresult = Parallel(n_workers=10)(slow_function)(input_iterable).compute()\n```\n\nYou can also use `Parallel` as a function decorator:\n```python\n@Parallel(n_workers=10)\ndef slow_function_decorated(arg):\n    sleep(1.0)\n    return arg + 5\n\nresult = slow_function_decorated(input_iterable).run()\n```\n\nSimilarly, you can decorate a class. It will be instantiated within a worker. Iterable items will be fed to the `__call__` method. Note how the additional init arguments are provided:\n```python\n@Parallel(n_workers=1)\nclass Reader:\n    def __init__(self, arg1):\n        self.arg1 = arg1\n    def __call__(self, item):\n        return item + self.arg1\n\nresult = Reader(arg1=0.5)(input_iterable).compute()\n```\n\nFinally, you can create pipelines of chained jobs. In this example, we have a single threaded reader and consumer, but a parallel processing stage (an example use case is sequentially reading a file, compressing chunks in parallel and then sequentially writing to an output file):\n```python\n@Parallel(n_workers=1)\nclass Producer:\n    def __init__(self, arg1):\n        self.arg1 = arg1\n    def __call__(self, item):\n        return item + self.arg1\n\n@Parallel(n_workers=10)\nclass Processor:\n    def __init__(self, arg1):\n        self.arg1 = arg1\n    def __call__(self, arg):\n        sleep(1.0) #simulating a slow function\n        return arg * self.arg1\n\n@Parallel(n_workers=1)\nclass Consumer:\n    def __init__(self, arg1):\n        self.arg1 = arg1\n    def __call__(self, arg):\n        return arg - self.arg1\n\ninput_iterable = range(100)\nstage1 = Producer(0.5)(input_iterable)\nstage2 = Processor(10.0)(stage1)\nstage3 = Consumer(1000)(stage2)\nresult = stage3.run()\n\n# or:\nresult = Consumer(1000)(Processor(10.0)(Producer(0.5)(input_iterable))).run()\n```\n\n\n# See also\n- [ArrayQueues](https://github.com/portugueslab/arrayqueues) \n- [joblib.Parallel](https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html)\n- [Déjà Q](https://en.wikipedia.org/wiki/Deja_Q)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanionella%2Fdejaq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanionella%2Fdejaq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanionella%2Fdejaq/lists"}