{"id":13578510,"url":"https://github.com/joblib/loky","last_synced_at":"2025-05-14T08:06:19.645Z","repository":{"id":39629638,"uuid":"48578152","full_name":"joblib/loky","owner":"joblib","description":"Robust and reusable Executor for joblib","archived":false,"fork":false,"pushed_at":"2025-04-29T14:25:17.000Z","size":1363,"stargazers_count":563,"open_issues_count":51,"forks_count":47,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-05-08T00:06:01.352Z","etag":null,"topics":["multiprocessing-library","python"],"latest_commit_sha":null,"homepage":"http://loky.readthedocs.io/en/stable/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joblib.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-12-25T11:16:10.000Z","updated_at":"2025-05-04T17:25:36.000Z","dependencies_parsed_at":"2024-06-18T13:44:52.553Z","dependency_job_id":"93a3de7a-40f7-4f93-880b-9009879266dd","html_url":"https://github.com/joblib/loky","commit_stats":{"total_commits":650,"total_committers":20,"mean_commits":32.5,"dds":0.6230769230769231,"last_synced_commit":"a9dc2eb790cf8e50eb75b2aeb70cb5f243c093d2"},"previous_names":[],"tags_count":39,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joblib%2Floky","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joblib%2Floky/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joblib%2Floky/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joblib%2Floky/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joblib","download_url":"https://codeload.github.com/joblib/loky/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254101615,"owners_count":22014909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["multiprocessing-library","python"],"created_at":"2024-08-01T15:01:31.247Z","updated_at":"2025-05-14T08:06:14.628Z","avatar_url":"https://github.com/joblib.png","language":"Python","readme":"\n\u003ca href=\"https://loky.readthedocs.io\"\u003e\n\u003cimg src=\"docs/_static/loky_logo.svg\"\nalt=\"Loky logo\" width=96/\u003e\u003c/a\u003e\n\n\n# Reusable Process Pool Executor\n[![Build Status](https://github.com/joblib/loky/actions/workflows/test.yml/badge.svg?branch=master)](https://github.com/joblib/loky/actions/workflows/test.yml?query=branch%3Amaster)\n[![Documentation Status](https://readthedocs.org/projects/loky/badge/?version=latest)](https://loky.readthedocs.io/en/latest/?badge=latest)\n[![codecov](https://codecov.io/gh/joblib/loky/branch/master/graph/badge.svg)](https://codecov.io/gh/joblib/loky)\n[![DOI](https://zenodo.org/badge/48578152.svg)](https://zenodo.org/badge/latestdoi/48578152)\n\n\n### Goal\n\nThe aim of this project is to provide a robust, cross-platform and\ncross-version implementation of the `ProcessPoolExecutor` class of\n`concurrent.futures`. It notably features:\n\n  * __Consistent and robust spawn behavior__: All processes are started\n    using fork + exec on POSIX systems. This ensures safer interactions with\n    third party libraries. On the contrary, `multiprocessing.Pool` uses\n    fork without exec by default, causing third party runtimes to crash\n    (e.g. OpenMP, macOS Accelerate...).\n\n  * __Reusable executor__: strategy to avoid re-spawning a complete\n    executor every time. A singleton executor instance can be reused (and\n    dynamically resized if necessary) across consecutive calls to limit\n    spawning and shutdown overhead. The worker processes can be shutdown\n    automatically after a configurable idling timeout to free system\n    resources.\n\n  * __Transparent cloudpickle integration__: to call interactively\n    defined functions and lambda expressions in parallel. It is also\n    possible to register a custom pickler implementation to handle\n    inter-process communications.\n\n  * __No need for ``if __name__ == \"__main__\":`` in scripts__: thanks\n    to the use of ``cloudpickle`` to call functions defined in the\n    ``__main__`` module, it is not required to protect the code calling\n    parallel functions under Windows.\n\n  * __Deadlock free implementation__: one of the major concern in\n    standard `multiprocessing` and `concurrent.futures` modules is the\n    ability of the `Pool/Executor` to handle crashes of worker\n    processes. This library intends to fix those possible deadlocks and\n    send back meaningful errors. Note that the implementation of\n    `concurrent.futures.ProcessPoolExecutor` that comes with Python 3.7+\n    is as robust as the executor from loky but the latter also works for\n    older versions of Python.\n\n\n### Installation\n\nThe recommended way to install `loky` is with `pip`,\n```bash\npip install loky\n```\n\n`loky` can also be installed from sources using\n```bash\ngit clone https://github.com/joblib/loky\ncd loky\npython setup.py install\n```\n\nNote that `loky` has an optional dependency on [`psutil`][1] to allow early\nmemory leak detections.\n\n### Usage\n\nThe basic usage of `loky` relies on the `get_reusable_executor`, which\ninternally manages a custom `ProcessPoolExecutor` object, which is reused or\nre-spawned depending on the context.\n\n```python\nimport os\nfrom time import sleep\nfrom loky import get_reusable_executor\n\n\ndef say_hello(k):\n    pid = os.getpid()\n    print(f\"Hello from {pid} with arg {k}\")\n    sleep(.01)\n    return pid\n\n\n# Create an executor with 4 worker processes, that will\n# automatically shutdown after idling for 2s\nexecutor = get_reusable_executor(max_workers=4, timeout=2)\n\nres = executor.submit(say_hello, 1)\nprint(\"Got results:\", res.result())\n\nresults = executor.map(say_hello, range(50))\nn_workers = len(set(results))\nprint(\"Number of used processes:\", n_workers)\nassert n_workers == 4\n```\n\nFor more advance usage, see our\n[documentation](https://loky.readthedocs.io/en/stable/)\n\n### Workflow to contribute\n\nTo contribute to **loky**, first create an account on\n[github](http://github.com/). Once this is done, fork the\n[loky repository](http://github.com/loky/loky) to have your own repository,\nclone it using 'git clone' on the computers where you want to work. Make your\nchanges in your clone, push them to your github account, test them on several\ncomputers, and when you are happy with them, send a pull request to the main\nrepository.\n\n### Running the test suite\n\nTo run the test suite, you need the `pytest` (version \u003e= 3) and `psutil`\nmodules. From the root of the project, run the test suite using:\n\n```sh\n    pip install -e .\n    pytest .\n```\n\n### Why was the project named `loky`?\n\nWhile developping `loky`, we had some bad experiences trying to debug deadlocks\nwhen using `multiprocessing.Pool` and `concurrent.futures.ProcessPoolExecutor`,\nespecially when calling functions with non-picklable arguments or returned\nvalues at the beginning of the project. When we had to chose a name, we had\ndealt with so many deadlocks that we wanted some kind of invocation to repel\nthem! Hence `loky`: a mix of a god, locks and the `y` that make it somehow\ncooler and nicer : (and also less likely to result in name conflict in google\nresults ^^).\n\nFixes to avoid those deadlocks in `concurrent.futures` were also contributed\nupstream in Python 3.7+, as a less mystical way to repel the deadlocks :D\n\n### Acknowledgement\n\nThis work is supported by the Center for Data Science, funded by the IDEX\nParis-Saclay, ANR-11-IDEX-0003-02\n\n\n[1]: https://github.com/giampaolo/psutil\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoblib%2Floky","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoblib%2Floky","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoblib%2Floky/lists"}