https://github.com/34j/nest-joblib
Patch joblib to allow nested parallelism
https://github.com/34j/nest-joblib
hacktoberfest joblib multiprocessing parallel python
Last synced: about 1 month ago
JSON representation
Patch joblib to allow nested parallelism
- Host: GitHub
- URL: https://github.com/34j/nest-joblib
- Owner: 34j
- License: mit
- Created: 2023-03-12T07:21:01.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2025-04-07T16:59:18.000Z (6 months ago)
- Last Synced: 2025-04-13T06:14:29.146Z (6 months ago)
- Topics: hacktoberfest, joblib, multiprocessing, parallel, python
- Language: Python
- Homepage:
- Size: 175 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 12
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
# nest-joblib
Patch joblib to allow nested parallelism.
## Installation
Install this via pip (or your favourite package manager):
```shell
pip install nest-joblib
```
## Usage
```python
from nest_joblib import apply
apply()
```
With the above code, LokyBackend supports nested-parallelism.
## Advanced Usage
The following joblib specification of not doing nested-parallelism may be inefficient in an environment with sufficient memory.
`joblib/_parallel_backends.py`:
```python
def get_nested_backend(self):
"""Backend instance to be used by nested Parallel calls.
By default a thread-based backend is used for the first level of
nesting. Beyond, switch to sequential backend to avoid spawning too
many threads on the host.
"""
nesting_level = getattr(self, 'nesting_level', 0) + 1
if nesting_level > 1:
return SequentialBackend(nesting_level=nesting_level), None
else:
return ThreadingBackend(nesting_level=nesting_level), None
```
After calling `nest_joblib.apply()`, when `joblib.parallel.register_parallel_backend(name, backend)` is called, a subclass of `backend` with modified `get_nested_backend` is dynamically generated and registered with the name `f"nested-{name}"`.
```python
from joblib.parallel import parallel_backend
from nest_joblib import apply
from ray.util.joblib import register_ray
# use LokyBackend
apply()
# use DaskDistributedBackend
apply()
parallel_backend("nested-dask")
# use RayBackend
apply()
register_ray()
parallel_backend("nested-ray")
# use custom backend
from joblib.parallel import LokyBackend
apply()
class MyBackend(LokyBackend):
pass
register_parallel_backend("custom", MyBackend)
parallel_backend("nested-custom")
```
## Contributors ✨
Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):
This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!
