https://github.com/finos/opengris-parfun
Lightweight parallelisation library for Python
https://github.com/finos/opengris-parfun
Last synced: 6 months ago
JSON representation
Lightweight parallelisation library for Python
- Host: GitHub
- URL: https://github.com/finos/opengris-parfun
- Owner: finos
- License: apache-2.0
- Created: 2024-07-26T15:42:35.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-06-24T15:24:06.000Z (7 months ago)
- Last Synced: 2025-07-12T16:28:51.774Z (6 months ago)
- Language: Python
- Homepage: https://citi.github.io/parfun/
- Size: 802 KB
- Stars: 17
- Watchers: 4
- Forks: 12
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
Lightweight parallelization library for Python.
OpenGRIS Parfun is a lightweight library **making it easy to write and run Python in parallel and distributed systems**.
The main feature of the library is its `@parallel` decorator that transparently executes standard Python functions in parallel
following the [map-reduce](https://en.wikipedia.org/wiki/MapReduce) pattern:
```Python
from typing import List
import parfun as pf
@pf.parallel(
# parallelize by chunking the argument list (map)
split=pf.per_argument(
values=pf.py_list.by_chunk
),
# merge the output by concatenating the results (reduce)
combine_with=pf.py_list.concat,
)
def list_pow(values: List[float], factor: float) -> List[float]:
"""compute powers of a list of numbers"""
return [v**factor for v in values]
if __name__ == "__main__":
with pf.set_parallel_backend_context("local_multiprocessing"): # use a local pool of processes
print(list_pow([1, 2, 3], 2)) # runs in parallel, prints [1, 4, 9]
```
## Features
* **Provides significant speedups** to existing Python functions.
* **Only requires basic understanding of parallel and distributed computing systems**.
* **Automatically estimates the optimal sub-task splitting strategy** (the *partition size*).
* **Transparently handles data transmission, caching, and synchronization**.
* **Supports various distributed computing backends**:
* Python's built-in [multiprocessing module](https://docs.python.org/3/library/multiprocessing.html).
* [Scaler](https://github.com/citi/scaler).
* [Dask](https://www.dask.org/).
## Quick Start
Install Parfun directly from PyPI:
```bash
pip install parfun
pip install "parfun[pandas,scaler,dask]" # with optional dependencies
```
The official documentation is available at [citi.github.io/parfun/](https://citi.github.io/parfun/).
Take a look at our documentation's [quickstart tutorial](https://citi.github.io/parfun/tutorials/quickstart.html) to get
more examples and a deeper overview of the library.
Alternatively, you can build the documentation from source:
```bash
cd docs
pip install -r requirements.txt
make html
```
The documentation's main page can then be found at `docs/build/html/index.html`.
## Benchmarks
**Parfun effectively parallelizes even short-duration functions**.
For example, when running a short 0.28-second machine learning function on an AMD Epyc 7313 16-Core Processor, we found that Parfun
provided an impressive **7.4x speedup**. Source code for this experiment [here](examples/california_housing/main.py).
## Contributing
Your contributions are at the core of making this a true open source project. Any contributions you make are **greatly appreciated**.
We welcome you to:
* Fix typos or touch up documentation
* Share your opinions on [existing issues](https://github.com/finos/opengris-parfun/issues)
* Help expand and improve our library by [opening a new issue](https://github.com/finos/opengris-parfun/issues/new)
Please review [functional contribution guidelines](./CONTRIBUTING.md) to get started 👍.
_NOTE:_ Commits and pull requests to FINOS repositories will only be accepted from those contributors with an active, executed Individual Contributor License Agreement (ICLA) with FINOS OR contributors who are covered under an existing and active Corporate Contribution License Agreement (CCLA) executed with FINOS. Commits from individuals not covered under an ICLA or CCLA will be flagged and blocked by the ([EasyCLA](https://community.finos.org/docs/governance/Software-Projects/easycla)) tool. Please note that some CCLAs require individuals/employees to be explicitly named on the CCLA.
*Need an ICLA? Unsure if you are covered under an existing CCLA? Email [help@finos.org](mailto:help@finos.org)*
## Code of Conduct
Please see the FINOS [Community Code of Conduct](https://www.finos.org/code-of-conduct).
## License
Copyright 2023 Citigroup, Inc.
This project is distributed under the [Apache-2.0 License](https://www.apache.org/licenses/LICENSE-2.0). See
[`LICENSE`](./LICENSE) for more information.
SPDX-License-Identifier: [Apache-2.0](https://spdx.org/licenses/Apache-2.0).
## Contact
If you have a query or require support with this project, [raise an issue](https://github.com/Citi/parfun/issues).
Otherwise, reach out to [opensource@citi.com](mailto:opensource@citi.com).