Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/nipype/pydra
Pydra Dataflow Engine
https://github.com/nipype/pydra
brainweb dataflow-engine python3
Last synced: about 1 month ago
JSON representation
Pydra Dataflow Engine
- Host: GitHub
- URL: https://github.com/nipype/pydra
- Owner: nipype
- License: other
- Created: 2018-10-08T18:26:17.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2024-05-27T16:51:19.000Z (7 months ago)
- Last Synced: 2024-05-28T01:23:09.688Z (7 months ago)
- Topics: brainweb, dataflow-engine, python3
- Language: Python
- Homepage: https://nipype.github.io/pydra/
- Size: 19.3 MB
- Stars: 119
- Watchers: 14
- Forks: 57
- Open Issues: 213
-
Metadata Files:
- Readme: README.rst
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
|GHAction| |CircleCI| |codecov|
|Pydralogo|
.. |Pydralogo| image:: https://raw.githubusercontent.com/nipype/pydra/master/docs/logo/pydra_logo.jpg
:width: 200px
:alt: pydra logo
.. |GHAction| image:: https://github.com/nipype/pydra/workflows/Pydra/badge.svg
:alt: GitHub Actions CI
:target: https://github.com/nipype/Pydra/actions
.. |CircleCI| image:: https://circleci.com/gh/nipype/pydra.svg?style=svg
:alt: CircleCI
.. |codecov| image:: https://codecov.io/gh/nipype/pydra/branch/master/graph/badge.svg
:alt: codecov
======================
Pydra: Dataflow Engine
======================
A simple dataflow engine with scalable semantics.
Pydra is a rewrite of the Nipype engine with mapping and joining as
first-class operations. It forms the core of the Nipype 2.0 ecosystem.
The goal of pydra is to provide a lightweight Python dataflow engine for DAG
construction, manipulation, and distributed execution.
Feature list:
=============
1. Python 3.7+ using type annotation and `attrs `_
2. Composable dataflows with simple node semantics. A dataflow can be a node of another dataflow.
3. `splitter` and `combiner` provides many ways of compressing complex loop semantics
4. Cached execution with support for a global cache across dataflows and users
5. Distributed execution, presently via ConcurrentFutures, SLURM, and Dask (this is an experimental implementation with limited testing)
`API Documentation `_
Learn more about Pydra
======================
* `SciPy 2020 Proceedings `_
* `PyCon 2020 Poster `_
* `Explore Pydra interactively `_ (the tutorial can be also run using Binder service)
|Binder|
.. |Binder| image:: https://mybinder.org/badge_logo.svg
:alt: Binder
Please note that mybinder times out after an hour.
Installation
============
::
pip install pydra
Note that installation fails with older versions of pip on Windows. Upgrade pip before installing:
::
pip install –upgrade pip
pip install pydra
Developer installation
======================
Pydra requires Python 3.7+. To install in developer mode:
::
git clone [email protected]:nipype/pydra.git
cd pydra
pip install -e ".[dev]"
In order to run pydra's test locally:
::
pytest -vs pydra
If you want to test execution with Dask:
::
git clone [email protected]:nipype/pydra.git
cd pydra
pip install -e ".[dask]"
It is also useful to install pre-commit:
::
pip install pre-commit
pre-commit