{"id":17933030,"url":"https://github.com/joerihermans/awflow","last_synced_at":"2025-09-21T15:32:43.693Z","repository":{"id":45077263,"uuid":"405937904","full_name":"JoeriHermans/awflow","owner":"JoeriHermans","description":"Reproducible research and reusable acyclic workflows in Python. Execute code on HPC systems as if you executed them on your personal computer!","archived":false,"fork":false,"pushed_at":"2022-01-11T21:48:57.000Z","size":250,"stargazers_count":18,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-04-26T12:03:53.683Z","etag":null,"topics":["automation","hpc","hpc-tools","python","reproducible-research","reproducible-science","slurm","workflow","workflow-engine"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JoeriHermans.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-09-13T11:05:02.000Z","updated_at":"2024-01-04T17:01:22.000Z","dependencies_parsed_at":"2022-08-27T18:41:02.613Z","dependency_job_id":null,"html_url":"https://github.com/JoeriHermans/awflow","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoeriHermans%2Fawflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoeriHermans%2Fawflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoeriHermans%2Fawflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoeriHermans%2Fawflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JoeriHermans","download_url":"https://codeload.github.com/JoeriHermans/awflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233764254,"owners_count":18726664,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","hpc","hpc-tools","python","reproducible-research","reproducible-science","slurm","workflow","workflow-engine"],"created_at":"2024-10-28T21:36:11.202Z","updated_at":"2025-09-21T15:32:38.422Z","avatar_url":"https://github.com/JoeriHermans.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Reproducible research and reusable acyclic workflows in Python. Execute code on HPC systems as if you executed them on your machine!\n\n## Motivation\n\nWould you like fully reproducible research or reusable workflows that seamlessly run on HPC clusters?\nTired of writing and managing large Slurm submission scripts? Do you have comment out large parts of your pipeline whenever its results have been generated? Hate YAML?\nDon't waste your precious time! `awflow` allows you to directly describe complex pipelines in Python, that run on your personal computer and large HPC clusters.\n\n\n```python\nimport glob\nimport numpy as np\nimport os\n\nfrom awflow import after, ensure, job, schedule\n\nn = 10000\ntasks = 10\n\n@ensure(lambda i: os.path.exists(f'pi-{i}.npy'))\n@job(cpus='4', memory='4GB', array=tasks)\ndef estimate(i: int):\n    print(f'Executing task {i + 1} / {tasks}.')\n    x = np.random.random(n)\n    y = np.random.random(n)\n    pi_estimate = (x**2 + y**2 \u003c= 1)\n    np.save(f'pi-{i}.npy', pi_estimate)\n\n@after(estimate)\n@ensure(lambda: os.path.exists('pi.npy'))\n@job(cpus='4')\ndef merge():\n    files = glob.glob('pi-*.npy')\n    stack = np.vstack([np.load(f) for f in files])\n    pi_estimate = stack.sum() / (n * tasks) * 4\n    print('π ≅', pi_estimate)\n    np.save('pi.npy', pi_estimate)\n\nmerge.prune()  # Prune jobs whose postconditions have been satisfied\n\nschedule(merge, backend='local')  # Executes merge and its dependencies\n```\nExecuting this Python program (`python examples/pi.py --backend slurm`) on a Slurm HPC cluster will launch the following jobs.\n```\n           1803299       all    merge username PD       0:00      1 (Dependency)\n     1803298_[6-9]       all estimate username PD       0:00      1 (Resources)\n         1803298_3       all estimate username  R       0:01      1 compute-xx\n         1803298_4       all estimate username  R       0:01      1 compute-xx\n         1803298_5       all estimate username  R       0:01      1 compute-xx\n```\nThe following example shows how workflow graphs can be dynamically allocated:\n```python\nfrom awflow import after, job, schedule, terminal_nodes\n\n@job(cpus='2', memory='4GB', array=5)\ndef generate(i: int):\n    print(f'Generating data block {i}.')\n\n@after(generate)\n@job(cpus='1', memory='2GB', array=5)\ndef postprocess(i: int):\n    print(f'Postprocessing data block {i}.')\n\ndef do_experiment(parameter):\n    r\"\"\"This method allocates a `fit` and `make_plot` job\n    based on the specified parameter.\"\"\"\n\n    @after(postprocess)\n    @job(name=f'fit_{parameter}')  # By default, the name is equal to the function name\n    def fit():\n        print(f'Fit {parameter}.')\n\n    @after(fit)\n    @job(name=f'plt_{parameter}')  # Simplifies the identification of the logfile\n    def make_plot():\n        print(f'Plot {parameter}.')\n\n# Programmatically build workflow\nfor parameter in [0.1, 0.2, 0.3, 0.4, 0.5]:\n    do_experiment(parameter)\n\nleafs = terminal_nodes(generate, prune=True)  # Find terminal nodes of workflow graph\nschedule(*leafs, backend='local')\n```\n\n\nCheck the [examples](examples/) directory to explore the functionality.\n\n## Available backends\n\nCurrently, `awflow.schedule` only supports a `local` and `slurm` backend.\n\n## Installation\n\nThe `awflow` package is available on [PyPi](https://pypi.org/project/awflow/), which means it is installable via `pip`.\n```console\nyou@local:~ $ pip install awflow\n```\nIf you would like the latest features, you can install it using this Git repository.\n```console\nyou@local:~ $ pip install git+https://github.com/JoeriHermans/awflow\n```\nIf you would like to run the examples as well, be sure to install the *optional* example dependencies.\n```console\nyou@local:~ $ pip install 'awflow[examples]'\n```\n\n## License\n\nAs described in the [`LICENSE`](LICENSE.txt) file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoerihermans%2Fawflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoerihermans%2Fawflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoerihermans%2Fawflow/lists"}