{"id":19705787,"url":"https://github.com/llnl/maestrowf","last_synced_at":"2025-04-08T09:09:16.587Z","repository":{"id":22600858,"uuid":"96822206","full_name":"LLNL/maestrowf","owner":"LLNL","description":"A tool to easily orchestrate general computational workflows both locally and on supercomputers","archived":false,"fork":false,"pushed_at":"2025-03-31T09:53:56.000Z","size":3117,"stargazers_count":142,"open_issues_count":75,"forks_count":43,"subscribers_count":18,"default_branch":"develop","last_synced_at":"2025-04-01T07:51:36.702Z","etag":null,"topics":["hpc","python","radiuss","science-research","simulation-study","workflow","workflow-processes","workflows"],"latest_commit_sha":null,"homepage":"https://maestrowf.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LLNL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-10T21:18:45.000Z","updated_at":"2025-02-13T00:35:25.000Z","dependencies_parsed_at":"2023-12-16T04:10:03.328Z","dependency_job_id":"cabac17d-8f20-4c55-8c1d-f252dda70fb7","html_url":"https://github.com/LLNL/maestrowf","commit_stats":{"total_commits":273,"total_committers":31,"mean_commits":8.806451612903226,"dds":0.5787545787545787,"last_synced_commit":"b9ea2f01dbcfc44000cf730019d8db3189209446"},"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LLNL%2Fmaestrowf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LLNL%2Fmaestrowf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LLNL%2Fmaestrowf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LLNL%2Fmaestrowf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LLNL","download_url":"https://codeload.github.com/LLNL/maestrowf/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247809964,"owners_count":20999816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hpc","python","radiuss","science-research","simulation-study","workflow","workflow-processes","workflows"],"created_at":"2024-11-11T21:30:11.136Z","updated_at":"2025-04-08T09:09:16.570Z","avatar_url":"https://github.com/LLNL.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![](https://github.com/LLNL/maestrowf/raw/develop/assets/logo.png?raw=true \"Orchestrate your workflows with ease!\")\n\n# Maestro Workflow Conductor ([maestrowf](https://pypi.org/project/maestrowf/))\n\n[![Build Status](https://travis-ci.org/LLNL/maestrowf.svg?branch=develop)](https://travis-ci.org/LLNL/maestrowf)\n[![PyPI](https://img.shields.io/pypi/v/maestrowf.svg)](https://pypi.python.org/pypi?name=maestrowf\u0026version=1.0.0\u0026:action=display)\n![Spack](https://img.shields.io/spack/v/py-maestrowf)\n[![Issues](https://img.shields.io/github/issues/LLNL/maestrowf.svg)](https://github.com/LLNL/maestrowf/issues)\n[![Forks](https://img.shields.io/github/forks/LLNL/maestrowf.svg)](https://github.com/LLNL/maestrowf/network)\n[![Stars](https://img.shields.io/github/stars/LLNL/maestrowf.svg)](https://github.com/LLNL/maestrowf/stargazers)\n[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/LLNL/maestrowf/master/LICENSE)\n\n[![Downloads](https://static.pepy.tech/personalized-badge/maestrowf?period=total\u0026units=international_system\u0026left_color=grey\u0026right_color=blue\u0026left_text=Downloads)](https://pepy.tech/project/maestrowf)\n[![Downloads](https://static.pepy.tech/personalized-badge/maestrowf?period=month\u0026units=international_system\u0026left_color=grey\u0026right_color=blue\u0026left_text=Downloads/Month)](https://pepy.tech/project/maestrowf)\n\nMaestro can be installed via [pip](https://pip.pypa.io/):\n\n    pip install maestrowf\n\n## Documentation\n\n* [Maestro Documentation](https://maestrowf.readthedocs.io) - Official Maestro documentation.\n* [Maestro Sheetmusic](https://github.com/LLNL/maestro_sheetmusic) - A collection of sample and user contributed Maestro study examples.\n* [Maestro Samples](/samples) - Maestro sample studies.\n\n## Getting Started is Quick and Easy\n\nCreate a `YAML` file named `study.yaml` and paste the following content into the file:\n\n``` yaml\ndescription:\n    name: hello_world\n    description: A simple 'Hello World' study.\n\nstudy:\n    - name: say-hello\n      description: Say hello to the world!\n      run:\n          cmd: |\n            echo \"Hello, World!\" \u003e hello_world.txt\n```\n\n\u003e *PHILOSOPHY*: Maestro believes in the principle of a clearly defined process, specified as a list of tasks, that are self-documenting and clear in their intent.\n\nRunning the `hello_world` study is as simple as...\n\n    maestro run study.yaml\n\n## Creating a Parameter Study is just as Easy\n\nWith the addition of the `global.parameters` block, and a few simple tweaks to your `study` block, the complete specification should look like this:\n\n``` yaml\ndescription:\n    name: hello_planet\n    description: A simple study to say hello to planets (and Pluto)\n\nstudy:\n    - name: say-hello\n      description: Say hello to a planet!\n      run:\n          cmd: |\n            echo \"Hello, $(PLANET)!\" \u003e hello_$(PLANET).txt\n\nglobal.parameters:\n    PLANET:\n        values: [Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, Pluto]\n        label: PLANET.%%\n```\n\n\u003e *PHILOSOPHY*: Maestro believes that a workflow should be easily parameterized with minimal modifications to the core process.\n\nMaestro will automatically expand each parameter into its own isolated workspace, generate a script for each parameter, and automatically monitor execution of each task.\n\nAnd, running the study is still as simple as:\n\n``` bash\n    maestro run study.yaml\n```\n\n## Scheduling Made Simple\n\nBut wait there's more! If you want to schedule a study, it's just as simple. With some minor modifications, you are able to run on an [HPC](https://en.wikipedia.org/wiki/Supercomputer) system.\n\n``` yaml\ndescription:\n    name: hello_planet\n    description: A simple study to say hello to planets (and Pluto)\n\nbatch:\n    type:  slurm\n    queue: pbatch\n    host:  quartz\n    bank:  science\n\nstudy:\n    - name: say-hello\n      description: Say hello to a planet!\n      run:\n          cmd: |\n            echo \"Hello, $(PLANET)!\" \u003e hello_$(PLANET).txt\n          nodes: 1\n          procs: 1\n          walltime: \"00:02:00\"\n\nglobal.parameters:\n    PLANET:\n        values: [Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, Pluto]\n        label: PLANET.%%\n```\n\n\u003e **NOTE**: This specification is configured to run on LLNL's quartz cluster. Under the `batch` header, you will need to make the necessary changes to schedule onto other HPC resources.\n\u003e\n\u003e *PHILOSOPHY*: Maestro believes that how a workflow is defined should be decoupled from how it's run. We achieve this capability by providing a seamless interface to multiple schedulers that allows Maestro to readily port workflows to multiple platforms.\n\nFor other samples, see the [samples](/samples) subfolder. To continue with our Hello World example, see the [Basics of Study Construction](https://maestrowf.readthedocs.io/en/latest/hello_world.html) in our [documentation](https://maestrowf.readthedocs.io/en/latest/index.html).\n\n## An Example Study using LULESH\n\nMaestro comes packed with a basic example using [LULESH](https://github.com/LLNL/LULESH), a proxy application provided by LLNL. You can find the example [here](https://maestrowf.readthedocs.io/en/latest/quick_start.html#).\n\n## What is Maestro?\n\nMaestro is an open-source HPC software tool that defines a YAML-based study specification for defining multistep workflows and automates execution of software flows on HPC resources. The core design tenants of Maestro focus on encouraging clear workflow communication and documentation, while making consistent execution easier to allow users to focus on science. Maestro's study specification helps users think about complex workflows in a step-wise, intent-oriented, manner that encourages modularity and tool reuse. These principles are becoming increasingly important as computational science is continuously more present in scientific fields and has started to require a similar rigor to physical experiment. Maestro is currently in use for multiple projects at Lawrence Livermore National Laboratory and has been used to run existing codes including MFEM, and other simulation codes. It has also been used in other areas including in the training of machine-learned models and more.\n\n### Maestro's Foundation and Core Concepts\n\nThere are many definitions of workflow, so we try to keep it simple and define the term as follows:\n\n``` text\nA set of high level tasks to be executed in some order, with or without dependencies on each other.\n```\n\nWe have designed Maestro around the core concept of what we call a \"study\". A study is defined as a set of steps that are executed (a workflow) over a set of parameters. A study in Maestro's context is analogous to an actual tangible scientific experiment, which has a set of clearly defined and repeatable steps which are repeated over multiple specimen.\n\nMaestro's core tenets are defined as follows:\n\n##### Repeatability\n\nA study should be easily repeatable. Like any well-planned and implemented science experiment, the steps themselves should be executed the exact same way each time a study is run over each set of parameters or over different runs of the study itself.\n\n##### Consistent\n\nStudies should be consistently documented and able to be run in a consistent fashion. The removal of variation in the process means less mistakes when executing studies, ease of picking up studies created by others, and uniformity in defining new studies.\n\n##### Self-documenting\n\nDocumentation is important in computational studies as much as it is in physical science. The YAML specification defined by Maestro provides a few required key encouraging human-readable documentation. Even further, the specification itself is a documentation of a complete workflow.\n\n----------------\n\n## Setting up your Python Environment\n\nTo get started, we recommend using virtual environments. If you do not have the\nPython `virtualenv` package installed, take a look at their official [documentation](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/) to get started.\n\nTo create a new virtual environment:\n\n    python -m virtualenv maestro_venv\n    source maestro_venv/bin/activate\n\n### Getting Started for Contributors\n\nIf you plan to develop on Maestro, install the repository directly using:\n\n    pip install poetry\n    poetry install\n\nOnce set up, test the environment. The paths should point to a virtual environment folder.\n\n    which python\n    which pip\n\n----------------\n\n## Using Maestro Dockerfiles\n\n\nMaestro comes packaged with a set of Docker files for testing things out. The two primary files\nare:\n\n- A standard `Dockerfile` in the root of the Maestro repository. This file is a standard install\n  of Maestro meant to try out Maestro on the demo samples provided with this repository. In order\n  to try Maestro locally, with [Docker](https://www.docker.com/) installed run:\n\n  ```\n  docker build -t maestrowf .\n  docker run -ti maestrowf\n  ```\n\n  From within the container run the following:\n\n  ```\n  maestro run ./maestrowf/samples/lulesh/lulesh_sample1_unix.yaml\n  ```\n\n- In order to try out Flux 0.19.0 integration, from the root of the Maestro repository run the\n  following:\n\n  ```\n  docker build -t flux_0190 -f ./docker/flux/0.19.0/Dockerfile .\n  docker run -ti flux_0190\n  ```\n\n  From within the container run the following:\n\n  ```\n  maestro run ./maestrowf/samples/lulesh/lulesh_sample1_unix_flux.yaml\n  ```\n\n----------------\n\n## Contributors\n\nMany thanks go to MaestroWF's [contributors](https://github.com/LLNL/maestrowf/graphs/contributors).\n\nIf you have any questions or to submit feature requests please [open a ticket](https://github.com/llnl/maestrowf/issues).\n\n----------------\n\n## Release\nMaestroWF is released under an MIT license.  For more details see the\nNOTICE and LICENSE files.\n\n``LLNL-CODE-734340``\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fllnl%2Fmaestrowf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fllnl%2Fmaestrowf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fllnl%2Fmaestrowf/lists"}