{"id":23127605,"url":"https://github.com/outerbounds/metaflow-slurm","last_synced_at":"2025-04-04T05:42:18.151Z","repository":{"id":267520550,"uuid":"883621913","full_name":"outerbounds/metaflow-slurm","owner":"outerbounds","description":"SLURM integration for Metaflow","archived":false,"fork":false,"pushed_at":"2024-12-10T20:25:42.000Z","size":41,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-02-09T17:14:42.989Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/outerbounds.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-05T09:40:46.000Z","updated_at":"2024-12-10T20:25:18.000Z","dependencies_parsed_at":"2024-12-10T20:36:40.231Z","dependency_job_id":"086abb65-8b80-4d9d-879d-7b4224d28013","html_url":"https://github.com/outerbounds/metaflow-slurm","commit_stats":null,"previous_names":["outerbounds/metaflow-slurm"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-slurm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-slurm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-slurm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-slurm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/outerbounds","download_url":"https://codeload.github.com/outerbounds/metaflow-slurm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247128739,"owners_count":20888234,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-17T09:14:09.881Z","updated_at":"2025-04-04T05:42:18.112Z","avatar_url":"https://github.com/outerbounds.png","language":"Python","funding_links":[],"categories":["Orchestration \u0026 Scheduling"],"sub_categories":[],"readme":"# SLURM extension for Metaflow\n\nThis extension adds support for executing steps in Metaflow Flows on SLURM clusters.\n\n## Basic Usage\n\n- Have a SLURM cluster that you have public access for.\n    - This includes the username, the IP address and the PEM file (at minimum)\n- Simply add the `@slurm` decorator to the step you want to run on the SLURM cluster.\n\n```py\n@slurm(\n    username=\"ubuntu\",\n    address=\"A.B.C.D\",\n    ssh_key_file=\"~/path/to/ssh/pem/file.pem\"\n)\n```\n\nNote that the above parameters can also be configured via the following environment variables:\n- `METAFLOW_SLURM_USERNAME`\n- `METAFLOW_SLURM_ADDRESS`\n- `METAFLOW_SLURM_SSH_KEY_FILE`\n\nThe step that is decorated with `@slurm` will create the following directory structure on the cluster.\n\n```bash\nmetaflow/\n├── assets\n│   └── madhurMovies218892mid13433160\n│       └── metaflow\n│           ├── INFO\n│           ├── demo.py\n│           ├── job.tar\n│           ├── linux-64\n│           ├── metaflow\n│           ├── metaflow_extensions\n│           └── micromamba\n├── madhurMovies218892mid13433160.sh\n├── stderr\n│   └── madhurMovies218892mid13433160.stderr\n└── stdout\n    └── madhurMovies218892mid13433160.stdout\n```\n\nIn the above output, `demo.py` was the name of our flow file.\n\nOne can pass `cleanup=True` in the decorator to clear up the contents of the `assets` folder.\nThis clears up all the `artifacts` created by Metaflow.\n\nUsing `cleanup=True` will not delete:\n- `stdout` folder\n- `stderr` folder\n- the generated shell script i.e. `madhurMovies218892mid13433160.sh`\n\nThis is useful for debugging later and may be manually deleted by logging into the slurm cluster.\n\n## Supplying Credentials\n\nCredentials need to be supplied to be able to download the code package. They can:\n\n- either exist on the Slurm cluster itself, i.e. compute instances have access to the blob store\n- supplied via the `@environment` decorator\n\n```py\n@environment(vars={\n    \"AWS_ACCESS_KEY_ID\": \"XXXX\",\n    \"AWS_SECRET_ACCESS_KEY\": \"YYYY\"\n})\n```\n\nNote that this will expose the credentials in the shell script that is generated i.e.\n\n`madhurMovies218892mid13433160.sh` will have the following contents present:\n\n```bash\nexport AWS_ACCESS_KEY_ID='XXXX'\nexport AWS_SECRET_ACCESS_KEY='YYYY'\n```\n\n- hydrating environment variables with the @secrets decorator from a secret manager.\n\nPS -- If you are on the [Outerbounds](https://outerbounds.com/) platform, the auth is taken care of and there is no need to fiddle with it.\n\n## Things to be taken care of\n\n- The extension runs workloads via shell scripts and `sbatch` in a linux native environment\n    - i.e. the workloads are NOT run inside docker containers\n    - As such, the compute instances should have `python3` installed (above 3.8 preferrably)\n    - If the default `python` points to `python2`, one can use the `path_to_python3` argument of the decorator i.e.\n\n```py\n@slurm(\n    path_to_python3=\"/usr/bin/python3\",\n)\n```\n\n### Fin.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fouterbounds%2Fmetaflow-slurm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fouterbounds%2Fmetaflow-slurm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fouterbounds%2Fmetaflow-slurm/lists"}