{"id":19227951,"url":"https://github.com/snakemake/snakemake-executor-plugin-flux","last_synced_at":"2025-04-21T01:31:53.409Z","repository":{"id":175922293,"uuid":"654452162","full_name":"snakemake/snakemake-executor-plugin-flux","owner":"snakemake","description":"Snakemake custom executor plugin for Flux Framework (under development)","archived":false,"fork":false,"pushed_at":"2024-09-07T18:14:48.000Z","size":48,"stargazers_count":1,"open_issues_count":1,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-01T07:54:02.879Z","etag":null,"topics":["flux","flux-framework","snakemake"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snakemake.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-16T06:54:30.000Z","updated_at":"2024-09-07T18:14:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"bb71426c-f844-4590-9a96-71d341e5c2fd","html_url":"https://github.com/snakemake/snakemake-executor-plugin-flux","commit_stats":{"total_commits":21,"total_committers":3,"mean_commits":7.0,"dds":0.5238095238095238,"last_synced_commit":"580099076de2db565b0a46cee79e4ece6d3dfdc6"},"previous_names":["snakemake/snakemake-executor-flux","snakemake/snakemake-executor-plugin-flux"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snakemake%2Fsnakemake-executor-plugin-flux","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snakemake%2Fsnakemake-executor-plugin-flux/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snakemake%2Fsnakemake-executor-plugin-flux/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snakemake%2Fsnakemake-executor-plugin-flux/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snakemake","download_url":"https://codeload.github.com/snakemake/snakemake-executor-plugin-flux/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249982555,"owners_count":21355718,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flux","flux-framework","snakemake"],"created_at":"2024-11-09T15:26:00.500Z","updated_at":"2025-04-21T01:31:53.127Z","avatar_url":"https://github.com/snakemake.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Snakemake Executor Flux\n\nThis is an example implementation for an external snakemake plugin.\nSince we already have one for Flux (and it can run in a container) the example\nis for Flux. You can use this repository as a basis to design your own executor to work\nwith snakemake!\n\n## Usage\n\n### Tutorial\n\nFor this tutorial you will need Docker installer.\n\n[Flux-framework](https://flux-framework.org/) is a flexible resource scheduler that can work on both high performance computing systems and cloud (e.g., Kubernetes).\nSince it is more modern (e.g., has an official Python API) we define it under a cloud resource. For this example, we will show you how to set up a \"single node\" local\nFlux container to interact with snakemake using the plugin here. You can use the [Dockerfile](examples/Dockerfile) that will provide a container with Flux and snakemake\nNote that we install from source and bind to `/home/fluxuser/snakemake` with the intention of being able to develop (if desired).\n\nFirst, build the container:\n\n```bash\n$ docker build -f example/Dockerfile -t flux-snake .\n```\n\nWe will add the plugin here to `/home/fluxuser/plugin`, install it, and shell in as the fluxuser to optimally interact with flux.\nAfter the container builds, shell in:\n\n```bash\n$ docker run -it flux-snake bash\n```\n\nAnd start a flux instance:\n\n```bash\n$ flux start --test-size=4\n```\n\nGo into the examples directory (where the Snakefile is) and run snakemake, targeting your executor plugin.\n\n```bash\n$ cd ./example\n\n# This says \"use the custom executor module named snakemake_executor_plugin_flux\"\n$ snakemake --jobs 1 --executor flux\n```\n```console\nBuilding DAG of jobs...\nUsing shell: /bin/bash\nJob stats:\njob                         count    min threads    max threads\n------------------------  -------  -------------  -------------\nall                             1              1              1\nmultilingual_hello_world        2              1              1\ntotal                           3              1              1\n\nSelect jobs to execute...\n\n[Fri Jun 16 19:24:22 2023]\nrule multilingual_hello_world:\n    output: hola/world.txt\n    jobid: 2\n    reason: Missing output files: hola/world.txt\n    wildcards: greeting=hola\n    resources: tmpdir=/tmp\n\nJob 2 has been submitted with flux jobid ƒcjn4t3R (log: .snakemake/flux_logs/multilingual_hello_world/greeting_hola.log).\n[Fri Jun 16 19:24:32 2023]\nFinished job 2.\n1 of 3 steps (33%) done\nSelect jobs to execute...\n\n[Fri Jun 16 19:24:32 2023]\nrule multilingual_hello_world:\n    output: hello/world.txt\n    jobid: 1\n    reason: Missing output files: hello/world.txt\n    wildcards: greeting=hello\n    resources: tmpdir=/tmp\n\nJob 1 has been submitted with flux jobid ƒhAPLa79 (log: .snakemake/flux_logs/multilingual_hello_world/greeting_hello.log).\n[Fri Jun 16 19:24:42 2023]\nFinished job 1.\n2 of 3 steps (67%) done\nSelect jobs to execute...\n\n[Fri Jun 16 19:24:42 2023]\nlocalrule all:\n    input: hello/world.txt, hola/world.txt\n    jobid: 0\n    reason: Input files updated by another job: hello/world.txt, hola/world.txt\n    resources: tmpdir=/tmp\n\n[Fri Jun 16 19:24:42 2023]\nFinished job 0.\n3 of 3 steps (100%) done\nComplete log: .snakemake/log/2023-06-16T192422.186675.snakemake.log\n```\n\nAnd that's it! Continue reading to learn more about plugin design, and how you can also design your own executor\nplugin for use or development (that doesn't need to be added to upstream snakemake).\n\n## Developer\n\nTo do the same run but bind the local plugin directory:\n\n```bash\ndocker run -it -v $PWD/:/home/fluxuser/plugin flux-snake bash\n```\n\nThe instructions for creating and scaffolding this plugin are [here](https://github.com/snakemake/poetry-snakemake-plugin#scaffolding-an-executor-plugin).\nInstructions for writing your plugin with examples are provided via the [snakemake-executor-plugin-interface](https://github.com/snakemake/snakemake-executor-plugin-interface).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnakemake%2Fsnakemake-executor-plugin-flux","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnakemake%2Fsnakemake-executor-plugin-flux","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnakemake%2Fsnakemake-executor-plugin-flux/lists"}