{"id":23673737,"url":"https://github.com/carpentries-incubator/hpc-workflows","last_synced_at":"2025-09-02T02:32:40.161Z","repository":{"id":165781698,"uuid":"630048447","full_name":"carpentries-incubator/hpc-workflows","owner":"carpentries-incubator","description":"HPC Workflow Management with Snakemake","archived":false,"fork":false,"pushed_at":"2024-12-24T01:13:01.000Z","size":7399,"stargazers_count":3,"open_issues_count":9,"forks_count":4,"subscribers_count":11,"default_branch":"main","last_synced_at":"2024-12-24T02:23:08.542Z","etag":null,"topics":["carpentries-incubator","english","hpc-carpentry","lesson","pre-alpha","snakemake","workflows"],"latest_commit_sha":null,"homepage":"https://carpentries-incubator.github.io/hpc-workflows/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/carpentries-incubator.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["carpentries"],"custom":["https://give.communityin.org/TheCarpentries"]}},"created_at":"2023-04-19T14:58:48.000Z","updated_at":"2024-12-05T17:03:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"c922869b-354a-4bcf-ba66-811b5bebaba0","html_url":"https://github.com/carpentries-incubator/hpc-workflows","commit_stats":{"total_commits":31,"total_committers":7,"mean_commits":4.428571428571429,"dds":0.7096774193548387,"last_synced_commit":"ba7a81f0d9b367f2ad53ce3002abaafb735ec16a"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":"carpentries/workbench-template-rmd","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carpentries-incubator%2Fhpc-workflows","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carpentries-incubator%2Fhpc-workflows/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carpentries-incubator%2Fhpc-workflows/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carpentries-incubator%2Fhpc-workflows/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/carpentries-incubator","download_url":"https://codeload.github.com/carpentries-incubator/hpc-workflows/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231737321,"owners_count":18418996,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["carpentries-incubator","english","hpc-carpentry","lesson","pre-alpha","snakemake","workflows"],"created_at":"2024-12-29T12:55:46.748Z","updated_at":"2024-12-29T12:55:47.848Z","avatar_url":"https://github.com/carpentries-incubator.png","language":"Python","funding_links":["https://github.com/sponsors/carpentries","https://give.communityin.org/TheCarpentries"],"categories":[],"sub_categories":[],"readme":"# Tame Your Workflow with Snakemake\n\nIn [HPC Intro][hpc-intro], learners explored the scheduler on their cluster by\nlaunching a program called [`amdahl`][amdahl]. The objective of this lesson is\nto adapt the manual job submission process into a repeatable, reusable workflow\nwith minimal human intervention. This is accomplished using\n[Snakemake][snakemake], a modern workflow engine.\n\nIf you are interested in learning more about workflow tools, please visit\n[The Workflows Community][workflows-community].\n\n\u003e ## Snakemake is best for single-node jobs\n\u003e\n\u003e NERSC's [Snakemake docs][nersc-snake] lists Snakemake's \"cluster mode\" as a\n\u003e _disadvantage_, since it submits each \"rule\" as a separate job, thereby\n\u003e spamming the scheduler with dependent tasks. The main Snakemake process also\n\u003e resides on the login node until all jobs have finished, occupying some\n\u003e resources.\n\u003e\n\u003e If you wish to adapt your Python-based program for multi-node cluster\n\u003e execution, consider applying the workflow principles learned from this lesson\n\u003e to the [Parsl][parsl] framework. Again, NERSC's [Parsl docs][nersc-parsl]\n\u003e provide helpful tips.\n\n## Contributing\n\nThis is a translation of the [old HPC Workflows lesson][workflows] using\n[The Carpentries Workbench][workbench] and [R Markdown (Rmd)][rmd].\nYou are cordially invited to contribute! Please check the list of\n[issues][issues] if you're unsure where to start.\n\n### Building Locally\n\nIf you edit the lesson, it is important to verify that the changes are rendered\nproperly in the online version. The best way to do this is to build the lesson\nlocally. You will need an R environment to do this: as described in the\n[`{sandpaper}` docs][sandpaper], the environment can be either your terminal or\nRStudio.\n\n#### Setup\n\nThe `environment.yml` file describes a [Conda][conda] virtual environment that\nincludes [R][r-project], [Snakemake][snakemake], [amdahl][amdahl],\n[pandoc][pandoc], and [termplotlib][termplotlib]: the tools you'll need to\ndevelop and run this lesson, as well as some depencencies. To prepare the\nenvironment, install [Miniconda][miniconda] following the official\ninstructions. Then open a shell application and create a new environment:\n\n``` shell\nyou@yours:~$ cd path/to/local/hpc-workflows\nyou@yours:hpc-workflows$ conda env create -f environment.yaml\n```\n\n\u003e _N.B.:_ the environment will be named \"workflows\" by default.\n\u003e If you prefer another name, add `-n «alternate_name»` to the command.\n\n#### {sandpaper}\n\n[{sandpaper}][sandpaper] is the engine behind The Carpentries Workbench lesson\nlayout and static website generator. It is an R package, and has not yet been\ninstalled. Paraphrasing the installation instructions, start R or\n[radian][radian], then install:\n\n``` shell\nyou@yours:hpc-workflows$ R --no-restore --no-save\n```\n\n``` R\ninstall.packages(c(\"sandpaper\", \"varnish\", \"pegboard\", \"tinkr\"),\n repos = c(\"https://carpentries.r-universe.dev/\", getOption(\"repos\")))\n```\n\nNow you can render the site! From your R session,\n\n``` R\nlibrary(\"sandpaper\")\nsandpaper::serve()\n```\n\nThis should output something like the following:\n\n``` plain\nOutput created: hpc-workflows/site/docs/index.html\nTo stop the server, run servr::daemon_stop(1) or restart your R session\nServing the directory hpc-workflows/site/docs at http://127.0.0.1:4321\n```\n\nClick on the link to \u003chttp://127.0.0.1:4321\u003e or copy and paste it in your\nbrowser. You should see any changes you've made to the lesson on the\ncorresponding page(s). If it looks right, you're set to proceed!\n\n\u003c!-- HPC Carpentry links --\u003e\n\n[amdahl]: https://github.com/hpc-carpentry/amdahl\n[hpc-intro]: https://carpentries-incubator.github.io/hpc-intro/\n[issues]: https://github.com/carpentries-incubator/hpc-workflows/issues\n[workflows]: https://github.com/hpc-carpentry/hpc-workflows.old\n\n\u003c!-- The Carpentries links --\u003e\n[workbench]: https://carpentries.github.io/sandpaper-docs/\n\n\u003c!-- world-wide web links --\u003e\n[conda]: https://docs.conda.io/en/latest/\n[miniconda]: https://docs.conda.io/projects/miniconda/en/latest/\n[nersc-parsl]: https://docs.nersc.gov/jobs/workflow/parsl/\n[nersc-snake]: https://docs.nersc.gov/jobs/workflow/snakemake/\n[pandoc]: https://pandoc.org\n[parsl]: http://parsl-project.org\n[r-project]: https://www.r-project.org\n[radian]: https://github.com/randy3k/radian\n[rmd]: https://rmarkdown.rstudio.com\n[sandpaper]: https://carpentries.github.io/sandpaper-docs/\n[snakemake]: https://snakemake.readthedocs.io/en/stable/\n[termplotlib]: https://github.com/nschloe/termplotlib\n[workflows-community]: https://workflows.community\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcarpentries-incubator%2Fhpc-workflows","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcarpentries-incubator%2Fhpc-workflows","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcarpentries-incubator%2Fhpc-workflows/lists"}