{"id":19227935,"url":"https://github.com/snakemake/snakemake-spack-tutorial","last_synced_at":"2025-04-21T01:31:53.822Z","repository":{"id":108942200,"uuid":"367432895","full_name":"snakemake/snakemake-spack-tutorial","owner":"snakemake","description":"The snakemake tutorial run with spack! (under development)","archived":false,"fork":false,"pushed_at":"2021-08-17T23:52:33.000Z","size":4830,"stargazers_count":4,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-01T07:54:01.082Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snakemake.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-14T17:18:36.000Z","updated_at":"2025-01-28T18:54:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"f7b5f9c6-e2b8-4f1c-b4da-369e9bc46a86","html_url":"https://github.com/snakemake/snakemake-spack-tutorial","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snakemake%2Fsnakemake-spack-tutorial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snakemake%2Fsnakemake-spack-tutorial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snakemake%2Fsnakemake-spack-tutorial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snakemake%2Fsnakemake-spack-tutorial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snakemake","download_url":"https://codeload.github.com/snakemake/snakemake-spack-tutorial/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249982553,"owners_count":21355716,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T15:25:57.534Z","updated_at":"2025-04-21T01:31:53.816Z","avatar_url":"https://github.com/snakemake.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Snakemake Tutorial with Spack\n\nThis is a small repository that shows running the snakemake tutorial using\nspack. For the full tutorial, you should see [the snakemake documentation](https://snakemake.readthedocs.io/en/stable/tutorial/tutorial.html). This tutorial is a subset of that, and we skip over the details\nto explain the differences of running natively or with conda and with spack.\n\n## Install Snakemake\n\nFor this tutorial, we are currently working off of the [add/spack](https://github.com/snakemake/snakemake/tree/add/spack) \nbranch of the snakemake repository. This will eventually be integrated with the\nsnakemake software, but for now is there. This means that you should currently\nclone the branch:\n\n```bash\ngit clone -b add/spack https://github.com/snakemake/snakemake\ncd snakemake\n```\n\nCheck out a virtual environment:\n\n```bash\npython -m venv env\nsource env/bin/activate\n```\n\nAnd install snakemake locally! This is considered a development install so you\ndon't muck around with your system or main snakemake.\n\n```bash\n$ pip install -e .\n```\n\n## Clone the workflow\n\nYou can then clone the tutorial here.\n\n```bash\n$ git clone https://github.com/snakemake/snakemake-spack-tutorial\ncd snakemake-spack-tutorial\n```\n\nTake a look around - you'll see an input data directory (data), a script\ndirectory with the Python script to make a plot, and then a folder of spack\nenvironments.\n\n```bash\n├── data\n├── envs\n├── README.md\n├── scripts\n└── Snakefile\n```\n\n## Changes to the Snakefile\n\nThe Snakefile is our main configuration file or recipe for defining and\nthen executing our workflow. If you compare this workfile to the \n[one in the tutorial](https://snakemake.readthedocs.io/en/stable/tutorial/basics.html#summary),\nthe only change is that the `conda` directives are now `spack`. Here is an example\"\n\n```yaml\nrule bwa:\n    input:\n        \"data/genome.fa\",\n        \"data/samples/{sample}.fastq\"\n    output:\n        temp(\"mapped/{sample}.bam\")\n    spack:\n        \"envs/spack-mapping.yml\"\n    threads: 8\n    shell:\n        \"bwa mem -t {threads} {input} | samtools view -Sb - \u003e {output}\"\n```\n\nAnd the environments folder includes yaml files that define spack environments!\nHere is the one that we see above:\n\n```yaml\nspack:\n    specs:\n      - bwa\n      - samtools\n    concretization: together\n```\n\n## Running the workflow\n\nSince we need spack, this means that it should be on your path before starting.\nYou can check as follows:\n\n```bash\n$ which spack\n```\n\nIf you need help installing spack, see [here](https://spack.readthedocs.io/en/latest/).\nThe spack executable is located in the bin of the repository that you clone.\nWe then can run the workflow!\n\n```bash\n$ snakemake --cores 1 --rerun-incomplete --use-spack\nBuilding DAG of jobs...\nUsing shell: /bin/bash\nProvided cores: 1 (use --cores to define parallelism)\nRules claiming more threads will be scaled down.\nJob counts:\n\tcount\tjobs\n\t1\tall\n\t2\tbwa\n\t1\tcall\n\t2\tsort\n\t1\tstats\n\t7\n\n[Fri May 14 10:56:54 2021]\nrule bwa:\n    input: data/genome.fa, data/samples/B.fastq\n    output: mapped/B.bam\n    jobid: 5\n    wildcards: sample=B\n\n[M::bwa_idx_load_from_disk] read 0 ALT contigs\n[M::process] read 25000 sequences (2525000 bp)...\n[M::mem_process_seqs] Processed 25000 reads in 0.709 CPU sec, 0.709 real sec\n[main] Version: 0.7.17-r1188\n[main] CMD: bwa mem -t 1 data/genome.fa data/samples/B.fastq\n[main] Real time: 0.982 sec; CPU: 0.735 sec\n[Fri May 14 10:56:57 2021]\nFinished job 5.\n1 of 7 steps (14%) done\n\n[Fri May 14 10:56:57 2021]\nrule sort:\n    input: mapped/B.bam\n    output: mapped/B.sorted.bam\n    jobid: 4\n    wildcards: sample=B\n\nRemoving temporary output file mapped/B.bam.\n[Fri May 14 10:56:58 2021]\nFinished job 4.\n2 of 7 steps (29%) done\n\n[Fri May 14 10:56:58 2021]\nrule bwa:\n    input: data/genome.fa, data/samples/A.fastq\n    output: mapped/A.bam\n    jobid: 3\n    wildcards: sample=A\n\n[M::bwa_idx_load_from_disk] read 0 ALT contigs\n[M::process] read 25000 sequences (2525000 bp)...\n[M::mem_process_seqs] Processed 25000 reads in 0.707 CPU sec, 0.706 real sec\n[main] Version: 0.7.17-r1188\n[main] CMD: bwa mem -t 1 data/genome.fa data/samples/A.fastq\n[main] Real time: 0.962 sec; CPU: 0.732 sec\n[Fri May 14 10:57:00 2021]\nFinished job 3.\n3 of 7 steps (43%) done\n\n[Fri May 14 10:57:00 2021]\nrule sort:\n    input: mapped/A.bam\n    output: mapped/A.sorted.bam\n    jobid: 2\n    wildcards: sample=A\n\nRemoving temporary output file mapped/A.bam.\n[Fri May 14 10:57:01 2021]\nFinished job 2.\n4 of 7 steps (57%) done\n\n[Fri May 14 10:57:01 2021]\nrule call:\n    input: data/genome.fa, mapped/A.sorted.bam, mapped/B.sorted.bam\n    output: calls/all.vcf\n    jobid: 1\n\n[warning] samtools mpileup option `g` is functional, but deprecated. Please switch to using bcftools mpileup in future.\n[mpileup] 2 samples in 2 input files\nNote: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid\n[Fri May 14 10:57:02 2021]\nFinished job 1.\n5 of 7 steps (71%) done\n\n[Fri May 14 10:57:02 2021]\nrule stats:\n    input: calls/all.vcf\n    output: plots/quals.svg\n    jobid: 6\n\n[Fri May 14 10:57:03 2021]\nFinished job 6.\n6 of 7 steps (86%) done\n\n[Fri May 14 10:57:03 2021]\nlocalrule all:\n    input: calls/all.vcf, plots/quals.svg\n    jobid: 0\n\n[Fri May 14 10:57:03 2021]\nFinished job 0.\n7 of 7 steps (100%) done\nComplete log: /home/vanessa/Desktop/Code/snakemake/tutorial/.snakemake/log/2021-05-14T105654.746372.snakemake.log\n```\n\nAfter running you will see result data files in newly generated folders mapped, calls, and plots!\n\n```bash\n$ tree -L 1\n.\n├── calls\n├── data\n├── envs\n├── mapped\n├── plots\n├── README.md\n├── scripts\n└── Snakefile\n```\n\nThe resulting plot is kept alongside this repository to show you:\n\n[img/quals.svg](img/quals.svg)\n\nWhat will happen when you run the workflow with `--use-spack` is that a spack\nview will be created for each of your environments. This means that a `spack install`\nwill be run for each environment to create the executables and other supporting\nlibraries. We can peek into the hidden .snakemake directory and see our spack\nenvironments:\n\n```bash\n$ tree .snakemake/spack\n.snakemake/spack\n├── 4ff8d68292f72d69cfaa19e8a6ddff03\n│   ├── env_setup_done\n│   ├── env_setup_start\n│   ├── spack.lock\n│   └── spack.yaml\n├── 6433cc1f09897cb43ae31919ad8e1892\n│   ├── env_setup_done\n│   ├── env_setup_start\n│   ├── spack.lock\n│   └── spack.yaml\n└── 827d39341264ddaff183ebb496b21eb2\n    ├── env_setup_done\n    ├── env_setup_start\n    ├── spack.lock\n    └── spack.yaml\n```\n\nWithin each folder named by the hash, you see the actual spack view (which\nmimics a small filesystem with include, bin, lib, etc.\n\n```bash\n.snakemake/spack/827d39341264ddaff183ebb496b21eb2/.spack-env/\n├── repos\n│   └── builtin\n├── transaction_lock\n└── view\n    ├── bin\n    ├── docs\n    ├── etc\n    ├── include\n    ├── lib\n    ├── libexec\n    ├── man\n    ├── sbin\n    └── share\n```\n\nNote that although spack has support for copy/relocate of environments, the path\nis limited in its length, so already in the snakemake metadata directory\nwith a hash for the name, it's really too long to feasibly try to support.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnakemake%2Fsnakemake-spack-tutorial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnakemake%2Fsnakemake-spack-tutorial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnakemake%2Fsnakemake-spack-tutorial/lists"}