{"id":26329201,"url":"https://github.com/murrellgroup/porpidpipeline","last_synced_at":"2026-01-02T10:54:13.884Z","repository":{"id":70649768,"uuid":"519796318","full_name":"MurrellGroup/PORPIDpipeline","owner":"MurrellGroup","description":"A pipeline for long-read UMI/PID sequencing of HIV amplicons.","archived":false,"fork":false,"pushed_at":"2024-12-22T18:41:29.000Z","size":91047,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-12-22T19:36:58.571Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Julia","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MurrellGroup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-31T14:18:55.000Z","updated_at":"2024-01-03T20:47:06.000Z","dependencies_parsed_at":"2024-12-09T18:31:14.042Z","dependency_job_id":null,"html_url":"https://github.com/MurrellGroup/PORPIDpipeline","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MurrellGroup%2FPORPIDpipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MurrellGroup%2FPORPIDpipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MurrellGroup%2FPORPIDpipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MurrellGroup%2FPORPIDpipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MurrellGroup","download_url":"https://codeload.github.com/MurrellGroup/PORPIDpipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243790971,"owners_count":20348386,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-15T21:19:44.721Z","updated_at":"2026-01-02T10:54:13.878Z","avatar_url":"https://github.com/MurrellGroup.png","language":"Julia","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PORPIDpipeline\n\nby Alec Pankow and Ben Murrell, now maintained by Hugh Murrell\n\nnow upgraded to Julia version 1.10.5\n\n## Branch: master (including the new artefactfilter)\n\n## Quick start\n\n### Dependencies (on an ubuntu machine)\n\n- first update all apps\n   - `apt update`\n   - `apt upgrade`\n- Snakemake\n   - `apt-get install -y snakemake`\n- python3 packages\n  - `apt-get install python3-pandas`\n  - `apt-get install python3-seaborn`\n\n\n### Julia version 1.10.5\n\nWe recommend you use the `juliaup` version manager to install julia.\nfrom a terminal you can do this as follows:\n\n```bash\ncurl -fsSL https://install.julialang.org | sh\n```\n\nThis should install the Julia version manager, `juliaup` as well as\nthe latest version of Julia. To find out how to use the version manager \nto makesure you have version 1.10.5 as your default, go here:\n\n[https://github.com/JuliaLang/juliaup]\n\nOnce Julia is installed, make sure you can enter the julia REPL from \nthe command line and check the version number by logging out and in again \n\n```bash\nexit\nssh root@.....\n```\n\nand then from your new terminal session:\n\n```bash\njuliaup status\njulia --version\n```\n\nIf the version number is not 1.10.5 then you need to use `juliaup` to install\nthat version and make it the default. \n\n```bash\njuliaup add 1.10.5\njuliaup default 1.10.5\n```\n\nfor further details concerning `juliaup` go here:\n\n[https://github.com/JuliaLang/juliaup?tab=readme-ov-file#using-juliaup]\n\n### cloning the PORPIDpipeline repository\n\nNow that the dependencies are setup we clone the PORPIDpipeline repository\n\n```bash\ngit clone https://github.com/MurrellGroup/PORPIDpipeline.git\n```\n\n### setting up the Julia package environment\n\nthen navigate to the `PORPIDpipeline` project folder and start the Julia REPL. \nEnter the package manager using `]` and then enter\n\n```julia\nactivate .\ninstantiate\nprecompile\n```\n\nThis will activate, install, and precompile the `julia` environment specified by the \n`Project.toml` and `Manifest.toml` files. The `precompile` command\nabove is not strictly needed but is useful if there are issues with installing\nthe `julia` packages listed in `Project.toml`\n\n### Workflow\n\nThe graph below summarizes the overall organization of the workflow. \nEach node in the graph is a *rule* in the The [Snakefile](Snakefile).\n\n![rulegraph](rulegraph.png)\n\nSnakemake makes use of a `Snakefile` and a `config` file to specify \ninput and output files for each rule and to set parameters for each \nrule in the workflow. Global parameters that can be changed by the \nuser editing the `Snakefile` are as follows:\n\n```\n# PORPIDpipeline parameters\n# demux\nchunk_size = 100000      # default 100000\nerror_rate = 0.01        # default 0.01\nmin_length = 2100        # default 2100\nmax_length = 4300        # default 4300\nmax_reads = 100000       # default 100000 reads per sample,\nverbose = \"false\"        # default \"false\", use \"true\" to debug demux\n#porpid\nfs_thresh = 1            # default 1 (or use 5 if af_thresh is 0)\nlda_thresh = 0.995       # default 0.995\n#consensus\nagreement_thresh = 0.7   # default 0.7\naf_thresh = 0.35         # default 0.35 (drops smallest 35% of CCS reads)\n#contam\ncluster_thresh = 0.015   # default 0.015\nproportion_thresh = 0.2  # default 0.2\ndist_thresh = 0.015      # default 0.015\ncontam_toggle = \"on\"     # default \"on\", use \"off\" to disable\n#postproc\npanel_thresh = 50        # default 50\n#tar\ndegap = \"true\"           # default \"true\", use \"false\" to disable\ncollapse = \"true\"        # default \"true\", use \"false\" to disable\nporpid_archive = \"full\"  # default \"full\", use \"part\" for partial archive\n```\n\nNote that with the advent of PacBio Revio sequencer, the number of reads\nper sample has grown to outstrip memory available on standard CPUs. \nTo enable a trouble free pipeline run, we now allow the user to specify\nthe maximum number of reads per sample using the `max_reads` parameter above.\nSamples with reads exceeding this limit are then randomly sub-sampled to\nreduce the number of reads accordingly.\n\nWe also introduce the option of a **partial** archive of the intermediate\n**porpid** directory. This option is made available to ameliorate the \ninordinately long time it can take to archive and gzip the huge directory\nstructures produced when processing PacBio Revio datasets.\n\n### Sample configuration\n\nParameters for each sample are provided in the `config.yaml` file. This file\nshould reflect your library construction, amplicon identity and any override\nparameter settings. \n\nIt should follow the same format shown in the **demo** example below.\n\n```yaml\ndemo:\n  donor_1_REN:\n    cDNA_primer: CCGCTCCGTCCGACGACTCACTATAacagtgNNNNNNNNGTCATTGGTCTTAAAGGTACCTG\n    sec_str_primer: TAGGCATCTCCT\n    panel: \"panels/HIV1_COM_2017_5970-8994_DNA_stripped.fasta\"\n  donor_2_REN:\n    cDNA_primer: CCGCTCCGTCCGACGACTCACTATAcactcaNNNNNNNNGTCATTGGTCTTAAAGGTACCTG\n    sec_str_primer: TAGGCATCTCCT\n    panel: \"panels/HIV1_COM_2017_5970-8994_DNA_stripped.fasta\"\n    af_override: 0.4\n  donor_3_REN:\n    cDNA_primer: CCGCTCCGTCCGACGACTCACTATAggtagcNNNNNNNNGTCATTGGTCTTAAAGGTACCTG\n    sec_str_primer: TAGGCATCTCCT\n    panel: \"panels/HIV1_COM_2017_5970-8994_DNA_stripped.fasta\"\n  donor_1_GP:\n    cDNA_primer: CCGCTCCGTCCGACGACTCACTATAacagtgNNNNNNNNGTATGTCATTGACAGTCCAGC\n    sec_str_primer: TTGACTAGCGGAGGCTAGAAGGAGA\n    panel: \"panels/HIV1_COM_2017_787-3300_DNA_stripped.fasta\"\n    af_override: 0.3\n  donor_2_GP:\n    cDNA_primer: CCGCTCCGTCCGACGACTCACTATAcactcaNNNNNNNNGTATGTCATTGACAGTCCAGC\n    sec_str_primer: TTGACTAGCGGAGGCTAGAAGGAGA\n    panel: \"panels/HIV1_COM_2017_787-3300_DNA_stripped.fasta\"\n    af_override: 0.0\n    fs_override: 14\n  donor_3_GP:\n    cDNA_primer: CCGCTCCGTCCGACGACTCACTATAggtagcNNNNNNNNGTATGTCATTGACAGTCCAGC\n    sec_str_primer: TTGACTAGCGGAGGCTAGAAGGAGA\n    panel: \"panels/HIV1_COM_2017_787-3300_DNA_stripped.fasta\"\n```\n\nNote that the **donor ID barcode** is in lowercase\nand the **Unique Molecular Identifier (UMI) barcode** is indicated with N's. \nThe primer sequences provided will be used for demultiplexing and will be trimmed\nfrom the final sequences. \n\nThe **panel** arg should be a path to a `.fasta` alignment spanning your amplicon, \nwith all gaps stripped. This will be used only in the postproccessing step to remove \noff-target seqs and trim to the correct coordinates.\n\nTo generate your own panel file you are encouraged to visit:\n\n[https://www.hiv.lanl.gov/content/sequence/NEWALIGN/align.html](https://www.hiv.lanl.gov/content/sequence/NEWALIGN/align.html)\n\nwhere you can download an alignment and then use `aliview` to trim \nthe alignment to your region of interest.\n\ngzipped CCS .fastq files should be placed in the `raw-reads/` subdirectory and named \naccording to the the dataset name used in the `config.yaml` file, ie, `demo.fastq.gz`\nfor the *demo* dataset.\n\nWe also use the `config` file to allow an **override** for a particular sample, \nof the default artefact filter threshold, and/or, the default family size filter. \nSee the demo config file above.\n\n### Preview and Execution\n\nPreview jobs with Snakemake and run with {n} cores.\n\n```bash\n#preview jobs\nsnakemake -np\n\n#run\nsnakemake -j{n}\n\n#in the cloud\nnohup snakemake -j{n}\u0026\ncat nohup.out\n```\n\nFor more info on Snakemake, see:\n\n[https://snakemake.readthedocs.io/en/stable/]\n\n## Conda setup\n\nSome (without root access) may prefer to setup PORPIDpipeline in a **conda** environment.\n\nTo accomplish this, first install `anaconda` locally. (the install script allows you to choose\nthe location for anaconda, by default `/home/user` but choose something else if\nyou want something accessable to a group of users)\n\n```bash\ncurl –O https://repo.anaconda.com/archive/Anaconda3-2021.05-Linux-x86_64.sh \u003e Anaconda3-2021.05-Linux-x86_64.sh\nbash Anaconda3-2021.05-Linux-x86_64.sh\n```\n\nthen log out and log in again and check that you are in the `base` environment.\n\n`conda` is very slow, so we suggest installing `mamba` in the conda `base` environment:\n\n```bash\nconda install -n base -c conda-forge mamba\n```\nclone the PORPIDpipeline repository\n\n```bash\ncd ~  # or some other directory used for your anaconda installation\ngit clone https://github.com/MurrellGroup/PORPIDpipeline.git\n```\n\nand then all the PORPIDpipeline dependencies including `julia` version `1.10.5`\n( as listed in the PORPIDpipeline conda environment spec in `environment.yaml`),\ncan be installed in a `conda` environment via `mamba` using the commands:\n\n```bash\nconda config --add channels conda-forge\nconda config --add channels bioconda\nmamba env create --file environment.yaml\n```\n\nNote that if you did use *some other directory* than your home directory for\ninstalling the PORPIDpipeline repository then you have to inform Julia where\nyour packages are stored by placing the following command in your `.bashrc`\nfile:\n\n```bash\n# set path to .julia files\nexport JULIA_DEPOT_PATH=\"/some/other/directory/.julia\"\n```\n\nto complete the setup, activate the new PORPIDpipeline conda environment, \n\n```bash\nconda activate PORPIDpipeline\n```\n\nand continue with the `julia` package environment setup as outlined above in the *quick start* section.\n\n## Cluster setup\n\nSeting up a `snakemake` pipeline on a cluster is a *dark* art. Here we describe an attempt\nat installing PORPIDpipeline on a two node cluster, (one node a *controller* node with 16 cores\nand the other node a *compute* node with 64 cores).\n\n**Firstly**, since the cluster administrator is hardly likely to give you root access we\nsuggest you follow the `conda` installation for PORPIDpipeline. If you expect more\nthan one user of your PORPIDpipeline then install in a directory that all\nyour users can see and that is visible from both the *contoller* and *compute* nodes. \nie use `some other directory` rather than the standard home directory and make \nsure to inform `julia` about this choice of directory as\noutlined in the `conda` section above.\n\n**Secondly**, cluster administrators usually insist that large data sets are stored\nin an appropriate volume and **not** in the usual user's space. On our cluster the\nadministrator required the PORPIDpipeline code to be installed in a `\\tools\\porpid\\`\ndirectory and the large data sets (input, output and temporary) to be stored in\na `\\data\\porpid\\` directory so we installed PORPIDpipeline into `\\tools\\porpid\\porpidpipeline`\nand then replaced some of the directories in the `porpidpipeline`\ndirectory with symbolic links to an appropriate directory in the `\\data\\porpid\\` directory\nas shown below\n\n```\nconfig.yaml -\u003e /raw/porpid/config/demo.yaml\npanels -\u003e /raw/porpid/panels/\nporpid -\u003e /raw/porpid/porpid/\npostproc -\u003e /raw/porpid/postproc/\nraw-reads -\u003e /raw/porpid/raw-reads/\n```\n\nNaturally, one must copy contents of the installation to the `/raw/porpid/` directory\nbefore deleting the installation directory and replacing it with a symbolic link to the\nappropriate place on the `raw` volume.\n\n**Job submission**, after setting up like this we are ready to run the `demo` study through \nthe PORPIDpipeline\nby submitting the `snakemake` command to the cluster managemant system.\nOn our cluster that management system is `slurm` and the following shell script\nstored in `porpid_job.sh` facilitated that submission:\n\n```bash\n#!/bin/bash\n#SBATCH --job-name==porpid\n#SBATCH --time=1:0:0\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=7\n#SBATCH --partition=main\n\nif [ \"$#\" -lt 1 ]; then\n    echo \"please supply a config file name as first parameter\"\n    exit\nfi\necho \"config file is $1\"\n\necho \"${SLURM_JOB_NAME} job submited using ${SLURM_NTASKS} cores\"\n\n# create a symbolic link for the snakemake config file to point to the config for the current study\nrm -f /tools/PORPIDpipeline/porpidpipeline/config.yaml\nln -s /RAW/PORPID/CONFIG/$1.yaml /tools/PORPIDpipeline/porpidpipeline/config.yaml\n\n# tell slurm where anaconda is and conda activate the PORPIDpipeline environment\nsource /tools/PORPIDpipeline/anaconda3/etc/profile.d/conda.sh\nconda activate PORPIDpipeline\n\n# navigate to the porpidpipeline directory and run snakemake\n# add -F to to the snakemake command to force re-run of all rules\ncd /tools/PORPIDpipeline/porpidpipeline\nsnakemake --rerun-incomplete -j${SLURM_NTASKS}  \n```\n\nTo submit the `demo` to run as a `slurm` batch job one just uses\n\n```bash\nsbatch porpid_job.sh demo\n```\nThe script above sets some environment variables for `slurm` and then resets\nthe symbolic link to the appropriate config file for the `demo` study.\nIt then activates the conda environment switches to the installation\ndirectory and runs the snakemake pipeline.\n\nWith this structure it is easy to run a new study through PORPIDpipeline.\nOne copies the new config file into the `/raw/porpid/config/` directory,\ntransfers the `fastq` data to the `/raw/porpid/raw-reads/` directory\nand then issues the `sbatch` command using the appropriate study name\ninstead of `demo`\n\nNote that with this method you must predetermine the number of cores\nyou intend to use on your cluster's node. In the `demo` study this is set\nto 7 ( 6 cores for the samples to run in parallel plus 1 core for snakemake )\n\nEach study will be different. To see how many samples can be run in parallel\nyou can do a `snakemake` dry run using the `porpid_dry_run.sh` script below:\n\n```bash\n#!/bin/bash\nif [ \"$#\" -lt 1 ]; then\n        echo \"please supply a config file name as first parameter\"\n        exit\nfi\necho \"config file is $1\"\n# create a symbolic link for the snakemake config file to\n# point to the config for the current study\nrm -f /tools/PORPIDpipeline/porpidpipeline/config.yaml\nln -s /RAW/PORPID/CONFIG/$1.yaml /tools/PORPIDpipeline/porpidpipeline/config.yaml\n# activate the conda environment\nsource /tools/PORPIDpipeline/anaconda3/etc/profile.d/conda.sh\nconda activate PORPIDpipeline\n# perform a snakemake dry run\n# remove the -f for a partial dry run of what's left to do\ncd /tools/PORPIDpipeline/porpidpipeline\nsnakemake -F --rerun-incomplete -np\n```\n\nNote that this dry run is not compute intensive and can ve executed on the\n*controller* machine without using the `sbatch` command as follows:\n\n```bash\n./porpid_dry_run.sh demo\n```\n\n### Caveat\n\nThe above suggestion for running a `snakemake` pipeline under `slurm`\nis rudamentary. Maximum cores must be requested at the start of execution\nand they are probably held throughout the run.\n\nHowever, it is alledged that `snakemake` can play nicely with `slurm` and\nit should be possible to have `snakemake` invoke `slurm` for each rule in\nthe pipeline. In this case `snakemake` would request the optimal number\nof cores needed for each step in the pipeline.\n\nWe have not attempted this yet, and it would probably require writing a\n`slurm` efficient version of the `snakefile`. \n\n## Documentation\n\n\n\nAn introduction to PacBio sequencing and an explanation for each *PORPIDpipeline* rule \nis given in the set of introductory slides packaged with this repository.\n[docs/slides/PORPIDpipeline.pdf](docs/slides/PORPIDpipeline.pdf)\n\nTo understand how this pipeline was designed and tested please read our\n[Optimized SMRT-UMI protocol](https://academic.oup.com/ve/article/10/1/veae019/7640831)\npaper.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmurrellgroup%2Fporpidpipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmurrellgroup%2Fporpidpipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmurrellgroup%2Fporpidpipeline/lists"}