{"id":13835791,"url":"https://github.com/rc-biotech/massiveNGSpipe","last_synced_at":"2025-07-10T12:33:00.696Z","repository":{"id":159477530,"uuid":"575149552","full_name":"rc-biotech/massiveNGSpipe","owner":"rc-biotech","description":null,"archived":false,"fork":false,"pushed_at":"2024-06-11T12:20:10.000Z","size":748,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-08-05T15:04:25.563Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rc-biotech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-06T21:38:21.000Z","updated_at":"2024-06-11T12:20:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"e791fa94-0aad-4c61-8eae-4ef046b127e4","html_url":"https://github.com/rc-biotech/massiveNGSpipe","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rc-biotech%2FmassiveNGSpipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rc-biotech%2FmassiveNGSpipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rc-biotech%2FmassiveNGSpipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rc-biotech%2FmassiveNGSpipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rc-biotech","download_url":"https://codeload.github.com/rc-biotech/massiveNGSpipe/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225638889,"owners_count":17500647,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T15:00:19.509Z","updated_at":"2024-11-20T22:30:58.085Z","avatar_url":"https://github.com/rc-biotech.png","language":"R","funding_links":[],"categories":["Data Processing and Analysis"],"sub_categories":[],"readme":"\n![](inst/images/massiveNGSpipe_overview.png)\n\nThis package is under heavy development to include more features, API is not\nstable!\n\n### About\n\nmassiveNGSpipe is a R package for full process integration of\nNGS studies, it can handle thousands of samples from multiple organisms\nin a single run.\n\nThe pipelines flow is built on 4 steps:\n\n- 1. Set up pipeline config (What steps to do: download, trimming, alignmnets) (presets supported)\n- 2. Curate data (define which samples you want to run \u0026 metadata)\n- 3. Download organism annotations from linked metadata per study (set up pipeline objects)\n- 4. Run pipeline\n\nIt currently supports to set up easily a fail safe and fallback supported \npipeline that does the specific steps:\n- Find candidate accessions for studies you might be interested in\n- A clean way of helping with annotating samples using\neither google sheet integration or local csv files.\n- Automatic download of genome and gff for each organism (supports\nfixing malformed gffs, adding pseudo 5' UTRs, contamants etc)\n- Automatic download of samples with fallback options:\n 1. AWS (amazon, fastest), 2. ENA (fast), 3. fastq-dump (slowest)\n- Automatic detection of 3' adapters and trimming (fastp)\n- Collapsing of duplicated reads for faster processing\n- barcode / UMI detection / removal (To be implemented)\n- Autmoatic removal of contaminants: phix, rRNA, tRNA,..\n- Genome alignment with STAR (with QC)\n- Create ORFik.experiment object for easy analysis in R\n- Convert output files to any format: bigwig, wig, bed, ofst etc.\n- Merging of samples by type\n\nRibo-seq specific:\n- automatic pshifting and quality validation\n\nVisulization specific:\nOur data output is directly supported to be browsed by the\nRiboCrypt visuzliation tool which supports among others:\n- Both genomic and transciptomic view browser\n- Differential expression\n- Read length heatmaps\n- Codon usage\n- And much more\n\n### Primary pipeline (Data from online repository)\n\nPackage is also available here on github\n```r\nlibrary(massiveNGSpipe)\n## Setup\nconfig \u003c- pipeline_config(preset = \"Ribo-seq\") # \u003c- set up paths\n## Curate metadata\naccessions \u003c- accessions_to_use(\"GSE152850\", \"Saccharomyces cerevisiae\", FALSE)\ncurate_metadata(accessions, config, \"Saccharomyces cerevisiae\")\n## Init pipeline and show current progress report\npipelines \u003c- pipeline_init_all(config) # Initialize pipeline configuration for all experiments\nprogress_report(pipelines, config)\n# Start the pipeline (In RStudio start this step with the 'background jobs' tab)\nrun_pipeline(pipelines, config, wait = 100, BPPARAM = bpparam())\n```  \n\n### Installation\n\nPackage is currently only available here on github\n```r\nif (!requireNamespace(\"devtools\", quietly=TRUE))\n    install.packages(\"devtools\")\ndevtools::install_github(\"rc-biotech/massive_NGS_pipe\")\n```  \n\n### Requirements\n\nSoftware\n\n- R programming language\n- Unix OS (Linux or macOS), (or Windows Subsystem for Linux (WSL2))\n\nHardware\n\n- Large genomes (like human, mouse, ...): (minimum 32GB memory)\n- For small genomes (Yeast, E.coli, ...): (minimum 16GB memory)\n\nRunning with many cores will use more memory, we have seen\ncases of a 800 sample study using up to 120GB memory on a large server when using \u003e 50 cores.\n\n\n### Local pipeline (Data from personal hard drive / local server)\n\nTo run your local data, you run the pipeline, but skips the steps of \ndownloading. You also need to annotate the basic sample metadata yourself,\nsince it does not exist online yet. \n\n```r\nlibrary(massiveNGSpipe)\nconfig_dirs \u003c- ORFik::config() # Where all data is stored (all your pipelines)\n# -\u003e project_dir specifies log, status and metadata folder for specific pipeline\nproject_dir \u003c- file.path(dirname(config_dirs)[1], \"test_pipeline\")\nmode \u003c- \"local\" # Data already exists on your computer / server\ngoogle_url \u003c- NULL # Do not use google sheet for metadata storage\npreset \u003c- \"Ribo-seq\" # What kind of pipeline (use Ribo-seq preset)\n\n# Init config (set up paths and pipeline settings)\nconfig \u003c- pipeline_config(project_dir, config_dirs, mode = mode,\n                          preset = preset, google_url = google_url)\n\n# Now, for local data you need to define a data.table with local file metadata\n# Do this for each experiment (split experiment for each organism)\nname \u003c- \"local_mNGSp_run_test\" # This is the folder name in the fastq folder\norganism \u003c- \"Homo sapiens\"\norganism_caps \u003c- gsub(\" \", \"_\", trimws(tolower(organism)))\npath \u003c-  paste(name, organism_caps, sep = \"-\")\npath_full \u003c- file.path(config$config[\"fastq\"], path)\nstopifnot(basename(path) == path) # relative path only!\nstopifnot(dir.exists(path_full)) # Must exist!\n\nsave_file \u003c- file.path(config[[\"metadata\"]],\n                       paste0(\"SraRunInfo_\", name, \".csv\"))\n# Add all information needed here\ndt \u003c- local_study_csv(path_full, name, organism = organism,\n                      paired = \"SINGLE\", AUTHOR = \"Håkon\",\n                      sample_title = c(\"WT_ribo_cycloheximide_rep1\",\n                                       \"WT_ribo_cycloheximide_rep2\"))\ndir.create(dirname(save_file), FALSE, TRUE)\nfwrite(dt, save_file)\n\n# Now for all experimental directories, validate metadata\n# massiveNGSpipe requires a unique set of rows for metadata to be valid\nall_names \u003c- c(name) # Add all here (here we did only 1)\n# Now run this, when you get to validation either open your csv and set\n# the KEEP column to TRUE, for sample you want to run.\n# Or do as bellow to run and update KEEP column in R\ncurate_metadata(all_names, config)\n\n# data.table way of setting samples to run.\ntemp \u003c- fread(config$temp_metadata)\ntemp$KEEP \u003c- TRUE\nfwrite(temp, config$complete_metadata)\ncurate_metadata(all_names, config)\n\n# Now init pipeline objects and run pipeline\npipelines \u003c- pipeline_init_all(config)\n\nrun_pipeline(pipelines, config, wait = 20, BPPARAM = SerialParam())\n```\n\n### The flow of data through the pipeline (theory)\n\nThe pipeline consists of several steps where the user can add their custom\nsteps. Here we will walk through the 4 main steps:\n\n\n\n#### pipeline_config (Where should the pipeline save stuff, what functions to run, how to get data)\n\npipeline_config is the first function you will call to init a new pipeline.\n\nIt sets up 3 main steps:\n\n- Directories (for data and metadata (with google sheet integration))\n- Steps of action (function to be done, and how they are grouped), we call these\n'flags' and each flag has a function to implements the logic.\n- Setup for Parallel processing and how to log errors.\n\nImportant to remember is that the pipeline has 2 main modes:\n- 'online': for online repository data\n- 'local': for local machine / server data\n\nSince the input steps are just a set of input function, you as a user can fully\ncustomize the pipeline. \n\n#### curate_metadata (Which samples to run and what are they)\n\ncurate_metadata is the function to set up and validate metadata.\nThere are 3 main steps:\n\nFor each study you have, per organism, we define a data structure:\n\n- Which samples to use: For online mode, samples are identified by Run ids (SRR, ERR, DRR,...) and study id will be the bioproject id, SRP, etc.,\nfor local mode, samples are identified by relative file names, and studies are \nfolders (one folder per organism, if local study is multi-species).\n- Metadata extraction: When primary metadata is set up (where are the samples, which organism), we need to make sure we have a unique experimental identifier per sample. Like Ribo-seq Wild type cyclohexamide rep1. \n- Automatic validation: We then run through a automatic validation\nof the metadata and the user will be informed if the information is not sufficient. \n\n#### pipeline_init_all (Create pipeline objects (per study per organism), download genomes)\n\nWhen the curate_metadata step is complete and you have got a valid set of input data, you are good to go. This steps sets up the pipeline object: the object you actually feed into the main run. \n\nIt takes the config and the completed metadata table, it then sets up 3 important things:\n- Where should raw files, processed files and experiments be stored for this specific study?\n- Which genome and index does it use?\n- What is the metadata for this specific study?\n\n#### run_pipeline (Start processing)\n\nWe advice you to always run this in a background job in RStudio, \nto \nrun_pipeline uses the results from the parts:\n- pipeline_init_all object (all data needed per study/organism to run )\n- pipeline_config (the pipeline config, what to do etc.)\n\nIt then runs each function in a safe wrapper that if it fails it stores\nerror to a log and let's you know. It then continues to the next pipeline object.\nWhen all objects are done it will retry the ones that failed.\n\nYou can also get a progress report of how long you have come. \n\n### The flow of data through the pipeline (code examples)\n\nThe pipeline consists of several steps where the user can add their custom\nsteps. Here we will walk through the 4 main steps with code examples (theory described in section above):\n\n#### Running custom functions\n\nTemplate for new functions\n```r\n# If you want to work on finished ORFik experiments\npipe_doX \u003c- function(pipelines, config, step = \"doX\") {\n  for (pipeline in pipelines) { # Per pipeline object\n    try \u003c- try({\n      # For each species in study\n      df_list \u003c- lapply(experiments, function(e) read.experiment(e, validate = FALSE, output.env = new.env()))\n      # Your code here ----\u003e\n      for (df in df_list) {\n        # First skip if step is done\n        if (!step_is_next_not_done(config, step, name(df))) next\n        # Do something\n        # Then set flag when done\n        set_flag(config, step, name(df))\n      }\n    })\n    if (is(try, \"try-error\"))\n      warning(\"Failed at step, \", step ,\", study: \", pipeline$accession)\n  }\n}\n\n# If you want to work on pipeline objects\npipe_count_reades \u003c- function(pipelines, config, step = \"countReads\") {\n  for (pipeline in pipelines) { # Per pipeline object\n    try \u003c- try({\n      conf \u003c- pipeline$organisms[[organism]]$conf\n      if (!step_is_next_not_done(config, step, conf[\"exp\"])) next\n      \n      index \u003c- pipeline$organisms[[organism]]$index\n      source_dir \u003c- conf[\"fastq\"]\n      runs \u003c- study[ScientificName == organism]\n      # Files to run (Single end / Paired end)\n      all_files \u003c- massiveNGSpipe:::run_files_organizer(runs, source_dir)\n      lengths \u003c- sapply(all_files, function(file) length(readDNAStringSet(file)))\n      saveRDS(file.path(source_dir, \"count_reads.rds\"))\n      }\n    })\n    if (is(try, \"try-error\"))\n      warning(\"Failed at step, \", step ,\", study: \", pipeline$accession)\n  }\n}\n```\n\nActuall example:\n```r\nlibrary(massiveNGSpipe)\n\n# Setting up custom pipelines\nconfig_dirs \u003c- ORFik::config() # Where all data is stored (all your pipelines)\n# -\u003e project_dir specifies log, status and metadata folder for specific pipeline\nproject_dir \u003c- file.path(dirname(config_dirs)[1], \"local_pipeline_ribo\")\nmode \u003c- \"local\" # Data already exists on your computer / server\ngoogle_url \u003c- NULL # Do not use google sheet for metadata storage\npreset \u003c- \"Ribo-seq\" # What kind of pipeline (use Ribo-seq preset)\n# Init config (set up paths and pipeline settings)\nconfig \u003c- pipeline_config(project_dir, config_dirs, mode = mode,\n                          preset = preset, google_url = google_url,\n                          keep_contaminants = TRUE,\n                          keep.unaligned.genome = TRUE)\n                          \n# Adding new step\n\n# 3 things: flag_id name, name of function, and the function itself\n\n#' Differential expression analysis on primary contrast\n#' @inheritParams run_pipeline\n#' @param step name_id of current step\npipe_difexp \u003c- function(pipelines, config, step = \"difexp\") {\n  for (pipeline in pipelines) { # Per pipeline object\n    try \u003c- try({\n      # For each species in study\n      df_list \u003c- lapply(experiments, function(e) read.experiment(e, validate = FALSE, output.env = new.env()))\n      # Your code here ----\u003e\n      for (df in df_list) {\n        # First skip if step is done\n        if (!step_is_next_not_done(config, step, name(df))) next\n        DEG.analysis(df)\n        # Then set flag when done\n        set_flag(config, step, name(df))\n      }\n    })\n    if (is(try, \"try-error\"))\n      warning(\"Failed at step, \", step ,\", study: \", pipeline$accession)\n  }\n}\n\nflag_id \u003c- \"difexp\" # The id of the flag step\n# By default appends as last step, you can also set new steps to be appended in front.\nnew_config \u003c- add_step_to_pipeline(config, flag_id, function_name, FUN = pipe_difexp)\npipeline_init_all(config)\n```\n\n# Overview of making metadata for massiveNGSpipe:\n\n\n## Metadata introduction\nSamples need to be structured be able to run through a pipeline.\n\nMore specifically, we require a minimum set of datapoints, and \nfor each library (Run), the set of datapoints must be unique.\n\n### ENTREZ run summary\n\nThe way we do this in is to use a csv file (a google sheet):\n\nIt is structured in a following way:\n\nFor each study we get the run summary, a csv file from the ENTREZ database:\n\nEach row in the run summary represents a run (a fastq file or a pair of fastq files in case of paired-end)\n\nThese are the important columns:\n\n- Run number: SRR, ERR, DRR (Run)\n- Study accession (BioProject)\n- Single end or paired end (LibraryLayout)\n- Organism it came from (ScientificName)\n\nTo gather what the sample is:\n\n- library type (e.g. Ribo-seq, RNA-seq)\n- condition (e.g. wild type, knock out or overexpression)\n- timepoint (e.g. 2 days post fertilization, 2 hours of drug treatment)\n- replicate (e.g. replicate 1, replicate 2, ...),\n\nWe need to know where to find these datapoints.\n\nHere we order columns to look at in probability of finding useful information:\n\n- sample_title (Sample title given by author)\n- sample_source (Source of sample given by author)\n- LibraryName (Name of specific library, this is usually empty)\n\nIf not existing there, then we sadly have not enough data, and need to do a manual check in the web-page for now:\n\nWe advice using geo like this:\n\nhttps://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM2664262\n\nWhere GSM... in ?acc=GSM is the GSM id found in the SampleName column.\n\n## Automatic metadata detection with massiveNGSpipe\n\nWhen you start curating data, massiveNGSpipe will try to auto-guess some columns with \nlook up rules using regex and uniqueness of runs.\n\nThese columns will be added:\n- LIBRARYTYPE\t(RNA-seq, Ribo-seq, ..)\n- REPLICATE\t(Minimum index seperator, rep1, rep2, rep3,..)\n- CONDITION\t(WT (wild type), KO (knock out), mut (mutant),...)\n- INHIBITOR\t(chx (cyclohexamide), harr (harringtonin), ...)\n- BATCH\t(maximum index seperator, 1, 2, 3, ..) \n- TIMEPOINT\t(2h (2 hours), 1d (1 day), ...)\n- TISSUE\t(kidney, liver, heart, ...)\n- CELL_LINE\t(HEK293, HeLa, ...)\n- GENE\t\t(RPL11, TOR, ATF4, ...)\n- FRACTION\t(Remaining uniqueness seperator: cytosolic, nuclear, dmso, silvesterol, ...)\n\nThe following is the goal of manual annotation:\nPer study, per organism: Get a unique set of the above columns.\nIf 2 rows (Runs) are identical, the validation test should fail.\n\n\nSome additional rules:\n\nBATCH vs replicate:\nIf there are technical replicates of biological replicates which are themselves grouped in batches, \nthen replicate should be what seperates them most down the line. \nThe replicate is technical replicate etc.\n\nHow to use fraction column:\nWhen additional data-points are required to split data data, this should be put in the fraction column, seperated by underscores \"_\".\n\n## FAQ\n\n### What software does the preset pipelines use:\nThese are the current presets\n\n#### Ribo-seq preset\nConfig Call: preset = \"Ribo-seq\"\n\n- Fastq files are download with ORFik download.sra using a fallback system: Amazone\n- Adapter is detected with either fastqc (sequence detection) and falls back to fastp auto detection.\n- Reads are then trimmed with fastp\n- Read are collapsed (get the set of unique reads and put duplication count in read header)\n- Reads are aligned with the STAR aligner (using the wrapper in ORFik), that supports contamination removal. Settings:\n- genomic coordinates (to allow both genomic and transcriptomic coordinates) local alignment (to remove unknown flank effects)\n    minimum read size (20nt)\n- When all samples of study are aligned, an ORFik experiment is created that connects each sample to metadata (condition, inhibitor, fraction, replicate etc)\n- Bam files are then converted to ORFik ofst format\n- These ofst files are then pshifted\n- Faster formats are then created (bigwig and covRLE) for faster visualization\n- Count tables are made with ORFik as Summarized experiments objects, for fast count loading later.\n\n\n### The genome download failed\n\nIf this happens, it means genome could not be automatically fetched \nfrom ensembl.\n\nLuckily there is a way to either manually fix this problem.\n\nTo fix, run get_annotation(\"Homo sapiens\") \u003c- replace with your organism, and see what the problem is.\n\nIf the problem is deeper try, some info is needed:\nWhen init_pipelines_all is run, it checks per organism for a file\ncalled ouputs.rds in the organism reference folder.\nIf this file exists the step will pass.\n\n- Find your gtf and fasta genome file and put manually in folder.\n- Call:\n\nget_annotation(organism, GTF = path_to_gtf, genome = path_to_genome, gene_symbols = FALSE)\n\nThis should now work. If it still fails, contact us on github.\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frc-biotech%2FmassiveNGSpipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frc-biotech%2FmassiveNGSpipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frc-biotech%2FmassiveNGSpipe/lists"}