{"id":41443630,"url":"https://github.com/cellgeni/nf-irods-to-fastq","last_synced_at":"2026-01-23T14:59:19.681Z","repository":{"id":196726778,"uuid":"697010909","full_name":"cellgeni/nf-irods-to-fastq","owner":"cellgeni","description":"Get CRAMs from iRODS and convert them to FASTQ","archived":false,"fork":false,"pushed_at":"2025-11-27T10:48:12.000Z","size":165,"stargazers_count":2,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-11-30T03:57:02.360Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Nextflow","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cellgeni.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-09-26T21:45:39.000Z","updated_at":"2025-11-27T10:48:15.000Z","dependencies_parsed_at":"2024-11-18T19:19:53.162Z","dependency_job_id":"ad6fd600-99da-4f6c-bf0f-f4d125414653","html_url":"https://github.com/cellgeni/nf-irods-to-fastq","commit_stats":null,"previous_names":["cellgeni/nf-irods-to-fastq"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/cellgeni/nf-irods-to-fastq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cellgeni%2Fnf-irods-to-fastq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cellgeni%2Fnf-irods-to-fastq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cellgeni%2Fnf-irods-to-fastq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cellgeni%2Fnf-irods-to-fastq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cellgeni","download_url":"https://codeload.github.com/cellgeni/nf-irods-to-fastq/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cellgeni%2Fnf-irods-to-fastq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28694459,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-23T14:15:13.573Z","status":"ssl_error","status_checked_at":"2026-01-23T14:09:05.534Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-23T14:59:18.969Z","updated_at":"2026-01-23T14:59:19.669Z","avatar_url":"https://github.com/cellgeni.png","language":"Nextflow","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nf-irods-to-fastq\n\n## Overview\n\nThis Nextflow pipeline retrieves samples from iRODS storage, converts CRAM/BAM files to FASTQ format, and optionally uploads the results to FTP servers. The pipeline supports comprehensive metadata management and provides three main operations: metadata discovery, CRAM-to-FASTQ conversion, and FTP upload.\n\n## Contents of Repo\n\n* `main.nf` — the main Nextflow pipeline that orchestrates all workflows\n* `nextflow.config` — configuration script for IBM LSF submission on Sanger's HPC with Singularity containers and global parameters\n* `subworkflows/` — collection of subworkflows for different pipeline stages\n* `modules/` — collection of reusable modules for various tasks\n* `configs/` — configuration files for different pipeline components\n* `examples/` — example input files demonstrating various input formats\n\n## Pipeline Workflow\n\n1. **Sample Discovery**: Reads sample information from CSV, TSV, or JSON input files\n2. **Metadata Retrieval**: Searches iRODS for CRAM files associated with samples and retrieves metadata\n3. **File Download**: Downloads CRAM/BAM files from iRODS storage\n4. **Format Conversion**: Converts CRAM/BAM files to FASTQ format using samtools\n5. **Quality Control**: Calculates read lengths and applies ATAC-seq specific formatting if needed\n6. **File Concatenation**: Combines FASTQ files by sample and read type\n7. **Checksum Calculation**: Generates MD5 checksums for data integrity verification\n8. **FTP Upload**: Optionally uploads processed FASTQ files to specified FTP servers\n\n## Pipeline Parameters\n\n### Required Parameters (choose one):\n* `--samples` — Path to a CSV, TSV, or JSON file containing sample information with a `sample` or `sample_id` column\n* `--crams` — Path to a CSV or TSV file containing CRAM file information with columns: `sample`, `cram_path`, `fastq_prefix`\n* `--fastqs` — Path to a CSV file containing FASTQ file information with columns: `sample`, `path`\n\n### Operation Flags:\n* `--cram2fastq` — Enable CRAM-to-FASTQ conversion (used with `--samples` or `--crams`)\n* `--toftp` — Enable FTP upload (used with `--fastqs`)\n\n### Optional Parameters:\n* `--output_dir` — Output directory for pipeline results (default: `\"results\"`)\n* `--publish_mode` — File publishing mode (default: `\"copy\"`)\n* `--index_format` — Index format formula for samtools (default: `\"i*i*\"`)\n* `--format_atac` — Apply ATAC-seq specific formatting (default: `true`)\n* `--ignore_patterns` — Comma-separated patterns to ignore when finding CRAMs (default: `\"*_phix.cram,*yhuman*,*#888.cram\"`)\n* `--irods_zone` — iRODS zone to search (default: `\"seq\"`)\n\n### FTP Parameters (required when using `--toftp`):\n* `--ftp_host` — FTP server hostname (default: `\"ftp-private.ebi.ac.uk\"`)\n* `--username` — FTP username\n* `--password` — FTP password\n* `--ftp_path` — Target path on FTP server\n\nNote: When using `--toftp`, you must also provide `--fastqs` with a CSV file containing FASTQ paths.\n\n## Input File Formats\n\nThe pipeline supports multiple input formats for different operation modes:\n\n### Option 1: Sample Discovery (`--samples`)\n\nSpecify `sample` or `sample_id` along with other useful metadata columns to find CRAM files on iRODS.\n\n**CSV format:**\n```csv\nsample,study_title\n4861STDY7135911,Study_Name\n4861STDY7135912,Study_Name\nHuman_colon_16S8000511,Human_colon_16S\n```\n\n**TSV format:**\n```tsv\nsample\tstudy_title\n4861STDY7135911\tStudy_Name\n4861STDY7135912\tStudy_Name\n```\n\n**JSON format:**\n```json\n[\n  {\"sample\": \"4861STDY7135911\", \"study_title\": \"Study_Name\"},\n  {\"sample\": \"4861STDY7135912\", \"study_title\": \"Study_Name\"}\n]\n```\n\n### Option 2: Direct CRAM Processing (`--crams`)\n\nSpecify `sample`, `cram_path`, and `fastq_prefix` columns to directly process known CRAM files.\n\n**CSV format:**\n```csv\nsample,cram_path,fastq_prefix\n4861STDY7135911,/seq/24133/24133_1#4.cram,4861STDY7135911_S1_L001\n4861STDY7135911,/seq/24133/24133_2#2.cram,4861STDY7135911_S1_L002\n```\n\n### Option 3: FASTQ Upload (`--fastqs`)\n\nSpecify `sample` and `path` columns for FASTQ files to upload. Note: this requires a CSV file, not a directory path.\n```\nsample,path\n4861STDY7135911,results/fastqs/4861STDY7135911/4861STDY7135911_S1_L001_I1_001.fastq.gz\n4861STDY7135911,results/fastqs/4861STDY7135911/4861STDY7135911_S1_L001_R1_001.fastq.gz\n```\n\n## Examples\n\n### System Requirements Setup\nPrepare your environment on Sanger's farm22:\n\n```bash\nmodule load cellgen/nextflow/24.10.0\nmodule load cellgen/irods\nmodule load cellgen/singularity\nmodule load python-3.11.6\nexport LSB_DEFAULT_USERGROUP=\u003cYOURGROUP\u003e\n```\n\nInitialize iRODS connection:\n```bash\niinit\n```\n\n### Basic Usage Examples\n\n**1. Sample Metadata Discovery:**\n```bash\nnextflow run main.nf --samples ./examples/samples.csv\n```\n\nThis generates a `metadata/` directory with:\n```\nmetadata/\n├── getmetadata.log     # warnings and processing information\n└── metadata.tsv       # sample metadata from iRODS\n```\n\n**2. CRAM-to-FASTQ Conversion:**\n```bash\nnextflow run main.nf --cram2fastq --crams metadata/metadata.tsv\n```\n\n**3. Complete Pipeline (Discovery + Conversion):**\n```bash\nnextflow run main.nf --samples ./examples/samples.csv --cram2fastq\n```\n\nNote: The pipeline does not currently support end-to-end operation combining CRAM conversion with FTP upload in a single command. To upload converted FASTQ files, you must first run the conversion step, then use the generated `fastqs.csv` file for FTP upload in a separate command.\n\n**4. FTP Upload:**\n```bash\nnextflow run main.nf --toftp --fastqs ./examples/fastqs.csv --username \"annotare\" --password \"annotare1\" --ftp_host \"ftp-private.ebi.ac.uk\" --ftp_path \"/path/to/ftp/dir\"\n```\n\n**5. End-to-End Pipeline (two-step process):**\n```bash\n# Step 1: Discovery and conversion\nnextflow run main.nf --samples ./examples/samples.csv --cram2fastq\n\n# Step 2: Upload the generated fastqs.csv (after step 1 completes)\nnextflow run main.nf --toftp --fastqs ./results/fastqs.csv --username \"annotare\" --password \"annotare1\" --ftp_host \"ftp-private.ebi.ac.uk\" --ftp_path \"/path/to/ftp/dir\"\n```\n\n### Advanced Usage Examples\n\n**Custom Output Directory:**\n```bash\nnextflow run main.nf \\\n    --samples ./examples/samples.csv \\\n    --cram2fastq \\\n    --output_dir \"my_results\"\n```\n\n**Disable ATAC Formatting:**\n```bash\nnextflow run main.nf \\\n    --samples ./examples/samples.csv \\\n    --cram2fastq \\\n    --format_atac false\n```\n\n## Expected Output Structure\n\n### After Metadata Discovery:\n```\nmetadata/\n├── getmetadata.log\n└── metadata.tsv\n```\n\n### After CRAM-to-FASTQ Conversion:\n```\nresults/\n├── fastqs/\n│   └── {sample}/\n│       ├── {sample}_S1_L001_I1_001.fastq.gz\n│       ├── {sample}_S1_L001_R1_001.fastq.gz\n│       ├── {sample}_S1_L001_R2_001.fastq.gz\n│       └── ...\n├── fastqs.csv                    # Generated CSV file listing all FASTQ paths\n└── metadata_final.tsv            # Final metadata file\n```\n\n### After FTP Upload:\nAdditional files in `results/`:\n```\n├── concatenated/                  # Concatenated FASTQ files by sample\n│   ├── {sample}_S1_I1_001.fastq.gz\n│   ├── {sample}_S1_R1_001.fastq.gz\n│   └── {sample}_S1_R2_001.fastq.gz\n└── md5checksums.txt              # MD5 checksums of uploaded files\n```\n\n## System Requirements\n\n- **Nextflow**: Version 25.04.4 or higher\n- **Singularity**: For containerized execution\n- **iRODS client**: Access to iRODS commands (`iget`, `imeta`, etc.)\n- **LSF**: For job submission on HPC clusters (configured for Sanger's environment)\n\n## Error Handling\n\n- **Invalid input files**: Pipeline validates CSV/TSV headers and JSON structure\n- **Missing samples**: Warnings are logged for samples not found in iRODS\n- **Missing required fields**: Pipeline validates presence of required columns (`sample`/`sample_id`, `cram_path`, `fastq_prefix`)\n- **Empty sample values**: Pipeline checks for non-empty sample identifiers\n- **Checksum verification**: MD5 checksums are calculated for data integrity verification\n- **FTP upload failures**: Failed uploads are logged with detailed error messages\n\n## Monitoring and Logging\n\nThe pipeline generates comprehensive reports in the `reports/` directory:\n- **Timeline report**: Visual timeline of task execution\n- **Execution report**: Detailed resource usage and performance metrics\n- **Trace file**: Complete execution trace for debugging\n\n## Pipeline Flow Diagram\n\n```mermaid\n---\ntitle: Nextflow pipeline for retrieving CRAM files from iRODS and converting them to FASTQ\n---\nflowchart TB\n    subgraph findcrams[\"IRODS_FINDCRAMS\"]\n        direction LR\n        v0([IRODS_FIND])\n        v1([IRODS_GETMETADATA])\n        v2([makeFastqPrefix])\n        v3([COMBINE_METADATA])\n    end\n    \n    subgraph downloadcrams[\"IRODS_DOWNLOADCRAMS\"]\n        direction LR\n        v4([IRODS_GETFILE])\n        v5([CRAM2FASTQ])\n        v6([COMBINE_METADATA])\n    end\n    \n    subgraph fastq2ftp[\"FASTQS2FTP\"]\n        direction LR\n        v7([CONCATENATE_FASTQS])\n        v8([CALCULATE_MD5])\n        v9([UPLOAD2FTP])\n    end\n    \n    v0 --\u003e v1 --\u003e v2 --\u003e v3\n    v4 --\u003e v5 --\u003e v6\n    v7 --\u003e v8\n    v7 --\u003e v9\n    \n    findcrams -.-\u003e downloadcrams -.-\u003e fastq2ftp\n```\n\n## Usage Notes\n\n- Only one input mode can be used per pipeline run (`--samples`, `--crams`, OR `--fastqs`)\n- When using `--samples`, the pipeline will automatically discover associated CRAM files in iRODS\n- Sample names must contain either a `sample` or `sample_id` column in input files\n- The pipeline automatically handles 10X ATAC-seq specific file naming conventions\n- FASTQ files are concatenated by sample and read type for easier downstream processing\n- FTP uploads require both `--toftp` flag AND `--fastqs` parameter with a CSV file (not directory)\n- End-to-end processing (CRAM conversion + FTP upload) requires two separate pipeline runs\n- Large CRAM files may take considerable time to download and convert depending on network bandwidth\n- The pipeline is optimized for batch processing of multiple samples simultaneously\n- The pipeline writes a `fastqs.csv` file to the output directory after CRAM conversion, which can be used for subsequent FTP uploads\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcellgeni%2Fnf-irods-to-fastq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcellgeni%2Fnf-irods-to-fastq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcellgeni%2Fnf-irods-to-fastq/lists"}