{"id":30237990,"url":"https://github.com/seqeralabs/nf-aggregate","last_synced_at":"2025-08-15T02:59:09.950Z","repository":{"id":203501258,"uuid":"709299314","full_name":"seqeralabs/nf-aggregate","owner":"seqeralabs","description":"Pipeline to aggregate pertinent metrics across pipeline runs on the Seqera Platform (beta)","archived":false,"fork":false,"pushed_at":"2025-06-17T10:57:57.000Z","size":3114,"stargazers_count":8,"open_issues_count":10,"forks_count":10,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-08-10T05:59:37.352Z","etag":null,"topics":["nextflow","pipeline","qc","seqera","workflow"],"latest_commit_sha":null,"homepage":"https://seqera.io/","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/seqeralabs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATIONS.md","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-10-24T12:47:04.000Z","updated_at":"2025-05-05T14:02:37.000Z","dependencies_parsed_at":"2024-05-20T16:19:34.962Z","dependency_job_id":"5b39b3da-3d79-4181-a823-f36ea2af28da","html_url":"https://github.com/seqeralabs/nf-aggregate","commit_stats":null,"previous_names":["seqeralabs/nf-aggregate"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/seqeralabs/nf-aggregate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seqeralabs%2Fnf-aggregate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seqeralabs%2Fnf-aggregate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seqeralabs%2Fnf-aggregate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seqeralabs%2Fnf-aggregate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/seqeralabs","download_url":"https://codeload.github.com/seqeralabs/nf-aggregate/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seqeralabs%2Fnf-aggregate/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270515666,"owners_count":24598440,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-15T02:00:12.559Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nextflow","pipeline","qc","seqera","workflow"],"created_at":"2025-08-15T02:59:05.487Z","updated_at":"2025-08-15T02:59:09.941Z","avatar_url":"https://github.com/seqeralabs.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# seqeralabs/nf-aggregate\n\n[![GitHub Actions CI Status](https://github.com/seqeralabs/nf-aggregate/actions/workflows/ci.yml/badge.svg)](https://github.com/seqeralabs/nf-aggregate/actions/workflows/ci.yml)\n[![GitHub Actions Linting Status](https://github.com/seqeralabs/nf-aggregate/actions/workflows/linting.yml/badge.svg)](https://github.com/seqeralabs/nf-aggregate/actions/workflows/linting.yml)\n[![nf-test](https://img.shields.io/badge/unit_tests-nf--test-337ab7.svg)](https://www.nf-test.com)\n\n[![Nextflow](https://img.shields.io/badge/version-%E2%89%A524.04.2-green?style=flat\u0026logo=nextflow\u0026logoColor=white\u0026color=%230DC09D\u0026link=https%3A%2F%2Fnextflow.io)](https://www.nextflow.io/)\n[![nf-core template version](https://img.shields.io/badge/nf--core_template-3.3.0.dev0-green?style=flat\u0026logo=nfcore\u0026logoColor=white\u0026color=%2324B064\u0026link=https%3A%2F%2Fnf-co.re)](https://github.com/nf-core/tools/releases/tag/3.3.0.dev0)\n[![run with conda](http://img.shields.io/badge/run%20with-conda-3EB049?labelColor=000000\u0026logo=anaconda)](https://docs.conda.io/en/latest/)\n[![run with docker](https://img.shields.io/badge/run%20with-docker-0db7ed?labelColor=000000\u0026logo=docker)](https://www.docker.com/)\n[![run with singularity](https://img.shields.io/badge/run%20with-singularity-1d355c.svg?labelColor=000000)](https://sylabs.io/docs/)\n[![Launch on Seqera Platform](https://img.shields.io/badge/Launch%20%F0%9F%9A%80-Seqera%20Platform-%234256e7)](https://cloud.seqera.io/launch?pipeline=https://github.com/seqeralabs/nf-aggregate)\n\n## Introduction\n\n**seqeralabs/nf-aggregate** is a Nextflow pipeline to aggregate pertinent metrics across pipeline runs on the Seqera Platform.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/multiqc_screenshot.png\" alt=\"MultiQC screenshot\" width=\"75%\"/\u003e\n\u003c/p\u003e\n\nThe pipeline performs the following steps:\n\n1. Downloads run information via the Seqera CLI in parallel\n2. Runs MultiQC to aggregate all of the run metrics into a single report\n\nYou can download an example MultiQC report [here](assets/multiqc_report.html).\n\n## Prerequisites\n\n- [Nextflow](https://www.nextflow.io/docs/latest/getstarted.html#installation) \u003e=23.10.0\n- Account in [Seqera Platform](https://seqera.io/platform/)\n- [Access token](https://docs.seqera.io/platform/23.3.0/api/overview#authentication) which is your personal authorization token for the Seqera Platform CLI. This can be created in the user menu under **Your tokens**. Export the token as a shell variable directly into your terminal if running the pipelie locally. You will not need to set this if running the pipeline within the Seqera Platform as it will automatically be inherited from the executing environment.\n\n  ```bash\n  export TOWER_ACCESS_TOKEN=\u003cyour access token\u003e\n  ```\n\n## Usage\n\nThe primary input to the pipeline is a file containing a list of run identifiers from the Seqera Platform. These can be obtained from details in the runs page for any pipeline execution. For example, we can create a file called `run_ids.csv` with the following contents:\n\n```\nid,workspace\n4Bi5xBK6E2Nbhj,community/showcase\n4LWT4uaXDaGcDY,community/showcase\n38QXz4OfQDpwOV,community/showcase\n2lXd1j7OwZVfxh,community/showcase\n```\n\nThis pipeline can then be executed with the following command:\n\n```\nnextflow run seqeralabs/nf-aggregate \\\n    --input run_ids.csv \\\n    --outdir ./results \\\n    -profile docker\n```\n\nIf you are using a Seqera Platform Enterprise instance that is secured with a private CA SSL certificate not recognized by default Java certificate authorities, you can specify a custom `cacerts` store path through the `--java_truststore_path` parameter and optionally, a password with the `--java_truststore_password`. This certificate will be used to achieve connectivity with your Seqera Platform instance through API and CLI.\n\n### Benchmark reports\n\nIf you want to generate a benchmark report comparing multiple runs, you can include a `group` column in your `run_ids.csv` file. This allows you to organize and analyze runs based on custom groupings in the final report.\n\n```\nid,workspace,group\n3VcLMAI8wyy0Ld,community/showcase,group1\n4VLRs7nuqbAhDy,community/showcase,group2\n```\n\n## Use logs from an external Seqera Platform deployment\n\nSometimes we want to compile benchmark reports from runs from two different Seqera platform deployments, for example a dev and a production environment to compare performance. External logs in nf-aggregate can be used by specifying the workspace as `external` and providing some additional optional columns that point to the log folder and specify whether these external logs contain fusion logs (did you export them with the `--add-fusion-logs` flag in your `tw run dumps`. If they do contain fusion logs, you can generate a gannt plot for them, as for runs supplied only via id.)\n\nHere is an example of using a mix of run ids for which we want to extract logs from our platform deployment and some run logs from another deployment we want to compare. In the example below, `1JI5B1avuj3o58` is a run that contains fusion logs, while `1vsww7GjKBsVNa` does not contain fusion logs.\n\n```\nid,workspace,group,logs,fusion\n3VcLMAI8wyy0Ld,community/showcase,group1,\n1JI5B1avuj3o58,external,group2,/path/to/my/run_dumps_tarball.tar.gz,true\n1vsww7GjKBsVNa,external,group2,/path/to/my/run_dumps_folder,false\n```\n\n## Incorporate AWS split cost allocation data\n\nTo incorporate AWS cost data into the benchmark report, use the `benchmark_aws_cur_report` parameter. This should point to a valid AWS Cost and Usage Report (CUR) file in Parquet format, currently only supporting CUR 1.0. The file can be stored locally or in a cloud bucket.\nTo run nf-aggregate and generate benchmark reports, you can use the following command:\n\n```\nnextflow run seqeralabs/nf-aggregate \\\n    --input run_ids.csv \\\n    --outdir ./results \\\n    --generate_benchmark_report \\\n    --benchmark_aws_cur_report ./aws_cost_report.parquet\n```\n\nThe benchmark report can be generated without cost data - simply omit the `--benchmark_aws_cur_report` parameter if cost analysis is not needed.\n\n## Output\n\nThe results from the pipeline will be published in the path specified by the `--outdir` and will consist of the following contents:\n\n```\n./results\n├── multiqc/\n│   ├── multiqc_data/\n│   ├── multiqc_plots/\n│   └── multiqc_report.html                 ## MultiQC report\n├── nf-core_rnaseq/\n│   ├── gantt/\n│   │   └── 4Bi5xBK6E2Nbhj_gantt.html       ## Gantt plot for run\n│   └── runs_dump/\n│       └── 4Bi5xBK6E2Nbhj/                 ## Output of 'tw runs dump'\n│           ├── service-info.json\n│           ├── workflow-launch.json\n│           ├── workflow-load.json\n│           ├── workflow-metrics.json\n│           ├── workflow-tasks.json\n│           └── workflow.json\n└── pipeline_info/\n```\n\n\u003e [!NOTE]\n\u003e Gantt plots depend on information derived from the Fusion logs. For that reason, Gantt plots will be ommitted from the pipeline outputs for non-Fusion runs, irrespective of whether the `--skip_run_gantt` parameter has been set.\n\n## Contributions and Support\n\nIf you would like to contribute to this pipeline, please see the [contributing guidelines](.github/CONTRIBUTING.md).\n\n## Credits\n\nnf-aggregate was written by the Scientific Development and MultiQC teams at [Seqera Labs](https://seqera.io/).\n\n## Citations\n\nThis pipeline uses code and infrastructure developed and maintained by the [nf-core](https://nf-co.re) community, reused here under the [MIT license](https://github.com/nf-core/tools/blob/master/LICENSE).\n\nYou can cite the `nf-core` publication as follows:\n\n\u003e **The nf-core framework for community-curated bioinformatics pipelines.**\n\u003e\n\u003e Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso \u0026 Sven Nahnsen.\n\u003e\n\u003e _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseqeralabs%2Fnf-aggregate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fseqeralabs%2Fnf-aggregate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseqeralabs%2Fnf-aggregate/lists"}