{"id":29224547,"url":"https://github.com/sing-group/seda-compi-pipelines","last_synced_at":"2025-08-11T02:07:11.299Z","repository":{"id":194967367,"uuid":"662056671","full_name":"sing-group/seda-compi-pipelines","owner":"sing-group","description":"This project provides a framework for easily creating pipelines of SEDA commands using Compi.","archived":false,"fork":false,"pushed_at":"2025-02-17T11:34:05.000Z","size":2268,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-07-03T06:07:16.080Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sing-group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-07-04T09:05:33.000Z","updated_at":"2025-02-17T11:34:09.000Z","dependencies_parsed_at":"2023-09-16T01:15:09.652Z","dependency_job_id":"b3de9fc6-5edf-411d-85f8-f3799e939fe2","html_url":"https://github.com/sing-group/seda-compi-pipelines","commit_stats":null,"previous_names":["sing-group/seda-compi-pipelines"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/sing-group/seda-compi-pipelines","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing-group%2Fseda-compi-pipelines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing-group%2Fseda-compi-pipelines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing-group%2Fseda-compi-pipelines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing-group%2Fseda-compi-pipelines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sing-group","download_url":"https://codeload.github.com/sing-group/seda-compi-pipelines/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing-group%2Fseda-compi-pipelines/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269819032,"owners_count":24480087,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-11T02:00:10.019Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-03T06:07:15.069Z","updated_at":"2025-08-11T02:07:11.274Z","avatar_url":"https://github.com/sing-group.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SEDA pipelines with Compi [![license](https://img.shields.io/github/license/sing-group/seda-compi-pipelines?style=flat-square)](https://github.com/sing-group/seda-compi-pipelines)\n\nThis project provides a framework for easily creating pipelines made up of [SEDA](https://www.sing-group.org/seda/) commands using [Compi](https://www.sing-group.org/compi/).\n\nCompi pipelines are defined in an XML file that contains the pipeline parameters, task definitions and task dependencies. In a pipeline made up of SEDA commands all tasks would be essentially the same: running the specific SEDA command with a set of parameters over a set of input files; then, the input of each command would be the output of its predecessor commands. As the SEDA CLI can read command parameters, this execution can be generalized so that pipeline developers should only declare the pipeline tasks (i.e. the SEDA CLI commands) and their execution order. \n\nThis is what this framework provides: a generic execution engine for Compi-based pipelines that only requires developers to follow some conventions regarding input, output and parameter files. The image below illustrates these conventions, which are further explained before. \n\n\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"./docs/figure_1.png\" alt=\"seda-cheatsheet\" title=\"SEDA-Compi Pipelines, Figure 1\" width=\"70%;\" /\u003e\n\t\u003c/br\u003e\n\u003c/p\u003e\n\n# Table of Contents\n- [SEDA pipelines with Compi ](#seda-pipelines-with-compi-)\n- [Table of Contents](#table-of-contents)\n- [Requirements](#requirements)\n- [Getting started](#getting-started)\n\t- [Testing the sample pipeline](#testing-the-sample-pipeline)\n\t- [Creating your own SEDA pipeline](#creating-your-own-seda-pipeline)\n- [Understanding Compi-based SEDA pipelines](#understanding-compi-based-seda-pipelines)\n\t- [Defining pipeline tasks: inputs and outputs](#defining-pipeline-tasks-inputs-and-outputs)\n\t- [SEDA command parameters](#seda-command-parameters)\n\t- [Running tasks for non-SEDA commands](#running-tasks-for-non-seda-commands)\n\t- [Running commands in batches](#running-commands-in-batches)\n\t- [An alternative way of specifying the input files](#an-alternative-way-of-specifying-the-input-files)\n\t- [Logging](#logging)\n- [SEDA-Compi Pipelines](#seda-compi-pipelines)\n- [Contributors](#contributors)\n\n# Requirements\n\n- [Docker](https://www.docker.com/products/docker-desktop/), for running pipelines.\n- [Compi Development Kit](https://www.sing-group.org/compi/), only for building pipeline Docker images.\n\n# Getting started\n\nYou can do two things with this project:\n1. Understand how SEDA pipelines can be easily created with Compi and try the example pipeline (`pipeline.xml`) available in the `sample-working-dir` directory.\n2. Clone it and start creating your own SEDA pipeline.\n\n## Testing the sample pipeline\n\nTo test the sample pipeline, simply:\n\n1. Build the Docker image with `compi-dk build -drd -tv`. It will be named `test/seda-cli-pipeline`.\n2. Run the pipeline with `run.sh /path/to/seda-compi-pipelines/sample-working-dir`.\n\nIf you are using Visual Studio Code, the `.vscode/tasks.json` defines tasks for doing this: *build*, *run-docker*, and *build-and-run-docker*.\n\n## Creating your own SEDA pipeline\n\nTo create your own SEDA pipeline, clone the project and then:\n\n1. Edit the `compi.project` file to set the name of your Docker image. Edit the `run.sh` file accordingly to use this Docker image.\n\n2. Edit the `pipeline.xml` file to add one task for each SEDA command to be executed. Also:\n\n    - Edit the `pipeline-runner.xml` to specify the non-SEDA tasks if needed.\n\n\t- Edit the `Dockerfile` to update the base SEDA version to be used (`FROM pegi3s/seda:1.7.2`).\n\n3. Build the image with `compi-dk build -drd -tv` (or run the *build* task within Visual Studio Code).\n\nNote that the list of SEDA commands can be obtained with `docker run --rm pegi3s/seda:1.7.2 /opt/SEDA/run-cli.sh help` (or with the *show SEDA help* task).\n\n# Understanding Compi-based SEDA pipelines\n\n## Defining pipeline tasks: inputs and outputs\n\nThe minimal pipeline task looks like the following:\n\n```xml\n\u003ctask id=\"filtering\"/\u003e\n```\n\nWhich means that the SEDA `filtering` command should be executed. Therefore, task identifiers are the names of the SEDA commands.\n\nAs this task has not dependencies (i.e. previous tasks that must be executed before), its input files are taken from `input/filtering`. This means that all files in that directory will be processed by the command. The corresponding output files will be placed at `output/filtering` (as you should be expecting).\n\nWhat if the same command must be executed more than once? In this case, each command must be suffixed with `_\u003cnumber\u003e`. \n\nLet's make our pipeline starts with two `filtering` commands:\n\n```xml\n\u003ctask id=\"filtering_1\"/\u003e\n\u003ctask id=\"filtering_2\"/\u003e\n```\n\nIn this case, the input files for the two initial pipeline tasks are taken from  `input/filtering_1` and  `input/filtering_2`. The corresponding output files will be created at `output/filtering_1` and `output/filtering_2` (again, no surprises here).\n\nLets add a task that comes after `filtering_1` and `filtering_2`. This task should run the `reformat` command, so it is as simple as:\n\n```xml\n\u003ctask id=\"reformat\" after=\"filtering_1 filtering_2\"/\u003e\n```\n\nThis tasks takes as input all files from the output files of its predecessor tasks, so it will process all files in `output/filtering_1` and `output/filtering_2` and create its output files in `output/reformat`.\n\n\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"./docs/figure_2.png\" alt=\"seda-cheatsheet\" title=\"SEDA-Compi Pipelines, Figure 2\" width=\"90%;\" /\u003e\n\t\u003c/br\u003e\n\u003c/p\u003e\n\nTo sum up what we have seen, as illustrated by the image above:\n\n1. Task ids are just the name of the SEDA commands to be executed. The list of SEDA commands can be obtained with `docker run --rm pegi3s/seda:1.7.2 /opt/SEDA/run-cli.sh help`.\n2. If the same SEDA command is executed more than once, then tasks are disambiguated by adding the `\u003cseda_command\u003e_\u003cnumber\u003e` suffix.\n3. Input files:\n   1. For tasks without dependencies: all files in `input/\u003cseda_command\u003e`.\n   2. For tasks with dependencies: all files each `output/\u003cseda_command_after\u003e` directory (where `\u003cseda_command_after\u003e` is each ID of the predecessor tasks, defined in the `after` property).\n4. Output files are created in `output/\u003cseda_command\u003e`.\n\nThese logic is handled by the `pipeline-runner.sh` script and all tasks are by default using this runner as specified in the `pipeline.xml`.\n\n## SEDA command parameters\n\nWhat about the parameters of the commands? For instance, the `filtering_1` task is being executed with `--minimum-sequence-length 7` and `filtering_2` with `--minimum-sequence-length 12`. The parameters of this tasks are defined in `params/filtering_1.cliParams` and `params/filtering_2.cliParams`.\n\nThe `pipeline-runner.sh` script looks for SEDA command parameter files in the `params` directory. These files should have one of these two names:\n\n- `\u003cseda_command\u003e.cliParams`: plain-text files containing only one line with the SEDA command CLI parameters (e.g. `--minimum-sequence-length 7`).\n- `\u003cseda_command\u003e.sedaParams`: JSON files containing command parameters. These files are generated by the GUI or CLI of SEDA.\n\nBoth files can be used at the same time, meaning that the command will be executed with the corresponding CLI parameters and also the `\u003cseda_command\u003e.sedaParams`.\n\nFor instance, the task `disambiguate` of the sample `pipeline.xml` takes its parameters from the file `disambiguate.sedaParams`.\n\n\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"./docs/figure_3.png\" alt=\"seda-cheatsheet\" title=\"SEDA-Compi Pipelines, Figure 3\" width=\"90%;\" /\u003e\n\t\u003c/br\u003e\n\u003c/p\u003e\n\n## Running tasks for non-SEDA commands\n\nIf your pipeline needs to execute tasks for non-SEDA commands, you must:\n\n1. Add the `src` property to the task definition to specify the path to the executable script. For instance, this is the case of the task `final-task` of the sample `pipeline.xml`:\n\n```xml\n\u003ctask id=\"final-task\" after=\"reverse-complement\" src=\"task-scripts/final-task.sh\"\u003e\u003c/task\u003e\n```\n\n2. Edit the `pipeline-runner.xml` file to specify which tasks must be executed as shell scripts.\n\nThe image below illustrates this situation, where task `Z` comes after task `Y` and its source code is provided by the `script` specified in the `src` property. In these cases, pipeline developers are responsible of creating the output files in the appropriate output directory, so that following tasks can work properly in case of being SEDA commands.\n\n\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"./docs/figure_1.png\" alt=\"seda-cheatsheet\" title=\"SEDA-Compi Pipelines, Figure 1\" width=\"70%;\" /\u003e\n\t\u003c/br\u003e\n\u003c/p\u003e\n\n## Running commands in batches\n\nImagine that some tasks take as input a large amount of files (e.g. 10K) and you want to process them in smaller batches (e.g. 10K in 100 batches of 100 files), so that one SEDA command is executed for each batch of files. This way, you can control how many batches are executed in parallel and avoid memory exhaustion or other performance-related problems.\n\nTo specify the batch size for a specific task you should create a new `\u003cparam\u003e` in the `pipeline.xml` file named `batchSize_\u003cseda_command\u003e`:\n```\n\u003cparam name=\"batchSize_filtering_2\" shortName=\"bsf2\" defaultValue=\"2\"\u003e\u003c/param\u003e\n```\n\nThis way, the task `filtering_2` will be executed in batches of two files.\n\nNote that if a task/command name contains the `-` character, it must be replaced with `_` when creating the `batchSize_\u003cseda_command\u003e` parameter (as it is the case for the task `reverse-complement` in the sample `pipeline.xml`):\n```\n\u003cparam name=\"batchSize_reverse_complement\" shortName=\"bsrc\" defaultValue=\"1\"\u003e\u003c/param\u003e\n```\n\n## An alternative way of specifying the input files\n\nSo far, tasks could take as input files all output files from its predecessor tasks or the files given in `input/\u003cseda_command\u003e`. There is a third way of specifying the input files: creating a special file at `input/lists/\u003cseda_command\u003e.txt`. The `pipeline-runner.sh` will look in first place if such file exists and, if so, take the file contents as the input file list.\n\nIn the example pipeline, files for task `filtering_1` are listed in `input/lists/filtering_1.txt`.\n\nThis feature, illustrated in the image below, is useful when used in conjunction with the feature shown in the next section.\n\n\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"./docs/figure_4.png\" alt=\"seda-cheatsheet\" title=\"SEDA-Compi Pipelines, Figure 4\" width=\"90%;\" /\u003e\n\t\u003c/br\u003e\n\u003c/p\u003e\n\n## Logging\n\nThe `pipeline-runner.sh` also generates useful logs:\n\n- It creates a directory named `_stats` in the output directory containing one CSV file for each pipeline task (`\u003ctask_id\u003e.csv`). This CSV contains the number of input and output files produced by the task. This is useful for quickly checking that commands produce the right number of output files (e.g. n to n in `rename`, n to 1 in `merge`, etc.).\n- In case any batch fails running the corresponding SEDA command, a new directory named `failed/\u003ctask_id\u003e` in the output directory will be created. This directory will contain the file lists corresponding to the failed batches.\n\nIf one or more batches of the same task fail, one should take a look at the execution logs and, in some cases, re-run only such task for the failed files. To do so, all file lists can be merged into a single file at `input/lists/\u003cseda_command\u003e.txt` and run the pipeline with the `run.sh \"--single-task seda_command\"`. This way, the `pipeline-runner.sh` will only run the specified `seda_command` using the files in the given list. After a successful re-run, just delete the file list at `input/lists/\u003cseda_command\u003e.txt`.\n\n# SEDA-Compi Pipelines\n\n- [Preparing datasets for large scale phylogenetic analyses](https://www.sing-group.org/seda/manual/protocols.html#preparing-datasets-for-large-scale-phylogenetic-analyses) [![dockerhub](https://img.shields.io/badge/Docker%20Hub-7289da.svg?style=flat-square)](https://hub.docker.com/r/pegi3s/seda-pipeline-phylogenetics-datasets) [![github](https://img.shields.io/badge/GitHub-7289da.svg?style=flat-square)](https://github.com/pegi3s/seda-pipeline-phylogenetics-datasets)\n- [Protocol for a phylogenomics study](https://www.sing-group.org/seda/manual/protocols.html#protocol-for-a-phylogenomics-study) [![dockerhub](https://img.shields.io/badge/Docker%20Hub-7289da.svg?style=flat-square)](https://hub.docker.com/r/pegi3s/seda-pipeline-phylogenomics-study) [![github](https://img.shields.io/badge/GitHub-7289da.svg?style=flat-square)](https://github.com/pegi3s/seda-pipeline-phylogenomics-study)\n- [Obtaining protein family members](https://www.sing-group.org/seda/manual/protocols.html#obtaining-protein-family-members) [![dockerhub](https://img.shields.io/badge/Docker%20Hub-7289da.svg?style=flat-square)](https://hub.docker.com/r/pegi3s/seda-pipeline-protein-family-members) [![github](https://img.shields.io/badge/GitHub-7289da.svg?style=flat-square)](https://github.com/pegi3s/seda-pipeline-protein-family-members)\n\n# Contributors\n\n\u003ca href=\"https://github.com/sing-group/seda-compi-pipelines/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=sing-group/seda-compi-pipelines\"/\u003e\n\u003c/a\u003e\n\n\u003csup\u003eMade with [contrib.rocks](https://contrib.rocks).\u003c/sup\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsing-group%2Fseda-compi-pipelines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsing-group%2Fseda-compi-pipelines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsing-group%2Fseda-compi-pipelines/lists"}