Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/tom-tan/shaft
A workflow engine for CWL CommandLineTool
https://github.com/tom-tan/shaft
commonwl workflow-engine
Last synced: 17 days ago
JSON representation
A workflow engine for CWL CommandLineTool
- Host: GitHub
- URL: https://github.com/tom-tan/shaft
- Owner: tom-tan
- License: apache-2.0
- Created: 2022-01-18T05:16:51.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-08-30T03:31:45.000Z (4 months ago)
- Last Synced: 2024-10-16T13:34:27.192Z (2 months ago)
- Topics: commonwl, workflow-engine
- Language: D
- Homepage:
- Size: 879 KB
- Stars: 4
- Watchers: 3
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Shaft: A workflow engine for CommandLineTool in local machine
[](https://github.com/tom-tan/shaft/releases/latest)
[](https://github.com/tom-tan/shaft/blob/main/LICENSE)
[](https://github.com/tom-tan/shaft/actions/workflows/ci.yml)
Shaft is a workflow engine to be used as a "shaft" of other workflow engines for the [Common Workflow Language](https://www.commonwl.org/) (CWL).
The main purpose of this engine is to be embedded in more enhanced workflow engines such as [ep3](https://github.com/tom-tan/ep3) and therefore it focuses on supporting [CommandLineTool](https://www.commonwl.org/v1.2/CommandLineTool.html) and [ExpressionTool](https://www.commonwl.org/v1.2/Workflow.html#ExpressionTool) documents in local machines.
## Main features
- Easy to deploy (statically-linked binary)
- [Enable to take inherited process requirements and hints via input object](#passing-inherited-process-requirements-and-hints-via-input-object)
- it is necessary to integrate shaft with other workflow engine that supports `Workflow`
## Conformance tests for CWL v1.0
[](https://github.com/tom-tan/shaft/releases/latest) 
### Classes
[](https://github.com/tom-tan/conformance/blob/master/shaft/cwl_v1.0/shaft_latest/command_line_tool.md) [](https://github.com/tom-tan/conformance/blob/master/shaft/cwl_v1.0/shaft_latest/expression_tool.md)
### Required features
[](https://github.com/tom-tan/conformance/blob/master/shaft/cwl_v1.0/shaft_latest/required.md)
### Optional features
[](https://github.com/tom-tan/conformance/blob/master/shaft/cwl_v1.0/shaft_latest/docker.md) [](https://github.com/tom-tan/conformance/blob/master/shaft/cwl_v1.0/shaft_latest/env_var.md) [](https://github.com/tom-tan/conformance/blob/master/shaft/cwl_v1.0/shaft_latest/initial_work_dir.md) [](https://github.com/tom-tan/conformance/blob/master/shaft/cwl_v1.0/shaft_latest/inline_javascript.md) [](https://github.com/tom-tan/conformance/blob/master/shaft/cwl_v1.0/shaft_latest/resource.md) [](https://github.com/tom-tan/conformance/blob/master/shaft/cwl_v1.0/shaft_latest/schema_def.md) [](https://github.com/tom-tan/conformance/blob/master/shaft/cwl_v1.0/shaft_latest/shell_command.md)
## Shaft extension
### Taking inherited process requirements and hints via the input object
The [CWL specification](https://www.commonwl.org/v1.2/CommandLineTool.html#Requirements_and_hints) says that the step processes in a workflow must inherit requirements and hints declared in the parent workflow.
To integrate shaft with other workflow engine for `Workflow`, shaft can take inherited requirements and hints via `shaft:inherited-requirements` and `shaft:inherited-hints` in the input object. They must have an array of process requirements that are inherited from the parent workflow.
- Examples: Inheriting `EnvVarRequirement` via `shaft:inherited-requirements`
- tests/tools/env-tool1-noenv.cwl
```cwl
# based on https://github.com/common-workflow-language/common-workflow-language/blob/main/v1.0/v1.0/env-tool1.cwl
class: CommandLineTool
cwlVersion: v1.0
hints:
ResourceRequirement:
ramMin: 8
inputs:
in: string
outputs:
out:
type: File
outputBinding:
glob: out
# requirements:
# EnvVarRequirement:
# envDef:
# TEST_ENV: $(inputs.in)
baseCommand: ["/bin/sh", "-c", "echo $TEST_ENV"]
stdout: out
```
- test/jobs/senv-job.yml
```yml
in: "hello test env"
shaft:inherited-requirements:
- class: EnvVarRequirement
envDef:
TEST_ENV: override
```
- Execution result:
```console
$ ./bin/shaft --quiet tests/tools/env-tool1-noenv.cwl tests/jobs/env-job.yml | jq .
{
"out": {
"basename": "out",
"checksum": "sha1$cdc1e84968261d6a7575b5305945471f8be199b6",
"class": "File",
"dirname": "/workspaces/shaft",
"location": "file:///workspaces/shaft/out",
"nameext": "",
"nameroot": "out",
"path": "/workspaces/shaft/out",
"size": 9
}
}
$ cat out
override
```
- Examples: Inheriting `EnvVarRequirement` via `shaft:inherited-requirements` but not overriding requirements in the process
- tests/tools/env-tool1.cwl
```cwl
class: CommandLineTool
cwlVersion: v1.0
hints:
ResourceRequirement:
ramMin: 8
inputs:
in: string
outputs:
out:
type: File
outputBinding:
glob: out
requirements:
EnvVarRequirement:
envDef:
TEST_ENV: $(inputs.in)
baseCommand: ["/bin/sh", "-c", "echo $TEST_ENV"]
stdout: out
```
- test/jobs/senv-job.yml (same as previous example)
```yml
in: "hello test env"
shaft:inherited-requirements:
- class: EnvVarRequirement
envDef:
TEST_ENV: override
```
- Execution result:
```console
$ ./bin/shaft --quiet tests/tools/env-tool1.cwl tests/jobs/env-job.yml | jq .
{
"out": {
"basename": "out",
"checksum": "sha1$b3ec4ed1749c207e52b3a6d08c59f31d83bff519",
"class": "File",
"dirname": "/workspaces/shaft",
"location": "file:///workspaces/shaft/out",
"nameext": "",
"nameroot": "out",
"path": "/workspaces/shaft/out",
"size": 15
}
}
$ cat out
hello test env
```