Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/pwwang/pipen-cli-require
Checking the requirements for processes of a pipeline
https://github.com/pwwang/pipen-cli-require
Last synced: 14 days ago
JSON representation
Checking the requirements for processes of a pipeline
- Host: GitHub
- URL: https://github.com/pwwang/pipen-cli-require
- Owner: pwwang
- Created: 2022-05-19T18:02:40.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2024-07-23T22:27:20.000Z (5 months ago)
- Last Synced: 2024-12-13T03:51:30.849Z (21 days ago)
- Language: Python
- Size: 175 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# pipen-cli-require
Checking the requirements for processes of a [pipen][1] pipeline
## Install
```shell
pip install -U pipen-cli-require
```
## Usage
### Defining requirements of a process
```python
# example_pipeline.py
from pipen import Pipen, Proc
class P1(Proc):
"""Process 1
Requires:
pipen: Run `pip install -U pipen` to install
- check: |
{{proc.lang}} -c "import pipen"
liquidpy: Run `pip install -U liquidpy` to install
- check: |
{{proc.lang}} -c "import liquid"
nonexist: Run `pip install -U nonexist` to install
- check: |
{{proc.lang}} -c "import nonexist"
conditional:
- if: {{envs.require_conditional}}
- check:
{{proc.lang}} -c "import optional"
"""
input = "a"
output = "outfile:file:out.txt"
envs = {"require_conditional": False}
lang = "python"
# Setup the pipeline
# Must be outside __main__
# Or define a function to return the pipeline
class Pipeline(Pipen):
starts = P1
if __name__ == '__main__':
# Pipeline must run with __main__
Pipeline().run()
```
### Parsing process requirements using API
```python
from pipen_cli_require import parse_proc_requirements
def parse_proc_requirements(
proc: Type[Proc]
) -> Tuple[OrderedDiot, OrderedDiot]:
"""Parse the requirements of a process
Args:
proc: The process class
Returns:
A tuple of two OrderedDiot's.
The first one is the annotated sections by pipen_annotate
The second one is the requirements. The key is the name of the
requirement, the value is a dict with message, check and if_ keys.
"""
```
## Checking the requirements via the CLI
```shell
> pipen require --verbose --ncores 2 example_pipeline.py:pipeline
Checking requirements for pipeline: PIPEN-0
│
└── P1: Process 1
├── ✅ pipen
├── ✅ liquidpy
├── ❎ nonexist: Run `pip install -U nonexist` to install
│ └── Traceback (most recent call last):
│ File "", line 1, in
│ ModuleNotFoundError: No module named 'nonexist'
│
└── ⏩ conditional (skipped by if-statement)
```
## Checking requirements with runtime arguments
For example, when I use a different python to run the pipeline:
Add this to the head of `example_pipeline.py`:
```python
import pipen_args
```
See also `tests/pipen_args_pipeline.py`
Then specify the path of the python to use:
```shell
pipen require example_pipeline.py:pipeline --P1.lang /path/to/another/python
```
[1]: https://github.com/pwwang/pipen