https://github.com/dsblank/picopipe

A small pipeline framework built on top of functions
https://github.com/dsblank/picopipe

Last synced: 2 months ago
JSON representation

A small pipeline framework built on top of functions

• Host: GitHub
• URL: https://github.com/dsblank/picopipe
• Owner: dsblank
• License: apache-2.0
• Created: 2023-09-01T12:15:44.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2023-11-29T19:51:17.000Z (about 1 year ago)
• Last Synced: 2024-10-04T17:50:06.273Z (3 months ago)
• Language: Python
• Size: 22.5 KB
• Stars: 9
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# picopipe

A small pipeline framework built on top of functions

An article describing development and use of picopipe: 

[Towards Data Science: The Worlds Smallest Data Pipeline Framework](https://medium.com/towards-data-science/the-worlds-smallest-data-pipeline-framework-408eaf1a4ce4)

## Installation

```shell

pip install picopipe

```

## Examples

```python

def add1(value):

    return value + 1

def add2(value):

    return value + 2

def add3(value):

    return value + 3

from picopipe import pipeline

p = pipeline(add1, add2, add3)

p([10, 20, 30])

```

See tests for more examples.

## Visualization

```python

with open("pipeline.mmd", "w") as fp:

    fp.write(to_mermaid(p))

```

Mermaid file (renders in github):

```mermaid

flowchart

subgraph pipeline_98e33e0628b546268abb2af5f74e50f1 ["pipeline"]

end

subgraph pipeline_205813c806fd4b37b40309497768f7c1 ["pipeline"]

    node0["identity"]

    node1["is_not_none"]

    node2["identity"]

end

node0 --> node1

node1 --> node2

click node0 console.log "def identity(value):
    return value
"

click node1 console.log "@pfilter
def is_not_none(v):
    return v is not None
"

click node2 console.log "def identity(value):
    return value
"

subgraph pipeline_d2f205ef957a428abbaa208241125819 ["pipeline"]

    node3["add1"]

    node4["[lambda]"]

end

node3 --> node4

click node3 console.log "def add1(value):
    return value + 1
"

click node4 console.log "lambda ...: ..."

pipeline_98e33e0628b546268abb2af5f74e50f1 --> pipeline_205813c806fd4b37b40309497768f7c1

pipeline_205813c806fd4b37b40309497768f7c1 --> pipeline_d2f205ef957a428abbaa208241125819

```