https://github.com/maxfischer2781/chainlet
Python module for linking generators/iterators to processing chains
https://github.com/maxfischer2781/chainlet
chain iterator pipeline processing-chain python2 python3
Last synced: 12 months ago
JSON representation
Python module for linking generators/iterators to processing chains
- Host: GitHub
- URL: https://github.com/maxfischer2781/chainlet
- Owner: maxfischer2781
- License: mit
- Created: 2017-02-07T20:47:35.000Z (about 9 years ago)
- Default Branch: master
- Last Pushed: 2018-06-12T08:14:43.000Z (almost 8 years ago)
- Last Synced: 2025-03-07T17:18:53.467Z (about 1 year ago)
- Topics: chain, iterator, pipeline, processing-chain, python2, python3
- Language: Python
- Size: 527 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# Chainlet
[](http://chainlet.readthedocs.io/en/latest/?badge=latest)
[](https://travis-ci.org/maxfischer2781/chainlet)
[](https://landscape.io/github/maxfischer2781/chainlet/master)
[](https://codecov.io/gh/maxfischer2781/chainlet)
Framework for linking generator/iterators to create processing chains and pipelines.
With its operator based syntax, it is easy to create complex sequences from simple building blocks.
Chainlets are suitable for incremental, iterative and stream processing and beyond.
## Simplistic Chains with Chainlets
Consider the following use case:
Read data from an XML, flatten it, then write to a csv.
This can be expressed with a chain of generators:
csv_writer(flatten(xml_reader(path='data.xml'), join='.'.join), path='data.csv')
When written using chainlets, generator sequence and arguments are much easier to read.
The chainlets are joined using the `>>` operator:
xml_reader(path='data.xml') >> flatten(join='.'.join) >> csv_writer(path='data.csv')
In addition, chainlets can be composed much more freely.
Instead of deeply nested call structures, chainlets have simple, flat call sequences.
## Custom Chainlets for Pipelines
Writing new chainlets does not require any special techniques or conventions.
You can directly convert existing coroutines, functions and objects.
Implementing a moving average requires exactly one line specific to the ``chainlet`` library:
@chainlet.genlet
def moving_average(window_size=8):
buffer = collections.deque([(yield)], maxlen=window_size)
while True:
new_value = yield(sum(buffer)/len(buffer))
buffer.append(new_value)
All the gluing and binding is done automatically for you.
Instead of bloating existing code, it is often easier to create and bind another simple chainlet.
## Extended Pipelines with Chainlets
Chainlets are not limited to 1-to-1 relations, but actually allow n-to-n links.
Each link can have multiple parents and children.
The following example reads XML messages via UDP, and logs them in two different verbosity levels.
udp_digest(port=31137) >> xml_converter() >> (
json_writer(path='raw.json'),
moving_average(window_size=60) >> json_writer(path='avg1m.json'),
)
## Quick Overview
* The *smallest* building blocks is a `ChainLink`, ready to be subclassed and made _**big**_
* Generate awesome *generator* pipelines, and let `GeneratorLink` put that awesome into use
* State is for wusses, real programmers use functions; real programmers use `FunctionLink`
* Don't wrap your head with wrappers, wrap with decorators - `linklet` and let link
## Tell me more!
Chainlets are simple at their core, and quick to understand.
If you want to know more, just read the fabulous manual:
[](http://chainlet.readthedocs.io/en/latest/?badge=latest)
The module is hosted on [github](https://github.com/maxfischer2781/chainlet).
If you have issues or want to propose changes, check out the [issue tracker](https://github.com/maxfischer2781/chainlet/issues).