https://github.com/qiwi/protopipe
Graph-driven data processor
https://github.com/qiwi/protopipe
utils
Last synced: 5 days ago
JSON representation
Graph-driven data processor
- Host: GitHub
- URL: https://github.com/qiwi/protopipe
- Owner: qiwi
- License: mit
- Created: 2018-12-02T10:54:21.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2025-03-15T14:38:39.000Z (about 2 months ago)
- Last Synced: 2025-04-17T16:55:20.413Z (15 days ago)
- Topics: utils
- Language: TypeScript
- Homepage:
- Size: 1.53 MB
- Stars: 4
- Watchers: 2
- Forks: 2
- Open Issues: 20
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# protopipe
> Graph-driven data processor. [qiwi.github.io/protopipe](https://qiwi.github.io/protopipe/)
[](https://github.com/qiwi/protopipe/actions/workflows/ci.yaml)
[](https://www.npmjs.com/package/protopipe)
[](https://codeclimate.com/github/qiwi/protopipe/maintainability)
[](https://codeclimate.com/github/qiwi/protopipe/test_coverage)
[](https://github.com/qiwi/lint-config-qiwi)
## Idea
We often come across the problem of atomic data processing ([logwrap](https://github.com/qiwi/logwrap), [uniconfig](https://github.com/qiwi/uniconfig), [cyclone](https://github.com/qiwi/cyclone), etc), and it seems to be useful to make the _one pipeline to rule them all_.
Not universal, not high-performance. But dumb and clear.
## TL;DR
#### Install
```bash
yarn add protopipe
```
#### Usage
```javascript
import {Graph, IAny, ISpace, NetProcessor} from 'protopipe'
const graph = new Graph({
edges: ['AB', 'BC'],
vertexes: ['A', 'B', 'C'],
incidentor: {
type: 'EDGE_LIST_INCDR',
value: {
'AB': ['A', 'B'],
'BC': ['B', 'C'],
},
},
})
const handler = {
// Default handler
graph: (space: ISpace): IAny => (NetProcessor.getData(space) || {value: 0}).value * 2,
// Vertex specific handlers
vertexes: {
'B': (space: ISpace): IAny => (NetProcessor.getData(space, 'A') || {value: 10}).value * 3,
},
}
const protopipe = new NetProcessor({
graph,
handler,
})
const space = protopipe.impact(true, ['A', 1]) as ISpace
console.log(NetProcessor.getData(space, 'C').value) // 6
```
##### Features
* Sync / async execution modes.
* Stateful and stateless contracts.
* Deep customization.
* Typings for both worlds — TS and flow.
##### Limitations (of default executor, traverser, etc)
* Protopipe supports digraphs only.
* The lib does not solve the declaration problem (you know, _adjacency/incidence matrix_, _adjacency list_, DSL).
* No consistency checks out of box: graph is being processed as is. But you're able to add custom assertions (for example, BFS-based check).
## Definitions and contracts
* Space is a set with some added structure.
* Vertex is a graph atom.
* Edge — bond.
* Incidentor — the _rule_ of connecting vertexes and edges.
* Graph — a class that implements [`IGraph`](./src/main/ts/types.ts) — stores vertexes and edges collections, features and incidentor.
* Sequence — any possible transition.
* Walk: vertices may repeat, edges may repeat (closed / open)
* Trail: vertices may repeat, edges cannot repeat (open)
* Circuit: vertices may repeat, edges cannot repeat (closed)
* Path: vertices cannot repeat, edges cannot repeat (open)
* Cycle : vertices cannot repeat, edges cannot repeat (closed)
* Pipe is an executable segment of pipeline, any directed sequence with attached handler(s)
* Handler — lambda-function, which implements [`IHandler`](./src/main/ts/interface.ts) iface.
### IGraph
```javascript
export type IGraph = {
vertexes: Array,
edges: Array,
incidentor: IGraphIncidentor
}
export type IGraphRepresentation = any
export type IGraphIncidentor = {
type: IGraphIncidentorType,
value: IGraphRepresentation
}
```
### Sync / async
Pass `mode` flag as the first `.impact()` argument to get result or promise.
```javascript
const graph = new Graph({
edges: ['AB', 'AC', 'BC', 'BD', 'CD', 'AD'],
vertexes: ['A', 'B', 'C', 'D'],
incidentor: {
type: 'EDGE_LIST_INCDR',
value: {
'AB': ['A', 'B'],
'AC': ['A', 'C'],
'BC': ['B', 'C'],
'BD': ['B', 'D'],
'CD': ['C', 'D'],
'AD': ['A', 'D'],
},
},
})
const handler = (space: ISpace) => NetProcessor.requireElt('ANCHOR', space).value.vertex
const netProcessor = new NetProcessor({graph, handler})
// SYNC
const res1 = netProcessor.impact(true,'A') as ISpace
NetProcessor.getData(res1, 'D') // 'D'
// ASYNC
netProcessor.impact(false,'A').then((res) => {
NetProcessor.getData(res, 'D') // 'D'
})
```
### Bundles
The lib exposes its inners as ES5, ES6 and TS formats.
### Customization
You're able to override everything.