Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/maty21/api-server
https://github.com/maty21/api-server
Last synced: 16 days ago
JSON representation
- Host: GitHub
- URL: https://github.com/maty21/api-server
- Owner: maty21
- License: mit
- Created: 2017-12-26T09:29:49.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2020-12-09T01:24:18.000Z (about 4 years ago)
- Last Synced: 2024-11-08T00:37:32.818Z (2 months ago)
- Language: JavaScript
- Size: 1.12 MB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 12
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://travis-ci.org/kube-HPC/api-server)
[](https://coveralls.io/github/kube-HPC/api-server?branch=master)
## Documentation
The full doc can be found [here](https://kube-hpc.github.io/api-server/docs.html)
The Swagger-UI can found at http://localhost:3000/swagger-ui/
## Pipeline
The basic concept behind the Hkube pipeline is [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph)
Directed Acyclic Graph is a graph consisting of nodes connected with edges that have a direction: A -> B -> C
The reasons for choosing this structure are:
- represent an orderings processing of nodes
- data flowing between the nodes
- parallel and batch processing
### Example
Here we can see the most basic pipeline
```js
"nodes": [{
"nodeName": "green",
"algorithmName": "green-alg",
"input": [false, "OK"]
},
{
"nodeName": "yellow",
"algorithmName": "yellow-alg",
"input": [true, "@green"]
},
{
"nodeName": "red",
"algorithmName": "red-alg",
"input": ["@yellow", 512]
}]
```
The DAG of this pipeline will look like:
The blue circle is the pipeline driver which responsible for
executing nodes with the right order and the right input.
each result from any node is always return to the pipeline driver which
decide what to next. Here we have three nodes: green, yellow and red.
The green node -> pass it to yellow -> pass it to red.
The input of one node is the output of another.
The green node will run first because it does not depend on any other node.
Green node input will be: [false, "OK", 256].
Yellow node depends on green node, see the "@green" in the input of the yellow node.
So the input of the yellow node will be: [true, green node output].
The last node to run will be the red node because it depend on the completion of
the yellow node. The input of the red node will be: [yellow node output, 512].
The final results of this pipeline will be the output of the red node.
That because the red node is the last node in the pipeline.
### Batch Example
You can also execute nodes in parallel and reduce the results into a single node.
This example is exactly like the first one, except one detail.
The batch sign: # in the input of the green node.
```js
"nodes": [{
"nodeName": "green",
"algorithmName": "green-alg",
"input": [false, "#[1,2,3]"]
},
{
"nodeName": "yellow",
"algorithmName": "yellow-alg",
"input": [true, "@green"]
},
{
"nodeName": "red",
"algorithmName": "red-alg",
"input": ["@yellow", 512]
}]
```
The DAG of this pipeline will look like:
The green node will run as a batch because of the # sign in the input.
This pipeline will create three different tasks from type green-alg,
each with a different input:
```js
task 1: [false, "1"]
task 2: [false, "2"]
task 3: [false, "3"]
```
The yellow node will wait until all tasks of the green node will finish.
The input of the yellow node will be: [true, green node output].
The input of the red node will be: [yellow node output, 512].
### Another Batch Example
You can also create a batch processing on node results.
Lets say that green node returns an array: ['A', 'B', 'C'].
```js
"nodes": [{
"nodeName": "green",
"algorithmName": "green-alg",
"input": [false, "OK"]
},
{
"nodeName": "yellow",
"algorithmName": "yellow-alg",
"input": [false, "#@green"]
},
{
"nodeName": "red",
"algorithmName": "red-alg",
"input": ["@yellow", 512]
}]
```
The DAG of this pipeline will look like:
## License
[MIT](LICENSE)