Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/phillippohlandt/pipelinex
Elixir library to build simple data pipelines
https://github.com/phillippohlandt/pipelinex
elixir elixir-lang pipeline pipelines
Last synced: 4 days ago
JSON representation
Elixir library to build simple data pipelines
- Host: GitHub
- URL: https://github.com/phillippohlandt/pipelinex
- Owner: PhillippOhlandt
- License: mit
- Created: 2017-06-30T12:12:23.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2017-06-30T13:04:32.000Z (over 7 years ago)
- Last Synced: 2024-03-14T19:31:44.997Z (8 months ago)
- Topics: elixir, elixir-lang, pipeline, pipelines
- Language: Elixir
- Size: 7.81 KB
- Stars: 6
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Pipelinex
Pipelinex is a simple library to build data pipelines in a clean and structured way.
It's mainly built for personal usage to help with structuring big data processing flows
and automatically apply things like logging, which otherwise would make the code very unclean.
## Installation
The package can be installed by adding `pipelinex`
to your list of dependencies in `mix.exs`:
```elixir
def deps do
[{:pipelinex, "~> 0.1.0"}]
end
```
## Usage
Pipelinex provides two macros to build pipelines, `task` and `pipeline`.
### Simple Example
Build simple pipelines using the `task` macro:
```elixir
defmodule MyPipeline do
use Pipelinex
task "add 5", num do
num + 5
end
task "add 10", num do
num + 10
end
end
```
Execute the pipeline via it's injected `run/1` function:
```elixir
result = MyPipeline.run(5)
result == 20
```
### Reference Other Pipelines
Pipelines can execute other pipelines using the `pipeline` macro:
```elixir
defmodule MyOtherPipeline do
use Pipelinex
pipeline MyPipeline
task "add 20", num do
num + 20
end
end
result = MyOtherPipeline.run(5)
result == 40
```
### Pattern Matching
The `task` macro supports pattern matching for its data argument.
```elixir
defmodule MyPipeline do
use Pipelinex
task "add 5", %{:value => val} = map do
Map.put map, :value, val + 5
end
task "add 10", %{:value => val} = map do
Map.put map, :value, val + 10
end
end
```
If two or more tasks with the same name are defined one after another,
the task will be called once and normal pattern matching behaviour will be applied.
```elixir
defmodule MyPipeline do
use Pipelinex
task "add 5", %{:value => val} = map do
Map.put map, :value, val + 5
end
task "add 5", num do
num + 5
end
end
value = MyPipeline.run(%{:value => 5})
value == %{value: 10}
value = MyPipeline.run(5)
value == 10
```
### Logging
Pipelinex logs when a task starts and ends using `Logger.info`
```elixir
defmodule MyPipeline do
use Pipelinex
task "add 5", num do
num + 5
end
task "add 10", num do
num + 10
end
end
MyPipeline.run(5)
# 20
#
# 13:27:43.870 [info] Start: add 5
#
# 13:27:43.870 [info] End: add 5
#
# 13:27:43.870 [info] Start: add 10
#
# 13:27:43.870 [info] End: add 10
```