https://github.com/jackdoe/elixir-vowpal-fleet
vowpal wabbit distributed supervisor using swarm and elixir (including handoff of the models)
https://github.com/jackdoe/elixir-vowpal-fleet
Last synced: 8 months ago
JSON representation
vowpal wabbit distributed supervisor using swarm and elixir (including handoff of the models)
- Host: GitHub
- URL: https://github.com/jackdoe/elixir-vowpal-fleet
- Owner: jackdoe
- License: mit
- Created: 2018-11-18T15:45:15.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2018-12-17T14:52:30.000Z (over 7 years ago)
- Last Synced: 2025-09-15T00:44:41.090Z (9 months ago)
- Language: Elixir
- Size: 39.1 KB
- Stars: 3
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# VowpalFleet
Vowpal Fleet - manage [Vowpal Wabbit](https://github.com/VowpalWabbit/vowpal_wabbit) instances usint `Swarm`
## Info
* create one cluster per model using `VowpalFleet.start_worker/2`
* `VowpalFleet.train/3` goes to all living instance
* `VowpalFleet.predict/2` picks random instance to do the prediction
* configure auto save interval, root directory and vw options with mix config `:vowpal_fleet` in `config.exs`
## Installation
* Make sure you have [Vowpal Wabbit](https://github.com/VowpalWabbit/vowpal_wabbit) installed and it is findable in `$PATH`
* add the dependency to your mix.exs
```elixir
def deps do
[
{:vowpal_fleet, "~> 0.1.0"}
]
end
ef application do
[
extra_applications: [:vowpal_fleet]
]
end
```
* configure the parameters, edit `config/config.exs`
```
config :vowpal_fleet,
root: "/tmp/vw",
some_cluster_id: %{:autosave => 300_000, :args => ["--random_seed", "123"]}
```
## Work In Progress
More testing is needed to ensure that the failure scenarios are covered, at the moment the code just works but.. well take it with grain of salt
## Examples
iex> VowpalFleet.start_worker(:some_cluster_id, :instance_1)
...
:ok
iex> VowpalFleet.start_worker(:some_cluster_id, :instance_2)
...
:ok
iex> VowpalFleet.train(:some_cluster_id, 1, [{"features", [1, 2, 3]}])
:ok
iex> VowpalFleet.predict(:some_cluster_id, [{"features", [1, 2, 3]}])
1.0
## Handoff
When the process has to be moved to a different node, the working model is saved, and then handed off to the starting process
## Links
[issues](https://github.com/jackdoe/elixir-vowpal-fleet/issues) [fork](https://github.com/jackdoe/elixir-vowpal-fleet) [license - MIT](https://en.wikipedia.org/wiki/MIT_License)
## credit
Icons made by [Freepik](https://www.freepik.com) from [Flaticon](https://www.flaticon.com/) is licensed by [CC 3.0 BY](http://creativecommons.org/licenses/by/3.0/)