https://github.com/jazware/white-water
A distributed key value store written in go
https://github.com/jazware/white-water
Last synced: 9 months ago
JSON representation
A distributed key value store written in go
- Host: GitHub
- URL: https://github.com/jazware/white-water
- Owner: jazware
- Created: 2018-05-05T18:17:13.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2018-06-07T21:15:42.000Z (almost 8 years ago)
- Last Synced: 2025-09-09T07:37:03.983Z (9 months ago)
- Language: Go
- Size: 465 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# White Water
White Water is a distributed Key Value Store built using the Raft algorithm in Golang.
## Project Structure
White Water is separated into 3 modules
1. `messages` which drives mesage sending and receiving
- `messages.go` - Contains logic for ZeroMQ clients and ways to register message handlers and subscribe to specific types of messages.
- `hello.go` - Contains a handler for hello messages from the ZeroMQ broker
2. `raft` which handles Raft consensus logic
- `append.go` - Handles processing of Append Entries RPC messages
- `get.go` - Handles processing of Get messages from a client.
- `raft.go` - Contains core Raft logic with helper functions for message interaction and routing RPCs to respective handlers.
- `requestVote.go` - Handles processing of Request Vote RPC messages and all other election process logic
- `stuctures.go` - Holds all the data for the node structs and the RPC messages we use in Raft
- `timer.go` - Contains utility functions for reseting the timer used in raft elections, timeouts, and heartbeats
3. `storage` which handles the state machine interface
- `storage.go` - Contains structures for managing the state of the node and logs. Implements the get and set interfaces for updating the local model after a consensus round. Allows leaders to get local values from the model.
## Running White Water
To run White Water, first ensure you have Golang installed and correctly configured, we recommend Go 1.8+.
For additional instructions, see [Golang Installation](https://golang.org/doc/install).
Next, ensure you have the ZeroMQ dependencies by installing `ZeroMQ version 4.0.1 or above. To use CURVE security in versions prior to 4.2, ZeroMQ must be installed with libsodium enabled.`
For additional instructions, see the [ZMQ4 Github Project](https://github.com/pebbe/zmq4).
Finally, run `go build` in our source repository.
Note that this requires pulling from GitHub and thus we have to make our project public in order for you to pull the dependencies for White Water.
Once you've built the source, you can stay in the directory that contains `node.go` and run `chistributed` as you would normally.
Our tests are inside `scripts/` and can be run with the `--run` flag in chistributed.
For tests with 5 nodes, please include a `--config-file scripts/five-nodes.conf` flag when running chistributed.
Example outputs of our test runs are provided in the `scripts/` directory.
You'll note that our tests don't always look the same because leader election in Raft is nondeterministic, thus the leader node is not predictable when causing specific failures.
To get around this, we provided output from non-leader nodes that point to the current leader and also provide answers to get requests so tests are easier to run.
## Tests
Run tests from the impl directory:
1. Multiple sets
`chistributed --run scripts/multi-sets.chi`
2. Numerous Sets and gets (> 80)
`chistributed --run scripts/big-sets.chi`
3. Basic Failures
`chistributed --run scripts/simple-fail.chi`
4. Multiple Different Failures
`chistributed --run scripts/multi-fail.chi`
5. Basis Partition
`chistributed --run scripts/basic-partition.chi`
6. Partition with Leader in majority
`chistributed --config-file scripts/five-nodes.conf --run scripts/big-1-partition.chi`
7. Partition with Leader in minority
`chistributed --config-file scripts/five-nodes.conf --run scripts/big-2-partition.chi`
Tests outputs from the creators of White Water are in impl/scripts
and are of the form [testname]-sample.out
These tests were run on two machines.
Machine 1: UChicago CS VM 201718.3 (Ubuntu 16.04.3 LTS)
Machine 2: Dell Precision 5510, 32GB of RAM, Intel(R) Xeon(R) CPU E3-1505M v5 @ 2.80GHz (Ubuntu 16.04.4 LTS)