https://github.com/ds2-lab/bespokv

Last synced: about 1 year ago
JSON representation

Host: GitHub
URL: https://github.com/ds2-lab/bespokv
Owner: ds2-lab
Created: 2019-10-01T21:49:12.000Z (over 6 years ago)
Default Branch: master
Last Pushed: 2019-10-01T21:49:45.000Z (over 6 years ago)
Last Synced: 2025-02-16T18:42:59.564Z (over 1 year ago)
Language: C
Size: 902 KB
Stars: 0
Watchers: 2
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

# BespoKV - README #

## Dependencies ##
* gcc 4.8 (required by folly)
* autoconf-2.69 (requires v >= 2.69)
* google-glog
* protobuf
* libopenssl
* gflags-master
* boost_1_55_0: ./b2 install
* folly (requires double_conversion)
* libuuid
* libevent
* lz4 1.7.1 (redlock requires v >= 1.7.1)
* zookeeper (server + C binding lib)

## How to compile? ##

### Compile BespoKV ###

Go to the src directory, for debugging mode, run 'make'

To enable compiler level optimization, run 'make opti'

### Compile datalet application ###

Go to the apps directory, for debugging mode, run 'make'

To enable compiler level optimization, run 'make opti'

### Compile client lib ###

To compile client lib, go to the libckv dir and compile proto file:
```
cd apps/clibs/libckv
protoc --cpp_out=. ckv_proto.proto
```

Move the header file generated to the include folder:
```
cp *.h include/
```

Next, create a directory to compile the lib:
```
mkdir build
cd build/
cmake ..
make
```
libckv.a will be available in build directory

### Compile benchmark ###

First, compile the client lib as shown in previous step.

Then go to the bench directory and run 'make'

## How to run? ##

### Datalet backend ###

First, you should have a backend datalet running, e.g., a Redis node:
Go to the Redis dir:
```
$ ./redis-server --port 12346
```

Under apps/ , we implemented two applications for BespoKV. If you want the datalet backend to be a key-value store, type:
```
$ cd apps/ckv
$ ./conkv -l 192.168.0.170 -p 11111 -t 1
```

### BespoKV ###

To run the BespoKV executable, go to the src dir:
```
$ ./conproxy --config /root/conrun/conf/c1.json --datalets /root/conrun/conf/d1.cfg --shard shard1 --proxyAddr 192.168.0.170 --proxyClientPort 12345
```

Configuration files include:
A JSON formatted file specifying all options (num_replicas option might be a bit confusing and here it indicates how many replicas excluding the master replica), as below:
```
{
"zk_host": "192.168.0.173:2181",
"kafka_broker": "192.168.0.173:9092",
"consistency_model": "strong",
"consistency_tech": "cr",
"topology": "ms",
"num_replicas": "2",
}
```
And a datalet list file specifying all datalets, as below:
```
# 0: master; 1: slave
192.168.0.171:11111:0
192.168.0.171:11112:1
192.168.0.171:11113:1
```
### Benchmarking ###

To run the YCSB trace bench:
```
$ cd bench
$ ./bench_client -d 40 -l trace_dir/kv1M_op1M_uniform_text.run -t 32 -m 6 -r 2 -f hosts.cfg -R 0 -W 0
```
Where hosts.cfg is an example host file including all hosts, -m indicates how many hosts, -r indicates how many replicas, -R specifies which replica to serve READ (-1 means any replica), and -W indicates which replica to serve WRITE (again, -1 means any replica for active/active topology).
Example host file is shown as below:
```
192.168.0.171:12345 192.168.0.171:12348
192.168.0.171:12346 192.168.0.171:12349
192.168.0.171:12347 192.168.0.171:12350
192.168.0.172:12345 192.168.0.172:12348
192.168.0.172:12346 192.168.0.172:12349
192.168.0.172:12347 192.168.0.172:12350
```

To run the Redis benchmark:
```
$ ./redis-benchmark -h hulk0 -p 12345 -c 50 -n 100000 -t set,get -P 32 -r 1000000
```
This will send requests to conproxy, which will serve as a proxy forwarding requests between benchmark clients and Redis backend datalets.

### Deployment ###

To launch zk and MQ on cloud, run:
```
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
bin/MQ-server-start.sh -daemon config/server.properties
```

To launch a cluster of BespoKV + conkv nodes, first add the data node info in slap.sh, then run:
```
$ cd scripts
$ ./slap.sh runckv
$ ./slap.sh runcon
```
Docker based BespoKV is ==PARTIALLY== supported. To run containerized deployment:
```
$ ./slap.sh docker_runckv
$ ./slap.sh docker_runcon
```

### Configuration file setting for different scenarios ###
```
-----Master-Slave topology with strong consistency-----

Master:
{
"zk_host": "192.168.0.173:2181",
"kafka_broker": "192.168.0.173:9092",
"consistency_model": "strong",
"consistency_tech": "cr",
"topology": "ms",
"num_replicas": "2",
}

Slave:
{
"zk_host": "192.168.0.173:2181",
"kafka_broker": "192.168.0.173:9092",
"consistency_model": "strong",
"consistency_tech": "cr",
"topology": "slave",
"num_replicas": "0",
}

-----Master-Slave topology with eventual consistency-----

Master:
{
"zk_host": "192.168.0.173:2181",
"kafka_broker": "192.168.0.173:9092",
"consistency_model": "eventual",
"consistency_tech": "cr",
"topology": "ms",
"num_replicas": "0",
}

Slave:
{
"zk_host": "192.168.0.173:2181",
"kafka_broker": "192.168.0.173:9092",
"consistency_model": "evenual",
"consistency_tech": "cr",
"topology": "slave",
"num_replicas": "0",
}

-----Master-Slave topology without consistency-----

Master:
{
"zk_host": "192.168.0.173:2181",
"kafka_broker": "192.168.0.173:9092",
"consistency_model": "without",
"consistency_tech": "none",
"topology": "ms",
"num_replicas": "2",
}

Slave:
{
"zk_host": "192.168.0.173:2181",
"kafka_broker": "192.168.0.173:9092",
"consistency_model": "without",
"consistency_tech": "none",
"topology": "slave",
"num_replicas": "0",
}

-----No topology and without consistency-----
{
"zk_host": "172.17.0.2:2181",
"kafka_broker": "172.17.0.3:9092",
"consistency_model": "without",
"consistency_tech": "none",
"topology": "no",
"num_replicas": "0",
}

-----Active-Active toplogy with strong consistency-----

{
"zk_host": "172.17.0.2:2181",
"kafka_broker": "172.17.0.3:9092",
"consistency_model": "strong",
"consistency_tech": "rl",
"topology": "aa",
"num_replicas": "0",
"rl_host": "127.0.0.1:12121"
}

-----Active-Active toplogy with eventual consistency-----

{
"zk_host": "192.168.0.173:2181",
"kafka_broker": "192.168.0.173:9092",
"consistency_model": "eventual",
"consistency_tech": "cr",
"topology": "aa",
"num_replicas": "0",
}

```

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/ds2-lab/bespokv

Awesome Lists containing this project

README