{"id":18470496,"url":"https://github.com/ds2-lab/bespokv","last_synced_at":"2025-05-12T03:11:58.381Z","repository":{"id":132149662,"uuid":"212206007","full_name":"ds2-lab/bespokv","owner":"ds2-lab","description":null,"archived":false,"fork":false,"pushed_at":"2019-10-01T21:49:45.000Z","size":924,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-16T18:42:59.564Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ds2-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-01T21:49:12.000Z","updated_at":"2019-10-01T21:49:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"d91e4db0-9c9f-4809-ad94-c96de83fd10d","html_url":"https://github.com/ds2-lab/bespokv","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2Fbespokv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2Fbespokv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2Fbespokv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2Fbespokv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ds2-lab","download_url":"https://codeload.github.com/ds2-lab/bespokv/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253176418,"owners_count":21866141,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T10:14:08.384Z","updated_at":"2025-05-09T01:39:57.918Z","avatar_url":"https://github.com/ds2-lab.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BespoKV - README #\n\n\n## Dependencies ##\n* gcc 4.8 (required by folly)\n* autoconf-2.69 (requires v \u003e= 2.69)\n* google-glog\n* protobuf\n* libopenssl\n* gflags-master\n* boost_1_55_0: ./b2 install\n* folly (requires double_conversion)\n* libuuid\n* libevent\n* lz4 1.7.1 (redlock requires v \u003e= 1.7.1)\n* zookeeper (server + C binding lib)\n\n\n## How to compile? ##\n\n### Compile BespoKV ###\n\nGo to the src directory, for debugging mode, run 'make'\n\nTo enable compiler level optimization, run 'make opti'\n\n\n### Compile datalet application ###\n\nGo to the apps directory, for debugging mode, run 'make'\n\nTo enable compiler level optimization, run 'make opti'\n\n\n### Compile client lib ###\n\nTo compile client lib, go to the libckv dir and compile proto file:\n```\ncd apps/clibs/libckv\nprotoc --cpp_out=. ckv_proto.proto\n```\n\nMove the header file generated to the include folder:\n```\ncp *.h include/\n```\n\nNext, create a directory to compile the lib:\n```\nmkdir build\ncd build/\ncmake ..\nmake\n```\nlibckv.a will be available in build directory\n\n### Compile benchmark ###\n\nFirst, compile the client lib as shown in previous step.\n\nThen go to the bench directory and run 'make'\n\n\n## How to run? ##\n\n### Datalet backend ###\n\nFirst, you should have a backend datalet running, e.g., a Redis node:  \nGo to the Redis dir:  \n```\n$ ./redis-server --port 12346\n```   \n\nUnder apps/ , we implemented two applications for BespoKV. If you want the datalet backend to be a key-value store, type:\n```\n$ cd apps/ckv\n$ ./conkv -l 192.168.0.170 -p 11111 -t 1 \n```   \n\n### BespoKV ###\n\nTo run the BespoKV executable, go to the src dir:  \n```\n$ ./conproxy --config /root/conrun/conf/c1.json --datalets /root/conrun/conf/d1.cfg --shard shard1 --proxyAddr 192.168.0.170 --proxyClientPort 12345 \n```\n\nConfiguration files include:   \nA JSON formatted file specifying all options (num_replicas option might be a bit confusing and here it indicates how many replicas excluding the master replica), as below:\n```\n{\n\t\"zk_host\": \"192.168.0.173:2181\",\n\t\"kafka_broker\": \"192.168.0.173:9092\",\n\t\"consistency_model\": \"strong\",\n\t\"consistency_tech\": \"cr\",\n\t\"topology\": \"ms\",\n\t\"num_replicas\": \"2\",\n}\n```   \nAnd a datalet list file specifying all datalets, as below:\n```\n# 0: master; 1: slave\n192.168.0.171:11111:0\n192.168.0.171:11112:1\n192.168.0.171:11113:1\n```\n### Benchmarking ###\n\nTo run the YCSB trace bench:\n```\n$ cd bench\n$ ./bench_client -d 40 -l trace_dir/kv1M_op1M_uniform_text.run -t 32 -m 6 -r 2 -f hosts.cfg -R 0 -W 0\n```\nWhere hosts.cfg is an example host file including all hosts, -m indicates how many hosts, -r indicates how many replicas, -R specifies which replica to serve READ (-1 means any replica), and -W indicates which replica to serve WRITE (again, -1 means any replica for active/active topology).\nExample host file is shown as below:\n```\n192.168.0.171:12345 192.168.0.171:12348\n192.168.0.171:12346 192.168.0.171:12349\n192.168.0.171:12347 192.168.0.171:12350\n192.168.0.172:12345 192.168.0.172:12348\n192.168.0.172:12346 192.168.0.172:12349\n192.168.0.172:12347 192.168.0.172:12350\n```\n\nTo run the Redis benchmark:  \n```\n$ ./redis-benchmark -h hulk0 -p 12345 -c 50 -n 100000 -t set,get -P 32 -r 1000000  \n```   \nThis will send requests to conproxy, which will serve as a proxy forwarding requests between benchmark clients and Redis backend datalets.\n\n### Deployment ###\n\nTo launch zk and MQ on cloud, run:\n```\nbin/zookeeper-server-start.sh -daemon config/zookeeper.properties\nbin/MQ-server-start.sh -daemon config/server.properties\n```\n\nTo launch a cluster of BespoKV + conkv nodes, first add the data node info in slap.sh, then run:  \n```\n$ cd scripts  \n$ ./slap.sh runckv  \n$ ./slap.sh runcon\n```\nDocker based BespoKV is ==PARTIALLY== supported. To run containerized deployment:  \n```\n$ ./slap.sh docker_runckv\n$ ./slap.sh docker_runcon\n```\n\n### Configuration file setting for different scenarios ###\n```\n-----Master-Slave topology with strong consistency-----\n\nMaster:\n{\n\t\"zk_host\": \"192.168.0.173:2181\",\n\t\"kafka_broker\": \"192.168.0.173:9092\",\n\t\"consistency_model\": \"strong\",\n\t\"consistency_tech\": \"cr\",\n\t\"topology\": \"ms\",\n\t\"num_replicas\": \"2\",\n}\n\nSlave:\n{\n\t\"zk_host\": \"192.168.0.173:2181\",\n\t\"kafka_broker\": \"192.168.0.173:9092\",\n\t\"consistency_model\": \"strong\",\n\t\"consistency_tech\": \"cr\",\n\t\"topology\": \"slave\",\n\t\"num_replicas\": \"0\",\n}\n\n\n-----Master-Slave topology with eventual consistency-----\n\nMaster:\n{\n\t\"zk_host\": \"192.168.0.173:2181\",\n\t\"kafka_broker\": \"192.168.0.173:9092\",\n\t\"consistency_model\": \"eventual\",\n\t\"consistency_tech\": \"cr\",\n\t\"topology\": \"ms\",\n\t\"num_replicas\": \"0\",\n}\n\nSlave:\n{\n\t\"zk_host\": \"192.168.0.173:2181\",\n\t\"kafka_broker\": \"192.168.0.173:9092\",\n\t\"consistency_model\": \"evenual\",\n\t\"consistency_tech\": \"cr\",\n\t\"topology\": \"slave\",\n\t\"num_replicas\": \"0\",\n}\n\n\n-----Master-Slave topology without consistency-----\n\nMaster:\n{\n\t\"zk_host\": \"192.168.0.173:2181\",\n\t\"kafka_broker\": \"192.168.0.173:9092\",\n\t\"consistency_model\": \"without\",\n\t\"consistency_tech\": \"none\",\n\t\"topology\": \"ms\",\n\t\"num_replicas\": \"2\",\n}\n\nSlave:\n{\n\t\"zk_host\": \"192.168.0.173:2181\",\n\t\"kafka_broker\": \"192.168.0.173:9092\",\n\t\"consistency_model\": \"without\",\n\t\"consistency_tech\": \"none\",\n\t\"topology\": \"slave\",\n\t\"num_replicas\": \"0\",\n}\n\n-----No topology and without consistency-----\n{\n\t\"zk_host\": \"172.17.0.2:2181\",\n\t\"kafka_broker\": \"172.17.0.3:9092\",\n\t\"consistency_model\": \"without\",\n\t\"consistency_tech\": \"none\",\n\t\"topology\": \"no\",\n\t\"num_replicas\": \"0\",\n}\n\n\n-----Active-Active toplogy with strong consistency-----\n\n{\n\t\"zk_host\": \"172.17.0.2:2181\",\n\t\"kafka_broker\": \"172.17.0.3:9092\",\n\t\"consistency_model\": \"strong\",\n\t\"consistency_tech\": \"rl\",\n\t\"topology\": \"aa\",\n\t\"num_replicas\": \"0\",\n\t\"rl_host\": \"127.0.0.1:12121\"\n}\n\n-----Active-Active toplogy with eventual consistency-----\n\n{\n\t\"zk_host\": \"192.168.0.173:2181\",\n\t\"kafka_broker\": \"192.168.0.173:9092\",\n\t\"consistency_model\": \"eventual\",\n\t\"consistency_tech\": \"cr\",\n\t\"topology\": \"aa\",\n\t\"num_replicas\": \"0\",\n} \n\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fds2-lab%2Fbespokv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fds2-lab%2Fbespokv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fds2-lab%2Fbespokv/lists"}