{"id":16766391,"url":"https://github.com/mosuka/cete","last_synced_at":"2025-04-08T04:19:35.844Z","repository":{"id":91545172,"uuid":"178297688","full_name":"mosuka/cete","owner":"mosuka","description":"Cete is a distributed key value store server written in Go built on top of BadgerDB.","archived":false,"fork":false,"pushed_at":"2020-09-22T16:31:55.000Z","size":249,"stargazers_count":236,"open_issues_count":15,"forks_count":27,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-24T09:29:10.956Z","etag":null,"topics":["go","golang","grpc","key-value-store","kvs","raft","raft-consensus-algorithm","rest","rest-api"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mosuka.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-03-28T23:31:28.000Z","updated_at":"2025-03-19T12:46:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"c77959db-1471-432b-8c04-abafa9724209","html_url":"https://github.com/mosuka/cete","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mosuka%2Fcete","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mosuka%2Fcete/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mosuka%2Fcete/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mosuka%2Fcete/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mosuka","download_url":"https://codeload.github.com/mosuka/cete/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247773739,"owners_count":20993640,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","grpc","key-value-store","kvs","raft","raft-consensus-algorithm","rest","rest-api"],"created_at":"2024-10-13T06:06:14.957Z","updated_at":"2025-04-08T04:19:35.801Z","avatar_url":"https://github.com/mosuka.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cete\n\nCete is a distributed key value store server written in [Go](https://golang.org) built on top of [BadgerDB](https://blog.dgraph.io/post/badger/).  \nIt provides functions through [gRPC](http://www.grpc.io) ([HTTP/2](https://en.wikipedia.org/wiki/HTTP/2) + [Protocol Buffers](https://developers.google.com/protocol-buffers/)) or traditional [RESTful](https://en.wikipedia.org/wiki/Representational_state_transfer) API ([HTTP/1.1](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol) + [JSON](http://www.json.org)).  \nCete implements [Raft consensus algorithm](https://raft.github.io/) by [hashicorp/raft](https://github.com/hashicorp/raft). It achieve consensus across all the instances of the nodes, ensuring that every change made to the system is made to a quorum of nodes, or none at all.  \nCete makes it easy bringing up a cluster of BadgerDB (a cete of badgers) .\n\n\n## Features\n\n- Easy deployment\n- Bringing up cluster\n- Database replication\n- An easy-to-use HTTP API\n- CLI is also available\n- Docker container image is available\n\n\n## Building Cete\n\nWhen you satisfied dependencies, let's build Cete for Linux as following:\n\n```bash\n$ mkdir -p ${GOPATH}/src/github.com/mosuka\n$ cd ${GOPATH}/src/github.com/mosuka\n$ git clone https://github.com/mosuka/cete.git\n$ cd cete\n$ make build\n```\n\nIf you want to build for other platform, set `GOOS`, `GOARCH` environment variables. For example, build for macOS like following:\n\n```bash\n$ make GOOS=darwin build\n```\n\n### Binaries\n\nYou can see the binary file when build successful like so:\n\n```bash\n$ ls ./bin\ncete\n```\n\n\n## Testing Cete\n\nIf you want to test your changes, run command like following:\n\n```bash\n$ make test\n```\n\n\n## Packaging Cete\n\n###  Linux\n\n```bash\n$ make GOOS=linux dist\n```\n\n### macOS\n\n```bash\n$ make GOOS=darwin dist\n```\n\n## Configure Cete\n\n| CLI Flag | Environment variable | Configuration File | Description |\n| --- | --- | --- | --- |\n| --config-file | - | - | config file. if omitted, cete.yaml in /etc and home directory will be searched |\n| --id | CETE_ID | id | node ID |\n| --raft-address | CETE_RAFT_ADDRESS | raft_address | Raft server listen address |\n| --grpc-address | CETE_GRPC_ADDRESS | grpc_address | gRPC server listen address |\n| --http-address | CETE_HTTP_ADDRESS | http_address | HTTP server listen address |\n| --data-directory | CETE_DATA_DIRECTORY | data_directory | data directory which store the key-value store data and Raft logs |\n| --peer-grpc-address | CETE_PEER_GRPC_ADDRESS | peer_grpc_address | listen address of the existing gRPC server in the joining cluster |\n| --certificate-file | CETE_CERTIFICATE_FILE | certificate_file | path to the client server TLS certificate file |\n| --key-file | CETE_KEY_FILE | key_file | path to the client server TLS key file |\n| --common-name | CETE_COMMON_NAME | common_name | certificate common name |\n| --log-level | CETE_LOG_LEVEL | log_level | log level |\n| --log-file | CETE_LOG_FILE | log_file | log file |\n| --log-max-size | CETE_LOG_MAX_SIZE | log_max_size | max size of a log file in megabytes |\n| --log-max-backups | CETE_LOG_MAX_BACKUPS | log_max_backups | max backup count of log files |\n| --log-max-age | CETE_LOG_MAX_AGE | log_max_age | max age of a log file in days |\n| --log-compress | CETE_LOG_COMPRESS | log_compress | compress a log file |\n\n\n## Starting Cete node\n\nStarting cete is easy as follows:\n\n```bash\n$ ./bin/cete start --id=node1 --raft-address=:7000 --grpc-address=:9000 --http-address=:8000 --data-directory=/tmp/cete/node1\n```\n\nYou can get the node information with the following command:\n\n```bash\n$ ./bin/cete node | jq .\n```\n\nor the following URL:\n\n```bash\n$ curl -X GET http://localhost:8000/v1/node | jq .\n```\n\nThe result of the above command is:\n\n```json\n{\n  \"node\": {\n    \"raft_address\": \":7000\",\n    \"metadata\": {\n      \"grpc_address\": \":9000\",\n      \"http_address\": \":8000\"\n    },\n    \"state\": \"Leader\"\n  }\n}\n```\n\n## Health check\n\nYou can check the health status of the node.\n\n```bash\n$ ./bin/cete healthcheck | jq .\n```\n\nAlso provides the following REST APIs\n\n### Liveness prove\n\nThis endpoint always returns 200 and should be used to check Cete health.\n\n```bash\n$ curl -X GET http://localhost:8000/v1/liveness_check | jq .\n```\n\n### Readiness probe\n\nThis endpoint returns 200 when Cete is ready to serve traffic (i.e. respond to queries).\n\n```bash\n$ curl -X GET http://localhost:8000/v1/readiness_check | jq .\n```\n\n## Putting a key-value\n\nTo put a key-value, execute the following command:\n\n```bash\n$ ./bin/cete set 1 value1\n```\n\nor, you can use the RESTful API as follows:\n\n```bash\n$ curl -X PUT 'http://127.0.0.1:8000/v1/data/1' --data-binary value1\n$ curl -X PUT 'http://127.0.0.1:8000/v1/data/2' -H \"Content-Type: image/jpeg\" --data-binary @/path/to/photo.jpg\n```\n\n## Getting a key-value\n\nTo get a key-value, execute the following command:\n\n```bash\n$ ./bin/cete get 1\n```\n\nor, you can use the RESTful API as follows:\n\n```bash\n$ curl -X GET 'http://127.0.0.1:8000/v1/data/1'\n```\n\nYou can see the result. The result of the above command is:\n\n```text\nvalue1\n```\n\n## Deleting a key-value\n\nDeleting a value by key, execute the following command:\n\n```bash\n$ ./bin/cete delete 1\n```\n\nor, you can use the RESTful API as follows:\n\n```bash\n$ curl -X DELETE 'http://127.0.0.1:8000/v1/data/1'\n```\n\n\n## Bringing up a cluster\n\nCete is easy to bring up the cluster. Cete node is already running, but that is not fault tolerant. If you need to increase the fault tolerance, bring up 2 more data nodes like so:\n\n```bash\n$ ./bin/cete start --id=node2 --raft-address=:7001 --grpc-address=:9001 --http-address=:8001 --data-directory=/tmp/cete/node2 --peer-grpc-address=:9000\n$ ./bin/cete start --id=node3 --raft-address=:7002 --grpc-address=:9002 --http-address=:8002 --data-directory=/tmp/cete/node3 --peer-grpc-address=:9000\n```\n\n_Above example shows each Cete node running on the same host, so each node must listen on different ports. This would not be necessary if each node ran on a different host._\n\nThis instructs each new node to join an existing node, each node recognizes the joining clusters when started.\nSo you have a 3-node cluster. That way you can tolerate the failure of 1 node. You can check the cluster with the following command:\n\n```bash\n$ ./bin/cete cluster | jq .\n```\n\nor, you can use the RESTful API as follows:\n\n```bash\n$ curl -X GET 'http://127.0.0.1:8000/v1/cluster' | jq .\n```\n\nYou can see the result in JSON format. The result of the above command is:\n\n```json\n{\n  \"cluster\": {\n    \"nodes\": {\n      \"node1\": {\n        \"raft_address\": \":7000\",\n        \"metadata\": {\n          \"grpc_address\": \":9000\",\n          \"http_address\": \":8000\"\n        },\n        \"state\": \"Leader\"\n      },\n      \"node2\": {\n        \"raft_address\": \":7001\",\n        \"metadata\": {\n          \"grpc_address\": \":9001\",\n          \"http_address\": \":8001\"\n        },\n        \"state\": \"Follower\"\n      },\n      \"node3\": {\n        \"raft_address\": \":7002\",\n        \"metadata\": {\n          \"grpc_address\": \":9002\",\n          \"http_address\": \":8002\"\n        },\n        \"state\": \"Follower\"\n      }\n    },\n    \"leader\": \"node1\"\n  }\n}\n```\n\nRecommend 3 or more odd number of nodes in the cluster. In failure scenarios, data loss is inevitable, so avoid deploying single nodes.\n\nThe above example, the node joins to the cluster at startup, but you can also join the node that already started on standalone mode to the cluster later, as follows:\n\n```bash\n$ ./bin/cete join --grpc-addr=:9000 node2 127.0.0.1:9001\n```\n\nor, you can use the RESTful API as follows:\n\n```bash\n$ curl -X PUT 'http://127.0.0.1:8000/v1/cluster/node2' --data-binary '\n{\n  \"raft_address\": \":7001\",\n  \"metadata\": {\n    \"grpc_address\": \":9001\",\n    \"http_address\": \":8001\"\n  }\n}\n'\n```\n\nTo remove a node from the cluster, execute the following command:\n\n```bash\n$ ./bin/cete leave --grpc-addr=:9000 node2\n```\n\nor, you can use the RESTful API as follows:\n\n```bash\n$ curl -X DELETE 'http://127.0.0.1:8000/v1/cluster/node2'\n```\n\nThe following command indexes documents to any node in the cluster:\n\n```bash\n$ ./bin/cete set 1 value1 --grpc-address=:9000 \n```\n\nSo, you can get the document from the node specified by the above command as follows:\n\n```bash\n$ ./bin/cete get 1 --grpc-address=:9000\n```\n\nYou can see the result. The result of the above command is:\n\n```text\nvalue1\n```\n\nYou can also get the same document from other nodes in the cluster as follows:\n\n```bash\n$ ./bin/cete get 1 --grpc-address=:9001\n$ ./bin/cete get 1 --grpc-address=:9002\n```\n\nYou can see the result. The result of the above command is:\n\n```text\nvalue1\n```\n\n\n## Cete on Docker\n\n### Building Cete Docker container image on localhost\n\nYou can build the Docker container image like so:\n\n```bash\n$ make docker-build\n```\n\n### Pulling Cete Docker container image from docker.io\n\nYou can also use the Docker container image already registered in docker.io like so:\n\n```bash\n$ docker pull mosuka/cete:latest\n```\n\nSee https://hub.docker.com/r/mosuka/cete/tags/\n\n\n### Pulling Cete Docker container image from docker.io\n\nYou can also use the Docker container image already registered in docker.io like so:\n\n```bash\n$ docker pull mosuka/cete:latest\n```\n\n\n### Running Cete node on Docker\n\nRunning a Cete data node on Docker. Start Cete node like so:\n\n```bash\n$ docker run --rm --name cete-node1 \\\n    -p 7000:7000 \\\n    -p 8000:8000 \\\n    -p 9000:9000 \\\n    mosuka/cete:latest cete start \\\n      --id=node1 \\\n      --raft-address=:7000 \\\n      --grpc-address=:9000 \\\n      --http-address=:8000 \\\n      --data-directory=/tmp/cete/node1\n```\n\nYou can execute the command in docker container as follows:\n\n```bash\n$ docker exec -it cete-node1 cete node --grpc-address=:9000\n```\n\n## Securing Cete\n\nCete supports HTTPS access, ensuring that all communication between clients and a cluster is encrypted.\n\n### Generating a certificate and private key\n\nOne way to generate the necessary resources is via [openssl](https://www.openssl.org/). For example:\n\n```bash\n$ openssl req -x509 -nodes -newkey rsa:4096 -keyout ./etc/cete-key.pem -out ./etc/cete-cert.pem -days 365 -subj '/CN=localhost'\nGenerating a 4096 bit RSA private key\n............................++\n........++\nwriting new private key to 'key.pem'\n```\n\n### Secure cluster example\n\nStarting a node with HTTPS enabled, node-to-node encryption, and with the above configuration file. It is assumed the HTTPS X.509 certificate and key are at the paths server.crt and key.pem respectively.\n\n```bash\n$ ./bin/cete start --id=node1 --raft-address=:7000 --grpc-address=:9000 --http-address=:8000 --data-directory=/tmp/cete/node1 --peer-grpc-address=:9000 --certificate-file=./etc/cert.pem --key-file=./etc/key.pem --common-name=localhost\n$ ./bin/cete start --id=node2 --raft-address=:7001 --grpc-address=:9001 --http-address=:8001 --data-directory=/tmp/cete/node2 --peer-grpc-address=:9000 --certificate-file=./etc/cert.pem --key-file=./etc/key.pem --common-name=localhost\n$ ./bin/cete start --id=node3 --raft-address=:7002 --grpc-address=:9002 --http-address=:8002 --data-directory=/tmp/cete/node3 --peer-grpc-address=:9000 --certificate-file=./etc/cert.pem --key-file=./etc/key.pem --common-name=localhost\n```\n\nYou can access the cluster by adding a flag, such as the following command:\n\n```bash\n$ ./bin/cete cluster --grpc-address=:9000 --certificate-file=./cert.pem --common-name=localhost | jq .\n```\n\nor\n\n```bash\n$ curl -X GET https://localhost:8000/v1/cluster --cacert ./cert.pem | jq .\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmosuka%2Fcete","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmosuka%2Fcete","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmosuka%2Fcete/lists"}