{"id":13411992,"url":"https://github.com/chrislusf/vasto","last_synced_at":"2025-05-12T14:32:26.459Z","repository":{"id":57496846,"uuid":"117634649","full_name":"chrislusf/vasto","owner":"chrislusf","description":"A distributed key-value store. On Disk. Able to grow or shrink without service interruption.","archived":false,"fork":false,"pushed_at":"2019-03-07T20:29:11.000Z","size":1025,"stargazers_count":263,"open_issues_count":4,"forks_count":30,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-03-31T23:18:12.408Z","etag":null,"topics":["distributed-storage","key-value","replication","rocksdb"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chrislusf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-01-16T05:16:57.000Z","updated_at":"2025-02-21T12:55:21.000Z","dependencies_parsed_at":"2022-09-03T02:30:52.987Z","dependency_job_id":null,"html_url":"https://github.com/chrislusf/vasto","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrislusf%2Fvasto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrislusf%2Fvasto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrislusf%2Fvasto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrislusf%2Fvasto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chrislusf","download_url":"https://codeload.github.com/chrislusf/vasto/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253754959,"owners_count":21958933,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-storage","key-value","replication","rocksdb"],"created_at":"2024-07-30T20:01:19.877Z","updated_at":"2025-05-12T14:32:26.430Z","avatar_url":"https://github.com/chrislusf.png","language":"Go","funding_links":[],"categories":["Database","Distributed Data Stores","Go","数据库  `go语言实现的数据库`","数据库","Uncategorized","Generators","Data Integration Frameworks"],"sub_categories":["Databases Implemented in Go","Advanced Console UIs","Go中实现的数据库","标准 CLI"],"readme":"# Vasto\n\n[![Build Status](https://travis-ci.org/chrislusf/vasto.svg?branch=master)](https://travis-ci.org/chrislusf/vasto)\n[![GoDoc](https://godoc.org/github.com/chrislusf/vasto/goclient/vs?status.svg)](https://godoc.org/github.com/chrislusf/vasto/goclient/vs)\n[![Go Report Card](https://goreportcard.com/badge/github.com/chrislusf/vasto)](https://goreportcard.com/report/github.com/chrislusf/vasto)\n[![codecov](https://codecov.io/gh/chrislusf/vasto/branch/master/graph/badge.svg)](https://codecov.io/gh/chrislusf/vasto)\n\nA distributed high-performance key-value store. On Disk. Eventual consistent. HA. Able to grow or shrink without service interruption.\n\nVasto scales embedded [RocksDB](https://github.com/facebook/rocksdb) into a distributed key-value store,\nadding sharding, replication, and support operations to\n1. create a new keyspace\n1. delele an existing keyspace\n1. grow a keyspace\n1. shrink a keyspace\n1. replace a node in a keyspace\n\n# Why\nA key-value store is often re-invented. Why there is another one?\n\nVasto enables developers to setup a distributed key-value store as simple as creating a map object.\n\nThe operations, such as creating/deleting the store, partitioning, replications, seamlessly adding/removing servers, etc,\nare managed by a few commands.\nClient connection configurations are managed automatically.\n\nIn a sense, Vasto is an in-house cloud providing distributed key-value stores as a service, \nminus the need to balance performance and cloud service costs, plus consistent and low latency.\n\n# Architecture\n\nThere are one Vasto master and N number of Vasto stores, plus Vasto clients or Vasto proxies/gateways.\n1. The Vasto stores are basically simple wrapper of RocksDB. \n1. The Vasto master manages all the Vasto stores and Vasto clients.\n1. Vasto clients rely on the master to connect to the right Vasto stores.\n1. Vasto gateways use Vasto client libraries to support different APIs.\n\nThe master is the brain of the system. Vasto does not use gossip protocols, or other consensus algorithms.\nInstead, Vasto uses a single master for simple setup, fast failure detection, fast topology changes,\nand precise coordinations.\nThe master only contains soft states and is only required when topology changes. \nSo even if it ever crashes, a simple restart will recover everything.\n\nThe Vasto stores simply pass get/put/delete/scan requests to RocksDB. \nOne Vasto store can host multiple db instances.\n\nGo applications can use the client library directly.\n\nApplications in other languages can talk to the Vasto gateway, which uses the client library and reverse proxy the requests\nto the Vasto stores. The number of Vasto gateways are unlimited. \nThey can be installed on any application machines to reduce one network hop. \nOr can be on its dedicated machine to reduce number of connections to the Vasto stores if both the number of stores and the number of clients are very high.\n\n\n# Life cycle\n\nOne Vasto cluster has one master and multiple Vasto stores. When the store joins the cluster, it is just empty.\n\nWhen the master receives a request to create a keyspace with x shards and replication factor = y, the master would\n1. find x stores that meet the requirement and assign it to the keyspace\n1. ask the stores to create the shards, including replicas.\n1. inform the clients of the store locations\n\nWhen the master receives a request to resize the keyspace from m shards to n shards, the master would\n1. if size increased, find n-m stores that meet the requirement and assign it to the keyspace\n1. ask the stores to create the shards, including replicas.\n1. prepare the data to the new stores\n1. direct the clients traffic to the new stores\n1. remove retiring shards\n\n# Hashing algorithm\n\nVasto used [Jumping Consistent Hash](https://arxiv.org/abs/1406.2294) to allocate data. This algorithm\n1. requires no storage. The master only need soft state to manage all store servers. It is OK to restart master.\n1. evenly distribute the data into buckets.\n1. when the number of bucket changes, it can also evenly dividing the workload.\n\nWith this jumping hash, the cluster resizing is rather simple, flexible, and efficient:\n1. Cluster can resize up or down freely.\n1. Resizing is well coordinated.\n1. Data can be moved via the most efficient SSTable writes.\n1. Clients aware of the cluster change and can redirect traffic only when the new whole new server are ready.\n\n# Eventual Consistency and Active-Active Replication\n\nAll Vasto stores can be used to read and write. The changes will be propagated to other replicas within a few\nmilliseconds. Only the primary replica participate in the normal operations. The replica are participating\nwhen the primary replica is down, or in a different data center.\n\nVasto assumes the data already has the event time. It should be the time when the event really happens, not the\ntime when the data is feed into Vasto system. If the system fails over to the replica partition, and there are\nmultiple changes to one key, the one with latest event times will win.\n\n# Client APIs\n\nSee https://godoc.org/github.com/chrislusf/vasto/goclient/vs\n\n## Example\n\n```go\n    // create a vasto client talking to master at localhost:8278\n    vc := vs.NewVastoClient(context.Background(), \"client_name\", \"localhost:8278\")\n    \n    // create a cluster for keyspace ks1, with one server, and one copy of data.\n    vc.CreateCluster(\"ks1\", 1, 1)\n    \n    // get a cluster client for ks1\n    cc := vc.NewClusterClient(\"ks1\")\n\n    // operate with the cluster client\n    var key, value []byte\n    cc.Put(key, value)    \n    cc.Get(vs.Key(key))\n    ...\n\n    // change cluster size to 3 servers\n    vc.ResizeCluster(\"ks1\", 3)\n\n    // operate with the existing cluster client\n    cc.Put(key, value)    \n    cc.Get(vs.Key(key))\n    ...\n\n```\n\n\nCurrently only basic go library is provided. The gateway is not ready yet.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrislusf%2Fvasto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchrislusf%2Fvasto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrislusf%2Fvasto/lists"}