{"id":13509650,"url":"https://github.com/deroproject/graviton","last_synced_at":"2025-03-30T13:32:40.341Z","repository":{"id":41382838,"uuid":"292903178","full_name":"deroproject/graviton","owner":"deroproject","description":"Graviton Database: ZFS for key-value stores.","archived":false,"fork":false,"pushed_at":"2022-01-30T07:06:39.000Z","size":446,"stargazers_count":418,"open_issues_count":4,"forks_count":26,"subscribers_count":18,"default_branch":"master","last_synced_at":"2024-08-02T02:14:23.962Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deroproject.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-04T17:03:50.000Z","updated_at":"2024-06-15T06:27:45.000Z","dependencies_parsed_at":"2022-09-26T18:31:16.702Z","dependency_job_id":null,"html_url":"https://github.com/deroproject/graviton","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deroproject%2Fgraviton","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deroproject%2Fgraviton/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deroproject%2Fgraviton/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deroproject%2Fgraviton/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deroproject","download_url":"https://codeload.github.com/deroproject/graviton/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222552993,"owners_count":17002160,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T02:01:10.979Z","updated_at":"2024-11-01T09:31:57.848Z","avatar_url":"https://github.com/deroproject.png","language":"Go","funding_links":[],"categories":["Go","Key-value Data Model","others"],"sub_categories":[],"readme":"\n# Graviton Database: ZFS for key-value stores.\n\nGraviton Database is simple, fast, versioned, authenticated, embeddable key-value store database in pure GOLANG.  \nGraviton Database in short is like \"ZFS for key-value stores\" in which every write is tracked, versioned and authenticated with cryptographic proofs. Additionally it is possible to take snapshots of database. Also it is possible to use simple copy,rsync commands for database backup even during live updates without any possibilities of database corruption.\n\n![Graviton: ZFS for key-value stores](images/GRAVITON.png?raw=true \"Graviton: ZFS for key-value stores\")\n\n## Project Status\nGraviton is currently alpha software. Almost full unit test coverage and randomized black box testing are used to ensure database consistency and thread safety. The project already has 100% code coverage. A number of decisions such as change,rename APIs, handling errors, hashing algorithms etc. are being evaluated and open for improvements and suggestions.\n\n## Features\nGraviton Database in short is  \"ZFS for key-value stores\".\n\n* Authenticated data store (All keys, values are backed by blake 256 bit checksum).\n* Append only data store.\n* Support of 2^64 trees (Theoretically) within a single data store. Trees can be named and thus used as buckets.\n* Support of values version tracking. All committed changes are versioned with ability to visit them at any point in time. \n* Snapshots (Multi tree commits in a single version causing multi bucket sync, each snapshot can be visited, appended and further modified, keys deleted, values modified etc., new keys, values stored.)\n* Ability to iterate over all key-value pairs in a tree.\n* Ability to diff between 2 trees in linear time and report all changes of Insertions, Deletions, Modifications.)\n* Minimal and simplified API.\n* Theoretically support Exabyte data store, Multi TeraByte tested internally.\n* Decoupled storage layer, allowing use of object stores such as Ceph, AWS etc.\n* Ability to generate cryptographic proofs which can prove key existance or non-existance (Cryptographic Proofs are around 1 KB.)\n* Superfast proof generation time of around 1000 proofs per second per core.\n* Support for disk based filesystem based persistant stores.\n* Support for memory based non-persistant stores.\n* 100% code coverage\n\n\n\n## Table of Contents\n1. [Getting Started](#getting-started) \n1. [Installing](#installing) \n1. [Opening and Using the Database](#opening-and-using-the-database) \n1. [Graviton Tree](#graviton-tree) \n1. [Using key,value pairs](#using-keyvalue-pairs) \n1. [Iterating over keys](#iterating-over-keys) \n1. [Snapshots](#snapshots) \n1. [Diffing](#diffing) (Diffing of 2 trees to detect changes between versions or compare 2 arbitrary trees in linear time.)\n1. [GravitonDB Backups](#gravitondb-backups) \n1. [Stress testing](#stress-testing) \n1. [Graviton Internals](#graviton-internals) \n1. [Lines of Code](#lines-of-Code) \n1. [TODO](#todo) \n1. [Comparison with other databases](#comparison-with-other-databases) (Mysql, Postgres, LevelDB, RocksDB, LMDB, Bolt etc.)\n1. [License](#license) \n\n\nGNU General Public License v3.0\n\n## Getting Started\n### Installing\nTo start using Graviton DB, install Go and run go get:\n\n```go get github.com/deroproject/graviton/...```\n\nThis will retrieve the library and build the library\n\n\n### Opening and Using the Database\n\nThe top-level object in Graviton is a Store. It is represented as a directory with multiple files on server's disk and represents a consistent snapshot of your data at all times.\n\nExample code to open database:\n\n    package main\n\n    import \"fmt\"\n    import \"github.com/deroproject/graviton\"\n\n    func main() {\n\t   //store, _ := graviton.NewDiskStore(\"/tmp/testdb\")   // create a new testdb in \"/tmp/testdb\"\n        store, _ := graviton.NewMemStore()            // create a new  DB in RAM\n        ss, _ := store.LoadSnapshot(0)           // load most recent snapshot\n        tree, _ := ss.GetTree(\"root\")            // use or create tree named \"root\"\n        tree.Put([]byte(\"key\"), []byte(\"value\")) // insert a value\n        graviton.Commit(tree)                  // commit the tree\n        value, _ := tree.Get([]byte(\"key\"))\n        fmt.Printf(\"value retrived from DB \\\"%s\\\"\\n\", string(value))\n    }\n\n    //NOTE: Linux (or other platforms) have open file limit for 1024. \n    //    Default limits allows upto 2TB of Graviton databases.\n\n### Graviton Tree\nA Tree in Graviton DB acts like a bucket in BoltDB or a ZFS dataset. It is named and can contain upto 128 byte names. Any store can contain infinite trees. Each tree can also contain infinite key-value pairs. However, practically being limited by the server or system storage space.\n\nEach tree can be accessed with its merkle root hash using \"*GetTreeWithRootHash*\" API. Also each tree maintains its own separate version number and any specific version can be used *GetTreeWithVersion*. Note that each tree can also have arbitrary tags and any tagged tree can be accessed using the tag *GetTreeWithTag*. Also, 2 arbitrary trees can diffed in linear time and relevant changes detected.\n\n    NOTE: Tree tags or names cannot start with ':' .\n\n### Using key,value pairs\n\nTo save a key/value pair to a tree ( or bucket), use the `tree.Put()` function:\n\n```go\n        tree, _ := ss.GetTree(\"root\") \n        tree.Put([]byte(\"answer\"), []byte(\"44\")) // insert a value\n        graviton.Commit(tree)  // make the tree persistant by storing it in backend disk\n```\n\nThis will set the value of the `\"answer\"` key to `\"44\"` in the `root`\ntree. To retrieve this value, we can use the `tree.Get()` function:\n\n```go\n\ttree, _ := ss.GetTree(\"root\") \n\tv,_ := tree.Get([]byte(\"answer\"))\n\tfmt.Printf(\"The answer is: %s\\n\", v)\n```\n\nThe `Get()` function returns an error because its operation is guaranteed to work (unless there is some kind of system failure which we try to report). If the key exists then it will return its byte slice value. If it doesn't exist then it\nwill return  an error. \n\n### Iterating over keys\n\nGraviton stores its keys in hash byte-sorted order within a tree. This makes sequential\niteration over these keys extremely fast. To iterate over keys GravitonDB uses a\n`Cursor`:\n\n```go\n\t// Assume \"root\" tree exists and has keys\n    tree, _ := store.GetTree(\"root\") \n\tc := tree.Cursor()\n\n\tfor k, v, err := c.First(); err == nil; k, v, err = c.Next() { \n\t\tfmt.Printf(\"key=%s, value=%s\\n\", k, v)\n\t}\n```\n\nThe cursor allows you to move to a specific point in the list of keys and move\nforward or backward through the keys one at a time.\n\nThe following functions are available on the cursor:\n\n```\nFirst()  Move to the first key.\nLast()   Move to the last key.\nNext()   Move to the next key.\nPrev()   Move to the previous key.\n```\n\nEach of those functions has a return signature of `(key []byte, value []byte, err error)`.\nWhen you have iterated to the end of the cursor then `Next()` will return an error `ErrNoMoreKeys`.  You must seek to a position using `First()`, `Last()`\nbefore calling `Next()` or `Prev()`. If you do not seek to a position then these functions will return an error.\n\n\n### Snapshots\nSnapshot refers to collective state of all buckets + data + history. Each commit( tree.Commit() or Commit(tree1, tree2 .....)) creates a new snapshot in the store.Each snapshot is represented by an incremental uint64 number, 0 represents most recent snapshot.\nSnapshots can be used to access any arbitrary state of entire database at any point in time.\n\nExample code for snapshots:\n\n    package main\n\n    import \"fmt\"\n    import \"github.com/deroproject/graviton\"\n\n    func main() {\n    \t   key := []byte(\"key1\")\n\t   //store, _ := graviton.NewDiskStore(\"/tmp/testdb\")   // create a new testdb in \"/tmp/testdb\"\n\t   store, _ := graviton.NewMemStore()          // create a new  DB in RAM\n\t   ss, _ := store.LoadSnapshot(0)         // load most recent snapshot\n\t   tree, _ := ss.GetTree(\"root\")          // use or create tree named \"root\"\n\t   tree.Put(key, []byte(\"commit_value1\")) // insert a value\n\t   commit1, _ := graviton.Commit(tree)         // commit the tree\n\t   tree.Put(key, []byte(\"commit_value2\")) // overwrite existing value\n\t   commit2, _ := graviton.Commit(tree)         // commit the tree again\n\n\t   // at this point, you have done 2 commits\n\t   // at first commit or snapshot,  \"root\" tree contains  \"key1 : commit_value1\"\n\t   // at second commit or snapshot,  \"root\" tree contains  \"key1 : commit_value2\"\n\n\t   // we will traverse now commit1 snapshot\n\t   ss, _ = store.LoadSnapshot(commit1)\n\t   tree, _ = ss.GetTree(\"root\")\n\t   value, err := tree.Get(key)\n\t   fmt.Printf(\" snapshot%d  key %s value %s err %s\\n\", ss.GetVersion(), string(key), string(value), err)\n\n\t   // we will traverse now commit2 snapshot\n\t   ss, _ = store.LoadSnapshot(commit2)\n\t   tree, _ = ss.GetTree(\"root\")\n\t   value, err = tree.Get(key)\n\t   fmt.Printf(\" snapshot%d  key %s value %s err %s\\n\", ss.GetVersion(), string(key), string(value), err)\n    }\n\n### Diffing\n#### Diffing of 2 trees to detect changes between versions or compare 2 arbitrary trees in linear time.\nTwo arbitrary trees can be diffed in linear time to detect changes. Changes are of 3 types insertions, deletions and modifications (Same key but value changed). If the reported changes are applied to base tree, it will be equivalent to the head tree being compared.\n\n    func Diff(base_tree, head_tree *Tree, deleted, modified, inserted DiffHandler) (err error)\n\nDiffhandler is a callback function of the following type having k,v as arguments\n\n    type DiffHandler func(k, v []byte)\n\nThe algorithm is linear time in the number of changes. Eg. a tree with billion KVs can be diffed with parent almost instantaneously.\n\n\n\n### GravitonDB Backups\nUse simple commands like cp, copy or rsync to sync a Graviton database even while the database is being updated. However, as the database might be continuously appending, backup will always lag a bit. And note that the database or backups will NEVER get corrupted during copying while commits are being done.\n\n### Stress Testing\nA mini tool to do single thread testing is provided which can be used to perform various tests on memory or disk backend.\n\n    go run github.com/deroproject/graviton/cmd/stress\n\nSee help using `--help` argument. To use disk backend, use `--memory=false`\n\n\n### Graviton Internals\nInternally, all trees are stored within a base-2 merkle with collapsing path. This means if tree has 4 billion key-value pairs, it will only be 32 level deep.This leads to tremendous savings in storage space.This also means when you modify an existing key-value, only limited amount of nodes are touched.\n\n\n### Lines of Code\n    ~/tools/gocloc   --by-file  node_inner.go tree.go snapshot.go proof.go node_leaf.go  store.go node.go  hash.go  const.go doc.go  diff_tree.go cursor.go \n    -----------------------------------------------------------------\n    File           files          blank        comment           code\n    -----------------------------------------------------------------\n    node_inner.go                    76             33            364\n    store.go                         69             22            250\n    tree.go                          75             71            250\n    proof.go                         30             16            171\n    snapshot.go                      36             18            155\n    node_leaf.go                     29              3            150\n    diff_tree.go                     34             33            133\n    cursor.go                        21             15            106\n    node.go                           5              3             35\n    const.go                          4              0             21\n    hash.go                           7              2             19\n    doc.go                           16             42              1\n    -----------------------------------------------------------------\n    TOTAL             12            402            258           1655\n    -----------------------------------------------------------------\n\n## TODO \n* Currently it is not optimized for speed and GC (Garbage collection).\n* Expose/build metrics.\n* Currently, we have error reportingapi to reports rot bits, but nothing about disks corruption, should we discard such error design and make the API simpler (except snapshots, tree loading, commiting, no more errors ). More discussion required on this hard-disk failures,errors etc. required.\n\n\n### Comparison with other databases\nNone of the following databases provides ability to traverse back-in-time for each and every commit. GravitonDB is the only DB which provides back-in-time. Also presently GravitonDB is the only database which can diff between 2 trees in linear time. Let's compare between other features of some databases.\n\n#### Postgres, MySQL, \u0026 other relational databases\n\nRelational databases structure data into rows and are only accessible through\nthe use of SQL. This approach provides flexibility in how you store and query\nyour data but also incurs overhead in parsing and planning SQL statements. GravitonDB\naccesses all data by a byte slice key. This makes GravitonDB fast to read and write\ndata by key but provides no built-in support for joining values together.\n\nMost relational databases (with the exception of SQLite) are standalone servers\nthat run separately from the application. This gives systems\nflexibility to connect multiple application servers to a single database\nserver but also adds overhead in serializing and transporting data over the\nnetwork. Graviton runs as a library included in your application so all data access\nhas to go through your application's process. This brings data closer to your\napplication but limits multi-process access to the data.\n\n\n\n#### LevelDB, RocksDB\n\nLevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Graviton in that\nthey are libraries bundled into the application, However, their underlying\nstructure is a log-structured merge-tree (LSM tree). An LSM tree optimizes\nrandom writes by using a write ahead log and multi-tiered, sorted files called\nSSTables. Graviton uses a base 2 merkle tree internally. Both approaches\nhave trade-offs.\n\nIf you require a high random write throughput or you need to use\nspinning disks then LevelDB could be a good choice unless there are requirements of versioning, authenticated proofs or other features of Graviton database.\n\n#### LMDB, BoltDB\n\nLMDB, Bolt are architecturally similar. Both use a B+ tree, have ACID semantics with fully serializable transactions, and support lock-free MVCC using a single writer and multiple readers.\n\nIn-addition LMDB heavily focuses on raw performance while BoltDB focus on simplicity and ease of use. For example, LMDB allows several unsafe actions such as direct writes for the sake of performance. Bolt opts to disallow actions which can leave the database in a corrupted state. The only exception to this in Bolt is `DB.NoSync`.GravitonDB does not leave the database in corrupted state at any point in time.\n\n\nIn-addition LMDB, BoltDB doesn't support versioning, snapshots, linear diffing etc. features only Graviton provides such features for now.\n\n\n### License \n[GNU General Public License v3.0](https://github.com/deroproject/graviton/blob/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderoproject%2Fgraviton","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fderoproject%2Fgraviton","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderoproject%2Fgraviton/lists"}