{"id":13439255,"url":"https://github.com/aergoio/litetree","last_synced_at":"2025-05-16T12:09:10.854Z","repository":{"id":48273823,"uuid":"143778005","full_name":"aergoio/litetree","owner":"aergoio","description":"SQLite with Branches","archived":false,"fork":false,"pushed_at":"2020-03-06T17:50:00.000Z","size":8153,"stargazers_count":1627,"open_issues_count":8,"forks_count":34,"subscribers_count":38,"default_branch":"master","last_synced_at":"2025-04-02T07:38:25.126Z","etag":null,"topics":["aergo","database","sqlite3"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aergoio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-06T20:12:23.000Z","updated_at":"2025-03-29T22:25:51.000Z","dependencies_parsed_at":"2022-08-12T19:41:28.853Z","dependency_job_id":null,"html_url":"https://github.com/aergoio/litetree","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aergoio%2Flitetree","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aergoio%2Flitetree/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aergoio%2Flitetree/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aergoio%2Flitetree/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aergoio","download_url":"https://codeload.github.com/aergoio/litetree/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248008628,"owners_count":21032556,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aergo","database","sqlite3"],"created_at":"2024-07-31T03:01:12.406Z","updated_at":"2025-04-09T09:03:08.802Z","avatar_url":"https://github.com/aergoio.png","language":"C","funding_links":[],"categories":["C","database","other","\u003ca name=\"C\"\u003e\u003c/a\u003eC"],"sub_categories":[],"readme":"\u003ca href=\"https://aergo.io\"\u003e\u003cimg src=\"http://litesync.io/litetree/aergo-presents-black-round.svg\" width=\"20%\"\u003e\u003c/a\u003e [![Build Status](https://travis-ci.org/aergoio/litetree.svg?branch=master)](https://travis-ci.org/aergoio/litetree) [![Build Status](https://ci.appveyor.com/api/projects/status/github/aergoio/litetree?branch=master\u0026svg=true)](https://ci.appveyor.com/project/kroggen/litetree)\n\n# LiteTree: SQLite with Branches\n\n![](http://litesync.io/litetree/graph-litetree.png)\n\nImagine being able to have many connections to the same database, each one reading a separate branch or commit at the same time. Or even writing to separate branches.\n\nThis is possible with **LiteTree**. It is a modification of the SQLite engine to support branching, like git!\n\nDatabase branching is a very useful tool for blockchain implementations and LiteTree will be at the core of [Aergo](https://github.com/aergoio/aergo).\n\nThis is how it works:\n\nEach database transaction is saved as a commit, and each commit has an incremental number. Let's consider an empty db in which we run this first SQL command:\n\n```\nCREATE TABLE t1 (name)\n```\n\nNow it will have the first commit (number 1) in the automatically created `master` branch:\n\n![](http://litesync.io/litetree/graph-1-commit.png)\n\nWhen we execute new transactions it will add new commits to the current branch:\n\n```\nINSERT INTO t1 VALUES ('first')\nINSERT INTO t1 VALUES ('second')\n```\nNow we have 3 commits:\n\n![](http://litesync.io/litetree/graph-3-commits.png)\n\nTo include many SQL commands in a single commit we must enclose them in `BEGIN` and `COMMIT`  commands.\n\nWe create new branches informing the source branch and commit number:\n\n```\nPRAGMA new_branch=test at master.2\n```\n\nAfter this command is executed the new branch is created but without any new data added to it. The database connection also moves to this new branch, having it as the current branch.\n\nWe can check the current branch with the command:\n\n```\nPRAGMA branch\n```\n\nIn this case it will return: `test`\n\nIf we execute a SQL command on this db connection the commit will be saved in the connection's current branch:\n\n```\nINSERT INTO t1 VALUES ('from test branch')\n```\n\nNow the graph state will be:\n\n![](http://litesync.io/litetree/graph-2-branches.png)\n\nWe can also read the database at this new branch:\n```\nSELECT * FROM t1\n```\n\nIt will return these values:\n\n\n\u003e first\n\u003e\n\u003e from test branch\n\n\nWe can move to the master branch:\n\n```\nPRAGMA branch=master\n```\n\nAnd executing the same SELECT command (but now in the master branch) it will return:\n\n\u003e first\n\u003e\n\u003e second\n\nDifferent content for the same table on separate branches.\n\nCommits in separate branches have the same numbering based on the distance from the first commit:\n\n![](http://litesync.io/litetree/graph-commit-numbers.png)\n\nWe can read the database in a previous point-in-time by moving to that commit, like this:\n\n```\nPRAGMA branch=master.2\n```\n\nAt this point the table `t1` has a single row and if we do a SELECT it will return just `first`.\n\nWe cannot write to the database when we are in a defined commit, writing is only possible at the head of each branch. If you want to make modifications to some previous commit you must create a new branch that starts at that commit.\n\nIt is also possible to truncate a branch at a specific commit, rename a branch, delete it and retrieve branch info.\n\n## Supported commands\n\n- Selecting the active branch:\n\t```\n\tPRAGMA branch=\u003cname\u003e\n\t```\n- Selecting a specific commit in a branch:\n\t```\n\tPRAGMA branch=\u003cname\u003e.\u003ccommit\u003e\n\t```\n- Retrieving the current/active branch:\n\t```\n\tPRAGMA branch\n\t```\n- Listing the existing branches:\n\t```\n\tPRAGMA branches\n\t```\n- Creating a new branch:\n\t```\n\tPRAGMA new_branch=\u003cname\u003e at \u003csource\u003e.\u003ccommit\u003e\n\t```\n- Deleting a branch:\n\t```\n\tPRAGMA del_branch(\u003cname\u003e)\n\t```\n- Renaming a branch:\n\t```\n\tPRAGMA rename_branch \u003cold_name\u003e \u003cnew_name\u003e\n\t```\n- Truncating a branch at a specific commit:\n\t```\n\tPRAGMA branch_truncate(\u003cname\u003e.\u003ccommit\u003e)\n\t```\n- Displaying the tree structure:\n\t```\n\tPRAGMA branch_tree\n\t```\n- Retrieving the branch info:\n\t```\n\tPRAGMA branch_info(\u003cname\u003e)\n\t```\n- Showing the commit and SQL log/history for a branch:\n\t```\n\tPRAGMA branch_log(\u003cname\u003e)\n\t```\n\n#### Not yet available\n\nSome of these commands are being developed:\n\n- Modifying a commit:\n\t```\n\tPRAGMA branch_log [--set|--add|--del] \u003cname\u003e \u003csql commands\u003e\n\t```\n- Showing the diff between 2 branches or commits:\n\t```\n\tPRAGMA branch_diff \u003cfrom_branch\u003e[.\u003ccommit\u003e] \u003cto_branch\u003e[.\u003ccommit\u003e]\n\t```\n- [Save metadata to each branch and/or commit](https://github.com/aergoio/litetree/wiki/Storing-metadata)\n- [Merging 2 branches](https://github.com/aergoio/litetree/wiki/Merging-branches)\n\nAnd maybe these extended features could be supported:\n\n- Access control by branch\n\nCheck the roadmap on our [wiki](https://github.com/aergoio/litetree/wiki). Feature requests and suggestions are welcome.\n\n\n## Technologies\n\nWe can use LiteTree with big databases (many gigabytes). There is no data copying when a new branch is created. When a new transaction is commited only the modified database pages are copied.\n\nLiteTree is implemented storing the SQLite db pages on LMDB.\n\nThe data is not compressed, and each db page is stored on just one disk sector (4096 bytes by default). This is achieved by reserving some bytes at each SQLite db page so it can fit into one LMDB overflow page, that can hold 4080 (4096 - 16) bytes.\n\n## Performance\n\nLiteTree is way faster than normal SQLite (journal mode) with comparable performance to WAL mode.\n\nHere are the some results:\n\n##### Linux\n\n```\nwriting:\n--------\nnormal   = 22.8921730518 seconds\nwal      = 10.7780168056 seconds\nmmap     = 10.4009709358 seconds\nlitetree = 10.8633410931 seconds\n\nreading:\n--------\nnormal   = 0.817955970764 seconds\nwal      = 0.660045146942 seconds\nmmap     = 0.592491865158 seconds\nlitetree = 0.619393110275 seconds\n```\n\n##### MacOSX\n\n```\nwriting:\n--------\nnormal   = 1.9102909565 seconds\nwal      = 1.30300784111 seconds\nmmap     = 1.21677088737 seconds\nlitetree = 0.988132953644 seconds\n\nreading:\n--------\nnormal   = 0.999235868454 seconds\nwal      = 0.776713132858 seconds\nmmap     = 0.653935909271 seconds\nlitetree = 0.714652061462 seconds\n```\n\n##### Windows\n\n```\nwriting:\n--------\nnormal   = 68.0931215734 seconds\nlitetree = 39.239919979 seconds\n\nreading:\n--------\nnormal   = 0.012673914421 seconds\nlitetree = 0.00631055510799 seconds\n```\n\nYou can make your own benchmark (after installing LiteTree) with this command:\n\n```\nmake benchmark\n```\n\n## Current Limits\n\nNumber of branches: 1024 branches  (can be increased)\n\nNumber of commits per branch: 2^64 = 18,446,744,073,709,551,615 commits\n\nConcurrent db connections to the same db: XXX readers\n\n## Some Limitations\n\nA database file created in one architecture cannot be used in another. This is a limitation of LMDB. We need to dump the database using `mdb_dump` and load it using `mdb_load`.\n\nThe db file cannot be opened by unmodified SQLite libraries.\n\nSavepoints are not yet supported.\n\n\n## How to use\n\nLiteTree can be used in many programming languages via existing SQLite wrappers.\n\n1. Update your app to open the database file using an URI containing the `branches` parameter, like this:\n\t```\n\t“file:data.db?branches=on”\n\t```\n\n2. Make your app use this new library instead of the pre-installed SQLite library:\n\n### On Linux\n\n This can be achieved in 4 ways:\n\n- Using the `LD_LIBRARY_PATH` environment variable:\n\t```\n\tLD_LIBRARY_PATH=/usr/local/lib/litetree ./myapp\n\t```\n\tThis can be used with all programming languages and wrappers.\n\n- Patching your wrapper or app to search for the library in the new path:\n\t```\n\tpatchelf --set-rpath /usr/local/lib/litetree lib_or_app\n\t```\n\n- Setting the `rpath` at the link time:\n\t```\n\tLIBPATH = /usr/local/lib/litetree\n\tgcc myapp.c -Wl,-rpath,$(LIBPATH) -L$(LIBPATH) -lsqlite3\n\t```\n\tYou can use this if your app is linking directly to the LiteTree library.\n\n- Replacing the pre-installed SQLite library on your system\n\n\tThis can also be used with many programming languages. But use it with care because the native library may have been compiled with different directives.\n\n\n### On Mac OSX\n\n This can be achieved in these ways:\n\n- Patching your wrapper or app to search for the library in the new path:\n\n\t```\n\tinstall_name_tool -change /old/path/to/libsqlite3.dylib /usr/local/lib/litetree/libsqlite3.dylib lib_or_app\n\t```\n\n\tYou can check the old path with this command:\n\n\t```\n\totool -L lib_or_app\n\t```\n\n\tThis method can be used with all programming languages and wrappers as long as they are not protected by the OS.\n\n\tIt it is protected then you will need to install a new copy of the wrapper, modify it and use it instead of the protected one.\n\n- Using the `DYLD_LIBRARY_PATH` environment variable:\n\n\t```\n\tDYLD_LIBRARY_PATH=/usr/local/lib/litetree ./myapp\n\t```\n\n\tThis can be used if the wrapper was linked to just the library name and does not contain any path.\n\n\tIf it does not work we can patch the wrapper to not contain any path:\n\n\t```\n\tinstall_name_tool -change /old/path/to/libsqlite3.dylib libsqlite3.dylib lib_or_app\n\t```\n\n\tBut if you are able to modify the wrapper with `install_name_tool` then the first method above may be better.\n\n- Linking to the LiteTree library:\n\n\t```\n\tgcc myapp.c -L/usr/local/lib/litetree -lsqlite3\n\t```\n\n\n### On Windows\n\nCopy the modified SQLite library to the system folder.\n\n- On 64 bit Windows:\n\n\tC:\\Windows\\System32 (if 64 bit DLL)\n\n\tC:\\Windows\\SysWOW64 (if 32 bit DLL)\n\n- On 32 bit Windows:\n\n\tC:\\Windows\\System32\n\n\n## Compiling and installing\n\n### On Linux and Mac OSX\n\nInstall [LMDB](https://github.com/lmdb/lmdb) if not already installed:\n\n```\ngit clone https://github.com/lmdb/lmdb\ncd lmdb/libraries/liblmdb\nmake\nsudo make install\n```\n\nThen install LiteTree:\n\n```\ngit clone https://github.com/aergoio/litetree\ncd litetree\nmake\nsudo make install\n```\n\n### On Windows\n\nYou can use these pre-compiled binaries: (can be outdated)\n\n- [32 bit DLLs](http://litesync.io/litetree/litetree-binaries-win-x86.zip)\n- [64 bit DLLs](http://litesync.io/litetree/litetree-binaries-win-x64.zip)\n\nOr follow these steps:\n\n1. Compile LMDB using [MinGW](https://github.com/kroggen/lmdb) or Visual Studio ([1](https://github.com/Ri0n/lmdb) or [2](https://github.com/htaox/lightningdb-win))\n\n2. Compile LiteTree using MinGW or Visual Studio\n\n3. Copy the libraries to the Windows System folder\n\n\n## Running the Tests\n\nThe tests are written in Python using the [pysqlite](https://github.com/ghaering/pysqlite) wrapper.\n\nOn **MacOSX** we cannot use a modified SQLite library with the pre-installed system python due to the System Integrity Protection so we need to install another copy of pysqlite and link it to the LiteTree library:\n\n```\ngit clone https://github.com/ghaering/pysqlite\ncd pysqlite\necho \"include_dirs=/usr/local/include\" \u003e\u003e setup.cfg\necho \"library_dirs=/usr/local/lib/litetree\" \u003e\u003e setup.cfg\npython setup.py build\nsudo python setup.py install\n```\n\nTo run the tests:\n\n```\nmake test\n```\n\n## License\n\nMIT\n\n## Creator\n\nDeveloped by Bernardo Ramos at\n\n\u003ca href=\"https://aergo.io\"\u003e\u003cimg src=\"http://litesync.io/litetree/aergo-logo-black-slim.svg\" width=\"30%\"\u003e\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faergoio%2Flitetree","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faergoio%2Flitetree","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faergoio%2Flitetree/lists"}