{"id":13561964,"url":"https://github.com/aeternity/mnesia_rocksdb","last_synced_at":"2025-04-04T10:04:09.885Z","repository":{"id":29179291,"uuid":"120424479","full_name":"aeternity/mnesia_rocksdb","owner":"aeternity","description":"A RocksDB backend plugin for mnesia, based on mnesia_eleveldb","archived":false,"fork":false,"pushed_at":"2024-12-12T07:01:37.000Z","size":2535,"stargazers_count":153,"open_issues_count":6,"forks_count":43,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-03-28T09:04:49.589Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aeternity.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-06T08:29:44.000Z","updated_at":"2025-03-24T15:37:34.000Z","dependencies_parsed_at":"2023-10-13T06:21:08.144Z","dependency_job_id":"19d205ef-25d7-45e0-a6a3-5a4d535ee7d4","html_url":"https://github.com/aeternity/mnesia_rocksdb","commit_stats":{"total_commits":71,"total_committers":15,"mean_commits":4.733333333333333,"dds":"0.43661971830985913","last_synced_commit":"9cea41cb9a6dbc1cef016e92f7a824d369d0bb28"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aeternity%2Fmnesia_rocksdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aeternity%2Fmnesia_rocksdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aeternity%2Fmnesia_rocksdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aeternity%2Fmnesia_rocksdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aeternity","download_url":"https://codeload.github.com/aeternity/mnesia_rocksdb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247156556,"owners_count":20893199,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:01:03.076Z","updated_at":"2025-04-04T10:04:09.860Z","avatar_url":"https://github.com/aeternity.png","language":"Erlang","funding_links":[],"categories":["Erlang"],"sub_categories":[],"readme":"\n\n# Mnesia Rocksdb - Rocksdb backend plugin for Mnesia #\n\nCopyright (c) 2013-21 Klarna AB\n\n__Authors:__ Ulf Wiger ([`ulf@wiger.net`](mailto:ulf@wiger.net)).\n\nThe Mnesia DBMS, part of Erlang/OTP, supports 'backend plugins', making\nit possible to utilize more capable key-value stores than the `dets`\nmodule (limited to 2 GB per table). Unfortunately, this support is\nundocumented. Below, some informal documentation for the plugin system\nis provided.\n\n\n### \u003ca name=\"Table_of_Contents\"\u003eTable of Contents\u003c/a\u003e ###\n\n\n1. [Usage](https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/README.md#Usage)\n1. [Prerequisites](https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/README.md#Prerequisites)\n1. [Getting started](https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/README.md#Getting_started)\n1. [Special features](https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/README.md#Special_features)\n1. [Customization](https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/README.md#Customization)\n1. [Handling of errors in write operations](https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/README.md#Handling_of_errors_in_write_operations)\n1. [Caveats](https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/README.md#Caveats)\n\n1. [Mnesia backend plugins](https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/README.md#Mnesia_backend_plugins)\n1. [Background](https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/README.md#Background)\n1. [Design](https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/README.md#Design)\n\n1. [Mnesia index plugins](https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/README.md#Mnesia_index_plugins)\n\n1. [Rocksdb](https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/README.md#Rocksdb)\n\n\n\n### \u003ca name=\"Usage\"\u003eUsage\u003c/a\u003e ###\n\n\n#### \u003ca name=\"Prerequisites\"\u003ePrerequisites\u003c/a\u003e ####\n\n* rocksdb (included as dependency)\n\n* sext (included as dependency)\n\n* Erlang/OTP 21.0 or newer (https://github.com/erlang/otp)\n\n\n\n#### \u003ca name=\"Getting_started\"\u003eGetting started\u003c/a\u003e ####\n\nCall `mnesia_rocksdb:register()` immediately after\nstarting mnesia.\n\nPut `{rocksdb_copies, [node()]}` into the table definitions of\ntables you want to be in RocksDB.\n\n\n#### \u003ca name=\"Special_features\"\u003eSpecial features\u003c/a\u003e ####\n\nRocksDB tables support efficient selects on _prefix keys_.\n\nThe backend uses the `sext` module (see\n[`https://github.com/uwiger/sext`](https://github.com/uwiger/sext)) for mapping between Erlang terms and the\nbinary data stored in the tables. This provides two useful properties:\n\n* The records are stored in the Erlang term order of their keys.\n\n* A prefix of a composite key is ordered just before any key for which\n  it is a prefix. For example, `{x, '_'}` is a prefix for keys `{x, a}`,`{x, b}` and so on.\n\n\nThis means that a prefix key identifies the start of the sequence of\nentries whose keys match the prefix. The backend uses this to optimize\nselects on prefix keys.\n\n### Customization\n\nRocksDB supports a number of customization options. These can be specified\nby providing a `{Key, Value}` list named `rocksdb_opts` under `user_properties`,\nfor example:\n\n```erlang\nmnesia:create_table(foo, [{rocksdb_copies, [node()]},\n                          ...\n                          {user_properties,\n                              [{rocksdb_opts, [{max_open_files, 1024}]}]\n                          }])\n```\n\nConsult the [RocksDB documentation](https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning)\nfor information on configuration parameters. Also see the section below on handling write errors.\n\nThe default configuration for tables in `mnesia_rocksdb` is:\n\n```erlang\ndefault_open_opts() -\u003e\n    [ {create_if_missing, true}\n      , {cache_size,\n         list_to_integer(get_env_default(\"ROCKSDB_CACHE_SIZE\", \"32212254\"))}\n      , {block_size, 1024}\n      , {max_open_files, 100}\n      , {write_buffer_size,\n         list_to_integer(get_env_default(\n                           \"ROCKSDB_WRITE_BUFFER_SIZE\", \"4194304\"))}\n      , {compression,\n         list_to_atom(get_env_default(\"ROCKSDB_COMPRESSION\", \"true\"))}\n      , {use_bloomfilter, true}\n    ].\n```\n\nIt is also possible, for larger databases, to produce a tuning parameter file.\nThis is experimental, and mostly copied from `mnesia_leveldb`. Consult the\nsource code in `mnesia_rocksdb_tuning.erl` and `mnesia_rocksdb_params.erl`.\nContributions are welcome.\n\n\n#### \u003ca name=\"Caveats\"\u003eCaveats\u003c/a\u003e ####\n\nAvoid placing `bag` tables in RocksDB. Although they work, each write\nrequires additional reads, causing substantial runtime overheads. There\nare better ways to represent and process bag data (see above about\n_prefix keys_).\n\nThe `mnesia:table_info(T, size)` call always returns zero for RocksDB\ntables. RocksDB itself does not track the number of elements in a table, and\nalthough it is possible to make the `mnesia_rocksdb` backend maintain a size\ncounter, it incurs a high runtime overhead for writes and deletes since it\nforces them to first do a read to check the existence of the key. If you\ndepend on having an up to date size count at all times, you need to maintain\nit yourself. If you only need the size occasionally, you may traverse the\ntable to count the elements.\n\n\n### \u003ca name=\"Mnesia_backend_plugins\"\u003eMnesia backend plugins\u003c/a\u003e ###\n\n\n#### \u003ca name=\"Background\"\u003eBackground\u003c/a\u003e ####\n\nMnesia was initially designed to be a RAM-only DBMS, and Erlang's\n`ets` tables were developed for this purpose. In order to support\npersistence, e.g. for configuration data, a disk-based version of `ets`\n(called `dets`) was created. The `dets` API mimicks the `ets` API,\nand `dets` is quite convenient and fast for (nowadays) small datasets.\nHowever, using a 32-bit bucket system, it is limited to 2GB of data.\nIt also doesn't support ordered sets. When used in Mnesia, dets-based\ntables are called `disc_only_copies`.\n\nTo circumvent these limitations, another table type, called `disc_copies`\nwas added. This is a combination of `ets` and `disk_log`, where Mnesia\nperiodically snapshots the `ets` data to a log file on disk, and meanwhile\nmaintains a log of updates, which can be applied at startup. These tables\nare quite performant (especially on read access), but all data is kept in\nRAM, which can become a serious limitation.\n\nA backend plugin system was proposed by Ulf Wiger in 2016, and further\ndeveloped with Klarna's support, to finally become included in OTP 19.\nKlarna uses a LevelDb backend, but Aeternity, in 2017, instead chose\nto implement a Rocksdb backend plugin.\n\n\n### \u003ca name=\"Design\"\u003eDesign\u003c/a\u003e ###\n\nAs backend plugins were added on a long-since legacy-stable Mnesia,\nthey had to conform to the existing code structure. For this reason,\nthe plugin callbacks hook into the already present low-level access\nAPI in the `mnesia_lib` module. As a consequence, backend plugins have\nthe same access semantics and granularity as `ets` and `dets`. This\nisn't much of a disadvantage for key-value stores like LevelDb and RocksDB,\nbut a more serious issue is that the update part of this API is called\non _after_ the point of no return. That is, Mnesia does not expect\nthese updates to fail, and has no recourse if they do. As an aside,\nthis could also happen if a `disc_only_copies` table exceeds the 2 GB\nlimit (mnesia will not check it, and `dets` will not complain, but simply\ndrop the update.)\n\n\n### \u003ca name=\"Mnesia_index_plugins\"\u003eMnesia index plugins\u003c/a\u003e ###\n\nWhen adding support for backend plugins, index plugins were also added. Unfortunately, they remain undocumented.\n\nAn index plugin can be added in one of two ways:\n\n1. When creating a schema, provide `{index_plugins, [{Name, Module, Function}]}` options.\n\n1. Call the function `mnesia_schema:add_index_plugin(Name, Module, Function)`\n\n\n`Name` must be an atom wrapped as a 1-tuple, e.g. `{words}`.\n\nThe plugin callback is called as `Module:Function(Table, Pos, Obj)`, where `Pos=={words}` in\nour example. It returns a list of index terms.\n\n\u003cstrong\u003eExample\u003c/strong\u003e\n\nGiven the following index plugin implementation:\n\n```erlang\n-module(words).\n-export([words_f/3]).\n\nwords_f(_,_,Obj) when is_tuple(Obj) -\u003e\n    words_(tuple_to_list(Obj)).\n\nwords_(Str) when is_binary(Str) -\u003e\n    string:lexemes(Str, [$\\s, $\\n, [$\\r,$\\n]]);\nwords_(L) when is_list(L) -\u003e\n    lists:flatmap(fun words_/1, L);\nwords_(_) -\u003e\n    [].\n```\n\nWe can register the plugin and use it in table definitions:\n\n```erlang\nEshell V12.1.3  (abort with ^G)\n1\u003e mnesia:start().\nok\n2\u003e mnesia_schema:add_index_plugin({words}, words, words_f).\n{atomic,ok}\n3\u003e mnesia:create_table(i, [{index, [{words}]}]).\n{atomic,ok}\n```\n\nNote that in this case, we had neither a backend plugin, nor even a persistent schema.\nIndex plugins can be used with all table types. The registered indexing function (arity 3) must exist\nas an exported function along the node's code path.\n\nTo see what happens when we insert an object, we can turn on call trace.\n\n```erlang\n4\u003e dbg:tracer().\n{ok,\u003c0.108.0\u003e}\n5\u003e dbg:tp(words, x).\n{ok,[{matched,nonode@nohost,3},{saved,x}]}\n6\u003e dbg:p(all,[c]).\n{ok,[{matched,nonode@nohost,60}]}\n7\u003e mnesia:dirty_write({i,\u003c\u003c\"one two\"\u003e\u003e, [\u003c\u003c\"three\"\u003e\u003e, \u003c\u003c\"four\"\u003e\u003e]}).\n(\u003c0.84.0\u003e) call words:words_f(i,{words},{i,\u003c\u003c\"one two\"\u003e\u003e,[\u003c\u003c\"three\"\u003e\u003e,\u003c\u003c\"four\"\u003e\u003e]})\n(\u003c0.84.0\u003e) returned from words:words_f/3 -\u003e [\u003c\u003c\"one\"\u003e\u003e,\u003c\u003c\"two\"\u003e\u003e,\u003c\u003c\"three\"\u003e\u003e,\n                                             \u003c\u003c\"four\"\u003e\u003e]\n(\u003c0.84.0\u003e) call words:words_f(i,{words},{i,\u003c\u003c\"one two\"\u003e\u003e,[\u003c\u003c\"three\"\u003e\u003e,\u003c\u003c\"four\"\u003e\u003e]})\n(\u003c0.84.0\u003e) returned from words:words_f/3 -\u003e [\u003c\u003c\"one\"\u003e\u003e,\u003c\u003c\"two\"\u003e\u003e,\u003c\u003c\"three\"\u003e\u003e,\n                                             \u003c\u003c\"four\"\u003e\u003e]\nok\n8\u003e dbg:ctp('_'), dbg:stop().\nok\n9\u003e mnesia:dirty_index_read(i, \u003c\u003c\"one\"\u003e\u003e, {words}).\n[{i,\u003c\u003c\"one two\"\u003e\u003e,[\u003c\u003c\"three\"\u003e\u003e,\u003c\u003c\"four\"\u003e\u003e]}]\n```\n\n(The fact that the indexing function is called twice, seems like a performance bug.)\n\nWe can observe that the indexing callback is able to operate on the whole object.\nIt needs to be side-effect free and efficient, since it will be called at least once for each update\n(if an old object exists in the table, the indexing function will be called on it too, before it is\nreplaced by the new object.)\n\n\n### \u003ca name=\"Rocksdb\"\u003eRocksdb\u003c/a\u003e ###\n\n\n### \u003ca name=\"Usage\"\u003eUsage\u003c/a\u003e ###\n\n\n\n## Modules ##\n\n\n\u003ctable width=\"100%\" border=\"0\" summary=\"list of modules\"\u003e\n\u003ctr\u003e\u003ctd\u003e\u003ca href=\"https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/mnesia_rocksdb.md\" class=\"module\"\u003emnesia_rocksdb\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\u003ca href=\"https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/mnesia_rocksdb_admin.md\" class=\"module\"\u003emnesia_rocksdb_admin\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\u003ca href=\"https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/mnesia_rocksdb_app.md\" class=\"module\"\u003emnesia_rocksdb_app\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\u003ca href=\"https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/mnesia_rocksdb_lib.md\" class=\"module\"\u003emnesia_rocksdb_lib\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\u003ca href=\"https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/mnesia_rocksdb_params.md\" class=\"module\"\u003emnesia_rocksdb_params\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\u003ca href=\"https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/mnesia_rocksdb_sup.md\" class=\"module\"\u003emnesia_rocksdb_sup\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\u003ca href=\"https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/mnesia_rocksdb_tuning.md\" class=\"module\"\u003emnesia_rocksdb_tuning\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\u003ca href=\"https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/mrdb.md\" class=\"module\"\u003emrdb\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\u003ca href=\"https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/mrdb_index.md\" class=\"module\"\u003emrdb_index\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\u003ca href=\"https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/mrdb_mutex.md\" class=\"module\"\u003emrdb_mutex\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\u003ca href=\"https://github.com/aeternity/mnesia_rocksdb/blob/g3553-refactor-plugin-migration-tmp-220318/doc/mrdb_select.md\" class=\"module\"\u003emrdb_select\u003c/a\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faeternity%2Fmnesia_rocksdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faeternity%2Fmnesia_rocksdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faeternity%2Fmnesia_rocksdb/lists"}