{"id":15056391,"url":"https://github.com/rabbitmq/khepri","last_synced_at":"2025-05-14T14:08:10.947Z","repository":{"id":37799665,"uuid":"347983856","full_name":"rabbitmq/khepri","owner":"rabbitmq","description":"Khepri is a tree-like replicated on-disk database library for Erlang and Elixir.","archived":false,"fork":false,"pushed_at":"2025-05-07T10:12:40.000Z","size":4280,"stargazers_count":401,"open_issues_count":8,"forks_count":21,"subscribers_count":25,"default_branch":"main","last_synced_at":"2025-05-14T02:47:17.066Z","etag":null,"topics":["database","distributed","distributed-database","elixir","elixir-library","erlang","erlang-library","raft"],"latest_commit_sha":null,"homepage":"https://rabbitmq.github.io/khepri/","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rabbitmq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-Apache-2.0","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-03-15T13:38:36.000Z","updated_at":"2025-05-02T21:24:20.000Z","dependencies_parsed_at":"2023-02-16T11:15:42.886Z","dependency_job_id":"c4cb9b20-37c4-4be6-9e69-41d2e79cecdf","html_url":"https://github.com/rabbitmq/khepri","commit_stats":{"total_commits":563,"total_committers":9,"mean_commits":62.55555555555556,"dds":"0.29840142095914746","last_synced_commit":"5186e9bf0038f2a9377ae3202b52d54bbf25e0c7"},"previous_names":[],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rabbitmq%2Fkhepri","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rabbitmq%2Fkhepri/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rabbitmq%2Fkhepri/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rabbitmq%2Fkhepri/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rabbitmq","download_url":"https://codeload.github.com/rabbitmq/khepri/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254159689,"owners_count":22024564,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","distributed","distributed-database","elixir","elixir-library","erlang","erlang-library","raft"],"created_at":"2024-09-24T21:50:38.400Z","updated_at":"2025-05-14T14:08:10.938Z","avatar_url":"https://github.com/rabbitmq.png","language":"Erlang","readme":"# The Khepri database library\n\n[![Hex.pm](https://img.shields.io/hexpm/v/khepri)](https://hex.pm/packages/khepri/)\n[![Test](https://github.com/rabbitmq/khepri/actions/workflows/test-and-release.yaml/badge.svg)](https://github.com/rabbitmq/khepri/actions/workflows/test-and-release.yaml)\n[![Codecov](https://codecov.io/gh/rabbitmq/khepri/branch/main/graph/badge.svg?token=R0OGKZ2RK2)](https://codecov.io/gh/rabbitmq/khepri)\n\nKhepri is a tree-like replicated on-disk database library for Erlang and\nElixir, built on top of the [Raft consensus algorithm](https://raft.github.io/).\n\n\u003cimg align=\"right\" width=\"100\" src=\"/doc/khepri-logo.svg\"\u003e\n\n## The basics\n\nData are stored in a **tree structure**. Each node in the tree is referenced by\nits path from the root node. A path is a list of Erlang atoms and/or binaries.\nFor ease of use, Unix-like path strings are accepted as well.\n\nFor **consistency and replication** and to manage data on disk, Khepri relies\non [Ra](https://github.com/rabbitmq/ra), an Erlang implementation of the [Raft\nconsensus algorithm](https://raft.github.io/). In Ra parlance, Khepri is a\nstate machine in a Ra cluster.\n\n## Project maturity\n\nKhepri is still under active development and should be considered *Beta* at\nthis stage.\n\n## Fundamental Assumptions\n\nAs a Raft-based system, Khepri **requires durable storage**, including\nbetween node restarts, and stable node (replica) identities.\n\nLikewise, as a Raft-based system, Khepri requires a majority of nodes to be\nonline at all times. When this is not the case, the cluster will lose its availability,\nspecifically for writes.\n\n## Known limitations\n\nKhepri currently hosts the entire data set **in memory as well as on disk**, so\nthere is a realistic limit to how large a data set can be stored in it.\n\nFor this reason and others, storing **blobs of files** in Khepri is not\nrecommended. For that, use an external blob store.\n\n## Documentation\n\n* A short tutorial in the [Getting started](#getting-started) section below\n* [Documentation and API reference](https://rabbitmq.github.io/khepri/)\n\n## Getting started\n\n### Add as a dependency\n\nAdd Khepri as a dependency of your project:\n\nUsing Rebar:\n\n```erlang\n%% In rebar.config\n{deps, [{khepri, \"0.17.1\"}]}.\n```\n\nUsing Erlang.mk:\n\n```make\n# In your Makefile\nDEPS += khepri\ndep_khepri = hex 0.17.1\n```\n\nUsing Mix:\n\n```elixir\n# In mix.exs\ndefp deps do\n  [\n    {:khepri, \"0.17.1\"}\n  ]\nend\n```\n\n### Start default Khepri store\n\nTo start the default store, use `khepri:start/0`:\n\n```erlang\nkhepri:start().\n```\n\nThe default Khepri store uses the default Ra system. Data is stored in the\nconfigured default Ra system data directory, which is `khepri#$NODENAME` in\nthe current working directory.\n\nIt is fine to get started and play with Khepri. However, it is recommended to\nconfigure your own Ra system and Ra cluster to select the directory where data\nis stored and to be able to have multiple Khepri database instances running on\nthe same Erlang node.\n\n### Insert data\n\nHere's how to **insert** a piece of data, say, an email address of Alice:\n\n```erlang\n%% Using a native path:\nok = khepri:put([emails, \u003c\u003c\"alice\"\u003e\u003e], \"alice@example.org\").\n\n%% Using a Unix-like path string:\nok = khepri:put(\"/:emails/alice\", \"alice@example.org\").\n```\n\n### Read data back\n\nTo get Alice's email address back, **query** the same path:\n\n```erlang\n{ok, \"alice@example.org\"} = khepri:get(\"/:emails/alice\").\n```\n\n### Delete data\n\nTo **delete** Alice's email address:\n\n```erlang\nok = khepri:delete(\"/:emails/alice\").\n```\n\nThe `emails` parent node was automatically created when the `alice` node was\ninserted earlier. It has no data attached to it. However, after the `alice`\nnode is deleted, the `emails` node will stay around. It is possible to tell\nKhepri to automatically remove `emails` as soon as its last child node is\ndeleted. Khepri supports many more conditions by the way.\n\n### Transactional Operations\n\nIt is also possible to perform **transactional queries and updates** using\nanonymous functions, similar to Mnesia:\n\n```erlang\n%% This transaction checks the quantity of wood left and returns `true` or\n%% `false` if we need to process a new order.\nkhepri:transaction(\n    fun() -\u003e\n        case khepri_tx:get([stock, wood]) of\n            {ok, Quantity} when Quantity \u003e= 100 -\u003e\n                %% There is enough wood left.\n                false;\n            _ -\u003e\n                %% There is less than 100 pieces of wood, or there is none\n                %% at all (the node does not exist in Khepri). We need to\n                %% request a new order.\n                ok = khepri_tx:put([order, wood], 1000),\n                true\n        end\n    end).\n```\n\nIn this example, the transaction returns a boolean indicating if orders are\nready to be processed. It does not send a message to a process or write\nsomething on disk for instance.\n\nBecause of the nature of the Raft consensus algorithm, transactions are not\nallowed to have side effects or take non-deterministic inputs such as the node\nname or the current date \u0026 time.\n\n### Triggers\n\nKhepri supports *stored procedures* and *triggers*. They allow to store code in\nthe database itself and automatically execute it after some event occurs.\n\n1.  Store an anonymous function in the tree:\n\n    ```erlang\n    StoredProcPath = [path, to, stored_procedure],\n\n    Fun = fun(Props) -\u003e\n              #{path := Path,\n                on_action := Action} = Props\n          end,\n\n    khepri:put(StoreId, StoredProcPath, Fun).\n    ```\n\n2.  Register a trigger using an event filter:\n\n    ```erlang\n    %% A path is automatically considered a tree event filter.\n    EventFilter = [stock, wood, \u003c\u003c\"oak\"\u003e\u003e],\n\n    ok = khepri:register_trigger(\n           StoreId,\n           TriggerId,\n           EventFilter,\n           StoredProcPath).\n    ```\n\nIn the example above, as soon as the `[stock, wood, \u003c\u003c\"oak\"\u003e\u003e]` node is\ncreated, updated or deleted, the anonymous function will be executed.\n\nThe function is executed at least once on the Ra leader's Erlang node. It may\nbe executed multiple times if the leader changes and thus should be idempotent.\n\nUnlike transaction functions, stored procedures may have whatever side effects\nthey want.\n\n## Migrating from Mnesia\n\nTo help you migrate an existing Mnesia database, you can use [the\n`khepri_mnesia_migration`\napplication](https://github.com/rabbitmq/khepri_mnesia_migration/). It can take\ncare of:\n* synchronizing the cluster membership and\n* copying Mnesia tables to a Khepri store.\n\n## How to build\n\n### Build\n\n```\nrebar3 compile\n```\n\n### Build documentation\n\n```\nrebar3 edoc\n```\n\n### Test\n\n```\nrebar3 xref\nrebar3 eunit\nrebar3 proper\nrebar3 ct --sname ct\nrebar3 as test dialyzer\n```\n\n## Copyright and License\n\n© 2021-2025 Broadcom. All Rights Reserved. The term \"Broadcom\" refers to\nBroadcom Inc. and/or its subsidiaries.\n\nThis work is dual-licensed under the Apache License 2.0 and the Mozilla Public\nLicense 2.0. Users can choose any of these licenses according to their needs.\n\nThe logo (`doc/khepri-logo.svg`) and the favicon (`doc/khepri-favicon.svg`) are\nbased on the following two resources:\n* https://www.svgrepo.com/svg/55105/rabbit (license: CC0)\n* https://www.svgrepo.com/svg/336625/database-point (license: MIT)\n\nSPDX-License-Identifier: Apache-2.0 OR MPL-2.0\n","funding_links":[],"categories":["Erlang"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frabbitmq%2Fkhepri","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frabbitmq%2Fkhepri","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frabbitmq%2Fkhepri/lists"}