{"id":13395119,"url":"https://github.com/mirage/irmin","last_synced_at":"2025-05-13T19:12:55.518Z","repository":{"id":8172828,"uuid":"9595928","full_name":"mirage/irmin","owner":"mirage","description":"Irmin is a distributed database that follows the same design principles as Git","archived":false,"fork":false,"pushed_at":"2025-03-19T16:10:25.000Z","size":57581,"stargazers_count":1880,"open_issues_count":135,"forks_count":159,"subscribers_count":61,"default_branch":"main","last_synced_at":"2025-04-27T05:43:21.638Z","etag":null,"topics":["database","git","irmin","mirageos","ocaml","storage"],"latest_commit_sha":null,"homepage":"https://irmin.org","language":"OCaml","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mirage.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2013-04-22T09:52:58.000Z","updated_at":"2025-04-25T16:19:57.000Z","dependencies_parsed_at":"2024-03-11T16:43:04.268Z","dependency_job_id":"003ed4c2-4b20-46b7-b2b9-51362f33326d","html_url":"https://github.com/mirage/irmin","commit_stats":{"total_commits":4043,"total_committers":76,"mean_commits":53.19736842105263,"dds":0.5021023992085085,"last_synced_commit":"0ffa99014c57db9244e4d9f258aba2a6888d485c"},"previous_names":[],"tags_count":82,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirage%2Firmin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirage%2Firmin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirage%2Firmin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirage%2Firmin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mirage","download_url":"https://codeload.github.com/mirage/irmin/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251094580,"owners_count":21535324,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","git","irmin","mirageos","ocaml","storage"],"created_at":"2024-07-30T17:01:42.639Z","updated_at":"2025-04-27T05:43:26.859Z","avatar_url":"https://github.com/mirage.png","language":"OCaml","readme":"\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://irmin.org\"\u003e\n    \u003cimg src=\"./logo.svg\" alt=\"Irmin logo\"/\u003e\n  \u003c/a\u003e\n  \u003cbr /\u003e\n  \u003cstrong\u003eA Distributed Database Built on the Same Principles as Git\u003c/strong\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\u003cbr /\u003e\n\n[![OCaml-CI Build Status](https://img.shields.io/endpoint?url=https%3A%2F%2Fci.ocamllabs.io%2Fbadge%2Fmirage%2Firmin%2Fmain\u0026logo=ocaml\u0026style=flat-square)](https://ci.ocamllabs.io/github/mirage/irmin)\n[![codecov](https://codecov.io/gh/mirage/irmin/branch/main/graph/badge.svg?token=n4mWfgURqT)](https://codecov.io/gh/mirage/irmin)\n[![GitHub release (latest by date)](https://img.shields.io/github/v/release/mirage/irmin?style=flat-square\u0026color=09aa89)](https://github.com/mirage/irmin/releases/latest)\n[![docs](https://img.shields.io/badge/doc-online-blue.svg?style=flat-square)](https://mirage.github.io/irmin/)\n\n\u003c/div\u003e\n\n\u003chr /\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cem\u003e\n    Irmin is an OCaml library for building mergeable, branchable distributed\n    data stores.\n  \u003c/em\u003e\n\u003c/div\u003e\n\n\u003chr /\u003e\n\nIrmin is based on distributed version-control systems (DVCs),\nextensively used in software development to track data \nprovenance and show modifications in the source\ncode. Irmin applies DVC's principles to large-scale distributed data\nand includes similar functions to Git (clone, push, pull, branch,\nrebase). The Git workflow was initially designed for humans to manage\nchanges within source code. Irmin scales this to handle automatic\nprograms performing a very high number of operations per second, with\nfully-automated conflict handling.\n\nIrmin is highly customisable. Users can define their types to\nstore application-specific values. They can also define custom storage layers (in\nmemory, on disk, in a remote Redis database, in the browser,\netc.).  Finally, Irmin contains an event-driven API to define programmable dynamic behaviours and to\nprogram distributed dataflow pipelines.\n\nIrmin was created at the University of Cambridge in 2013 to be the\ndefault storage layer for [MirageOS][] applications (both to store and\norchestrate unikernel binaries and the data that these unikernels are\nusing). As such, Irmin is not, strictly speaking, a complete database\nengine. Instead, similarly to other MirageOS components, it is a\ncollection of libraries designed to solve different flavours of the\nchallenges raised by the [CAP Theorem][]. Each application\ncan select the right combination of libraries to solve its particular\ndistributed problem.\n\nIrmin is built on a core of well-defined, low-level data structures that \ndictate how data should be persisted and shared across nodes. \nIt defines algorithms for efficient synchronisation of those\ndistributed low-level constructs. It also builds a collection of\nhigher-level data structures that developers can use without knowing\nprecisely how Irmin works underneath. Some of these components even\nhave [formal semantics][], including [Conflict-free Replicated\nData-Types (CRDT)][]. Since it's a part of MirageOS, Irmin does not\nmake strong assumptions about the OS environment, which makes the system\nvery portable. It works well for in-memory databases\nand slower persistent serialisation, such as SSDs, hard drives, web\nbrowser local storage, or even the Git file format.\n\nIrmin is primarily developed and maintained by [Tarides][], with\ninvolvement by [contributors][] from various\norganisations. External maintainers and contributors are welcome.\n\n[MirageOS]: https://mirage.io\n[CAP Theorem]: http://en.wikipedia.org/wiki/CAP_theorem\n[formal semantics]: https://kcsrk.info/papers/banyan_aplas20.pdf\n[Conflict-free Replicated Data-Types (CRDT)]: https://arxiv.org/abs/2203.14518\n[Tarides]: https://tarides.com\n[contributors]: https://github.com/mirage/irmin/graphs/contributors\n\n\u003cdiv class=\"toc\"\u003e\n\n* [Features](#Features)\n* [Documentation](#Documentation)\n* [Installation](#Installation)\n  * [Prerequisites](#Prerequisites)\n  * [Development Version](#Development-Version)\n* [Usage](#Usage)\n  * [Example](#Example)\n  * [Command Line](#Commandline)\n* [Context](#Context)\n  * * [Irmin as a portable and efficient structured key-value store](#Irmin-as-a-portable-and-efficient-structured-keyvalue-store)\n    * [Irmin as a distributed store](#Irmin-as-a-distributed-store)\n    * [Irmin as a dataflow scheduler](#Irmin-as-a-dataflow-scheduler)\n* [Issues](#Issues)\n* [License](#License)\n* [Acknowledgements](#Acknowledgements)\n\n\u003c/div\u003e\n\n## Features\n\n- **Built-In Snapshotting** - backup and restore\n- **Storage Agnostic** - use Irmin on top of your own storage layer\n- **Custom Datatypes** - (de)serialisation for custom data types, derivable via\n  [`ppx_irmin`][ppx_irmin-readme]\n- **Highly Portable** - runs anywhere from Linux to web browsers and Xen unikernels\n- **Git Compatibility** - `irmin-git` uses an on-disk format that can be\n  inspected and modified using Git\n- **Dynamic Behavior** - allows the users to define custom merge functions,\n  use in-memory transactions (to keep track of reads as well as writes), and\n  to define event-driven workflows using a notification mechanism\n\n## Documentation\n\nAPI documentation can be found online at [https://mirage.github.io/irmin](https://mirage.github.io/irmin)\n\n## Installation\n\n### Prerequisites\n\nPlease ensure to install the minimum `opam` and `ocaml` versions. Find the latest\nversion and install instructions on [ocaml.org](https://ocaml.org/docs/install.html).\n\nTo install Irmin with the command-line tool and all Unix backends using `opam`:\n\n\u003c!-- $MDX skip --\u003e\n```bash\n  opam install irmin-cli\n```\n\nA minimal installation containing the reference in-memory backend can be\ninstalled by running:\n\n\u003c!-- $MDX skip --\u003e\n```bash\n  opam install irmin\n```\n\nThe following packages are available on `opam`:\n\n- `irmin` - the base package, plus an in-memory storage implementation\n- `irmin-chunk` - chunked storage\n- `irmin-cli` - a simple command-line tool\n- `irmin-fs` - filesystem-based storage using `bin_prot`\n- `irmin-git` - Git compatible storage\n- `irmin-graphql` - GraphQL server\n- `irmin-mirage` - MirageOS compatibility\n- `irmin-mirage-git` - Git compatible storage for MirageOS\n- `irmin-mirage-graphql` - MirageOS compatible GraphQL server\n- `irmin-pack` - compressed, on-disk, POSIX backend\n- `ppx_irmin` - PPX deriver for Irmin content types (see [README_PPX.md][ppx_irmin-readme])\n- `irmin-containers` - collection of simple, ready-to-use mergeable data structures\n\nTo install a specific package, simply run:\n\n\u003c!-- $MDX skip --\u003e\n```bash\n  opam install \u003cpackage-name\u003e\n```\n\n### Development Version\n\nTo install the development version of Irmin in your current `opam switch`, clone\nthis repository and `opam install` the packages inside:\n\n\u003c!-- $MDX skip --\u003e\n```bash\n  git clone https://github.com/mirage/irmin\n  cd irmin/\n  opam install .\n```\n\n## Usage\n\n### Example\n\nBelow is a simple example of setting a key and getting the value out of a\nGit-based, filesystem-backed store.\n\n\u003c!-- $MDX file=examples/readme.ml --\u003e\n```ocaml\nopen Lwt.Syntax\n\n(* Irmin store with string contents *)\nmodule Store = Irmin_git_unix.FS.KV (Irmin.Contents.String)\n\n(* Database configuration *)\nlet config = Irmin_git.config ~bare:true \"/tmp/irmin/test\"\n\n(* Commit author *)\nlet author = \"Example \u003cexample@example.com\u003e\"\n\n(* Commit information *)\nlet info fmt = Irmin_git_unix.info ~author fmt\n\nlet main =\n  (* Open the repo *)\n  let* repo = Store.Repo.v config in\n\n  (* Load the main branch *)\n  let* t = Store.main repo in\n\n  (* Set key \"foo/bar\" to \"testing 123\" *)\n  let* () =\n    Store.set_exn t ~info:(info \"Updating foo/bar\") [ \"foo\"; \"bar\" ]\n      \"testing 123\"\n  in\n\n  (* Get key \"foo/bar\" and print it to stdout *)\n  let+ x = Store.get t [ \"foo\"; \"bar\" ] in\n  Printf.printf \"foo/bar =\u003e '%s'\\n\" x\n\n(* Run the program *)\nlet () = Lwt_main.run main\n```\n\nThe example is contained in [examples/readme.ml](./examples/readme.ml) It can\nbe compiled and executed with Dune:\n\n\u003c!-- $MDX skip --\u003e\n```bash\n$ dune build examples/readme.exe\n$ dune exec examples/readme.exe\nfoo/bar =\u003e 'testing 123'\n```\n\nThe [examples](./examples/) directory also contains more advanced examples,\nwhich can be executed in the same way.\n\n### Command Line\n\nThe same thing can also be accomplished using `irmin`, the command-line\napplication installed with `irmin-cli`, by running:\n\n```bash\n$ echo \"root: .\" \u003e irmin.yml\n$ irmin init\n$ irmin set foo/bar \"testing 123\"\n$ irmin get foo/bar\ntesting 123\n```\n\n`irmin.yml` allows for `irmin` flags to be set on a per-directory basis. You\ncan also set flags globally using `$HOME/.irmin/config.yml`. Run\n`irmin help irmin.yml` for further details.\n\nAlso see `irmin --help` for a list of all commands and either\n`irmin \u003ccommand\u003e --help` or `irmin help \u003ccommand\u003e` for more help with a\nspecific command.\n\n## Context\n\nIrmin's initial design is directly inspired from\n[XenStore](https://dl.acm.org/doi/10.1145/1631687.1596581), with:\n\n- the need for efficient optimistic concurrency control features to\n  let thousands of virtual machine concurrently access and\n  modify a central configuration database (the Xen stack uses XenStore\n  as an RPC mechanism to setup VM configuration on boot). Very early\n  on, the initial focus was to specify and handle [potential\n  conflicts](https://hal.inria.fr/hal-01099136v1/document) when the\n  optimistic assumptions do not usually work so well.\n- the need for a convenient way to debug and audit possible issues\n  that might happen in that system. Our [initial\n  experiments](https://mirage.io/blog/introducing-irmin-in-xenstore)\n  showed that it was possible to design a reliable system using Git as\n  backend to persist configuation data reliably (to safely restart\n  after a crash), while making system debugging easy and go really\n  fast, thanks to efficient merging strategy.\n\nIn 2014, the first release of [Irmin was announced](https://mirage.io/blog/introducing-irmin) \nas part of the MirageOS 2.0 release. Since\nthen, several projects started using and improving Irmin. These can\nroughly be split into three categories: \n1. Use Irmin as a portable,\nstructured key-value store (with expressive, mergeable types)\n2. Use Irmin as distributed database (with a customisable consistency\nsemantics)\n3. Use Irmin as an event-driven dataflow engine.\n\n\n#### Irmin as a portable and efficient structured key-value store\n\n- [XenStored](https://github.com/xen-project/xen/tree/master/tools/ocaml/xenstored)\n  is an information storage space shared between all the Xen virtual\n  machines running in the same host. Each virtual machine gets its\n  own path in the store. When values are changed, the\n  appropriate drivers are notified. The initial OCaml implementation\n  was later [extended to use Irmin](https://github.com/mirage/ocaml-xenstore-server).\n  More details\n  [here](https://mirage.io/blog/introducing-irmin-in-xenstore).\n- [Jitsu](https://github.com/mirage/jitsu) is an experimental\n  orchestrator for unikernels. It uses Irmin to store the unikernel\n  configuration (and manage dynamic DNS entries). See more details\n  [here](https://www.usenix.org/system/files/conference/nsdi15/nsdi15-paper-madhavapeddy.pdf).\n- [Cuekeeper](https://github.com/talex5/cuekeeper) is a web-based GTD\n  (a fancy TODO list) that runs entirely in the browser. It uses Irmin\n  to store data locally with support for structured\n  concurrent editing and snapshot export and import. More details\n  [here](https://roscidus.com/blog/blog/2015/04/28/cuekeeper-gitting-things-done-in-the-browser/).\n- [Canopy](https://github.com/Engil/Canopy) and\n  [Unipi](https://github.com/roburio/unipi) both use Irmin to serve\n  static websites pulled from Git repositories and deployed as\n  unikernels.\n- [Caldav](https://github.com/roburio/caldav) uses Irmin to store\n  calendar entries and back them into a Git repository. More\n  information [here](https://robur.io/Our%20Work/Projects).\n- [Datakit](https://github.com/moby/datakit) was developed at Docker\n  and provided a 9p interface to the Irmin API. It was used to manage\n  the configuration of Docker for Desktop with merge policies on\n  upgrade, full auditing, and snapshot/rollback capabilites.\n- [Tezos](https://gitlab.com/tezos/tezos/) started using Irmin in 2017\n  to store the\n  ledger state. The first prototype used `irmin-git` before switching to\n  `irmin-lmdb` and `irmin-leveldb` (and now `irmin-pack`). More details\n  [here](https://tarides.com/blog/2019-11-21-irmin-v2#tezos-and-irmin-pack).\n\n#### Irmin as a distributed store\n\n- An [IMAP](ttps://github.com/gregtatcam/imaplet-lwt) server using\n  Irmin to store emails. More details\n  [here](https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-918.pdf). The\n  goal of that project was both to use Irmin to store emails (so using\n  Irmin as a local key-value store) but also to experiment with\n  replacing the IMAP on-wire protocol by an explicit Git push/pull\n  mechanism.\n- [`irmin-ARP`](https://github.com/yomimono/irmin-arp) uses Irmin to\n  store and audit ARP configuration. It's using Irmin as a local\n  key-value store for very low-level information (which are normally\n  stored very deep in the kernel layers), but the main goal was really\n  to replace the broadcasting on-wire protocol by point-to-point\n  pull/push synchronisation primitives, with a full audit log of ARP\n  operations over a network. More details\n  [here](http://somerandomidiot.com/blog/2015/04/24/what-a-distributed-version-controlled-ARP-cache-gets-you/).\n- [Banyan](https://github.com/prismlab/irmin-scylla) uses Irmin to\n  implement a distributed cache over a geo-replicated cluster. It's\n  using [Cassandra](https://cassandra.apache.org/_/index.html) as a\n  storage backend. More information\n  [here](https://kcsrk.info/papers/banyan_aplas20.pdf).\n- [`irmin-fdb`](https://github.com/andreas/irmin-fdb) implements an\n  Irmin store backed by\n  [FoundationDB](https://www.foundationdb.org/). More details\n  [here](https://www.youtube.com/watch?v=NArvw-9axeg\u0026ab_channel=TheLinuxFoundation).\n\n#### Irmin as a dataflow scheduler\n\n- [Datakit CI](https://github.com/moby/datakit/tree/master/ci) is a\n  continuous integration service that monitors GitHub projects and\n  tests each branch, tag, and pull request. It displays the test\n  results as status indicators in the GitHub UI. It keeps all of its\n  state and logs in DataKit rather than a traditional relational\n  database, allowing review with the usual Git tools. The core of the\n  project is a scheduler that manages dataflow pipelines across Git\n  repositories. For a few years, it was used as Docker for Desktop's CI system test\n  on bare-metal and virtual machines, as well as\n  all the new opam package submissions to `ocaml/opam-repository`. More\n  details\n  [here](https://www.docker.com/blog/docker-unikernels-open-source/).\n- [Causal RPC](https://github.com/CraigFe/causal-rpc) implements an\n  RPC framework using Irmin as a network substrate. More details\n  [here](https://www.craigfe.io/causalrpc.pdf).\n- [CISO](https://github.com/samoht/ciso) is an experimental\n  (distributed) Continuous Integration engine for opam. It was\n  designed as a replacement of Datakit-CI and finally turned into\n  [OCurrent](https://github.com/ocurrent/ocurrent).\n\n## Issues\n\nFeel free to report any issues using the [GitHub bugtracker](https://github.com/mirage/irmin/issues).\n\n## License\n\nSee the [LICENSE file](./LICENSE.md).\n\n## Acknowledgements\n\nDevelopment of Irmin was supported in part by the EU FP7 User-Centric Networking\nproject, Grant No. 611001.\n\n[ppx_irmin-readme]: ./README_PPX.md\n","funding_links":[],"categories":["OCaml","By Language","git","数据库管理系统","ocaml","Databases","\u003ca name=\"OCaml\"\u003e\u003c/a\u003eOCaml"],"sub_categories":["Other","网络服务_其他"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmirage%2Firmin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmirage%2Firmin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmirage%2Firmin/lists"}