{"id":17480038,"url":"https://github.com/dacr/zio-lmdb","last_synced_at":"2025-04-15T17:46:06.534Z","repository":{"id":65508469,"uuid":"521968358","full_name":"dacr/zio-lmdb","owner":"dacr","description":"Lightning Memory Database (LMDB) for scala ZIO ","archived":false,"fork":false,"pushed_at":"2025-04-05T07:40:12.000Z","size":150,"stargazers_count":39,"open_issues_count":2,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-05T08:27:31.761Z","etag":null,"topics":["database","lmdb","lmdb-scala","lmdbjava","scala","scala3","zio","zio-lmdb"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dacr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-06T14:26:41.000Z","updated_at":"2025-04-05T07:40:16.000Z","dependencies_parsed_at":"2024-02-03T14:31:12.609Z","dependency_job_id":"1d83be5c-af9e-4781-871c-a228253f0b48","html_url":"https://github.com/dacr/zio-lmdb","commit_stats":null,"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dacr%2Fzio-lmdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dacr%2Fzio-lmdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dacr%2Fzio-lmdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dacr%2Fzio-lmdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dacr","download_url":"https://codeload.github.com/dacr/zio-lmdb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249123814,"owners_count":21216569,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","lmdb","lmdb-scala","lmdbjava","scala","scala3","zio","zio-lmdb"],"created_at":"2024-10-18T21:42:29.985Z","updated_at":"2025-04-15T17:46:06.521Z","avatar_url":"https://github.com/dacr.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lightning Memory Database (LMDB) for ZIO\n[![][ZIOLMDBManagerImg]][ZIOLMDBManagerLnk] [![scaladoc][ScalaDocImg]][ScalaDoc]\n\nWhy ZIO-lmdb ? Because I wanted a very simple **embedded** (in the same process) ACID database for small\napplications while keeping deployment, maintenance, upgrades as simple as possible.\n\nZIO-lmdb is based on the powerful [lmdb-java][JLMDB] library and bring a higher level API in order\nto enhance the developer experience.\n\nSo ZIO-lmdb is an embedded key/value database, with an easy to use opinionated API, choices have been made\nto make the developer experience as simple as possible :\n- JSON based default storage using zio-json,\n  - *Custom serialization is supported*\n- Safe update by using a lambda which will be called with the previous value if it exists and returns the new value,\n- Identifiers are managed by the developer, just use [UUID][UUID] or [ULID][ZIO-ULID].\n  - Remember that identifiers are automatically lexicographically sorted :)\n\nAPI is designed to not lie, all functions signatures describe precisely\nwhat you must expect from them, thanks to [ZIO][ZIO] and [Scala3][Scala3].  \n\n## Definitions\n\nFor a better understanding, this library use a slightly different vocabulary from LMDB original one :  \n- **Database** :  (*LMDB talk about Environment*)\n  - The place where the database file is stored on your file system\n  - A set of configuration for this database (expected maximum size, expected collection number)\n- **Collection** : (*LMDB talk about Database*) \n  - A sorted map where to store your data\n  - One database contains multiple collection\n- **Transaction** : (*the same for LMDB*)\n  - for global coherency within the same database\n  - only one simultaneous write access is possible within the same database \n\n\n## Configuration\n\nConfiguration is based on the standard ZIO config mechanism, the default configuration provider uses environnment variables\nor java properties to resolve this library configuration parameters.\n\n \n| Configuration key   | Environment variable | Description                                                    | Default value    |\n|---------------------|----------------------|----------------------------------------------------------------|------------------|\n| lmdb.name           | LMDB_NAME            | Database name, which will be also used as the directory name   | default          |\n| lmdb.home           | LMDB_HOME            | Where to store the database directory                          | $HOME/.lmdb      |\n| lmdb.sync           | LMDB_SYNC            | Synchronize the file system with all database write operations | false            |\n| lmdb.maxReaders     | LMDB_MAXREADERS      | The maximum number of readers                                  | 100              |\n| lmdb.maxCollections | LMDB_MAXCOLLECTIONS  | The maximum number of collections which can be created         | 10_000           |\n| lmdb.mapSize        | LMDB_MAPSIZE         | The maximum size of the whole database including metadata      | 100_000_000_000L |\n\n\n## Usages example\n\n```scala\n//\u003e using scala  \"3.6.4\"\n//\u003e using dep \"fr.janalyse::zio-lmdb:2.0.0\"\n//\u003e using javaOpt \"--add-opens\", \"java.base/java.nio=ALL-UNNAMED\", \"--add-opens\", \"java.base/sun.nio.ch=ALL-UNNAMED\"\n\nimport zio.*, zio.json.*,zio.lmdb.*,zio.lmdb.json.*\nimport java.io.File, java.util.UUID, java.time.OffsetDateTime\n\ncase class Record(uuid: UUID, name: String, age: Int, addedOn: OffsetDateTime) derives LMDBCodecJson\n\nobject SimpleExample extends ZIOAppDefault {\n  override def run = example.provide(LMDB.liveWithDatabaseName(\"lmdb-data-simple-example\"), Scope.default)\n\n  val collectionName = \"examples\"\n  val example        = for {\n    examples  \u003c- LMDB.collectionCreate[Record](collectionName, failIfExists = false)\n    recordId  \u003c- Random.nextUUID\n    dateTime  \u003c- Clock.currentDateTime\n    record     = Record(recordId, \"John Doe\", 42, dateTime)\n    _         \u003c- examples.upsertOverwrite(recordId.toString, record)\n    gotten    \u003c- examples.fetch(recordId.toString).some\n    collected \u003c- examples.collect()\n    _         \u003c- Console.printLine(s\"collection $collectionName contains ${collected.size} records\")\n    _         \u003c- ZIO.foreach(collected)(record =\u003e Console.printLine(record))\n    lmdb      \u003c- ZIO.service[LMDB]\n    _         \u003c- Console.printLine(\"\"\"LMDB standard tools can be used to manage the database content : sudo apt-get install lmdb-utils\"\"\")\n    _         \u003c- Console.printLine(s\"\"\"To get some statistics     : mdb_stat -s $collectionName ${lmdb.databasePath}/\"\"\")\n    _         \u003c- Console.printLine(s\"\"\"To dump collection content : mdb_dump -p -s $collectionName ${lmdb.databasePath}/\"\"\")\n  } yield ()\n}\n\nSimpleExample.main(Array.empty)\n```\n\nTo run the previous logic, you'll have to provide the LMDB layer, two layers are available :\n- `LMDB.live` : Fully configurable using standard zio-config\n- `LMDB.liveWithDatabaseName(\"chosen-database-name\")` : to override/force the database name\n  (quite useful when writing scala scripts)\n\n### ZIO-LMDB based Applications\n- [sotohp - photos management][SOTOHP] which uses zio-lmdb intensively\n- [code-examples-manager - snippets/gists management][CEM] lmdb used for caching and data sharing\n- [zwords - wordle like game][ZWORDS-CODE] which can be played [zwords game][ZWORDS-LIVE]\n\n### Code snippets using ZIO-LMDB, runnable with [scala-cli][SCL]\n- [ZIO LMDB simple example (scala-3)](https://gist.github.com/dacr/dcb8a11f095ef0a2a95c24701e6eb804)\n- [ZIO LMDB feeding with French town postal codes](https://gist.github.com/dacr/6d24baf827ae0c590133e0f27f1ef20b)\n- [ZIO LMDB using custom configuration provider](https://gist.github.com/dacr/790df1705c7ec19ae2fe4098dad8d762)\n- [Extract photos records from elasticsearch and save them into LMDB](https://gist.github.com/dacr/6ea121f251ad316a64657cbe78085ab7)\n- [Export code examples and executions results from lmdb to elastisearch](https://gist.github.com/dacr/f25da8222b2ac644c3195c5982b7367e)\n\n## Operating lmdb databases\n\nLMDB standard tools can be used to manage the databases content : `sudo apt-get install lmdb-utils`\n- to get some database statistics : `mdb_stat -a database_directory_path/`\n- to dump the content of a database : `mdb_dump -a -p database_directory_path/`\n- to dump the content of a database collection : `mdb_dump -s collectionName -p database_directory_path/`\n- to restore some collection or the entire database use the command named `mdb_load` which uses the same format as for `mdb_dump` \n\nAs zio-lmdb is using json format, dumps are just text, which can be edited and then loaded back. So simple data migration is straightforward.\n\n## Requirements\n\nWhen LVMDB is used as persistence store with recent JVM it requires some JVM options :\n\n```\n--add-opens java.base/java.nio=ALL-UNNAMED\n--add-opens java.base/sun.nio.ch=ALL-UNNAMED\n```\n\n## Contributors :)\n\n- [François Armand](https://github.com/fanf) : for scala 2.13 support initiative\n\n\n[ZIOLMDBManager]:    https://github.com/dacr/zio-lmdb\n[ZIOLMDBManagerImg]: https://img.shields.io/maven-central/v/fr.janalyse/zio-lmdb_3.svg\n[ZIOLMDBManagerLnk]: https://mvnrepository.com/artifact/fr.janalyse/zio-lmdb\n[ZIO]: https://zio.dev/\n[Scala3]: https://docs.scala-lang.org/scala3/reference/\n[JLMDB]: https://github.com/lmdbjava/lmdbjava\n[LMDB]: https://www.symas.com/lmdb\n[ZIO-ULID]: https://zio-ulid.bilal-fazlani.com/\n[UUID]: https://en.wikipedia.org/wiki/Universally_unique_identifier\n[ZWORDS-CODE]: https://github.com/dacr/zwords\n[ZWORDS-LIVE]: https://zwords.mapland.fr/\n[CEM]: https://github.com/dacr/code-examples-manager\n[SOTOHP]: https://github.com/dacr/sotohp\n[SCL]: https://scala-cli.virtuslab.org/\n[ScalaDocImg]: https://javadoc.io/badge2/fr.janalyse/zio-lmdb_3/scaladoc.svg\n[ScalaDoc]: https://javadoc.io/doc/fr.janalyse/zio-lmdb_3/latest/zio/lmdb/LMDB$.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdacr%2Fzio-lmdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdacr%2Fzio-lmdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdacr%2Fzio-lmdb/lists"}