{"id":18409605,"url":"https://github.com/portasynthinca3/sidx","last_synced_at":"2025-10-14T00:37:47.739Z","repository":{"id":65133721,"uuid":"582786789","full_name":"portasynthinca3/sidx","owner":"portasynthinca3","description":"Simple key-value store with subindex support for the BEAM implemented in pure Elixir. Mainly intended to be used by the `markov` library","archived":false,"fork":false,"pushed_at":"2023-10-19T16:30:34.000Z","size":187,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-16T18:14:52.530Z","etag":null,"topics":["database","elixir","key-value"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"wtfpl","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/portasynthinca3.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-27T21:39:04.000Z","updated_at":"2023-01-09T19:20:57.000Z","dependencies_parsed_at":"2024-11-06T03:40:56.727Z","dependency_job_id":null,"html_url":"https://github.com/portasynthinca3/sidx","commit_stats":{"total_commits":7,"total_committers":1,"mean_commits":7.0,"dds":0.0,"last_synced_commit":"81fa477ec4188eeb553505b5fe7f759a4aeb6913"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/portasynthinca3/sidx","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/portasynthinca3%2Fsidx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/portasynthinca3%2Fsidx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/portasynthinca3%2Fsidx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/portasynthinca3%2Fsidx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/portasynthinca3","download_url":"https://codeload.github.com/portasynthinca3/sidx/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/portasynthinca3%2Fsidx/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279017357,"owners_count":26086052,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","elixir","key-value"],"created_at":"2024-11-06T03:26:13.850Z","updated_at":"2025-10-14T00:37:47.720Z","avatar_url":"https://github.com/portasynthinca3.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Subindex\nSimple key-value store with subindex support for the BEAM implemented in pure\nElixir. Mainly intended to be used by the\n[markov](https://github.com/portasynthinca3/markov) library.\n\n## Goals\nThis database was created for a very specific purpose - to support the\naforementioned markov library, which in turn powers the\n[Deuterium](https://github.com/portasynthinca3/deutexrium) Discord bot. It\nstrives to achieve the following goals:\n  - Integrated into the BEAM VM as its own OTP app.\n  - Write speed at the expense of durability; this is because Deuterium receives\n  about 10 write requests for every read request.\n  - Support for \"subindexing\", or partial keys.\n  - No need for entire tables to be loaded into memory; this requirement arose\n  after trying `ets` and `mnesia`.\n\n## Installation\n```elixir\ndefp deps do\n  [\n    {:sidx, \"~\u003e 0.1\"}\n  ]\nend\n```\n\n## Usage\nExample workflow:\n```elixir\n# loads table at \"./path\" or creates one if it doesn't exist\ntable = Sidx.open!(\"./path\", keys: 3)\n\n:ok = Sidx.insert(table, [:a, :b, :c], :d) # the 3 keys and a value\n:ok = Sidx.insert(table, [:e, :f, :g], :h)\n:ok = Sidx.insert(table, [:e, :f, :i], :j)\n:ok = Sidx.insert(table, [:e, :k, :l], :m)\n\nSidx.select(table, [:a, :b, :c]) # {:ok, [{[], :d}]}\nSidx.select(table, [:e, :f, :g]) # {:ok, [{[], :h}]}\nSidx.select(table, [:e, :f, :i]) # {:ok, [{[], :j}]}\nSidx.select(table, [:e, :f])     # {:ok, [{[:g], :h}, {[:i], :j}]}\nSidx.select(table, [:e, :k])     # {:ok, [{[:l], :m}]}\nSidx.select(table, [:e])         # {:ok, [\n                                 #   {[:g, :f], :h}   # subkeys in reverse order\n                                 #   {[:i, :f], :j},\n                                 #   {[:l, :k], :m}]\n                                 # }\n\n# inserts with keys to an existing row overwrite the value\n:ok = Sidx.insert(table, [:a, :b, :c], 0)\nSidx.select(table, [:a, :b, :c]) # {:ok, [{[], 0}]}\n\n# update is atomic and more efficient than select+insert\nSidx.update(table, [:a, :b, :c], fn _path, value -\u003e value + 1 end)\nSidx.select(table, [:a, :b, :c]) # {:ok, [{[], 1}]}\n\nSidx.close!(table)\n```\n\n## Inner workings\n  - The database is partitioned by the first key in the key list\n  - Every partition consists of a **key tree** and a **value store**\n  - Partitions are **serialized** into a binary\n  - A serialized partition is **fragmented** into equally-sized **slots**\n  - Multiple serialized partitions' slots are intertwined and put into one file\n  in a process called **unification**\n  - When a partition needs to be read from disk, the unifier will first\n  **collect** the slots corresponding to that partition which will then be\n  **concatenated** and finally **deserialized**\n\n![Visual representation of the above operations](./inner_workings.png)\n\n## Results\nThis database has achieved the required performance target, however it fails to\nfail gracefully. A sudden stop of the VM or an interruption of the writing\nprocess leads to a corrupted database. Ideally, this should not happen, however\nit is by design _okay_ to lose writes in such an event.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fportasynthinca3%2Fsidx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fportasynthinca3%2Fsidx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fportasynthinca3%2Fsidx/lists"}