{"id":19473781,"url":"https://github.com/wtdcode/mdbx-remote","last_synced_at":"2025-04-25T12:31:40.883Z","repository":{"id":259475917,"uuid":"877959455","full_name":"wtdcode/mdbx-remote","owner":"wtdcode","description":"Access your MDBX database over network safely","archived":false,"fork":false,"pushed_at":"2024-12-21T08:22:57.000Z","size":744,"stargazers_count":20,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-14T19:56:44.378Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wtdcode.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-24T14:33:36.000Z","updated_at":"2025-04-11T12:56:20.000Z","dependencies_parsed_at":"2024-10-25T21:10:38.575Z","dependency_job_id":"f16e10fc-4b51-4a2d-b45c-253cf8a43894","html_url":"https://github.com/wtdcode/mdbx-remote","commit_stats":null,"previous_names":["wtdcode/mdbx-remote"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wtdcode%2Fmdbx-remote","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wtdcode%2Fmdbx-remote/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wtdcode%2Fmdbx-remote/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wtdcode%2Fmdbx-remote/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wtdcode","download_url":"https://codeload.github.com/wtdcode/mdbx-remote/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250817724,"owners_count":21492210,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T19:20:17.257Z","updated_at":"2025-04-25T12:31:38.525Z","avatar_url":"https://github.com/wtdcode.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MDBX-Remote\n\n`mdbx-remote` is a tool to access your MDBX databases over network safely without any concern of corrupting the databases or getting dirty data. `mdbx-remote` also provides the _same_ but _async_ programming interface as the original `libmdbx-rs` bindings, which _passes_ all tests of origin `libmdbx-rs`.\n\n## Motivation\n\n[MDBX](https://libmdbx.dqdkfa.ru/) is a fast and easy-to-use KV database offering ACID properties. However, one of the limitations of MDBX is that it tries hard to prevent users from accessing the database via network to avoid potential corruption. For example, opening a database in the common collaborative mode will immediately get `MDBX_EREMOTE`. Even if you open the database via `MDBX_EXCLUSIVE`, it is still highly likely to corrupt the database or get inconsistent data if there are remote readers, which MDBX can not always detect. This is super inconvenient in many cases, like just changing one or two entries.\n\n`mdbx-remote` solves this by spinning up an RPC service to expose all raw MDBX primitives.\n\n## Usages\n\n### Build\n\nSimply do\n\n```\ncargo build --release\n```\n\nNote `libclang-dev` is required to generate the libmdbx bindings.\n\n### Docker Image\n\nAlternatively, there is a [docker image](https://hub.docker.com/r/lazymio/mdbx-remote) provided to help setup.\n\n### MDBX Utility\n\n`mdbx-remote` is shipped with an `mdbx` utlity. For example, spin up an RPC endpoint:\n\n```\n# By default, this will listen at 0.0.0.0:1899\nRUST_LOG=mdbx=info cargo run --release server\n# Or \ndocker run --rm -it --env RUST_LOG=info lazymio/mdbx-remote server\n```\n\nAnd stat any database with a url:\n\n```\n# Stat a remote reth database\ncargo run --release stat mdbx://127.0.0.1/remote/path/to/db?ro=1\n# Or\ndocker run --rm -it --env RUST_LOG=info --network host \\\n    lazymio/mdbx-remote stat mdbx://127.0.0.1/remote/path/to/db?ro=1\n\nDatabase:\n\tPage size: 4096\n\tEntries: 19265135020\n\tLeaf Pages: 365773779\n\tBranch Pages: 23212389\n\tDepth: 102\n\tOverflow Pages: 5270586\nTable Main:\n\tPage size: 4096\n\tEntries: 29\n\tLeaf Pages: 1\n\tBranch Pages: 0\n\tDepth: 1\n\tOverflow Pages: 0\nTable AccountChangeSets:\n\tPage size: 4096\n\tEntries: 3298505382\n\tLeaf Pages: 42669897\n\tBranch Pages: 13964000\n\tDepth: 4\n\tOverflow Pages: 0\n...\n```\n\nThis exactly mimics the output of `mdbx_stat` utility from `libmdbx`. Note the `mdbx://` URL supports a subset of MDBX features by parameters including:\n\n- `ro`: Read-only\n- `rw`: Read-write\n- `accede`: Use options from existing environment\n- `exclusive`: Open in the exclusive mode\n- `no_sub_dir`: No directory for the mdbx database\n- `max_readers`: Max readers\n- `max_dbs`: Max databases\n- `deadline`: The seconds to wait before giving up an RPC request\n\nIt is also worth mentioning that, if the host part is empty, like `mdbx:///path/to/local/db`, it refers to a local MDBX database located at `/path/to/local/db`.\n\n### Access Reth Database\n\nOne of the motivation to build `mdbx-remote` is to access reth database. `mdbx reth` can get a value from one reth table.\n\n```\ncargo run --release reth \\\n    -u mdbx://127.0.0.1/path/to/reth/db \\\n    -t PlainAccountState \\\n    0x95222290DD7278Aa3Ddd389Cc1E1d165CC4BAfe5\n# Or\ndocker run --rm -it --network host lazymio/mdbx-remote reth \\\n    -u mdbx://127.0.0.1/path/to/reth/db \\\n    -t PlainAccountState \\\n    0x95222290DD7278Aa3Ddd389Cc1E1d165CC4BAfe5\n\nThe account is:\n{\n  \"nonce\": 1569575,\n  \"balance\": \"0x7db63f1578c045b9\",\n  \"bytecode_hash\": null\n}\n```\n\n### For Developers\n\n`mdbx-remote` reaches almost the full parity with `libmdbx-rs` interfaces, except forcing everything to _async_ because of `tarpc`. Generally, the table below shows the types for different scenarios.\n\n||Environment|Database|Transaction|Cursor|\n|-|-|-|-|-|\n|Local|`Environment`|`Database`|`Transaction\u003cK\u003e`|`Cursor\u003cK\u003e`|\n|Remote|`RemoteEnvironment`|`RemoteDatabase`|`RemoteTransaction\u003cK\u003e`|`RemoteCursor\u003cK\u003e`|\n|Local or Remote|`EnvironmentAny`|`DatabaseAny`|`TransactionAny\u003cK\u003e`|`CursorAny\u003cK\u003e`|\n\nAdd this to your Cargo.toml:\n\n```\nlibmdbx-remote = \"0.3\"\n```\n\n### Batch RPC Calls\n\nYou certainly don't want to make 1 RPC call per data fetching, which wastes your bandwith and is super slow.\n\n`mdbx-remote` provides buffered cursor operations for high performance data accessing.\n\nCheck the methods with `_buffered` suffix for details.\n\n## Caveats\n\n- Only expose the RPC endpoint to trusted network because it doesn't have any protection or authentication. Tunnel the traffic via something like SSH if you really would like to expose it to public network.\n- The same database can't be opened twice as inherited from MDBX.\n\n## TODO\n\n- Batch cursor (taking a `self`) get/put to improve throughput\n- Implmenet more MDBX utilities, especially `mdbx_dump`.\n- Reach full parity with `libmdbx-rs`.\n- Reach more parity with `reth db` operations.\n- Benchmark the performance.\n- Run Reth with remote storage.\n\n## Credits\n\nThe bindings is forked from reth v1.1.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwtdcode%2Fmdbx-remote","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwtdcode%2Fmdbx-remote","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwtdcode%2Fmdbx-remote/lists"}