{"id":20327619,"url":"https://github.com/tokahuke/sled_to_postgres","last_synced_at":"2026-04-17T18:03:32.538Z","repository":{"id":118776233,"uuid":"272068294","full_name":"tokahuke/sled_to_postgres","owner":"tokahuke","description":"Replicate your Sled database to Postgres ","archived":false,"fork":false,"pushed_at":"2020-12-03T13:11:34.000Z","size":168,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-04-03T10:57:56.495Z","etag":null,"topics":["key-value-database","key-value-store","postgres","postgresql","rust","rust-lang","sled"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tokahuke.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-13T18:55:17.000Z","updated_at":"2022-04-13T07:26:48.000Z","dependencies_parsed_at":null,"dependency_job_id":"3c7c5fc3-a0ad-4b8b-b5e5-9342d8532ee4","html_url":"https://github.com/tokahuke/sled_to_postgres","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tokahuke/sled_to_postgres","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokahuke%2Fsled_to_postgres","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokahuke%2Fsled_to_postgres/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokahuke%2Fsled_to_postgres/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokahuke%2Fsled_to_postgres/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tokahuke","download_url":"https://codeload.github.com/tokahuke/sled_to_postgres/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokahuke%2Fsled_to_postgres/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31939788,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-17T17:29:20.459Z","status":"ssl_error","status_checked_at":"2026-04-17T17:28:47.801Z","response_time":62,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["key-value-database","key-value-store","postgres","postgresql","rust","rust-lang","sled"],"created_at":"2024-11-14T19:48:18.616Z","updated_at":"2026-04-17T18:03:32.531Z","avatar_url":"https://github.com/tokahuke.png","language":"Rust","readme":"# Replicate your Sled database to Postgres\n\n\u003ca href=\"https://docs.rs/sled_to_postgres\"\u003e\u003cimg src=\"https://docs.rs/sled_to_postgres/badge.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://crates.io/crates/sled_to_postgres\"\u003e\u003cimg src=\"https://img.shields.io/crates/v/sled_to_postgres.svg\"\u003e\u003c/a\u003e\n\n**WIP: although this is part of a bigger project, it has not yet been used in production. My use case allows for _some_ small inconsistency. If yours doesn't, _caveat emptor._**\n\n## Introduction\n\nThis crate provides replication from Sled (a Key-Value database) to Postgres (a RDBMS). It uses `tokio_postgres` to connect to Postgres and the `Subscriber` API in Sled to watch for updates. Since Sled is agnostic to the type in the database and Postgres is strongly typed, you must provide the serialization/deserialization functions explicitly whe setting the replication up.\n\nAgain, this is still a _work in progress_. Do _not_ use this crate (yet) if you need high reliability. \n\n## Quickstart \n\nThis is a sample usage example for a single tree. For more trees, just chain more `Replication::push` calls. First of all, let's set up the replication itself:\n\n```rust\n// (`ToSql` is reexported from `tokio_postgres`)\nuse sled_to_postgres::{Replication, ReplicateTree, ToSql};\n\n// Open your database:\nlet db = sled::open(\"data/db\").unwrap();\n// Open your tree.\nlet tree = db.open_tree(\"a_tree\").unwrap();\n\n/// Makeshift decode.\nfn decode(i: \u0026[u8]) -\u003e i32 {\n    i32::from_be_bytes([i[0], i[1], i[2], i[3]])\n}\n\n// This is how you set up a replication:\nlet setup = Replication::new(\n        // Put in your credentials.\n        \"host=localhost dbname=a_db user=someone password=idk\",\n        // You will need a location in the disk for temporary data.\n        \"data/replication\",\n    ).push(ReplicateTree {\n        // This is a replication on the tree `a_tree`.\n        tree: tree.clone(),\n        // You may specify a replication only over a given prefix.\n        prefix: vec![],\n        // This are the commands for table (and index) creation.\n        // This needs to be *idempotent* (create _if not exists_).\n        create_commands: \"\n            create table if not exists a_table (\n                x int primary key,\n                y int not null\n            );\n        \",\n        // This is the command for one insertion. The replication might need\n        // to call this repeatedly for the same data. \n        insert_statement: \"\n            insert into a_table values ($1::int, $2::int)\n            on conflict (x) do update set x = excluded.x;\n        \",\n        // This is how you transform a `(key, value)` into the parameters for\n        // the above statement.\n        // ... this is the general and complicated form. You can simplify \n        // stuff using `params!`.\n        insert_parameters: |key: \u0026[u8], value: \u0026[u8]| {\n            vec![\n                Box::new(decode(\u0026*key)) as Box\u003cdyn ToSql + Send + Sync\u003e,\n                Box::new(decode(\u0026*value)) as Box\u003cdyn ToSql + Send + Sync\u003e,\n            ]\n        },\n        // This is the command for one removal. The replication needs to call\n        // this repeatedly for the same data. \n        remove_statement: \"delete from a_table where x = $1::int\",\n        // This is how you transform a `key` into the parameters for the above\n        // statement.\n        // ... using `params!` makes it more ergonomic.\n        remove_parameters: |key: \u0026[u8]| params![decode(\u0026*key)],\n    });\n```\n\nNow, after we have configured the replication, let's put it to run. Since `tokio_postgres` is based on `tokio`, we will use the `tokio` runtime.\n\n```rust\ntokio::spawn(async move {\n    // Do not insert anything before starting the replication. These events will not be logged.\n    // tree.insert(\u0026987i32.to_be_bytes(), \u0026654i32.to_be_bytes()).unwrap();\n    \n    // Although the current state of the database *will* be dumped with the \n    // replication when it starts for the first time.\n\n    // Start the replication.\n    let stopper = replication.start().await.unwrap();\n\n    // Now, insert something in `a_tree`.\n    tree.insert(\u0026123i32.to_be_bytes(), \u0026456i32.to_be_bytes()).unwrap();\n    \n    // When you are done, trigger shutdown:\n    // It is understood that _there will be no more db operations after this\n    // point._\n    // Shutdown doesn't happen immediately. It takes at least 500ms.\n    stopper.stop().await;\n});\n```\n\n## Limitations and _caveats_\n\nThere are some limitations on the current implementation:\n1. Do not use foreign key constraints on the replicated tables. Since Sled\ndoesn't have such a concept and updates are inserted in small batches \nconcurrently, the table might not obey this constraint during brief moments.\n2. Be careful to start the replications before any updates are done to the\ndatabase, preferably giving it a head-start of a couple of milliseconds.\n3. Be careful to only trigger the end of the replication when you are\nabsolutely sure no more updates are going to be made from that point on.\n\n## License\n\nThis code is licensed under the Apache 2.0 license. See the attached `license` file for further details.","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftokahuke%2Fsled_to_postgres","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftokahuke%2Fsled_to_postgres","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftokahuke%2Fsled_to_postgres/lists"}