{"id":16248330,"url":"https://github.com/cmars/ddcp","last_synced_at":"2025-06-14T07:39:56.588Z","repository":{"id":194691392,"uuid":"691377052","full_name":"cmars/ddcp","owner":"cmars","description":"Database-to-Database Copy (DDCP) over Veilid ","archived":false,"fork":false,"pushed_at":"2024-07-03T08:31:59.000Z","size":286,"stargazers_count":8,"open_issues_count":3,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-17T10:12:23.758Z","etag":null,"topics":["database","distributed","local-first","privacy","sqlite","veilid"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cmars.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-14T04:00:56.000Z","updated_at":"2025-02-27T02:48:04.000Z","dependencies_parsed_at":"2023-10-16T11:23:06.456Z","dependency_job_id":"a4912308-2b7e-4f0f-a977-b5af01e53e9f","html_url":"https://github.com/cmars/ddcp","commit_stats":null,"previous_names":["cmars/velouria","cmars/ddcp"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmars%2Fddcp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmars%2Fddcp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmars%2Fddcp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmars%2Fddcp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cmars","download_url":"https://codeload.github.com/cmars/ddcp/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244493111,"owners_count":20461609,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","distributed","local-first","privacy","sqlite","veilid"],"created_at":"2024-10-10T14:41:16.959Z","updated_at":"2025-03-19T19:31:55.045Z","avatar_url":"https://github.com/cmars.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DDCP\n\nDatabase-to-Database Copy (DDCP) over [Veilid](https://veilid.com).\n\nNo servers, no cloud. No gods, no masters.\n\n# What is DDCP?\n\nDDCP (Database-to-Database Copy) is two things:\n\n- A Git-like CLI for [VLCN](https://vlcn.io/)'s CR-SQLite.\n- A Rust-native networking layer, which coordinates VLCN database changes among remote peers.\n\nBoth operate over [Veilid](https://veilid.com) for simple, private, and secure p2p networking with strong cryptographic identities.\n\n# Why would I use DDCP?\n\nRapid development of local-first distributed applications. Cut out the boilerplate and yak-shaving of APIs and focus on the data.\n\nConvert traditional RDBMS apps into local-first without having to re-architect around low-level CRDTs.\n\n# How do I use it?\n\nDDCP is still in the early stages of development but is already pretty interesting. Here's what you can do with it:\n\n## Initialize a SQLite database\n\n```bash\nddcp init\n```\n\nThis registers a Veilid DHT public key for your database.\n\n```bash\n2023-09-27T16:50:03.646448Z  INFO ddcp: Registered database at DHT key=\"VLD0:73iT7NuqplS2nab1AH4P7lJYiQROR_k4NyUhag_K4DY\"\n```\n\n## Publish it\n\n```bash\nddcp serve\n```\n\n```bash\n2023-09-27T16:50:36.319921Z  INFO ddcp: Serving database at DHT key=\"VLD0:73iT7NuqplS2nab1AH4P7lJYiQROR_k4NyUhag_K4DY\"\n2023-09-27T16:50:36.321171Z  INFO ddcp: Database changed, updated status db_version=2 key=\"VLD0:73iT7NuqplS2nab1AH4P7lJYiQROR_k4NyUhag_K4DY\"\n```\n\nThis process must remain running in the background in order for remotes to be able to fetch changes from this database.\n\nThis process also prints the Veilid public key to give to your peers.\n\n## Populate it\n\nCreate some VLCN [CRRs (Conflict-free Replicated Relations)](https://vlcn.io/docs/appendix/crr) in your database. This can be done while DDCP is running and publishing your database. Changes will be automatically picked up by remote subscribers.\n\n```bash\nddcp shell \u003c fixtures/cr_tables.sql\nddcp shell \u003c fixtures/ins_test_schema_alice.sql\n```\n\n## Replicate it\n\nAdd remote peer databases to synchronize with:\n\n```bash\nddcp remote add alice 73iT7NuqplS2nab1AH4P7lJYiQROR_k4NyUhag_K4DY\n```\n\nThe `VLD0:` prefix is optional.\n\nChanges from remotes are automatically applied to the local database while publishing changes.\n\n```bash\nddcp serve\n```\n\n```bash\n2023-09-27T17:01:30.615367Z  INFO ddcp: Pulled changes from remote database remote_name=\"alice\" db_version=5\n```\n\n# VLCN Caveats\n\nCRRs have certain restrictions in order to work as expected, or even at all:\n\n- Changes in CRRs MUST happen after loading the crsqlite.so extension. Running `ddcp shell` does this automatically for you. Something to keep in mind when using cr-sqlite from other applications though.\n- Peers have to share a common schema in order to replicate changes. VLCN supports schema migrations.\n- Replication happens on primary keys. Avoid auto-incrementing primary keys, these will not merge well. Prefer strong content identifiers or large random UUID-like keys.\n\n# Development\n\n## OCI image\n\nProbably the easiest way to evaluate DDCP. OCI image build is based on Debian Bookworm.\n\n```bash\ndocker build -t ddcp .\n```\n\nUsage:\n\n```bash\ndocker run -v $(pwd)/data:/data ddcp:latest init\ndocker run -v $(pwd)/data:/data ddcp:latest remote add alice 73iT7NuqplS2nab1AH4P7lJYiQROR_k4NyUhag_K4DY\ncat fixtures/cr_test_schema.sql | docker run -v $(pwd)/data:/data -i ddcp:latest shell\ndocker run -v $(pwd)/data:/data ddcp:latest pull\n```\n\n## Nix\n\nDDCP currently builds in a Nix flake devshell. If you Nix,\n\n```bash\nnix develop\ncargo build\n```\n\n## Other platforms\n\nThis might probably work, with compilers and development libraries installed:\n\n```bash\n# Install serde tooling; used in cargo build (for now)\n./scripts/install_capnproto.sh\n./scripts/install_protoc.sh\n\ncargo build --release\n```\n\nYou can also try `cargo install ddcp` but the build will still require the serde tooling to be installed.\n\n# Roadmap\n\nDDCP is still a tech preview. Wire protocol, schemas, and APIs are all unstable and subject to breaking changes.\n\nSeveral areas where DDCP still needs improvements:\n\n### Private sharing\n\nDHT addresses are currently publicly accessible; if you know the address, you can pull from the database. This is OK for public distribution of data sets (blogs, \"everybody edits\", stuff like that), but not for private apps.\n\nSupport sharing for authenticated DHTs.\n\n### Changeset filtering and transformation\n\nMore control over how changes are merged. In some use cases, you want to merge all peers' changes in a consistent state. In others, you probably want to keep peers' content separate but linked. Primitives that support these different synchronization patterns. Filters \u0026 transformations on `crsql_changes`. Authn and authz (who can change or pull what).\n\n### Large changesets\n\nOvercome `app_call` message size limits. Currently there's no checks on this so sending large changesets (like pictures of cats) will likely fail in uncontrolled ways. Veilid messages are typically limited to 32k.\n\n32k is probably enough for a row without large column data. Large column data should probably reference block storage, like PostgreSQL's Toast.\n\n### Missing Veilid features\n\nReplace polling with a watch on DHT changes (currently [not implemented](https://gitlab.com/veilid/veilid/-/blob/bd4b4233bfed5bdca4da3cacda3ad960e28daab5/veilid-core/src/storage_manager/mod.rs#L485) AFAICT).\n\nVeilid blockstore integration when it's ready.\n\n### Some apps\n\nIdeas for kinds of data that could be shared:\n\n- Distributed BBS\n- Bookmarks \u0026 link sharing\n- Network scanning results\n- Remote sensor aggregation\n- Code hosting to get this project off Git\\*\\*b\n\n# About\n\nName takes inspiration from [UUCP](https://en.wikipedia.org/wiki/UUCP) and [NNCP](https://www.complete.org/nncp/).\n\nSome of the Veilid core integration was derived from examples in [vldpipe](https://gitlab.com/vatueil/vldpipe).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmars%2Fddcp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcmars%2Fddcp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmars%2Fddcp/lists"}