{"id":21436754,"url":"https://github.com/modfin/zdap","last_synced_at":"2025-03-16T23:23:50.294Z","repository":{"id":57620055,"uuid":"388490391","full_name":"modfin/zdap","owner":"modfin","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-10T09:16:38.000Z","size":168,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-10T10:26:17.683Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/modfin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-22T14:25:46.000Z","updated_at":"2025-03-10T09:16:41.000Z","dependencies_parsed_at":"2022-09-16T19:23:05.551Z","dependency_job_id":"5cd173c9-8b1a-4905-ba5f-b6a247e6ca36","html_url":"https://github.com/modfin/zdap","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fzdap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fzdap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fzdap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fzdap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/modfin","download_url":"https://codeload.github.com/modfin/zdap/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243945787,"owners_count":20372932,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-23T00:14:58.827Z","updated_at":"2025-03-16T23:23:50.288Z","avatar_url":"https://github.com/modfin.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n`zdap`, `zfs-database-access-proxy`, is a commandline tool and server ware to offload large databases in a docker compose dev environment to other machines. \n\n# Background\nWith a sample size of 1, our experience at Modular Finance, is as follows\n \nWe often use, censored, production data in our development environment and access to the full production databases is often required for efficient development. Starting out, we had databases which were around 5-10gb, just cloning them every once in a while to your local environment add adding restoring them to a postgres docker image was little trouble. However, time goes on and now we have multiple databases running at more than 200gb which makes cloning a much more tidies process which can take multiple hours.\n\n## Environment \nHistorically most have of us has used stationary computers in order to handle the load of these large datasets needed for much of the daily work. Covid and the general growth of the company has made flexibility and laptops much more of a requirement. \n\nWe have for a very long time been completely dockerized, which to us means, that we run our development environment insider docker containers using docker-compose, a long with k8s for production. Our databases for our development environment is simply postgres images running through docker-compose with a mounted volume.     \n\n## Concept\nThe basic concept for `zdap` is to override a docker-compose file with a proxy, that offloads the database, to a server.\n\nEveryone want there own databses, so the basic idea is for our server to clone the database into a zfs volume. Since zfs has copy-on-write support, we can utilize snapshots and zfs-clones to provide everyone with their own database instance with zero overhead. \n\n\n# Components\nThere are three components to `zdap`\n* `zdapd`       - server \n* `zdap`        - client commandline tool\n* `zdap-proxyd` - internal proxy\n\n \n# zdapd\n\n`zdapd` is the daemon running on the database server that exposes a http api for management.\n\n## Dependency\n* `zfs`\n* `zfs dev`\n* `docker`\n\n## zfs dependency  \nInstall zfs\n```bash \napt-get install zfsutils-linux libzfslinux-dev\n```\n\nSetting up a pool for zdap to use for storing the databases\n```bash \nzpool create zdap-pool /dev/sdx1 /dev/sdx2 ...\nzpool list \n#NAME        SIZE   ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT\n#zdap-pool  4,56T  94,5K   4,56T        -         -     0%     0%  1.00x    ONLINE  -\n\nzfs list\n#NAME        USED  AVAIL     REFER  MOUNTPOINT\n#zdap-pool    87K  4,56T       24K  /zdap-pool\n\n```\n\n\n## Installing\n\n\n```bash\n## Installing\ngo install github.com/modfin/zdap/cmd/zdapd@latest\n\n## Running\nzdapd --zpool=zdap-pool \\\n      --config-dir=/path/to/config/dir \\\n      --network-address=\u003cip address of the machine\u003e \\\n      --api-port=43210 \\\n      serv\n```\n\n\n# zdap\n\n`zdap` is the cli tool that is used by a user in order to create instances of database and attach them to the docker-compose environment    \n\n## Installing \u0026 Running\n\n```bash \n## Installing\ngo install github.com/modfin/zdap/cmd/zdap@latest\n\nzdap auto-complete [bash|zsh|fish] # prints auto-compleat installation instructions\n\nzdap set user \u003cname@host\u003e\nzdap add origin \u003cip\u003e:\u003cport\u003e # you can add mutiple origins and this way zdap balences\n                            # resource creation over all servers running zdapd.\n\n## in you project where a docker-compose.yml file is located\nzdap init               # initilizes zdap for the docker-compose context\nzdap list resources     # list the databases that can be mounted\nzdap attach \u003cresource\u003e  # creates a clone of the resource in the at the zdapd \n                        # server and attaches it to docker-compose.override file.\n                        # docker-compose up \u003cresource\u003e will now use the zdapd server\n                        # instance of the database\nzdap detach \u003cresource\u003e  # detaches the resource from the docker-compose.override \n                        # and destroys the resource-clone on the zdap server\n```\n\n# kubernetes\nTo access a zdap database from within a kubernetes cluster, just deploy a zdap-proxy image where the\n`ZDAP_CLONE_OWNER_NAME`, `ZDAP_RESOURCE` and `ZDAP_SERVERS` environment variable are set. `ZDAP_CLONE_OWNER_NAME` must \na be unique identifier for the cluster.\n\nThere is also another optional\nparameter (`ZDAP_RESET_AT_HH_MM`) that can be set to automatically take a new snapshot each day.\n\nDeployment example:\n```yaml\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: postgres-x\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: postgres-x\n  strategy:\n    type: Recreate\n  template:\n    metadata:\n      labels:\n        app: postgres-x\n    spec:\n      containers:\n        - name: postgres-x\n          image: modfin/zdap-proxy:latest\n          env:\n            - name: ZDAP_CLONE_OWNER_NAME\n              value: \"staging@example.com\"\n            - name: ZDAP_RESOURCE\n              value: \"postgres-x\"\n            - name: ZDAP_SERVERS\n              value: \"192.168.0.x,192.168.0.y\"\n            - name: ZDAP_RESET_AT_HH_MM\n              value: \"0130\"\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodfin%2Fzdap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmodfin%2Fzdap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodfin%2Fzdap/lists"}