{"id":20290614,"url":"https://github.com/gocardless/draupnir","last_synced_at":"2025-08-21T11:33:01.088Z","repository":{"id":44454459,"uuid":"76380203","full_name":"gocardless/draupnir","owner":"gocardless","description":"Anonymised database instances as-a-service","archived":false,"fork":false,"pushed_at":"2025-01-27T08:39:04.000Z","size":17946,"stargazers_count":46,"open_issues_count":29,"forks_count":7,"subscribers_count":62,"default_branch":"master","last_synced_at":"2025-04-06T10:03:22.771Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gocardless.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-12-13T17:01:08.000Z","updated_at":"2025-03-23T18:37:41.000Z","dependencies_parsed_at":"2024-01-09T16:48:46.570Z","dependency_job_id":"239f2da1-f750-4b77-84c1-f67bb0e3f2c0","html_url":"https://github.com/gocardless/draupnir","commit_stats":null,"previous_names":[],"tags_count":36,"template":false,"template_full_name":null,"purl":"pkg:github/gocardless/draupnir","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fdraupnir","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fdraupnir/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fdraupnir/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fdraupnir/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gocardless","download_url":"https://codeload.github.com/gocardless/draupnir/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fdraupnir/sbom","scorecard":{"id":433120,"data":{"date":"2025-08-11","repo":{"name":"github.com/gocardless/draupnir","commit":"471788b9762c5e2648644fbf2c230299ce4573bc"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.1,"checks":[{"name":"Code-Review","score":10,"reason":"all changesets reviewed","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/build-integration.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact v5.3.6 not signed: https://api.github.com/repos/gocardless/draupnir/releases/182932878","Warn: release artifact v5.3.4 not signed: https://api.github.com/repos/gocardless/draupnir/releases/182900049","Warn: release artifact v5.3.3 not signed: https://api.github.com/repos/gocardless/draupnir/releases/137779571","Warn: release artifact v5.3.1 not signed: https://api.github.com/repos/gocardless/draupnir/releases/72742617","Warn: release artifact v5.3.0 not signed: https://api.github.com/repos/gocardless/draupnir/releases/72594593","Warn: release artifact v5.3.6 does not have provenance: https://api.github.com/repos/gocardless/draupnir/releases/182932878","Warn: release artifact v5.3.4 does not have provenance: https://api.github.com/repos/gocardless/draupnir/releases/182900049","Warn: release artifact v5.3.3 does not have provenance: https://api.github.com/repos/gocardless/draupnir/releases/137779571","Warn: release artifact v5.3.1 does not have provenance: https://api.github.com/repos/gocardless/draupnir/releases/72742617","Warn: release artifact v5.3.0 does not have provenance: https://api.github.com/repos/gocardless/draupnir/releases/72594593"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Security-Policy","score":10,"reason":"security policy file detected","details":["Info: security policy file detected: github.com/gocardless/.github/SECURITY.md:1","Info: Found linked content: github.com/gocardless/.github/SECURITY.md:1","Info: Found disclosure, vulnerability, and/or timelines in security policy: github.com/gocardless/.github/SECURITY.md:1","Info: Found text in security policy: github.com/gocardless/.github/SECURITY.md:1"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Pinned-Dependencies","score":1,"reason":"dependency not pinned by hash detected -- score normalized to 1","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build-integration.yml:20: update your workflow using https://app.stepsecurity.io/secureworkflow/gocardless/draupnir/build-integration.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build-integration.yml:21: update your workflow using https://app.stepsecurity.io/secureworkflow/gocardless/draupnir/build-integration.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build-integration.yml:29: update your workflow using https://app.stepsecurity.io/secureworkflow/gocardless/draupnir/build-integration.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/build-integration.yml:30: update your workflow using https://app.stepsecurity.io/secureworkflow/gocardless/draupnir/build-integration.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build-integration.yml:38: update your workflow using https://app.stepsecurity.io/secureworkflow/gocardless/draupnir/build-integration.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build-integration.yml:39: update your workflow using https://app.stepsecurity.io/secureworkflow/gocardless/draupnir/build-integration.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build-integration.yml:51: update your workflow using https://app.stepsecurity.io/secureworkflow/gocardless/draupnir/build-integration.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build-integration.yml:52: update your workflow using https://app.stepsecurity.io/secureworkflow/gocardless/draupnir/build-integration.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build-integration.yml:61: update your workflow using https://app.stepsecurity.io/secureworkflow/gocardless/draupnir/build-integration.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build-integration.yml:62: update your workflow using https://app.stepsecurity.io/secureworkflow/gocardless/draupnir/build-integration.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/build-integration.yml:67: update your workflow using https://app.stepsecurity.io/secureworkflow/gocardless/draupnir/build-integration.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build-integration.yml:80: update your workflow using https://app.stepsecurity.io/secureworkflow/gocardless/draupnir/build-integration.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build-integration.yml:81: update your workflow using https://app.stepsecurity.io/secureworkflow/gocardless/draupnir/build-integration.yml/master?enable=pin","Warn: containerImage not pinned by hash: Dockerfile:1: pin your Docker image by updating ubuntu:22.04 to ubuntu:22.04@sha256:1aa979d85661c488ce030ac292876cf6ed04535d3a237e49f61542d8e5de5ae0","Info:   0 out of  11 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   2 third-party GitHubAction dependencies pinned","Info:   0 out of   1 containerImage dependencies pinned","Info:   1 out of   1 goCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"29 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-22f2-v57c-j9cx","Warn: Project is vulnerable to: GHSA-3h57-hmj3-gj3p","Warn: Project is vulnerable to: GHSA-54rr-7fvw-6x8f","Warn: Project is vulnerable to: GHSA-65f5-mfpf-vfhj","Warn: Project is vulnerable to: GHSA-7g2v-jj9q-g3rg","Warn: Project is vulnerable to: GHSA-7wqh-767x-r66v","Warn: Project is vulnerable to: GHSA-8cgq-6mh2-7j6v","Warn: Project is vulnerable to: GHSA-93pm-5p5f-3ghx","Warn: Project is vulnerable to: GHSA-c6qg-cjj8-47qp","Warn: Project is vulnerable to: GHSA-gjh7-p2fx-99vx","Warn: Project is vulnerable to: GHSA-rqv2-275x-2jq5","Warn: Project is vulnerable to: GHSA-vpfw-47h7-xj4g","Warn: Project is vulnerable to: GHSA-xj5v-6v4g-jfw6","Warn: Project is vulnerable to: GHSA-2rxp-v6pw-ch6m","Warn: Project is vulnerable to: GHSA-4xqq-m2hx-25v8","Warn: Project is vulnerable to: GHSA-5866-49gr-22v4","Warn: Project is vulnerable to: GHSA-r55c-59qm-vjw6","Warn: Project is vulnerable to: GHSA-vg3r-rm7w-2xgh","Warn: Project is vulnerable to: GHSA-vmwr-mc7x-5vc3","Warn: Project is vulnerable to: GO-2024-3321 / GHSA-v778-237x-gjrc","Warn: Project is vulnerable to: GO-2025-3487 / GHSA-hcg3-q754-cr77","Warn: Project is vulnerable to: GO-2023-1988 / GHSA-2wrh-6pvc-2jm9","Warn: Project is vulnerable to: GO-2023-2102 / GHSA-4374-p667-p6c8","Warn: Project is vulnerable to: GHSA-qppj-fm5r-hxr3","Warn: Project is vulnerable to: GO-2024-2687 / GHSA-4v7x-pqxf-cx7m","Warn: Project is vulnerable to: GO-2024-3333","Warn: Project is vulnerable to: GO-2025-3503 / GHSA-qxp5-gwg8-xv66","Warn: Project is vulnerable to: GO-2025-3595 / GHSA-vvgc-356p-c3xw","Warn: Project is vulnerable to: GO-2025-3488 / GHSA-6v2p-p543-phr9"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-19T03:51:35.398Z","repository_id":44454459,"created_at":"2025-08-19T03:51:35.398Z","updated_at":"2025-08-19T03:51:35.398Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271470280,"owners_count":24765363,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-21T02:00:08.990Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T15:08:29.016Z","updated_at":"2025-08-21T11:32:59.595Z","avatar_url":"https://github.com/gocardless.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"Draupnir\n========\n\nDraupnir is a tool that provides on-demand Postgres databases with preloaded data.\n\n\u003e *Odin laid upon the pyre the gold ring called Draupnir; this quality attended it: that every ninth night there fell from it eight gold rings of equal weight.*\n\nDevelopment\n-----------\n\nPrerequisites:\n- Go\n- Postgresql\n- Ruby\n\nCreate the database\n```\ncreatedb draupnir\n```\n\nMigrate the database\n```\nmake migrate\n```\n\nDevelopment (Vagrant VM)\n------------------------\n\n**!! Disclaimer, the vagrant VM is currently unsupported on Apple Silicon**\nIt will often be desirable to run a full virtual machine, with btrfs, in order\nto test the complete Draupnir flow. This can be achieved via the included\nVagrant configuration.\n\nInstall prerequisites:\n```\nbrew cask install virtualbox vagrant\n```\n\nBuild the Linux binary:\n```\nmake build-linux\n```\n\nBoot Vagrant VM:\n```\nvagrant up\n```\n\nLogin and use Draupnir:\n```\nvagrant ssh\n$ sudo su -\n# eval $(draupnir new)\n# psql -d myapp\n```\n\nAfter making changes to the code, to restart the server:\n```\nmake build-linux \u0026\u0026 vagrant up --provision\n```\n\nTests\n-----\n\nTo run the unit tests:\n```\nmake test\n```\n\nTo run the integration tests, ensure you've run `make build-linux` before running:\n```\nmake test-integration\n```\n\n# Releases\nFor releases, this project uses [GoReleaser](https://goreleaser.com/). The configuration was done in such a way that\nreleases happen on any commit to the main branch that also updates [DRAUPNIR_VERSION](./DRAUPNIR_VERSION), and should be\naccompanied by an update to [CHANGELOG.md](CHANGELOG.md) to make it explicit what has changed.\n\nVersion updates should follow the [Semantic Versioning](https://semver.org/) guidelines.\n\nUsage\n=====\n\nDraupnir provides an API to create, use and manage instances of your database.\nThere are two API resources: _Images_ and _Instances_. An Image is a database\nbackup that you upload to Draupnir. Instances are lightweight copies of a\nparticular Image that you can create, use and destroy with ease. We'll walk\nthrough the basics of using Draupnir. The full API reference is at the bottom of\nthis document.\n\n### Creating an Image\nCreate a new Image by `POST`ing to `/images`, providing a timestamp for the\nbackup and an anonymisation script that will be run against the backup. You can\nuse this to remove any sensitive data from your backup before serving it to\nusers.\n```http\nPOST /images HTTP/1.1\nContent-Type: application/json\nDraupnir-Version: 1.0.0\nAuthorization: Bearer 123\n\n{\n  \"data\": {\n    \"type\": \"images\",\n    \"attributes\": {\n      \"backed_up_at\": \"2017-05-01T12:00:00Z\",\n      \"anonymisation_script\": \"\\c my_db\\nDELETE FROM secret_tokens;\"\n    }\n  }\n}\n\n201 Created\n{\n  \"data\": {\n    \"type\": \"images\",\n    \"id\": 1,\n    \"attributes\": {\n      \"backed_up_at\": \"2017-05-01T12:00:00Z\",\n      \"created_at\": \"2017-05-01T15:00:00Z\",\n      \"updated_at\": \"2017-05-01T15:00:00Z\",\n      \"ready\": false\n    }\n  }\n}\n```\n\n### Uploading an Image\nOnce you've created an Image, you can upload it. This is done by `scp`ing a\ntarball of the database data directory to Draupnir. The upload is authenticated\nwith an ssh key which you'll create when setting up Draupnir.\n```\nscp -i key.pem db_backup.tar.gz upload@my-draupnir.tld:/draupnir/image_uploads/1\n```\n\nOnce you've uploaded the backup, inform Draupnir that you're ready to finalise\nthe image. This may take some time, as Draupnir will spin up Postgres and run\nthe anonymisation script.\n```http\nPOST /images/1/done HTTP/1.1\nContent-Type: application/json\nDraupnir-Version: 1.0.0\nAuthorization: Bearer 123\n\n200 OK\n{\n  ...\n}\n```\n\n### Creating Instances\nNow you've got an image, you can create instances of it. The process for this is\nvery simple.\n```http\nPOST /instances HTTP/1.1\nContent-Type: application/json\nDraupnir-Version: 1.0.0\nAuthorization: Bearer 123\n\n{\n  \"data\": {\n    \"type\": \"instances\",\n    \"attributes\": {\n      \"image_id\": 1\n    }\n  }\n}\n\n201 Created\n{\n  \"data\": {\n    \"type\": \"instances\",\n    \"id\": 1,\n    \"attributes\": {\n      \"created_at\": \"2017-05-01T16:00:00Z\",\n      \"updated_at\": \"2017-05-01T16:00:00Z\",\n      \"image_id\": 1,\n      \"port\": \"5678\"\n    }\n  }\n}\n```\n\nYou now have a Postgres server up and running, containing a copy of your\ndatabase. You can connect to it like you would any other database.\n```\nPGHOST=my-draupnir.tld PGPORT=5678 psql my-db\n```\n\nYou can make any modifications to this database and they won't affect the\noriginal backup. When you're done, just destroy the instance.\n```http\nDELETE /instances/1\nAuthorization: Bearer 123\nDraupnir-Version: 1.0.0\n\n204 No Content\n```\n\nYou can create as many instances of a particular image as you want, without\nworrying about disk space. Draupnir will only consume disk space for new data\nthat you write to your instances.\n\nConfiguration\n------------\n\nWhen draupnir boots it looks for a config file at `/etc/draupnir/config.toml`.\nThis file must specify all required configuration variables in order for\nDraupnir to boot. The variables are as follows:\n\n| Field                          | Required | Description\n|--------------------------------|----------|---------------------------------------|\n| `database_url`                 | True     | A postgresql [connection URI](https://www.postgresql.org/docs/9.5/static/libpq-connect.html#LIBPQ-CONNSTRING) for draupnir's internal database.\n| `data_path`                    | True     | The path to draupnir's data directory, where all images and instances will be stored.\n| `environment`                  | True     | The environment. This can be any value, but if it is set to \"test\", draupnir will use a stubbed authentication client which allows all requests specifying an access token of `the-integration-access-token`. This is intended for integration tests - don't use it in production. The environment will be included in all log messages.\n| `shared_secret`                | True     | A hardcoded access token that can be used by automated scripts which can't authenticate via OAuth. At GoCardless we use this to automatically create new images.\n| `trusted_user_email_domain`    | True     | The domain under which users are considered \"trusted\". This is draupnir's rudimentary form of authentication: if a user athenticates via OAuth and their email address is under this domain, they will be allowed to use the service. This domain must start with a `@`, e.g. `@gocardless.com`.\n| `public_hostname`              | True     | The hostname that will be set as PGHOST. This is configurable as it may be different to the hostname of the _API address_ that clients communicate with.\n| `sentry_dsn`                   | False    | The DSN for your [Sentry](https://sentry.io/) project, if you're using Sentry.\n| `clean_interval`               | True     | The interval at which Draupnir checks and removes any instance associated with a user that no longer has a valid refresh token. Valid values are a sequence of digits followed by a unit, such as \"30m\", \"6h\". See [time.ParseDuration](https://golang.org/pkg/time/#ParseDuration).\n| `min_instance_port`            | True     | The minimum port number (inclusive) that may be used when creating a Draupnir instance.\n| `max_instance_port`            | True     | The maximum port number (exclusive) that may be used when creating a Draupnir instance.\n| `enable_ip_whitelisting`       | False    | Whether to enable the [IP whitelisting module](#ip-address-whitelisting).\n| `whitelist_reconcile_interval` | False    | If IP whitelisting is enabled, this is the interval at which Draupnir reconciles the IP address whitelist with what's in iptables, in order to clean up incorrect state. Uses the same format as `clean_interval`.\n| `use_x_forwarded_for`          | False    | Whether to use the `X-Forwarded-For` header when determining the real user IP address. See [documentation](#identification-of-user-ip-addresses).\n| `trusted_proxy_cidrs`          | False    | A list of CIDRs that will match your load balancer IP addresses. Example: `[\"10.32.0.0/16\"]`. See [documentation](#identification-of-user-ip-addresses).\n| `http.listen_address`          | False    | The address and port that the HTTPS server will bind to.\n| `http.insecure_listen_address` | False    | The address and port that the HTTP server will bind to.\n| `http.tls_certificate`         | False    | The path to the TLS certificate file that the HTTPS server will use.\n| `http.tls_private_key`         | False    | The path to the TLS private key that the HTTPS server will use.\n| `oauth.redirect_url`           | True     | The redirect URL for the OAuth flow.\n| `oauth.client_id`              | True     | The OAuth client ID.\n| `oauth.client_secret`          | True     | The OAuth client secret.\n\nFor a complete example of this file, see `spec/fixtures/config.toml`.\n\nCLI\n---\n\nDraupnir ships as a single binary which can be used to run the server or use as a client\nto manage your instances.\n\nThe CLI has built-in help (`draupnir help`). For help on sub-commands, use an invocation\nlike `draupnir images help` instead of `draupnir help images`.\n\n#### Authenticate\n```\ndraupnir authenticate\n```\n\n#### List Images\n```\ndraupnir images list\n```\n\n#### Create an instance of Image 3\n```\ndraupnir instances create 3\n```\n\n#### Connect to instance 4\n```\neval $(draupnir env 4)\npsql\n```\n\n#### Destroy instance 4\n```\ndraupnir instances destroy 4\n```\n\nAPI\n===\n\nThe Draupnir API roughly follows the JSON API spec, with a few deviations.\nThe only supported `Content-Type` is `application/json`. Authentication is\nrequired for most API endpoints and is provided in the form of an access token\nin the `Authorization` header.\n\nThe API also requires a `Draupnir-Version` header to be set. This version must\nbe exactly equal to the version of Draupnir serving the API. The CLI and server\nare distributed as one, and share a version number. We enforce equality here as\na conservative measure to ensure that the CLI and API can interoperate\nseamlessly. In the future we might relax this constraint.\n\n### Images\n#### List Images\n```http\nGET /images HTTP/1.1\nContent-Type: application/json\nDraupnir-Version: 1.0.0\nAuthorization: Bearer 123\n\n200 OK\n{\n  \"data\": [\n    {\n      \"type\": \"images\",\n      \"attributes\": {\n        \"backed_up_at\": \"2017-05-01T12:00:00Z\",\n        \"anonymisation_script\": \"\\c my_db\\nDELETE FROM secret_tokens;\"\n      }\n    }\n  ]\n}\n```\n\n#### Get Image\n```http\nGET /images/1 HTTP/1.1\nContent-Type: application/json\nDraupnir-Version: 1.0.0\nAuthorization: Bearer 123\n\n200 OK\n{\n  \"data\": {\n    \"type\": \"images\",\n    \"attributes\": {\n      \"backed_up_at\": \"2017-05-01T12:00:00Z\",\n      \"anonymisation_script\": \"\\c my_db\\nDELETE FROM secret_tokens;\"\n    }\n  }\n}\n```\n\n#### Create Image\n```http\nPOST /images HTTP/1.1\nContent-Type: application/json\nDraupnir-Version: 1.0.0\nAuthorization: Bearer 123\n\n{\n  \"data\": {\n    \"type\": \"images\",\n    \"attributes\": {\n      \"backed_up_at\": \"2017-05-01T12:00:00Z\",\n      \"anonymisation_script\": \"\\c my_db\\nDELETE FROM secret_tokens;\"\n    }\n  }\n}\n\n201 Created\n{\n  \"data\": {\n    \"type\": \"images\",\n    \"id\": 1,\n    \"attributes\": {\n      \"backed_up_at\": \"2017-05-01T12:00:00Z\",\n      \"created_at\": \"2017-05-01T15:00:00Z\",\n      \"updated_at\": \"2017-05-01T15:00:00Z\",\n      \"ready\": false\n    }\n  }\n}\n```\n\n#### Finalise Image\n```http\nPOST /images/1/done HTTP/1.1\nContent-Type: application/json\nDraupnir-Version: 1.0.0\nAuthorization: Bearer 123\n\n200 OK\n{\n  \"data\": {\n    \"type\": \"images\",\n    \"id\": 1,\n    \"attributes\": {\n      \"backed_up_at\": \"2017-05-01T12:00:00Z\",\n      \"created_at\": \"2017-05-01T15:00:00Z\",\n      \"updated_at\": \"2017-05-01T15:01:00Z\",\n      \"ready\": true\n    }\n  }\n}\n```\n\n#### Destroy Image\n```http\nDELETE /images/1\nAuthorization: Bearer 123\n\n204 No Content\n```\n\n### Instances\n#### List Instances\n```http\nGET /instances HTTP/1.1\nContent-Type: application/json\nDraupnir-Version: 1.0.0\nAuthorization: Bearer 123\n\n200 Ok\n{\n  \"data\": [\n    {\n      \"type\": \"instances\",\n      \"id\": 1,\n      \"attributes\": {\n        \"created_at\": \"2017-05-01T16:00:00Z\",\n        \"updated_at\": \"2017-05-01T16:00:00Z\",\n        \"image_id\": 1,\n        \"port\": \"5678\"\n      }\n    }\n  ]\n}\n```\n\n#### Get Instance\n```http\nGET /instances HTTP/1.1\nContent-Type: application/json\nDraupnir-Version: 1.0.0\nAuthorization: Bearer 123\n\n200 Ok\n{\n  \"data\": {\n    \"type\": \"instances\",\n    \"id\": 1,\n    \"attributes\": {\n      \"created_at\": \"2017-05-01T16:00:00Z\",\n      \"updated_at\": \"2017-05-01T16:00:00Z\",\n      \"image_id\": 1,\n      \"port\": \"5678\"\n    }\n  }\n}\n```\n\n#### Create Instance\n```http\nPOST /instances HTTP/1.1\nContent-Type: application/json\nDraupnir-Version: 1.0.0\nAuthorization: Bearer 123\n\n{\n  \"data\": {\n    \"type\": \"instances\",\n    \"attributes\": {\n      \"image_id\": 1\n    }\n  }\n}\n\n201 Created\n{\n  \"data\": {\n    \"type\": \"instances\",\n    \"id\": 1,\n    \"attributes\": {\n      \"created_at\": \"2017-05-01T16:00:00Z\",\n      \"updated_at\": \"2017-05-01T16:00:00Z\",\n      \"image_id\": 1,\n      \"port\": \"5678\"\n    }\n  }\n}\n```\n\n#### Destroy Instance\n```\nDELETE /instances/1 HTTP/1.1\nDraupnir-Version: 1.0.0\nAuthorization: Bearer 123\n\n204 No Content\n```\n\n# Internal Architecture\n\nDraupnir is basically two things: a manager for [BTRFS](https://btrfs.wiki.kernel.org/index.php/Main_Page)\nvolumes and a supervisor of PostgreSQL processes.\nEach image is stored in its own BTRFS subvolume, and instances are created by\ncreating a snapshot of the image's subvolume, and booting a Postgres instance in\nit. In order to do this, Draupnir requires read-write access to a disk formatted\nwith BTRFS. The path to this disk is specified at runtime by the `DRAUPNIR_DATA_PATH` environment variable.\nThe whole process looks like this (assuming `DRAUPNIR_DATA_PATH=/draupnir`):\n\n1. An image is created via the API (`POST /images`). This creates a record in Draupnir's\n   internal database and an empty subvolume is created at\n   `/draupnir/image_uploads/1` (where `1` is the image ID). The user may specify\n   an anonymisation script to be run on the data before it is made available. At\n   this point, the image is marked as \"not ready\", meaning it cannot be used to\n   create instances.\n2. A PostgreSQL backup, in the form of a tar file, is pushed into the server\n   over SCP. The ssh credentials for this operation are set when the machine is\n   provisioned, via the [chef cookbook](https://github.com/gocardless/chef-draupnir).\n   The backup is pushed directly into `/draupnir/image_uploads/1`.\n3. The image is finalised via the API (`POST /images/1/done`). This indicates to Draupnir that the\n   backup has completed and no more data needs to be pushed. Draupnir prepares\n   the directory so Postgres will boot from it, and runs the anonymisation\n   script. For more detail on this step see `cmd/draupnir-finalise-image`.\n   Finally, Draupnir will create a BTRFS snapshot of the subvolume at\n   `/draupnir/image_snapshots/1`. This snapshot is read-only and ensures that the image\n   will not change from now on. At this point Draupnir marks the image as\n   \"ready\", meaning that instances can be created from it.\n4. A user creates an instance from this image via the API (`POST /instances`).\n   First, draupnir creates a corresponding record in its database. Then it will\n   take a further snapshot of the image: `/draupnir/image_snapshots/1 -\u003e\n   /draupnir/instances/1` (where `1` is the instance ID). It will start a\n   Postgres process, setting the data directory to `/draupnir/instances/1` and\n   binding it to a random port (which we persist in the database as part of the\n   instance).\n5. The instance is now running and can accept external connections (the port\n   range used for instances is exposed via an iptables rule in the cookbook).\n   The user can connect to the instance as if it were any other database, simply\n   by specifying the host (whatever server Draupnir is running on), the port\n   (serialised in the API) and valid user credentials.  We expect that the user\n   already knows the credentials for a user in their database, or alternatively\n   they can use the `postgres` user which we create (with no password) as part\n   of step 3.\n6. The user destroys the instance via the API (`DELETE /instances/1`). Draupnir\n   stops the Postgres process for that instance and deletes the snapshot\n   `/draupnir/instances/1`.\n7. The image is destroyed via an API call (`DELETE /images`). All instances of\n   this image are destroyed as per step 6, and then the image is destroyed by\n   removing the directories `/draupnir/image_snapshots/1` and\n   `/draupnir/image_uploads/1`.\n\nAll interaction with BTFS and Postgres is done via a collection of small shell\nscripts in the `cmd` directory - read them if you want to know more.\n\nRight now modifications to images (creation, finalisation, deletion) are\nrestricted to a single \"upload\" user, who authenticates with the API via a\nshared secret.\n\n## Security model\n\nDraupnir has been designed to be deployed on a publicly-accessible instance, but\nrestrict access to the potentially sensitive data in the Draupnir images to\nauthorised users only.\n\n### API access\n\nAccess to the API is secured via Google OAuth. A user must have a valid token in\norder to create, retrieve or destroy a Draupnir instance.\n\n### Connecting to Draupnir Postgres instances\n\nAccess to a Draupnir Postgres instance is secured via a client-authenticated TLS\nconnection.\nThe client certificate and key are served via the API and then\nstored in a secure temporary location on the client machine. The paths to these\nfiles are then used to set `PGSSLCERT` and `PGSSLKEY`.\n\nAdditionally, `PGSSLMODE` is set to `verify-ca`, meaning that the Postgres\nclient will *only* attempt to connect via TLS, and will also only successfully\nconnect to the instance if it provides the expected CA certificate.\n\nOn the server side, when an image is finalised the `pg_hba.conf` file is setup\nso that the only method of access is client-authenticated TLS, and this\ntherefore propagates to every instance created from the image.\nThis property ensures that even if a user was to login as a Postgres superuser\non their instance and set a blank password for a given database user, then still\nnobody would be able to connect without a valid client certificate and key.\n\nEach instance has a unique CA, server and client certificate, all generated at\ncreation time, meaning that certificates and keys cannot be reused across\ninstances and that once the instance is created the locally-stored credentials\nare useless.\nGiven that an instance's details (and therefore credentials) can only\nbe retrieved by the user that created that instance, it also means that only the\nowning user has access to connect to the instance.\n\n### IP address whitelisting\n\nDraupnir provides the ability to dynamically whitelist user IP addresses\nto further secure the Postgres instances that it creates, protecting them from\nautomated scans and attacks.\n\nThis is achieved via iptables rules. If this component is enabled\n(`enable_ip_whitelisting = true` in the server config) then the Draupnir\ndaemon will maintain an iptables chain named `DRAUPNIR-WHITELIST`. It is\ntherefore **the administrator's responsibility** to provision other iptables\nrules that reference this chain.\n\nWhen a user creates an instance, or retrieves the details of one of their own\ninstances, this chain will be populated with a rule that allows _new\nconnections_ to the Postgres instance port, from their IP address only. The rule\nwill be removed as soon as the instance is destroyed.\n\nAn example configuration is provided below. This will work in configurations\nwhere the default policy of the `INPUT` chain is `ACCEPT`. For those with\ndefaults of `DROP`, the third rule can be omitted.\n\n```\n# In this example, our Draupnir port range is 6432-7432.\n\n# Setup the DRAUPNIR-WHITELIST chain. The server will create this itself if it's\n# missing, but we  won't be able to reference the chain unless we do this.\niptables -N DRAUPNIR-WHITELIST\n\n# Allow all connections on the loopback interface\niptables -A INPUT -i lo -p tcp -m tcp --dport 6432:7432 -j ACCEPT\n\n# For any connections which have been successfully opened, allow further\n# communication.\niptables -A INPUT -p tcp -m tcp --dport 6432:7432 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT\n\n# For any new connections, pass them through to the DRAUPNIR-WHITELIST chain\niptables -A INPUT -p tcp -m tcp --dport 6432:7432 -m conntrack --ctstate NEW -j DRAUPNIR-WHITELIST\n\n# For any connections that have not been accepted by the whitelist, drop the\n# packet\niptables -A INPUT -p tcp -m tcp --dport 6432:7432 -j DROP\n```\n\nThe iptables wrapper library used in this project requires root access, and [does\nnot support sudo](https://github.com/coreos/go-iptables/issues/55). Because it\nis strongly recommended to _not_ run the Draupnir server as the root user, this\ncan be worked around by using the provided [wrapper script](./scripts/iptables)\nwhich is installed into the `/usr/lib/draupnir/bin` directory by the Debian\npackage.\nThe Draupnir server process must be executed with a `PATH` variable that places\nthis directory at the beginning, in order to ensure it is used instead of the\nreal `iptables` binary.\n\n#### Identification of user IP addresses\n\nThe Draupnir server creates whitelist rules based on the IP address of the\nuser, which it determines by inspecting the HTTP request that was made to its\nAPI.\n\nIf your Draupnir API server is fronted by a load balancer, then the HTTP\nconnection that the Draupnir server receives will originate from that, rather\nthan the user directly. In this instance a separate mechanism of determining the\nuser's IP address must be employed; the `X-Forwarded-For` header.\n\nIf this scenario applies to you then the following steps must be taken:\n1. Ensure that your load balancer places the 'real' user IP address in the\n   `X-Forwarded-For` header.\n2. Enable the use of the `X-Forwarded-For` header for IP address identification\n   by setting the `use_x_forwarded_for` variable to `true`.\n3. Define a list of trusted proxies, via the `trusted_proxy_cidrs` setting.\n   Any IP addresses in the `X-Forwarded-For` header that match any of these\n   CIDRs will be ignored.\n   The real user IP address is then determined by taking the resulting list of\n   elements of the `X-Forwarded-For` header and using the last one (under the\n   assumption that this is the one that your load balancer has added).\n\nIf you are not using a load balancer then it is imperative that the\n`use_x_forwarded_for` setting remains disabled. If it is enabled without a load\nbalancer present, rewriting the contents of the header, then it's possible for\nan authenticated user to send API requests with a fabricated `X-Forwarded-For`\nheader and therefore open up their instance(s) to unauthorized IP addresses.\n\n### Cleanup of revoked user instances\n\nWhen a user creates an instance Draupnir stores the user's refresh token so that\nit can, at the `clean_interval`, check that the refresh token is still valid.\nIn the event that the token isn't valid, the instance is deleted. This ensures\nthat instances don't remain available longer than the users have access to\nDraupnir.\n\nCommon causes for an invalid refresh token are:\n- The user has revoked the application's third-party access in the Google\n  account dashboard.\n- The user is suspended via G Suite.\n- The user has been deleted.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgocardless%2Fdraupnir","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgocardless%2Fdraupnir","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgocardless%2Fdraupnir/lists"}