{"id":13650845,"url":"https://github.com/myENA/consul-backinator","last_synced_at":"2025-04-22T18:32:58.383Z","repository":{"id":42017786,"uuid":"43525589","full_name":"myENA/consul-backinator","owner":"myENA","description":"Command line Consul backup and restore utility supporting KVs, ACLs and Queries","archived":false,"fork":false,"pushed_at":"2022-12-24T01:23:15.000Z","size":2358,"stargazers_count":226,"open_issues_count":3,"forks_count":22,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-10-04T00:45:15.346Z","etag":null,"topics":["backup","consul","consul-acl-backup","consul-backup","consul-kv-backup","consul-query-backup","docker","encrypted-backup","nomad"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/myENA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-10-01T22:59:18.000Z","updated_at":"2023-11-30T06:51:14.000Z","dependencies_parsed_at":"2023-01-30T19:50:15.180Z","dependency_job_id":null,"html_url":"https://github.com/myENA/consul-backinator","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myENA%2Fconsul-backinator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myENA%2Fconsul-backinator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myENA%2Fconsul-backinator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myENA%2Fconsul-backinator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/myENA","download_url":"https://codeload.github.com/myENA/consul-backinator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223472814,"owners_count":17150745,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","consul","consul-acl-backup","consul-backup","consul-kv-backup","consul-query-backup","docker","encrypted-backup","nomad"],"created_at":"2024-08-02T02:00:41.559Z","updated_at":"2024-11-10T01:31:16.529Z","avatar_url":"https://github.com/myENA.png","language":"Go","funding_links":[],"categories":["Infrastructure setup","Projects","Go"],"sub_categories":["Tools and Utilities","Backup and Restore"],"readme":"[![Mozilla Public License](https://img.shields.io/badge/license-MPL-blue.svg)](https://www.mozilla.org/MPL)\n[![Go Report Card](https://goreportcard.com/badge/github.com/myENA/consul-backinator)](https://goreportcard.com/report/github.com/myENA/consul-backinator)\n[![GoDoc](https://godoc.org/github.com/myENA/consul-backinator/common?status.svg)](https://godoc.org/github.com/myENA/consul-backinator/common)\n[![Build Status](https://github.com/myENA/consul-backinator/workflows/Build/badge.svg)](https://github.com/myENA/consul-backinator/actions?query=workflow%3ABuild)\n[![Downloads](https://img.shields.io/github/downloads/myENA/consul-backinator/total.svg)](https://github.com/myENA/consul-backinator/releases)\n[![Docker Pulls](https://img.shields.io/docker/pulls/myena/consul-backinator.svg)](https://hub.docker.com/r/myena/consul-backinator)\n[![Docker Automated Build](https://img.shields.io/docker/automated/myena/consul-backinator.svg)](https://hub.docker.com/r/myena/consul-backinator)\n[![Gitter Chat](https://badges.gitter.im/consul-backinator/Lobby.svg)](https://gitter.im/consul-backinator/Lobby?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\n# consul-backinator\n\n## Summary\n\nFlexible Consul KV pair backup and restore tool with a few unique features\nincluding ACL token and prepared query backup and restoration.\n\n## Key Features\n\n* Written in Golang using the official Consul API\n* No limits on the number of keys that can be backed up or restored\n* Backup files are written as gzip compressed and AES256 encrypted JSON data\n* Data integrity validation via HMAC-SHA256 signature of the raw data\n* Optional path transformation (path replacement) on key backup and/or restore\n* Clean well documented code that's simple to follow\n* Direct AWS/S3 support for backup and restoration of KVs, ACLs and queries\n* Node auto discovery in cloud environments via [go-discover](https://github.com/hashicorp/go-discover)\n\n## Installing\n\nOS X/macOS Homebrew users ...\n\n```\nbrew install consul-backinator\n```\n\nUsers with a proper Go environment (1.8+ required) ...\n\n```\ngo get -u github.com/myENA/consul-backinator\n```\n\nDevelopers that wish to take advantage of vendoring and other options ...\n\n```\ngit clone https://github.com/myENA/consul-backinator.git\ncd consul-backinator\nmake\n```\n\nTo build as a Docker container ...\n\n```\ngit clone https://github.com/myENA/consul-backinator.git\ncd consul-backinator\nmake docker\n```\n\nTo use the latest container from Docker Hub ...\n\n```\ndocker pull myena/consul-backinator\ndocker run myena/consul-backinator\n```\nSee [DOCKER.md](DOCKER.md) for some Docker use cases.\n\nTo run on Nomad ...\n\n```\ngit clone https://github.com/myENA/consul-backinator.git\ncd consul-backinator\n```\n\nEdit the job specification file `consul-backinator.nomad` to suit your environment. It uses S3 by default and must be\nconfigured with the correct bucket URI, access key, and secret key.  It's recommended to use a dedicated\nconsul-backinator user with IAM permissions to just this bucket for security purposes.  The job runs every\n15 minutes by default and logs to STDERR and STDOUT.\n\n## Security\n\nReleases after 1.4 will be accompanied by a GPG signed SHA256SUM file.\n\nReleases from 1.6.6 will use this GPG key:\n\n````\nEducation Networks of America (ENA github code signing key) \u003crd@ena.com\u003e\n````\n\nThe public key [720E001BD902187F83763C62C924F0AC9DA36FEB](https://pgp.key-server.io/search/0x720E001BD902187F83763C62C924F0AC9DA36FEB)\nis available via public servers and [gpg/rd@ena.com.asc](gpg/rd@ena.com.asc) in this repository.\n\n\nFor previous releases, they are signed by this key:\n````\nENA R\u0026D Team (ENA R\u0026D code signing key) \u003cr\u0026d@ena.com\u003e 73F17750\n````\n\nThis old key is still available at [62859FAD5BAEA13C3839D5053CA59EE673F17750](http://pgp.key-server.io/search/0x62859FAD5BAEA13C3839D5053CA59EE673F17750)\nvia public servers.\n\nWhen verifying releases please note that only the SHA256SUM file is signed with the GPG key.\nThe following process should be more than satisfactory to verify the authenticity of any release.\n\n### Verifying Releases\n\nUsing the key downloaded from the public server or contained in this repository you\nshould import the gpg key into your local key store.\n\n```\ngpg --import \"rd@ena.com.asc\"\n```\n\nNext, with the release archive, shasum and signature file downloaded you should\nverify integrity of the checksum file using the GPG signature.\n\n```\ngpg --verify consul-backinator-1.6.6-SHA256SUMS.sig consul-backinator-1.6.6-SHA256SUMS\n```\n\nFinally, ensure the downloaded archive matches the verified checksums file.\nOnly downloaded archives in the current directory will be checked.\n\n```\nshasum -a 256 -c consul-backinator-1.5-SHA256SUMS\n```\n\nIf the above steps completed without error you have a verified release!\n\n## Usage\n\n### Summary\n\n```\nahurt$ ./consul-backinator --help\nusage: consul-backinator [--version] [--help] \u003ccommand\u003e [\u003cargs\u003e]\n\nAvailable commands are:\n    backup     Perform a backup operation\n    dump       Dump a backup file\n    restore    Perform a restore operation\n\n```\n\n### Backup Options\n\n| Option      | Description |\n|-------------|-------------|\n| `file`      | The backup file target.  The signature will be the same with a `.sig` extension appended.  The default names are `consul.bak` and `consul.bak.sig`\n| `key`       | The passphrase used for data encryption and signature generation.  The default string `password` will be used if none specified.  This should be a secure pseudo random string.\n| `nokv`      | Do not attempt to backup kv data.  This only makes sense if also passing the `acls` and/or `queries` option below.\n| `acls`      | Optional backup filename or S3 location for acl tokens.\n| `queries`   | Optional backup filename or S3 location for prepared queries.\n| `transform` | Optional argument that affects the key paths written to the backup file.  See the transformation notes below for more information.\n| `prefix`    | Optional argument that specifies the starting point for the backup tree.  The default prefix is the root `/` prefix.  To perform a partial tree backup specify a prefix.\n\n### Restore Options\n\n| Option    | Description |\n|-----------|-------------|\n| `file`    | The source file. The default is `consul.bak`\n| `key`     | The passphrase used for data decryption and signature validation.  This must match the key used when the backup was created.\n| `nokv`    | Do not attempt to restore kv data.  This only makes sense if also passing the `acls` option below.\n| `acls`    | Optional source filename or S3 location for acl tokens.\n| `queries` | Optional source filename or S3 location for query definitions.\n| `delete`  | Optionally delete all keys under the specified prefix prior to restoring the backup file.  The default is false.\n| `prefix`  | The prefix with the `delete` option.  The default is `/` root.  __THIS WILL DELETE ALL DATA IN YOUR KEYSTORE__ if not changed when using `-delete`.\n\n### Shared Consul Options (backup/restore)\n\n| Option   | Description |\n|----------|-------------|\n| `addr`            | Optional consul agent address and port.  The default is read from the `CONSUL_HTTP_ADDR` environment variable if specified or set to `127.0.0.1:8500`.\n| `scheme`          | Optional scheme `http` or `https` used when connecting to the consul agent.  The default is set to `https` if the `CONSUL_HTTP_SSL` environment variable is set to `true` otherwise the default is `http`.\n| `dc`              | Optional datacenter specification.  The default value is the datacenter of the agent to which you are connecting.\n| `token`           | Optional consul access token.  The default value is read from the `CONSUL_HTTP_TOKEN` environment variable if specified.\n| `ca-cert`         | Optional path to a PEM encoded CA cert file.  This may also be a certificate bundle (concatenation of CA certificates).\n| `client-cert`     | Optional path to a PEM encoded client certificate.  This certificate must match the client key.\n| `client-key`      | Optional path to an unencrypted PEM encoded private key. This key should obviously match the client cert.  Passing this or `client-cert` alone will probably not work.\n| `tls-skip-verify` | Optional bool for verifying a TLS certificate.  This is a very clear security risk and is not reccomended.  This option alone has the same affect as the `CONSUL_HTTP_SSL_VERIFY` environment variable.\n\n### Dump Options\n\n| Option    | Description |\n|-----------|-------------|\n| `file`    | The source file.  The default `consul.bak` will be used if not specified.\n| `key`     | The passphrase for the backup file to be dumped.  The default is `password` if not passed.\n| `plain`   | Decrypt and dump the full raw payload contained within the backup file.\n| `acls`    | Dump a limited set of data in a more concise format than the `plain` option above.  This is only relevant for ACL backup files.\n| `queries` | Dump a limited set of data in a more concise format than the `plain` option above.  This is only relevant for query backup files. \n\n## Transformations\n\nTransformations are simple string operations and will affect the path anywhere\nthere is a match.  For example, passing `-transform=\"foo,bar\"` would rewrite\n`/apple/foo/key` =\u003e `/apple/bar/key` as well as `/orange/thing/foo/key` =\u003e `/orange/thing/bar/key`.\nTo avoid potential errors in transformations you should always use the most exact path possible.\nUsing the previous example if you only wanted to affect keys under `apple` you should pass\n`-transform=\"apple/foo,apple/bar\"` to prevent other paths from being modified inadvertently.\n\n## S3 Support\n\nSupport for S3 is implemented by passing an S3 URI to the standard ```-file``` option.  The full format for the URI is as follows:\n\n```\ns3://access-key:secret-key@my-bucket/path/to/object?region=us-east-1\u0026endpoint=my-s3gw:9000\u0026secure=false\n```\n\nThe minimal URI when using environment variables would be: ```s3://my-bucket/path/object```\n\nThe current URI parsing accepts `s3://` and `s3n://` scheme prefixes.\n\nThis table describes all the S3 URI options and corresponding environment variables.\n\n| Paramater    | Environment             | Required    | Description             | Default          |\n|--------------|-------------------------|-------------|-------------------------|------------------|\n| `access-key` | `AWS_ACCESS_KEY_ID`     | yes         | Your S3/AWS access key  |                  |\n| `secret-key` | `AWS_SECRET_ACCESS_KEY` | yes         | Your S3/AWS secret key  |                  |\n| `region`     | `AWS_REGION`            | yes         | Your S3/AWS region      |                  |\n| `endpoint`   |                         | no          | Optional endpoint       | s3.amazonaws.com |\n| `secure`     |                         | no          | Optional secure flag    | true             |\n| `pathstyle`  |                         | no          | Optional pathstyle flag | false            |\n\n## Example\n\n```\nahurt$ ./consul-backinator backup -key=superSecretStuff\n2016/05/09 17:14:11 [Success] Backed up 289 keys from / to consul.bak\nKeep your backup (consul.bak) and signature (consul.bak.sig) files in a safe place.\nYou will need both to restore your data.\n```\n\n```\nahurt$ ls -la *.sig *.bak\n-rw-------  1 ahurt  staff  11167 May  9 17:14 consul.bak\n-rw-------  1 ahurt  staff     44 May  9 17:14 consul.bak.sig\n```\n\n## Thanks\n\n* The [HashiCorp](https://github.com/hashicorp) folks for the excellent Consul service discovery daemon and the\nexcellent embedded API and golang package.  In addition to some very nice references back to this utility in a few issues.\n* [Adam Avilla](https://github.com/hekaldama) for his time and contributions to the Docker portions of the build.\n* All the other contributors, testers and stargazers.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FmyENA%2Fconsul-backinator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FmyENA%2Fconsul-backinator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FmyENA%2Fconsul-backinator/lists"}