{"id":13645985,"url":"https://github.com/hashicorp/consul-replicate","last_synced_at":"2025-05-15T12:04:52.817Z","repository":{"id":17856491,"uuid":"20779425","full_name":"hashicorp/consul-replicate","owner":"hashicorp","description":"Consul cross-DC KV replication daemon.","archived":false,"fork":false,"pushed_at":"2025-03-26T09:20:58.000Z","size":1560,"stargazers_count":517,"open_issues_count":29,"forks_count":70,"subscribers_count":60,"default_branch":"master","last_synced_at":"2025-04-14T22:18:06.897Z","etag":null,"topics":["consul","go","replication"],"latest_commit_sha":null,"homepage":"https://www.hashicorp.com/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hashicorp.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2014-06-12T19:02:46.000Z","updated_at":"2025-03-26T09:20:41.000Z","dependencies_parsed_at":"2023-02-17T08:31:15.693Z","dependency_job_id":"f886858e-8e99-4792-865d-606a4454614f","html_url":"https://github.com/hashicorp/consul-replicate","commit_stats":{"total_commits":146,"total_committers":25,"mean_commits":5.84,"dds":0.4246575342465754,"last_synced_commit":"e06f7c1623307fa58d5917dd390d5fca6fe9a91d"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashicorp%2Fconsul-replicate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashicorp%2Fconsul-replicate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashicorp%2Fconsul-replicate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashicorp%2Fconsul-replicate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hashicorp","download_url":"https://codeload.github.com/hashicorp/consul-replicate/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254337612,"owners_count":22054253,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consul","go","replication"],"created_at":"2024-08-02T01:02:46.268Z","updated_at":"2025-05-15T12:04:47.780Z","avatar_url":"https://github.com/hashicorp.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Consul Replicate\n\n![Build Status](https://github.com/hashicorp/consul-replicate/actions/workflows/lint-and-test.yml/badge.svg)\n\nThis project provides a convenient way to replicate values from one\n[Consul][consul] datacenter to another using the `consul-replicate` daemon.\n\nThe daemon `consul-replicate` integrates with [Consul][consul] to perform\ncross-data-center K/V replication. This makes it possible to manage application\nconfiguration from a central data center, with low-latency asynchronous\nreplication to other data centers, thus avoiding the need for smart clients that\nwould need to write to all data centers and queue writes to handle network\nfailures.\n\n---\n\n**The documentation in this README corresponds to the master branch of Consul Replicate. It may contain unreleased features or different APIs than the most recently released version.**\n\n**Please see the [Git tag](https://github.com/hashicorp/consul-replicate/releases) that corresponds to your version of Consul Replicate for the proper documentation.**\n\n---\n\n\n## Installation\n\n1. Download a pre-compiled, released version from the [Consul Replicate releases page][releases].\n\n1. Extract the binary using `unzip` or `tar`.\n\n1. Move the binary into `$PATH`.\n\nTo compile from source, please see the instructions in the\n[contributing section](#contributing).\n\n## Usage\n\nFor the full list of options:\n\n```shell\n$ consul-replicate -h\n```\n\n### Command Line Flags\n\nThe CLI interface supports all options in the configuration file and visa-versa.\nHere are a few examples of common integrations on the command line.\n\nReplicate all keys under \"global\" from the nyc1 data center:\n\n```sh\n$ consul-replicate \\\n  -prefix \"global@nyc1\"\n```\n\nReplicate all keys under \"global\" from the nyc1 data center, renaming the key to\n\"default\" in the replicated stores:\n\n```sh\n$ consul-replicate \\\n  -prefix \"global@nyc1:default\"\n```\n\nReplicate all keys under \"global\" from the nyc1 data center, but do not poll or\nwatch for changes (just do it one time):\n\n```sh\n$ consul-replicate \\\n  -prefix \"global@nyc1\" \\\n  -once\n```\n\nReplicate all keys under \"global\" from the nyc1 data center, but exclude the\nglobal/private prefix:\n\n```sh\n$ consul-replicate \\\n  -prefix \"global@nyc1\" \\\n  -exclude \"global/private\" \\\n  -once\n```\n\n### Configuration File Format\n\nConfiguration files are written in the [HashiCorp Configuration Language][hcl].\nBy proxy, this means the configuration is also JSON compatible.\n\n```hcl\n# This denotes the start of the configuration section for Consul. All values\n# contained in this section pertain to Consul.\nconsul {\n  # This block specifies the basic authentication information to pass with the\n  # request. For more information on authentication, please see the Consul\n  # documentation.\n  auth {\n    enabled  = true\n    username = \"test\"\n    password = \"test\"\n  }\n\n  # This is the address of the Consul agent. By default, this is\n  # 127.0.0.1:8500, which is the default bind and port for a local Consul\n  # agent. It is not recommended that you communicate directly with a Consul\n  # server, and instead communicate with the local Consul agent. There are many\n  # reasons for this, most importantly the Consul agent is able to multiplex\n  # connections to the Consul server and reduce the number of open HTTP\n  # connections. Additionally, it provides a \"well-known\" IP address for which\n  # clients can connect.\n  address = \"127.0.0.1:8500\"\n\n  # This is the ACL token to use when connecting to Consul. If you did not\n  # enable ACLs on your Consul cluster, you do not need to set this option.\n  #\n  # This option is also available via the environment variable CONSUL_TOKEN.\n  token = \"abcd1234\"\n\n  # This controls the retry behavior when an error is returned from Consul.\n  # Consul Replicate is highly fault tolerant, meaning it does not exit in the\n  # face of failure. Instead, it uses exponential back-off and retry functions\n  # to wait for the cluster to become available, as is customary in distributed\n  # systems.\n  retry {\n    # This enabled retries. Retries are enabled by default, so this is\n    # redundant.\n    enabled = true\n\n    # This specifies the number of attempts to make before giving up. Each\n    # attempt adds the exponential backoff sleep time. Setting this to\n    # zero will implement an unlimited number of retries.\n    attempts = 12\n\n    # This is the base amount of time to sleep between retry attempts. Each\n    # retry sleeps for an exponent of 2 longer than this base. For 5 retries,\n    # the sleep times would be: 250ms, 500ms, 1s, 2s, then 4s.\n    backoff = \"250ms\"\n\n    # This is the maximum amount of time to sleep between retry attempts.\n    # When max_backoff is set to zero, there is no upper limit to the\n    # exponential sleep between retry attempts.\n    # If max_backoff is set to 10s and backoff is set to 1s, sleep times\n    # would be: 1s, 2s, 4s, 8s, 10s, 10s, ...\n    max_backoff = \"1m\"\n  }\n\n  # This block configures the SSL options for connecting to the Consul server.\n  ssl {\n    # This enables SSL. Specifying any option for SSL will also enable it.\n    enabled = true\n\n    # This enables SSL peer verification. The default value is \"true\", which\n    # will check the global CA chain to make sure the given certificates are\n    # valid. If you are using a self-signed certificate that you have not added\n    # to the CA chain, you may want to disable SSL verification. However, please\n    # understand this is a potential security vulnerability.\n    verify = false\n\n    # This is the path to the certificate to use to authenticate. If just a\n    # certificate is provided, it is assumed to contain both the certificate and\n    # the key to convert to an X509 certificate. If both the certificate and\n    # key are specified, Consul Replicate will automatically combine them into an\n    # X509 certificate for you.\n    cert = \"/path/to/client/cert\"\n    key  = \"/path/to/client/key\"\n\n    # This is the path to the certificate authority to use as a CA. This is\n    # useful for self-signed certificates or for organizations using their own\n    # internal certificate authority.\n    ca_cert = \"/path/to/ca\"\n\n    # This is the path to a directory of PEM-encoded CA cert files. If both\n    # `ca_cert` and `ca_path` is specified, `ca_cert` is preferred.\n    ca_path = \"path/to/certs/\"\n\n    # This sets the SNI server name to use for validation.\n    server_name = \"my-server.com\"\n  }\n}\n\n# This is the list of keys to exclude if they are found in the prefix. This can\n# be specified multiple times to exclude multiple keys from replication.\nexclude {\n  source = \"my-key\"\n}\n\n# This is the signal to listen for to trigger a graceful stop. The default value\n# is shown below. Setting this value to the empty string will cause Consul\n# Replicate to not listen for any graceful stop signals.\nkill_signal = \"SIGINT\"\n\n# This is the log level. If you find a bug in Consul Replicate, please enable\n# debug logs so we can help identify the issue. This is also available as a\n# command line flag.\nlog_level = \"warn\"\n\n# This is the maximum interval to allow \"stale\" data. By default, only the\n# Consul leader will respond to queries; any requests to a follower will\n# forward to the leader. In large clusters with many requests, this is not as\n# scalable, so this option allows any follower to respond to a query, so long\n# as the last-replicated data is within these bounds. Higher values result in\n# less cluster load, but are more likely to have outdated data.\nmax_stale = \"10m\"\n\n# This is the path to store a PID file which will contain the process ID of the\n# Consul Replicate process. This is useful if you plan to send custom signals\n# to the process.\npid_file = \"/path/to/pid\"\n\n# This is the prefix and datacenter to replicate and the resulting destination.\nprefix {\n  source      = \"global\"\n  datacenter  = \"nyc1\"\n  destination = \"default\"\n}\n\n# This is the signal to listen for to trigger a reload event. The default value\n# is shown below. Setting this value to the empty string will cause Consul\n# Replicate to not listen for any reload signals.\nreload_signal = \"SIGHUP\"\n\n# This is the path in Consul to store replication and leader status.\nstatus_dir = \"service/consul-replicate/statuses\"\n\n# This block defines the configuration for connecting to a syslog server for\n# logging.\nsyslog {\n  # This enables syslog logging. Specifying any other option also enables\n  # syslog logging.\n  enabled = true\n\n  # This is the name of the syslog facility to log to.\n  facility = \"LOCAL5\"\n}\n\n# This is the quiescence timers; it defines the minimum and maximum amount of\n# time to wait for the cluster to reach a consistent state before rendering a\n# replicating. This is useful to enable in systems that have a lot of flapping,\n# because it will reduce the the number of times a replication occurs.\nwait {\n  min = \"5s\"\n  max = \"10s\"\n}\n```\n\nNote that not all fields are required. If you are not logging to syslog, you do\nnot need to specify a syslog configuration.\n\nFor additional security, tokens may also be read from the environment using the\n`CONSUL_TOKEN` environment variable. It is highly recommended that you do not\nput your tokens in plain-text in a configuration file.\n\nInstruct Consul Replicate to use a configuration file with the `-config` flag:\n\n```shell\n$ consul-replicate -config \"/my/config.hcl\"\n```\n\nThis argument may be specified multiple times to load multiple configuration\nfiles. The right-most configuration takes the highest precedence. If the path to\na directory is provided (as opposed to the path to a file), all of the files in\nthe given directory will be merged in\n[lexical order](http://golang.org/pkg/path/filepath/#Walk), recursively. Please\nnote that symbolic links are _not_ followed.\n\n**Commands specified on the CLI take precedence over a config file!**\n\n## Debugging\n\nConsul Replicate can print verbose debugging output. To set the log level for\nConsul Replicate, use the `-log-level` flag:\n\n```shell\n$ consul-replicate -log-level info ...\n```\n\n```text\n\u003ctimestamp\u003e [INFO] (cli) received redis from Watcher\n\u003ctimestamp\u003e [INFO] (cli) invoking Runner\n# ...\n```\n\nYou can also specify the level as trace:\n\n```shell\n$ consul-replicate -log-level trace ...\n```\n\n```text\n\u003ctimestamp\u003e [DEBUG] (cli) creating Runner\n\u003ctimestamp\u003e [DEBUG] (cli) creating Consul API client\n\u003ctimestamp\u003e [DEBUG] (cli) creating Watcher\n\u003ctimestamp\u003e [DEBUG] (cli) looping for data\n\u003ctimestamp\u003e [DEBUG] (watcher) starting watch\n\u003ctimestamp\u003e [DEBUG] (watcher) all pollers have started, waiting for finish\n\u003ctimestamp\u003e [DEBUG] (redis) starting poll\n\u003ctimestamp\u003e [DEBUG] (service redis) querying Consul with \u0026{...}\n\u003ctimestamp\u003e [DEBUG] (service redis) Consul returned 2 services\n\u003ctimestamp\u003e [DEBUG] (redis) writing data to channel\n\u003ctimestamp\u003e [DEBUG] (redis) starting poll\n\u003ctimestamp\u003e [INFO] (cli) received redis from Watcher\n\u003ctimestamp\u003e [INFO] (cli) invoking Runner\n\u003ctimestamp\u003e [DEBUG] (service redis) querying Consul with \u0026{...}\n# ...\n```\n\n## FAQ\n\n**Q: Can I use this for master-master replication?**\u003cbr\u003e\nA: Master-master replication is not possible. A leader would never be elected.\n\n## Contributing\n\nTo build and install Consul Replicate locally, you will need to install the\nDocker engine:\n\n- [Docker for Mac](https://docs.docker.com/engine/installation/mac/)\n- [Docker for Windows](https://docs.docker.com/engine/installation/windows/)\n- [Docker for Linux](https://docs.docker.com/engine/installation/linux/ubuntulinux/)\n\nClone the repository:\n\n```shell\n$ git clone https://github.com/hashicorp/consul-replicate.git\n```\n\nTo compile the `consul-replicate` binary for your local machine:\n\n```shell\n$ make dev\n```\n\nThis will compile the `consul-replicate` binary into `bin/consul-replicate` as\nwell as your `$GOPATH` and run the test suite.\n\nIf you want to compile a specific binary, set `XC_OS` and `XC_ARCH` or run the\nfollowing to generate all binaries:\n\n```shell\n$ make bin\n```\n\nIf you just want to run the tests:\n\n```shell\n$ make test\n```\n\nOr to run a specific test in the suite:\n\n```shell\ngo test ./... -run SomeTestFunction_name\n```\n\n[consul]: https://www.consul.io \"Consul by HashiCorp\"\n[hcl]: https://github.com/hashicorp/hcl \"HashiCorp Configuration Language (hcl)\"\n[releases]: https://releases.hashicorp.com/consul-replicate \"Consul Replicate Releases\"\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhashicorp%2Fconsul-replicate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhashicorp%2Fconsul-replicate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhashicorp%2Fconsul-replicate/lists"}