{"id":13582455,"url":"https://github.com/adamringhede/influxdb-ha","last_synced_at":"2025-06-10T17:39:24.710Z","repository":{"id":148039728,"uuid":"80944051","full_name":"adamringhede/influxdb-ha","owner":"adamringhede","description":"High-availability and horizontal scalability for InfluxDB","archived":false,"fork":false,"pushed_at":"2019-07-28T09:25:55.000Z","size":1043,"stargazers_count":46,"open_issues_count":1,"forks_count":12,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-05-16T13:44:48.896Z","etag":null,"topics":["clustering","database","influxdb","replication","time-series"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/adamringhede.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-02-04T19:49:00.000Z","updated_at":"2024-08-17T14:27:46.000Z","dependencies_parsed_at":"2023-05-28T07:00:24.866Z","dependency_job_id":null,"html_url":"https://github.com/adamringhede/influxdb-ha","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamringhede%2Finfluxdb-ha","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamringhede%2Finfluxdb-ha/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamringhede%2Finfluxdb-ha/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamringhede%2Finfluxdb-ha/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/adamringhede","download_url":"https://codeload.github.com/adamringhede/influxdb-ha/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamringhede%2Finfluxdb-ha/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259118814,"owners_count":22808061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clustering","database","influxdb","replication","time-series"],"created_at":"2024-08-01T15:02:43.934Z","updated_at":"2025-06-10T17:39:24.660Z","avatar_url":"https://github.com/adamringhede.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# InfluxDB Cluster (unofficial)\nA distributed clustering system for production deployments of InfluxDB with high availability and horizontal scalability via partitioning.\n\n*Partitioning is not different from what is often called \"sharding\" in NoSQL database systems. However, InfluxDB already uses the term \"sharding\" to refer to\nsplitting individual series into chunks, called \"shards\", based on time. In order to scale writes, it would not work to shard data using only time chunks as most writes would be made\nto the same chunk on the same node. Instead, in this implementation, entire series are distributed to different nodes which we call \"partitioning\".* \n\n## Not ready for production\nAltough most functionality is implemented, it has not yet been tested with high load. There are possibly memory leaks, performance issues, bugs and missing critical functionality that will surface when running in production.\n\nAlso, some of the instructions below refer to functionality that is not yet implemented completely. \n\n## Testing it locally\nOne can easily start a cluster using either docker-compose or Kubernetes configurations available in this repo. The following guides will result in starting an InfluxDB shell conneted to the cluster nodes. In the shell, you can make inserts and queries just like a regular InfluxDB shell. Inserts will be distributed for redundancy by default. If a partition key is defined, it will also partition the data. Queries will also be rediriected to a node that holds data automatically based on the partition key. \n\n### Using Docker Compose\nRun the following command to start the cluster in the project's root directory:\n```\ndocker-compose up -d\n```\nIt will create three cluster nodes on the same machine with one container for the cluster proxy (this software) and a container just for InfluxDB for each node. \n\nOne of the cluster proxies will listen of the default InfluxDB port 8086, so you can connect to it by just running `influx` in your terminal if you have the InfluxDB CLI installed. The command `SHOW NODES` is handled by the cluster proxy so you can see that the cluster is up and running.\n\n```\n➜ influxdb-ha git:(master) ✗ influx \nInfluxDB shell version: v1.7.7\n\u003e show nodes\nname: nodes\nname             data location\n----             -------------\ninfluxdb-handle  influxdb-1:8086\ninfluxdb-handle2 influxdb-2:8086\ninfluxdb-handle3 influxdb-3:8086\n\n```\n\n### Using Kubernetes\nSee the Readme file in the kubernetes folder in this repo to learn more. \n\n```\ncd kubernetes\nminikube start\nbin/clean\nbin/start\nbin/influx\nInfluxDB shell version: v1.7.7\n\u003e show nodes\nname: nodes\nname             data location\n----             -------------\ninflux-cluster-0 influx-cluster-0.influxc:8086\ninflux-cluster-2 influx-cluster-2.influxc:8086\n\u003e CREATE USER admin WITH PASSWORD 'password' WITH ALL PRIVILEGES\n\u003e auth\nusername: admin\npassword: \n```\n\n## Etcd\nThe cluster is dependent on etcd for storing data used for clustering mechanisms. It is preferable \nto use a cluster of 3 etcd instances for high availability but it is not required as the cluster can function with degraded functionality while Etcd is unavailable.\n\n## Extend existing deployment\n\n### High availability via replication\nBy adding one ore more nodes, data will automatically be replicated\n\n#### 1. Start a cluster agent \nStart a cluster agent process referencing the old - still running - single node on the same \nmachine using the `-data` option. The ip should be reachable from all other nodes in the cluster \nas well for replication and forwarding of queries. \n\n```\ninfluxc -data 10.2.3.4:8086 -cluster-id 1 -etcd \"10.3.4.5:2379,10.4.5.6:2379,10.5.6.7:2379\"\n```\n\nIn order to migrate data in InfluxDB, the cluster agent needs to have admin access\n\n#### 2. Add more nodes\nOn another machine, start InfluxDB and the clustering agent like above but with a new \ninfluxd process. The cluster-id has to be the same and it should also specity the etcd nodes.\nData will then automatically start being imported to the new node. \n\n##### Recovery of passwords\nWith an existing single node deployment, all meta data needs to be replicated which includes users and passwords. If the deployment is set up in a way that the cluster agent can’t access meta data files where hashed passwords are stored, then it will not be able to recover passwords, and all users' passwords will be reset and need to be set again manually. \n\n##### Greater number of nodes than replication factor\nIf you add more nodes than the configured replication factor, the additional nodes can be used as well if you have more than one measurement. If you only have a few very large measurements and some other much smaller, some nodes will not be used to its fullest capacity and it could be more beneficial in terms of hardware costs to host the smaller measurements seperately in more appropriate instances. \n\n### Scale via partitioning\nPartitioning is possible to do on a running InfluxDB deployment without downtime. Note that after it has moved some of the data to the new nodes, it will delete the data from the original node permanently, so it is recommended to make a backup before transitioning to a partitioned deployment. \n\n\n#### 1. Start a cluster agent \nStart a cluster agent process referencing the old single node on the same machine.\n\n```\ninfluxc -data 10.2.3.4:8086 -cluster-id 1 -etcd \"10.3.4.5:2379,10.4.5.6:2379,10.5.6.7:2379\"\n```\n\n#### 2. Create a partition key\nCreate a partition key on the database or database and measurement combination. Read **Selecting partition key tags** before doing this.\n\n```sql\nCREATE PARTITION KEY meter_id,region ON mydb.mymeasurement\n```\n\n#### 3. (optional) Change replication factor\nBy default, data will be replicated to 2 nodes. So if the deployment consists of 2 nodes, then partitioning does not have any effect until another node is added. The replication factor can be changed later, however it is preferable to set it to the preferred number now rather than after the data is partitioned to not spend time on moving around data.\n\nThe replication factor can either be defined as a default, on a single database or just on a specific measurement by specifying an `ON` clause\n\nChanging the default to replicate to 3 nodes\n```sql\nSET REPLICATION FACTOR 3\n```\n\nSetting the replication factor to 1 to avoid replication in order to preserve storage space on a single measurement. \n```sql\nSET REPLICATION FACTOR 1 ON mydb.mymeasurement\n```\n\nFor a measurement that receives a lot of read from different systems or users at the same time, you may want to increase the replication factor. \nNote that, if you have less nodes in the cluster than the set replicatin factor, then it will not replicate more times than there are nodes. \nAs more nodes are added, data will automatically be replicated until the number of replicas requested by the replication factor is met. \n```sql\nSET REPLICATION FACTOR 10 ON mydb.mymeasurement\n```\n\n#### 4. (optional) Add more nodes\nOn another machine, start InfluxDB and the clustering agent like above but with the new influxdb process. The cluster-id has to be the same and it should also specity the etcd nodes.\n\nAbout 50% of the data will then automatically start being imported to the new node. After the data is imported, it will be deleted from from the original node unless that node should still have some of the data according to the replication factor.\n\nRepeat the last step until you have as many nodes as you want.\n\n*A complete repartitioning is not required when adding nodes as this implementation is using what's called \"consistent hashing\" which makes adding another node require has a constant duration, rather than a linear increase. This makes adding and removing nodes efficient.*\n\n## Selecting partition key tags\nA partition key requires one or more tags to partition data. To be able to query partitioned data efficiently without having to broadcast the query the entire cluster the tags need to be in the `WHERE` clause of the query in `=` conditions. That means having fewer tags can give more freedom when making queries. However, if the tag has low cardinality or very disproportionate, then it may not be possible to partition the data evenly across the nodes in the cluster. This can then be resolved by adding another tag to the partition key. \n\nChanging partition key later is possible – even without downtime – but requires creating a new measurement and copying all data to that measurement where it will be distributed differently. \n\n## Managing nodes\nThe following commands can be used in the normal influx client by connecting to any of the nodes. \n\n### Show nodes\nGet a list of all nodes currently in the cluster.\n\n```sql\nSHOW NODES\n```\n\n### Removing a node\nRemoving a node will cause other nodes to import data as needed from it. If the data is not replicated (that is the replication factor is set to 1), then it is important that the underlying influxd process is up so that the other nodes can import data from it. \n\n```sql\nREMOVE NODE nodename\n```\n\n## Hardware sizing guidelines\nThe guidelines for single nodes given by the official InfluxDB docuentations can be extrapolated here. The cluster agents do not add much overhead, but need additional storage to save data temporarily that could not be written when the target node is unavailable.\n\n### Upgrading storage\nThere are two main ways of upgrading the storage capacity which should work regardless of your infrastructure. \n\nThe first, which is commonly used for database deployments, is to stop the node, attach a new larger volume, copy the data to it from the old one and then start it.\nData that was to be written to it while it was down will be recovered after starting the node.\nNote that this implementation distributes data evenly; so to increase storage capacity, all nodes need to be upgraded  \n\nThe second, which does not require stopping any node and may be more appropriate if I/O rate is a bottleneck  is\nto add one or more nodes. \nIf you have more nodes than the configured replication factor, data will be distributed evenly as another node is added.\n\n### Recovery data storage \nThe cluster agent process need access to additional persistent storage for recovery data. The amount required depends on the volumes of points written when a node is unavailable and how fast it can be recovered which depends on disk io.    \n\n## Distributed queries\n\n### Query to multiple partitions without aggregations\nThe query is distributed to nodes and results are then merged and sorted to give the impression that the request was made to a single node.  \n\n### Query to multiple partitions with aggregations\nAggregations make it a lot more complicated but is possible and will be completely automatic. The query is first decompiled and an abstract syntax tree (AST) is created that is then\nused to perform the aggregation from results from the nodes. \n\n## Limitations\n\n### Sub queries\nCurrently does not work at all\n\nFuture:\n* May work on partitioned data if the entire query can be contained to a single node or set of nodes\n\n### Multiple FROM clauses\nNot supported when querying partitioned data but works for other nodes\n\nFuture:\n* Can work by combining the results locally.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadamringhede%2Finfluxdb-ha","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadamringhede%2Finfluxdb-ha","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadamringhede%2Finfluxdb-ha/lists"}