Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/HariSekhon/HAProxy-configs
80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Kubernetes, Elasticsearch, SolrCloud, HBase, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, Hue, ZooKeeper, SSH, RabbitMQ, Redis, Riak, Cloudera, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, Rancher etc.
https://github.com/HariSekhon/HAProxy-configs
apache-drill cassandra cloudera elasticsearch hacktoberfest hadoop haproxy hbase hive influxdb mapr mysql nosql opentsdb postgresql presto prometheus redis solrcloud zookeeper
Last synced: 3 months ago
JSON representation
80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Kubernetes, Elasticsearch, SolrCloud, HBase, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, Hue, ZooKeeper, SSH, RabbitMQ, Redis, Riak, Cloudera, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, Rancher etc.
- Host: GitHub
- URL: https://github.com/HariSekhon/HAProxy-configs
- Owner: HariSekhon
- License: mit
- Created: 2018-06-08T19:04:36.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2024-02-09T03:47:06.000Z (9 months ago)
- Last Synced: 2024-02-14T21:52:39.367Z (9 months ago)
- Topics: apache-drill, cassandra, cloudera, elasticsearch, hacktoberfest, hadoop, haproxy, hbase, hive, influxdb, mapr, mysql, nosql, opentsdb, postgresql, presto, prometheus, redis, solrcloud, zookeeper
- Language: Shell
- Homepage: https://www.linkedin.com/in/HariSekhon
- Size: 496 KB
- Stars: 220
- Watchers: 14
- Forks: 79
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# Advanced HAProxy Configs for Big Data, NoSQL, Web and Infrastructure technologies
[](https://github.com/HariSekhon/HAProxy-configs/stargazers)
[](https://github.com/HariSekhon/HAProxy-configs/network)
[](https://github.com/HariSekhon/HAProxy-configs)
[](https://github.com/HariSekhon/HAProxy-configs/blob/master/LICENSE)
[](https://github.com/HariSekhon/HAProxy-configs/commits/master)
[](https://www.codacy.com/app/harisekhon/haproxy-configs)
[](https://www.codefactor.io/repository/github/harisekhon/HAProxy-configs)
[](https://sonarcloud.io/dashboard?id=HariSekhon_HAProxy-configs)
[](https://sonarcloud.io/dashboard?id=HariSekhon_HAProxy-configs)
[](https://sonarcloud.io/dashboard?id=HariSekhon_HAProxy-configs)
[](https://sonarcloud.io/dashboard?id=HariSekhon_HAProxy-configs)
[](https://sonarcloud.io/summary/new_code?id=HariSekhon_HAProxy-configs)
[](https://github.com/HariSekhon/HAProxy-configs)
[](https://github.com/HariSekhon/HAProxy-configs)
[](https://hub.docker.com/_/haproxy)
[](https://seladb.github.io/StarTrack-js/#/preload?r=HariSekhon,Nagios-Plugins&r=HariSekhon,Dockerfiles&r=HariSekhon,DevOps-Python-tools&r=HariSekhon,DevOps-Perl-tools&r=HariSekhon,DevOps-Bash-tools&r=HariSekhon,HAProxy-configs&r=HariSekhon,SQL-scripts)
[](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/STARCHARTS.md)
[](https://harisekhon.github.io/CI-CD/)
[](https://github.com/HariSekhon/HAProxy-configs/blob/master/Jenkinsfile)
[](https://github.com/HariSekhon/HAProxy-configs/blob/master/cicd/.concourse.yml)
[](https://github.com/HariSekhon/HAProxy-configs/blob/master/cicd/.gocd.yml)
[](https://github.com/HariSekhon/TeamCity-CI)
[](https://circleci.com/gh/HariSekhon/HAProxy-configs)
[](https://buildkite.com/hari-sekhon/haproxy-configs)
[](https://ci.appveyor.com/project/HariSekhon/HAProxy-configs/branch/master)
[](https://cloud.drone.io/HariSekhon/HAProxy-configs)
[](https://g.codefresh.io/pipelines/edit/new/builds?id=5e58e31b3953b7fe454b7904&pipeline=HAProxy-configs&projects=GitHub&projectId=5e52ca8ea284e00f882ea992&context=github&filter=page:1;pageSize:10;timeFrameStart:week)
[](https://cirrus-ci.com/github/HariSekhon/HAProxy-configs)
[](https://harisekhon.semaphoreci.com/projects/HAProxy-configs)
[](https://github.com/HariSekhon/HAProxy-configs/blob/master/buddy.yml)
[](https://github.com/HariSekhon/HAProxy-configs/blob/master/shippable.yml)
[](https://github.com/HariSekhon/HAProxy-configs/blob/master/.travis.yml)
[](https://dev.azure.com/harisekhon/GitHub/_build/latest?definitionId=10&branchName=master)
[](https://gitlab.com/HariSekhon/HAProxy-configs/pipelines)
[](https://bitbucket.org/harisekhon/haproxy-configs/addon/pipelines/home#!/)
[](https://github.com/HariSekhon/HAProxy-configs/blob/master/cicd/buildspec.yml)
[](https://github.com/HariSekhon/HAProxy-configs/blob/master/cicd/cloudbuild.yaml)
[](https://dev.azure.com/harisekhon/GitHub/_git/HAProxy-configs)
[](https://github.com/HariSekhon/HAProxy-configs)
[](https://gitlab.com/HariSekhon/HAProxy-configs)
[](https://bitbucket.org/HariSekhon/HAProxy-configs)
[](https://github.com/HariSekhon/HAProxy-configs/actions/workflows/shellcheck.yaml)
[](https://github.com/HariSekhon/HAProxy-configs/actions/workflows/validate.yaml)
[](https://github.com/HariSekhon/HAProxy-configs/actions/workflows/kics.yaml)
[](https://github.com/HariSekhon/HAProxy-configs/actions/workflows/grype.yaml)
[](https://github.com/HariSekhon/HAProxy-configs/actions/workflows/semgrep.yaml)
[](https://github.com/HariSekhon/HAProxy-configs/actions/workflows/semgrep-cloud.yaml)
[](https://github.com/HariSekhon/HAProxy-configs/actions/workflows/trivy.yaml)
[](https://github.com/HariSekhon/HAProxy-configs/actions?query=workflow%3A%22GitHub+Actions+Ubuntu%22)
[](https://github.com/HariSekhon/HAProxy-configs/actions/workflows/mac.yaml)
[](https://github.com/HariSekhon/HAProxy-configs/actions/workflows/mac_11.yaml)
[](https://github.com/HariSekhon/HAProxy-configs/actions/workflows/mac_12.yaml)
[](https://github.com/HariSekhon/HAProxy-configs/actions?query=workflow%3A%22Ubuntu%22)
[](https://github.com/HariSekhon/HAProxy-configs/actions?query=workflow%3A%22Ubuntu+20.04%22)
[](https://github.com/HariSekhon/HAProxy-configs/actions?query=workflow%3A%22Ubuntu+22.04%22)
[](https://github.com/HariSekhon/HAProxy-configs/actions?query=workflow%3A%22Debian%22)
[](https://github.com/HariSekhon/HAProxy-configs/actions?query=workflow%3A%22Debian+10%22)
[](https://github.com/HariSekhon/HAProxy-configs/actions?query=workflow%3A%22Debian+11%22)
[](https://github.com/HariSekhon/HAProxy-configs/actions?query=workflow%3A%22Debian+12%22)
[](https://github.com/HariSekhon/HAProxy-configs/actions?query=workflow%3A%22CentOS%22)
[](https://github.com/HariSekhon/HAProxy-configs/actions?query=workflow%3A%22CentOS+7%22)
[](https://github.com/HariSekhon/HAProxy-configs/actions?query=workflow%3A%22CentOS+8%22)
[](https://github.com/HariSekhon/HAProxy-configs/actions?query=workflow%3A%22Fedora%22)
[](https://github.com/HariSekhon/HAProxy-configs/actions?query=workflow%3A%22Alpine%22)
[](https://github.com/HariSekhon/HAProxy-configs/actions?query=workflow%3A%22Alpine+3%22)
[git.io/haproxy-configs](https://git.io/haproxy-configs)
Advanced HAProxy configurations for Multi-Master, Active-Standby (Hadoop, HBase, Presto) and Peer-to-Peer technologies (Elasticsearch, SolrCloud etc).
Designed both for production-grade High Availability and also to make scripting and monitoring easier when connecting to APIs.
Forked as a submodule from the [Advanced Nagios Plugins Collection](https://github.com/HariSekhon/Nagios-Plugins#advanced-nagios-plugins-collection), these configurations contain specialised Health Checks for each system based on experience and code from the [Advanced Nagios Plugins Collection](https://github.com/HariSekhon/Nagios-Plugins#advanced-nagios-plugins-collection) and [DevOps Python Tools](https://github.com/HariSekhon/DevOps-Python-tools) github repos.
-*Recommended to run with High Availability HAProxy using VRRP to create full production-grade High Availability load balancer solutions.**
Hari Sekhon
Cloud & Big Data Contractor, United Kingdom
(ex-Cloudera, former Hortonworks Consultant)
[](https://www.linkedin.com/in/HariSekhon/)
###### (you're welcome to connect with me on LinkedIn)
### Features
Each config comes pre-tuned with:
1. Specialised Health Checks per technology's API
2. Load Balancing algorithm selected best suited per technology (depending on Active/Passive, Peer-to-Peer etc)
3. Various buffer sizes, keepalives, connection limits and server retries already tuned (many via shared global config `10-global.cfg`)
4. ACL protections - limits access to private IP addresses - safer by default but easy to disable if you need to serve the internet
5. Additional logging
6. Optional Stats Admin UI (excellent for admin, monitoring and debugging - password protected, change the default password from 'test' in `20-stats.cfg` if you source that config)
You should use an expert consultant to fine tune to your needs but these should be extremely close to your finished production configurations. **In most cases all you need to do is put in your addresses for the backend servers**.
## Usage
Configurations are split by service in the form of ```.cfg``` for mix-and-match convenience and must be combined with ```10-global.cfg``` settings like so:
```sh
$ haproxy -f 10-global.cfg -f elasticsearch.cfg
```
For multiple services just add those service configurations to the command line options:
```sh
$ haproxy -f 10-global.cfg -f elasticsearch.cfg -f solrcloud.cfg
```
Some technologies default to the same port number (eg. Ambari and Presto both use port 8080), so if running both on the same HAProxy host then just modify one of the frontend listening port numbers.
If you want to add a Stats Admin UI then include the ```20-stats.cfg``` configuration (**remember to change the default password `test`**):
```sh
$ haproxy -f 10-global.cfg -f 20-stats.cfg -f elasticsearch.cfg ...
```
#### Shortcut
To shorten the command line you can simply use the `run.sh` script instead:
```sh
./run.sh elasticsearch.cfg
```
and easily with multiple configs
```sh
./run.sh elasticsearch.cfg solrcloud.cfg ...
```
### Backend Server Addresses (set these to your cluster hosts)
-*In most cases all you'll need to do is add your server addresses in to the backend config and then start HAProxy**.
Common backend server addresses have been pre-populated for convenience including:
- `````` - generic service name matching the proxied technology - could be resolved by DNS to multiple IPs to be balanced across
- ```192.168.99.100``` - the common [Docker Machine](https://docs.docker.com/machine/overview/) / [MiniKube](https://minikube.sigs.k8s.io) / [MiniShift](https://www.okd.io/minishift/) IP address
- ```docker``` - convenient to use DNS or `/etc/hosts` to point this to your Docker host
These addresses are used in Continuous Integration tests run on this repo from the [Advanced Nagios Plugins Collection](https://github.com/HariSekhon/Nagios-Plugins#advanced-nagios-plugins-collection).
### More Configs
See the ```untested/``` directory for a few more including SSL config versions I haven't got round to testing yet but should work.
### Tests
Forked as a submodule from the [Advanced Nagios Plugins Collection](https://github.com/HariSekhon/Nagios-Plugins#advanced-nagios-plugins-collection). These HAProxy configs are tested against Docker containers as part of CI tests in that repo.
## Related Repositories
- ```find_active_server.py``` from the [DevOps Python Tools](https://github.com/HariSekhon/DevOps-Python-tools) repo and its related adjacent programs for on-the-fly command line determination of active masters or first responding peers across many of these same technologies
- [DevOps Bash Tools](https://github.com/HariSekhon/DevOps-Bash-tools) - 1000+ DevOps Bash Scripts, Advanced `.bashrc`, `.vimrc`, `.screenrc`, `.tmux.conf`, `.gitconfig`, CI configs & Utility Code Library - AWS, GCP, Kubernetes, Docker, Kafka, Hadoop, SQL, BigQuery, Hive, Impala, PostgreSQL, MySQL, LDAP, DockerHub, Jenkins, Spotify API & MP3 tools, Git tricks, GitHub API, GitLab API, BitBucket API, Code & build linting, package management for Linux / Mac / Python / Perl / Ruby / NodeJS / Golang, and lots more random goodies
- [SQL Scripts](https://github.com/HariSekhon/SQL-scripts) - 100+ SQL Scripts - PostgreSQL, MySQL, AWS Athena, Google BigQuery
- [Jenkins](https://github.com/HariSekhon/Jenkins) - Advanced Jenkinsfile & Jenkins Groovy Shared Library
- [GitHub-Actions](https://github.com/HariSekhon/GitHub-Actions) - GitHub Actions master template & GitHub Actions Shared Workflows library
- [Templates](https://github.com/HariSekhon/Templates) - dozens of Code & Config templates - AWS, GCP, Docker, Jenkins, Terraform, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions Workflows, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml, M4 etc.
- [Kubernetes configs](https://github.com/HariSekhon/Kubernetes-configs) - Kubernetes YAML configs - Best Practices, Tips & Tricks are baked right into the templates for future deployments
- [Terraform](https://github.com/HariSekhon/Terraform) - Terraform templates for AWS / GCP / Azure / GitHub management
- [DevOps Python Tools](https://github.com/HariSekhon/DevOps-Python-tools) - 80+ DevOps CLI tools for AWS, GCP, Hadoop, HBase, Spark, Log Anonymizer, Ambari Blueprints, AWS CloudFormation, Linux, Docker, Spark Data Converters & Validators (Avro / Parquet / JSON / CSV / INI / XML / YAML), Elasticsearch, Solr, Travis CI, Pig, IPython
- [DevOps Perl Tools](https://github.com/harisekhon/perl-tools) - 25+ DevOps CLI tools for Hadoop, HDFS, Hive, Solr/SolrCloud CLI, Log Anonymizer, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Dockerfiles & SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Ambari FreeIPA Kerberos, Datameer, Linux...
- [The Advanced Nagios Plugins Collection](https://github.com/HariSekhon/Nagios-Plugins) - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)
- [Nagios Plugin Kafka](https://github.com/HariSekhon/Nagios-Plugin-Kafka) - Kafka API pub/sub Nagios Plugin written in Scala with Kerberos support
- [Dockerfiles](https://github.com/HariSekhon/Dockerfiles) - 50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Mesos, Consul, Riak, OpenTSDB, Jython, Advanced Nagios Plugins & DevOps Tools repos on Alpine, CentOS, Debian, Fedora, Ubuntu, Superset, H2O, Serf, Alluxio / Tachyon, FakeS3
- [HashiCorp Packer templates](https://github.com/HariSekhon/Packer-templates) - Linux automated bare-metal installs and portable virtual machines OVA format appliances using HashiCorp Packer, Redhat Kickstart, Debian Preseed and Ubuntu AutoInstaller / Cloud-Init
- [Diagrams-as-Code](https://github.com/HariSekhon/Diagrams-as-Code) - Cloud & Open Source architecture diagrams with Python & D2 source code provided - automatically regenerated via GitHub Actions CI/CD - AWS, GCP, Kubernetes, Jenkins, ArgoCD, Traefik, Kong API Gateway, Nginx, Redis, PostgreSQL, Kafka, Spark, web farms, event processing...
- [Knowledge-Base](https://github.com/HariSekhon/Knowledge-Base) - IT Knowledge Base from 20 years in DevOps, Linux, Cloud, Big Data, AWS, GCP etc.
### Stargazers over time
[](https://starchart.cc/HariSekhon/HAProxy-configs)
[git.io/haproxy-configs](https://git.io/haproxy-configs)