{"id":26649024,"url":"https://github.com/lexxai/es_cluster_docker_helper","last_synced_at":"2026-05-06T06:33:00.809Z","repository":{"id":283920773,"uuid":"951449747","full_name":"lexxai/es_cluster_docker_helper","owner":"lexxai","description":"The docker assistant for creating a docker-compose.yml file for organizing a cluster for an elastic search server. ","archived":false,"fork":false,"pushed_at":"2025-03-23T03:02:31.000Z","size":15,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-12T18:54:53.764Z","etag":null,"topics":["cluster","docker","docker-compose","elasticsearch","elasticsearch-cluster","python","truenas","truenas-scale"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lexxai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-19T17:39:51.000Z","updated_at":"2025-03-23T03:02:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"f2e74396-e138-4674-a3b1-2ec5cf745ba0","html_url":"https://github.com/lexxai/es_cluster_docker_helper","commit_stats":null,"previous_names":["lexxai/es_cluster_docker_helper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lexxai/es_cluster_docker_helper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lexxai%2Fes_cluster_docker_helper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lexxai%2Fes_cluster_docker_helper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lexxai%2Fes_cluster_docker_helper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lexxai%2Fes_cluster_docker_helper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lexxai","download_url":"https://codeload.github.com/lexxai/es_cluster_docker_helper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lexxai%2Fes_cluster_docker_helper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278838111,"owners_count":26054720,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cluster","docker","docker-compose","elasticsearch","elasticsearch-cluster","python","truenas","truenas-scale"],"created_at":"2025-03-25T00:47:44.227Z","updated_at":"2025-10-07T19:51:06.711Z","avatar_url":"https://github.com/lexxai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Docker-Compose Generator for ElasticSearch Cluster\n\nThis script helps generate a `docker-compose.yml` file for setting up an ElasticSearch cluster using template files and environment variables. The cluster is built as a set of Docker containers, including ElasticSearch nodes with different roles, Kibana, and additional service containers.\n\n## Usage\n\n1. **Configure Environment Variables**:  \n   Use a `.env` file to set the required environment variables. The main variable that configures the cluster is `DOCKER_SERVICES`.  \n\n   Example:  \n   ```plaintext\n   DOCKER_SERVICES=\"setup:1,node-master:1,node-data:3,node-ingest:2,node-coordinator:1,kibana:1\"\n   ```\n   Here, nodes are separated by commas, and the count of each node is specified after a colon.\n\n2. **Run script**:\n   Execute the script to generate the docker-compose.yml file:\n```bash\npython ./gen-compose.py\n```\n\n## Installation \n### Dependencies\nYou can install the required dependencies using either pip or poetry.\n\nUsing pip:\n```bash\npip install -r requirements.txt\n```\nUsing poetry:\n```bash\npoetry install\n```\n\n## Example Usage\n\nTo configure a cluster with the following setup:\n\n    1 setup node\n\n    1 master node\n\n    3 data nodes\n\n    2 ingest nodes\n\n    1 coordinator node\n\n    1 Kibana instance\n\nSet the DOCKER_SERVICES variable in your .env file as follows:\n\n## Example of run:\n\n### Logs\n```\n2025-03-19 02:23:24 [INFO] Running...\n2025-03-19 02:23:24 [INFO] Node with role in cluster: 'master', count: 1\n2025-03-19 02:23:24 [INFO] Node with role in cluster: 'data', count: 3\n2025-03-19 02:23:24 [INFO] Node with role in cluster: 'ingest', count: 2\n2025-03-19 02:23:24 [INFO] Node with role in cluster: 'coordinator', count: 1\n2025-03-19 02:23:24 [INFO] Total services with nodes roles in cluster: 7\n2025-03-19 02:23:24 [INFO] Other additional services is: 'setup, kibana'\n2025-03-19 02:23:24 [INFO] Total services for docker compose file: 9\n2025-03-19 02:23:25 [INFO] Saved results to 'docker-compose.yml'\n```\n\n### docker-compose.yml\n```yaml\nservices:\n  kibana:\n    cap_drop:\n    - ALL\n    deploy:\n      resources:\n        limits:\n          cpus: '8'\n          memory: 8096M\n    environment:\n    - SERVERNAME=kibana\n    - ELASTICSEARCH_HOSTS=es-server-0.example.com:9200\n    - ELASTICSEARCH_USERNAME=kibana_system\n    - ELASTICSEARCH_PASSWORD=12345678\n    - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt\n    healthcheck:\n      interval: 10s\n      retries: 120\n      test:\n      - CMD-SHELL\n      - curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'\n      timeout: 10s\n    image: docker.elastic.co/kibana/kibana:8.17.3\n    platform: linux/amd64\n    ports:\n    - 5601:5601\n    privileged: false\n    restart: unless-stopped\n    security_opt:\n    - no-new-privileges=true\n    user: 1000:1000\n    volumes:\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: true\n      source: /mnt/mdata/database/es_data/cluster/certs\n      target: /usr/share/kibana/config/certs\n      type: bind\n    - kibanadata:/usr/share/kibana/data\n  node_1:\n    cap_drop:\n    - ALL\n    container_name: node_1\n    depends_on:\n      setup:\n        condition: service_healthy\n    deploy:\n      resources:\n        limits:\n          cpus: '8'\n          memory: 8096M\n    environment:\n    - node.name=node_1\n    - ES_NETWORK_HOST=\"0.0.0.0,[::]\"\n    - cluster.name=es-cluster\n    - discovery.seed_hosts=es-server-0.example.com:9300\n    - cluster.initial_master_nodes=node_1\n    - node.roles=master\n    - http.port=9200\n    - transport.port=9300\n    - ELASTIC_PASSWORD=12345678\n    - xpack.security.http.ssl.enabled=true\n    - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.key\n    - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.crt\n    - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt\n    - xpack.security.http.ssl.supported_protocols=\"TLSv1.2,TLSv1.3\"\n    - xpack.security.http.ssl.verification_mode=certificate\n    - xpack.security.transport.ssl.enabled=true\n    - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.key\n    - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.crt\n    - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt\n    - xpack.security.transport.ssl.supported_protocols=\"TLSv1.2,TLSv1.3\"\n    - xpack.security.transport.ssl.verification_mode=certificate\n    - TZ=\"Etc/UTC\"\n    - UMASK=\"002\"\n    - UMASK_SET=\"002\"\n    healthcheck:\n      interval: 10s\n      retries: 120\n      test:\n      - CMD-SHELL\n      - 'curl --silent --output /dev/null --show-error --fail --cacert config/certs/ca/ca.crt\n        --header \"Authorization: Basic $(echo -n elastic:12345678 | base64)\" https://es-server-0.example.com:9200/_cluster/health?local=true'\n      timeout: 10s\n    image: elasticsearch:8.17.3\n    network_mode: host\n    platform: linux/amd64\n    privileged: false\n    restart: unless-stopped\n    security_opt:\n    - no-new-privileges=true\n    ulimits:\n      memlock:\n        hard: -1\n        soft: -1\n    user: 1000:1000\n    volumes:\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: true\n      source: /mnt/mdata/database/es_data/cluster/certs\n      target: /usr/share/elasticsearch/config/certs\n      type: bind\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: false\n      source: /mnt/mdata/database/es_data/cluster/node_1\n      target: /usr/share/elasticsearch/data\n      type: bind\n  node_2:\n    cap_drop:\n    - ALL\n    container_name: node_2\n    depends_on:\n      setup:\n        condition: service_healthy\n    deploy:\n      resources:\n        limits:\n          cpus: '8'\n          memory: 16384M\n    environment:\n    - node.name=node_2\n    - ES_NETWORK_HOST=\"0.0.0.0,[::]\"\n    - cluster.name=es-cluster\n    - discovery.seed_hosts=es-server-0.example.com:9300\n    - cluster.initial_master_nodes=''\n    - node.roles=data\n    - http.port=9201\n    - transport.port=9301\n    - ELASTIC_PASSWORD=12345678\n    - xpack.security.http.ssl.enabled=true\n    - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.key\n    - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.crt\n    - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt\n    - xpack.security.http.ssl.supported_protocols=\"TLSv1.2,TLSv1.3\"\n    - xpack.security.http.ssl.verification_mode=certificate\n    - xpack.security.transport.ssl.enabled=true\n    - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.key\n    - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.crt\n    - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt\n    - xpack.security.transport.ssl.supported_protocols=\"TLSv1.2,TLSv1.3\"\n    - xpack.security.transport.ssl.verification_mode=certificate\n    - TZ=\"Etc/UTC\"\n    - UMASK=\"002\"\n    - UMASK_SET=\"002\"\n    healthcheck:\n      interval: 10s\n      retries: 120\n      test:\n      - CMD-SHELL\n      - 'curl --silent --output /dev/null --show-error --fail --cacert config/certs/ca/ca.crt\n        --header \"Authorization: Basic $(echo -n elastic:12345678 | base64)\" https://es-server-0.example.com:9201/_cluster/health?local=true'\n      timeout: 10s\n    image: elasticsearch:8.17.3\n    network_mode: host\n    platform: linux/amd64\n    privileged: false\n    restart: unless-stopped\n    security_opt:\n    - no-new-privileges=true\n    ulimits:\n      memlock:\n        hard: -1\n        soft: -1\n    user: 1000:1000\n    volumes:\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: true\n      source: /mnt/mdata/database/es_data/cluster/certs\n      target: /usr/share/elasticsearch/config/certs\n      type: bind\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: false\n      source: /mnt/mdata/database/es_data/cluster/node_2\n      target: /usr/share/elasticsearch/data\n      type: bind\n  node_3:\n    cap_drop:\n    - ALL\n    container_name: node_3\n    depends_on:\n      setup:\n        condition: service_healthy\n    deploy:\n      resources:\n        limits:\n          cpus: '8'\n          memory: 16384M\n    environment:\n    - node.name=node_3\n    - ES_NETWORK_HOST=\"0.0.0.0,[::]\"\n    - cluster.name=es-cluster\n    - discovery.seed_hosts=es-server-0.example.com:9300\n    - cluster.initial_master_nodes=''\n    - node.roles=data\n    - http.port=9202\n    - transport.port=9302\n    - ELASTIC_PASSWORD=12345678\n    - xpack.security.http.ssl.enabled=true\n    - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.key\n    - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.crt\n    - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt\n    - xpack.security.http.ssl.supported_protocols=\"TLSv1.2,TLSv1.3\"\n    - xpack.security.http.ssl.verification_mode=certificate\n    - xpack.security.transport.ssl.enabled=true\n    - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.key\n    - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.crt\n    - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt\n    - xpack.security.transport.ssl.supported_protocols=\"TLSv1.2,TLSv1.3\"\n    - xpack.security.transport.ssl.verification_mode=certificate\n    - TZ=\"Etc/UTC\"\n    - UMASK=\"002\"\n    - UMASK_SET=\"002\"\n    healthcheck:\n      interval: 10s\n      retries: 120\n      test:\n      - CMD-SHELL\n      - 'curl --silent --output /dev/null --show-error --fail --cacert config/certs/ca/ca.crt\n        --header \"Authorization: Basic $(echo -n elastic:12345678 | base64)\" https://es-server-0.example.com:9202/_cluster/health?local=true'\n      timeout: 10s\n    image: elasticsearch:8.17.3\n    network_mode: host\n    platform: linux/amd64\n    privileged: false\n    restart: unless-stopped\n    security_opt:\n    - no-new-privileges=true\n    ulimits:\n      memlock:\n        hard: -1\n        soft: -1\n    user: 1000:1000\n    volumes:\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: true\n      source: /mnt/mdata/database/es_data/cluster/certs\n      target: /usr/share/elasticsearch/config/certs\n      type: bind\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: false\n      source: /mnt/mdata/database/es_data/cluster/node_3\n      target: /usr/share/elasticsearch/data\n      type: bind\n  node_4:\n    cap_drop:\n    - ALL\n    container_name: node_4\n    depends_on:\n      setup:\n        condition: service_healthy\n    deploy:\n      resources:\n        limits:\n          cpus: '8'\n          memory: 16384M\n    environment:\n    - node.name=node_4\n    - ES_NETWORK_HOST=\"0.0.0.0,[::]\"\n    - cluster.name=es-cluster\n    - discovery.seed_hosts=es-server-0.example.com:9300\n    - cluster.initial_master_nodes=''\n    - node.roles=data\n    - http.port=9203\n    - transport.port=9303\n    - ELASTIC_PASSWORD=12345678\n    - xpack.security.http.ssl.enabled=true\n    - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.key\n    - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.crt\n    - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt\n    - xpack.security.http.ssl.supported_protocols=\"TLSv1.2,TLSv1.3\"\n    - xpack.security.http.ssl.verification_mode=certificate\n    - xpack.security.transport.ssl.enabled=true\n    - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.key\n    - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.crt\n    - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt\n    - xpack.security.transport.ssl.supported_protocols=\"TLSv1.2,TLSv1.3\"\n    - xpack.security.transport.ssl.verification_mode=certificate\n    - TZ=\"Etc/UTC\"\n    - UMASK=\"002\"\n    - UMASK_SET=\"002\"\n    healthcheck:\n      interval: 10s\n      retries: 120\n      test:\n      - CMD-SHELL\n      - 'curl --silent --output /dev/null --show-error --fail --cacert config/certs/ca/ca.crt\n        --header \"Authorization: Basic $(echo -n elastic:12345678 | base64)\" https://es-server-0.example.com:9203/_cluster/health?local=true'\n      timeout: 10s\n    image: elasticsearch:8.17.3\n    network_mode: host\n    platform: linux/amd64\n    privileged: false\n    restart: unless-stopped\n    security_opt:\n    - no-new-privileges=true\n    ulimits:\n      memlock:\n        hard: -1\n        soft: -1\n    user: 1000:1000\n    volumes:\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: true\n      source: /mnt/mdata/database/es_data/cluster/certs\n      target: /usr/share/elasticsearch/config/certs\n      type: bind\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: false\n      source: /mnt/mdata/database/es_data/cluster/node_4\n      target: /usr/share/elasticsearch/data\n      type: bind\n  node_5:\n    cap_drop:\n    - ALL\n    container_name: node_5\n    depends_on:\n      setup:\n        condition: service_healthy\n    deploy:\n      resources:\n        limits:\n          cpus: '16'\n          memory: 16384M\n    environment:\n    - node.name=node_5\n    - ES_NETWORK_HOST=\"0.0.0.0,[::]\"\n    - cluster.name=es-cluster\n    - discovery.seed_hosts=es-server-0.example.com:9300\n    - cluster.initial_master_nodes=''\n    - node.roles=ingest\n    - http.port=9204\n    - transport.port=9304\n    - ELASTIC_PASSWORD=12345678\n    - xpack.security.http.ssl.enabled=true\n    - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.key\n    - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.crt\n    - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt\n    - xpack.security.http.ssl.supported_protocols=\"TLSv1.2,TLSv1.3\"\n    - xpack.security.http.ssl.verification_mode=certificate\n    - xpack.security.transport.ssl.enabled=true\n    - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.key\n    - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.crt\n    - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt\n    - xpack.security.transport.ssl.supported_protocols=\"TLSv1.2,TLSv1.3\"\n    - xpack.security.transport.ssl.verification_mode=certificate\n    - TZ=\"Etc/UTC\"\n    - UMASK=\"002\"\n    - UMASK_SET=\"002\"\n    healthcheck:\n      interval: 10s\n      retries: 120\n      test:\n      - CMD-SHELL\n      - 'curl --silent --output /dev/null --show-error --fail --cacert config/certs/ca/ca.crt\n        --header \"Authorization: Basic $(echo -n elastic:12345678 | base64)\" https://es-server-0.example.com:9204/_cluster/health?local=true'\n      timeout: 10s\n    image: elasticsearch:8.17.3\n    network_mode: host\n    platform: linux/amd64\n    privileged: false\n    restart: unless-stopped\n    security_opt:\n    - no-new-privileges=true\n    ulimits:\n      memlock:\n        hard: -1\n        soft: -1\n    user: 1000:1000\n    volumes:\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: true\n      source: /mnt/mdata/database/es_data/cluster/certs\n      target: /usr/share/elasticsearch/config/certs\n      type: bind\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: false\n      source: /mnt/mdata/database/es_data/cluster/node_5\n      target: /usr/share/elasticsearch/data\n      type: bind\n  node_6:\n    cap_drop:\n    - ALL\n    container_name: node_6\n    depends_on:\n      setup:\n        condition: service_healthy\n    deploy:\n      resources:\n        limits:\n          cpus: '16'\n          memory: 16384M\n    environment:\n    - node.name=node_6\n    - ES_NETWORK_HOST=\"0.0.0.0,[::]\"\n    - cluster.name=es-cluster\n    - discovery.seed_hosts=es-server-0.example.com:9300\n    - cluster.initial_master_nodes=''\n    - node.roles=ingest\n    - http.port=9205\n    - transport.port=9305\n    - ELASTIC_PASSWORD=12345678\n    - xpack.security.http.ssl.enabled=true\n    - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.key\n    - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.crt\n    - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt\n    - xpack.security.http.ssl.supported_protocols=\"TLSv1.2,TLSv1.3\"\n    - xpack.security.http.ssl.verification_mode=certificate\n    - xpack.security.transport.ssl.enabled=true\n    - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.key\n    - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.crt\n    - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt\n    - xpack.security.transport.ssl.supported_protocols=\"TLSv1.2,TLSv1.3\"\n    - xpack.security.transport.ssl.verification_mode=certificate\n    - TZ=\"Etc/UTC\"\n    - UMASK=\"002\"\n    - UMASK_SET=\"002\"\n    healthcheck:\n      interval: 10s\n      retries: 120\n      test:\n      - CMD-SHELL\n      - 'curl --silent --output /dev/null --show-error --fail --cacert config/certs/ca/ca.crt\n        --header \"Authorization: Basic $(echo -n elastic:12345678 | base64)\" https://es-server-0.example.com:9205/_cluster/health?local=true'\n      timeout: 10s\n    image: elasticsearch:8.17.3\n    network_mode: host\n    platform: linux/amd64\n    privileged: false\n    restart: unless-stopped\n    security_opt:\n    - no-new-privileges=true\n    ulimits:\n      memlock:\n        hard: -1\n        soft: -1\n    user: 1000:1000\n    volumes:\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: true\n      source: /mnt/mdata/database/es_data/cluster/certs\n      target: /usr/share/elasticsearch/config/certs\n      type: bind\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: false\n      source: /mnt/mdata/database/es_data/cluster/node_6\n      target: /usr/share/elasticsearch/data\n      type: bind\n  node_7:\n    cap_drop:\n    - ALL\n    container_name: node_7\n    depends_on:\n      setup:\n        condition: service_healthy\n    deploy:\n      resources:\n        limits:\n          cpus: '8'\n          memory: 16384M\n    environment:\n    - node.name=node_7\n    - ES_NETWORK_HOST=\"0.0.0.0,[::]\"\n    - cluster.name=es-cluster\n    - discovery.seed_hosts=es-server-0.example.com:9300\n    - cluster.initial_master_nodes=''\n    - node.roles=[]\n    - http.port=9206\n    - transport.port=9306\n    - ELASTIC_PASSWORD=12345678\n    - xpack.security.http.ssl.enabled=true\n    - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.key\n    - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.crt\n    - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt\n    - xpack.security.http.ssl.supported_protocols=\"TLSv1.2,TLSv1.3\"\n    - xpack.security.http.ssl.verification_mode=certificate\n    - xpack.security.transport.ssl.enabled=true\n    - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.key\n    - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/es-server-0.example.com/es-server-0.example.com.crt\n    - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca/ca.crt\n    - xpack.security.transport.ssl.supported_protocols=\"TLSv1.2,TLSv1.3\"\n    - xpack.security.transport.ssl.verification_mode=certificate\n    - TZ=\"Etc/UTC\"\n    - UMASK=\"002\"\n    - UMASK_SET=\"002\"\n    healthcheck:\n      interval: 10s\n      retries: 120\n      test:\n      - CMD-SHELL\n      - 'curl --silent --output /dev/null --show-error --fail --cacert config/certs/ca/ca.crt\n        --header \"Authorization: Basic $(echo -n elastic:12345678 | base64)\" https://es-server-0.example.com:9206/_cluster/health?local=true'\n      timeout: 10s\n    image: elasticsearch:8.17.3\n    network_mode: host\n    platform: linux/amd64\n    privileged: false\n    restart: unless-stopped\n    security_opt:\n    - no-new-privileges=true\n    ulimits:\n      memlock:\n        hard: -1\n        soft: -1\n    user: 1000:1000\n    volumes:\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: true\n      source: /mnt/mdata/database/es_data/cluster/certs\n      target: /usr/share/elasticsearch/config/certs\n      type: bind\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: false\n      source: /mnt/mdata/database/es_data/cluster/node_7\n      target: /usr/share/elasticsearch/data\n      type: bind\n  setup:\n    command: \"bash -c 'if [ x12345678 == x ]; then\\n    echo \\\"Set the ELASTIC_PASSWORD\\\n      \\ environment variable in the .env file\\\";\\n    exit 1;\\n  elif [ x12345678\\\n      \\ == x ]; then\\n    echo \\\"Set the KIBANA_PASSWORD environment variable in the\\\n      \\ .env file\\\";\\n    exit 1;\\n  fi;\\n  if [ ! -f config/certs/ca.zip ]; then\\n\\\n      \\    echo \\\"Creating CA\\\";\\n    bin/elasticsearch-certutil ca --silent --pem\\\n      \\ -out config/certs/ca.zip;\\n    unzip config/certs/ca.zip -d config/certs;\\n\\\n      \\  fi;\\n  if [ ! -f config/certs/certs.zip ]; then\\n    echo \\\"Creating certs\\\"\\\n      ;\\n    echo -ne \\\\\\n    \\\"instances:\\\\n\\\"\\\\\\n    \\\"- name: es-server-0.example.com\\\\\\\n      n\\\"\\\\\\n    \\\"  dns:\\\\n\\\"\\\\\\n    \\\"  - es-server-0.example.com\\\\n\\\"\\\\\\n    \\\"\\\n      \\  - localhost\\\\n\\\"\\\\\\n    \\\"  ip:\\\\n\\\"\\\\\\n    \\\"  - 10.1.10.1\\\\n\\\"\\\\\\n    \\\"\\\n      \\  - 127.0.0.1\\\\n\\\"\\\\\\n    \u003e config/certs/instances.yml;\\n    bin/elasticsearch-certutil\\\n      \\ cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml\\\n      \\ --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;\\n    unzip\\\n      \\ config/certs/certs.zip -d config/certs;\\n  fi;\\n  echo \\\"Waiting for Elasticsearch\\\n      \\ availability\\\";\\n  until curl -s --cacert config/certs/ca/ca.crt https://es-server-0.example.com:9200\\\n      \\ | grep -q \\\"missing authentication credentials\\\"; do sleep 30; done;\\n  echo\\\n      \\ \\\"Setting kibana_system password\\\";\\n  until curl -s -X POST --cacert config/certs/ca/ca.crt\\\n      \\ -u \\\"elastic:12345678\\\" -H \\\"Content-Type: application/json\\\" https://es-server-0.example.com:9200/_security/user/kibana_system/_password\\\n      \\ -d \\\"{\\\\\\\"password\\\\\\\":\\\\\\\"12345678\\\\\\\"}\\\" | grep -q \\\"^{}\\\"; do sleep 10;\\\n      \\ done;\\n  echo \\\"All done!\\\";'\\n\"\n    healthcheck:\n      interval: 1s\n      retries: 120\n      test:\n      - CMD-SHELL\n      - '[ -f config/certs/es-server-0.example.com/es-server-0.example.com.crt ]'\n      timeout: 5s\n    image: elasticsearch:8.17.3\n    user: 1000:1000\n    volumes:\n    - bind:\n        create_host_path: false\n        propagation: rprivate\n      read_only: false\n      source: /mnt/mdata/database/es_data/cluster/certs\n      target: /usr/share/elasticsearch/config/certs\n      type: bind\nvolumes:\n  kibanadata:\n    driver: local\n```\n\n### services-resources.yml\n```yaml\nservices:\n  node-master:\n    limits:\n      cpu: \"8\"\n      memory: \"8096M\"\n  node-data:\n    limits:\n      cpu: \"8\"\n      memory: \"16384M\"\n  node-ingest:\n    limits:\n      cpu: \"16\"\n      memory: \"16384M\"\n  node-coordinator:\n    limits:\n      cpu: \"8\"\n      memory: \"16384M\"\n  node-data_ingest:\n    limits:\n      cpu: \"16\"\n      memory: \"24384M\"      \n  node-master_data_ingest:\n    limits:\n      cpu: \"32\"\n      memory: \"64000M\"\n  node-master_data:\n    limits:\n      cpu: \"16\"\n      memory: \"32768M\"      \n  setup:\n    limits:\n      cpu: \"1\"\n      memory: \"2048M\"\n  kibana:\n    limits:\n      cpu: \"8\"\n      memory: \"8096M\"\n```\n\n## Test Environment\n\nThe generated ElasticSearch cluster was tested using the resulting `docker-compose.yml` file on a **TrueNAS SCALE 24.10.2** server. The cluster was deployed as a custom application by importing the YAML file into TrueNAS SCALE.\n\n### Key Details:\n- **TrueNAS SCALE Version**: 24.10.2\n- **Storage Configuration**: Mount points were created as **ZFS Datasets** on the TrueNAS server.\n- **Deployment Method**: The `docker-compose.yml` file was used to deploy the cluster as a custom application.\n\n### Network Configuration:\n- **Network Mode**: ElasticSearch nodes were configured with `network_mode: host` in the `docker-compose.yml` file.\n- **IP and Ports**: All nodes share the same IP address but use different, incrementally assigned ports for **HTTPS** and **transport connections**.\n\n### Load Balancing:\n- A separate **HAProxy server** was later set up as a **round-robin reverse proxy** to balance traffic across the ElasticSearch nodes.\n- The HAProxy backend was configured to forward requests only to nodes with the **data-only role**, providing a single entry point for the ElasticSearch cluster.\n\nThis setup ensures reliable storage, performance, and scalability, leveraging the robustness of ZFS on TrueNAS SCALE and the flexibility of HAProxy for load balancing.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flexxai%2Fes_cluster_docker_helper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flexxai%2Fes_cluster_docker_helper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flexxai%2Fes_cluster_docker_helper/lists"}