{"id":15517094,"url":"https://github.com/deric/puppet-clickhouse","last_synced_at":"2025-12-11T21:44:11.768Z","repository":{"id":136386418,"uuid":"284262167","full_name":"deric/puppet-clickhouse","owner":"deric","description":"Pupppet module to manage Clickhouse installation","archived":false,"fork":false,"pushed_at":"2025-12-03T09:52:11.000Z","size":167,"stargazers_count":2,"open_issues_count":0,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-12-05T20:21:05.334Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deric.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-08-01T13:09:30.000Z","updated_at":"2025-12-03T09:52:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"596b79ad-c0da-4a7f-bb05-80d773ba83f6","html_url":"https://github.com/deric/puppet-clickhouse","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/deric/puppet-clickhouse","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deric%2Fpuppet-clickhouse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deric%2Fpuppet-clickhouse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deric%2Fpuppet-clickhouse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deric%2Fpuppet-clickhouse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deric","download_url":"https://codeload.github.com/deric/puppet-clickhouse/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deric%2Fpuppet-clickhouse/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27670620,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-11T02:00:11.302Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T10:11:33.449Z","updated_at":"2025-12-11T21:44:11.749Z","avatar_url":"https://github.com/deric.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Puppet Clickhouse\n\n[![Puppet Forge](http://img.shields.io/puppetforge/v/deric/clickhouse.svg)](https://forge.puppet.com/modules/deric/clickhouse) [![Tests](https://github.com/deric/puppet-clickhouse/actions/workflows/spec.yml/badge.svg)](https://github.com/deric/puppet-clickhouse/actions/workflows/spec.yml)\n\nThe clickhouse module installs, configures and manages the Clickhouse Server service.\n\nIt also allows to create and manage users, quotas, profiles, dictionaries, databases as well as configure replication and sharding.\n\n## Setup\n\n### Setup Requirements\n\nThis module requires xml-simple gem, which is used to translate Hash configuration to Clickhouse XML format configuration files.\nTo install it you need to execute following command on your puppetmaster server:\n\n```bash\nsudo puppetserver gem install xml-simple\n```\n\n### Beginning with clickhouse\n\nTo install a server with the default options:\n\n`include 'clickhouse::server'`.\n\nTo customize Clickhouse server options, you must also pass in an override hash:\n\n```puppet\nclass { 'clickhouse::server':\n  override_options =\u003e $override_options\n}\n```\n\nSee [**Customize Server Options**](#customize-server-options) below for examples of the hash structure for $override_options.\n\n## Usage\n\nAll server configuration is done via `clickhouse::server`. To install client separatly, use `clickhouse::client`\n\n### Customize server options\n\nTo define server options, pass a hash structure of overrides in `clickhouse::server`. Server configuration parameters can be found at https://clickhouse.yandex/docs/en/operations/server_settings/settings/\n\n```puppet\n$override_options = {\n  parent_xml_tag =\u003e {\n    item =\u003e thing,\n  }\n}\n```\n\nFor example, to configure zstd compression pass following override options hash:\n\n```puppet\n$override_options = {\n  compression =\u003e {\n    case =\u003e {\n      min_part_size       =\u003e 10000000000,\n      min_part_size_ratio =\u003e 0.01,\n      method              =\u003e 'zstd',\n    }\n  }\n}\n```\nIt will add following section to Clickhouse configuration file:\n\n```xml\n\u003ccompression\u003e\n    \u003ccase\u003e\n        \u003cmin_part_size\u003e10000000000\u003c/min_part_size\u003e\n        \u003cmin_part_size_ratio\u003e0.01\u003c/min_part_size_ratio\u003e\n        \u003cmethod\u003ezstd\u003c/method\u003e\n    \u003c/case\u003e\n\u003c/compression\u003e\n```\n\n### Create a database\n\nTo create a database:\n\n```puppet\nclickhouse_database { 'my_database':\n  ensure =\u003e 'present',\n}\n```\n\n### Specify passwords for users\n\nIn addition to passing passwords in plain text, you can input them in sha256 encoding. For example:\n\n```puppet\nclass { 'clickhouse::server':\n  users =\u003e {\n    myuser =\u003e {\n      password =\u003e '02472d6a1e23f73b37481bbd67949a5d16cbaf3d71770696f20a0cd773a2e682',\n    }\n  }\n}\n```\n\n### Install Clickhouse Server\n\nThis example shows how to install Clickhouse Server, configure zstd compression for it, create production profile and quota, create user alice and assign production profile and quota to her and also configure replication and three different sharding schemas: replicated - just a basic replication with single shard and two replicas; segmented - two shards without replicas; segmented_replicated - two shards each having two replicas:\n\n```puppet\nclass { 'clickhouse::server':\n  override_options =\u003e {\n    compression =\u003e {\n      case =\u003e {\n        min_part_size       =\u003e 10000000000,\n        min_part_size_ratio =\u003e 0.01,\n        method              =\u003e 'zstd',\n      },\n    },\n  },\n  profiles         =\u003e {\n    production =\u003e {\n      use_uncompressed_cache =\u003e 0,\n      log_queries =\u003e 1,\n      max_memory_usage =\u003e ceiling($facts['memory']['system']['total_bytes'] * 0.7),\n    },\n  },\n  quotas           =\u003e {\n    production =\u003e {\n      interval =\u003e [\n        {\n          duration       =\u003e 3600,\n          queries        =\u003e 200,\n          erros          =\u003e 10,\n          result_rows    =\u003e 0,\n          read_rows      =\u003e 0,\n          execution_time =\u003e 0,\n        }\n      ],\n    },\n  },\n  users            =\u003e {\n    alice =\u003e {\n      password =\u003e '02472d6a1e23f73b37481bbd67949a5d16cbaf3d71770696f20a0cd773a2e682',\n      quota    =\u003e 'production',\n      profile  =\u003e 'production',\n      networks  =\u003e {\n        ip =\u003e ['::/0'],\n      },\n    },\n  },\n  replication      =\u003e {\n    zookeeper_servers =\u003e ['zookeeper1.local:2181', 'zookeeper2.local:2181', 'zookeeper3.local:2181'],\n    macros =\u003e {\n      cluster =\u003e 'Clickhouse_cluster',\n      replica =\u003e $facts['networking']['fqdn'],\n    },\n  },\n  remote_servers   =\u003e {\n    replicated           =\u003e {\n      shard =\u003e {\n        internal_replication =\u003e true,\n        replica              =\u003e ['host1.local:9000', 'host2.local:9000'],\n      },\n    },\n    segmented            =\u003e {\n      shard1 =\u003e {\n        weight               =\u003e 1,\n        internal_replication =\u003e true,\n        replica              =\u003e ['host1.local:9000'],\n      },\n      shard2 =\u003e {\n        weight               =\u003e 2,\n        internal_replication =\u003e true,\n        replica              =\u003e ['host2.local:9000'],\n      },\n    },\n    segmented_replicated =\u003e {\n      shard1 =\u003e {\n        internal_replication =\u003e true,\n        replica              =\u003e ['host1.local:9000', 'host2.local:9000'],\n      },\n      shard2 =\u003e {\n        internal_replication =\u003e true,\n        replica              =\u003e ['host3.local:9000', 'host4.local:9000'],\n      },\n    },\n  },\n}\n```\n\n### Hiera support\n\nConfiguration from YAML backend:\n```yaml\nclickhouse::server::override_options:\n  listen_host: 0.0.0.0\n  default_profile: default\n  default_database: default\n  mark_cache_size: 5368709120\n  uncompressed_cache_size: 8589934592\n  path: /var/lib/clickhouse/\n  interserver_http_port: 9009\n  interserver_http_host: \"%{::fqdn}\"\n  max_connections: 4096\n  max_concurrent_queries: 100\n  http_port: 8123\n  tcp_port: 9000\n  prometheus:\n    endpoint: /metrics\n    port: 8001\n    metrics: true\n    events: true\n    asynchronous_metrics: true\nclickhouse::server::profiles:\n  default:\n    max_memory_usage: 10000000000 #  Maximum memory usage for processing single query, in bytes\n    use_uncompressed_cache: 0 # Use cache of uncompressed blocks of data. Meaningful only for processing many of very short queries\n    load_balancing: random\n  readonly:\n    readonly: 1\n```\n\n\n### Replicas\n\nConfigure host properties:\n```yaml\nclickhouse::server::remote_servers:\n  replicated:\n    shard:\n      internal_replication: true\n      replicas:\n        host1.local:\n          port: 9000\n          priority: 1\n          default_database: foo\n          tcp_port_secure: 9100\n          secure: 1\n          user: click\n          password: house\n        host2.local:\n          port: 9000\n```\n\n### Crash reports\n\nSettings for opt-in sending crash reports.\n\n```\nclickhouse::server::crash_reports:\n  enabled: true\n  endpoint: http://sentry.localhost # Overrides the Sentry endpoint\n  debug: false\n```\n\n\n## Reference\n\n**Classes**\n\n_Public Classes_\n\n* [`clickhouse::client`](./REFERENCE.md#clickhouseclient): Installs and configures Clickhouse client.\n* [`clickhouse::repo`](./REFERENCE.md#clickhouserepo): Installs repository for Clickhouse.\n* [`clickhouse::server`](./REFERENCE.md#clickhouseserver): Installs and configures Clickhouse server.\n\n_Private Classes_\n\n* `clickhouse::client::install`: Private class for managing Clickhouse client package.\n* `clickhouse::params`: Private class for setting default Clickhouse parameters.\n* `clickhouse::server::config`: Private class for Clickhouse server configuration.\n* `clickhouse::server::install`: Private class for managing Clickhouse server package.\n* `clickhouse::server::resources`: Private class for applying Clickhouse resources.\n* `clickhouse::server::service`: Private class for managing the Clickhouse service.\n\n**Defined types**\n\n* [`clickhouse::server::dictionary`](./REFERENCE.md#clickhouseserverdictionary): Create and manage Clickhouse dictionary.\n* [`clickhouse::server::macros`](./REFERENCE.md#clickhouseservermacros): Create and manage Clickhouse macros file for replication.\n* [`clickhouse::server::profiles`](./REFERENCE.md#clickhouseserverprofiles): Create and manage Clickhouse profiles.\n* [`clickhouse::server::quotas`](./REFERENCE.md#clickhouseserverquotas): Create and manage Clickhouse quotas.\n* [`clickhouse::server::remote_servers`](./REFERENCE.md#clickhouseserverremote_servers): Create and manage Clickhouse remote servers for Distributed engine.\n* [`clickhouse::server::user`](./REFERENCE.md#clickhouseserveruser): Create and manage Clickhouse user.\n\n**Resource types**\n\n* [`clickhouse_database`](./REFERENCE.md#clickhouse_database): Manages a Clickhouse database.\n\n**Functions**\n\n* [`clickhouse_config`](./REFERENCE.md#clickhouse_config): Convert hash to Clickhouse XML config.\n\n## Limitations\n\nFor a list of supported operating systems, see [metadata.json](https://github.com/MaxFedotov/puppet-clickhouse/blob/master/metadata.json)\n\n## Development\n\nPlease feel free to fork, modify, create issues, bug reports and pull requests.\n\n### Testing epp templates\n\nRender template from CLI:\n```\n$ puppet epp render templates/zookeeper.xml.epp --values \"{'zookeeper_servers' =\u003e ['172.0.0.1:2181'], distributed_ddl=\u003e {path =\u003e '/foo' }}\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderic%2Fpuppet-clickhouse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fderic%2Fpuppet-clickhouse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderic%2Fpuppet-clickhouse/lists"}