{"id":13795385,"url":"https://github.com/MrAlias/druid","last_synced_at":"2025-05-12T22:31:06.498Z","repository":{"id":36028351,"uuid":"40324457","full_name":"MrAlias/druid","owner":"MrAlias","description":"Puppet module that sets up and configures a full druid cluster","archived":false,"fork":false,"pushed_at":"2015-12-12T11:55:49.000Z","size":201,"stargazers_count":4,"open_issues_count":0,"forks_count":7,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-10-17T03:50:51.170Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Puppet","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MrAlias.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-06T20:05:02.000Z","updated_at":"2019-01-18T16:09:54.000Z","dependencies_parsed_at":"2022-08-17T22:25:15.707Z","dependency_job_id":null,"html_url":"https://github.com/MrAlias/druid","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrAlias%2Fdruid","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrAlias%2Fdruid/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrAlias%2Fdruid/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrAlias%2Fdruid/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MrAlias","download_url":"https://codeload.github.com/MrAlias/druid/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224993016,"owners_count":17403943,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T23:00:55.465Z","updated_at":"2024-11-18T09:32:07.056Z","avatar_url":"https://github.com/MrAlias.png","language":"Puppet","funding_links":[],"categories":[],"sub_categories":[],"readme":"# druid\n\n#### Table of Contents\n\n1. [Module Description](#module-description)\n2. [Setup](#setup)\n    * [What Druid Affects](#what-druid-affects)\n    * [What Druid Does Not Manage](#what-druid-does-not-manage)\n    * [Setup Requirements](#setup-requirements)\n    * [Beginning With Druid](#beginning-with-druid)\n3. [Usage](#usage)\n    * [Standalone Druid Packages](#standalone-druid-packages)\n    * [Hiera Support](#hiera-support)\n    * [Historical Node](#historical-node)\n4. [Reference](#reference)\n    * [Private Classes](#private-classes)\n       * [druid::indexing](#druidindexing)\n    * [Public Classes](#public-classes)\n       * [druid](#druid-1)\n       * [druid::broker](#druidbroker)\n       * [druid::coordinator](#druidcoordinator)\n       * [druid::historical](#druidhistorical)\n       * [druid::indexing::overlord](#druidindexingoverlord)\n       * [druid::indexing::middle_manager](#druidindexingmiddle_manager)\n       * [druid::realtime](#druidrealtime)\n    * [Private Defined Types](#private-defined-types)\n       * [druid::service](#druidservice)\n5. [Limitations](#limitations)\n\n## Module Description\n\n[Druid](druid.io) is a data store solution designed for online analytical processing of time-series data.  This module is used to manage the Druid services used throughout a Druid cluster, namely the Historical, Broker, Coordinator, Indexing Services, and Realtime services.\n\n## Setup\n\n### What Druid Affects\n\n* Installs the Druid jars to the local filesystem.\n* Created Druid configuration files on the local filesystem.\n* Creates and manages a [systemd](http://www.freedesktop.org/wiki/Software/systemd/) service for each Druid service to run.\n\n### What Druid Does Not Manage\n\nThis module will not setup all the additional services a full Druid cluster will need.  Specificaly it does not manage the following.\n\n* Cluster deep storage\n* Metadata storage\n* ZooKeeper\n\n### Setup Requirements\n\n* Puppet \u003e= 3.x  (It might work with a previous version, but no guarantees are made).\n* Ruby \u003e= 1.9.3\n* [systemd](http://www.freedesktop.org/wiki/Software/systemd/)\n\n### Beginning With Druid\n\nTo install the Druid jars and setup basic configurations the following is all you need.\n\n```puppet\ninclude druid\n```\n\n## Usage\n\nThe module is designed to be called differently depending on the type of Druid service the node should be running.\n\n### Standalone Druid Packages\n\nIf you want to run the druid service on your own the base `druid` class can be used to maintain settings. The following will install the Druid-0.8.0 release in `/opt/`, create a symlink from the installed directory to `/opt/druid/`, and then install a basic common properties file in `/opt/druid/configs/`\n\n```puppet\nclass { 'druid':\n  version     =\u003e '0.8.0',\n  install_dir =\u003e '/opt',\n  config_dir  =\u003e '/opt/druid/configs',\n}\n```\n\n### Hiera Support\n\nAll parameters for all druid classes are definable in hiera.  The following hiera excerpt shows how the [above example](#standalone-druid-packages) could have been accomplished by defining the parameters in hiera.\n\n```yaml\ndruid::version: '0.8.0',\ndruid::install_dir: '/opt'\ndruid::config_dir: '/opt/druid/configs'\n```\n\n### Historical Node\n\nThe `druid::historical` class is used to setup and manage Druid as a historical node.\n\n```puppet\nclass { 'druid::historical':\n  host =\u003e $::ipaddress_lo,\n  port =\u003e 8080,\n}\n```\n\nIf you need to manage common configuration options you can additionally call the base `druid` class.\n\n```puppet\nclass { 'druid':\n  storage_type        =\u003e 's3',\n  s3_access_key       =\u003e 'some_access_key',\n  s3_secret_key       =\u003e 'super_secret_fake_key',\n  s3_bucket           =\u003e 'druid',\n  s3_base_key         =\u003e 'test-druid',\n  s3_archive_bucket   =\u003e 'druid-archive',\n  s3_archive_base_key =\u003e 'test_archive_base_key',\n}\n```\n\nHowever, a simpler way to accomplish this would be to define the whole configuration via hiera.\n\n```yaml\ndruid::storage_type: 's3'\ndruid::s3_access_key: 'some_access_key'\ndruid::s3_secret_key: 'super_secret_fake_key'\ndruid::s3_bucket: 'druid'\ndruid::s3_base_key: 'test-druid'\ndruid::s3_archive_bucket: 'druid-archive'\ndruid::s3_archive_base_key: 'test_archive_base_key'\ndruid::historical::host: %{::ipaddress_lo}\ndruid::historical::port: 8080\n```\n\n## Reference\n\n### Private Classes\n\nThese classes should not be called outside of the module.\n\n#### druid::indexing\n\nProvides management of all common indexing related configurations.\n\n##### `druid::indexing::azure_logs_container`\n\nThe Azure Blob Store container to store logs at. \n\n##### `druid::indexing::azure_logs_prefix`\n\nThe path to prepend to logs for Azure storage. \n\n##### `druid::indexing::hdfs_logs_directory`\n\nThe HDFS directory to store logs. \n\n##### `druid::indexing::local_logs_directory`\n\nLocal filesystem path to store logs. \n\nDefault value: `'/var/log'`\n\n##### `druid::indexing::logs_type`\n\nWhere to store task logs. \n\nValid values: `'noop'`, `'s3'`, `'azure'`, `'hdfs'`, `'file'`. \n\nDefault value: `'file'`. \n\n##### `druid::indexing::s3_logs_bucket`\n\nS3 bucket name to store logs at. \n\n##### `druid::indexing::s3_logs_prefix`\n\nS3 key prefix. \n\n### Public Classes\n\n#### druid\n\nThe main module class.  This class is not intended to be called directly. All parameters a expected to be configured via hiera, but support is still provided to directly configure this class as a puppet class resource directly.\n\n##### `druid::announcer_max_bytes_per_node`\n\nMax byte size for Znode. \n\nDefault value: `524288`.\n\n##### `druid::announcer_segments_per_node`\n\nEach Znode contains info for up to this many segments. \n\nDefault value: `50`. \n\n##### `druid::announcer_type`\n\nThe type of data segment announcer to use. \n\nValid values: `'legacy'` or `'batch'`. \n\nDefault value: `'batch'`. \n\n##### `druid::cache_expiration`\n\nMemcached expiration time. \n\nDefault value: `2592000` (30 days). \n\n##### `druid::cache_hosts`\n\nArray of Memcached hosts (`'host:port'`). \n\nDefault value: `[]`. \n\n##### `druid::cache_initial_size`\n\nInitial size of the hashtable backing the local cache. \n\nDefault value: `500000`. \n\n##### `druid::cache_log_eviction_count`\n\nIf non-zero, log local cache eviction for that many items. \n\nDefault value: `0`. \n\n##### `druid::cache_max_object_size`\n\nMaximum object size in bytes for a Memcached object. \n\nDefault value: `52428800` (50 MB). \n\n##### `druid::cache_memcached_prefix`\n\nKey prefix for all keys in Memcached. \n\nDefault value: `'druid'`. \n\n##### `druid::cache_size_in_bytes`\n\nMaximum local cache size in bytes. Zero disables caching. \n\nDefault value: `0`. \n\n##### `druid::cache_timeout`\n\nMaximum time in milliseconds to wait for a response from Memcached. \n\nDefault value: `500`. \n\n##### `druid::cache_type`\n\nThe type of cache to use for queries. \n\nValid values: `'local'` or `'memcached'`. \n\nDefault value: `'local'`. \n\n##### `druid::cache_uncacheable`\n\nAll query types to not cache. \n\n##### `druid::cassandra_host`\n\nCassandra host to access for deep storage. \n\n##### `druid::cassandra_keyspace`\n\nCassandra key space. \n\n##### `druid::config_dir`\n\nDirectory druid will keep configuration files. \n\nDefault value: `'/etc/druid'`. \n\n##### `druid::curator_compress`\n\nBoolean flag for whether or not created Znodes should be compressed. \n\nDefault value: `true`. \n\n##### `druid::discovery_curator_path`\n\nServices announce themselves under this ZooKeeper path. \n\nDefault value: `'/druid/discovery'`. \n\n##### `druid::emitter_http_flush_count`\n\nHow many messages can the internal message buffer hold before flushing (sending). \n\nDefault value: `500`. \n\n##### `druid::emitter_http_flush_millis`\n\nHow often to internal message buffer is flushed (data is sent). \n\nDefault value: `60000`. \n\n##### `druid::emitter_http_recipient_base_url`\n\nThe base URL to emit messages to. Druid will POST JSON to be consumed at the HTTP endpoint specified by this property. \n\nDefault value: `''`. \n\n##### `druid::emitter_http_time_out`\n\nThe timeout for data reads. \n\nDefault value: `'PT5M'`. \n\n##### `druid::emitter_logging_log_level`\n\nThe log level at which message are logged. \n\nValid values: `'debug'`, `'info'`, `'warn'`, or `'error'`\n\nDefault value: `'info'`. \n\n##### `druid::emitter_logging_logger_class`\n\nThe class used for logging. \n\nValid values: `'HttpPostEmitter'`, `'LoggingEmitter'`, `'NoopServiceEmitter'`, or `'ServiceEmitter'`\n\nDefault value: `'LoggingEmitter'`. \n\n##### `druid::emitter`\n\nEmitter module to use. \n\nValid values: `'noop'`, `'logging'`, or `'http'`. \n\nDefault value: `'logging'`. \n\n##### `druid::extensions_coordinates`\n\nArray of \"groupId:artifactId[:version]\" maven coordinates. \n\nDefault value: `[]`. \n\n##### `druid::extensions_default_version`\n\nVersion to use for extension artifacts without version information. \n\n##### `druid::extensions_local_repository`\n\nThe way maven gets dependencies is that it downloads them to a \"local repository\" on your local disk and then collects the paths to each of the jars. This specifies the directory to consider the \"local repository\". If this is set, `extensions_remote_repositories` is not required. \n\nDefault value: `'~/.m2/repository'`. \n\n##### `druid::extensions_remote_repositories`\n\nArray of remote repositories to load dependencies from. \n\nDefault value: `['http://repo1.maven.org/maven2/', 'https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local', ]`. \n\n##### `druid::extensions_search_current_classloader`\n\nThis is a boolean flag that determines if Druid will search the main classloader for extensions. It defaults to true but can be turned off if you have reason to not automatically add all modules on the classpath. \n\nDefault value: `true`. \n\n##### `druid::hdfs_directory`\n\nHDFS directory to use as deep storage. \n\n##### `druid::install_dir`\n\nDirectory druid will be installed in. \n\nDefault value: `'/usr/local/lib'`. \n\n##### `druid::java_pkg`\n\nName of the java package to ensure installed on system. \n\nDefault value: `'openjdk-7-jre-headless'`. \n\n##### `druid::metadata_storage_connector_create_tables`\n\nSpecifies to create tables in the metadata storage if they do not exits. \n\nDefault value: `true`. \n\n##### `druid::metadata_storage_connector_password`\n\nThe password to connect with. \n\nDefault value: `'insecure_pass'`. \n\n##### `druid::metadata_storage_connector_uri`\n\nThe URI for the metadata storage. \n\nDefault value: `'jdbc:mysql://localhost:3306/druid?characterEncoding=UTF-8'`\n\n##### `druid::metadata_storage_connector_user`\n\nThe username to connect to the metadata storage with. \n\nDefault value: `'druid'`. \n\n##### `druid::metadata_storage_tables_audit`\n\nThe table to use for audit history of configuration changes. \n\nDefault value: `'druid_audit'`. \n\n##### `druid::metadata_storage_tables_base`\n\nThe base name for tables. \n\nDefault value: `'druid'`. \n\n##### `druid::metadata_storage_tables_config_table`\n\nThe table to use to look for configs. \n\nDefault value: `'druid_config'`. \n\n##### `druid::metadata_storage_tables_rule_table`\n\nThe table to use to look for segment load/drop rules. \n\nDefault value: `'druid_rules'`. \n\n##### `druid::metadata_storage_tables_segment_table`\n\nThe table to use to look for segments. \n\nDefault value: `'druid_segments'`. \n\n##### `druid::metadata_storage_tables_task_lock`\n\nUsed by the indexing service to store task locks. \n\nDefault value: `'druid_taskLock'`. \n\n##### `druid::metadata_storage_tables_task_log`\n\nUsed by the indexing service to store task logs. \n\nDefault value: `'druid_taskLog'`. \n\n##### `druid::metadata_storage_tables_tasks`\n\nThe table used by the indexing service to store tasks. \n\nDefault value: `'druid_tasks'`. \n\n##### `druid::metadata_storage_type`\n\nThe type of metadata storage to use. \n\nValid values: `'mysql'`, `'postgres'`, or `'derby'`\n\nDefault value: `'mysql'`. \n\n##### `druid::monitoring_emission_period`\n\nHow often metrics are emitted. \n\nDefault value: `'PT1m'`. \n\n##### `druid::monitoring_monitors`\n\nArray of Druid monitors used by a node. See below for names and more information. \n\nValid array values:\n\n| Array Value | Description |\n| --- | --- |\n| `'io.druid.client.cache.CacheMonitor'` | Emits metrics (to logs) about the segment results cache for Historical and Broker nodes. Reports typical cache statistics include hits, misses, rates, and size (bytes and number of entries), as well as timeouts and errors. |\n| `'com.metamx.metrics.SysMonitor'` | This uses the SIGAR library to report on various system activities and statuses. Make sure to add the sigar library jar to your classpath if using this monitor. |\n| `'io.druid.server.metrics.HistoricalMetricsMonitor'` | Reports statistics on Historical nodes. |\n| `'com.metamx.metrics.JvmMonitor'` | Reports JVM-related statistics. |\n| `'io.druid.segment.realtime.RealtimeMetricsMonitor'` | Reports statistics on Realtime nodes. |\n\nDefault value: `[]`. \n\n##### `druid::request_logging_dir`\n\nHistorical, Realtime and Broker nodes maintain request logs of all of the requests they get (interacton is via POST, so normal request logs don’t generally capture information about the actual query), this specifies the directory to store the request logs in. \n\nDefault value: `''`. \n\n##### `druid::request_logging_feed`\n\nFeed name for requests. \n\nDefault value: `'druid'`. \n\n##### `druid::request_logging_type`\n\nSpecifies the type of logging used. \n\nValid values: `'noop'`, `'file'`, `'emitter'`. \n\nDefault value: `'noop'`. \n\n##### `druid::s3_access_key`\n\nThe access key to use to access S3. \n\n##### `druid::s3_archive_base_key`\n\nS3 object key prefix for archiving. \n\n##### `druid::s3_archive_bucket`\n\nS3 bucket name for archiving when running the indexing-service archive task. \n\n##### `druid::s3_base_key`\n\nS3 object key prefix for storage. \n\n##### `druid::s3_bucket`\n\nS3 bucket name. \n\n##### `druid::s3_secret_key`\n\nThe secret key to use to access S3. \n\n##### `druid::selectors_indexing_service_name`\n\nThe service name of the indexing service Overlord node. To start the Overlord with a different name, set it with this property. \n\nDefault value: `'druid/overlord'`. \n\n##### `druid::storage_directory`\n\nDirectory on disk to use as deep storage if `$storage_type` is set to `'local'`. \n\nDefault value: `'/tmp/druid/localStorage'`. \n\n##### `druid::storage_disable_acl`\n\nBoolean flag for ACL. \n\nDefault value: `false`. \n\n##### `druid::storage_type`\n\nThe type of deep storage to use. \n\nValid values: `'local'`, `'noop'`, `'s3'`, `'hdfs'`, or `'c*'`\n\nDefault value: `'local'`. \n\n##### `druid::version`\nVersion of druid to install. \n\nDefault value: `'0.8.1'`. \n\n##### `druid::zk_paths_announcements_path`\n\nDruid node announcement path. \n\n##### `druid::zk_paths_base`\n\nBase Zookeeper path. \n\nDefault value: `'/druid'`. \n\n##### `druid::zk_paths_coordinator_path`\n\nUsed by the coordinator for leader election. \n\n##### `druid::zk_paths_indexer_announcements_path`\n\nMiddle managers announce themselves here. \n\n##### `druid::zk_paths_indexer_base`\n\nBase zookeeper path for indexers. \n\n##### `druid::zk_paths_indexer_leader_latch_path`\n\nUsed for Overlord leader election. \n\n##### `druid::zk_paths_indexer_status_path`\n\nParent path for announcement of task statuses. \n\n##### `druid::zk_paths_indexer_tasks_path`\n\nUsed to assign tasks to middle managers. \n\n##### `druid::zk_paths_live_segments_path`\n\nCurrent path for where Druid nodes announce their segments. \n\n##### `druid::zk_paths_load_queue_path`\n\nEntries here cause historical nodes to load and drop segments. \n\n##### `druid::zk_paths_properties_path`\n\nZookeeper properties path. \n\n##### `druid::zk_service_host`\n\nThe ZooKeeper hosts to connect to. \n\nDefault value: `'localhost'`. \n\n##### `druid::zk_service_session_timeout_ms`\n\nZooKeeper session timeout, in milliseconds. \n\nDefault value: `30000`. \n\n#### druid::broker\n\n##### `druid::broker::host`\n\nHost address for the service to listen on.\n\nDefault value: The `$ipaddress` fact.\n\n##### `druid::broker::port`\n\nPort the service listens on.\n\nDefault value: `8082`.\n\n##### `druid::broker::service`\n\nThe name of the service.\n\nThis is used as a dimension when emitting metrics and alerts.  It is used to differentiate between the various services\n\nDefault value: `'druid/broker'`.\n\n##### `druid::broker::balancer_type`\n\nThe way connections to historical nodes are balanced.\n\nValid values:\n* `'random'`: Choose randomly.\n* `'connectionCount'`: Use node with the fewest active connections.\n\nDefault value: `'random'`.\n\n##### `druid::broker::cache_expiration`\n\nMemcached expiration time.\n\nDefault value: `2592000` (30 days).\n\n##### `druid::broker::cache_hosts`\n\nArray of Memcached hosts (i.e. '\u003chost:port\u003e').\n\nDefault value: `[]`.\n\n##### `druid::broker::cache_initial_size`\n\nInitial size of the hashtable backing the cache.\n\nDefault value: `500000`.\n\n##### `druid::broker::cache_log_eviction_count`\n\nIf non-zero, log cache eviction every logEvictionCount items.\n\nDefault value: `0`.\n\n##### `druid::broker::cache_max_object_size`\n\nMaximum object size in bytes for a Memcached object.\n\nDefault value: `52428800` (50 MB).\n\n##### `druid::broker::cache_memcached_prefix`\n\nKey prefix for all keys in Memcached.\n\nDefault value: `'druid'`.\n\n##### `druid::broker::cache_populate_cache`\n\nPopulate the cache on the broker.\n\nValid values: `true`, `false`.\n\nDefault value: `false`.\n\n##### `druid::broker::cache_size_in_bytes`\n\nMaximum cache size in bytes.\n\nZero disables caching.\n\nDefault value: `0`.\n\n##### `druid::broker::cache_timeout`\n\nMaximum time in milliseconds to wait for a response from Memcached.\n\nDefault value: `500`.\n\n##### `druid::broker::cache_type`\n\nType of cache to use for queries.\n\nValid values:\n* `'local'`: Use local file cache.\n* `'memcached'`: Use Memcached.\n\nDefault value: `'local'`.\n\n##### `druid::broker::cache_uncacheable`\n\nAll query types to not cache.\n\nDefault value: `['groupBy', 'select']`.\n\n##### `druid::broker::cache_use_cache`\n\nEnable the cache on the broker.\n\nValid values: `true`, `false`.\n\nDefault value: `false`.\n\n##### `druid::broker::http_num_connections`\n\nConnection pool size.\n\nSpecifically, this is the number of connections the Broker uses to connect to historical and real-time nodes. If there are more queries than this number that all need to speak to the same node, then they will queue up.\n\nDefault value: `5`.\n\n##### `druid::broker::http_read_timeout`\n\nThe timeout for data reads.\n\nDefault value: `'PT15M'`.\n\n##### `druid::broker::jvm_opts`\n\nArray of options to set for the JVM running the service.\n\nDefault value: `['-server', '-Duser.timezone=UTC', '-Dfile.encoding=UTF-8', '-Djava.io.tmpdir=/tmp', '-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager']`\n\n##### `druid::broker::processing_buffer_size_bytes`\n\nBuffer size for the storage of intermediate results.\n\nThe computation engine in both the Historical and Realtime nodes will use a scratch buffer of this size to do all of their intermediate computations off-heap. Larger values allow for more aggregations in a single pass over the data while smaller values can require more passes depending on the query that is being executed.\n\nDefault value: `1073741824` (1GB).\n\n##### `druid::broker::processing_column_cache_size_bytes`\n\nMaximum size in bytes for the dimension value lookup cache.\n\nAny value greater than 0 enables the cache. It is currently disabled by default. Enabling the lookup cache can significantly improve the performance of aggregators operating on dimension values, such as the JavaScript aggregator, or cardinality aggregator, but can slow things down if the cache hit rate is low (i.e. dimensions with few repeating values). Enabling it may also require additional garbage collection tuning to avoid long GC pauses.\n\nDefault value: `0` (disabled).\n\n##### `druid::broker::processing_format_string`\n\nFormat string to name processing threads.\n\nDefault value: `'processing-%s'`.\n\n##### `druid::broker::processing_num_threads`\n\nNumber of processing threads available for processing of segments.\n\nRule of thumb is num\\_cores - 1, which means that even under heavy load there will still be one core available to do background tasks like talking with ZooKeeper and pulling down segments. If only one core is available, this property defaults to the value 1.\n\n##### `druid::broker::query_group_by_max_intermediate_rows`\n\nMaximum number of intermediate rows.\n\nDefault value: `50000`.\n\n##### `druid::broker::query_group_by_max_results`\n\nMaximum number of results.\n\nDefault value: `500000`.\n\n##### `druid::broker::query_group_by_single_threaded`\n\nRun single threaded group by queries.\n\nValid values: `true`, `false`.\n\nDefault value: `false`.\n\n##### `druid::broker::query_search_max_search_limit`\n\nMaximum number of search results to return.\n\nDefault value: `1000`.\n\n##### `druid::broker::retry_policy_num_tries`\n\nNumber of tries.\n\nDefault value: `1`.\n\n##### `druid::broker::select_tier_custom_priorities`\n\nArray of integer priorities to select server tiers with.\n\nDefault value: `[]`.\n\n##### `druid::broker::select_tier`\n\nPreference of tier to select based on priority.\n\nIf segments are cross-replicated across tiers in a cluster, you can tell the broker to prefer to select segments in a tier with a certain priority.\n\nValid values:\n* `'highestPriority'`\n* `'lowestPriority'`\n* `'custom'`\n\nDefault value: `'highestPriority'`.\n\n##### `druid::broker::server_http_max_idle_time`\n\nThe Jetty max idle time for a connection.\n\nDefault value: `'PT5m'`.\n\n##### `druid::broker::server_http_num_threads`\n\nNumber of threads for HTTP requests.\n\nDefault value: `10`.\n\n#### druid::coordinator\n\nSets up configuration and manages the Druid coordinator service.\n\n##### `druid::coordinator::host`\n\nHost address the service listens on.\n\nDefault value: The `$ipaddress` fact.\n\n##### `druid::coordinator::port`\n\nPort the service listens on.\n\nDefault value: `8081`.\n\n##### `druid::coordinator::service`\n\nThe name of the service.\n\nThis is used as a dimension when emitting metrics and alerts.  It is used to differentiate between the various services\n\nDefault value: `'druid/coordinator'`.\n\n##### `druid::coordinator::conversion_on`\n\nSpecify if old segment indexing versions should be converted to the latest.\n\nValid values: `true` or `false`.\n\nDefault value: `false`.\n\n##### `druid::coordinator::jvm_opts`\n\nArray of options to set for the JVM running the service.\n\nDefault value: `[\n    '-server',\n    '-Duser.timezone=UTC',\n    '-Dfile.encoding=UTF-8',\n    '-Djava.io.tmpdir=/tmp',\n    '-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager'\n  ]`\n\n##### `druid::coordinator::load_timeout`\n\nTime before the coordinator assigns a segment to a historical node.\n\nDefault value: `'PT15M'`.\n\n##### `druid::coordinator::manager_config_poll_duration`\n\nTime between polls of the config table for updates.\n\nDefault value: `'PT1m'`.\n\n##### `druid::coordinator::manager_rules_alert_threshold`\n\nThe duration after a failed poll upon which an alert should be emitted.\n\nDefault value: `'PT10M'`.\n\n##### `druid::coordinator::manager_rules_default_tier`\n\nThe default tier which default rules will be loaded from.\n\nDefault value: `'_default'`.\n\n##### `druid::coordinator::manager_rules_poll_duration`\n\nTime between polls of the set of active rules for updates.\n\nDefines the amount of lag time it can take for the coordinator to notice rule changes.\n\nDefault value: `'PT1M'`.\n\n##### `druid::coordinator::manager_segment_poll_duration`\n\nTime between polls of the set of active segments for updates.\n\nDefines the amount of lag time it can take for the coordinator to notice segment changes.\n\nDefault value: `'PT1M'`.\n\n##### `druid::coordinator::merge_on`\n\nSpecify if the coordinator should try and merge small segments.\n\nThis helps keep segements in a more optimal segment size.\n\nDefault value: `false`.\n\n##### `druid::coordinator::period`\n\nThe run period for the coordinator.\n\nThe coordinator’s operates by maintaining the current state of the world in memory and periodically looking at the set of segments available and segments being served to make decisions about whether any changes need to be made to the data topology. This parameter sets the delay between each of these runs.\n\nDefault value: `'PT60S'`.\n\n##### `druid::coordinator::period_indexing_period`\n\nHow often to send indexing tasks to the indexing service.\n\nOnly applies if merge or conversion is turned on.\n\nDefault value: `'PT1800S'` (30 mins).\n\n##### `druid::coordinator::start_delay`\n\nTime to let the service think it has all data.\n\nThe operation of the Coordinator works on the assumption that it has an up-to-date view of the state of the world when it runs, the current ZK interaction code, however, is written in a way that doesn’t allow the Coordinator to know for a fact that it’s done loading the current state of the world. This delay is a hack to give it enough time to believe that it has all the data.\n\nDefault value: `'PT300S'`.\n\n#### druid::historical\n\nHistorical node class.\n\n##### `druid::historical::host`\n\nThe host for the current node. \n\nDefault value: The `$ipaddress` fact. \n\n##### `druid::historical::port`\n\nThis is the port to actually listen on; unless port mapping is used, this will be the same port as is on `druid::historical::host`. \n\nDefault value: `8083`. \n\n##### `druid::historical::service`\n\nThe name of the service. This is used as a dimension when emitting metrics and alerts to differentiate between the various services\n\nDefault value: `'druid/historical'`. \n\n##### `druid::historical::jvm_opts`\n\nArray of options to set for the JVM running the service.\n\nDefault value: `['-server', '-Duser.timezone=UTC', '-Dfile.encoding=UTF-8', '-Djava.io.tmpdir=/tmp', '-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager']`\n\n##### `druid::historical::populate_cache`\n\nPopulate the cache on the historical. \n\nDefault value: `false`. \n\n##### `druid::historical::processing_buffer_size_bytes`\n\nThis specifies a buffer size for the storage of intermediate results. The computation engine in both the Historical and Realtime nodes will use a scratch buffer of this size to do all of their intermediate computations off-heap. Larger values allow for more aggregations in a single pass over the data while smaller values can require more passes depending on the query that is being executed. \n\nDefault value: `1073741824` (1GB). \n\n##### `druid::historical::processing_column_cache_size_bytes`\n\nMaximum size in bytes for the dimension value lookup cache. Any value greater than 0 enables the cache.  Enabling the lookup cache can significantly improve the performance of aggregators operating on dimension values, such as the JavaScript aggregator, or cardinality aggregator, but can slow things down if the cache hit rate is low (i.e. dimensions with few repeating values).  Enabling it may also require additional garbage collection tuning to avoid long GC pauses. \n\nDefault value: `0` (disabled). \n\n##### `druid::historical::processing_format_string`\n\nRealtime and historical nodes use this format string to name their processing threads. \n\nDefault value: `'processing-%s'`. \n\n##### `druid::historical::processing_num_threads`\n\nThe number of processing threads to have available for parallel processing of segments.  Druid will default to using 'num\\_cores - 1 || 1'. \n\n##### `druid::historical::query_groupBy_max_intermediate_rows`\n\nMaximum number of intermediate rows. \n\nDefault value: `50000`. \n\n##### `druid::historical::query_groupBy_max_results`\n\nMaximum number of results. \n\nDefault value: `500000`. \n\n##### `druid::historical::query_groupBy_single_threaded`\n\nRun single threaded group By queries. \n\nDefault value: `false`. \n\n##### `druid::historical::query_search_max_search_limit`\n\nMaximum number of search results to return. \n\nDefault value: `1000`. \n\n##### `druid::historical::segment_cache_announce_interval_millis`\n\nHow frequently to announce segments while segments are loading from cache.  Set this value to zero to wait for all segments to be loaded before announcing. \n\nDefault value: `5000` (5 seconds). \n\n##### `druid::historical::segment_cache_delete_on_remove`\n\nDelete segment files from cache once a node is no longer serving a segment. \n\nDefault value: `true`. \n\n##### `druid::historical::segment_cache_drop_segment_delay_millis`\n\nHow long a node delays before completely dropping segment. \n\nDefault value: `30000` (30 seconds). \n\n##### `druid::historical::segment_cache_info_dir`\n\nHistorical nodes keep track of the segments they are serving so that when the process is restarted they can reload the same segments without waiting for the Coordinator to reassign. This path defines where this metadata is kept. \n\n##### `druid::historical::segment_cache_locations`\n\nSegments assigned to a Historical node are first stored on the local file system (in a disk cache) and then served by the Historical node. These locations define where that local cache resides. \n\nValid values: 'none' (also `undef`) or an absolute file path. \n\n##### `druid::historical::segment_cache_num_loading_threads`\n\nHow many segments to load concurrently from from deep storage. \n\nDefault value: `1`. \n\n##### `druid::historical::server_http_max_idle_time`\n\nThe Jetty max idle time for a connection. \n\nDefault value: `'PT5m'`. \n\n##### `druid::historical::server_http_num_threads`\n\nNumber of threads for HTTP requests. \n\nDefault value: `10`. \n\n##### `druid::historical::server_max_size`\n\nThe maximum number of bytes-worth of segments that the node wants assigned to it. This is not a limit that Historical nodes actually enforces, just a value published to the Coordinator node so it can plan accordingly. \n\nDefault value: `0`. \n\n##### `druid::historical::server_priority`\n\nIn a tiered architecture, the priority of the tier, thus allowing control over which nodes are queried. Higher numbers mean higher priority. The Druid default (no priority) works for architecture with no cross replication (tiers that have no data-storage overlap). Data centers typically have equal priority. \n\nDefault value: `0`. \n\n##### `druid::historical::server_tier`\n\nA string to name the distribution tier that the storage node belongs to.  Many of the rules Coordinator nodes use to manage segments can be keyed on tiers. \n\nDefault value: `'_default_tier'`. \n\n##### `druid::historical::uncacheable`\n\nAll query types to not cache. \n\nDefault value: `[\"groupBy\", \"select\"]`. \n\n##### `druid::historical::use_cache`\n\nEnable the cache on the historical. \n\nDefault value: `false`. \n\n#### druid::indexing::overlord\n\nManages and sets up the indexing overlord.\n\n##### `druid::indexing::overlord::host`\n\nHost address the service listens on.\n\nDefault value: The `$ipaddress` fact.\n\n##### `druid::indexing::overlord::port`\n\nPort the service listens on.\n\nDefault value: `8090`.\n\n##### `druid::indexing::overlord::service`\n\nThe name of the service.\n\nThis is used as a dimension when emitting metrics and alerts.  It is used to differentiate between the various services\n\nDefault value: `'druid/overlord'`.\n\n##### `druid::indexing::overlord::autoscale`\n\nEnable autoscaling.\n\nValid values: `true` or `false`.\n\nDefault value: `false`.\n\n##### `druid::indexing::overlord::autoscale_max_scaling_duration`\n\nTime to wait for a middle manager to appear before giving up.\n\nDefault value: `'PT15M'`.\n\n##### `druid::indexing::overlord::autoscale_num_events_to_track`\n\nThe number of autoscaling events to track.\n\nThis includes node creation and termination.\n\nDefault value: `10`.\n\n##### `druid::indexing::overlord::autoscale_origin_time`\n\nStarting time stamp the terminate period increments upon.\n\nDefault value: `'2012-01-01T00:55:00.000Z'`.\n\n##### `druid::indexing::overlord::autoscale_pending_task_timeout`\n\nTime a task can be \"pending\" before scaling up.\n\nDefault value: `'PT30S'`.\n\n##### `druid::indexing::overlord::autoscale_provision_period`\n\nTime window to check if new middle managers should be added.\n\nDefault value: `'PT1M'`.\n\n##### `druid::indexing::overlord::autoscale_strategy`\n\nStrategy to run when autoscaling is required.\n\nValid values: `'noop'` or `'ec2'`.\n\nDefault value: `'noop'`.\n\n##### `druid::indexing::overlord::autoscale_terminate_period`\n\nTime window to check if middle managers should be removed.\n\nDefault value: `'PT5M'`.\n\n##### `druid::indexing::overlord::autoscale_worker_idle_timeout`\n\nIdle time of a worker before it is considered for termination.\n\nDefault value: `'PT90M'`.\n\n##### `druid::indexing::overlord::autoscale_worker_port`\n\nPort that middle managers will listen on.\n\nDefault value: `8080`.\n\n##### `druid::indexing::overlord::autoscale_worker_version`\n\nNode versions to create.\n\nIf set, the overlord will only create nodes of set version during autoscaling. This overrides dynamic configuration.\n\n##### `druid::indexing::overlord::jvm_opts`\n\nArray of options to set for the JVM running the service.\n\nDefault value: `['-server', '-Duser.timezone=UTC', '-Dfile.encoding=UTF-8', '-Djava.io.tmpdir=/tmp', '-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager']`\n\n##### `druid::indexing::overlord::queue_max_size`\n\nMaximum number of active tasks at one time.\n\n##### `druid::indexing::overlord::queue_restart_delay`\n\nSleep time when queue management throws an exception.\n\nDefault value: `'PT30S'`.\n\n##### `druid::indexing::overlord::queue_start_delay`\n\nSleep time before starting overlord queue management.\n\nThis can be useful to give a cluster time to re-orient itself (e.g.  after a widespread network issue).\n\nDefault value: `'PT1M'`.\n\n##### `druid::indexing::overlord::queue_storage_sync_rate`\n\nRate to sync state with an underlying task persistence mechanism.\n\nDefault value: `'PT1M'`.\n\n##### `druid::indexing::overlord::runner_compress_znodes`\n\nExpect middle managers to compress Znodes.\n\nValid values: `true` or `false`.\n\nDefault value: `true`.\n\n##### `druid::indexing::overlord::runner_max_znode_bytes`\n\nMaximum Znode size in bytes that can be created in Zookeeper.\n\nDefault value: `524288`.\n\n##### `druid::indexing::overlord::runner_min_worker_version`\n\nMinimum middle manager version to send tasks to.\n\nDefault value: `'0'`.\n\n##### `druid::indexing::overlord::runner_task_assignment_timeout`\n\nTime to wait after a task has been assigned to a middle manager.\n\nTasks that take longer then this results in an error being thrown.\n\nDefault value: `'PT5M'`.\n\n##### `druid::indexing::overlord::runner_task_cleanup_timeout`\n\nTime to wait before failing a task when the middle manager assigned the task becomes disconnected from Zookeeper.\n\nDefault value: `'PT15M'`.\n\n##### `druid::indexing::overlord::runner_type`\n\nSpecify if tasks are run locally or in a distributed environment.\n\nValid values: `'local'` or `'remote'`.\n\nDefault value: `'local'`.\n\n##### `druid::indexing::overlord::storage_recently_finished_threshold`\n\nTime to store task results.\n\nDefault value: `'PT24H'`.\n\n##### `druid::indexing::overlord::storage_type`\n\nSpecify where incoming task are stored.\n\nStoring incoming tasks in metadata storage allows for tasks to be resumed if the overlord should fail.\n\nValid values:\n\n    * `'local'`: In heap\n    * `'metadata'`: Metadata storage.\n\nDefault value: `'local'`.\n\n#### druid::indexing::middle\\_manager\n\nManages and sets up the indexing middle managers.\n\n##### `druid::indexing::middle_manager::host`\n\nHost address the service listens on.\n\nDefault value: The `$ipaddress` fact.\n\n##### `druid::indexing::middle_manager::port`\n\nPort the service listens on.\n\nDefault value: `8080`.\n\n##### `druid::indexing::middle_manager::service`\n\nThe name of the service.\n\nThis is used as a dimension when emitting metrics and alerts.  It is used to differentiate between the various services\n\nDefault value: `'druid/middlemanager'`.\n\n\n##### `druid::indexing::middle_manager::fork_properties`\n\nHash of explicit child peon config options.\n\nPeons inherit the configurations of their parent middle managers, but if this is undesired for certain config options they can be explicitly passed here.\n\nThese key value pairs are expected in Druid config format and are unvalidated.  The keys should NOT include `'druid.indexer.fork.property'` as a prefix.\n\nExample:\n\n```puppet\n{\n  \"druid.monitoring.monitors\" =\u003e \"[\\\"com.metamx.metrics.JvmMonitor\\\"]\",\n  \"druid.processing.numThreads\" =\u003e 2,\n}\n```\n\nDefault value: `{}`\n\n##### `druid::indexing::middle_manager::jvm_opts`\n\nArray of options to set for the JVM running the service.\n\nDefault value: `['-server', '-Duser.timezone=UTC', '-Dfile.encoding=UTF-8', '-Djava.io.tmpdir=/tmp', '-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager']`\n\n##### `druid::indexing::middle_manager::peon_mode`\n\nMode peons are run in.\n\nValid values:\n\n    * `'local'`: Standalone mode (Not recommended).\n    * `'remote'`: Pooled.\n\nDefault value: `'remote'`.\n\n##### `druid::indexing::middle_manager::remote_peon_max_retry_count`\n\nMax retries a remote peon makes communicating with the overlord.\n\nDefault value: `10`.\n\n##### `druid::indexing::middle_manager::remote_peon_max_wait`\n\nMax retry time a remote peon makes communicating with the overlord.\n\nDefault value: `'PT10M'`.\n\n##### `druid::indexing::middle_manager::remote_peon_min_wait`\n\nMin retry time a remote peon makes communicating with the overlord.\n\nDefault value: `'PT1M'`.\n\n##### `druid::indexing::middle_manager::runner_allowed_prefixes`\n\nArray of prefixes of configs that are passed down to peons.\n\nDefault value: `['com.metamx', 'druid', 'io.druid', 'user.timezone', 'file.encoding']`.\n\n##### `druid::indexing::middle_manager::runner_classpath`\n\nJava classpath for the peons.\n\n##### `druid::indexing::middle_manager::runner_compress_znodes`\n\nSpecify if Znodes are compressed.\n\nDefault value: `true`.\n\n##### `druid::indexing::middle_manager::runner_java_command`\n\nCommand for peons to use to execute java.\n\nDefault value: `'java'`.\n\n##### `druid::indexing::middle_manager::runner_java_opts`\n\nJava \"-X\" options for the peon to use in its own JVM.\n\n##### `druid::indexing::middle_manager::runner_max_znode_bytes`\n\nMaximum Znode size in bytes that can be created in Zookeeper.\n\nDefault value: `524288`.\n\n##### `druid::indexing::middle_manager::runner_start_port`\n\nPort peons begin running on.\n\nDefault value: `8100`.\n\n##### `druid::indexing::middle_manager::task_base_dir`\n\nBase temporary working directory.\n\nDefault value: `'/tmp'`.\n\n##### `druid::indexing::middle_manager::task_base_task_dir`\n\nBase temporary working directory for tasks.\n\nDefault value: `'/tmp/persistent/tasks'`.\n\n##### `druid::indexing::middle_manager::task_chat_handler_type`\n\nSpecify service discovery type.\n\nCertain tasks will use service discovery to announce an HTTP endpoint that events can be posted to.\n\nValid values: `'noop'` or `'announce'`.\n\nDefault value: `'noop'`.\n\n##### `druid::indexing::middle_manager::task_default_hadoop_coordinates`\n\nArray of default Hadoop version to use.\n\nThis is used with HadoopIndexTasks that do not request a particular version.\n\nDefault value: `['org.apache.hadoop:hadoop-client:2.3.0']`.\n\n##### `druid::indexing::middle_manager::task_default_row_flush_boundary`\n\nHighest row count before persisting to disk.\n\nUsed for indexing generating tasks.\n\nDefault value: `50000`.\n\n##### `druid::indexing::middle_manager::task_hadoop_working_path`\n\nTemporary working directory for Hadoop tasks.\n\nDefault value: `'/tmp/druid-indexing'`.\n\n##### `druid::indexing::middle_manager::worker_capacity`\n\nMaximum number of tasks to accept.\n\n##### `druid::indexing::middle_manager::worker_ip`\n\nThe IP of the worker.\n\nDefault value: `'localhost'`.\n\n##### `druid::indexing::middle_manager::worker_version`\n\nVersion identifier for the middle manager.\n\nDefault value: `'0'`.\n\n#### druid::realtime\n\nSets up configuration and manages the Druid realtime service.\n\n##### `druid::realtime::host`\n\nHost address the service listens on.\n\nDefault value: The `$ipaddress` fact.\n\n##### `druid::realtime::port`\n\nPort the service listens on.\n\nDefault value: `8084`.\n\n##### `druid::realtime::service`\n\nThe name of the service.\n\nThis is used as a dimension when emitting metrics and alerts.  It is used to differentiate between the various services\n\nDefault value: `'druid/realtime'`.\n\n##### `druid::realtime::jvm_opts`\n\nArray of options to set for the JVM running the service.\n\nDefault value: `[\n    '-server',\n    '-Duser.timezone=UTC',\n    '-Dfile.encoding=UTF-8',\n    '-Djava.io.tmpdir=/tmp',\n    '-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager'\n  ]`\n\n##### `druid::realtime::processing_buffer_size_bytes`\n\nBuffer size for the storage of intermediate results.\n\nThe computation engine uses a scratch buffer of this size to do all intermediate computations off-heap. Larger values allow for more aggregations in a single pass over the data while smaller values can require more passes depending on the query that is being executed.\n\nDefault value: `1073741824` (1GB).\n\n##### `druid::realtime::processing_column_cache_size_bytes`\n\nMaximum size in bytes for the dimension value lookup cache.\n\nAny value greater than `0` enables the cache. Enabling the lookup cache can significantly improve the performance of aggregators operating on dimension values, such as the JavaScript aggregator, or cardinality aggregator, but can slow things down if the cache hit rate is low (i.e.  dimensions with few repeating values). Enabling it may also require additional garbage collection tuning to avoid long GC pauses.\n\nDefault value: `0` (disabled).\n\n##### `druid::realtime::processing_format_string`\n\nFormat string to name processing threads.\n\nDefault value: `'processing-%s'`.\n\n##### `druid::realtime::processing_num_threads`\n\nNumber of processing threads for processing of segments.\n\nRule of thumb is num\\_cores - 1, which means that even under heavy load there will still be one core available to do background tasks like talking with ZooKeeper and pulling down segments.\n\n##### `druid::realtime::publish_type`\n\nWhere to publish segments.\n\nValid values: `'noop'` or `'metadata'`.\n\nDefault value: `'metadata'`.\n\n##### `druid::realtime::query_group_by_max_intermediate_rows`\n\nMaximum number of intermediate rows.\n\nDefault value: `50000`.\n\n##### `druid::realtime::query_group_by_max_results`\n\nMaximum number of results.\n\nDefault value: `500000`.\n\n##### `druid::realtime::query_group_by_single_threaded`\n\nRun single threaded `groupBy` queries.\n\nDefault value: `false`.\n\n##### `druid::realtime::query_search_max_search_limit`\n\nMaximum number of search results to return.\n\nDefault value: `1000`.\n\n##### `druid::realtime::segment_cache_locations`\n\nWhere intermediate segments are stored.\n\n##### `druid::realtime::spec_file`\n\nFile location of realtime specFile.\n\n##### `druid::realtime::spec_file_content`\n\nContent to ensure in spec\\_file.\n\n### Private Defined Types\n\n#### druid::service\n\nResource used to setup common files needed for Druic services.\n\n##### `druid::indexing::middle_manager::service_name`\n\nName the service is known by (e.g historical, broker, realtime, ...).\n\nDefault value: `$namevar`\n\n##### `druid::service::config_content`\n\nRequired content of the service properties file.\n\n##### `druid::serviceservice_content`\n\nRequired content of the systemd service file.\n\n## Limitations\n\nThe module has been designed to run on a Debian based system using systemd as a service manager.\n\nTesting is currently only being done on Debian 8 (Jessie).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMrAlias%2Fdruid","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMrAlias%2Fdruid","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMrAlias%2Fdruid/lists"}