Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/deric/puppet-zookeeper

Puppet module for managing Apache ZooKeeper
https://github.com/deric/puppet-zookeeper

puppet zookeeper

Last synced: 3 days ago
JSON representation

Puppet module for managing Apache ZooKeeper

Awesome Lists containing this project

README 

        
# puppet-zookeeper



[![Puppet

Forge](http://img.shields.io/puppetforge/v/deric/zookeeper.svg)](https://forge.puppetlabs.com/deric/zookeeper) [![Static & Spec Tests](https://github.com/deric/puppet-zookeeper/actions/workflows/spec.yml/badge.svg)](https://github.com/deric/puppet-zookeeper/actions/workflows/spec.yml) [![Puppet Forge

Downloads](http://img.shields.io/puppetforge/dt/deric/zookeeper.svg)](https://forge.puppetlabs.com/deric/zookeeper/scores)



A puppet receipt for [Apache Zookeeper](http://zookeeper.apache.org/). ZooKeeper is a high-performance coordination service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.



## Requirements



  * Puppet

  * Binary or ZooKeeper source code archive



## Basic Usage:



```puppet

class { 'zookeeper': }

```



## Cluster setup



When running ZooKeeper in the distributed mode each node must have unique ID (`1-255`). The easiest way how to setup multiple ZooKeepers, is by using Hiera.



`hiera/host/zk1.example.com.yaml`:

```yaml

zookeeper::id: '1'

```

`hiera/host/zk2.example.com.yaml`:

```yaml

zookeeper::id: '2'

```

`hiera/host/zk3.example.com.yaml`:

```yaml

zookeeper::id: '3'

```

A ZooKeeper quorum should consist of odd number of nodes (usually `3` or `5`).

For defining a quorum it is enough to list all IP addresses of all its members.



```puppet

class { 'zookeeper':

  servers => {

    1 => '192.168.1.1',

    2 => '192.168.1.2',

    3 => '192.168.1.3',

  },

}

```

In case that an array is passed as `servers`, first ZooKeeper will be assigned `ID = 1`. This would produce following configuration:

```

server.1=192.168.1.1:2888:3888

server.2=192.168.1.2:2888:3888

server.3=192.168.1.3:2888:3888

```

where first port is `election_port` and second one `leader_port`. Both ports could be customized for each ZooKeeper instance.



```puppet

class { 'zookeeper':

  election_port => 2889,

  leader_port   => 3889,

  servers       => {

    1 => '192.168.1.1',

    2 => '192.168.1.2',

    3 => '192.168.1.3',

  }

}

```



### Observers



[Observers](http://zookeeper.apache.org/doc/r3.3.0/zookeeperObservers.html) were introduced in ZooKeeper 3.3.0. To enable this feature simply state which of ZooKeeper servers are observing:



```puppet

class { 'zookeeper':

  servers   => ['192.168.1.1', '192.168.1.2', '192.168.1.3', '192.168.1.4', '192.168.1.5'],

  observers => ['192.168.1.4', '192.168.1.5']

}

```

**Note**: Currently observer server needs to be listed between standard servers (this behavior might change in feature).



### Set binding interface



By default ZooKeeper should bind to all interfaces. When you specify `client_ip` only single interface

will be used. If `$::ipaddress` is not your public IP (e.g. you are using Docker) make sure to setup correct IP:



```puppet

class { 'zookeeper':

  client_ip => $::ipaddress_eth0

}

```



or in Hiera:



```yaml

zookeeper::client_ip: "%{::ipaddress_eth0}"

```



This is a workaround for a a [Facter issue](https://tickets.puppetlabs.com/browse/FACT-380).



### ZooKeeper service



Use `service_provider` to override Puppet detection for starting service.



```puppet

class { 'zookeeper':

  service_provider    => 'init',

  manage_service_file => false,

}

```



Some reasonable values are:



  * `init`

  * `systemd`

  * `runit`

  * `exhibitor` - zookeeper process and config will be managed by exhibitor (https://github.com/soabase/exhibitor). Exhibitor is not managed by this module.

  * `none` - service won't be installed



Parameter `manage_service_file` controls whether service definition should be managed by Puppet (default: `false`). Currently supported for `systemd` and `init`.



### Systemd Unit 'After' and 'Want' control

By default the module will create the following Unit section in /etc/systemd/system/multi-user.target.wants/zookeeper.service

````

[Unit]

Description=Apache ZooKeeper

After=network.target

````



Both After and Want (omitted when using the module defaults) can be controled using this module.



E.g on CentOS 7 those might have to be configured for 'netwrok-online.target' using the following syntax:



```puppet

class { 'zookeeper':

   systemd_unit_after => 'network-online.target',

   systemd_unit_want => 'network-online.target',

}

```



Which will modify the Unit section to look like:



````

[Unit]

Description=Apache ZooKeeper

Want=network-online.target

After=network-online.target

````



##  Parameters



   - `id` - cluster-unique zookeeper's instance id (1-255)

   - `datastore`

   - `datalogstore` - Defining `dataLogDir` allows ZooKeeper transaction logs to be stored in a different location, might improve I/O performance (e.g. if path is mounted on dedicated disk)

   - `log_dir`

   - `purge_interval` - automatically will delete ZooKeeper logs (available since ZooKeeper 3.4.0)

   - `snap_retain_count` - number of snapshots that will be kept after purging (since ZooKeeper 3.4.0)

   - `min_session_timeout` - the minimum session timeout in milliseconds that the server will allow the client to negotiate. Defaults to 2 times the **tickTime** (since ZooKeeper 3.3.0)

   - `max_session_timeout` - the maximum session timeout in milliseconds that the server will allow the client to negotiate. Defaults to 20 times the **tickTime** (since ZooKeeper 3.3.0)

   - `global_outstanding_limit` - ZooKeeper will throttle clients so that there is no more than `global_outstanding_limit` outstanding requests in the system.

   - `manage_service` (default: `true`) whether Puppet should ensure running service

   - `manage_service_file` when enabled on RHEL 7.0 a systemd config will be managed

   - `ensure_account` controls whether `zookeeper` user and group will be ensured (set to `false` to disable this feature)

   - `install_method` controls whether ZooKeeper is installed from binary (`package`) or source (`archive`) packages

   - `archive_version` allows to specify an arbitrary version of ZooKeeper when using source packages

   - `archive_install_dir` controls the installation directory when using source packages (defaults to `/opt`)

   - `archive_symlink` controls the name of a version-independent symlink when using source packages

   - `archive_dl_url` allows to change the download URL for source packages (defaults to apache.org)

   - `systemd_path` where to put `systemd` service files (applies only if `manage_service_file` and `service_provider == 'systemd'`)

   - `restart_on_change` whether ZooKeeper service should be restarted on configuration files change (default: `true`)

   - `remove_host_principal` whether to remove host from Kerberos principal (default: `false`)

   - `remove_realm_principal` whether to remove relam from Kerberos principal (default: `false`)

   - `whitelist_4lw` Fine grained control over the set of commands ZooKeeper can execute (an array e.g. `whitelist_4lw = ['*']`)



and many others, see the `params.pp` file for more details.



If your distribution has multiple packages for ZooKeeper, you can provide all package names

as an array.



```puppet

class { 'zookeeper':

  packages => ['zookeeper', 'zookeeper-java']

}

```

## Logging



ZooKeeper uses log4j, following variables can be configured:



```puppet

class { 'zookeeper':

  console_threshold     => 'INFO',

  rollingfile_threshold => 'INFO',

  tracefile_threshold   => 'TRACE',

  maxfilesize           => '256MB',

  maxbackupindex        => 20,

}

```

Threshold supported values are: `ALL`, `DEBUG`, `ERROR`, `FATAL`, `INFO`, `OFF`, `TRACE` and `WARN`.



[Maxfilesize](https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/RollingFileAppender.html#maxFileSize)



[MaxBackupIndex](https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/RollingFileAppender.html#maxBackupIndex)



By default console, rolling file and trace logging can be configured. Additional log appenders (vulgo log methods) can be configured

by adding a hash `extra_appenders`. The following sets up syslog logging and points the root logger towards syslog (note that you must

have syslog listening on port 514/udp for this to work):



```puppet

class { 'zookeeper':

  log4j_prop      => 'INFO,SYSLOG',

  extra_appenders => {

    'Syslog' => {

      'class'                    => 'org.apache.log4j.net.SyslogAppender',

      'layout'                   => 'org.apache.log4j.PatternLayout',

      'layout.conversionPattern' => "${hostname} zookeeper[id:%X{myid}] - %-5p [%t:%C{1}@%L][%x] - %m%n",

      'syslogHost'               => 'localhost',

      'facility'                 => 'user',

    },

  },

}

```



## Hiera Support



All parameters could be defined in hiera files, e.g. `common.yaml`, `Debian.yaml` or `zookeeper.yaml`:



```yaml

zookeeper::id: 1

zookeeper::client_port: 2181

zookeeper::datastore: '/var/lib/zookeeper'

zookeeper::datalogstore: '/disk2/zookeeper'

```



### Custom RPM repository



Optionally you can specify a custom repository, using a hash configuration.



```puppet

class { 'zookeeper':

  repo       =>  {

    name      => 'myrepo',

    url       => 'http://custom.url',

    descr     => 'description'

    sslverify => 1,

    gpgcheck  => true,

  }

}

```



## Source package



Source packages provide the ability to install arbitrary versions of ZooKeeper on any platform. Note that you'll likely have to use the `manage_service_file` in order to be able to control the ZooKeeper service (because source packages do not install service files).



```puppet

class { 'zookeeper':

  install_method  => 'archive',

  archive_version => '3.4.8',

}

```



Optionally you can specify a `proxy_server`:



```puppet

class { 'zookeeper':

  install_method  => 'archive',

  archive_version => '3.4.8',

  proxy_server    => 'http://10.0.0.1:8080'

}

```



## Java installation



Default: `false`



By changing these two parameters you can ensure, that given Java package will be installed before ZooKeeper packages.



```puppet

class { 'zookeeper':

  install_java => true,

  java_package => 'openjdk-7-jre-headless'

}

```



## Install



### Librarian (recommended)



For [puppet-librarian](https://github.com/rodjek/librarian-puppet) just add to `Puppetfile`



from Forge:

```ruby

mod 'deric-zookeeper'

```



latest (development) version from GitHub

```ruby

mod 'deric-zookeeper', git: 'git://github.com/deric/puppet-zookeeper.git'

```



### submodules



If you are versioning your puppet conf with git just add it as submodule, from your repository root:



    git submodule add git://github.com/deric/puppet-zookeeper.git modules/zookeeper



## Dependencies



  * stdlib `> 2.3.3` - function `ensure_resources` is required

  * puppet-archive `> 0.4.4` - provides capabilities to use archives instead of binary packages



## Acceptance testing



Fastest way is to run tests on prepared Docker images:

```

BEAKER_set=debian9-6.3 bundle exec rake acceptance

```

For examining system state set Beaker's ENV variable `BEAKER_destroy=no`:



```

BEAKER_destroy=no BEAKER_set=default bundle exec rake acceptance

```

and after finishing tests connect to container:

```

docker exec -it adoring_shirley bash

```



When host machine is NOT provisioned (puppet installed, etc.):

```

PUPPET_install=yes BEAKER_set=debian-8 bundle exec rake acceptance

```



Run on specific OS (see `spec/acceptance/nodesets`), to see available sets:

```

rake beaker:sets

```



## Supported platforms



  * Debian/Ubuntu

  * RedHat/CentOS/Fedora/Rocky



### Tested on:



  * Debian (8, 9, 10)

  * Ubuntu (16.04, 18.04)

  * RHEL (6, 7)

  * CentOS (6, 7)

  * SLES (12)