Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/voxpupuli/puppet-logstash
Puppet module to manage Logstash
https://github.com/voxpupuli/puppet-logstash
centos-puppet-module debian-puppet-module hacktoberfest linux-puppet-module opensuse-puppet-module oraclelinux-puppet-module puppet redhat-puppet-module scientific-puppet-module ubuntu-puppet-module
Last synced: 3 days ago
JSON representation
Puppet module to manage Logstash
- Host: GitHub
- URL: https://github.com/voxpupuli/puppet-logstash
- Owner: voxpupuli
- License: apache-2.0
- Created: 2012-11-29T14:23:29.000Z (about 12 years ago)
- Default Branch: master
- Last Pushed: 2024-09-17T07:33:02.000Z (5 months ago)
- Last Synced: 2025-01-30T19:48:49.541Z (10 days ago)
- Topics: centos-puppet-module, debian-puppet-module, hacktoberfest, linux-puppet-module, opensuse-puppet-module, oraclelinux-puppet-module, puppet, redhat-puppet-module, scientific-puppet-module, ubuntu-puppet-module
- Language: Ruby
- Homepage: https://forge.puppet.com/elastic/logstash
- Size: 1.6 MB
- Stars: 191
- Watchers: 282
- Forks: 298
- Open Issues: 30
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# logstash
[](https://github.com/voxpupuli/puppet-logstash/actions?query=workflow%3ACI)
[](https://github.com/voxpupuli/puppet-logstash/actions/workflows/release.yml)
[](https://forge.puppetlabs.com/puppet/logstash)
[](https://forge.puppetlabs.com/puppet/logstash)
[](https://forge.puppetlabs.com/puppet/logstash)
[](https://forge.puppetlabs.com/puppet/logstash)
[](http://www.puppetmodule.info/m/puppet-logstash)
[](LICENSE)
[](#transfer-notice)
A Puppet module for managing and configuring [Logstash](https://www.elastic.co/logstash/).
Version 7 and newer of this module are released by Vox Pupuli. They now follow semantic versioning. Previously the module was maintained by Elastic.
## Requirements
* The [stdlib](https://forge.puppetlabs.com/puppetlabs/stdlib) module.
* Logstash < 7.0.0 requires Java. The [puppetlabs/java](https://forge.puppetlabs.com/modules/puppetlabs/java) module is recommended
for installing Java.
Optional:
* The [elastic_stack](https://forge.puppetlabs.com/elastic/elastic_stack) module
when using automatic repository management.
* The [apt](https://forge.puppetlabs.com/puppetlabs/apt) (>= 2.0.0) module when
using repo management on Debian/Ubuntu.
* The [zypprepo](https://forge.puppetlabs.com/darin/zypprepo) module when using
repo management on SLES/SuSE.
## Quick Start
This minimum viable configuration ensures that Logstash is installed, enabled, and running:
``` puppet
include logstash
# You must provide a valid pipeline configuration for the service to start.
logstash::configfile { 'my_ls_config':
content => template('path/to/config.file'),
}
```
## Package and service options
### Choosing a Logstash major version
This module uses the related "elastic/elastic_stack" module to manage package
repositories. Since there is a separate repository for each major version of
the Elastic stack, if you don't want the default version, it's necessary
to select which version to configure, like this:
``` puppet
class { 'elastic_stack::repo':
version => 6,
}
include logstash
```
### Choosing a Logstash minor version
``` puppet
class { 'logstash':
version => '6.8.0',
}
```
### Manual repository management
You may want to manage repositories manually. You can disable
automatic repository management like this:
``` puppet
class { 'logstash':
manage_repo => false,
}
```
### Using an explicit package source
Rather than use your distribution's repository system, you can specify an
explicit package to fetch and install.
#### From an HTTP/HTTPS/FTP URL
``` puppet
class { 'logstash':
package_url => 'https://artifacts.elastic.co/downloads/logstash/logstash-7.17.8-x86_64.rpm',
}
```
#### From a 'puppet://' URL
``` puppet
class { 'logstash':
package_url => 'puppet:///modules/my_module/logstash-7.17.8-x86_64.rpm',
}
```
#### From a local file on the agent
``` puppet
class { 'logstash':
package_url => 'file:///tmp/logstash-7.17.8-x86_64.rpm',
}
```
### Allow automatic point-release upgrades
``` puppet
class { 'logstash':
auto_upgrade => true,
}
```
### Use a different logstash home
``` puppet
class { 'logstash':
home_dir => '/opt/logstash',
}
```
### Do not run as a service
``` puppet
class { 'logstash':
status => 'disabled',
}
```
### Disable automatic restarts
Under normal circumstances, changing a configuration will trigger a restart of
the service. This behaviour can be disabled:
``` puppet
class { 'logstash':
restart_on_change => false,
}
```
### Disable and remove Logstash
``` puppet
class { 'logstash':
ensure => 'absent',
}
```
## Logstash config files
### Settings
Logstash uses several files to define settings for the service and associated
Java runtime. The settings files can be configured with class parameters.
#### `logstash.yml` with flat keys
``` puppet
class { 'logstash':
settings => {
'pipeline.batch.size' => 25,
'pipeline.batch.delay' => 5,
}
}
```
#### `logstash.yml` with nested keys
``` puppet
class { 'logstash':
settings => {
'pipeline' => {
'batch' => {
'size' => 25,
'delay' => 5,
}
}
}
}
```
#### `jvm.options`
``` puppet
class { 'logstash':
jvm_options => [
'-Xms1g',
'-Xmx1g',
]
}
```
#### `startup.options`
``` puppet
class { 'logstash':
startup_options => {
'LS_NICE' => '10',
}
}
```
#### `pipelines.yml`
``` puppet
class { 'logstash':
pipelines => [
{
"pipeline.id" => "pipeline_one",
"path.config" => "/usr/local/etc/logstash/pipeline-1/one.conf",
},
{
"pipeline.id" => "pipeline_two",
"path.config" => "/usr/local/etc/logstash/pipeline-2/two.conf",
}
]
}
```
Note that specifying `pipelines` will automatically remove the default
`path.config` setting from `logstash.yml`, since this is incompatible with
`pipelines.yml`.
Enabling centralized pipeline management with `xpack.management.enabled` will
also remove the default `path.config`.
### Pipeline Configuration
Pipeline configuration files can be declared with the `logstash::configfile`
type.
``` puppet
logstash::configfile { 'inputs':
content => template('path/to/input.conf.erb'),
}
```
or
``` puppet
logstash::configfile { 'filters':
source => 'puppet:///path/to/filter.conf',
}
```
For simple cases, it's possible to provide your Logstash config as an
inline string:
``` puppet
logstash::configfile { 'basic_ls_config':
content => 'input { heartbeat {} } output { null {} }',
}
```
You can also specify the exact path for the config file, which is
particularly useful with multiple pipelines:
``` puppet
logstash::configfile { 'config_for_pipeline_two':
content => 'input { heartbeat {} } output { null {} }',
path => '/usr/local/etc/logstash/pipeline-2/two.conf',
}
```
If you want to use Hiera to specify your configs, include the following
create_resources call in your manifest:
``` puppet
create_resources('logstash::configfile', hiera('my_logstash_configs'))
```
...and then create a data structure like this in Hiera:
``` yaml
---
my_logstash_configs:
nginx:
template: site_logstash/nginx.conf.erb
syslog:
template: site_logstash/syslog.conf.erb
```
In this example, templates for the config files are stored in the custom,
site-specific module "`site_logstash`".
### Patterns
Many plugins (notably [Grok](http://logstash.net/docs/latest/filters/grok)) use *patterns*. While many are included in Logstash already, additional site-specific patterns can be managed as well.
``` puppet
logstash::patternfile { 'extra_patterns':
source => 'puppet:///path/to/extra_pattern',
}
```
By default the resulting filename of the pattern will match that of the source. This can be over-ridden:
``` puppet
logstash::patternfile { 'extra_patterns_firewall':
source => 'puppet:///path/to/extra_patterns_firewall_v1',
filename => 'extra_patterns_firewall',
}
```
**IMPORTANT NOTE**: Using logstash::patternfile places new patterns in the correct directory, however, it does NOT cause the path to be included automatically for filters (example: grok filter). You will still need to include this path (by default, /etc/logstash/patterns/) explicitly in your configurations.
Example: If using 'grok' in one of your configurations, you must include the pattern path in each filter like this:
```
# Note: this example is Logstash configuration, not a Puppet resource.
# Logstash and Puppet look very similar!
grok {
patterns_dir => "/etc/logstash/patterns/"
...
}
```
## Plugin management
### Installing by name (from RubyGems.org)
``` puppet
logstash::plugin { 'logstash-input-beats': }
```
### Installing from a local Gem
``` puppet
logstash::plugin { 'logstash-input-custom':
source => '/tmp/logstash-input-custom-0.1.0.gem',
}
```
### Installing from a 'puppet://' URL
``` puppet
logstash::plugin { 'logstash-filter-custom':
source => 'puppet:///modules/my_ls_module/logstash-filter-custom-0.1.0.gem',
}
```
### Installing from an 'http(s)://' URL
``` puppet
logstash::plugin { 'x-pack':
source => 'https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.3.0.zip',
}
```
### Controling the environment for the `logstash-plugin` command
``` puppet
logstash::plugin { 'logstash-input-websocket':
environment => 'LS_JAVA_OPTS=-Xms1g -Xmx1g',
}
```
## Transfer Notice
This module was originally authored by [Elastic](https://www.elastic.co).
The maintainer preferred that Vox Pupuli take ownership of the module for future improvement and maintenance.
Existing pull requests and issues were transferred over, please fork and continue to contribute here instead of Elastic.