Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/vkhatri/chef-filebeat

Chef Cookbook to Manage Elastic Filebeat https://supermarket.chef.io/cookbooks/filebeat
https://github.com/vkhatri/chef-filebeat

beats chef-cookbook filebeat logging

Last synced: 3 months ago
JSON representation

Chef Cookbook to Manage Elastic Filebeat https://supermarket.chef.io/cookbooks/filebeat

Host: GitHub
URL: https://github.com/vkhatri/chef-filebeat
Owner: vkhatri
License: apache-2.0
Created: 2015-10-22T17:51:13.000Z (over 9 years ago)
Default Branch: master
Last Pushed: 2021-02-06T21:59:41.000Z (almost 4 years ago)
Last Synced: 2024-10-13T19:06:43.689Z (4 months ago)
Topics: beats, chef-cookbook, filebeat, logging
Language: Ruby
Size: 327 KB
Stars: 32
Watchers: 6
Forks: 83
Open Issues: 7
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE

Awesome Lists containing this project

README

filebeat Cookbook
================

[![Cookbook](https://img.shields.io/github/tag/vkhatri/chef-filebeat.svg)](https://github.com/vkhatri/chef-filebeat) [![Build Status](https://travis-ci.org/vkhatri/chef-filebeat.svg?branch=master)](https://travis-ci.org/vkhatri/chef-filebeat)

This is a [Chef] cookbook to manage [Filebeat].

>> For Production environment, always prefer the [most recent release](https://supermarket.chef.io/cookbooks/filebeat).

## Most Recent Release

```ruby
cookbook 'filebeat', '~> 2.4.0'
```

## From Git

```ruby
cookbook 'filebeat', github: 'vkhatri/chef-filebeat', tag: 'v2.4.0'
```

## Repository

```bash
https://github.com/vkhatri/chef-filebeat
```

## Supported OS

- Windows
- Amazon Linux
- CentOS
- Fedora
- Ubuntu
- Debian
- Mac OSX

Also works on Solaris zones given a physical Solaris 11.2 server. For that, use the .kitchen.zone.yml file. Check usage at (https://github.com/criticalmass/kitchen-zone). You will need an url to a filebeat package that works on Solaris 11.2. Checkout Building-Filebeat-On-Solaris11.md for instructions to build a filebeat package.

## Supported Chef

This cookbook is tested against current Chef version. But, the cookbook is known to work with Chef Client version >=12.14.

## Supported Filebeat

- 6.x
- 7.x

## Cookbook Dependency

- `homebrew`
- `elastic_repo`
- `yum-plugin-versionlock`
- `runit`
- `windows`

## Recipes

- lwrp_test - LWRP examples recipe

## LWRP Resources

- `filebeat_config` - filebeat configuration resource

- `filebeat_install` - filebeat install resource

- `filebeat_install_preview` - filebeat preview package install resource

- `filebeat_service` - filebeat service resource

- `filebeat_runit_service` - filebeat service resource using runit

- `filebeat_prospector` - filebeat prospector resource (renamed filebeat inputs in version 6.3+)

## Limitations

The Mac OSX setup only allows for package installs and depends on brew, this means that version selection and preview build installs are not supported.

## LWRP filebeat_install

LWRP `filebeat_install` installs filebeat, creates log/prospectors directories, and also enable filebeat service.

Below attributes are derived using helper methods and also used by other LWRP.

- `conf_dir`
- `prospectors_dir`
- `log_dir`

**LWRP example**

```ruby
filebeat_install 'default' do
[options ..]
end
```

**LWRP Options**

- *action* (optional) - default `:create`, options: :create, :delete
- *version* (optional, String) - default `6.3.0`, filebeat version
- *release* (optional, String) - default `1`, filebeat release version, used by `rhel` family package resource
- *setup_repo* (optional, Boolean) - default `true`, set to `false`, to skip elastic repository setup using cookbook `elastic_repo`
- *ignore_package_version* (optional, Boolean) - default `false`, set to true, to install latest available yum/apt filebeat package
- *service_name* (optional, String) - default `filebeat`, filebeat service name, must be common across resources
- *disable_service* (optional, Boolean) - default `false`, set to `true`, to disable filebeat service
- *notify_restart* (optional, Boolean) - default `true`, set to `false`, to ignore filebeat service restart notify
- *delete_prospectors_dir* (optional, Boolean) - default `false`, set to `true`, to purge prospectors directory by deleting and recrating prospectors directory
- *conf_dir* (optional, String, NilClass) - default `nil`, filebeat configuration directory, this attribute is derived by helper method
- *prospectors_dir* (optional, String, NilClass) - default `nil`, filebeat prospectors directory, this attribute is derived by helper method
- *log_dir* (optional, String, NilClass) - default `nil`, filebeat log directory, this attribute is derived by helper method
- *windows_package_url* (optional, String) - default `auto`, windows filebeat package url
- *windows_base_dir* (optional, String) - default `C:/opt/filebeat`, filebeat windows base directory
- *apt_options* (optional, Array) - default `%w[stable main]`, filebeat package resource attribute for `debian` platform family
- *elastic_repo_options* (optional, Hash) - default `{}`, resource elastic_repo options, `filebeat_install` attribute `version` overrides `elasti_repo_options` key `version` value. Check out [elastic_repo cookbook](https://github.com/vkhatri/chef-elastic-repo) for more details.

## LWRP filebeat_service

LWRP `filebeat_service` configures `filebeat` service.

**LWRP example**

```ruby
filebeat_service 'default' do
[options ..]
end
```

**LWRP Options**

- *action* (optional) - default `:create`, options: :create, :delete
- *filebeat_install_resource_name* (optional, String) - default `default`, filebeat_install/filebeat_install_preview resource name, set this attribute if LWRP resource name is other than `default`
- *service_name* (optional, String) - default `filebeat`, filebeat service name, must be common across resources
- *disable_service* (optional, Boolean) - default `false`, set to `true`, to disable filebeat service
- *purge_prospectors_dir* (optional, Boolean) - default `false`, set to `true`, to purge prospectors
- *service_ignore_failure* (optional, Boolean) - default `false`, set to `true`, to ignore filebeat service failures
- *service_retries* (optional, Integer) - default `2`, filebeat service resource attribute
- *service_retry_delay* (optional, Integer) - default `0`, filebeat service resource attribute

## LWRP filebeat_config

LWRP `filebeat_config` creates filebeat configuration yaml file `/etc/filebeat/filebeat.yml`.

Below filebeat configuration parameters gets overwritten by the LWRP.

- `filebeat.registry_file`
- `filebeat.config_dir`
- `logging.files`

### Filebeat version < 6.0

```ruby
conf = {
'filebeat.modules' => [],
'prospectors' => [],
'logging.level' => 'info',
'logging.to_files' => true,
'logging.files' => { 'name' => 'filebeat' },
'output.elasticsearch' => { 'hosts' => ['127.0.0.1:9200'] }
}

filebeat_config 'default' do
config conf
action :create
end
```

Above LWRP Resource will create a file `/etc/filebeat/filebeat.yml` with content:

```yaml
filebeat.modules: []
prospectors: []
logging.level: info
logging.to_files: true
logging.files:
path: "/var/log/filebeat"
output.elasticsearch:
hosts:
- 127.0.0.1:9200
filebeat.registry_file: "/var/lib/filebeat/registry"
filebeat.config_dir: "/etc/filebeat/conf.d"
```

### Filebeat version 6.0+

```ruby
conf = {
'filebeat.modules' => [],
'filebeat.inputs' => [],
'logging.level' => 'info',
'logging.to_files' => true,
'logging.files' => { 'name' => 'filebeat' },
'output.elasticsearch' => { 'hosts' => ['127.0.0.1:9200'] }
}

filebeat_config 'default' do
config conf
action :create
end
```

Above LWRP Resource will create a file `/etc/filebeat/filebeat.yml` with content:

**LWRP Options**

- *action* (optional) - default `:create`, options: :create, :delete
- *filebeat_install_resource_name* (optional, String) - default `default`, filebeat_install/filebeat_install_preview resource name, set this attribute if LWRP resource name is other than `default`
- *config* (Hash) - default `{}` filebeat configuration options
- *config_sensitive* (optional, Boolean) - default `false`, filebeat configuration file chef resource attribute
- *service_name* (optional, String) - default `filebeat`, filebeat service name, must be common across resources
- *conf_file* (optional, String, NilClass) - default `nil`, filebeat configuration file, this attribute is derived by helper method
- *disable_service* (optional, Boolean) - default `false`, set to `true`, to disable filebeat service
- *notify_restart* (optional, Boolean) - default `true`, set to `false`, to ignore filebeat service restart notify
- *prefix* (optional, String) - default `lwrp-prospector-`, filebeat prospecteor filename prefix, set to '' if no prefix is desired

## LWRP filebeat_prospector (inputs)

### Filebeat version up to 6.3
LWRP `filebeat_prospector` creates a filebeat prospector configuration yaml file under prospectors directory with file name `lwrp-prospector-#{resource_name}.yml`.
`lwrp-prospector-` prefix can be changed with `prefix` property (see above).

**LWRP example**

```ruby
conf = {
'enabled' => true,
'paths' => ['/var/log/messages'],
'type' => 'log',
'fields' => {'type' => 'messages_log'}
}

filebeat_prospector 'messages_log' do
config conf
action :create
prefix 'my-custom-prefix-'
end
```

Above LWRP Resource will create a file `/etc/filebeat/conf.d/my-custom-prefix-messages_log.yml` with content:

```yaml
filebeat:
prospectors:
- enabled: true
paths:
- "/var/log/messages"
type: log
fields:
type: messages_log
```

### Filebeat version 6.3+

With `filebeat.config.inputs` set, Filebeat 6.3+ will read in the files as previous versions and the same config will work as expected

```ruby
conf = {
'type' => 'log',
'enabled' => true,
'paths' => ['/var/log/messages'],
'type' => 'log',
'fields' => {'type' => 'messages_log'}
}

filebeat_prospector 'messages_log' do
config conf
action :create
end
```

the file is created with content:

```yaml
- enabled: true
paths:
- "/var/log/messages"
type: log
fields:
type: messages_log
```

**LWRP Options**

- *action* (optional) - default `:create`, options: :create, :delete
- *filebeat_install_resource_name* (optional, String) - default `default`, filebeat_install/filebeat_install_preview resource name, set this attribute if LWRP resource name is other than `default`
- *config* (Hash) - default `{}` filebeat configuration options
- *config_sensitive* (optional, Boolean) - default `false`, filebeat configuration file chef resource attribute
- *service_name* (optional, String) - default `filebeat`, filebeat service name, must be common across resources
- *disable_service* (optional, Boolean) - default `false`, set to `true`, to disable filebeat service
- *notify_restart* (optional, Boolean) - default `true`, set to `false`, to ignore filebeat service restart notify

## LWRP filebeat_runit_service

LWRP `filebeat_runit_service` configures `filebeat` service using `runit`.

**LWRP example**

```ruby
filebeat_runit_service 'default' do
[options ..]
end
```

**LWRP Options**

- *action* (optional) - default `:create`, options: :create, :delete
- *filebeat_install_resource_name* (optional, String) - default `default`, filebeat_install/filebeat_install_preview resource name, set this attribute if LWRP resource name is other than `default`
- *service_name* (optional, String) - default `filebeat`, filebeat service name, must be common across resources
- *disable_service* (optional, Boolean) - default `false`, set to `true`, to disable filebeat service
- *purge_prospectors_dir* (optional, Boolean) - default `false`, set to `true`, to purge prospectors
- *runit_filebeat_cmd_options* (optional, Boolean) - default `''`, set to `true`, runit filebeat service command line options
- *service_ignore_failure* (optional, Boolean) - default `false`, set to `true`, to ignore filebeat service failures

## How to Create Filebeat LWRP Resources via Node Attribute

Check out filebeat test cookbook [filebeat_test](test/cookbooks/filebeat_test).

## TODO

- Add other platforms support to install_preview resource

## Contributing

1. Fork the repository on Github
2. Create a named feature branch (like `add_component_x`)
3. Write your change
4. Write tests for your change (if applicable)
5. Run the tests (`rake & rake knife`), ensuring they all pass
6. Write new resource/attribute description to `README.md`
7. Write description about changes to PR
8. Submit a Pull Request using Github

## Copyright & License

Authors:: Virender Khatri and [Contributors]

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

[Chef]: https://www.chef.io/
[Filebeat]: https://www.elastic.co/products/beats/filebeat
[Contributors]: https://github.com/vkhatri/chef-filebeat/graphs/contributors