https://github.com/jmaas/splunk-configs-clustering
Splunk configurations for a clustered architecture
https://github.com/jmaas/splunk-configs-clustering
splunk splunk-enterprise
Last synced: 3 months ago
JSON representation
Splunk configurations for a clustered architecture
- Host: GitHub
- URL: https://github.com/jmaas/splunk-configs-clustering
- Owner: jmaas
- License: bsd-2-clause
- Created: 2020-03-26T18:20:12.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2023-01-10T15:28:36.000Z (over 2 years ago)
- Last Synced: 2025-01-14T11:15:07.743Z (4 months ago)
- Topics: splunk, splunk-enterprise
- Homepage:
- Size: 38.1 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
This repository contains several basic configuration files required
by recent Splunk versions. Version used for testing is Splunk 9.0.0.
The instructions and files included in this repository allow you to
set-up a clustered environment relatively easy.
# Reference Architecture
This repository is based on an architecture I have running in my home lab.
The references to the hostnames should be replaced with the equivalents in your environment.
- 3x indexers cluster (`splunk-idxN`)
- 1x single-node cluster search head (`splunk-shN`)
- 1x universal forwarder (`splunk-ufN`)
- 1x management server (`splunk-mgt`) with roles:
- license master
- cluster master
- monitoring console
- deployer
# Manual Installation
Repeat all steps for every Splunk instance type in your architecture except for the Universal Forwarder instance (`splunk-ufN`).
## Splunk Enterprise
- install Splunk Enterprise using the package (rpm, deb, tgz) that best fits your environment
- switch to splunk user `sudo su - splunk` (when using tgz create user/group manually first)
- accept license and setup the admin account `$SPLUNK_HOME/bin/splunk start --accept-license`
- stop splunk `$SPLUNK_HOME/bin/splunk stop`
## Operating System
### Systemd based systems
- copy `systemd/disable-thp.service` over to `/etc/systemd/system/`
- copy `systemd/splunkd.service` over to `/etc/systemd/system/`
- make sure you don't `enable boot-start`, just to be sure `rm -f /etc/init.d/splunk`
- reload systemd unit files from disk `systemctl daemon-reload`
- enable the disable-thp service `systemctl enable disable-thp.service`
- enable the splunkd service `systemctl enable splunkd.service`
- start disable-thp `systemctl start disable-thp.service`
- start splunk `systemctl start splunkd.service`
### Sysvinit based systems
- copy `sysvinit/99-splunk.conf` over to `/etc/security/limits.d/`
- disable THP `cat sysvinit/rc.local >> /etc/rc.local`
- start splunk on boot `/opt/splunk/bin/splunk enable boot-start -user splunk`
- start splunk `/etc/init.d/splunk start`
## Verification
Verify that THP is disabled, please note that the output provided here is from CentOS 8.2.
```
[splunk@splunk-mgt ~]$ cat /sys/kernel/mm/transparent_hugepage/defrag
always defer defer+madvise madvise [never]
[splunk@splunk-mgt ~]$ cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
```
Verify that Splunk is not complaining about ulimits:
```
[root@splunk-idx3 ~]# grep limit /opt/splunk/var/log/splunk/splunkd.log | tail -n 12
09-15-2020 11:14:34.800 +0200 INFO ulimit - Linux vm.overcommit setting, value="0"
09-15-2020 11:23:24.781 +0200 INFO ulimit - Limit: virtual address space size: unlimited
09-15-2020 11:23:24.781 +0200 INFO ulimit - Limit: data segment size: unlimited
09-15-2020 11:23:24.781 +0200 INFO ulimit - Limit: resident memory size: unlimited
09-15-2020 11:23:24.781 +0200 INFO ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited]
09-15-2020 11:23:24.781 +0200 INFO ulimit - Limit: core file size: unlimited
09-15-2020 11:23:24.781 +0200 INFO ulimit - Limit: data file size: unlimited
09-15-2020 11:23:24.781 +0200 INFO ulimit - Limit: open files: 64000 files
09-15-2020 11:23:24.781 +0200 INFO ulimit - Limit: user processes: 16000 processes
09-15-2020 11:23:24.781 +0200 INFO ulimit - Limit: cpu time: unlimited
09-15-2020 11:23:24.782 +0200 INFO ulimit - Linux transparent hugepage support, enabled="never" defrag="never"
09-15-2020 11:23:24.782 +0200 INFO ulimit - Linux vm.overcommit setting, value="0"
```
# Configuration Apps
Apps are used as configuration bundles for the different instance roles in your environment.
## Master apps
These apps are installed on the `cluster master` in `/opt/splunk/etc/manager-apps` and pushed to all indexers.
- `cfg_base`: disables webserver in web.conf
- `cfg_indexes`: custom indexes should be defined here, includes examples
- `cfg_inputs`: enables the inputs on the indexers
- `cfg_license`: configures the license client
## Search-head Apps
The search head apps are installed on the `deployer` in `/opt/splunk/etc/shcluster/apps` and pushed to all search-heads.
- `cfg_outputs`: disables indexing on the SH and forwards data to the indexers
- `cfg_license`: configures the license client
## Deployment apps
Deployment apps are installed on the `deployment server` in `/opt/splunk/etc/deployment-apps` and pushed to all forwarders.
- `cfg_outputs`: setup forwarding to the indexers
# Notes
Instructions for the most common tasks are provided in the `notes/` directory.