https://github.com/crowdstrike/kubernetes-fluentbit-logging-falcon-logscale-integration
Integration to capture and send kubernetes logs and metrics to Falcon LogScale using the FluentBit observability pipeline
https://github.com/crowdstrike/kubernetes-fluentbit-logging-falcon-logscale-integration
Last synced: 5 months ago
JSON representation
Integration to capture and send kubernetes logs and metrics to Falcon LogScale using the FluentBit observability pipeline
- Host: GitHub
- URL: https://github.com/crowdstrike/kubernetes-fluentbit-logging-falcon-logscale-integration
- Owner: CrowdStrike
- License: apache-2.0
- Created: 2023-08-21T17:44:28.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-05-22T16:47:59.000Z (about 2 years ago)
- Last Synced: 2025-09-10T06:21:38.308Z (10 months ago)
- Size: 374 KB
- Stars: 1
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
[](https://twitter.com/CrowdStrike)
# Kubernetes log forwarding with the Fluent-Bit log shipper
Forward Kubernetes logs and system metrics to Falcon LogScale with the fluent-bit log shipper.
## Overview
This integration provides instructions on how to configure the Fluent-Bit log shipper to forward Kubernetes cluster logs and node metrics to Falcon LogScale for ingestion.
This integration was created as a companion to the LogScale [Kubernetes Logging Package](https://github.com/CrowdStrike/logscale-community-content/tree/main/Log-Sources/Kubernetes/Fluent-Bit-Logging) which is available on the [CrowdStrike Lgscale Community Content GitHub](https://github.com/CrowdStrike/logscale-community-content) repository.
*note: The Kubernetes Logging package dashboards rely on specific enrichments added by this integration.*
## Fluent-bit
[Fluent-Bit](https://fluentbit.io/) is a fast and lightweight end-to-end log processing pipeline that collects, filters, and forwards logs and metrics. It is an [open source](https://docs.fluentbit.io/manual/about/what-is-fluent-bit) sub-project of the [CNCF](https://www.cncf.io/).
Fluent-bit is written in C. It provides a variety of plugins to input, filter, and output logging telemetry. It can be run as system level service or as a DaemonSet within Kubernetes.
## Implementation Notes
This integration employs a split horizon setup of fluent-bit in order to capture a complete view of Kubernetes cluster logging telemetry: that is, fluent-bit is run simultaneously as a system service outside of Kubernetes, and as a DaemonSet within Kubernetes.
As a DaemonSet, fluent-bit monitors and forwards well known Kubernetes log sources: container, docker, audit.
As a system service, fluent-bit collects and forwards a variety of node metrics (cpu, network, disk, memory), and Docker events.
The split horizon implementation reflects the current state of fluent-bit namespace contraints among the various fluent-bit input plugins: metrics scraping is currently only possible from the system level where the fluent-bit metrics `[INPUT]` plugins have access to special device files in /proc and /sys.
Metrics capture is not currently possible when running fluent-bit from within a Kubernetes container.
Logs collected by the DaemonSet are automatic enriched with a set of Kubernetes properties: `i.e. "kubernetes": { ... }`
```json
{
"@timestamp": "2023-XX-XXTXX:XX:XX.XXXZ",
"log": "[2023/XX/XX XX:XX:XX] [ info] [engine] flush chunk '1-1111111111.111111111.flb' succeeded at retry 2: task_id=7, input=tail.1 > output=es.0 (out_id=0)\n",
"stream": "stderr",
"time": "2023-XX-XXTXX:XX:XX.000000000Z",
"source": "container",
"hostname": "host.name.com",
"kubernetes": {
"pod_name": "fluent-bit-tsv4f",
"namespace_name": "logging",
"pod_id": "nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn",
"labels": {
"app.kubernetes.io/instance": "fluent-bit",
"app.kubernetes.io/name": "fluent-bit",
"controller-revision-hash": "1111111111",
"pod-template-generation": "2"
},
"annotations": {
"checksum/config": "nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn",
"checksum/luascripts": "nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn",
"cni.projectcalico.org/containerID": "nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn",
"cni.projectcalico.org/podIP": "10.10.10.11/32",
"cni.projectcalico.org/podIPs": "10.10.10.11/32",
"kubectl.kubernetes.io/restartedAt": "2023-XX-XXTXX:XX:XX-XX:XX"
},
"host": "host.name.com",
"container_name": "fluent-bit",
"docker_id": "nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn",
"container_hash": "cr.fluentbit.io/fluent/fluent-bit@sha256:nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn",
"container_image": "cr.fluentbit.io/fluent/fluent-bit:2.0.8"
}
}
```
## Installation and Setup
### Install the Kubernetes Logging LogScale package
[Kubernetes Logging Package](https://github.com/CrowdStrike/logscale-community-content/tree/main/Log-Sources/Kubernetes/Fluent-Bit-Logging)
*note: the `fluentbit` parser is included in the package*
### (system) Install Fluent-Bit as a System Service
*note: this integration assumes Ubuntu Linux as the target platform*
#### Ubuntu APT install
[Apt Install Fluent-Bit](https://docs.fluentbit.io/manual/installation/linux/ubuntu)
`sudo apt-get install fluent-bit`
#### Configure Fluent-Bit
Prepare the `/etc/fluent-bit/fluent-bit.conf` file.
Use the `fluent-bit.conf` template file included with this integration to source the `[INPUT]`, `[FILTER]`, and `[OUTPUT]` stanzas.
Update the `[OUTPUT]` stanza with your LogScale ingest credentials.
*note: a fluent-bit timestamp is added to some metrics events that lack native timestamping.*
#### Start Fluent-Bit Service
`sudo systemctl start fluent-bit`
Verify that events are being ingested to the repository.
### (k8s) Install Fluent-Bit as a Kubernetes DaemonSet
Use the upstream fluentbit opensource distribution to install a daemonset through a helm chart.
#### Create a Logging namespace
`kubectl create namespace logging`
#### Prepare the Helm Chart
Follow the instructions in the [guide to installing Fluent-Bit via Helm](https://fluentbit.io/blog/2020/12/29/5-minute-guide-to-deploying-fluent-bit-on-kubernetes/) to install the base Fluent-Bit DaemonSet from the public helm chart https://github.com/fluent/helm-charts/
##### Configure the Fluent-Bit Helm Chart
###### ConfigMap
Use the included `fluent-bit.conf` template file to provide the stanzas for `[INPUT]`, `[FILTER]`, and `[OUTPUT]`.
Update the `[OUTPUT]` stanza with your LogScale ingest credentials.
###### Tolerations
Add a toleration to the DaemonSet, to allow Fluent-Bit to be scheduled on control-plane master nodes:
```
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
```
###### Auditing
Update the fluent-bit ClusterRole yaml to allow access to `events` resources:
```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
. . .
. . .
rules:
- apiGroups:
- ""
resources:
- namespaces
- pods
[[- events]] add this resource <-------
verbs:
- get
- list
- watch
```
*note: without `events` resource access, the following error will occur:*
```
{"log":"[2023/XX/XX XX:XX:XX] [error] [input:kubernetes_events:kubernetes_events.4] http_status=403:\n","stream":"stderr","time":"2023-XX-XXTXX:XX:XX.XXXXXXXXXZ"}
{"log":"{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"events is forbidden: User \\\"system:serviceaccount:logging:fluent-bit\\\" cannot list resource \\\"events\\\" in API group \\\"\\\" at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"events\"},\"code\":403}\n","stream":"stderr","time":"2023-XX-XXTXX:XX:XX.XXXXXXXXZ"}
```
##### Helm Install Fluent-Bit
`helm install fluent-bit .`
Once fluent-bit is enabled and the DaemonSet is running on all nodes, events should begin to appear in LogScale.
---
WE STOP BREACHES