{"id":18512827,"url":"https://github.com/vmware-archive/kube-fluentd-operator","last_synced_at":"2026-04-04T15:27:17.356Z","repository":{"id":32029020,"uuid":"126805536","full_name":"vmware/kube-fluentd-operator","owner":"vmware","description":"Auto-configuration of Fluentd daemon-set based on Kubernetes metadata","archived":false,"fork":false,"pushed_at":"2025-04-16T23:01:33.000Z","size":1116,"stargazers_count":320,"open_issues_count":47,"forks_count":102,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-04-28T17:03:01.721Z","etag":null,"topics":["fluentd","golang","hacktoberfest","kubernetes","operators","vmware"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vmware.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-03-26T09:29:35.000Z","updated_at":"2025-04-20T17:32:22.000Z","dependencies_parsed_at":"2023-12-19T20:28:44.004Z","dependency_job_id":"69b69a21-0c74-4505-b07d-091e357f9d54","html_url":"https://github.com/vmware/kube-fluentd-operator","commit_stats":{"total_commits":404,"total_committers":57,"mean_commits":7.087719298245614,"dds":0.6683168316831682,"last_synced_commit":"377758cac5b682c3262a92179fb677485b47f638"},"previous_names":[],"tags_count":59,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmware%2Fkube-fluentd-operator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmware%2Fkube-fluentd-operator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmware%2Fkube-fluentd-operator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmware%2Fkube-fluentd-operator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vmware","download_url":"https://codeload.github.com/vmware/kube-fluentd-operator/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254029003,"owners_count":22002283,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluentd","golang","hacktoberfest","kubernetes","operators","vmware"],"created_at":"2024-11-06T15:35:36.745Z","updated_at":"2026-04-04T15:27:17.222Z","avatar_url":"https://github.com/vmware.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kube-fluentd-operator (KFO)\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/vmware/kube-fluentd-operator)](https://goreportcard.com/report/github.com/vmware/kube-fluentd-operator)\n\n## Overview\n\nKubernetes Fluentd Operator (KFO) is a Fluentd config manager with batteries included, config validation, no needs to restart, with sensible defaults and best practices built-in. Use Kubernetes labels to filter/route logs per namespace!\n\n_kube-fluentd-operator_ configures Fluentd in a Kubernetes environment. It compiles a Fluentd configuration from configmaps (one per namespace) - similar to how an Ingress controller would compile nginx configuration from several Ingress resources. This way only one instance of Fluentd can handle all log shipping for an entire cluster and the cluster admin does NOT need to coordinate with namespace admins.\n\nCluster administrators set up Fluentd only once and namespace owners can configure log routing as they wish. _KFO_ will re-configure Fluentd accordingly and make sure logs originating from a namespace will not be accessible by other tenants/namespaces.\n\n_KFO_ also extends the Fluentd configuration language making it possible to refer to pods based on their labels and the container name pattern. This enables for very fined-grained targeting of log streams for the purpose of pre-processing before shipping. Writing a custom processor, adding a new Fluentd plugin, or writing a custom Fluentd plugin allow KFO to be extendable for any use case and any external logging ingestion system.\n\nFinally, it is possible to ingest logs from a file on the container filesystem. While this is not recommended, there are still legacy or misconfigured apps that insist on logging to the local filesystem.\n\n## Try it out\n\nThe easiest way to get started is using the Helm chart. Official images are not published yet, so you need to pass the image.repository and image.tag manually:\n\n```bash\ngit clone git@github.com:vmware/kube-fluentd-operator.git\nhelm install kfo ./kube-fluentd-operator/charts/log-router \\\n  --set rbac.create=true \\\n  --set image.tag=v1.18.1 \\\n  --set image.repository=vmware/kube-fluentd-operator\n```\n\nAlternatively, deploy the Helm chart from a Github release:\n\n```bash\nCHART_URL='https://github.com/vmware/kube-fluentd-operator/releases/download/v1.18.1/log-router-0.4.0.tgz'\n\nhelm install kfo ${CHART_URL} \\\n  --set rbac.create=true \\\n  --set image.tag=v1.18.1 \\\n  --set image.repository=vmware/kube-fluentd-operator\n```\n\nThen create a namespace `demo` and a configmap describing where all logs from `demo` should go to. The configmap must contain an entry called \"fluent.conf\". Finally, point the kube-fluentd-operator to this configmap using annotations.\n\n```bash\nkubectl create ns demo\n\ncat \u003e fluent.conf \u003c\u003c EOF\n\u003cmatch **\u003e\n  @type null\n\u003c/match\u003e\nEOF\n\n# Create the configmap with a single entry \"fluent.conf\"\nkubectl create configmap fluentd-config --namespace demo --from-file=fluent.conf=fluent.conf\n\n\n# The following step is optional: the fluentd-config is the default configmap name.\n# kubectl annotate namespace demo logging.csp.vmware.com/fluentd-configmap=fluentd-config\n\n```\n\nIn a minute, this configuration would be translated to something like this:\n\n```xml\n\u003cmatch demo.**\u003e\n  @type null\n\u003c/match\u003e\n```\n\nEven though the tag `**` was used in the `\u003cmatch\u003e` directive, the kube-fluentd-operator correctly expands this to `demo.**`. Indeed, if another tag which does not start with `demo.` was used, it would have failed validation. Namespace admins can safely assume that they has a dedicated Fluentd for themselves.\n\nAll configuration errors are stored in the annotation `logging.csp.vmware.com/fluentd-status`. Try replacing `**` with an invalid tag like 'hello-world'. After a minute, verify that the error message looks like this:\n\n```bash\n# extract just the value of logging.csp.vmware.com/fluentd-status\nkubectl get ns demo -o jsonpath='{.metadata.annotations.logging\\.csp\\.vmware\\.com/fluentd-status}'\nbad tag for \u003cmatch\u003e: hello-world. Tag must start with **, $thisns or demo\n```\n\nWhen the configuration is made valid again the `fluentd-status` is set to \"\".\n\nTo see kube-fluentd-operator in action you need a cloud log collector like logz.io, papertrail or ELK accessible from the K8S cluster. A simple logz.io configuration looks like this (replace TOKEN with your customer token):\n\n```xml\n\u003cmatch **\u003e\n   @type logzio_buffered\n   endpoint_url https://listener.logz.io:8071?token=$TOKEN\n\u003c/match\u003e\n```\n\n## Build\n\nGet the code using `go get` or git clone this repo:\n\n```bash\ngo get -u github.com/vmware/kube-fluentd-operator/config-reloader\ncd $GOPATH/src/github.com/vmware/kube-fluentd-operator\n\n# build a base-image\ncd base-image \u0026\u0026 make build-image\n\n# build helm chart\ncd charts/log-router \u0026\u0026 make helm-package\n\n# build the daemon\ncd config-reloader\nmake install\nmake build-image\n\n# run with mock data (loaded from the examples/ folder)\nmake run-once-fs\n\n# run with mock data in a loop (may need to ctrl+z to exit)\nmake run-loop-fs\n\n# inspect what is generated from the above command\nls -l tmp/\n```\n\n### Project structure\n\n- `charts/log-router`: Builds the Helm chart\n- `base-image`: Builds a Fluentd 1.2.x image with a curated list of plugins\n- `config-reloader`: Builds the daemon that generates fluentd configuration files\n\n### Config-reloader\n\nThis is where interesting work happens. The [dependency graph](config-reloader/godepgraph.png) shows the high-level package interaction and general dataflow.\n\n- `config`: handles startup configuration, reading and validation\n- `datasource`: fetches Pods, Namespaces, ConfigMaps from Kubernetes\n- `fluentd`: parses Fluentd config files into an object graph\n- `processors`: walks this object graph doing validations and modifications. All features are implemented as chained `Processor` subtypes\n- `generator`: serializes the processed object graph to the filesystem for Fluentd to read\n- `controller`: orchestrates the high-level `datasource` -\u003e `processor` -\u003e `generator` pipeline.\n\n### How does it work\n\nIt works be rewriting the user-provided configuration. This is possible because _kube-fluentd-operator_ knows about the kubernetes cluster, the current namespace and\nalso has some sensible defaults built in. To get a quick idea what happens behind the scenes consider this configuration deployed in a namespace called `monitoring`:\n\n```xml\n\u003cfilter $labels(server=apache)\u003e\n  @type parser\n  \u003cparse\u003e\n    @type apache2\n  \u003c/parse\u003e\n\u003c/filter\u003e\n\n\u003cfilter $labels(app=django)\u003e\n  @type detect_exceptions\n  language python\n\u003c/filter\u003e\n\n\u003cmatch **\u003e\n  @type es\n\u003c/match\u003e\n```\n\nIt gets processed into the following configuration which is then fed to Fluentd:\n\n```xml\n\u003cfilter kube.monitoring.*.*\u003e\n  @type record_transformer\n  enable_ruby true\n\n  \u003crecord\u003e\n    kubernetes_pod_label_values ${record[\"kubernetes\"][\"labels\"][\"app\"]\u0026.gsub(/[.-]/, '_') || '_'}.${record[\"kubernetes\"][\"labels\"][\"server\"]\u0026.gsub(/[.-]/, '_') || '_'}\n  \u003c/record\u003e\n\u003c/filter\u003e\n\n\u003cmatch kube.monitoring.*.*\u003e\n  @type rewrite_tag_filter\n\n  \u003crule\u003e\n    key kubernetes_pod_label_values\n    pattern ^(.+)$\n    tag ${tag}._labels.$1\n  \u003c/rule\u003e\n\u003c/match\u003e\n\n\u003cfilter kube.monitoring.*.*.**\u003e\n  @type record_modifier\n  remove_keys kubernetes_pod_label_values\n\u003c/filter\u003e\n\n\u003cfilter kube.monitoring.*.*._labels.*.apache _proc.kube.monitoring.*.*._labels.*.apache\u003e\n  @type parser\n  \u003cparse\u003e\n    @type apache2\n  \u003c/parse\u003e\n\u003c/filter\u003e\n\n\u003cmatch kube.monitoring.*.*._labels.django.*\u003e\n  @type rewrite_tag_filter\n\n  \u003crule\u003e\n    invert true\n    key _dummy\n    pattern /ZZ/\n    tag 3bfd045d94ce15036a8e3ff77fcb470e0e02ebee._proc.${tag}\n  \u003c/rule\u003e\n\u003c/match\u003e\n\n\u003cmatch 3bfd045d94ce15036a8e3ff77fcb470e0e02ebee._proc.kube.monitoring.*.*._labels.django.*\u003e\n  @type detect_exceptions\n  remove_tag_prefix 3bfd045d94ce15036a8e3ff77fcb470e0e02ebee\n  stream container_info\n\u003c/match\u003e\n\n\u003cmatch kube.monitoring.*.*._labels.*.* _proc.kube.monitoring.*.*._labels.*.*\u003e\n  @type es\n\u003c/match\u003e\n```\n\n## Configuration\n\n### Basic usage\n\nTo give the illusion that every namespace runs a dedicated Fluentd the user-provided configuration is post-processed. In general, expressions starting with `$` are macros that are expanded. These two directives are equivalent: `\u003cmatch **\u003e`, `\u003cmatch $thisns\u003e`. Almost always, using the `**` is the preferred way to match logs: this way you can reuse the same configuration for multiple namespaces.\n\n### The admin namespace\n\nKube-fluentd-operator defines one namespace to be the _admin_ namespace. By default this is set to `kube-system`. The _admin_ namespace is treated differently. Its configuration is not processed further as it is assumed only the cluster admin can manipulate resources in this namespace. If you don't plan to use any of the advanced features described bellow, you can just route all logs from all namespaces using this snippet in the _admin_ namespace:\n\n```xml\n\u003cmatch **\u003e\n @type ...\n # destination configuration omitted\n\u003c/match\u003e\n```\n\n`**` in this context is not processed and it means _literally_ everything.\n\nFluentd assumes it is running in a distro with systemd and generates logs with these Fluentd tags:\n\n- `systemd.{unit}`: the journal of a systemd unit, for example `systemd.docker.service`\n- `docker`: all docker logs, not containers. If systemd is used, the docker logs are in `systemd.docker.service`\n- `k8s.{component}`: logs from a K8S component, for example `k8s.kube-apiserver`\n- `kube.{namespace}.{pod_name}.{container_name}`: a log originating from (namespace, pod, container)\n\nAs the _admin_ namespace is processed first, a match-all directive would consume all logs and any other namespace configuration will become irrelevant (unless `\u003ccopy\u003e` is used).\nA recommended configuration for the _admin_ namespace is this one (assuming it is set to `kube-system`) - it captures all but the user namespaces' logs:\n\n```xml\n\u003cmatch systemd.** kube.kube-system.** k8s.** docker\u003e\n  # all k8s-internal and OS-level logs\n\n  # destination config omitted...\n\u003c/match\u003e\n```\n\nNote the `\u003cmatch systemd.**` syntax. A single `*` would not work as the tag is the full name - including the unit type, for example _systemd.nginx.service_\n\n### Using the $labels macro\n\nA very useful feature is the `\u003cfilter\u003e` and the `$labels` macro to define parsing at the namespace level. For example, the config-reloader container uses the `logfmt` format. This makes it easy to use structured logging and ingest json data into a remote log ingestion service.\n\n```xml\n\u003cfilter $labels(app=log-router, _container=reloader)\u003e\n  @type parser\n  reserve_data true\n  \u003cparse\u003e\n    @type logfmt\n  \u003c/parse\u003e\n\u003c/filter\u003e\n\n\u003cmatch **\u003e\n  @type logzio_buffered\n  # destination config omitted\n\u003c/match\u003e\n```\n\nThe above config will pipe all logs from the pods labelled with `app=log-router` through a [logfmt](https://github.com/vmware/kube-fluentd-operator/blob/master/base-image/plugins/parser_logfmt.rb) parser before sending them to logz.io. Again, this configuration is valid in any namespace. If the namespace doesn't contain any `log-router` components then the `\u003cfilter\u003e` directive is never activated. The `_container` is sort of a \"meta\" label and it allows for targeting the log stream of a specific container in a multi-container pod.\n\nIf you use [Kubernetes recommended labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/) for the pods and deployments, then KFO will rewrite `.` characters into `_`.\n\nFor example, let's assume the following labels exist in the fluentd-config in the `testing` namespace:\n\nThis label `$labels(_container=nginx-ingress-controller)` will filter by container name pattern. The label will convert to this for example: `kube.testing.*.nginx-ingress-controller._labels.*.*.`\n\nThis label `$labels(app.kubernetes.io/name=nginx-ingress, _container=nginx-ingress-controller)` converts to this `kube.testing.*.nginx-ingress-controller._labels.*.nginx_ingress`.\n\nThis label `$labels(app.kubernetes.io/name=nginx-ingress)` converts to this `$labels(kube.testing.*.*._labels.*.nginx_ingress)`.\n\nThis fluentd configmap in the `testing` namespace:\n\n```xml\n\u003cfilter **\u003e\n  @type concat\n  timeout_label @DISTILLERY_TYPES\n  key message\n  stream_identity_key cont_id\n  multiline_start_regexp /^(\\d{4}-\\d{1,2}-\\d{1,2} \\d{1,2}:\\d{1,2}:\\d{1,2}|\\[\\w+\\]\\s|\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\b|=\\w+ REPORT====|\\d{2}\\:\\d{2}\\:\\d{2}\\.\\d{3})/\n  flush_interval 10\n\u003c/filter\u003e\n\n\u003cmatch **\u003e\n  @type relabel\n  @label @DISTILLERY_TYPES\n\u003c/match\u003e\n\n\u003clabel @DISTILLERY_TYPES\u003e\n  \u003cfilter $labels(app_kubernetes_io/name=kafka)\u003e\n    @type parser\n    key_name log\n    format json\n    reserve_data true\n    suppress_parse_error_log true\n  \u003c/filter\u003e\n\n  \u003cfilter $labels(app.kubernetes.io/name=nginx-ingress, _container=controller)\u003e\n    @type parser\n    key_name log\n\n    \u003cparse\u003e\n      @type json\n      reserve_data true\n      time_format %FT%T%:z\n      emit_invalid_record_to_error false\n    \u003c/parse\u003e\n  \u003c/filter\u003e\n\n  \u003cmatch $labels(tag=noisy)\u003e\n    @type null\n  \u003c/match\u003e\n\u003c/label\u003e\n```\n\nwill be rewritten inside of KFO pods as this:\n\n```xml\n\u003cfilter kube.testing.**\u003e\n  @type concat\n  flush_interval 10\n  key message\n  multiline_start_regexp /^(\\d{4}-\\d{1,2}-\\d{1,2} \\d{1,2}:\\d{1,2}:\\d{1,2}|\\[\\w+\\]\\s|\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\b|=\\w+ REPORT====|\\d{2}\\:\\d{2}\\:\\d{2}\\.\\d{3})/\n  stream_identity_key cont_id\n  timeout_label @-DISTILLERY_TYPES-0e93f964a5b5f1760278744f1adf55d58d0e78ba\n\u003c/filter\u003e\n\n\u003cmatch kube.testing.**\u003e\n  @label @-DISTILLERY_TYPES-0e93f964a5b5f1760278744f1adf55d58d0e78ba\n  @type relabel\n\u003c/match\u003e\n\n\u003cmatch kube.testing.**\u003e\n  @label @-DISTILLERY_TYPES-0e93f964a5b5f1760278744f1adf55d58d0e78ba\n  @type null\n\u003c/match\u003e\n\n\u003clabel @-DISTILLERY_TYPES-0e93f964a5b5f1760278744f1adf55d58d0e78ba\u003e\n  \u003cfilter kube.testing.*.*._labels.*.kafka.*\u003e\n    @type parser\n    format json\n    key_name log\n    reserve_data true\n    suppress_parse_error_log true\n  \u003c/filter\u003e\n  \u003cfilter kube.testing.*.controller._labels.nginx_ingress.*.*\u003e\n    @type parser\n    key_name log\n\n    \u003cparse\u003e\n      @type json\n      emit_invalid_record_to_error false\n      reserve_data true\n      time_format %FT%T%:z\n    \u003c/parse\u003e\n  \u003c/filter\u003e\n  \u003cmatch kube.testing.*.*._labels.*.*.noisy\u003e\n    @type null\n  \u003c/match\u003e\n\u003c/label\u003e\n```\n\nAll plugins that change the fluentd tag are disabled for security reasons. Otherwise a rogue configuration may divert other namespace's logs to itself by prepending its name to the tag.\n\n### Ingest logs from a file in the container\n\nThe only allowed `\u003csource\u003e` directive is of type `mounted-file`. It is used to ingest a log file from a container on an `emptyDir`-mounted volume:\n\n```xml\n\u003csource\u003e\n  @type mounted-file\n  path /var/log/welcome.log\n  labels app=grafana, _container=test-container\n  \u003cparse\u003e\n    @type none\n  \u003c/parse\u003e\n\u003c/source\u003e\n```\n\nThe `labels` parameter is similar to the `$labels` macro and helps the daemon locate all pods that might log to the given file path. The `\u003cparse\u003e` directive is optional and if omitted the default `@type none` will be used. If you know the format of the log file you can explicitly specify it, for example `@type apache2` or `@type json`.\n\nThe above configuration would translate at runtime to something similar to this:\n\n```xml\n\u003csource\u003e\n  @type tail\n  path /var/lib/kubelet/pods/723dd34a-4ac0-11e8-8a81-0a930dd884b0/volumes/kubernetes.io~empty-dir/logs/welcome.log\n  pos_file /var/log/kfotail-7020a0b821b0d230d89283ba47d9088d9b58f97d.pos\n  read_from_head true\n  tag kube.kfo-test.welcome-logger.test-container\n\n  \u003cparse\u003e\n    @type none\n  \u003c/parse\u003e\n\u003c/source\u003e\n```\n\n### Dealing with multi-line exception stacktraces (since v1.3.0)\n\nMost log streams are line-oriented. However, stacktraces always span multiple lines. _kube-fluentd-operator_ integrates stacktrace processing using the [fluent-plugin-detect-exceptions](https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions). If a Java-based pod produces stacktraces in the logs, then the stacktraces can be collapsed in a single log event like this:\n\n```xml\n\u003cfilter $labels(app=jpetstore)\u003e\n  @type detect_exceptions\n  # you can skip language in which case all possible languages will be tried: go, java, python, ruby, etc...\n  language java\n\u003c/filter\u003e\n\n# The rest of the configuration stays the same even though quite a lot of tag rewriting takes place\n\n\u003cmatch **\u003e\n @type es\n\u003c/match\u003e\n```\n\nNotice how `filter` is used instead of `match` as described in [fluent-plugin-detect-exceptions](https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions). Internally, this filter is translated into several `match` directives so that the end user doesn't need to bother with rewriting the Fluentd tag.\n\nAlso, users don't need to bother with setting the correct `stream` parameter. _kube-fluentd-operator_ generates one internally based on the container id and the stream.\n\n### Reusing output plugin definitions (since v1.6.0)\n\nSometimes you only have a few valid options for log sinks: a dedicated S3 bucket, the ELK stack you manage, etc. The only flexibility you're after is letting namespace owners filter and parse their logs. In such cases you can abstract over an output plugin configuration - basically reducing it to a simple name which can be referenced from any namespace. For example, let's assume you have an S3 bucket for a \"test\" environment and you use logz.io for a \"staging\" environment. The first thing you do is define these two output in the _admin_ namespace:\n\n```xml\nadmin-ns.conf:\n\u003cmatch systemd.** docker kube.kube-system.** k8s.**\u003e\n  @type logzio_buffered\n  endpoint_url https://listener.logz.io:8071?token=$TOKEN\n\u003c/match\u003e\n\n\u003cplugin test\u003e\n  @type s3\n  aws_key_id  YOUR_AWS_KEY_ID\n  aws_sec_key YOUR_AWS_SECRET_KEY\n  s3_bucket   YOUR_S3_BUCKET_NAME\n  s3_region   AWS_REGION\n\u003c/plugin\u003e\n\n\u003cplugin staging\u003e\n  @type logzio_buffered\n  endpoint_url https://listener.logz.io:8071?token=$TOKEN\n\u003c/plugin\u003e\n```\n\nIn the above example for the admin configuration, the `match` directive is first defined to direct where to send logs for the `systemd`, `docker`, `kube-system`, and kubernetes control plane components. Below the `match` directive we have defined the `plugin` directives which define the log sinks that can be reused by namespace configurations.\n\nA namespace can refer to the `staging` and `test` plugins oblivious to the fact where exactly the logs end up:\n\n```xml\nacme-test.conf\n\u003cmatch **\u003e\n  @type test\n\u003c/match\u003e\n\n\nacme-staging.conf\n\u003cmatch **\u003e\n  @type staging\n\u003c/match\u003e\n```\n\nkube-fluentd-operator will insert the content of the `plugin` directive in the `match` directive. From then on, regular validation and postprocessing takes place.\n\n### Retagging based on log contents (since v1.12.0)\n\nSometimes you might need to split a single log stream to perform different processing based on the contents of one of the fields. To achieve this you can use the `retag` plugin that allows to specify a set of rules that match regular expressions against the specified fields. If one of the rules matches, the log is re-emitted with a new namespace-unique tag based on the specified tag.\n\nLogs that are emitted by this plugin can be consequently filtered and processed by using the `$tag` macro when specifiying the tag:\n\n```xml\n\u003cmatch $labels(app=apache)\u003e\n  @type retag\n  \u003crule\u003e\n    key message\n    pattern /^(ERROR) .*$/\n    tag notifications.$1 # refer to a capturing group using $number\n  \u003c/rule\u003e\n  \u003crule\u003e\n    key message\n    pattern /^(FATAL) .*$/\n    tag notifications.$1\n  \u003c/rule\u003e\n  \u003crule\u003e\n    key message\n    pattern /^(ERROR)|(FATAL) .*$/\n    tag notifications.other\n    invert true # rewrite tag when unmatch pattern\n  \u003c/rule\u003e\n\u003c/match\u003e\n\n\u003cfilter $tag(notifications.ERROR)\u003e\n  # perform some extra processing\n\u003c/filter\u003e\n\n\u003cfilter $tag(notifications.FATAL)\u003e\n  # perform different processing\n\u003c/filter\u003e\n\n\u003cmatch $tag(notifications.**)\u003e\n  # send to common output plugin\n\u003c/match\u003e\n```\n\n_kube-fluentd-operator_ ensures that tags specified using the `$tag` macro never conflict with tags from other namespaces, even if the tag itself is equivalent.\n\n### Sharing logs between namespaces\n\nBy default, you can consume logs only from your namespaces. Often it is useful for multiple namespaces (tenants) to get access to the logs streams of a shared resource (pod, namespace). _kube-fluentd-operator_ makes it possible using two constructs: the source namespace expresses its intent to share logs with a destination namespace and the destination namespace expresses its desire to consume logs from a source. As a result logs are streamed only when both sides agree.\n\nA source namespace can share with another namespace using the `@type share` macro:\n\nproducer namespace configuration:\n\n```xml\n\u003cmatch $labels(msg=nginx-ingress)\u003e\n  @type copy\n  \u003cstore\u003e\n    @type share\n    # share all logs matching the labels with the namespace \"consumer\"\n    with_namespace consumer\n  \u003c/store\u003e\n\u003c/match\u003e\n```\n\nconsumer namespace configuration:\n\n```xml\n# use $from(producer) to get all shared logs from a namespace called \"producer\"\n\u003clabel @$from(producer)\u003e\n  \u003cmatch **\u003e\n    # process all shared logs here as usual\n  \u003c/match\u003e\n\u003c/match\u003e\n```\n\nThe consuming namespace can use the usual syntax inside the `\u003clabel @$from...\u003e` directive. The fluentd tag is being rewritten as if the logs originated from the same namespace.\n\nThe producing namespace need to wrap `@type share` within a `\u003cstore\u003e` directive. This is done on purpose as it is very easy to just redirect the logs to the destination namespace and lose them. The `@type copy` clones the whole stream.\n\n### Log metadata\n\nOften you run mulitple Kubernetes clusters but you need to aggregate all logs to a single destination. To distinguish between different sources, `kube-fluentd-operator` can attach arbitrary metadata to every log event.\nThe metadata is nested under a key chosen with `--meta-key`. Using the helm chart, metadata can be enabled like this:\n\n```bash\nhelm install ... \\\n  --set meta.key=metadata \\\n  --set meta.values.region=us-east-1 \\\n  --set meta.values.env=staging \\\n  --set meta.values.cluster=legacy\n```\n\nEvery log event, be it from a pod, mounted-file or a systemd unit, will now carry this metadata:\n\n```json\n{\n  \"metadata\": {\n    \"region\": \"us-east-1\",\n    \"env\": \"staging\",\n    \"cluster\": \"legacy\"\n  }\n}\n```\n\nAll logs originating from a file look exactly as all other Kubernetes logs. However, their `stream` field is not set to `stdout` but to the path to the source file:\n\n```json\n{\n  \"message\": \"Some message from the welcome-logger pod\",\n  \"stream\": \"/var/log/welcome.log\",\n  \"kubernetes\": {\n    \"container_name\": \"test-container\",\n    \"host\": \"ip-11-11-11-11.us-east-2.compute.internal\",\n    \"namespace_name\": \"kfo-test\",\n    \"pod_id\": \"723dd34a-4ac0-11e8-8a81-0a930dd884b0\",\n    \"pod_name\": \"welcome-logger\",\n    \"labels\": {\n      \"msg\": \"welcome\",\n      \"test-case\": \"b\"\n    },\n    \"namespace_labels\": {}\n  },\n  \"metadata\": {\n    \"region\": \"us-east-2\",\n    \"cluster\": \"legacy\",\n    \"env\": \"staging\"\n  }\n}\n```\n\n### Go templting\n\nThe `ConfigMap` holding the fluentd configuration can be templated using `go` templting, you can use this for example to get a value from another kubernetes resource, like a secret, for example:\n\n```yaml\nkind: ConfigMap\napiVersion: v1\nmetadata:\n  annotations: {}\n  name: fluentd-config\n  namespace: my-namespace\ndata:\n  fluent.conf: |\n    {{- $s := k8sLookup \"Secret.v1\" \"my-namespace\" \"my-secret\" -}}\n    \u003cmatch **\u003e\n      @type logzio_buffered\n      endpoint_url https://listener.logz.io:8071?token={{ $s.data.token }}\u0026type=log-router\n      output_include_time true\n      output_include_tags false\n      http_idle_timeout 10\n\n      \u003cbuffer\u003e\n        @type file\n        path /var/log/my_namespace.log.buf\n        flush_thread_count 4\n        flush_interval 10s\n        chunk_limit_size 16m\n        queue_limit_length 4096\n      \u003c/buffer\u003e\n    \u003c/match\u003e\n```\n\nYou can limit what k8s objects can be looked up using the templting functionality by passing the `--allow-label` flag, for example `--allow-label=logs.vmware.com/allow`.\nYou can also override what label to use on specific Namespaces by passing the `--allow-label-annotation` flag and then setting what label to use in that annotation on the Namespace, for example, `--allow-label-annotation=logs.vmware.com/allow-label`\nAnd in the Namespace:\n\n```yaml\napiVersion: v1\nkind: Namespace\nmetadata:\n  name: my-namespace\n  annotations:\n    logs.vmware.com/allow-label: \"logs.vmware.com/my_namespace\"\nspec:\n  finalizers:\n    - kubernetes\n```\n\nAnd the templated config in this Namespace will only be allowed to lookup resources labeled with `logs.vmware.com/my_namespace=\"true\"`\n\n### Custom resource definition(CRD) support (since v1.13.0)\n\nCustom resources are introduced from v1.13.0 release onwards. It allows to have a dedicated resource for fluentd configurations, which enables to manage them in a more consistent way and move away from the generic ConfigMaps.\nIt is possible to create configs for a new application simply by attaching a FluentdConfig resource to the application manifests, rather than using a more generic ConfigMap with specific names and/or labels.\n\n```xml\napiVersion: logs.vdp.vmware.com/v1beta1\nkind: FluentdConfig\nmetadata:\n  name: fd-config\nspec:\n  fluentconf: |\n    \u003cmatch kube.ns.**\u003e\n      @type relabel\n      @label @NOTIFICATIONS\n    \u003c/match\u003e\n\n    \u003clabel @NOTIFICATIONS\u003e\n     \u003cmatch **\u003e\n       @type null\n     \u003c/match\u003e\n    \u003c/label\u003e\n```\n\nThe \"crd\" has been introduced as a new datasource, configurable through the helm chart values, to allow users that are currently set up with ConfigMaps and do not want to perform the switchover to FluentdConfigs, to be able to keep on using them. The config-reloader has been equipped with the capability of installing the CRD at startup if requested, so no manual actions to enable it on the cluster are needed.\nThe existing configurations though ConfigMaps can be migrated to CRDs through the following migration flow\n\n- A new user, who is installing kube-fluentd-operator for the first time, should set the datasource: crd option in the chart. This enables the crd support\n- A user who is already using kube-fluentd-operator with either datasource: default or datasource: multimap will have update to the new chart and set the 'crdMigrationMode' property to 'true'. This enables the config-reloader to launch with the crd datasource and the legacy datasource (either default or multimap depending on what was configured in the datasource property). The user can slowly migrate one by one all configmap resources to the corresponding fluentdconfig resources. When the migration is complete, the Helm release can be upgraded by changing the 'crdMigrationMode' property to 'false' and switching the datasource property to 'crd'. This will effectively disable the legacy datasource and set the config-reloader to only watch fluentdconfig resources.\n\n## Tracking Fluentd version\n\nThis projects tries to keep up with major releases for [Fluentd docker image](https://github.com/fluent/fluentd-docker-image/).\n\n| Fluentd version | Operator version |\n| --------------- | ---------------- |\n| 0.12.x          | 1.0.0            |\n| 1.15.3          | 1.17.1           |\n| 1.16.1          | 1.17.6           |\n| 1.16.1          | 1.18.0           |\n| 1.16.1          | 1.18.1           |\n\n## Plugins in latest release (1.18.1)\n\n`kube-fluentd-operator` aims to be easy to use and flexible. It also favors sending logs to multiple destinations using `\u003ccopy\u003e` and as such comes with many plugins pre-installed:\n\n- fluentd (1.16.1)\n- fluent-plugin-amqp (0.14.0)\n- fluent-plugin-azure-loganalytics (0.7.0)\n- fluent-plugin-cloudwatch-logs (0.14.3)\n- fluent-plugin-concat (2.5.0)\n- fluent-plugin-datadog (0.14.2)\n- fluent-plugin-elasticsearch (5.3.0)\n- fluent-plugin-opensearch (1.1.0)\n- fluent-plugin-gelf-hs (1.0.8)\n- fluent-plugin-google-cloud (0.13.0) - forked to allow fluentd v1.14.x\n- fluent-plugin-grafana-loki (1.2.20)\n- fluent-plugin-grok-parser (2.6.2)\n- fluent-plugin-json-in-json-2 (1.0.2)\n- fluent-plugin-kafka (0.18.1)\n- fluent-plugin-kinesis (3.4.2)\n- fluent-plugin-kubernetes_metadata_filter (3.2.0)\n- fluent-plugin-kubernetes_sumologic (2.4.2)\n- fluent-plugin-kubernetes (0.3.1)\n- fluent-plugin-logentries (0.2.10)\n- fluent-plugin-logzio (0.0.22)\n- fluent-plugin-mail (0.3.0)\n- fluent-plugin-mongo (1.5.0)\n- fluent-plugin-multi-format-parser (1.0.0)\n- fluent-plugin-papertrail (0.2.8)\n- fluent-plugin-prometheus (2.1.0)\n- fluent-plugin-record-modifier (2.1.0)\n- fluent-plugin-record-reformer (0.9.1)\n- fluent-plugin-redis (0.3.5)\n- fluent-plugin-remote_syslog (1.0.0)\n- fluent-plugin-rewrite-tag-filter (2.4.0)\n- fluent-plugin-route (1.0.0)\n- fluent-plugin-s3 (1.7.2)\n- fluent-plugin-splunk-hec (1.3.1)\n- fluent-plugin-splunkhec (2.3)\n- fluent-plugin-sumologic_output (1.7.3)\n- fluent-plugin-systemd (1.0.5)\n- fluent-plugin-uri-parser (0.3.0)\n- fluent-plugin-verticajson (0.0.6)\n- fluent-plugin-vmware-loginsight (1.4.1)\n- fluent-plugin-vmware-log-intelligence (2.0.8)\n- fluent-plugin-mysqlslowquery (0.0.9)\n- fluent-plugin-throttle (0.0.5)\n- fluent-plugin-webhdfs (1.5.0)\n- fluent-plugin-detect-exceptions (0.0.15)\n\nWhen customizing the image be careful not to uninstall plugins that are used internally to implement the macros.\n\nIf you need other destination plugins you are welcome to contribute a patch or just create an issue.\n\n## Synopsis\n\nThe config-reloader binary is the one that listens to changes in K8S and generates Fluentd files. It runs as a daemonset and is not intended to interact with directly. The synopsis is useful when trying to understand the Helm chart or just hacking.\n\n```txt\nusage: config-reloader [\u003cflags\u003e]\n\nRegenerates Fluentd configs based Kubernetes namespace annotations against templates, reloading\nFluentd if necessary\n\nFlags:\n  --help                        Show context-sensitive help (also try --help-long and\n                                --help-man).\n  --version                     Show application version.\n  --master=\"\"                   The Kubernetes API server to connect to (default: auto-detect)\n  --kubeconfig=\"\"               Retrieve target cluster configuration from a Kubernetes\n                                configuration file (default: auto-detect)\n  --datasource=default          Datasource to use (default|fake|fs|multimap|crd)\n  --crd-migration-mode          Enable the crd datasource together with the current datasource to facilitate the migration (used only with --datasource=default|multimap)\n  --fs-dir=FS-DIR               If datasource=fs is used, configure the dir hosting the files\n  --interval=60                 Run every x seconds\n  --allow-file                  Allow @type file for namespace configuration\n  --id=\"default\"                The id of this deployment. It is used internally so that two\n                                deployments don't overwrite each other's data\n  --fluentd-rpc-port=24444      RPC port of Fluentd\n  --log-level=\"info\"            Control verbosity of config-reloader logs\n  --fluentd-loglevel=\"info\"     Control verbosity of fluentd logs\n  --buffer-mount-folder=\"\"      Folder in /var/log/{} where to create all fluentd buffers\n  --annotation=\"logging.csp.vmware.com/fluentd-configmap\"\n                                Which annotation on the namespace stores the configmap name?\n  --default-configmap=\"fluentd-config\"\n                                Read the configmap by this name if namespace is not annotated.\n                                Use empty string to suppress the default.\n  --status-annotation=\"logging.csp.vmware.com/fluentd-status\"\n                                Store configuration errors in this annotation, leave empty to\n                                turn off\n  --kubelet-root=\"/var/lib/kubelet/\"\n                                Kubelet root dir, configured using --root-dir on the kubelet\n                                service\n  --namespaces=NAMESPACES ...   List of namespaces to process. If empty, processes all namespaces\n  --templates-dir=\"/templates\"  Where to find templates\n  --output-dir=\"/fluentd/etc\"   Where to output config files\n  --meta-key=META-KEY           Attach metadat under this key\n  --meta-values=META-VALUES     Metadata in the k=v,k2=v2 format\n  --fluentd-binary=FLUENTD-BINARY\n                                Path to fluentd binary used to validate configuration\n  --prometheus-enabled          Prometheus metrics enabled (default: false)\n  --admin-namespace=\"kube-system\"\n                                The namespace to be treated as admin namespace\n\n```\n\n## Helm chart\n\n| Parameter                    | Description                                                                                                          | Default                        |\n| ---------------------------- | -------------------------------------------------------------------------------------------------------------------- | ------------------------------ |\n| `rbac.create`                | Create a serviceaccount+role, use if K8s is using RBAC                                                               | `false`                        |\n| `serviceAccountName`         | Reuse an existing service account                                                                                    | `\"\"`                           |\n| `defaultConfigmap`           | Read the configmap by this name if the namespace is not annotated                                                    | `\"fluentd-config\"`             |\n| `image.repositiry`           | Repository                                                                                                           | `vmware/kube-fluentd-operator` |\n| `image.tag`                  | Image tag                                                                                                            | `latest`                       |\n| `image.pullPolicy`           | Pull policy                                                                                                          | `Always`                       |\n| `image.pullSecret`           | Optional pull secret name                                                                                            | `\"\"`                           |\n| `logLevel`                   | Default log level for config-reloader                                                                                | `info`                         |\n| `fluentdLogLevel`            | Default log level for fluentd                                                                                        | `info`                         |\n| `bufferMountFolder`          | Folder in /var/log/{} where to create all fluentd buffers                                                            | `\"\"`                           |\n| `kubeletRoot`                | The home dir of the kubelet, usually set using `--root-dir` on the kubelet                                           | `/var/lib/kubelet`             |\n| `namespaces`                 | List of namespaces to operate on. Empty means all namespaces                                                         | `[]`                           |\n| `interval`                   | How often to check for config changes (seconds)                                                                      | `45`                           |\n| `meta.key`                   | The metadata key (optional)                                                                                          | `\"\"`                           |\n| `meta.values`                | Metadata to use for the key                                                                                          | `{}`                           |\n| `extraVolumes`               | Extra volumes                                                                                                        |                                |\n| `fluentd.extraVolumeMounts`  | Mount extra volumes for the fluentd container, required to mount ssl certificates when elasticsearch has tls enabled |                                |\n| `fluentd.resources`          | Resource definitions for the fluentd container                                                                       | `{}`                           |\n| `fluentd.extraEnv`           | Extra env vars to pass to the fluentd container                                                                      | `{}`                           |\n| `reloader.extraVolumeMounts` | Mount extra volumes for the reloader container                                                                       |                                |\n| `reloader.resources`         | Resource definitions for the reloader container                                                                      | `{}`                           |\n| `reloader.extraEnv`          | Extra env vars to pass to the reloader container                                                                     | `{}`                           |\n| `tolerations`                | Pod tolerations                                                                                                      | `[]`                           |\n| `updateStrategy`             | UpdateStrategy for the daemonset. Leave empty to get the K8S' default (probably the safest choice)                   | `{}`                           |\n| `podAnnotations`             | Pod annotations for the daemonset                                                                                    |                                |\n| `adminNamespace`             | The namespace to be treated as admin namespace                                                                       | `kube-system`                  |\n\n## Cookbook\n\n### I want to use one destination for everything\n\nSimple, define configuration only for the _admin_ namespace (by default `kube-system`):\n\n```bash\nkube-system.conf:\n\u003cmatch **\u003e\n  # configure destination here\n\u003c/match\u003e\n```\n\n### I dont't care for systemd and docker logs\n\nSimple, exclude them at the _admin_ namespace level (by default `kube-system`):\n\n```bash\nkube-system.conf:\n\u003cmatch systemd.** docker\u003e\n  @type null\n\u003c/match\u003e\n\n\u003cmatch **\u003e\n  # all but systemd.** is still around\n  # configure destination\n\u003c/match\u003e\n```\n\n### I want to use one destination but also want to just exclude a few pods\n\nIt is not possible to handle this globally. Instead, provide this config for the noisy namespace and configure other namespaces at the cost of some code duplication:\n\n```xml\nnoisy-namespace.conf:\n\u003cmatch $labels(app=verbose-logger)\u003e\n  @type null\n\u003c/match\u003e\n\n# all other logs are captured here\n\u003cmatch **\u003e\n  @type ...\n\u003c/match\u003e\n```\n\nOn the bright side, the configuration of `noisy-namespace` contains nothing specific to noisy-namespace and the same content can be used for all namespaces whose logs we need collected.\n\n### I am getting errors \"namespaces is forbidden: ... cannot list namespaces at the cluster scope\"\n\nYour cluster is running under RBAC. You need to enable a serviceaccount for the log-router pods. It's easy when using the Helm chart:\n\n```bash\nhelm install ./charts/log-router --set rbac.create=true ...\n```\n\n### I have a legacy container that logs to /var/log/httpd/access.log\n\nFirst you need version 1.1.0 or later. At the namespace level you need to add a `source` directive of type `mounted-file`:\n\n```xml\n\u003csource\u003e\n  @type mounted-file\n  path /var/log/httpd/access.log\n  labels app=apache2\n  \u003cparse\u003e\n    @type apache2\n  \u003c/parse\u003e\n\u003c/source\u003e\n\n\u003cmatch **\u003e\n  # destination config omitted\n\u003c/match\u003e\n```\n\nThe type `mounted-file` is again a macro that is expanded to a `tail` plugin. The `\u003cparse\u003e` directive is optional and if not set a `@type none` will be used instead.\n\nIn order for this to work the pod must define a mount of type `emptyDir` at `/var/log/httpd` or any of it parent folders. For example, this pod definition is part of the test suite (it logs to /var/log/hello.log):\n\n```yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  name: hello-logger\n  namespace: kfo-test\n  labels:\n    msg: hello\nspec:\n  containers:\n    - image: ubuntu\n      name: greeter\n      command:\n        - bash\n        - -c\n        - while true; do echo `date -R` [INFO] \"Random hello number $((var++)) to file\"; sleep 2; [[ $(($var % 100)) == 0 ]] \u0026\u0026 :\u003e /var/log/hello.log ;done \u003e /var/log/hello.log\n      volumeMounts:\n        - mountPath: /var/log\n          name: logs\n  volumes:\n    - name: logs\n      emptyDir: {}\n```\n\nTo get the hello.log ingested by Fluentd you need at least this in the configuration for `kfo-test` namespace:\n\n```xml\n\u003csource\u003e\n  @type mounted-file\n  # need to specify the path on the container filesystem\n  path /var/log/hello.log\n\n  # only look at pods labeled this way\n  labels msg=hello\n  \u003cparse\u003e\n    @type none\n  \u003c/parse\u003e\n\u003c/source\u003e\n\n\u003cmatch $labels(msg=hello)\u003e\n  # store the hello.log somewhere\n  @type ...\n\u003c/match\u003e\n```\n\n### I want to push logs from namespace `demo` to logz.io\n\n```xml\ndemo.conf:\n\u003cmatch **\u003e\n  @type logzio_buffered\n  endpoint_url https://listener.logz.io:8071?token=TOKEN\u0026type=log-router\n  output_include_time true\n  output_include_tags true\n  \u003cbuffer\u003e\n    @type memory\n    flush_thread_count 4\n    flush_interval 3s\n    queue_limit_length 4096\n  \u003c/buffer\u003e\n\u003c/match\u003e\n```\n\nFor details you should consult the plugin documentation.\n\n### I want to push logs to a remote syslog server\n\nThe built-in `remote_syslog` plugin cannot be used as the fluentd tag may be longer than 32 bytes. For this reason there is a `truncating_remote_syslog` plugin that shortens the tag to the allowed limit. If you are currently using the `remote_syslog` output plugin you only need to change a single line:\n\n```xml\n\u003cmatch **\u003e\n  # instead of \"remote_syslog\"\n  @type truncating_remote_syslog\n\n  # the usual config for remote_syslog\n\u003c/match\u003e\n```\n\nTo get the general idea how truncation works, consider this table:\n\n| Original Tag                                  | Truncated tag                      |\n| --------------------------------------------- | ---------------------------------- |\n| `kube.demo.test.test`                         | `demo.test.test`                   |\n| `kube.demo.nginx-65899c769f-5zj6d.nginx`      | `demo.nginx-65899c769f-5zj*.nginx` |\n| `kube.demo.test.nginx11111111._lablels.hello` | `demo.test.nginx11111111`          |\n\n### I want to push logs to Humio\n\nHumio speaks the elasticsearh protocol so configuration is pretty similar to Elasticsearch. The example bellow is based on https://github.com/humio/kubernetes2humio/blob/master/fluentd/docker-image/fluent.conf.\n\n```xml\n\u003cmatch **\u003e\n  @type elasticsearch\n  include_tag_key false\n\n  host \"YOUR_HOST\"\n  path \"/api/v1/dataspaces/YOUR_NAMESPACE/ingest/elasticsearch/\"\n  scheme \"https\"\n  port \"443\"\n\n  user \"YOUR_KEY\"\n  password \"\"\n\n  logstash_format true\n\n  reload_connections \"true\"\n  logstash_prefix \"fluentd:kubernetes2humio\"\n  buffer_chunk_limit 1M\n  buffer_queue_limit 32\n  flush_interval 1s\n  max_retry_wait 30\n  disable_retry_limit\n  num_threads 8\n\u003c/match\u003e\n```\n\n### I want to push logs to papertrail\n\n```xml\ntest.conf:\n\u003cmatch **\u003e\n    @type papertrail\n    papertrail_host YOUR_HOST.papertrailapp.com\n    papertrail_port YOUR_PORT\n    flush_interval 30\n\u003c/match\u003e\n```\n\n### I want to push logs to an ELK cluster\n\n```xml\n\u003cmatch ***\u003e\n  @type elasticsearch\n  host ...\n  port ...\n  index_name ...\n  # many options available\n\u003c/match\u003e\n```\n\nFor details you should consult the plugin documentation.\n\n### I want to validate my config file before using it as a configmap\n\nThe container comes with a file validation command. To use it put all your \\*.conf file in a directory. Use the namespace name for the filename. Then use this one-liner, bind-mounting the folder and feeding it as a `DATASOURCE_DIR` env var:\n\n```bash\ndocker run --entrypoint=/bin/validate-from-dir.sh \\\n    --net=host --rm \\\n    -v /path/to/config-folder:/workspace \\\n    -e DATASOURCE_DIR=/workspace \\\n    vmware/kube-fluentd-operator:latest\n```\n\nIt will run fluentd in dry-run mode and even catch incorrect plug-in usage.\nThis is so common that it' already captured as a script [validate-logging-config.sh](https://github.com/vmware/kube-fluentd-operator/blob/master/config-reloader/validate-logging-config.sh).\nThe preferred way to use it is to copy it to your project and invoke it like this:\n\n```bash\nvalidate-logging-config.sh path/to/folder\n\n```\n\nAll `path/to/folder/*.conf` files will be validated. Check stderr and the exit code for errors.\n\n### I want to use Fluentd @label to simplify processing\n\nUse `\u003clabel\u003e` as usual, the daemon ensures that label names are unique cluster-wide. For example to route several pods' logs to destination X, and ignore a few others you can use this:\n\n```xml\n\u003cmatch $labels(app=foo)\u003e\n  @type relabel\n  @label @blackhole\n\u003c/match\u003e\n\n\u003cmatch $labels(app=bar)\u003e\n  @type relabel\n  @label @blackhole\n\u003c/match\u003e\n\n\u003clabel @blackhole\u003e\n  \u003cmatch **\u003e\n    @type null\n  \u003c/match\u003e\n\u003c/label\u003e\n\n# at this point, foo and bar's logs are being handled in the @blackhole chain,\n# the rest are still available for processing\n\u003cmatch **\u003e\n  @type ..\n\u003c/match\u003e\n```\n\n### I want to parse ingress-nginx access logs and send them to a different log aggregator\n\nThe ingress controller uses a format different than the plain Nginx. You can use this fragment to configure the namespace hosting the ingress-nginx controller:\n\n```xml\n\u003cfilter $labels(app=nginx-ingress, _container=nginx-ingress-controller)\u003e\n  @type parser\n  key_name log\n  reserve_data true\n  \u003cparse\u003e\n    @type regexp\n    expression /(?\u003cremote_addr\u003e[^ ]*) - \\[(?\u003cproxy_protocol_addr\u003e[^ ]*)\\] - (?\u003cremote_user\u003e[^ ]*) \\[(?\u003ctime\u003e[^\\]]*)\\] \"(?\u003cmethod\u003e\\S+)(?: +(?\u003crequest\u003e[^\\\"]*) +\\S*)?\" (?\u003ccode\u003e[^ ]*) (?\u003csize\u003e[^ ]*) \"(?\u003creferer\u003e[^\\\"]*)\" \"(?\u003cagent\u003e[^\\\"]*)\" (?\u003crequest_length\u003e[^ ]*) (?\u003crequest_time\u003e[^ ]*) \\[(?\u003cproxy_upstream_name\u003e[^ ]*)\\] (?\u003cupstream_addr\u003e[^ ]*) (?\u003cupstream_response_length\u003e[^ ]*) (?\u003cupstream_response_time\u003e[^ ]*) (?\u003cupstream_status\u003e[^ ]*)/\n    time_format %d/%b/%Y:%H:%M:%S %z\n  \u003c/parse\u003e\n\u003c/filter\u003e\n\n\u003cmatch **\u003e\n  # send the parsed access logs here\n\u003c/match\u003e\n```\n\nThe above configuration assumes you're using the Helm charts for Nginx ingress. If not, make sure to the change the `app` and `_container` labels accordingly. Given the horrendous regex above, you really should be outputting access logs in json format and just specify `@type json`.\n\n### I want to send logs to different sinks based on log contents\n\nThe `retag` plugin allows to split a log stream based on whether the contents of certain fields match the given regular expressions.\n\n```xml\n\u003cmatch $labels(app=apache)\u003e\n  @type retag\n  \u003crule\u003e\n    key message\n    pattern ^ERR\n    tag notifications.error\n  \u003c/rule\u003e\n  \u003crule\u003e\n    key message\n    pattern ^ERR\n    invert true\n    tag notifications.other\n  \u003c/rule\u003e\n\u003c/match\u003e\n\n\u003cmatch $tag(notifications.error)\u003e\n  # manage log stream with error severity\n\u003c/match\u003e\n\n\u003cmatch $tag(notifications.**)\u003e\n  # manage log stream with non-error severity\n\u003c/match\u003e\n```\n\n### I have my kubectl configured and my configmaps ready. I want to see the generated files before deploying the Helm chart\n\nYou need to run `make` like this:\n\n```bash\nmake run-once\n```\n\nThis will build the code, then `config-reloader` will connect to the K8S cluster, fetch the data and generate \\*.conf files in the `./tmp` directory. If there are errors the namespaces will be annotated.\n\n### I want to build a custom image with my own fluentd plugin\n\nUse the `vmware/kube-fluentd-operator:TAG` as a base and do any modification as usual. If this plugin is not top-secret consider sending us a patch :)\n\n### I run two clusters - in us-east-2 and eu-west-2. How to differentiate between them when pushing logs to a single location?\n\nWhen deploying the daemonset using Helm, make sure to pass some metadata:\n\nFor the cluster in USA:\n\n```bash\nhelm install ... \\\n  --set=meta.key=cluster_info \\\n  --set=meta.values.region=us-east-2\n```\n\nFor the cluster in Europe:\n\n```bash\nhelm install ... \\\n  --set=meta.key=cluster_info \\\n  --set=meta.values.region=eu-west-2\n```\n\nIf you are using ELK you can easily get only the logs from Europe using `cluster_info.region: +eu-west-2`. In this example the metadata key is `cluster_info` but you can use any key you like.\n\n### I don't want to annotate all my namespaces at all\n\nIt is possible to reduce configuration burden by using a default configmap name. The default value is `fluentd-config` - kube-fluentd-operator will read the configmap by that name if the namespace is not annotated.\nIf you don't like this default name or happen to use this configmap for other purposes then override the default with `--default-configmap=my-default`.\n\n### How can I be sure to use a valid path for the .pos and .buf files\n\n.pos files store the progress of the upload process and .buf are used for local buffering. Colliding .pos/.buf paths can lead to races in Fluentd. As such, `kube-fluentd-operator` tries hard to rewrite such path-based parameters in a predictable way. You only need to make sure they are unique for your namespace and `config-reloader` will take care to make them unique cluster-wide.\n\n### I dont like the annotation name logging.csp.vmware.com/fluentd-configmap\n\nUse `--annotation=acme.com/fancy-config` to use acme.com/fancy-config as annotation name. However, you'd also need to customize the Helm chart. Patches are welcome!\n\n## Known Issues\n\nCurrently space-delimited tags are not supported. For example, instead of `\u003cfilter a b\u003e`, you need to use `\u003cfilter a\u003e` and `\u003cfilter b\u003e`.\nThis limitation will be addressed in a later version.\n\n## Releases\n\n- [CHANGELOG.md](CHANGELOG.md).\n\n## Resoures\n\n- This plugin is used to provide kubernetes metadata https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter\n- This daemonset definition is used as a template: https://github.com/fluent/fluentd-kubernetes-daemonset/tree/master/docker-image/v0.12/debian-elasticsearch, however `kube-fluentd-operator` uses version 1.x version of fluentd and all the compatible plugin versions.\n- This [Github issue](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/issues/73) was the inspiration for the project. In particular it borrows the tag rewriting based on Kubernetes metadata to allow easier routing after that.\n\n## Contributing\n\nThe kube-fluentd-operator project team welcomes contributions from the community. If you wish to contribute code and you have not\nsigned our contributor license agreement (CLA), our bot will update the issue when you open a Pull Request. For any\nquestions about the CLA process, please refer to our [FAQ](https://cla.vmware.com/faq). For more detailed information,\nrefer to [CONTRIBUTING.md](CONTRIBUTING.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvmware-archive%2Fkube-fluentd-operator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvmware-archive%2Fkube-fluentd-operator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvmware-archive%2Fkube-fluentd-operator/lists"}