{"id":24504788,"url":"https://github.com/dennyzhang/challenges-fluent-bit","last_synced_at":"2026-03-19T19:33:39.557Z","repository":{"id":67455791,"uuid":"144055844","full_name":"dennyzhang/challenges-fluent-bit","owner":"dennyzhang","description":"Deep dive into fluent-bit","archived":false,"fork":false,"pushed_at":"2018-08-08T23:18:58.000Z","size":11,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-18T11:59:00.039Z","etag":null,"topics":["fluent-bit","logging"],"latest_commit_sha":null,"homepage":"https://www.dennyzhang.com/battle","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dennyzhang.png","metadata":{"files":{"readme":"README.org","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-08T18:58:04.000Z","updated_at":"2019-11-14T09:24:17.000Z","dependencies_parsed_at":"2023-06-11T02:46:42.112Z","dependency_job_id":null,"html_url":"https://github.com/dennyzhang/challenges-fluent-bit","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dennyzhang/challenges-fluent-bit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dennyzhang%2Fchallenges-fluent-bit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dennyzhang%2Fchallenges-fluent-bit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dennyzhang%2Fchallenges-fluent-bit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dennyzhang%2Fchallenges-fluent-bit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dennyzhang","download_url":"https://codeload.github.com/dennyzhang/challenges-fluent-bit/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dennyzhang%2Fchallenges-fluent-bit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29321276,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-10T20:44:44.282Z","status":"ssl_error","status_checked_at":"2026-02-10T20:44:43.393Z","response_time":65,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluent-bit","logging"],"created_at":"2025-01-21T23:26:19.220Z","updated_at":"2026-02-10T23:02:07.367Z","avatar_url":"https://github.com/dennyzhang.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"* org-mode configuration                                           :noexport:\n#+STARTUP: overview customtime noalign logdone showall\n#+TITLE:  Deep Dive Into Fluent-Bit\n#+DESCRIPTION: \n#+KEYWORDS: \n#+AUTHOR: Denny Zhang\n#+EMAIL:  denny@dennyzhang.com\n#+TAGS: noexport(n)\n#+PRIORITIES: A D C\n#+OPTIONS:   H:3 num:t toc:nil \\n:nil @:t ::t |:t ^:t -:t f:t *:t \u003c:t\n#+OPTIONS:   TeX:t LaTeX:nil skip:nil d:nil todo:t pri:nil tags:not-in-toc\n#+EXPORT_EXCLUDE_TAGS: exclude noexport\n#+SEQ_TODO: TODO HALF ASSIGN | DONE BYPASS DELEGATE CANCELED DEFERRED\n#+LINK_UP:   \n#+LINK_HOME: \n* Summary\n#+BEGIN_HTML\n\u003ca href=\"https://www.linkedin.com/in/dennyzhang001\"\u003e\u003cimg src=\"https://www.dennyzhang.com/wp-content/uploads/sns/linkedin.png\" alt=\"linkedin\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/DennyZhang\"\u003e\u003cimg src=\"https://www.dennyzhang.com/wp-content/uploads/sns/github.png\" alt=\"github\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://www.dennyzhang.com/slack\" target=\"_blank\" rel=\"nofollow\"\u003e\u003cimg src=\"https://slack.dennyzhang.com/badge.svg\" alt=\"slack\"/\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/DennyZhang\"\u003e\u003cimg align=\"right\" width=\"200\" height=\"183\" src=\"https://www.dennyzhang.com/wp-content/uploads/denny/watermark/github.png\" /\u003e\u003c/a\u003e\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n\u003ca href=\"http://makeapullrequest.com\" target=\"_blank\" rel=\"nofollow\"\u003e\u003cimg src=\"https://img.shields.io/badge/PRs-welcome-brightgreen.svg\" alt=\"PRs Welcome\"/\u003e\u003c/a\u003e\n#+END_HTML\n* Fluent-Bit\nFile me [[https://github.com/DennyZhang/challenges-fluent-bit/issues][Issues]] or star [[https://github.com/DennyZhang/challenges-fluent-bit][this repo]]\n\nSee more challenges from Denny: [[https://github.com/topics/denny-challenges][#denny-challenges]]\n\n** Deep Dive Into Source Code\n*** How fluent-bit kubernetes filter works\nIt calls something like: https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/%s/pods/%s\n\nThe logic is like this:\n\n#+BEGIN_EXAMPLE\n    kubernetes.c:cb_kube_filter -\u003e\n    kube_meta.c:flb_kube_meta_get/get_and_merge_meta -\u003e\n    kube_meta.c:get_api_server_info -\u003e\n    kube_meta.h:FLB_KUBE_API_FMT\n#+END_EXAMPLE\n\nhttps://github.com/fluent/fluent-bit/blob/master/plugins/filter_kubernetes/kube_meta.h#L54\n\nhttps://github.com/fluent/fluent-bit/blob/master/plugins/filter_kubernetes/kube_meta.c#L590\n\nhttps://github.com/fluent/fluent-bit/blob/master/plugins/filter_kubernetes/kube_meta.c#L138\n\nhttps://github.com/fluent/fluent-bit/blob/master/plugins/filter_kubernetes/kubernetes.c#L416\n\n**** How to run it manually\n- Get the token for your namespace account\n  https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/\n\n- Run curl test\n#+BEGIN_EXAMPLE\n# curl --header \"Authorization: Bearer $TOKEN\" -k https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/oratos/pods/dummy-input --insecure\ncurl --header \"Authorization: Bearer $TOKEN\" -k https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/oratos/pods/dummy-input --insecure\n{\n  \"kind\": \"Pod\",\n  \"apiVersion\": \"v1\",\n  \"metadata\": {\n    \"name\": \"dummy-input\",\n    \"namespace\": \"oratos\",\n    \"selfLink\": \"/api/v1/namespaces/oratos/pods/dummy-input\",\n    \"uid\": \"f57ec8e3-9a87-11e8-8dcd-080027dbaae1\",\n    \"resourceVersion\": \"8496\",\n    \"creationTimestamp\": \"2018-08-07T21:22:16Z\",\n    \"annotations\": {\n      \"kubectl.kubernetes.io/last-applied-configuration\": \"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"Pod\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"name\\\":\\\"dummy-input\\\",\\\"namespace\\\":\\\"oratos\\\"},\\\"spec\\\":{\\\"containers\\\":[{\\\"args\\\":[\\\"tail\\\",\\\"-f\\\",\\\"/dev/termination-log\\\"],\\\"image\\\":\\\"busybox\\\",\\\"name\\\":\\\"dummy-input\\\"}]}}\\n\"\n    }\n  },\n  \"spec\": {\n    \"volumes\": [\n      {\n        \"name\": \"default-token-xk9qq\",\n        \"secret\": {\n          \"secretName\": \"default-token-xk9qq\",\n          \"defaultMode\": 420\n        }\n      }\n    ],\n    \"containers\": [\n      {\n        \"name\": \"dummy-input\",\n        \"image\": \"busybox\",\n        \"args\": [\n          \"tail\",\n          \"-f\",\n          \"/dev/termination-log\"\n        ],\n        \"resources\": {\n          \n        },\n        \"volumeMounts\": [\n          {\n            \"name\": \"default-token-xk9qq\",\n            \"readOnly\": true,\n            \"mountPath\": \"/var/run/secrets/kubernetes.io/serviceaccount\"\n          }\n        ],\n        \"terminationMessagePath\": \"/dev/termination-log\",\n        \"terminationMessagePolicy\": \"File\",\n        \"imagePullPolicy\": \"Always\"\n      }\n    ],\n    \"restartPolicy\": \"Always\",\n    \"terminationGracePeriodSeconds\": 30,\n    \"dnsPolicy\": \"ClusterFirst\",\n    \"serviceAccountName\": \"default\",\n    \"serviceAccount\": \"default\",\n    \"nodeName\": \"minikube\",\n    \"securityContext\": {\n      \n    },\n    \"schedulerName\": \"default-scheduler\",\n    \"tolerations\": [\n      {\n        \"key\": \"node.kubernetes.io/not-ready\",\n        \"operator\": \"Exists\",\n        \"effect\": \"NoExecute\",\n        \"tolerationSeconds\": 300\n      },\n      {\n        \"key\": \"node.kubernetes.io/unreachable\",\n        \"operator\": \"Exists\",\n        \"effect\": \"NoExecute\",\n        \"tolerationSeconds\": 300\n      }\n    ]\n  },\n  \"status\": {\n    \"phase\": \"Running\",\n    \"conditions\": [\n      {\n        \"type\": \"Initialized\",\n        \"status\": \"True\",\n        \"lastProbeTime\": null,\n        \"lastTransitionTime\": \"2018-08-07T21:22:16Z\"\n      },\n      {\n        \"type\": \"Ready\",\n        \"status\": \"True\",\n        \"lastProbeTime\": null,\n        \"lastTransitionTime\": \"2018-08-07T21:22:21Z\"\n      },\n      {\n        \"type\": \"PodScheduled\",\n        \"status\": \"True\",\n        \"lastProbeTime\": null,\n        \"lastTransitionTime\": \"2018-08-07T21:22:16Z\"\n      }\n    ],\n    \"hostIP\": \"10.0.2.15\",\n    \"podIP\": \"172.17.0.7\",\n    \"startTime\": \"2018-08-07T21:22:16Z\",\n    \"containerStatuses\": [\n      {\n        \"name\": \"dummy-input\",\n        \"state\": {\n          \"running\": {\n            \"startedAt\": \"2018-08-07T21:22:20Z\"\n          }\n        },\n        \"lastState\": {\n          \n        },\n        \"ready\": true,\n        \"restartCount\": 0,\n        \"image\": \"busybox:latest\",\n        \"imageID\": \"docker-pullable://busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd\",\n        \"containerID\": \"docker://66f701a981bc2fa0db08fe9cdaf80468d2f7398c95db34e7502f839a909303d5\"\n      }\n    ],\n    \"qosClass\": \"BestEffort\"\n  }\n}\n#+END_EXAMPLE\n*** Sample message for fluent-bit kubernetes filter\n#+BEGIN_EXAMPLE\nkey: log, value: key: time, value: 2018-08-8T18:16:26.000979098Z\nkey: stream, value: stdout\nkey: time, value: 2018-08-08T18:16.27.002369384Z\nkey: kubernetes, value: map[pod_name:fluent-bit-n588c namespace_name: oratos pod_id:a987f335-9b36-11e8-9fa9-080027b477ce labels:map[controller-revision-hash:545891415 k8s-app:logging-agent kubernetes.io/cluster-service:true pod-template-generation:1 version:v1] host:minikube container_name:fluent-bit docker_id:5f5bcedc9a98d6c4705632cdb55d9bcb572b7fc80dbb1da3e440d092d56ea4f5]\n#+END_EXAMPLE\n*** For tail_file input plugin of fluent-bit, could I specify file path like: /var/log/*/apache/*.log?\nIt eventually use *glob* function of C. So probably, yes.\n\ntail_scan.c:flb_tail_scan -\u003e tail_scan.c:do_glob\n\nhttps://github.com/fluent/fluent-bit/blob/master/plugins/in_tail/tail_scan.c#L150\n\nhttps://github.com/fluent/fluent-bit/blob/master/plugins/in_tail/tail_scan.c#L202\n*** How fluent-bit notice a new log file creation or deletion?\nIt rescan the log folder every 10 seconds\n#+BEGIN_EXAMPLE\nDennyZhang [11:40 PM]\nHi XXX\n\nMay I ask one question about tail input plugin in fluent-bit\n\nWe’re using below input to parse k8s pod log files.\n    ```[INPUT]\n        Name              tail\n        Tag               kube.*\n        Path              /var/log/containers/*.log\n        Parser            docker\n        DB                /var/log/flb_kube.db\n        Mem_Buf_Limit     5MB\n        Skip_Long_Lines   On\n        Refresh_Interval  10```\n\nWhenever developers start or delete a pod, /var/log/containers will create or delete a log file.\n\n*We are wondering what the behavior fluent-bit would be*\n```1. Whether we need to restart or reload fluent-bit, if that happens\n2. What's the latency when the log file are created or deleted, but fluent-bit haven't detected.```\n\nChecking the code, here is my understanding. Would you help us to confirm?\n```1. When tail input plugin loads, it will register a timer to rescan the log folder. (in_tail_init function in tail.c. https://github.com/fluent/fluent-bit/blob/d2d3d363c6f852f155d45566e1c4155024327913/plugins/in_tail/tail.c#L221-L225)\n2. The rescan interval is every 60 seconds (https://github.com/fluent/fluent-bit/blob/d2d3d363c6f852f155d45566e1c4155024327913/plugins/in_tail/tail.h#L38)\n3. Whenever developers create a Pod, the associate log file will be detected after 60 seconds. And no service restart is required\n4. Whenever developers delete a Pod, file read will run into exception. Thus no more action will be required. (https://github.com/fluent/fluent-bit/blob/d2d3d363c6f852f155d45566e1c4155024327913/plugins/in_tail/tail_file.c#L658)```\n\nXXX [7:58 PM]\nHi,\n\nif I am not wrong when a Pod get's deleted, the log files for that container persists for a period of time. Now the re-scan of the path to find new logs happens by default every 60 seconds or based in the value of \"Refresh_Interval\" (which you have set to 10 seconds).\n#+END_EXAMPLE\n* More Resources\nLicense: Code is licensed under [[https://www.dennyzhang.com/wp-content/mit_license.txt][MIT License]].\n#+BEGIN_HTML\n\u003ca href=\"https://www.dennyzhang.com\"\u003e\u003cimg align=\"right\" width=\"201\" height=\"268\" src=\"https://raw.githubusercontent.com/USDevOps/mywechat-slack-group/master/images/denny_201706.png\"\u003e\u003c/a\u003e\n\u003ca href=\"https://www.dennyzhang.com\"\u003e\u003cimg align=\"right\" src=\"https://raw.githubusercontent.com/USDevOps/mywechat-slack-group/master/images/dns_small.png\"\u003e\u003c/a\u003e\n\n\u003ca href=\"https://www.linkedin.com/in/dennyzhang001\"\u003e\u003cimg align=\"bottom\" src=\"https://www.dennyzhang.com/wp-content/uploads/sns/linkedin.png\" alt=\"linkedin\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/DennyZhang\"\u003e\u003cimg align=\"bottom\"src=\"https://www.dennyzhang.com/wp-content/uploads/sns/github.png\" alt=\"github\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://www.dennyzhang.com/slack\" target=\"_blank\" rel=\"nofollow\"\u003e\u003cimg align=\"bottom\" src=\"https://slack.dennyzhang.com/badge.svg\" alt=\"slack\"/\u003e\u003c/a\u003e\n#+END_HTML\n* [#A] fluent bit                                                  :noexport:\n#+BEGIN_EXAMPLE\nhttps://fluentbit.io/documentation/current/installation/docker.html\n\ndocker run -t -d --name fluent-bit  -p 2020:2020 -p 24224:24224 -v fluent-bit.conf:/etc/fluent-bit.conf --entrypoint=/bin/bash fluent/fluent-bit:0.13\n\ndocker exec -it fluent-bit bash\n\ndocker cp fluent-test.conf fluent-bit:/root/fluent-test.conf\n/fluent-bit/bin/fluent-bit -c /root/fluent-test.conf\n\ndocker stop fluent-bit; docker rm fluent-bit\n#+END_EXAMPLE\n** #  --8\u003c-------------------------- separator ------------------------\u003e8-- :noexport:\n** HALF fluent bit restart\n** HALF how fluent bit monitor dynamic input files\n** #  --8\u003c-------------------------- separator ------------------------\u003e8-- :noexport:\n** TODO fluent bit compatible for promethus\n** TODO fluent bit performance metrics\n** TODO fluent bit multiple syslog output\n** TODO enable fluent-bit buffering with filesystem backend\nhttps://fluentbit.io/documentation/0.13/getting_started/buffer.html\n** #  --8\u003c-------------------------- separator ------------------------\u003e8-- :noexport:\n** TODO verify endpoint restart: whether fluent bit agent will auto healed\n** TODO fluent bit: how to detect backend status is error\n** #  --8\u003c-------------------------- separator ------------------------\u003e8-- :noexport:\n** DONE hello world with docker\n   CLOSED: [2018-06-27 Wed 00:35]\n*** [#A] run as a container\nhttps://fluentbit.io/documentation/current/installation/docker.html\n\ndocker run -t -d --name fluent-bit  -p 2020:2020 -p 24224:24224 -v fluent-bit.conf:/etc/fluent-bit.conf --entrypoint=/bin/bash fluent/fluent-bit:0.13\n\ndocker exec -it fluent-bit bash\n\ndocker cp fluent-bit.conf fluent-bit:/root/fluent-bit.conf\n/fluent-bit/bin/fluent-bit -c /root/fluent-bit.conf\n\ndocker stop fluent-bit; docker rm fluent-bit\n*** simple test\ndocker pull fluent/fluent-bit:0.13\ndocker run -ti fluent/fluent-bit:0.13 /fluent-bit/bin/fluent-bit -i cpu -o stdout -f 1\n*** more customization\n**** fluent-bit.conf\ncat \u003e fluent-bit.conf \u003c\u003c EOF\n[SERVICE]\n    HTTP_Server  On\n    HTTP_Listen  0.0.0.0\n    HTTP_PORT    2020\n\n[INPUT]\n    Name forward\n\n    # The Listen interface, by default we listen on all of them\n    Listen 0.0.0.0\n\n    # Default TCP listener port\n    Port 24224\n\n    # Buffer (Kilobytes)\n    # ------------------\n    # Specify the size of the receiver buffer. Incoming records\n    # must be inside this limit. By default 512KB.\n    Buffer 512000\n\n[OUTPUT]\n    Name file\n    Match *\n    Path /tmp/output.txt\nEOF\n**** run daemon service\n# docker run -p 2020:2020 -p 24224:24224 --name fluent-bit -v /Users/zdenny/Dropbox/private_data/work/vmware/fluent-bit/fluent-bit.conf:/etc/fluent-bit.conf -ti fluent/fluent-bit:0.13 /fluent-bit/bin/fluent-bit -c /etc/fluent-bit.conf\ndocker run -t -d --name fluent-bit -p 2020:2020 -p 24224:24224 -v /Users/zdenny/Dropbox/private_data/work/vmware/fluent-bit/fluent-bit.conf:/etc/fluent-bit.conf --entrypoint=/bin/bash fluent/fluent-bit:0.13\n\ndocker exec -it fluent-bit bash\n\n/fluent-bit/bin/fluent-bit -c /etc/fluent-bit.conf\n\nnc localhost 24224\n**** verify health check\nhttps://fluentbit.io/documentation/current/configuration/monitoring.html\n\ncurl -s http://127.0.0.1:2020/api/v1/metrics\n**** check log\ndocker exec -it fluent-bit tail /tmp/output.txt\n**** destroy\ndocker stop fluent-bit; docker rm fluent-bit\n** BYPASS hello world doens't work: doesn't work well in mac OS\n   CLOSED: [2018-06-27 Wed 00:33]\ncat \u003e fluent-bit.conf \u003c\u003c EOF\n[SERVICE]\n    HTTP_Server  On\n    HTTP_Listen  0.0.0.0\n    HTTP_PORT    2020\n\n[INPUT]\n    Name forward\n\n    # The Listen interface, by default we listen on all of them\n    Listen 0.0.0.0\n\n    # Default TCP listener port\n    Port 24224\n\n    # Buffer (Kilobytes)\n    # ------------------\n    # Specify the size of the receiver buffer. Incoming records\n    # must be inside this limit. By default 512KB.\n    Buffer 512000\n\n[OUTPUT]\n    Name stdout\n    Match *\nEOF\n\nbin/fluent-bit -c fluent-bit.conf\n** DONE Concept Routing Tag/Match\n   CLOSED: [2018-06-26 Tue 23:45]\nhttps://fluentbit.io/documentation/current/getting_started/routing.html\n\nTag is a human-readable indicator that helps to identify the data source.\n\nIf some data have a Tag that don't have a match upon routing time, the data it's deleted.\n#+BEGIN_EXAMPLE\n[INPUT]\n    Name cpu\n    Tag  my_cpu\n\n[INPUT]\n    Name mem\n    Tag  my_mem\n\n[OUTPUT]\n    Name   es\n    Match  my_c*\n\n[OUTPUT]\n    Name   stdout\n    Match  my_mem\n#+END_EXAMPLE\n** DONE use fluent bit to monitor http endpoint\n   CLOSED: [2018-06-26 Tue 23:31]\nhttps://fluentbit.io/documentation/current/input/health.html\n\n#+BEGIN_EXAMPLE\nzdenny-a01:~ zdenny$ nc -l 8080\nzdenny-a01:~ zdenny$\n#+END_EXAMPLE\n\n#+BEGIN_EXAMPLE\nzdenny-a01:build zdenny$ fluent-bit -i health://127.0.0.1:8080 -o stdout\nFluent-Bit v0.14.0\nCopyright (C) Treasure Data\n\n[2018/06/26 23:30:41] [ info] [engine] started (pid=23405)\n[2018/06/26 23:30:42] [error] [io] TCP connection failed: 127.0.0.1:8080 (Connection refused)\n[2018/06/26 23:30:43] [error] [io] TCP connection failed: 127.0.0.1:8080 (Connection refused)\n[2018/06/26 23:30:44] [error] [io] TCP connection failed: 127.0.0.1:8080 (Connection refused)\n[2018/06/26 23:30:45] [error] [io] TCP connection failed: 127.0.0.1:8080 (Connection refused)\n[0] health.0: [1530081042.000000000, {\"alive\"=\u003efalse}]\n[1] health.0: [1530081043.000000000, {\"alive\"=\u003efalse}]\n[2] health.0: [1530081044.000000000, {\"alive\"=\u003efalse}]\n[3] health.0: [1530081045.000000000, {\"alive\"=\u003efalse}]\n[2018/06/26 23:30:47] [error] [io] TCP connection failed: 127.0.0.1:8080 (Connection refused)\n[2018/06/26 23:30:48] [error] [io] TCP connection failed: 127.0.0.1:8080 (Connection refused)\n[2018/06/26 23:30:49] [error] [io] TCP connection failed: 127.0.0.1:8080 (Connection refused)\n[0] health.0: [1530081046.000000000, {\"alive\"=\u003etrue}]\n[1] health.0: [1530081047.000000000, {\"alive\"=\u003efalse}]\n[2] health.0: [1530081048.000000000, {\"alive\"=\u003efalse}]\n[3] health.0: [1530081049.000000000, {\"alive\"=\u003efalse}]\n[4] health.0: [1530081050.000000000, {\"alive\"=\u003etrue}]\n[2018/06/26 23:30:51] [error] [io] TCP connection failed: 127.0.0.1:8080 (Connection refused)\n[2018/06/26 23:30:52] [error] [io] TCP connection failed: 127.0.0.1:8080 (Connection refused)\n[2018/06/26 23:30:53] [error] [io] TCP connection failed: 127.0.0.1:8080 (Connection refused)\n^C[engine] caught signal (SIGINT)\n[2018/06/26 23:30:53] [ info] [input] pausing health.0\n#+END_EXAMPLE\n** DONE Enable a customized flent bit plugin\n   CLOSED: [2018-06-27 Wed 21:05]\n https://github.com/fluent/fluent-bit/blob/master/GOLANG_OUTPUT_PLUGIN.md\n\n** DONE fluent bit log error message\n   CLOSED: [2018-06-28 Thu 22:22]\n    flb_info(\"[out_syslog] addr=%s\", ctx-\u003eaddr);\n\n    fprintf(stderr, \"Here6\\n\");\n\n    flb_debug(\"[out_kafka_rest] host=%s port=%i\",\n              ins-\u003ehost.name, ins-\u003ehost.port);\n** DONE fluent-bit log to file\n   CLOSED: [2018-07-02 Mon 17:12]\n /bin/fluent-bit -i cpu -o file -p path=/tmp/output.txt -f 1\n** #  --8\u003c-------------------------- separator ------------------------\u003e8-- :noexport:\n** TODO backpressure\n** TODO loadbalancing\n** TODO syslog tls\n** TODO output retry\n** TODO output to syslog\n** TODO fluentbit backpressure\n https://fluentbit.io/documentation/0.13/configuration/backpressure.html\n** TODO fluentbit support client side loadbalancing / failover\n https://github.com/fluent/fluent-bit/issues/203#issuecomment-389579836\n\n** useful link\nhttp://edsiper.linuxchile.cl/blog/\n** HALF fluent bit multiple output: https://github.com/fluent/fluent-bit/issues/305\n github/fluent-bit/conf\n** #  --8\u003c-------------------------- separator ------------------------\u003e8-- :noexport:\n** DONE fluent-bit mechanism: retry when output plugin keeps returning FLB_RETRY\n   CLOSED: [2018-08-03 Fri 09:59]\n Last day, Warren and I were wondering what retry logic fluent-bit enginee would be.\n\n When the engine tries flush one message, but our output plugin returns FLB_RETRY. Maybe the syslog endpoint is unavailable, let's say that.\n\n When it retries, our plugin keep returning FLB_RETRY. What FB will do? Keep retrying, or drop it with some max retries mechanisms?\n\n After checking the source code, it enforces with max retries mechanism. And default max retries is 0.\n So it will drop it, when we return FLB_RETRY again. \n\n We can customize the max retries via retry_limit property.\n\n After checking the source code, it enforces with max retries mechanism. But default max retries is 0.\n\n So *it will drop the message*, when we return FLB_RETRY again. \n\n We can customize the max retries via retry_limit property.\n\n 1. retry_limit configuration\n   https://github.com/fluent/fluent-bit/blob/ed4b2f09d68f1b71bed53aaa37a048700adc2b5d/src/flb_output.c#L349-L364\n 2. When output returns retry, FB engine create a retry task.\n   https://github.com/fluent/fluent-bit/blob/ed4b2f09d68f1b71bed53aaa37a048700adc2b5d/src/flb_engine.c#L177-L188\n 3. And eash retry task has attribute of task-\u003eretries. And each output plugin has attribute of retry_limit.\n   https://github.com/fluent/fluent-bit/blob/ed4b2f09d68f1b71bed53aaa37a048700adc2b5d/src/flb_task.c#L88-L138\n 4. Based on task-\u003eretries and plugin-\u003eretry_limit, it determines whether keep trying or return error\n** TODO [#A] How to integrate fluentd with vRealize Log\n** TODO [#A] try kube-fluentd-operator: https://github.com/vmware/kube-fluentd-operator \n\n** TODO reload fluentbit\n** TODO try k8s fluentd-operator\n** TODO /Users/zdenny/Dropbox/private_data/work/vmware/fluent-bit/plugins/filter_kubernetes/kubernetes.c\n** TODO fluentd-kubernetes-daemonset: https://github.com/fluent/fluentd-kubernetes-daemonset\n** TODO fluent bit log to multiple endpoints\n* fluentd                                                          :noexport:\n** how fluentd get kubernetes information\nhttps://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter\n#+BEGIN_EXAMPLE\n```{\n  \"log\": \"2015/05/05 19:54:41 \\n\",\n  \"stream\": \"stderr\",\n  \"docker\": {\n    \"id\": \"df14e0d5ae4c07284fa636d739c8fc2e6b52bc344658de7d3f08c36a2e804115\",\n  }\n  \"kubernetes\": {\n    \"host\": \"jimmi-redhat.localnet\",\n    \"pod_name\":\"fabric8-console-controller-98rqc\",\n    \"pod_id\": \"c76927af-f563-11e4-b32d-54ee7527188d\",\n    \"container_name\": \"fabric8-console-container\",\n    \"namespace_name\": \"default\",\n    \"namespace_id\": \"23437884-8e08-4d95-850b-e94378c9b2fd\",\n    \"namespace_annotations\": {\n      \"fabric8.io/git-commit\": \"5e1116f63df0bac2a80bdae2ebdc563577bbdf3c\"\n    },\n    \"namespace_labels\": {\n      \"product_version\": \"v1.0.0\"\n    },\n    \"labels\": {\n      \"component\": \"fabric8Console\"\n    }\n  }\n}```\n\nAt least from this fluentd k8s filter plugin, I don’t see more meta data compared to fluent-bit\nLet’s see\n#+END_EXAMPLE\n* #  --8\u003c-------------------------- separator ------------------------\u003e8-- :noexport:\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdennyzhang%2Fchallenges-fluent-bit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdennyzhang%2Fchallenges-fluent-bit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdennyzhang%2Fchallenges-fluent-bit/lists"}