{"id":20578763,"url":"https://github.com/cclient/kubernetes-filebeat-collector","last_synced_at":"2026-04-11T00:50:40.889Z","repository":{"id":93977345,"uuid":"142630248","full_name":"cclient/kubernetes-filebeat-collector","owner":"cclient","description":"collector kubernetes docker contain json-file and mount path files by filebeat","archived":false,"fork":false,"pushed_at":"2018-10-17T14:37:22.000Z","size":3837,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-16T22:19:57.573Z","etag":null,"topics":["alert","collector","docker","filebeat","grafana","kubernetes","log","webhook"],"latest_commit_sha":null,"homepage":"https://www.cnblogs.com/zihunqingxin/p/9380509.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cclient.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-27T22:58:40.000Z","updated_at":"2023-06-30T03:00:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"af601ed0-a862-42c5-91be-c3e3a3ff6698","html_url":"https://github.com/cclient/kubernetes-filebeat-collector","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cclient%2Fkubernetes-filebeat-collector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cclient%2Fkubernetes-filebeat-collector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cclient%2Fkubernetes-filebeat-collector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cclient%2Fkubernetes-filebeat-collector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cclient","download_url":"https://codeload.github.com/cclient/kubernetes-filebeat-collector/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242205990,"owners_count":20089251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alert","collector","docker","filebeat","grafana","kubernetes","log","webhook"],"created_at":"2024-11-16T06:14:30.089Z","updated_at":"2025-12-31T00:52:01.370Z","avatar_url":"https://github.com/cclient.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"collect docker logs attach some simple kubernetes info as kubernetes contain log\n\n## work\n\n* 1 scan docker contains path(default /var/lib/docker/containers) get docker [contain log] and [contain config]\n\n* 2 cp filebeat.yml and modify\n\n* 3 start filebeat\n\nsee why\u0026how\n\n--\n\n### k8s start\n\n#### docker build(modify filebeat.yml \"localhost:9200\" to your es)\n\ndocker build -t cuidapeng/filebeat-k8s-collector:6.3.0 ./\n\nkubectl apply -f deploy-filebeat.yml\n\n### local start\n\npython filebeat.py -l /etc/hosts -m *.log -t start\n\n### local stop\n\npython filebeat.py -t stop\n\n--\n\n$ pgrep -a  -f filebeat\n\n14912 filebeat -e -c filebeat_consume_consume-577cd986c7-8mk4h_caed8938198152778e3715f4bd3c00795f40c812d2cdb87dadfe8b0bb058390f.yml\n\nfilebeat process info:\n\n```finfo\n{\n\"pid\": \"14912\",\n\"conf\": \"filebeat_consume_consume-577cd986c7-8mk4h_caed8938198152778e3715f4bd3c00795f40c812d2cdb87dadfe8b0bb058390f.yml\",\n\"cid\": \"caed8938198152778e3715f4bd3c00795f40c812d2cdb87dadfe8b0bb058390f\"\n}\n```\n\ndocker contain simple info:\n\n```cinfo\n{\n    \"io_kubernetes_docker_type\": \"container\",\n    \"io_kubernetes_pod_namespace\": \"defalt\",\n    \"hostname\": \"consume-577cd986c7-8mk4h\",\n    \"io_kubernetes_pod_name\": \"consume-577cd986c7-8mk4h\",\n    \"id\": \"caed8938198152778e3715f4bd3c00795f40c812d2cdb87dadfe8b0bb058390f\",\n    \"io_kubernetes_container_name\": \"consume\"\n}\n```\n\nfilebeat_[io_kubernetes_container_name]_[io_kubernetes_pod_name]_[id].yml\n\nfilebeat_consume_consume-577cd986c7-8mk4h_caed8938198152778e3715f4bd3c00795f40c812d2cdb87dadfe8b0bb058390f.yml\n\norgi filebeat.yml\n\n```orgi\nfilebeat.inputs:\n- type: log\n    - docker_logfile_path\n    - docker_mount_logfile_path\n  fields:\n    namespace: \"default\"\n    name: \"io_kubernetes_container_name\"\n    pod_name: \"io_kubernetes_pod_name\"\noutput.elasticsearch:\n  hosts: [\"localhost:9200\"]\nsetup.kibana:\n  host: \"localhost:5601\"\n```\n\nfilebeat_consume_consume-577cd986c7-8mk4h_caed8938198152778e3715f4bd3c00795f40c812d2cdb87dadfe8b0bb058390f.yml\n\n```desc\nfilebeat.inputs:\n- type: log\n  paths:\n    - /var/lib/docker/containers/caed8938198152778e3715f4bd3c00795f40c812d2cdb87dadfe8b0bb058390f/caed8938198152778e3715f4bd3c00795f40c812d2cdb87dadfe8b0bb058390f-json.log\n    - /var/lib/kubelet/pods/444e5054-8eec-11e8-b709-f01fafd51338/etc-hosts/*.log\n  fields:\n    namespace: \"default\"\n    name: \"consume\"\n    pod_name: \"consume-577cd986c7-8mk4h\"\noutput.elasticsearch:\n  hosts: [\"localhost:9200\"]\nsetup.kibana:\n  host: \"localhost:5601\"\n```  \n\nlog demo\n\n```demo\n{\n    \"_index\": \"filebeat-6.3.0-2018.07.25\",\n    \"_type\": \"doc\",\n    \"_id\": \"mOeJ0GQBE8FuUBDGk_Sj\",\n    \"_version\": 1,\n    \"_score\": null,\n    \"_source\": {\n        \"@timestamp\": \"2018-07-25T08:23:53.531Z\",\n        \"source\": \"/var/lib/docker/containers/caed8938198152778e3715f4bd3c00795f40c812d2cdb87dadfe8b0bb058390f/caed8938198152778e3715f4bd3c00795f40c812d2cdb87dadfe8b0bb058390f-json.log\",\n        \"offset\": 347825,\n        \"message\": \"{\\\"log\\\":\\\"exist /tmp/download/20180725_16:12:08.zip true\\\\n\\\",\\\"stream\\\":\\\"stdout\\\",\\\"time\\\":\\\"2018-07-25T08:23:47.947640042Z\\\"}\",\n        \"input\": {\n            \"type\": \"log\"\n        },\n        \"fields\": {\n            \"pod_name\": \"consume-577cd986c7-8mk4h\",\n            \"namespace\": \"default\",\n            \"name\": \"consume\"\n        },\n        \"prospector\": {\n            \"type\": \"log\"\n        },\n        \"beat\": {\n            \"name\": \"host0\",\n            \"hostname\": \"host0\",\n            \"version\": \"6.3.0\"\n        },\n        \"host\": {\n            \"name\": \"host0\"\n        }\n    },\n    \"fields\": {\n        \"@timestamp\": [\n            \"2018-07-25T08:23:53.531Z\"\n        ]\n    },\n    \"highlight\": {\n        \"fields.name\": [\n            \"@kibana-highlighted-field@consume@/kibana-highlighted-field@\"\n        ]\n    },\n    \"sort\": [\n        1532507033531\n    ]\n}\n```\n\nyaml demo\n\n```k8s\napiVersion: extensions/v1beta1\nkind: Deployment\nmetadata:\n  name: consume\n  namespace: default\n  labels:\n    app: consume\nspec:\n  replicas: 1\n  template:\n    metadata:\n      labels:\n        app: consume\n    spec:\n      containers:\n      - name: consume\n        imagePullPolicy: Always\n        image: cuidapeng/busybox-curl:v0.1\n        command: ['sh', '-c', 'echo The app is running! \u0026\u0026 sleep 3600']\n```\n\nspec.spec.containers.name=io_kubernetes_container_name","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcclient%2Fkubernetes-filebeat-collector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcclient%2Fkubernetes-filebeat-collector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcclient%2Fkubernetes-filebeat-collector/lists"}