{"id":17908278,"url":"https://github.com/laisky/go-fluentd","last_synced_at":"2025-09-03T21:33:40.912Z","repository":{"id":45566250,"uuid":"149221449","full_name":"Laisky/go-fluentd","owner":"Laisky","description":"rewrite fluentd in golang","archived":false,"fork":false,"pushed_at":"2023-05-05T02:28:39.000Z","size":1181,"stargazers_count":111,"open_issues_count":1,"forks_count":22,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-23T19:53:25.772Z","etag":null,"topics":["fluentd","golang","log-parser"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Laisky.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-18T03:09:34.000Z","updated_at":"2025-03-04T08:35:18.000Z","dependencies_parsed_at":"2024-06-18T18:40:13.794Z","dependency_job_id":"091956ec-33ba-45a9-8731-3851fa003cec","html_url":"https://github.com/Laisky/go-fluentd","commit_stats":null,"previous_names":[],"tags_count":55,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Laisky%2Fgo-fluentd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Laisky%2Fgo-fluentd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Laisky%2Fgo-fluentd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Laisky%2Fgo-fluentd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Laisky","download_url":"https://codeload.github.com/Laisky/go-fluentd/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245164804,"owners_count":20571186,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluentd","golang","log-parser"],"created_at":"2024-10-28T19:15:15.775Z","updated_at":"2025-03-23T20:31:00.728Z","avatar_url":"https://github.com/Laisky.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Go-Fluentd\n\nRewrite fluentd-server by Golang, Higher performance with less resource requirement.\n\n* At-Least-Once guarantee(disk WAL)\n* log concatenation by head regexp expression\n* log parsing by regexp expression(support embedded json)\n* log filter by custom plugins(acceptorfilters \u0026 tagfilters)\n* multiple receivers(support multiple protocols: msgpack, http, syslog, kafka, ...)\n* multiple senders(support multiple backend: elasticsearch, fluentd, ...)\n* multiple environments deployment(`--env`: sit, perf, uat, prod)\n\n![GitHub release](https://img.shields.io/github/release/Laisky/go-fluentd.svg)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Build Status](https://travis-ci.org/Laisky/go-fluentd.svg?branch=master)](https://travis-ci.org/Laisky/go-fluentd)\n[![codecov](https://codecov.io/gh/Laisky/go-fluentd/branch/master/graph/badge.svg)](https://codecov.io/gh/Laisky/go-fluentd)\n[![Commitizen friendly](https://img.shields.io/badge/commitizen-friendly-brightgreen.svg)](http://commitizen.github.io/cz-cli/)\n[![Go Report Card](https://goreportcard.com/badge/gofluentd)](https://goreportcard.com/report/gofluentd)\n[![GoDoc](https://godoc.org/gofluentd?status.svg)](https://godoc.org/gofluentd)\n\nAlready running on our PRODUCION since 2018/9.\n\nWhen processing 1000mbps logs flood:\n\n* dstat\n\n  ![dstat](https://s1.laisky.com/uploads/2020/01/dstat.png)\n* monitor\n\n  ![monitor](https://s2.laisky.com/uploads/2020/01/monitor.png)\n* profile\n\n  ![profile](https://s3.laisky.com/uploads/2020/01/profile.png)\n\nDocuments:\n\n- [中文文档](https://blog.laisky.com/p/go-fluentd/)\n- [Quick Start](docs/quickstart.md)\n- [Build with Docker](.docker)\n\n\n## Description\n\nLogAggregator + Concator + Parser + Producer.\n\nOrigin logs emitted from docker look like:\n\n```\n'{\"container_id\": \"xxxxx\", \"log\": \"2018-03-06 16:56:22.514 | mscparea | ERROR  | http-nio-8080-exec-1 | com.laisky.cloud.cp.core.service.impl.CPBusiness.reflectAdapterRequest | 84:'\n'{\"container_id\": \"xxxxx\", \"log\": \"Exception in thread \"main\" java.lang.IllegalStateException: A book has a null property\"}'\n'{\"container_id\": \"xxxxx\", \"log\": \"\\tat com.example.myproject.Author.getBookIds(Author.java:38)\"}'\n'{\"container_id\": \"xxxxx\", \"log\": \"\\tat com.example.myproject.Bootstrap.main(Bootstrap.java:14)\"}'\n'{\"container_id\": \"xxxxx\", \"log\": \"Caused by: java.lang.NullPointerException\"}'\n'{\"container_id\": \"xxxxx\", \"log\": \"\\tat com.example.myproject.Book.getId(Book.java:22)\"}'\n'{\"container_id\": \"xxxxx\", \"log\": \"\\tat com.example.myproject.Author.getBookIds(Author.java:35)\"}'\n'{\"container_id\": \"xxxxx\", \"log\": \"\\t... 1 more\"}'\n```\n\nAfter Concator(TagPipeline \u003e concator_f):\n\n```go\n\u0026FluentMsg{\n    Id: 12345,\n    Tag: \"spring.sit\",\n    Message: map[string]interface{}{\n        \"container_id\": \"xxxxx\",\n        \"log\": \"2018-03-06 16:56:22.514 | mscparea | ERROR  | http-nio-8080-exec-1 | com.laisky.cloud.cp.core.service.impl.CPBusiness.reflectAdapterRequest | 84: Exception in thread \"main\" java.lang.IllegalStateException: A book has a null property\\n\\tat com.example.myproject.Author.getBookIds(Author.java:38)\\n\\tat com.example.myproject.Bootstrap.main(Bootstrap.java:14)\\nCaused by: java.lang.NullPointerException\\n\\tat com.example.myproject.Book.getId(Book.java:22)\\n\\tat com.example.myproject.Author.getBookIds(Author.java:35)\\n\\t... 1 more\",\n    },\n}\n```\n\nAfter Parser(TagPipeline \u003e parser_f):\n\n```go\n\u0026FluentMsg{\n    Id: 12345,\n    Tag: \"spring.sit\",\n    Message: map[string]interface{}{\n        \"container_id\": \"xxxxx\",\n        \"time\": \"2018-03-06 16:56:22.514\",\n        \"level\": \"ERROR\",\n        \"app\": \"mscparea\",\n        \"thread\": \"http-nio-8080-exec-1\",\n        \"class\": \"com.laisky.cloud.cp.core.service.impl.CPBusiness.reflectAdapterRequest\",\n        \"line\": 84,\n        \"message\": \"Exception in thread \"main\" java.lang.IllegalStateException: A book has a null property\\n\\tat com.example.myproject.Author.getBookIds(Author.java:38)\\n\\tat com.example.myproject.Bootstrap.main(Bootstrap.java:14)\\nCaused by: java.lang.NullPointerException\\n\\tat com.example.myproject.Book.getId(Book.java:22)\\n\\tat com.example.myproject.Author.getBookIds(Author.java:35)\\n\\t... 1 more\",\n    },\n}\n```\n\nThen Producer can send logs to anywhere (depends on Senders).\n\n\n\n## Run\n\ndirectly run:\n\n```sh\ngo run -race main.go \\\n  --config=./docs/settings/tiny_settings.yml \\\n  --env=sit \\\n  --log-level=debug\n```\n\nrun by docker:\n\n```sh\ndocker run -itd --rm --name=go-fluentd -p 24225:24225 -p 8080:8080 \\\n    -v /etc/configs/go-fluentd:/etc/go-fluentd \\\n    -v /data/log/fluentd/go-fluentd:/data/log/fluentd/go-fluentd\n    ppcelery/go-fluentd:1.13.1 \\\n    ./go-fluentd \\\n        --config=/etc/go-fluentd/settings.yml \\\n        --env=perf \\\n        --addr=0.0.0.0:8080\n        --host=x.x.x.x\n        --enable-auto-gc\n```\n\n\n### docker images version\n\n- stable\n- release\n- dev\n- `\u003cfeature taskid\u003e`\n\n\n## Roles\n\n- Acceptor (consists of Recvs)\n- AcceptorPipeline (consists of AcceptorFilters)\n- Journal\n- Dispatcher\n- TagPipeline (consists of TagFilters)\n    - Concator\n    - Parser for each tag\n- PostPipeline (consists of PostFilters)\n- Producer (consists of Senders)\n\n\n![architecture](docs/architecture.jpg)\n\n\n### Acceptor\n\nContains multiply Recvs (such as KafkRecv \u0026 FluentdRecv),\ncan listening tcp port or fetch msg from kafka brokers.\n\n\n### AcceptorPipeline\n\nContains multiply AcceptorFilters, be used for ignore or retag specific messages.\nAll filters should return very fast to avoid blocking.\n\n\n### Journal\n\n...\n\n\n### Dispatcher\n\n...\n\n\n### TagPipeline\n\n...\n\n\n### PostPipeline\n\n...\n\n\n### Producer\n\n...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaisky%2Fgo-fluentd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flaisky%2Fgo-fluentd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaisky%2Fgo-fluentd/lists"}