{"id":17570666,"url":"https://github.com/kobsio/klogs","last_synced_at":"2026-01-12T08:02:22.990Z","repository":{"id":42621286,"uuid":"400601065","full_name":"kobsio/klogs","owner":"kobsio","description":"Fast, scalable and reliable logging using Fluent Bit and ClickHouse","archived":false,"fork":false,"pushed_at":"2025-10-02T05:34:21.000Z","size":1405,"stargazers_count":67,"open_issues_count":0,"forks_count":9,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-10-02T07:19:44.375Z","etag":null,"topics":["clickhouse","fluent-bit","kobs","kobsio","kubernetes","logging"],"latest_commit_sha":null,"homepage":"https://kobs.io/main/plugins/klogs/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kobsio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["ricoberger"],"custom":["https://www.paypal.me/ricoberger"]}},"created_at":"2021-08-27T18:26:24.000Z","updated_at":"2025-10-02T05:34:24.000Z","dependencies_parsed_at":"2024-03-14T09:38:40.325Z","dependency_job_id":"e761b084-50d5-4e6d-a13b-4ace46b75453","html_url":"https://github.com/kobsio/klogs","commit_stats":null,"previous_names":["kobsio/fluent-bit-clickhouse"],"tags_count":23,"template":false,"template_full_name":null,"purl":"pkg:github/kobsio/klogs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kobsio%2Fklogs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kobsio%2Fklogs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kobsio%2Fklogs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kobsio%2Fklogs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kobsio","download_url":"https://codeload.github.com/kobsio/klogs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kobsio%2Fklogs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28336970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T06:09:07.588Z","status":"ssl_error","status_checked_at":"2026-01-12T06:05:18.301Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clickhouse","fluent-bit","kobs","kobsio","kubernetes","logging"],"created_at":"2024-10-21T18:01:16.217Z","updated_at":"2026-01-12T08:02:22.979Z","avatar_url":"https://github.com/kobsio.png","language":"Go","funding_links":["https://github.com/sponsors/ricoberger","https://www.paypal.me/ricoberger"],"categories":["Integrations"],"sub_categories":["Data Transfer and Synchronization"],"readme":"# klogs\n\n**klogs** can be used to write the logs collected by\n[Fluent Bit](https://fluentbit.io) to [ClickHouse](https://clickhouse.com).\n\nYou can use [kobs](https://kobs.io) as interface to get the logs from\nClickHouse. More information regarding the klogs plugin for kobs can be found in\nthe [klogs](https://kobs.io/plugins/klogs/) documentation of kobs.\n\n![kobs](./assets/kobs.png)\n\n## Configuration\n\nAn example configuration file can be found in the\n[fluent-bit.yaml](./cluster/fluent-bit.yaml) file. The following options are\navailable:\n\n| Option                   | Description                                                                                                     | Default   |\n| ------------------------ | --------------------------------------------------------------------------------------------------------------- | --------- |\n| `Metrics_Server_Address` | The address, where the metrics server should listen on.                                                         | `:2021`   |\n| `Address`                | The address, where ClickHouse is listining on, e.g. `clickhouse-clickhouse.kube-system.svc.cluster.local:9000`. |           |\n| `Database`               | The name of the database for the logs.                                                                          | `logs`    |\n| `Username`               | The username, to authenticate to ClickHouse.                                                                    |           |\n| `Password`               | The password, to authenticate to ClickHouse.                                                                    |           |\n| `Dial_Timeout`           | ClickHouse dial timeout.                                                                                        | `10s`     |\n| `Conn_Max_Lifetime`      | ClickHouse maximum connection lifetime.                                                                         | `1h`      |\n| `Max_Idle_Conns`         | ClickHouse maximum number of idle connections.                                                                  | `1`       |\n| `Max_Open_Conns`         | ClickHouse maximum number of open connections.                                                                  | `1`       |\n| `Async_Insert`           | Use async inserts to write logs into ClickHouse.                                                                | `false`   |\n| `Wait_For_Async_Insert`  | Wait for the async insert operation.                                                                            | `false`   |\n| `Batch_Size`             | The size for how many log lines should be buffered, before they are written to ClickHouse.                      | `10000`   |\n| `Flush_Interval`         | The maximum amount of time to wait, before logs are written to ClickHouse.                                      | `60s`     |\n| `Force_Number_Fields`    | A list of fields which should be parsed as number.                                                              | `60s`     |\n| `Force_Underscores`      | Replace all `.` with `_` in keys.                                                                               | `false`   |\n| `Log_Format`             | The log format for the Fluent Bit ClickHouse plugin. Must be `console` or `json`.                               | `console` |\n| `Log_Level`              | The log level for the Fluent Bit ClickHouse plugin. Must be `DEBUG`, `INFO`, `WARN` or `ERROR`.                 | `INFO`    |\n\nThe SQL schema for ClickHouse must be created on each ClickHouse node and looks\nas follows:\n\n```sql\nCREATE DATABASE IF NOT EXISTS logs ON CLUSTER `{cluster}` ENGINE=Atomic;\n\nCREATE TABLE IF NOT EXISTS logs.logs_local ON CLUSTER `{cluster}`\n(\n    `timestamp` DateTime64(3) CODEC(Delta, LZ4),\n    `cluster` LowCardinality(String),\n    `namespace` LowCardinality(String),\n    `app` LowCardinality(String),\n    `pod_name` LowCardinality(String),\n    `container_name` LowCardinality(String),\n    `host` LowCardinality(String),\n    `fields_string` Map(LowCardinality(String), String),\n    `fields_number` Map(LowCardinality(String), Float64),\n    `log` String CODEC(ZSTD(1))\n)\nENGINE = ReplicatedMergeTree\nPARTITION BY toDate(timestamp)\nORDER BY (cluster, namespace, app, pod_name, container_name, host, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY;\n\nCREATE TABLE IF NOT EXISTS logs.logs ON CLUSTER '{cluster}' AS logs.logs_local ENGINE = Distributed('{cluster}', logs, logs_local, rand());\n```\n\nTo speedup queries for the most frequently queried fields we can create\ndedicated columns for specific fiels:\n\n```sql\nALTER TABLE logs.logs_local ON CLUSTER '{cluster}' ADD COLUMN content_level String DEFAULT fields_string['content.level']\nALTER TABLE logs.logs ON CLUSTER '{cluster}' ADD COLUMN content_level String DEFAULT fields_string['content.level']\n\nALTER TABLE logs.logs_local ON CLUSTER '{cluster}' ADD COLUMN content_response_code Float64 DEFAULT fields_number['content.response_code']\nALTER TABLE logs.logs ON CLUSTER '{cluster}' ADD COLUMN content_response_code Float64 DEFAULT fields_number['content.response_code']\n```\n\nBut those columns will be materialized only for new data and after merges. In\norder to materialize those columns for old data:\n\n- You can use `ALTER TABLE MATERIALIZE COLUMN` for ClickHouse version \u003e 21.10.\n\n```sql\nALTER TABLE logs.logs_local ON CLUSTER '{cluster}' MATERIALIZE COLUMN content_level;\nALTER TABLE logs.logs_local ON CLUSTER '{cluster}' MATERIALIZE COLUMN content_response_code;\n```\n\n- Or for older ClickHouse versions, `ALTER TABLE UPDATE`.\n\n```sql\nALTER TABLE logs.logs_local ON CLUSTER '{cluster}' UPDATE content_level = content_level WHERE 1;\nALTER TABLE logs.logs_local ON CLUSTER '{cluster}' UPDATE content_response_code = content_response_code WHERE 1;\n```\n\n## Development\n\nWe are using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) for local\ndevelopment. To create a new Kubernetes cluster using kind you can run the\n`cluster/cluster.sh` script, which will create such a cluster with a Docker\nregistry:\n\n```sh\n./cluster/cluster.sh\n```\n\nOnce the cluster is running we can build and push the Docker image for Fluent\nBit:\n\n```sh\ndocker build -f Dockerfile -t localhost:5001/klogs:latest .\ndocker push localhost:5001/klogs:latest\n```\n\nIn the next step we have to create our ClickHouse cluster via the\n[ClickHouse Operator](https://github.com/Altinity/clickhouse-operator):\n\n```sh\nkubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/refs/heads/master/deploy/operator/clickhouse-operator-install-bundle.yaml\nkubectl apply -f ./cluster/clickhouse.yaml\n```\n\nOnce ClickHouse is running we can to connect to the instance to check if the\ndatabase schema was created:\n\n```sh\nkubectl exec -n kube-system -it chi-clickhouse-example-0-0-0 -c clickhouse -- clickhouse-client -h 127.0.0.1\n```\n\n```sql\nSHOW DATABASES;\nUSE logs;\nSHOW TABLES;\nDESCRIBE logs_local;\nDESCRIBE logs;\n```\n\nNow we can deploy Fluent Bit to ingest all logs into ClickHouse:\n\n```sh\nkubectl apply -f ./cluster/fluent-bit.yaml\nkubectl logs -n kube-system -l app=fluent-bit -f\n```\n\nTo check if the logs are arriving in ClickHouse you can use the following SQL\ncommands:\n\n```sql\nSELECT count(*) FROM logs.logs;\nSELECT * FROM logs.logs LIMIT 10;\n\nSELECT count(*) FROM logs.logs_local;\nSELECT * FROM logs.logs_local LIMIT 10;\n```\n\nTo clean up all the created resources run the following commands:\n\n```sh\nkind delete cluster\ndocker stop kind-registry\ndocker rm kind-registry\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkobsio%2Fklogs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkobsio%2Fklogs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkobsio%2Fklogs/lists"}