{"id":13830332,"url":"https://github.com/samber/slog-kafka","last_synced_at":"2025-04-22T04:33:46.161Z","repository":{"id":159395692,"uuid":"634630611","full_name":"samber/slog-kafka","owner":"samber","description":"🚨 slog: Kafka handler","archived":false,"fork":false,"pushed_at":"2024-05-01T18:40:46.000Z","size":102,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-05-02T01:14:01.937Z","etag":null,"topics":["attribute","broker","error","go","golang","handler","kafka","log","log-level","logger","middleware","slog","structured-logging","topic"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/samber/slog-kafka/v2","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/samber.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["samber"]}},"created_at":"2023-04-30T18:43:38.000Z","updated_at":"2024-05-29T22:52:34.838Z","dependencies_parsed_at":"2023-05-21T03:15:37.695Z","dependency_job_id":"5ff030a3-34fd-4a4b-90a9-98ba92024a5c","html_url":"https://github.com/samber/slog-kafka","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samber%2Fslog-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samber%2Fslog-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samber%2Fslog-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samber%2Fslog-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/samber","download_url":"https://codeload.github.com/samber/slog-kafka/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223888365,"owners_count":17220083,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attribute","broker","error","go","golang","handler","kafka","log","log-level","logger","middleware","slog","structured-logging","topic"],"created_at":"2024-08-04T10:00:58.761Z","updated_at":"2024-11-09T22:09:29.483Z","avatar_url":"https://github.com/samber.png","language":"Go","funding_links":["https://github.com/sponsors/samber"],"categories":["Log forwarding"],"sub_categories":[],"readme":"\n# slog: Kafka handler\n\n[![tag](https://img.shields.io/github/tag/samber/slog-kafka.svg)](https://github.com/samber/slog-kafka/releases)\n![Go Version](https://img.shields.io/badge/Go-%3E%3D%201.21-%23007d9c)\n[![GoDoc](https://godoc.org/github.com/samber/slog-kafka?status.svg)](https://pkg.go.dev/github.com/samber/slog-kafka)\n![Build Status](https://github.com/samber/slog-kafka/actions/workflows/test.yml/badge.svg)\n[![Go report](https://goreportcard.com/badge/github.com/samber/slog-kafka)](https://goreportcard.com/report/github.com/samber/slog-kafka)\n[![Coverage](https://img.shields.io/codecov/c/github/samber/slog-kafka)](https://codecov.io/gh/samber/slog-kafka)\n[![Contributors](https://img.shields.io/github/contributors/samber/slog-kafka)](https://github.com/samber/slog-kafka/graphs/contributors)\n[![License](https://img.shields.io/github/license/samber/slog-kafka)](./LICENSE)\n\nA [Kafka](https://kafka.apache.org) Handler for [slog](https://pkg.go.dev/log/slog) Go library.\n\n\u003cdiv align=\"center\"\u003e\n  \u003chr\u003e\n  \u003csup\u003e\u003cb\u003eSponsored by:\u003c/b\u003e\u003c/sup\u003e\n  \u003cbr\u003e\n  \u003ca href=\"https://quickwit.io?utm_campaign=github_sponsorship\u0026utm_medium=referral\u0026utm_content=samber-slog-kafka\u0026utm_source=github\"\u003e\n    \u003cdiv\u003e\n      \u003cimg src=\"https://github.com/samber/oops/assets/2951285/49aaaa2b-b8c6-4f21-909f-c12577bb6a2e\" width=\"240\" alt=\"Quickwit\"\u003e\n    \u003c/div\u003e\n    \u003cdiv\u003e\n      Cloud-native search engine for observability - An OSS alternative to Splunk, Elasticsearch, Loki, and Tempo.\n    \u003c/div\u003e\n  \u003c/a\u003e\n  \u003chr\u003e\n\u003c/div\u003e\n\n**See also:**\n\n- [slog-multi](https://github.com/samber/slog-multi): `slog.Handler` chaining, fanout, routing, failover, load balancing...\n- [slog-formatter](https://github.com/samber/slog-formatter): `slog` attribute formatting\n- [slog-sampling](https://github.com/samber/slog-sampling): `slog` sampling policy\n- [slog-mock](https://github.com/samber/slog-mock): `slog.Handler` for test purposes\n\n**HTTP middlewares:**\n\n- [slog-gin](https://github.com/samber/slog-gin): Gin middleware for `slog` logger\n- [slog-echo](https://github.com/samber/slog-echo): Echo middleware for `slog` logger\n- [slog-fiber](https://github.com/samber/slog-fiber): Fiber middleware for `slog` logger\n- [slog-chi](https://github.com/samber/slog-chi): Chi middleware for `slog` logger\n- [slog-http](https://github.com/samber/slog-http): `net/http` middleware for `slog` logger\n\n**Loggers:**\n\n- [slog-zap](https://github.com/samber/slog-zap): A `slog` handler for `Zap`\n- [slog-zerolog](https://github.com/samber/slog-zerolog): A `slog` handler for `Zerolog`\n- [slog-logrus](https://github.com/samber/slog-logrus): A `slog` handler for `Logrus`\n\n**Log sinks:**\n\n- [slog-datadog](https://github.com/samber/slog-datadog): A `slog` handler for `Datadog`\n- [slog-betterstack](https://github.com/samber/slog-betterstack): A `slog` handler for `Betterstack`\n- [slog-rollbar](https://github.com/samber/slog-rollbar): A `slog` handler for `Rollbar`\n- [slog-loki](https://github.com/samber/slog-loki): A `slog` handler for `Loki`\n- [slog-sentry](https://github.com/samber/slog-sentry): A `slog` handler for `Sentry`\n- [slog-syslog](https://github.com/samber/slog-syslog): A `slog` handler for `Syslog`\n- [slog-logstash](https://github.com/samber/slog-logstash): A `slog` handler for `Logstash`\n- [slog-fluentd](https://github.com/samber/slog-fluentd): A `slog` handler for `Fluentd`\n- [slog-graylog](https://github.com/samber/slog-graylog): A `slog` handler for `Graylog`\n- [slog-quickwit](https://github.com/samber/slog-quickwit): A `slog` handler for `Quickwit`\n- [slog-slack](https://github.com/samber/slog-slack): A `slog` handler for `Slack`\n- [slog-telegram](https://github.com/samber/slog-telegram): A `slog` handler for `Telegram`\n- [slog-mattermost](https://github.com/samber/slog-mattermost): A `slog` handler for `Mattermost`\n- [slog-microsoft-teams](https://github.com/samber/slog-microsoft-teams): A `slog` handler for `Microsoft Teams`\n- [slog-webhook](https://github.com/samber/slog-webhook): A `slog` handler for `Webhook`\n- [slog-kafka](https://github.com/samber/slog-kafka): A `slog` handler for `Kafka`\n- [slog-nats](https://github.com/samber/slog-nats): A `slog` handler for `NATS`\n- [slog-parquet](https://github.com/samber/slog-parquet): A `slog` handler for `Parquet` + `Object Storage`\n- [slog-channel](https://github.com/samber/slog-channel): A `slog` handler for Go channels\n\n## 🚀 Install\n\n```sh\ngo get github.com/samber/slog-kafka/v2\n```\n\n**Compatibility**: go \u003e= 1.21\n\nNo breaking changes will be made to exported APIs before v3.0.0.\n\n## 💡 Usage\n\nGoDoc: [https://pkg.go.dev/github.com/samber/slog-kafka/v2](https://pkg.go.dev/github.com/samber/slog-kafka/v2)\n\n### Handler options\n\n```go\ntype Option struct {\n\t// log level (default: debug)\n\tLevel     slog.Leveler\n\n\t// Kafka Writer\n\tKafkaWriter *kafka.Writer\n\tTimeout time.Duration // default: 60s\n\n\t// optional: customize Kafka event builder\n\tConverter Converter\n\t// optional: custom marshaler\n\tMarshaler func(v any) ([]byte, error)\n\t// optional: fetch attributes from context\n\tAttrFromContext []func(ctx context.Context) []slog.Attr\n\n\t// optional: see slog.HandlerOptions\n\tAddSource   bool\n\tReplaceAttr func(groups []string, a slog.Attr) slog.Attr\n}\n```\n\nOther global parameters:\n\n```go\nslogkafka.SourceKey = \"source\"\nslogkafka.ContextKey = \"extra\"\nslogkafka.RequestKey = \"request\"\nslogkafka.ErrorKeys = []string{\"error\", \"err\"}\nslogkafka.RequestIgnoreHeaders = false\n```\n\n### Supported attributes\n\nThe following attributes are interpreted by `slogkafka.DefaultConverter`:\n\n| Atribute name    | `slog.Kind`       | Underlying type |\n| ---------------- | ----------------- | --------------- |\n| \"user\"           | group (see below) |                 |\n| \"error\"          | any               | `error`         |\n| \"request\"        | any               | `*http.Request` |\n| other attributes | *                 |                 |\n\nOther attributes will be injected in `extra` field.\n\nUsers must be of type `slog.Group`. Eg:\n\n```go\nslog.Group(\"user\",\n    slog.String(\"id\", \"user-123\"),\n    slog.String(\"username\", \"samber\"),\n    slog.Time(\"created_at\", time.Now()),\n)\n```\n\n### Example\n\n```go\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\tslogkafka \"github.com/samber/slog-kafka/v2\"\n\t\"github.com/segmentio/kafka-go\"\n\n\t\"log/slog\"\n)\n\nfunc main() {\n\t// docker-compose up -d\n\n\turi := \"127.0.0.1:9092\"\n\n\tdialer := \u0026kafka.Dialer{\n\t\tTimeout:   10 * time.Second,\n\t\tDualStack: true,\n\t}\n\n\tconn, err := dialer.DialContext(context.Background(), \"tcp\", uri)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\terr = conn.CreateTopics(kafka.TopicConfig{\n\t\tTopic:             \"logs\",\n\t\tNumPartitions:     12,\n\t\tReplicationFactor: 1,\n\t})\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\twriter := kafka.NewWriter(kafka.WriterConfig{\n\t\tBrokers: []string{uri},\n\t\tTopic:   \"logs\",\n\t\tDialer:  dialer,\n\n\t\tAsync:       true,\t// !\n\t\tBalancer:    \u0026kafka.Hash{},\n\t\tMaxAttempts: 3,\n\n\t\tLogger: kafka.LoggerFunc(func(msg string, args ...interface{}) {\n\t\t\tfmt.Printf(msg+\"\\n\", args...)\n\t\t}),\n\t\tErrorLogger: kafka.LoggerFunc(func(msg string, args ...interface{}) {\n\t\t\tfmt.Printf(msg+\"\\n\", args...)\n\t\t}),\n\t})\n\n\tdefer writer.Close()\n\tdefer conn.Close()\n\n\tlogger := slog.New(slogkafka.Option{Level: slog.LevelDebug, KafkaWriter: writer}.NewKafkaHandler())\n\tlogger = logger.With(\"release\", \"v1.0.0\")\n\n\tlogger.\n\t\tWith(\n\t\t\tslog.Group(\"user\",\n\t\t\t\tslog.String(\"id\", \"user-123\"),\n\t\t\t\tslog.Time(\"created_at\", time.Now()),\n\t\t\t),\n\t\t).\n\t\tWith(\"error\", fmt.Errorf(\"an error\")).\n\t\tError(\"a message\")\n}\n```\n\nKafka message:\n\n```json\n{\n  \"level\": \"ERROR\",\n\t\"logger\": \"samber/slog-kafka\",\n\t\"message\": \"a message\",\n\t\"timestamp\": \"2023-04-30T01:33:21.676768Z\",\n\t\"error\": {\n\t\t\"error\": \"an error\",\n\t\t\"kind\": \"*errors.errorString\",\n\t\t\"stack\": null\n\t},\n\t\"extra\": {\n\t\t\"release\": \"v1.0.0\"\n\t},\n\t\"user\": {\n\t\t\"created_at\": \"2023-04-30T01:33:21.676704Z\",\n\t\t\"id\": \"user-123\"\n\t}\n}\n```\n\n### Tracing\n\nImport the samber/slog-otel library.\n\n```go\nimport (\n\tslogkafka \"github.com/samber/slog-kafka\"\n\tslogotel \"github.com/samber/slog-otel\"\n\t\"go.opentelemetry.io/otel/sdk/trace\"\n)\n\nfunc main() {\n\ttp := trace.NewTracerProvider(\n\t\ttrace.WithSampler(trace.AlwaysSample()),\n\t)\n\ttracer := tp.Tracer(\"hello/world\")\n\n\tctx, span := tracer.Start(context.Background(), \"foo\")\n\tdefer span.End()\n\n\tspan.AddEvent(\"bar\")\n\n\tlogger := slog.New(\n\t\tslogkafka.Option{\n\t\t\t// ...\n\t\t\tAttrFromContext: []func(ctx context.Context) []slog.Attr{\n\t\t\t\tslogotel.ExtractOtelAttrFromContext([]string{\"tracing\"}, \"trace_id\", \"span_id\"),\n\t\t\t},\n\t\t}.NewKafkaHandler(),\n\t)\n\n\tlogger.ErrorContext(ctx, \"a message\")\n}\n```\n\n## 🤝 Contributing\n\n- Ping me on twitter [@samuelberthe](https://twitter.com/samuelberthe) (DMs, mentions, whatever :))\n- Fork the [project](https://github.com/samber/slog-kafka)\n- Fix [open issues](https://github.com/samber/slog-kafka/issues) or request new features\n\nDon't hesitate ;)\n\n```bash\n# Install some dev dependencies\nmake tools\n\n# Run tests\nmake test\n# or\nmake watch-test\n```\n\n## 👤 Contributors\n\n![Contributors](https://contrib.rocks/image?repo=samber/slog-kafka)\n\n## 💫 Show your support\n\nGive a ⭐️ if this project helped you!\n\n[![GitHub Sponsors](https://img.shields.io/github/sponsors/samber?style=for-the-badge)](https://github.com/sponsors/samber)\n\n## 📝 License\n\nCopyright © 2023 [Samuel Berthe](https://github.com/samber).\n\nThis project is [MIT](./LICENSE) licensed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamber%2Fslog-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsamber%2Fslog-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamber%2Fslog-kafka/lists"}