{"id":13830239,"url":"https://github.com/samber/slog-sampling","last_synced_at":"2025-04-22T04:33:55.596Z","repository":{"id":168753188,"uuid":"642949339","full_name":"samber/slog-sampling","owner":"samber","description":"🚨 slog sampling: drop repetitive log records","archived":false,"fork":false,"pushed_at":"2024-04-14T02:03:16.000Z","size":60,"stargazers_count":37,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-14T14:53:23.678Z","etag":null,"topics":["go","golang","handler","log-level","logger","logging","middleware","rate-limiting","sampling","slog","structured-logging","throughput"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/samber/slog-sample","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/samber.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["samber"]}},"created_at":"2023-05-19T18:09:20.000Z","updated_at":"2024-05-01T20:09:31.225Z","dependencies_parsed_at":"2023-10-16T11:11:50.585Z","dependency_job_id":"402f4bcf-1ef3-4d86-a0fc-7fd911da65df","html_url":"https://github.com/samber/slog-sampling","commit_stats":null,"previous_names":["samber/slog-sampling"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samber%2Fslog-sampling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samber%2Fslog-sampling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samber%2Fslog-sampling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samber%2Fslog-sampling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/samber","download_url":"https://codeload.github.com/samber/slog-sampling/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223888363,"owners_count":17220083,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","handler","log-level","logger","logging","middleware","rate-limiting","sampling","slog","structured-logging","throughput"],"created_at":"2024-08-04T10:00:57.678Z","updated_at":"2024-11-09T22:09:40.621Z","avatar_url":"https://github.com/samber.png","language":"Go","readme":"\n# Slog sampling policy\n\n[![tag](https://img.shields.io/github/tag/samber/slog-sampling.svg)](https://github.com/samber/slog-sampling/releases)\n![Go Version](https://img.shields.io/badge/Go-%3E%3D%201.21-%23007d9c)\n[![GoDoc](https://godoc.org/github.com/samber/slog-sampling?status.svg)](https://pkg.go.dev/github.com/samber/slog-sampling)\n![Build Status](https://github.com/samber/slog-sampling/actions/workflows/test.yml/badge.svg)\n[![Go report](https://goreportcard.com/badge/github.com/samber/slog-sampling)](https://goreportcard.com/report/github.com/samber/slog-sampling)\n[![Coverage](https://img.shields.io/codecov/c/github/samber/slog-sampling)](https://codecov.io/gh/samber/slog-sampling)\n[![Contributors](https://img.shields.io/github/contributors/samber/slog-sampling)](https://github.com/samber/slog-sampling/graphs/contributors)\n[![License](https://img.shields.io/github/license/samber/slog-sampling)](./LICENSE)\n\nA middleware that samples incoming records which caps the CPU and I/O load of logging while attempting to preserve a representative subset of your logs.\n\nSampling fixes throughput by dropping repetitive log entries.\n\n\u003cdiv align=\"center\"\u003e\n  \u003chr\u003e\n  \u003csup\u003e\u003cb\u003eSponsored by:\u003c/b\u003e\u003c/sup\u003e\n  \u003cbr\u003e\n  \u003ca href=\"https://quickwit.io?utm_campaign=github_sponsorship\u0026utm_medium=referral\u0026utm_content=samber-slog-sampling\u0026utm_source=github\"\u003e\n    \u003cdiv\u003e\n      \u003cimg src=\"https://github.com/samber/oops/assets/2951285/49aaaa2b-b8c6-4f21-909f-c12577bb6a2e\" width=\"240\" alt=\"Quickwit\"\u003e\n    \u003c/div\u003e\n    \u003cdiv\u003e\n      Cloud-native search engine for observability - An OSS alternative to Splunk, Elasticsearch, Loki, and Tempo.\n    \u003c/div\u003e\n  \u003c/a\u003e\n  \u003chr\u003e\n\u003c/div\u003e\n\n**See also:**\n\n- [slog-multi](https://github.com/samber/slog-multi): `slog.Handler` chaining, fanout, routing, failover, load balancing...\n- [slog-formatter](https://github.com/samber/slog-formatter): `slog` attribute formatting\n- [slog-sampling](https://github.com/samber/slog-sampling): `slog` sampling policy\n- [slog-mock](https://github.com/samber/slog-mock): `slog.Handler` for test purposes\n\n**HTTP middlewares:**\n\n- [slog-gin](https://github.com/samber/slog-gin): Gin middleware for `slog` logger\n- [slog-echo](https://github.com/samber/slog-echo): Echo middleware for `slog` logger\n- [slog-fiber](https://github.com/samber/slog-fiber): Fiber middleware for `slog` logger\n- [slog-chi](https://github.com/samber/slog-chi): Chi middleware for `slog` logger\n- [slog-http](https://github.com/samber/slog-http): `net/http` middleware for `slog` logger\n\n**Loggers:**\n\n- [slog-zap](https://github.com/samber/slog-zap): A `slog` handler for `Zap`\n- [slog-zerolog](https://github.com/samber/slog-zerolog): A `slog` handler for `Zerolog`\n- [slog-logrus](https://github.com/samber/slog-logrus): A `slog` handler for `Logrus`\n\n**Log sinks:**\n\n- [slog-datadog](https://github.com/samber/slog-datadog): A `slog` handler for `Datadog`\n- [slog-betterstack](https://github.com/samber/slog-betterstack): A `slog` handler for `Betterstack`\n- [slog-rollbar](https://github.com/samber/slog-rollbar): A `slog` handler for `Rollbar`\n- [slog-loki](https://github.com/samber/slog-loki): A `slog` handler for `Loki`\n- [slog-sentry](https://github.com/samber/slog-sentry): A `slog` handler for `Sentry`\n- [slog-syslog](https://github.com/samber/slog-syslog): A `slog` handler for `Syslog`\n- [slog-logstash](https://github.com/samber/slog-logstash): A `slog` handler for `Logstash`\n- [slog-fluentd](https://github.com/samber/slog-fluentd): A `slog` handler for `Fluentd`\n- [slog-graylog](https://github.com/samber/slog-graylog): A `slog` handler for `Graylog`\n- [slog-quickwit](https://github.com/samber/slog-quickwit): A `slog` handler for `Quickwit`\n- [slog-slack](https://github.com/samber/slog-slack): A `slog` handler for `Slack`\n- [slog-telegram](https://github.com/samber/slog-telegram): A `slog` handler for `Telegram`\n- [slog-mattermost](https://github.com/samber/slog-mattermost): A `slog` handler for `Mattermost`\n- [slog-microsoft-teams](https://github.com/samber/slog-microsoft-teams): A `slog` handler for `Microsoft Teams`\n- [slog-webhook](https://github.com/samber/slog-webhook): A `slog` handler for `Webhook`\n- [slog-kafka](https://github.com/samber/slog-kafka): A `slog` handler for `Kafka`\n- [slog-nats](https://github.com/samber/slog-nats): A `slog` handler for `NATS`\n- [slog-parquet](https://github.com/samber/slog-parquet): A `slog` handler for `Parquet` + `Object Storage`\n- [slog-channel](https://github.com/samber/slog-channel): A `slog` handler for Go channels\n\n## 🚀 Install\n\n```sh\ngo get github.com/samber/slog-sampling\n```\n\n**Compatibility**: go \u003e= 1.21\n\nNo breaking changes will be made to exported APIs before v2.0.0.\n\n## 💡 Usage\n\nGoDoc: [https://pkg.go.dev/github.com/samber/slog-sampling](https://pkg.go.dev/github.com/samber/slog-sampling)\n\n### Middlewares\n\n3 strategies are available:\n- [Uniform sampling](#uniform-sampling): drop % of logs\n- [Threshold sampling](#threshold-sampling): drop % of logs after a threshold\n- [Absolute sampling](#absolute-sampling): limit logs throughput to a fixed number of records\n- [Custom sampler](#custom-sampler)\n\nThe sampling middleware can be used standalone or with the `slog-multi` helpers.\n\nA combination of multiple sampling strategies can be chained. Eg:\n- drop when a single log message is produced more than 100 times per second\n- drop above 1000 log records per second (globally)\n\n### Matchers\n\nSimilar log records can be deduplicated and rate-limited using the `Matcher` API.\n\nAvailable `Matcher`:\n- `slogsampling.MatchByLevelAndMessage` (default)\n- `slogsampling.MatchAll`\n- `slogsampling.MatchByLevel`\n- `slogsampling.MatchByMessage`\n- `slogsampling.MatchBySource`\n- `slogsampling.MatchByAttribute`\n- `slogsampling.MatchByContextValue`\n\n### Uniform sampling\n\n```go\ntype UniformSamplingOption struct {\n    // The sample rate for sampling traces in the range [0.0, 1.0].\n    Rate float64\n\n    // Optional hooks\n    OnAccepted func(context.Context, slog.Record)\n    OnDropped  func(context.Context, slog.Record)\n}\n```\n\nExample using `slog-multi`:\n\n```go\nimport (\n    slogmulti \"github.com/samber/slog-multi\"\n    slogsampling \"github.com/samber/slog-sampling\"\n    \"log/slog\"\n)\n\n// Will print 33% of entries.\noption := slogsampling.UniformSamplingOption{\n\t// The sample rate for sampling traces in the range [0.0, 1.0].\n    Rate:       0.33,\n}\n\nlogger := slog.New(\n    slogmulti.\n        Pipe(option.NewMiddleware()).\n        Handler(slog.NewJSONHandler(os.Stdout, nil)),\n)\n```\n\n### Threshold sampling\n\n```go\ntype ThresholdSamplingOption struct {\n    // This will log the first `Threshold` log entries with the same hash,\n    // in a `Tick` interval as-is. Following that, it will allow `Rate` in the range [0.0, 1.0].\n    Tick       time.Duration\n    Threshold  uint64\n    Rate       float64\n\n    // Group similar logs (default: by level and message)\n    Matcher func(ctx context.Context, record *slog.Record) string\n\n    // Optional hooks\n    OnAccepted func(context.Context, slog.Record)\n    OnDropped  func(context.Context, slog.Record)\n}\n```\n\nIf `Rate` is zero, the middleware will drop all log entries after the first `Threshold` records in that interval.\n\nExample using `slog-multi`:\n\n```go\nimport (\n    slogmulti \"github.com/samber/slog-multi\"\n    slogsampling \"github.com/samber/slog-sampling\"\n    \"log/slog\"\n)\n\n// Will print the first 10 entries having the same level+message, then every 10th messages until next interval.\noption := slogsampling.ThresholdSamplingOption{\n    Tick:       5 * time.Second,\n    Threshold:  10,\n    Rate:       0.1,\n}\n\nlogger := slog.New(\n    slogmulti.\n        Pipe(option.NewMiddleware()).\n        Handler(slog.NewJSONHandler(os.Stdout, nil)),\n)\n```\n\n### Absolute sampling\n\n```go\ntype AbsoluteSamplingOption struct {\n    // This will log all entries with the same hash until max is reached,\n    // in a `Tick` interval as-is. Following that, it will reduce log throughput\n    // depending on previous interval.\n    Tick time.Duration\n    Max  uint64\n\n    // Group similar logs (default: by level and message)\n    Matcher Matcher\n\n    // Optional hooks\n    OnAccepted func(context.Context, slog.Record)\n    OnDropped  func(context.Context, slog.Record)\n}\n```\n\nExample using `slog-multi`:\n\n```go\nimport (\n    slogmulti \"github.com/samber/slog-multi\"\n    slogsampling \"github.com/samber/slog-sampling\"\n    \"log/slog\"\n)\n\n// Will print the first 10 entries during the first 5s, then a fraction of messages during the following intervals.\noption := slogsampling.AbsoluteSamplingOption{\n    Tick:       5 * time.Second,\n    Max:        10,\n\n    Matcher: slogsampling.MatchAll(),\n}\n\nlogger := slog.New(\n    slogmulti.\n        Pipe(option.NewMiddleware()).\n        Handler(slog.NewJSONHandler(os.Stdout, nil)),\n)\n```\n\n### Custom sampler\n\n```go\ntype CustomSamplingOption struct {\n    // The sample rate for sampling traces in the range [0.0, 1.0].\n    Sampler func(context.Context, slog.Record) float64\n\n    // Optional hooks\n    OnAccepted func(context.Context, slog.Record)\n    OnDropped  func(context.Context, slog.Record)\n}\n```\n\nExample using `slog-multi`:\n\n```go\nimport (\n    slogmulti \"github.com/samber/slog-multi\"\n    slogsampling \"github.com/samber/slog-sampling\"\n    \"log/slog\"\n)\n\n// Will print 100% of log entries during the night, or 50% of errors, 20% of warnings and 1% of lower levels.\noption := slogsampling.CustomSamplingOption{\n    Sampler: func(ctx context.Context, record slog.Record) float64 {\n        if record.Time.Hour() \u003c 6 || record.Time.Hour() \u003e 22 {\n            return 1\n        }\n\n        switch record.Level {\n        case slog.LevelError:\n            return 0.5\n        case slog.LevelWarn:\n            return 0.2\n        default:\n            return 0.01\n        }\n    },\n}\n\nlogger := slog.New(\n    slogmulti.\n        Pipe(option.NewMiddleware()).\n        Handler(slog.NewJSONHandler(os.Stdout, nil)),\n)\n```\n\n## 🤝 Contributing\n\n- Ping me on twitter [@samuelberthe](https://twitter.com/samuelberthe) (DMs, mentions, whatever :))\n- Fork the [project](https://github.com/samber/slog-sampling)\n- Fix [open issues](https://github.com/samber/slog-sampling/issues) or request new features\n\nDon't hesitate ;)\n\n```bash\n# Install some dev dependencies\nmake tools\n\n# Run tests\nmake test\n# or\nmake watch-test\n```\n\n## 👤 Contributors\n\n![Contributors](https://contrib.rocks/image?repo=samber/slog-sampling)\n\n## 💫 Show your support\n\nGive a ⭐️ if this project helped you!\n\n[![GitHub Sponsors](https://img.shields.io/github/sponsors/samber?style=for-the-badge)](https://github.com/sponsors/samber)\n\n## 📝 License\n\nCopyright © 2023 [Samuel Berthe](https://github.com/samber).\n\nThis project is [MIT](./LICENSE) licensed.\n","funding_links":["https://github.com/sponsors/samber"],"categories":["General"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamber%2Fslog-sampling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsamber%2Fslog-sampling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamber%2Fslog-sampling/lists"}