{"id":18103163,"url":"https://github.com/rubrikinc/fluent-plugin-throttle","last_synced_at":"2025-04-06T20:13:32.471Z","repository":{"id":37444882,"uuid":"125921759","full_name":"rubrikinc/fluent-plugin-throttle","owner":"rubrikinc","description":"Rate limiting for fluentd","archived":false,"fork":false,"pushed_at":"2023-07-16T17:52:10.000Z","size":32,"stargazers_count":41,"open_issues_count":12,"forks_count":24,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-04-25T13:21:05.809Z","etag":null,"topics":["fluentd","fluentd-plugin","ruby","throttling"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rubrikinc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-19T21:19:46.000Z","updated_at":"2024-06-21T05:47:29.411Z","dependencies_parsed_at":"2024-06-21T05:47:13.599Z","dependency_job_id":"cfe96265-820c-4a96-a817-decfb5ea2daa","html_url":"https://github.com/rubrikinc/fluent-plugin-throttle","commit_stats":{"total_commits":21,"total_committers":5,"mean_commits":4.2,"dds":"0.47619047619047616","last_synced_commit":"f54f1e3f7e8068d9bf33269a1827bf33d2cd01a3"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rubrikinc%2Ffluent-plugin-throttle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rubrikinc%2Ffluent-plugin-throttle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rubrikinc%2Ffluent-plugin-throttle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rubrikinc%2Ffluent-plugin-throttle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rubrikinc","download_url":"https://codeload.github.com/rubrikinc/fluent-plugin-throttle/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247543595,"owners_count":20955865,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluentd","fluentd-plugin","ruby","throttling"],"created_at":"2024-10-31T22:10:43.528Z","updated_at":"2025-04-06T20:13:32.398Z","avatar_url":"https://github.com/rubrikinc.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fluent-plugin-throttle\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/rubrikinc/fluent-plugin-throttle/blob/master/LICENSE)\n\nA sentry plugin to throttle logs. Logs are grouped by a configurable key. When\na group exceeds a configuration rate, logs are dropped for this group.\n\n## Installation\n\ninstall with `gem` or td-agent provided command as:\n\n```bash\n# for fluentd\n$ gem install fluent-plugin-throttle\n```\n\n## Usage\n\n```xml\n\u003cfilter **\u003e\n  @type throttle\n  group_key kubernetes.container_name\n  group_bucket_period_s   60\n  group_bucket_limit    6000\n  group_reset_rate_s     100\n\u003c/filter\u003e\n```\n\n## Configuration\n\n#### group\\_key\n\nDefault: `kubernetes.container_name`.\n\nUsed to group logs. Groups are rate limited independently.\n\nA dot indicates a key within a sub-object. As an example, in the following log,\nthe group key resolve to \"random\":\n```\n{\"level\": \"error\", \"msg\": \"plugin test\", \"kubernetes\": { \"container_name\": \"random\" } }\n```\n\nMultiple groups can be specified using the fluentd config array syntax, \ne.g. `kubernetes.container_name,kubernetes.pod_name`, in which case each unique pair\nof key values are rate limited independently.\n\nIf the group cannot be resolved, an anonymous (`nil`) group is used for rate limiting.\n\n#### group\\_bucket\\_period\\_s\n\nDefault: `60` (60 second).\n\nThis is the period of of time over which `group_bucket_limit` applies.\n\n#### group\\_bucket\\_limit\n\nDefault: `6000` (logs per `group_bucket_period_s`).\n\nMaximum number logs allowed per groups over the period of `group_bucket_period_s`.\n\nThis translate to a log rate of `group_bucket_limit/group_bucket_period_s`.\nWhen a group exceeds this rate, logs from this group are dropped.\n\nFor example, the default is 6000/60s, making for a rate of 100 logs per\nseconds.\n\nNote that this is not expressed as a rate directly because there is a\ndifference between the overall rate and the distribution of logs over a period\ntime. For example, a burst of logs in the middle of a minute bucket might not\nexceed the average rate of the full minute.\n\nConsider `60/60s`, 60 logs over a minute, versus `1/1s`, 1 log per second.\nOver a minute, both will emit a maximum of 60 logs. Limiting to a rate of 60\nlogs per minute. However `60/60s` will readily emit 60 logs within the first\nsecond then nothing for the remaining 59 seconds. While the `1/1s` will only\nemit the first log of every second.\n\n#### group\\_drop\\_logs\n\nDefault: `true`.\n\nWhen a group reaches its limit, logs will be dropped from further processing\nif this value is true (set by default). To prevent the logs from being dropped\nand only receive a warning message when rate limiting would have occurred, set\nthis value for false. This can be useful to fine-tune your group bucket limits\nbefore dropping any logs.\n\n#### group\\_reset\\_rate\\_s\n\nDefault: `group_bucket_limit/group_bucket_period_s` (logs per `group_bucket_period_s`).\nMaximum: `group_bucket_limit/group_bucket_period_s`.\n\nAfter a group has exceeded its bucket limit, logs are dropped until the rate\nper second falls below or equal to `group_reset_rate_s`.\n\nThe default value is `group_bucket_limits/group_bucket_period_s`. For example\nfor 3600 logs per hour, the reset will defaults to `3600/3600s = 1/s`, one log\nper second.\n\nTaking the example `3600 log/hour` with the default reset rate of `1 log/s`\nfurther:\n\n - Let's say we have a period of 10 hours.\n - During the first hour, 2 logs/s are produced. After 30 minutes, the hourly\n   bucket has reached its limit, and logs are dropped. At this point the rate\n   is still 2 logs/s for the remaining 30 minutes.\n - Because the last hour finished on 2 logs/s, which is higher that the\n   `1 log/s` reset, all logs are still dropped when starting the second hour. The\n   bucket limit is left untouched since nothing is being emitted.\n - Now, at 2 hours and 30 minutes, the log rate halves to `1 log/s`, which is\n   equal to the reset rate. Logs are emitted again, counting toward the bucket\n   limit as normal. Allowing up to 3600 logs for the last 30 minutes of the second\n   hour.\n\nBecause this could allow for some instability if the log rate hovers around the\n`group_bucket_limit/group_bucket_period_s` rate, it is possible to set a\ndifferent reset rate.\n\nNote that a value of `0` effectively means the plugin will drops logs forever\nafter a single breach of the limit until the next restart of fluentd.\n\nA value of `-1` disables the feature.\n\n#### group\\_warning\\_delay\\_s\n\nDefault: `10` (seconds).\n\nWhen a group reaches its limit and as long as it is not reset, a warning\nmessage with the current log rate of the group is emitted repeatedly. This is\nthe delay between every repetition.\n\n## License\n\nApache License, Version 2.0\n\n## Copyright\n\nCopyright © 2018 ([Rubrik Inc.](https://www.rubrik.com))\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frubrikinc%2Ffluent-plugin-throttle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frubrikinc%2Ffluent-plugin-throttle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frubrikinc%2Ffluent-plugin-throttle/lists"}