{"id":28762430,"url":"https://github.com/rubrikinc/fluent-plugin-quota-throttle","last_synced_at":"2025-07-01T03:07:25.520Z","repository":{"id":248485757,"uuid":"823527228","full_name":"rubrikinc/fluent-plugin-quota-throttle","owner":"rubrikinc","description":"Customisable Quota based Throttling for Fluentd","archived":false,"fork":false,"pushed_at":"2024-07-25T09:19:25.000Z","size":98,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-07-26T06:51:55.993Z","etag":null,"topics":["fluentd-plugin","ruby-gem","throttling"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rubrikinc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-03T07:50:16.000Z","updated_at":"2024-07-25T09:19:29.000Z","dependencies_parsed_at":"2024-07-15T09:17:07.727Z","dependency_job_id":"4cbdfef6-9f71-4a22-9ecd-76bf8e5444f1","html_url":"https://github.com/rubrikinc/fluent-plugin-quota-throttle","commit_stats":null,"previous_names":["rubrikinc/fluent-plugin-quota-throttle"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rubrikinc/fluent-plugin-quota-throttle","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rubrikinc%2Ffluent-plugin-quota-throttle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rubrikinc%2Ffluent-plugin-quota-throttle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rubrikinc%2Ffluent-plugin-quota-throttle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rubrikinc%2Ffluent-plugin-quota-throttle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rubrikinc","download_url":"https://codeload.github.com/rubrikinc/fluent-plugin-quota-throttle/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rubrikinc%2Ffluent-plugin-quota-throttle/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260318707,"owners_count":22991122,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluentd-plugin","ruby-gem","throttling"],"created_at":"2025-06-17T08:08:33.416Z","updated_at":"2025-06-17T08:08:34.251Z","avatar_url":"https://github.com/rubrikinc.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fluent-plugin-quota-throttle\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/rubrikinc/fluent-plugin-throttle/blob/master/LICENSE) ![Rake Test](https://github.com/rubrikinc/fluent-plugin-quota-throttle/actions/workflows/ruby.yml/badge.svg) [![Gem Version](https://badge.fury.io/rb/fluent-plugin-quota-throttle.svg)](https://badge.fury.io/rb/fluent-plugin-quota-throttle)\n\nA sentry plugin to throttle logs based on well defined quotas. Logs are grouped by configurable keys. When\na group exceeds a configuration rate, logs are dropped for this group.\n\n## Installation\n\ninstall with `gem` or td-agent provided command as:\n\n\n```bash\n\n# for fluentd\n\n$ gem install fluent-plugin-quota-throttle\n[OR]\n$ fluent-gem install fluent-plugin-quota-throttle\n\n```\n\n## Configuration\n\n#### name\n\nThe name of the quota. This is used for logging and debugging purposes.\n\n#### description\n\nA description of the quota. This is used for documentation of the defined quota\n\n#### group\\_by\n\nUsed to group logs into buckets. Quotas are applied to buckets independently.\n\nA dot indicates a key within a sub-object. As an example, in the following log,\nthe group by key `kubernetes.container_name` resolve to `random`:\n```\n{\"level\": \"error\", \"msg\": \"plugin test\", \"kubernetes\": { \"container_name\": \"random\" } }\n```\n\nMultiple groups can be specified, in which case each unique pair\nof key values are rate limited independently.\n\nIf the group cannot be resolved, an anonymous (`nil`) group is used for rate limiting.\n\n#### match\\_by\n\nUsed to match logs to their respective quotas. If a log does not match any quota, it will be matched to the default quota.\n\n#### bucket\\_size\n\nMaximum number logs allowed per groups over the period of `duration`.\n\nThis translate to a log rate of `bucket_size/duration`.\nWhen a group exceeds bucket limit, logs from this group are dropped/reemitted with new tag.\n\nFor example, the rate is 6000/60s, making for a rate of 100 logs per\nseconds.\n\nNote that this is not expressed as a rate directly because there is a\ndifference between the overall rate and the distribution of logs over a period\ntime. For example, a burst of logs in the middle of a minute bucket might not\nexceed the average rate of the full minute.\n\nConsider `60/60s`, 60 logs over a minute, versus `1/1s`, 1 log per second.\nOver a minute, both will emit a maximum of 60 logs. Limiting to a rate of 60\nlogs per minute. However `60/60s` will readily emit 60 logs within the first\nsecond then nothing for the remaining 59 seconds. While the `1/1s` will only\nemit the first log of every second.\n\n#### duration\n\nThis is the period of of time over which `bucket_size` applies. This should be given in seconds.\n\n#### action\n\nEither `drop` or `reemit`.\n\nWhen a group exceeds its rate limit, logs are either dropped or re-emitted with a new tag `secondary.\u003ctag\u003e`\n\n\n\n#### warning\\_delay\n\nDefault: `60` (seconds).\n\nWhen a group reaches its limit and as long as it is not reset, a warning\nmessage with the current log rate of the group is emitted repeatedly. This is\nthe delay between every repetition.\n\n## Usage\n\n```xml\n\u003cfilter **\u003e\n  @type quota_throttle\n  @path /etc/fluentd/quota_throttle.yaml\n  @warning_delay 30s\n\u003c/filter\u003e\n```\n```yaml\nquotas:\n- name: quota1\n  description: first quota\n  group_by:\n  - group1.a\n  match_by:\n    group1.a: value1\n  bucket_size: 100\n  duration: 60s\n  action: drop\n- name: quota2\n  description: second quota\n  group_by:\n  - group1.a\n  - group1.b\n  match_by:\n    group1.a: value2\n    group1.b: value3\n  bucket_size: 200\n  duration: 2m\n  action: reemit\n- name: quota3\n  description: third quota\n  group_by:\n  - group2\n  - group3\n  match_by:\n      group2: value2\n      group3: value3\n  bucket_size: 300\n  duration: 180s\n  action: drop\ndefault:\n  description: default quota\n  group_by:\n    - group1.a\n  bucket_size: 300\n  duration: 3m\n  action: reemit\n```\n\n## License\n\nApache License, Version 2.0\n\n## Copyright\n\nCopyright © 2018 ([Rubrik Inc.](https://www.rubrik.com))\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frubrikinc%2Ffluent-plugin-quota-throttle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frubrikinc%2Ffluent-plugin-quota-throttle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frubrikinc%2Ffluent-plugin-quota-throttle/lists"}