{"id":13530515,"url":"https://github.com/funbox/sidekiq-influxdb","last_synced_at":"2025-07-11T20:31:16.453Z","repository":{"id":52712398,"uuid":"112096083","full_name":"funbox/sidekiq-influxdb","owner":"funbox","description":"Writes Sidekiq job execution metrics to InfluxDB","archived":false,"fork":false,"pushed_at":"2021-04-26T10:42:55.000Z","size":79,"stargazers_count":18,"open_issues_count":0,"forks_count":1,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-06-06T10:53:08.795Z","etag":null,"topics":["influxdb","metrics","metrics-gathering","middleware","ruby","ruby-gem","ruby-library","sidekiq"],"latest_commit_sha":null,"homepage":"https://github.com/mperham/sidekiq/wiki/Related-Projects#monitoring","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/funbox.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-11-26T15:58:07.000Z","updated_at":"2021-09-10T09:12:07.000Z","dependencies_parsed_at":"2022-08-20T17:10:25.378Z","dependency_job_id":null,"html_url":"https://github.com/funbox/sidekiq-influxdb","commit_stats":null,"previous_names":["vassilevsky/sidekiq-influxdb"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/funbox/sidekiq-influxdb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/funbox%2Fsidekiq-influxdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/funbox%2Fsidekiq-influxdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/funbox%2Fsidekiq-influxdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/funbox%2Fsidekiq-influxdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/funbox","download_url":"https://codeload.github.com/funbox/sidekiq-influxdb/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/funbox%2Fsidekiq-influxdb/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264892154,"owners_count":23679241,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["influxdb","metrics","metrics-gathering","middleware","ruby","ruby-gem","ruby-library","sidekiq"],"created_at":"2024-08-01T07:00:51.213Z","updated_at":"2025-07-11T20:31:11.438Z","avatar_url":"https://github.com/funbox.png","language":"Ruby","funding_links":[],"categories":["Collecting data into InfluxDB"],"sub_categories":["Plugins"],"readme":"# Sidekiq::InfluxDB\n\n[![Gem Version](https://img.shields.io/gem/v/sidekiq-influxdb.svg)](https://rubygems.org/gems/sidekiq-influxdb)\n[![Travis CI](https://img.shields.io/travis/com/funbox/sidekiq-influxdb)](https://travis-ci.com/github/funbox/sidekiq-influxdb)\n[![Coveralls](https://img.shields.io/coveralls/funbox/sidekiq-influxdb.svg)](https://coveralls.io/github/funbox/sidekiq-influxdb)\n\n[Sidekiq](https://github.com/mperham/sidekiq/wiki) server middleware\nthat writes job lifecycle events as points to an [InfluxDB](http://docs.influxdata.com/influxdb/v1.3/) database.\nAlso includes classes that write global Sidekiq metrics and queue metrics.\n\n## Installation\n\nAdd this gem to your application's `Gemfile`:\n\n```bash\nbundle add sidekiq-influxdb\n```\n\n## Usage\n\nAdd included middleware to your application's Sidekiq middleware stack.\nThe following examples assume that you already have an InfluxDB client object\nin the `influxdb` variable.\nThis will create a middleware with all defaults (suitable for most deployments):\n\n```ruby\n# config/initializers/sidekiq.rb\n\nrequire \"sidekiq/middleware/server/influxdb\"\n\nSidekiq.configure_server do |config|\n  config.server_middleware do |chain|\n    chain.add Sidekiq::Middleware::Server::InfluxDB, influxdb_client: influxdb\n  end\nend\n```\n\nYou can customize the middleware by passing more options:\n\n```ruby\n# config/initializers/sidekiq.rb\n\nrequire \"sidekiq/middleware/server/influxdb\"\n\nSidekiq.configure_server do |config|\n  config.server_middleware do |chain|\n    chain.add Sidekiq::Middleware::Server::InfluxDB,\n                influxdb_client: influxdb,\n                series_name: 'sidekiq_jobs',  # This is the default one.\n                retention_policy: 'rp_name',  # In case you want to write metrics to a non-default RP.\n                start_events: true,           # Whether or not you want to know when jobs started. See `event` tag description below.\n                tags: {application: 'MyApp'}, # Anything you need on top. **Make sure that tag values have low cardinality!**\n                except: [UnimportantJob]      # These job classes will be executed without sending any metrics.\n  end\nend\n```\n\nThis library assumes that you already have an InfluxDB client object set up the way you like.\nIt does not try to create one for you.\nIf that is not the case, you can learn how to create a client\nin [InfluxDB client documentation](https://github.com/influxdata/influxdb-ruby#creating-a-client).\n\n**Warning:** This middleware is going to write _a lot_ of metrics.\nSet up your InfluxDB client accordingly:\n* either set `async: true` in the client's options to use its built-in batching feature,\n* or install Telegraf, set up aggregation inside it, and set up InfluxDB client to send metrics to it,\n* or both.\n\nWhen you deploy this code, you will have the following series in your InfluxDB database:\n\n```\n\u003e select * from sidekiq_jobs\nname: sidekiq_jobs\ntime                application  class  creation_time      error         event  jid                      queue   total              waited              worked\n----                -----------  -----  -------------      -----         -----  ---                      -----   -----              ------              ------\n1511707465061000000 MyApp        FooJob 1511707459.0186539               start  51cc82fe75fbeba37b1ff18f default                    6.042410135269165\n1511707465061000000 MyApp        FooJob 1511707459.0186539               finish 51cc82fe75fbeba37b1ff18f default 8.046684265136719  6.042410135269165   2.0042741298675537\n1511707467068000000 MyApp        BarJob 1511707461.019835                start  3891f241ab84d3aba728822e default                    6.049134016036987\n1511707467068000000 MyApp        BarJob 1511707461.019835  NoMethodError error  3891f241ab84d3aba728822e default 8.056788206100464  6.049134016036987   2.0076541900634766\n```\n\nTags (repetitive indexed data; for filtering and grouping by):\n\n* `time` — standard InfluxDB timestamp. Precision of the supplied client is respected.\n* `queue` — queue name.\n* `class` — job class name. Classes from `except:` keyword argument are skipped (no data is sent to InfluxDB).\n* `event` — what happened to the job at the specified `time`: `start`, `finish`, or `error`. If you initialize the middleware with `start_events: false`, there will be no `start` events.\n* `error` — if `event=error`, this tag contains the exception class name.\n* Your own tags from the initializer.\n\nValues (unique non-indexed data; for aggregation):\n\n* `jid` — unique job ID.\n* `creation_time` — job creation time.\n\nValues calculated by this gem (in seconds):\n\n* `waited` — how long the job waited in the `queue` until Sidekiq got around to starting it.\n* `worked` — how long it took to perform the job from start to finish or to an exception.\n* `total` — how much time passed from job creation to finish. How long it took to do the job, in total.\n\nThis schema allows querying various job metrics effectively.\n\nFor example, how many reports have been generated in the last day:\n\n```sql\nSELECT COUNT(jid) FROM sidekiq_jobs WHERE class = 'ReportGeneration' AND time \u003e now() - 1d\n```\n\nHow many different jobs were executed with errors in the last day:\n\n```sql\nSELECT COUNT(jid) FROM sidekiq_jobs WHERE event = 'error' AND time \u003e now() - 1d GROUP BY class\n```\n\nEt cetera.\n\n### Stats and Queues metrics\n\nTo collect metrics for task stats and queues, you need to run the following code periodically.\nFor example, you can use [Clockwork](https://rubygems.org/gems/clockwork) for that.\nYou can add settings like this to `clock.rb`:\n\n```ruby\nrequire \"sidekiq/metrics/stats\"\nrequire \"sidekiq/metrics/queues\"\n\ninflux = InfluxDB::Client.new(options)\n\nsidekiq_global_metrics = Sidekiq::Metrics::Stats.new(influxdb_client: influx)\nsidekiq_queues_metrics = Sidekiq::Metrics::Queues.new(influxdb_client: influx)\n\nevery(1.minute, 'sidekiq_metrics') do\n  sidekiq_global_metrics.publish\n  sidekiq_queues_metrics.publish\nend\n```\n\nFor stats metrics:\n\n```ruby\nrequire \"sidekiq/metrics/stats\"\n\nSidekiq::Metrics::Stats.new(\n  influxdb_client: InfluxDB::Client.new(options), # REQUIRED\n  series_name: 'sidekiq_stats',                   # optional, default shown\n  retention_policy: nil,                          # optional, default nil\n  tags: {},                                       # optional, default {}\n).publish\n```\n\nFor queues metrics:\n\n```ruby\nrequire \"sidekiq/metrics/queues\"\n\nSidekiq::Metrics::Queues.new(\n  influxdb_client: InfluxDB::Client.new(options), # REQUIRED\n  series_name: 'sidekiq_queues',                  # optional, default shown\n  retention_policy: nil,                          # optional, default nil\n  tags: {},                                       # optional, default {}\n).publish\n```\n\nWhen you run the code, you will have the following series in your InfluxDB database:\n\n```\n\u003e select * from sidekiq_stats\nname: sidekiq_stats\ntime                size     stat\n----                ----     ----\n1582502419000000000 9999     dead\n1582502419000000000 0        workers\n1582502419000000000 0        enqueued\n1582502419000000000 23020182 processed\n```\n\n```\n\u003e select * from sidekiq_queues\nname: sidekiq_queues\ntime                queue             size\n----                -----             ----\n1582502418000000000 default           0\n1582502418000000000 queue_name_1      0\n```\n\n## Visualization\n\n### Grafana\n\nYou can import a ready-made dashboard from [grafana_dashboard.json](grafana_dashboard.json).\n\n## Development\n\nSee [Contributing Guidelines](CONTRIBUTING.md).\n\n[![Sponsored by FunBox](https://funbox.ru/badges/sponsored_by_funbox_centered.svg)](https://funbox.ru)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffunbox%2Fsidekiq-influxdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffunbox%2Fsidekiq-influxdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffunbox%2Fsidekiq-influxdb/lists"}