{"id":14110047,"url":"https://github.com/fastly/sidekiq-prometheus","last_synced_at":"2025-04-05T21:11:28.136Z","repository":{"id":34088056,"uuid":"169797583","full_name":"fastly/sidekiq-prometheus","owner":"fastly","description":"Public repository with Prometheus instrumentation for Sidekiq","archived":false,"fork":false,"pushed_at":"2023-11-30T23:35:15.000Z","size":164,"stargazers_count":71,"open_issues_count":6,"forks_count":27,"subscribers_count":33,"default_branch":"main","last_synced_at":"2024-04-27T17:32:20.896Z","etag":null,"topics":["fastly-oss-tier2","prometheus","ruby","sidekiq"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fastly.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-02-08T20:53:44.000Z","updated_at":"2024-04-24T15:31:23.000Z","dependencies_parsed_at":"2024-01-08T08:05:51.904Z","dependency_job_id":null,"html_url":"https://github.com/fastly/sidekiq-prometheus","commit_stats":{"total_commits":98,"total_committers":20,"mean_commits":4.9,"dds":"0.29591836734693877","last_synced_commit":"db6f0f1e7cbfabeb0e2b8b931eee1b0db5bdcac2"},"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fastly%2Fsidekiq-prometheus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fastly%2Fsidekiq-prometheus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fastly%2Fsidekiq-prometheus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fastly%2Fsidekiq-prometheus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fastly","download_url":"https://codeload.github.com/fastly/sidekiq-prometheus/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247399887,"owners_count":20932880,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fastly-oss-tier2","prometheus","ruby","sidekiq"],"created_at":"2024-08-14T10:02:38.114Z","updated_at":"2025-04-05T21:11:28.107Z","avatar_url":"https://github.com/fastly.png","language":"Ruby","readme":"# Sidekiq Prometheus\n\n\n[![Status](https://github.com/fastly/sidekiq-prometheus/actions/workflows/ruby.yml/badge.svg)](https://github.com/fastly/sidekiq-prometheus/actions/workflows/ruby.yml)\n![Gem](https://img.shields.io/gem/v/sidekiq_prometheus.svg?color=blue)\n[![MIT license](http://img.shields.io/badge/license-MIT-brightgreen.svg)](http://opensource.org/licenses/MIT)\n\nPrometheus Instrumentation for Sidekiq. This gem provides:\n\n* Sidekiq server middleware for reporting job metrics\n* Global metrics reporter that uses the Sidekiq API for reporting Sidekiq cluster stats (*requires Sidekiq::Enterprise*)\n* Sidecar Rack server to provide scrape-able endpoint for Prometheus. This allows for metrics to be reported without having to run a separate prometheus exporter process.\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'sidekiq_prometheus'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install sidekiq_prometheus\n\n## Usage\n\nTo run with the defaults add this to your Sidekiq initializer\n\n```\nSidekiqPrometheus.setup\n```\n\nThis will register metrics, start the global reporter (if available), and start the Rack server for scraping. The default port is 9359 but this is easily configurable.\n\nOnce Sidekiq server is running you can see your metrics or scrape them with Prometheus:\n\n```\n$curl http://localhost:9359/metrics\n\n# TYPE sidekiq_job_count counter\n# HELP sidekiq_job_count Count of Sidekiq jobs\n# TYPE sidekiq_job_duration histogram\n# HELP sidekiq_job_duration Sidekiq job processing duration\n\n[etc]\n```\n\n[Full documentation](https://www.rubydoc.info/gems/sidekiq_prometheus)\n\n## Configuration\n\nYou can configure the gem by calling `configure`:\n\n```ruby\nSidekiqPrometheus.configure do |config|\n  config.preset_labels = { service: 'kubernandos_api' }\nend\n```\n\n`configure` will automatically call setup so\n\nIf you are running multiple services that will be reporting Sidekiq metrics you will want to take advantage of the `preset_labels` configuration option. For example:\n\n```ruby\nSidekiqPrometheus.configure do |config|\n  config.preset_labels  = { service: 'image_api' }\n  config.metrics_port = 9090\nend\n```\n\n#### Configuration options\n\n* `preset_labels`: Hash of labels that will be included with every metric when they are registered. `Hash{Symbol (label name) =\u003e String (label value)}`\n* `custom_labels`: Hash of metrics and labels that can be applied to specific metrics. The metric name must be a registered metric. `Hash{Symbol (metric name) =\u003e Array\u003cSymbol\u003e (label names)}`\n* `gc_metrics_enabled`: Boolean that determines whether to record object allocation metrics per job. The default is `true`. Setting this to `false` if you don't need this metric.\n* `global_metrics_enabled`: Boolean that determines whether to report global metrics from the PeriodicMetrics reporter. When `true` this will report on a number of stats from the Sidekiq API for the cluster. This requires Sidekiq::Enterprise as the reporter uses the leader election functionality to ensure that only one worker per cluster is reporting metrics.\n* `init_label_sets`: Hash of metrics and label sets that are initialized by calling `metric.init_label_set` after the metric is registered. See[ prometheus-client docs](https://github.com/prometheus/client_ruby/tree/e144d6225d3c346e9a4dd0a11f41f8acde386dd8#init_label_set) for details. `Hash{Symbol (metric name) =\u003e Array\u003cHash\u003e (label sets)}`.\n* `periodic_metrics_enabled`: Boolean that determines whether to run the periodic metrics reporter. `PeriodicMetrics` runs a separate thread that reports on global metrics (if enabled) as well worker GC stats (if enabled). It reports metrics on the interval defined by `periodic_reporting_interval`. Defaults to `true`.\n* `periodic_reporting_interval`: interval in seconds for reporting periodic metrics. Default: `30`\n* `metrics_server_enabled`: Boolean that determines whether to run the rack server. Defaults to `true`\n* `metrics_server_logger_enabled`: Boolean that determines if the metrics server will log access logs. Defaults to `true`\n* `metrics_host`: Host on which the rack server will listen. Defaults to\n  `localhost`\n* `metrics_port`: Port on which the rack server will listen. Defaults to `9359`\n* `registry`: An instance of `Prometheus::Client::Registry`. If you have a registry with defined metrics you can use this option to pass in your registry.\n\n```ruby\nSidekiqPrometheus.configure do |config|\n  config.preset_labels                 = { service: 'myapp' }\n  config.custom_labels                 = { sidekiq_job_count: [:worker_class, :job_type, :any_other_label] }\n  config.gc_metrics_enabled            = false\n  config.global_metrics_enabled        = true\n  config.init_label_sets               = { sidekiq_job_count: [{worker_class: \"class\", job_type: \"single\", any_other_label: \"value\"}, {worker_class: \"class\", job_type: \"batch\", any_other_label: \"other-value\"}] }\n  config.periodic_metrics_enabled      = true\n  config.periodic_reporting_interval   = 20\n  config.metrics_server_enabled        = true\n  config.metrics_port                  = 8675\nend\n```\n\nCustom labels may be added by defining the `prometheus_labels` method in the worker class,\nprior you need to register the custom labels as of the above example:\n\n```ruby\nclass SomeWorker\n  include Sidekiq::Worker\n\n  def prometheus_labels\n    { some: 'label' }\n  end\nend\n```\n\n## Metrics\n\n### JobMetrics\n\nAll Sidekiq job metrics are reported with these labels:\n\n* `class`: Sidekiq worker class name\n* `queue`: Sidekiq queue name\n\n| Metric | Type | Description |\n|--------|------|-------------|\n| sidekiq_job_count | counter | Count of Sidekiq jobs |\n| sidekiq_job_duration | histogram | Sidekiq job processing duration |\n| sidekiq_job_success | counter | Count of successful Sidekiq jobs |\n| sidekiq_job_allocated_objects | histogram | Count of ruby objects allocated by a Sidekiq job |\n| sidekiq_job_failed | counter | Count of failed Sidekiq jobs |\n| sidekiq_job_over_limit | counter | Count of over limit Sidekiq jobs |\n\n\nNotes:\n\n* when a job fails only `sidekiq_job_count` and `sidekiq_job_failed` will be reported.\n* when a job fails due to Sidekiq::Limiter::OverLimit error, only `sidekiq_job_count` and `sidekiq_job_over_limit` will be reported.\n* `sidekiq_job_allocated_objects` will only be reported if `SidekiqPrometheus.gc_metrics_enabled? == true`\n\n### Periodic GC Metrics\n\nThese require `SidekiqPrometheus.gc_metrics_enabled? == true` and `SidekiqPrometheus.periodic_metrics_enabled? == true`\n\n| Metric | Type | Description |\n|--------|------|-------------|\n| sidekiq_allocated_objects | counter | Count of allocated objects by the worker |\n| sidekiq_heap_free_slots | gauge | Number of free heap slots as reported by GC.stat |\n| sidekiq_heap_live_slots | gauge | Number of live heap slots as reported by GC.stat |\n| sidekiq_major_gc_count | counter | Count of major GC runs |\n| sidekiq_minor_gc_count | counter | Count of minor GC runs |\n| sidekiq_rss | gauge | RSS memory usage for worker process |\n\n### Periodic Global Metrics\n\nThese require `SidekiqPrometheus.global_metrics_enabled? == true` and `SidekiqPrometheus.periodic_metrics_enabled? == true`\n\nPeriodic metric reporting relies on Sidekiq Enterprise's leader election functionality ([Ent Leader Election ](https://github.com/mperham/sidekiq/wiki/Ent-Leader-Election))\nwhich ensures that metrics are only reported once per cluster.\n\n| Metric | Type | Description |\n|--------|------|-------------|\n| sidekiq_workers_size | gauge | Total number of workers processing jobs |\n| sidekiq_dead_size | gauge | Total Dead Size |\n| sidekiq_enqueued | gauge | Total Size of all known queues |\n| sidekiq_queue_latency | summary | Latency (in seconds) of all queues |\n| sidekiq_failed | gauge | Number of job executions which raised an error |\n| sidekiq_processed | gauge | Number of job executions completed (success or failure) |\n| sidekiq_retry_size | gauge | Total Retries Size |\n| sidekiq_scheduled_size | gauge | Total Scheduled Size |\n| sidekiq_redis_connected_clients | gauge | Number of clients connected to Redis instance for Sidekiq |\n| sidekiq_redis_used_memory | gauge | Used memory from Redis.info\n| sidekiq_redis_used_memory_peak | gauge | Used memory peak from Redis.info |\n| sidekiq_redis_keys | gauge | Number of redis keys |\n| sidekiq_redis_expires | gauge | Number of redis keys with expiry set |\n\nThe global metrics are reported with the only the `preset_labels` with the exception of `sidekiq_enqueued` which will add a `queue` label and record a metric per Sidekiq queue.\n\n## Custom Worker Metrics\n\nThere are a few different ways to register custom metrics with SidekiqPrometheus. Each custom metric should be defined as a Hash with the following form:\n\n```ruby\n{\n  name:        :metric_name,\n  type:        :gauge,\n  docstring:   'description',\n  preset_labels: { label_name: 'label_text' },\n}\n```\n\n* `:name` (required) - Unique name of the metric and should be a symbol.\n* `:type` (required) - Prometheus metric type. Supported values are: `:counter`, `:gauge`, `:histogram`, and `:summary`.\n* `:docstring` (required) - Human readable description of the metric.\n* `:preset_labels` (optional) - Hash of labels that will be applied to every instance of this metric.\n\n#### Registering custom metrics:\n\nRegistering a set of custom metrics is done by defining `custom_metrics` in the `configure` block:\n\n```ruby\nSidekiqPrometheus.configure do |config|\n  config.custom_metrics = [\n    { name: :imported_records, type: :counter, docstring: 'Count of successfully imported records' },\n    { name: :failed_records,   type: counter:, docstring: 'Count of failed records' },\n  ]\nend\n```\n\nMetrics can also be registered directly. This must done *after* `SidekiqPrometheus.configure` or `setup` has been run.\n\n```ruby\nSidekiqPrometheus::Metrics.register(name: :logged_in_users, type: :gauge, docstring: 'Logged in users')\n```\n\nThere is also a method to register more than one metric at a time:\n\n```ruby\ncustomer_worker_metrics = [\n  {\n    name: :file_count, type: :counter, docstring: 'Number of active files',\n    name: :file_size,  type: :gauge,   docstring: 'Size of files in bytes',\n  }\n]\n\nSidekiqPrometheus::Metrics.register_metrics(customer_worker_metrics)\n```\n\n#### Using custom metrics:\n\nOnce metrics are registered they can be used in your Sidekiq workers.\n\n```ruby\nclass ImportWorker\n  include Sidekiq::Worker\n\n  LABELS = {}\n\n  def perform(*args)\n    # worker code\n\n    SidekiqPrometheus[:file_count].increment(LABELS, new_file_count)\n    SidekiqPrometheus[:file_size].set(LABELS, total_file_size)\n  end\n\nend\n```\n\nSee the [documentation of the Prometheus::Client library](https://github.com/prometheus/client_ruby) for all of the available options for setting metric values.\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`.\n\n## Contributors\n\nSee: https://github.com/fastly/sidekiq-prometheus/graphs/contributors\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/fastly/sidekiq-prometheus. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.\n\n## Copyright\n\nCopyright 2019 Fastly, Inc.\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).\n\n## Code of Conduct\n\nEveryone interacting in the SidekiqPrometheus project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/fastly/sidekiq-prometheus/blob/main/CODE_OF_CONDUCT.md).\n","funding_links":[],"categories":["Ruby"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffastly%2Fsidekiq-prometheus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffastly%2Fsidekiq-prometheus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffastly%2Fsidekiq-prometheus/lists"}