{"id":13879613,"url":"https://github.com/Shopify/app_profiler","last_synced_at":"2025-07-16T15:32:33.694Z","repository":{"id":37802403,"uuid":"241714553","full_name":"Shopify/app_profiler","owner":"Shopify","description":"Collect performance profiles for your Rails application.","archived":false,"fork":false,"pushed_at":"2024-11-20T20:49:31.000Z","size":422,"stargazers_count":227,"open_issues_count":12,"forks_count":11,"subscribers_count":273,"default_branch":"main","last_synced_at":"2024-11-20T21:39:23.826Z","etag":null,"topics":["performance","profiling","rails","webscale"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Shopify.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-19T20:07:23.000Z","updated_at":"2024-11-11T18:10:44.000Z","dependencies_parsed_at":"2023-10-18T15:53:05.178Z","dependency_job_id":"6bbf363f-a09f-4712-98df-aafd38fdca0c","html_url":"https://github.com/Shopify/app_profiler","commit_stats":{"total_commits":102,"total_committers":26,"mean_commits":3.923076923076923,"dds":0.8529411764705882,"last_synced_commit":"31e445a4e879271a4cc95753b8c2a87b2a46411b"},"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shopify%2Fapp_profiler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shopify%2Fapp_profiler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shopify%2Fapp_profiler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shopify%2Fapp_profiler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Shopify","download_url":"https://codeload.github.com/Shopify/app_profiler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225637064,"owners_count":17500360,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["performance","profiling","rails","webscale"],"created_at":"2024-08-06T08:02:26.801Z","updated_at":"2025-07-16T15:32:33.683Z","avatar_url":"https://github.com/Shopify.png","language":"Ruby","readme":"# AppProfiler\n\nProfiling is critical to providing an understanding of an application's performance.\n\n`AppProfiler` aims to provide a common framework for performance profiling for Rails applications.\n\n## Installation\n\nTo install `app_profiler` you need to include it in your `Gemfile`.\n\n## Profiling middleware\n\n### Configuration\n\nThis gem automatically injects the `AppProfiler::Middleware` middleware into your Rails application.\n\nThis middleware can be disabled by using:\n\n```ruby\nAppProfiler.middleware.disabled = true\n# OR\nRails.application.config.app_profiler.middleware_disabled = true\n```\n\n### Trigger profiling\n\nProfiling can be triggered in one of two ways:\n1. Using the `profile` key in the query string of the URL.\n   - Query string format: `/?[key=value]\u0026...`\n2. Using the `X-Profile` key in the request headers.\n   - `X-Profile` header format: `[\u003ckey\u003e=\u003cvalue\u003e];...`\n\n\nIf `async` query string is provided, then the profile will be uploaded later, in an async manner. One use case for this is when we want to profile a certain % of traffic without incurring costs of inline profile uploads. Async background processing provides three callbacks:\n\n1. profile_enqueue_success: Called when a profile is successfully added to the queue, to be uploaded later.\n2. profile_enqueue_failure: Called when a profile fails to be enqueued due to no space left in the queue, `upload_queue_max_length` exceeded.\n3. after_process_queue: Called when profiles are uploaded from the background thread.\n\nThese callbacks can be configured the following manner:\n\n```ruby\nAppProfiler.profile_enqueue_success = -\u003e() {  StatsD.increment(\"profile_enqueue_success\") }\n# OR\nRails.application.config.app_profiler.profile_enqueue_success = -\u003e() {  StatsD.increment(\"profile_enqueue_success\") }\n```\n\n```ruby\nAppProfiler.profile_enqueue_failure = -\u003e(profile) {  Rails.logger.info(\"Profile #{profile.inspect} could not be enqueued.\") }\n# OR\nRails.application.config.app_profiler.profile_enqueue_failure = -\u003e(profile) {  Rails.logger.info(\"Profile #{profile.inspect} could not be enqueued.\")}\n```\n\n```ruby\nAppProfiler.after_process_queue = -\u003e(num_success, num_failures) { StatsD.gauge(\"async_profile_upload\", tags: { sucessful: num_success, failed: num_failures} )}\n# OR\nRails.application.config.app_profiler.after_process_queue = -\u003e(num_success, num_failures) { StatsD.gauge(\"async_profile_upload\", tags: { sucessful: num_success, failed: num_failures} )}\n```\n\n\n\nYou can configure the profile header using:\n\n```ruby\nAppProfiler.profile_header = \"X-Profile\"\n# OR\nRails.application.config.app_profiler.profile_header = \"X-Profile\"\n```\n\n### Here are some examples:\n\n1. `/?profile=cpu\u0026interval=2000\u0026autoredirect=1\u0026ignore_gc=1`\n2. Set `X-Profile` to `mode=wall;interval=1000;context=test-directory;autoredirect=1`\n\n### Possible keys:\n\n| Key | Value | Notes |\n| --- | ----- | ----- |\n| profile/mode | Supported profiling modes: `cpu`, `wall`, `object` for stackprof. | Use `profile` in (1), and `mode` in (2). Vernier backend only supports `wall` and `retained` at present time. |\n| async | Upload profile in a background thread. When this is set, profile redirect headers are not present in the response.\n| interval | Sampling interval in microseconds. | |\n| ignore_gc | Ignore garbage collection frames | |\n| autoredirect | Redirect request automatically to Speedscope's page after profiling. | |\n| context | Directory within the specified bucket in the selected storage where raw profile data should be written. | Only supported in (2). Defaults to `Rails.env` if not specified. |\n| backend | Profiler to use, either `stackprof` or `vernier`. Defaults to `stackprof`. Note that Vernier requires Ruby 3.2.1+. |\n\n\nNote that the `autoredirect` feature can be turned on for all requests by doing the following:\n\n```ruby\nAppProfiler.autoredirect = true\n# OR\nRails.application.config.app_profiler.autoredirect = true\n```\n\nFile names of profiles are prefixed by default with timezoned date and time, follow by profile mode, an id, and hostname of the machine where it was capture. For example: `20221006-121110-cpu-613fa8d2cdde5820d5312dea1cfa43d9-macbook-pro-work.lan.json`. To customize the prefix you can provide a proc:\n\n```ruby\nAppProfiler.profile_file_prefix = -\u003e { \"custom-prefix\" }\n# OR\nRails.application.config.app_profiler.profile_file_prefix = -\u003e { \"custom-prefix\" }\n\n```\nAs opposed to `profile_file_prefix`, which is used to customize the prefix of a file name, `profile_file_name` is used to provide the actual file name. If both are provided, `profile_file_name` takes precedence.\n\n```ruby\nAppProfiler.profile_file_name = -\u003e(metadata) { \"profile-#{metadata[:id]}-#{metadata[:hostname]}.json\" }\n# OR\nRails.application.config.app_profiler.profile_file_name = -\u003e(metadata) { \"profile-#{metadata[:id]}-#{metadata[:hostname]}.json\" }\n```\n\nTo customize the redirect location you can provide a proc:\n\n```ruby\nAppProfiler.profile_url_formatter = -\u003e(upload) { \"https://host.com/custom/#{upload.name}\" }\n# OR\nRails.application.config.app_profiler.profile_url_formatter = -\u003e(upload) { \"https://host.com/custom/#{upload.name}\" }\n```\n\n### OpenTelemetry Instrumentation\n\nApps using OpenTelemetry can add attributes to traces. This can be useful for correlating profiling data with traces. \n\nTo enable this feature, you can set  `otel_instrumentation_enabled` to `true`. The gem does not require the `opentelemetry` gem to be installed. If the gem is not installed or if it is not loaded, enabling it will raise an `ArgumentError`. By default, the feature is disabled.\n\n```ruby\nAppProfiler.otel_instrumentation_enabled = true\n# OR\nRails.application.config.app_profiler.otel_instrumentation_enabled = true\n```\n\nWhen profiling is triggered, the middleware will generate the profile through StackProf and upload the profiles to your specified storage. For example, the default configuration would upload profiles to file storage.\n\nWhen using a cloud storage provider, you can configure the target bucket name using:\n\n```ruby\nAppProfiler.storage.bucket_name = \"new-bucket-name\"\n# OR\nRails.application.config.app_profiler.storage_bucket_name = \"new-bucket-name\"\n```\n\n### Access control\n\nYou may restrict the storing of profiling results by defining your own Middleware based on `AppProfiler::Middleware` and changing the `after_profile` hook method to return `false` for such cases.\n\nFor example, the following middleware only stores the profiling results if a `disallow_profiling` key was not added to the `request.env` while processing the request.\n\n```ruby\nclass AppProfilerAuthorizedMiddleware \u003c AppProfiler::Middleware\n  def after_profile(env, params)\n    !env.key?(\"disallow_profiling\")\n  end\nend\n```\n\nYou can also restrict running profiling at all by using `before_profile`. For\nexample you may wish to prevent anonymous users triggering the profiler:\n\n```ruby\nclass AppProfilerAuthorizedMiddleware \u003c AppProfiler::Middleware\n  def before_profile(env, params)\n    current_user.present?\n  end\nend\n```\n\nThe custom middleware can then be configured like the following:\n\n```ruby\nRails.application.config.app_profiler.middleware = AppProfilerAuthorizedMiddleware\n```\n\nProfile's custom metadata can be passed on to the uploaded [GCS object](https://cloud.google.com/storage/docs/metadata) using:\n\n```ruby\nAppProfiler.forward_metadata_on_upload = true\n# OR\nRails.application.config.app_profiler.forward_metadata_on_upload = true\n```\n\n## Profile Server\n\nThis option allows for profiles to be passively collected via an HTTP endpoint,\ninspired by [golang's built-in pprof server](https://pkg.go.dev/net/http/pprof).\n\nA minimal Rack app runs a minimal (non-compliant) HTTP server, which exposes an\nendpoint that allows for profiling. For security purposes, the server is bound\nto localhost only. The HTTP server is built using standard library modules only,\nin order to keep dependencies minimal. Because it is an entirely separate server,\nlistening on an entirely separate socket, this should not interfere with any\nexisting application routes, and should even be usable in non-web apps.\n\nThis allows for two main use cases:\n\n- Passive profile collection in production\n  - Periodically profiling production apps to analyze them responding to real\nworkloads\n  - Providing a statistical, long-term view of the \"hot paths\" of the workload\n- Local development profiling\n  - Can be used to get a profile \"on demand\" against a development server\n\n### Configuration\n\nIf using as a railtie, only a single option needs to be set:\n\n```\nconfig.app_profiler.server_enabled = true\n```\n\nAlternatively, the server can be directly started by passing in a logger as follows:\n\n```\nAppProfiler::Server.start(logger)\n```\n\nThe default duration (in seconds), for requests without a duration parameter, can also be\nset via the railtie config.\n\n```\nAppProfiler.server.duration = 30\n# OR\nRails.application.config.app_profiler.server_duration = 30\n```\n\nThe server supports both TCP and Unix sockets for its transport. It is\nrecommended to use TCP for local development, and Unix sockets for production:\n\n```\nAppProfiler.server.transport = AppProfiler::Server::TRANSPORT_UNIX\n# OR\nRails.application.config.app_profiler.server_transport = AppProfiler::Server::TRANSPORT_UNIX\n```\n\nIt is possible, but not recommended, to hardcode the listen port to be used in\nTCP server mode with:\n\n```\nAppProfiler.server.port = 8080\n# OR\nRails.application.config.app_profiler.server_port = 8080\n```\n\nIf this is done in production and it can cause port conflicts with multiple\ninstances of the app, which is another reason why the Unix transport is\npreferred for production.\n\n#### Discovering the port or socket path\n\nIn general, the server should be run without setting the port, in which case\nany free TCP port may be used. To determine what the port is, check the\napplication logs, or resolve it from the special \"Magic file\" which contains\na mapping of pid to port:\n\n```\n$ PID=49825\n$ port_file=$(ls -1 /tmp/app_profiler/profileserver-$PID-port-*)\n$ PORT=$(echo $port_file | sed 's/.*port-\\([[:digit:]]*\\)-.*/\\1/g')\n$ echo $PORT\n60160\n```\n\nThis approach is intended to be \"machine friendly\" so that an external\nprofiling agent can easily detect what port to profile on.\n\nFor the Unix mode, this is even easier as the file simply includes the PID\nin it, and this will be the file handle to use for the Unix socket:\n\n```\n$ PID=41016\n$ SOCK=$(ls -1d /tmp/app_profiler/* | grep $PID.sock)\n$ echo $SOCK\n/tmp/app_profiler/app-profiler-41016.sock\n```\n\n### Collecting a profile\n\nThe API is very simple, and passes supported parameters directly to stackprof.\n\nSee the [possible keys](#possible-keys) for additional documentation on the\nsupported parameters.\n\nFor example, to collect a heap profile for 60 seconds, counting every 10th\nallocation:\n\n```\ncurl \"http://127.0.0.1:$PORT/profile?duration=60\u0026mode=object\u0026interval=10\"\n```\n\n#### Usage with speedscope directly\n\nBy default the server will allow CORS. This can be disabled if it presents a\nproblem, but it should be generally safe given that the server listens for\nrequests on localhost only, which is already a private network address.\n\nThis can be used with a local instance of speedscope to directly initiate\nprofiling from the browser. Assuming speedscope is running locally on port\n`9292`, and the profile server is running on port `57510`, the server address\ncan  be URL encoded, and passed to speedscope via `#profileURL`:\n\n```\nhttp://127.0.0.1:9292/#profileURL=http%3A%2F%2F127.0.0.1%3A57510%2Fprofile%3Fduration%3D1\n```\n\n## Profiling manually\n\n`AppProfiler` can be used more simply to profile blocks of code. Here's how:\n\n```ruby\nreport = AppProfiler.run(mode: :cpu) do\n  # ...\nend\n\nreport.view # opens the profile locally in speedscope by default\n```\n\nProfile files can be found locally in your rails app at `tmp/app_profiler/*.json`.\n\n**Note** In development, if using the `AppProfiler::Viewer::SpeedscopeRemoteViewer` for stackprof\nor if using Vernier, a route for `/app_profiler` will be added to the application.\nIf using Vernier, a route for `/from-url` is also added. These will be handled\nin middlewares, before any application routing logic. There is a small chance\nthat these could shadow existing routes in the application.\n\n## Storage backends\n\nProfiles are stored based on the defined storage class. At the moment, the gem only supports file-based and remote storage via Google Cloud Storage. the default backend is file storage.\n\nYou can use a different backend with the following configuration:\n\n```ruby\nAppProfiler.storage = AppProfiler::Storage::GoogleCloudStorage\n# OR\nRails.application.config.app_profiler.storage = AppProfiler::Storage::GoogleCloudStorage\n```\n\nCredentials for the selected storage can be set using the following configuration (Google Cloud Storage expects the path to a JSON file, or the JSON contents):\n\n```ruby\nAppProfiler.storage.credentials = { \"key\" =\u003e \"value\" }\n# OR\nRails.application.config.app_profiler.storage_credentials = { \"key\" =\u003e \"value\" }\n```\n\nNote that in `development` and `test` modes the file isn't uploaded. Instead, it is viewed via the `Middleware::ViewAction`. If you want to change that, use the `middleware_action` configuration:\n\n```ruby\nRails.application.config.app_profiler.middleware_action = AppProfiler::Middleware::UploadAction\n```\n\n## Profiler backends\n\nIt is possible to configure AppProfiler to use the [`vernier`](https://github.com/jhawthorn/vernier) or [`stackprof`](https://github.com/tmm1/stackprof). To use `vernier`, it must be added separately in the application Gemfile.\n\nThe backend can be selected dynamically at runtime using the `backend` parameter. The default backend to use when this parameter is not specified can be configured with:\n\n```ruby\nAppProfiler.backend = AppProfiler::StackprofBackend # or AppProfiler::VernierBackend\n# OR\nRails.application.config.app_profiler.backend = AppProfiler::StackprofBackend # or AppProfiler::VernierBackend\n```\n\nBy default, the stackprof backend will be used.\n\nIn local development, changing the backend will change whether the profile is viewed in Speedscope or Firefox Profiler.\n\n\n## Profile Sampling\n\nThe `AppProfiler` middleware can be configured to sample a percentage of requests for profiling. This can be useful for profiling a subset of requests in production without profiling every request.\n\nTo enable profile sampling, you can either set `profile_sampler_enabled` to `true` or provide your own custom `Proc`:\n\n```ruby\nAppProfiler.profile_sampler_enabled = true\n# OR\nRails.application.config.app_profiler.profile_sampler_enabled = true\n```\n\n```ruby\nAppProfiler.profile_sampler_enabled = -\u003e { sampler_enabled? }\n# OR\nRails.application.config.app_profiler.profile_sampler_enabled = -\u003e { sampler_enabled? }\n```\n\n\nBoth backends, StackProf and Vernier, can be configured separately.\n\nThese can be overridden like:\n\n```ruby\nAppProfiler.profile_sampler_config = AppProfiler::Sampler::Config.new(\n  sample_rate: 0.5,\n  targets: [\"/foo\"],\n  backends_probability: { stackprof: 0.5, vernier: 0.5 },\n  backend_configs: {\n    stackprof: AppProfiler::Sampler::StackprofConfig.new,\n    vernier: AppProfiler::Sampler::VernierConfig.new,\n)\n\n# OR\n\nRails.application.config.app_profiler = AppProfiler::Sampler::Config.new(\n  sample_rate: 0.5,\n  targets: [\"/foo\"],\n  backends_probability: { stackprof: 0.5, vernier: 0.5 },\n  backend_configs: {\n    stackprof: AppProfiler::Sampler::StackprofConfig.new,\n    vernier: AppProfiler::Sampler::VernierConfig.new,\n)\n\n```\n\nAll the configuration parameters are optional and have default values. The default values are:\n\n```ruby\n\n| Sampler Config                                         | Default         |\n| --------                                               | -------         |\n| sample_rate (0.0 - 1.0)                                | 0.001 (0.1 %)   |\n| targets (request paths, job_names etc )                | ['/']           |\n| exclude_targets (request paths, job_names etc )        | ['/ping']       |\n| stackprof_probability                                  | 1.0             |\n| vernier_probability                                    | 0.0             |\n\n| StackprofConfig                     | Default |\n| --------                            | ------- |\n| wall_interval                       | 5000    |\n| cpu_interval                        | 5000    |\n| object_interval                     | 1000    |\n| wall_mode_probability               | 0.8     |\n| cpu_mode_probability                | 0.1     |\n| object_mode_probability             | 0.1     |\n\n| VernierConfig                       | Default |\n| --------                            | ------- |\n| wall_interval                       | 5000    |\n| retained_interval                   | 5000    |\n| wall_mode_probability               | 1.0     |\n| retained_mode_probability           | 0.0     |\n```\n\nApps do not need have to configure anything if they are happy with the default values.\n\n## Running tests\n\n```\nbin/setup \u0026\u0026 bundle exec rake\n```\n","funding_links":[],"categories":["Ruby"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FShopify%2Fapp_profiler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FShopify%2Fapp_profiler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FShopify%2Fapp_profiler/lists"}