{"id":23813793,"url":"https://github.com/blacklane/kiev","last_synced_at":"2025-04-04T21:07:34.728Z","repository":{"id":26504561,"uuid":"108870445","full_name":"blacklane/kiev","owner":"blacklane","description":"A set of tools to do distributed logging for Ruby web applications","archived":false,"fork":false,"pushed_at":"2024-12-02T14:11:26.000Z","size":164,"stargazers_count":44,"open_issues_count":7,"forks_count":4,"subscribers_count":48,"default_branch":"master","last_synced_at":"2025-03-28T20:07:01.662Z","etag":null,"topics":["distributed-tracing","elk-stack","logging","ruby","sre"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blacklane.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-30T15:30:21.000Z","updated_at":"2024-02-09T10:27:37.000Z","dependencies_parsed_at":"2023-01-14T04:48:39.650Z","dependency_job_id":"384d1565-d73d-4198-8ff5-87e1113209e9","html_url":"https://github.com/blacklane/kiev","commit_stats":{"total_commits":34,"total_committers":14,"mean_commits":"2.4285714285714284","dds":0.7647058823529411,"last_synced_commit":"edda94115bdacd1cfc06df1720fce0575d534ff7"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blacklane%2Fkiev","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blacklane%2Fkiev/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blacklane%2Fkiev/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blacklane%2Fkiev/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blacklane","download_url":"https://codeload.github.com/blacklane/kiev/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247249524,"owners_count":20908212,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-tracing","elk-stack","logging","ruby","sre"],"created_at":"2025-01-02T03:36:14.787Z","updated_at":"2025-04-04T21:07:34.709Z","avatar_url":"https://github.com/blacklane.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kiev [![Build Status](https://github.com/blacklane/kiev/workflows/Main%20CI/badge.svg?branch=master)](https://github.com/blacklane/kiev/actions?query=workflow%3A%22Main+CI%22) [![Gem Version](https://badge.fury.io/rb/kiev.svg)](https://badge.fury.io/rb/kiev)\n\nKiev is a comprehensive logging library aimed at covering a wide range of frameworks and tools from the Ruby ecosystem:\n\n- Rails\n- Sinatra\n- Rack and other Rack-based frameworks\n- Sidekiq\n- Que\n- Shoryuken\n- Her and other Faraday-based libraries\n- HTTParty\n\nThe main goal of Kiev is consistent logging across distributed systems, like **tracking HTTP requests across various Ruby micro-services**. Kiev will generate and propagate request IDs and make it easy for you to identify service calls and branching requests, **including background jobs triggered by these requests**.\n\nAside from web requests and background jobs, which are tracked out of the box, Kiev makes it easy to append additional information or introduce **custom events**.\n\nKiev produces structured logs in the **JSON format**, which are ready to be ingested by ElasticSearch or other similar JSON-driven data stores. It eliminates the need for Logstash in a typical ELK stack.\n\nIn **development mode**, Kiev can print human-readable logs - pretty much like the default Rails logger, but including all the additional information that you've provided via Kiev events.\n\n## Install\n\nAdd the gem to your `Gemfile`:\n\n```ruby\ngem \"kiev\"\n```\n\nDon't forget to `bundle install`.\n\n## Configure\n\n### Rails\n\nPlace your configuration under `config/initializers/kiev.rb`:\n\n```ruby\nrequire \"kiev\"\n\nKiev.configure do |config|\n  config.app = :my_app\n  config.development_mode = Rails.env.development?\n  config.log_path = Rails.root.join(\"log\", \"structured.log\") unless Rails.env.development? || $stdout.isatty\nend\n```\n\nThe middleware stack is included automatically via a *Railtie*.\n\n### Sinatra\n\nSomewhere in your code, ideally before the server configuration, add the following lines:\n\n```ruby\nrequire \"kiev\"\n\nKiev.configure do |config|\n  config.app = :my_app\n  config.log_path = File.join(\"log\", \"structured.log\")\nend\n```\n\nWithin your `Sinatra::Base` implementation, include the `Kiev::Rack` module, in order to register the middleware stack:\n\n```ruby\nrequire \"kiev\"\nrequire \"sinatra/base\"\n\nclass MyController \u003c Sinatra::Base\n  include Kiev::Rack\n\n  use SomeOtherMiddleware\n\n  get \"/hello\" do\n    \"world\"\n  end\nend\n```\n\n### Rack\n\nSomewhere in your code, ideally before the server configuration, add the following lines:\n\n```ruby\nrequire \"kiev\"\n\nKiev.configure do |config|\n  config.app = :my_app\n  config.log_path = File.join(\"log\", \"structured.log\")\nend\n```\n\nWithin your `Rack::Builder` implementation, include the `Kiev::Rack` module, in order to register the middleware stack:\n\n```ruby\nrequire \"kiev\"\nrequire \"rack\"\n\napp = Rack::Builder.new do\n  include Kiev::Rack\n\n  use SomeOtherMiddleware\n\n  run labmda { |env| [ 200, {}, [ \"hello world\" ] ] }\nend\n\nrun(app)\n```\n\n### Hanami\n\nPlace your configuration under `config/initializers/kiev.rb`:\n\n```ruby\nrequire \"kiev\"\n\nKiev.configure do |config|\n  config.app = :my_app\n  config.development_mode = Hanami.env?(:development)\n  config.log_path = File.join(\"log\", \"structured.log\")\nend\n```\n\nWithin your `MyApp::Application` file, include the `Kiev::Hanami` module, in order to register the middleware stack.\nThe `include` should be added before `configure` block.\n\n```ruby\nmodule MyApp\n  class Application \u003c Hanami::Application\n    include Kiev::Hanami\n\n    configure do\n      # ...\n    end\n  end\nend\n```\n\n### Sidekiq\n\nAdd the following lines to your initializer code:\n\n```ruby\nKiev::Sidekiq.enable\n```\n\n### Shoryuken\n\nAdd the following lines to your initializer code:\n\n```ruby\nKiev::Shoryuken.enable\n```\n\nThe name of the worker class is not logged by default. Configure [`persistent_log_fields` option](#persistent_log_fields) to include `\"shoryuken_class\"` if you want this.\n\n### AWS SNS\n\nTo enhance messages published to SNS topics you can use the ContextInjector:\n\n```ruby\nsns_message = { topic_arn: \"...\",  message: \"{...}\" }\nKiev::Kafka.inject_context(sns_message[:message_attributes])\n\n```\n\nAfter this operation the message attributes will also include required context for the Kiev logger.\n\n### Kafka\n\nTo enhance messages published to Kafka topics you can use the ContextInjector:\n\n```ruby\nKiev::Kafka.inject_context(headers)\n```\n\nAfter this operation the headers variable will also include required context for the Kiev logger.\n\nIf you have a consumed `Kafka::FetchedMessage` you can extract logger context with:\n\n```ruby\nKiev::Kafka.extract_context(message)\n```\n\nThis will work regardless if headers are in HTTP format, e.g. `X-Tracking-Id` or plain field names: `tracking_id`. Plus the `message_key` field will contain the key of processed message. In case you want to log some more fields configure `persistent_log_fields` and `jobs_propagated_fields`.\n\n### Que\n\nAdd the following lines to your initializer code:\n\n```ruby\nrequire \"kiev/que/job\"\n\nclass MyJob \u003c Kiev::Que::Job\n  ...\nend\n```\n\n### Her\n\nAdd the following lines to your initializer code:\n\n```ruby\nHer::API.setup(url: \"https://api.example.com\") do |c|\n  c.use Kiev::HerExt::ClientRequestId\n  # other middleware\nend\n```\n\n## Loading only the required parts\n\nYou can load only parts of the gem, if you don't want to use all features:\n\n```ruby\nrequire \"kiev/her_ext/client_request_id\"\n```\n\n## Logging\n\n### Requests\n\nFor web requests the Kiev middleware will log the following information by default:\n\n```json\n{\n  \"application\":\"my_app\",\n  \"event\":\"request_finished\",\n  \"level\":\"INFO\",\n  \"timestamp\":\"2017-01-27T16:11:44.123Z\",\n  \"host\":\"localhost\",\n  \"verb\":\"GET\",\n  \"path\":\"/\",\n  \"params\":\"{\\\"hello\\\":\\\"world\\\",\\\"password\\\":\\\"[FILTERED]\\\"}\",\n  \"ip\":\"127.0.0.1\",\n  \"request_id\":\"UUID\",\n  \"request_depth\":0,\n  \"route\":\"RootController#index\",\n  \"user_agent\":\"curl/7.50.1\",\n  \"status\":200,\n  \"request_duration\":62.3773,\n  \"body\":\"See #log_response_body_condition\",\n  \"error_message\": \"...\",\n  \"error_class\": \"...\",\n  \"error_backtrace\": \"...\",\n  \"tree_path\": \"ACE\",\n  \"tree_leaf\": true\n}\n```\n\n* `params` attribute will store both query parameters and request body fields (as long as they are parseable). Sensitive fields will be filtered out - see the `#filtered_params` option.\n\n* `request_id` is the correlation ID and will be the same across all requests within a chain of requests. It's represented as a UUID (version 4). (currently deprecated in favor of a new name: `tracking_id`)\n\n* `tracking_id` is the correlation ID and will be the same across all requests within a chain of requests. It's represented as a UUID (version 4). If not provided the value is seeded from deprecated `request_id`.\n\n* `request_depth` represents the position of the current request within a chain of requests. It starts with 0.\n\n* `route` attribute will be set to either the Rails route (`RootController#index`) or Sinatra route (`/`) or the path, depending on the context.\n\n* `request_duration` is measured in miliseconds.\n\n* `body` attribute coresponds to the response body and will be logged depending on the `#log_response_body_condition` option.\n\n* `tree_path` attribute can be used to follow the branching of requests within a chain of requests. It's a lexicographically sortable string.\n\n* `tree_leaf` points out that this request is a leaf in the request chain tree structure.\n\n### Background jobs\n\nFor background jobs, Kiev will log the following information by default:\n\n```json\n{\n  \"application\":\"my_app\",\n  \"event\":\"job_finished\",\n  \"level\":\"INFO\",\n  \"timestamp\":\"2017-01-27T16:11:44.123Z\",\n  \"job_name\":\"name\",\n  \"params\": \"...\",\n  \"jid\":123,\n  \"request_id\":\"UUID\",\n  \"request_depth\":0,\n  \"request_duration\":0.000623773,\n  \"error_message\": \"...\",\n  \"error_class\": \"...\",\n  \"error_backtrace\": \"...\",\n  \"tree_path\": \"BDF\",\n  \"tree_leaf\": true\n}\n```\n\n### Appending data to the request log entry\n\nYou can also append **arbitrary data** to the request log by calling:\n\n```ruby\n# Append structured data (will be merged)\nKiev.payload(first_name: \"john\", last_name: \"smith\")\n\n# Same thing\nKiev[:first_name] = \"john\"\nKiev[:last_name] = \"smith\"\n```\n\n### Other events\n\nKiev allows you to log custom events as well.\n\nThe recommended way to do this is by using the `#event` method:\n\n```ruby\n# Log event without any data\nKiev.event(:my_event)\n\n# Log structured data (will be merged)\nKiev.event(:my_event, { some_array: [1, 2, 3] })\n\n# Log other data types (will be available under the `message` key)\nKiev.event(:my_event, \"hello world\")\n\n# Log with given severity [debug, info, warn, error, fatal]\nKiev.info(:my_event)\nKiev.info(:my_event, { some_array: [1, 2, 3] })\nKiev.info(:my_event, \"hello world\")\n```\n\nHowever, `Kiev.logger` implements the Ruby `Logger` class, so all the other methods are available as well:\n\n```ruby\nKiev.logger.info(\"hello world\")\nKiev.logger.debug({ first_name: \"john\", last_name: \"smith\" })\n```\n\nNote that, even when logging custom events, Kiev **will try to append request information** to the entries: the HTTP `verb` and `path` for web request or `job_name` and `jid` for background jobs. The payload, however, will be logged only for the `request_finished` or `job_finished` events. If you want to add a payload to a custom event, use the second argument of the `event` method.\n\n## Advanced configuration\n\n### development_mode\n\nKiev offers human-readable logging for development purposes. You can enable it via the `development_mode` option:\n\n```ruby\nKiev.configure do |config|\n  config.development_mode = Rails.env.development?\nend\n```\n\n### filtered_params\n\nBy default, Kiev filters out the values for the following parameters:\n\n- client_secret\n- token\n- password,\n- password_confirmation\n- old_password\n- credit_card_number\n- credit_card_cvv\n\nYou can override this behaviour via the `filtered_params` option:\n\n```ruby\nKiev.configure do |config|\n  config.filtered_params = %w(email first_name last_name)\nend\n```\n\n### ignored_params\n\nBy default, Kiev ignores the following parameters:\n\n- controller\n- action\n- format\n- authenticity_token\n- utf8\n\nYou can override this behaviour via the `ignored_params` option:\n\n```ruby\nKiev.configure do |config|\n  config.ignored_params = %w(some_field some_other_field)\nend\n```\n\n### log_request_condition\n\nBy default, Kiev doesn't log requests to `/ping`, `/health`, `/live` or `/ready` or requests to assets.\n\nYou can override this behaviour via the `log_request_condition` option, which should be a `proc` returning a `boolean`:\n\n```ruby\nKiev.configure do |config|\n  config.log_request_condition = proc do |request, response|\n    !%r{(^(/ping|/health))|(\\.(js|css|png|jpg|gif)$)}.match(request.path)\n  end\nend\n```\n\n### log_request_error_condition\n\nKiev logs Ruby exceptions. By default, it won't log the exceptions produced by 404s.\n\nYou can override this behaviour via the `log_request_error_condition` option, which should be a `proc` returning a `boolean`:\n\n```ruby\nKiev.configure do |config|\n  config.log_request_error_condition = proc do |request, response|\n    response.status != 404\n  end\nend\n```\n\n### log_response_body_condition\n\nKiev can log the response body. By default, it will only log the response body when the status code is in the 4xx range and the content type is JSON or XML.\n\nYou can override this behaviour via the `log_response_body_condition` option, which should be a `proc` returning a `boolean`:\n\n```ruby\nKiev.configure do |config|\n  config.log_response_body_condition = proc do |request, response|\n    response.status \u003e= 400 \u0026\u0026 response.status \u003c 500 \u0026\u0026 response.content_type =~ /(json|xml)/\n  end\nend\n```\n\n### persistent_log_fields\n\nIf you need to log some data for every event in the session (e.g. the user ID), you can do this via the `persistent_log_fields` option.\n\n```ruby\nKiev.configure do |config|\n  config.persistent_log_fields = [:user_id]\nend\n\n# Somewhere in application\nbefore do\n  Kiev[:user_id] = current_user.id\nend\n\nget \"/\" do\n  \"hello world\"\nend\n```\n\n### log_level\nYou can specify log level.\n\n```ruby\nKiev.configure do |config|\n  # One of: 0, 1, 2, 3, 4 (DEBUG, INFO, WARN, ERROR, FATAL)\n  config.log_level = 0\nend\n```\n\n### disable_filter_for_log_levels\nYou can specify for which log levels personal identifying information filter will NOT be applied.\n\n```ruby\nKiev.configure do |config|\n  # [DEBUG, INFO, WARN, ERROR, FATAL]\n  config.disable_filter_for_log_levels = [0, 1, 2, 3, 4]\nend\n```\n\n**By default enabled for all suppported log levels.**\n\n## nginx\n\nIf you want to log 499 and 50x errors in nginx, which will not be captured by Ruby application, consider adding this to your nginx configuration:\n\n```\nlog_format kiev '{\"application\":\"app_name\", \"event\":\"request_finished\",'\n  '\"timestamp\":\"$time_iso8601\", \"request_id\":\"$http_x_request_id\",'\n  '\"user_agent\":\"$http_user_agent\", \"status\":$status,'\n  '\"request_duration_seconds\":$request_time, \"host\":\"$host\",'\n  '\"verb\":\"$request_method\", \"path\":\"$request_uri\", \"tree_path\": \"$http_x_tree_path\"}';\n\nlog_format simple_log '$remote_addr - $remote_user [$time_local] '\n                       '\"$request\" $status $bytes_sent '\n                       '\"$http_referer\" \"$http_user_agent\"';\n\nmap $status $not_loggable {\n  ~(499) 0;\n  default 1;\n}\n\nmap $status $loggable {\n  ~(499) 1;\n  default 0;\n}\n\nserver {\n  access_log /var/log/nginx/access.kiev.log kiev if=$loggable;\n  access_log /var/log/nginx/access.log simple_log if=$not_loggable;\n\n  location = /50x.html {\n    access_log /var/log/nginx/access.kiev.log kiev;\n  }\n}\n```\n\nIf you'd like to measure nginx queue latency, add the following to your nginx configuration:\n\n```\nserver {\n  ...\n  proxy_set_header X-Request-Start \"${msec}\";\n  ...\n}\n```\n\nOther libs/technologies using `X-Request-Start` are [rack-timeout](https://github.com/heroku/rack-timeout) and [NewRelic](https://docs.newrelic.com/docs/apm/applications-menu/features/request-queue-server-configuration-examples). There's no [support for ELB](https://forums.aws.amazon.com/message.jspa?messageID=396283) :(\n\n## Logstash, Logrotate, Filebeat\n\nKiev does not provide facilities to log directly to ElasticSearch. This is done for simplicity. Instead we recommend using [Filebeat](https://www.elastic.co/products/beats/filebeat) to deliver logs to ElasticSearch.\n\nWhen storing logs on disk, we recommend using Logrotate in truncate mode.\n\nYou can use [jq](https://stedolan.github.io/jq/) to traverse JSON log files, when you're not running Kiev in *development mode*.\n\n## Suffixing `tree_path`\n\nKiev is built upon the assumption that one request is handled once. This isn't always true.\n\nA practical example: multiple Amazon SQS queues subscribed to one Amazon SNS topic. You send one message to SNS and queues receive identical copies that are impossible to distinguish in the trace without any help from the outside.\n\nYou can solve this by adding a fixed unique suffix inside each queue processor. Preferably a single character with an even number in the alphabet (B, D, F and so on), to maintain the notion of \"asynchronous processing\" used throughout Kiev.\n\nFor a combination of SNS and [Shoryuken](https://github.com/phstc/shoryuken) (SQS consumer). Here's how you can use it:\n\n* Enable \"Raw Message Delivery\" in your SQS-to-SNS subscriptions\n* On sender, write `Kiev::SubrequestHelper.payload` into the message attributes\n* On each receiver, use `Kiev::Shoryuken::suffix_tree_path` with a unique tag, like this:\n\n  ```ruby\n  # Suffix a single worker class:\n  class MyWorker\n    include Shoryuken::Worker\n    Kiev::Shoryuken.suffix_tree_path(self, \"B\")\n    # ...\n  end\n\n  # Or use a suffix process-wide:\n  Shoryuken.configure_server do |config|\n    Kiev::Shoryuken.suffix_tree_path(config, \"B\")\n  end\n  ```\n\nHere's an example of the possble `tree_path` sequence you could get by configuring two consumers with suffixes `1` and `2` (note ordering by `tree_path`):\n\n| `tree_path` |                             Meaning                             |\n|-------------|-----------------------------------------------------------------|\n| `A`         | An entry point into the system, a synchronous request           |\n| `AB`        | Background job caused by `A` executed                           |\n| `ABA`       | Synchoronous request made from `AB`                             |\n| `ABD`       | _(Not logged by Kiev itself)_ `AB` sends out an SNS message     |\n| `ABD1`      | Message `ABD` handled by susbcriber `1`                         |\n| `ABD1A`     | Synchronous request sent by `1` when handling the message `ABD` |\n| `ABD1C`     | Synchronous request sent by `1` when handling the message `ABD` |\n| `ABD2`      | Message `ABD` handled by susbcriber `2`                         |\n| `ABD2A`     | Synchronous request sent by `2` when handling the message `ABD` |\n| `ABF`       | Another backgound job from `AB` executed                        |\n| `AD`        | Background job caused by `A` executed                           |\n| `AE`        | Synchronous request made from `A`                               |\n\nWithout suffixing you won't see at a glance who made the request `ABDC` and you will have two entries for both `ABD` and `ABDA`. As different subscribers may log different fields, you might be able to tell apart `ABD`s. But both `ABDA`s could happen on the same node and be logged with the same lines of code.\n\n## Alternatives\n\n### Logging\n\n- [semantic_logger](http://rocketjob.github.io/semantic_logger/)\n- [lograge](https://github.com/roidrage/lograge)\n- [logging](https://github.com/TwP/logging)\n\n### Request-Id\n\n- [Pliny::Middleware::RequestID](https://github.com/interagent/pliny/blob/master/lib/pliny/middleware/request_id.rb)\n- [ActionDispatch::RequestId](http://api.rubyonrails.org/classes/ActionDispatch/RequestId.html)\n- [request_id](https://github.com/remind101/request_id)\n\n## Development\n\nPull the code:\n\n```\ngit clone git@github.com:blacklane/kiev.git\n```\n\nRun tests:\n\n```sh\nbundle exec rake\n```\n\nRun tests for different rubies, frameworks and framework versions:\n\n```sh\n# Create a Postgres test database for Que\ncreatedb que_test\n\n# Run the tests (replace myuser with your username)\nDATABASE_URL=postgres://myuser:@localhost/que_test bundle exec wwtd\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblacklane%2Fkiev","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblacklane%2Fkiev","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblacklane%2Fkiev/lists"}