{"id":15562130,"url":"https://github.com/mrkamel/redstream","last_synced_at":"2025-08-26T19:07:51.532Z","repository":{"id":34771885,"uuid":"156717280","full_name":"mrkamel/redstream","owner":"mrkamel","description":"Using redis streams to keep your primary database in sync with secondary datastores","archived":false,"fork":false,"pushed_at":"2024-09-21T17:04:03.000Z","size":101,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-08T04:41:00.558Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mrkamel.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-08T14:15:43.000Z","updated_at":"2024-09-21T17:04:06.000Z","dependencies_parsed_at":"2025-03-06T19:31:08.189Z","dependency_job_id":"e0047357-5ce6-4dd7-9050-25f10e8b8424","html_url":"https://github.com/mrkamel/redstream","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/mrkamel/redstream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fredstream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fredstream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fredstream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fredstream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mrkamel","download_url":"https://codeload.github.com/mrkamel/redstream/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fredstream/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265383482,"owners_count":23756548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T16:11:54.863Z","updated_at":"2025-07-15T01:09:26.504Z","avatar_url":"https://github.com/mrkamel.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Redstream\n\n**Using redis streams to keep your primary database in sync with secondary\ndatastores (e.g. elasticsearch).**\n\n[![Build Status](https://github.com/mrkamel/redstream/workflows/test/badge.svg?branch=master)](https://github.com/mrkamel/redstream/actions?query=workflow%3Atest)\n\n## Installation\n\nFirst, install redis. Then, add this line to your application's Gemfile:\n\n```ruby\ngem 'redstream'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install redstream\n\n## Reference Docs\n\nThe reference docs can be found at\n[https://www.rubydoc.info/github/mrkamel/redstream/master](https://www.rubydoc.info/github/mrkamel/redstream/master).\n\n## Usage\n\nInclude `Redstream::Model` in your model and add a call to\n`redstream_callbacks`.\n\n```ruby\nclass MyModel \u003c ActiveRecord::Base\n  include Redstream::Model\n\n  # ...\n\n  redstream_callbacks\n\n  # ...\nend\n```\n\n`redstream_callbacks` adds `after_save`, `after_touch`, `after_destroy` and,\nmost importantly, `after_commit` callbacks which write messages, containing the\nrecord id, to a redis stream. A background worker can then fetch those messages\nand update secondary datastores.\n\nIn a background process, you need to run a `Redstream::Consumer`, `Redstream::Delayer`\nand a `Redstream::Trimmer`:\n\n```ruby\nRedstream::Consumer.new(stream_name: Product.redstream_name, name: \"consumer\").run do |messages|\n  # Update seconday datastore\nend\n\n# ...\n\nRedstream::Delayer.new(stream_name: Product.redstream_name, delay: 5.minutes).run\n\n# ...\n\ntrimmer = RedStream::Trimmer.new(\n  stream_name: Product.redstream_name,\n  consumer_names: [\"indexer\", \"cacher\"],\n  interval: 30\n)\n\ntrimmer.run\n```\n\nAs all of them are blocking, you should run them in individual threads. But as\nnone of them must be stopped gracefully, this can be as simple as:\n\n```ruby\nThread.new do\n  Redstream::Consumer.new(\"...\").run do |messages|\n    # ...\n  end\nend\n```\n\nMore concretely, `after_save`, `after_touch` and `after_destroy` only write\n\"delay\" messages to an additional redis stream. Delay message are like any\nother messages, but they get processed by a `Redstream::Delayer` and the\n`Delayer` will wait for some (configurable) delay/time before processing them.\nAs the `Delayer` is neccessary to fix inconsistencies, the delay must be at\nleast as long as your maximum database transaction time. Contrary,\n`after_commit` writes messages to a redis stream from which the messages can\nbe fetched immediately to keep the secondary datastores updated in\nnear-realtime. The reasoning of all this is simple: usually, i.e. by using only\none way to update secondary datastores, namely `after_save` or `after_commit`,\nany errors occurring in between `after_save` and `after_commit` result in\ninconsistencies between your primary and secondary datastore. By using these\nkinds of \"delay\" messages triggered by `after_save` and fetched after e.g. 5\nminutes, errors occurring in between `after_save` and `after_commit` can be\nfixed when the delay message get processed.\n\nAny messages are fetched in batches, such that e.g. elasticsearch can be\nupdated using its bulk API. For instance, depending on which elasticsearch ruby\nclient you are using, the reindexing code regarding elasticsearch will look\nsimilar to:\n\n```ruby\nThread.new do\n  Redstream::Consumer.new(stream_name: Product.redstream_name, name: \"indexer\").run do |messages|\n    ids = messages.map { |message| message.payload[\"id\"] }\n\n    ProductIndex.import Product.where(id: ids)\n  end\nend\n\nThread.new do\n  Redstream::Delayer.new(stream_name: Product.redstream_name, delay: 5.minutes).run\nend\n\nThread.new do\n  RedStream::Trimmer.new(stream_name: Product.redstream_name, consumer_names: [\"indexer\"], interval: 30).run\nend\n```\n\nYou should run a consumer per `(stream_name, name)` tuple on multiple hosts for\nhigh availability. They'll use a redis based locking mechanism to ensure that\nonly one consumer is consuming messages per tuple while the others are\nhot-standbys, i.e. they'll take over in case the currently active instance\ndies. The same stands for delayers and trimmers.\n\nPlease note: if you have multiple kinds of consumers for a single model/topic,\nthen you must use distinct names. Assume you have an indexer, which updates a\nsearch index for a model and a cacher, which updates a cache store for a model:\n\n```ruby\nRedstream::Consumer.new(stream_name: Product.redstream_name, name: \"indexer\").run do |messages|\n  # ...\nend\n\nRedstream::Consumer.new(stream_name: Product.redstream_name, name: \"cacher\").run do |messages|\n  # ...\nend\n```\n\n# Consumer, Delayer, Trimmer, Producer\n\nA `Consumer` fetches messages that have been added to a redis stream via\n`after_commit` or by a `Delayer`, i.e. messages that are available for\nimmediate retrieval/reindexing/syncing.\n\n```ruby\n  Redstream::Consumer.new(stream_name: Product.redstream_name, name: \"indexer\").run do |messages|\n    ids = messages.map { |message| message.payload[\"id\"] }\n\n    ProductIndex.import Product.where(id: ids)\n  end\n```\n\nA `Delayer` fetches messages that have been added to a second redis stream via\n`after_save`, `after_touch` and `after_destroy` to be retrieved after a certain\nconfigurable amount of time (5 minutes usually) to fix inconsistencies. The\namount of time must be longer than your maximum database transaction time at\nleast.\n\n```ruby\n  Redstream::Delayer.new(stream_name: Product.redstream_name, delay: 5.minutes).run\n```\n\nA `Trimmer` is responsible to finally remove messages from redis streams.\nWithout a `Trimmer` messages will fill up your redis server and redis will\nfinally crash due to out of memory errors. To be able to trim a stream, you\nmust pass an array containing all consumer names reading from the respective\nstream. The `Trimmer` then continously checks how far each consumer already\nprocessed the stream and trims the stream up to the committed minimum.\nContrary, if there is nothing to trim, the `Trimmer` will sleep for a specified\n`interval`.\n\n```ruby\n  RedStream::Trimmer.new(stream_name: Product.redstream_name, consumer_names: [\"indexer\"], interval: 30).run\n```\n\nA `Producer` adds messages to the concrete redis streams, and you\ncan actually pass a concrete `Producer` instance via `redstream_callbacks`:\n\n```ruby\nclass Product \u003c ActiveRecord::Base\n  include Redstream::Model\n\n  # ...\n\n  redstream_callbacks producer: Redstream::Producer.new(\"...\")\n\n  # ...\nend\n```\n\nAs you might recognize, `Redstream::Model` is of course only able to send\nmessages to redis streams for model lifecyle callbacks. This is however not\nthe case for `#update_all`:\n\n```ruby\nProduct.where(on_stock: true).update_all(featured: true)\n```\n\nTo capture those updates as well, you need to change:\n\n```ruby\nProduct.where(on_stock: true).update_all(featured: true)\n```\n\nto\n\n```ruby\nRedstreamProducer = Redstream::Producer.new\n\nProduct.where(on_stock: true).find_in_batches do |products|\n  RedstreamProducer.bulk products do\n    Product.where(id: products.map(\u0026:id)).update_all(featured: true)\n  end\nend\n```\n\nThe `Producer` will write a message for every matched record into the delay\nstream before `update_all` is called and will write another message for every\nrecord to the main stream after `update_all` is called - just like it is done\nwithin the model lifecycle callbacks.\n\nThe `#bulk` method must ensure that the same set of records is used for the\ndelay messages and the instant messages. Thus, you better directly pass an\narray of records to `Redstream::Producer#bulk`, like shown above. If you pass\nan `ActiveRecord::Relation`, the `#bulk` method will convert it to an array,\ni.e. load the whole result set into memory.\n\n## Sharding\n\nWhen you want to attach multiple consumers to a single stream, you maybe want\nto add sharding. This can be accomplished by specifying a dynamic stream name\nwhere you compute the shard key by hashing the primary key.\n\n```ruby\nclass Product \u003c ActiveRecord::Base\n  include Redstream::Model\n\n  NUM_SHARDS = 4\n\n  def self.redstream_name(shard)\n    \"products-#{shard}\"\n  end\n\n  def redstream_name\n    self.class.redstream_name(Digest::SHA1.hexdigest(id.to_s)[0, 4].to_i(16) % NUM_SHARDS)\n  end\nend\n```\n\nThe sharding via hashing the primary key is neccessary, because we want each\nchange of a specific object to end up in the same stream. Otherwise the order\nof changes for a specific object gets mixed up. Subsequently, you can add\nconsumers, etc for each individual stream name.\n\n## Namespacing\n\nIn case you are using a shared redis, where multiple appications read/write\nfrom the same redis server using Redstream, key conflicts could occur.\nTo avoid that, you want to use namespacing:\n\n```ruby\nRedstream.namespace = 'my_app'\n```\n\nsuch that every application will have its own namespaced Redstream keys.\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/mrkamel/redstream\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrkamel%2Fredstream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrkamel%2Fredstream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrkamel%2Fredstream/lists"}