{"id":13835589,"url":"https://github.com/jhollinger/occams-record","last_synced_at":"2025-07-10T08:30:42.867Z","repository":{"id":47068204,"uuid":"99639871","full_name":"jhollinger/occams-record","owner":"jhollinger","description":"The missing high-efficiency query API for ActiveRecord","archived":false,"fork":false,"pushed_at":"2024-08-31T14:52:15.000Z","size":1657,"stargazers_count":357,"open_issues_count":1,"forks_count":7,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-08-31T15:05:27.084Z","etag":null,"topics":["activerecord","activerecord-queries","performance","ruby","sql"],"latest_commit_sha":null,"homepage":"https://occams.jordanhollinger.com/","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jhollinger.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-08T02:15:39.000Z","updated_at":"2024-08-31T14:51:59.000Z","dependencies_parsed_at":"2023-12-16T00:04:24.494Z","dependency_job_id":"d765a355-6f83-4d9b-a960-c9ed3ebffd97","html_url":"https://github.com/jhollinger/occams-record","commit_stats":{"total_commits":386,"total_committers":6,"mean_commits":64.33333333333333,"dds":0.2953367875647669,"last_synced_commit":"e75ddc8acc8ac172a1fda54fb2a5fa683a312472"},"previous_names":[],"tags_count":79,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jhollinger%2Foccams-record","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jhollinger%2Foccams-record/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jhollinger%2Foccams-record/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jhollinger%2Foccams-record/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jhollinger","download_url":"https://codeload.github.com/jhollinger/occams-record/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225627411,"owners_count":17498981,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["activerecord","activerecord-queries","performance","ruby","sql"],"created_at":"2024-08-04T14:01:05.922Z","updated_at":"2025-07-10T08:30:42.860Z","avatar_url":"https://github.com/jhollinger.png","language":"Ruby","funding_links":[],"categories":["Ruby","Gems"],"sub_categories":["Performance Optimization"],"readme":"# Occams Record\n\n\u003e Do not multiply entities beyond necessity. -- Occam's Razor\n\u003e\n\u003e \n\nLearn OccamsRecord by reading [The Book at occams.jordanhollinger.com](https://occams.jordanhollinger.com/).\n\nAPI documentation is available at [rubydoc.info/gems/occams-record](http://www.rubydoc.info/gems/occams-record).\n\n*Want OccamsRecord's flexible preloading in ActiveRecord itself? Check out [Uberloader](https://github.com/jhollinger/uberloader/)!*\n\nOccamsRecord is a high-efficiency, advanced query library for use alongside ActiveRecord. It is **not** an ORM or an ActiveRecord replacement. OccamsRecord can breathe fresh life into your ActiveRecord app by giving it two things:\n\n### 1) Huge performance gains\n\n* 3x-5x faster than ActiveRecord queries, *minimum*.\n* Uses 1/3 the memory of ActiveRecord query results.\n* Eliminates the N+1 query problem. (This often exceeds the baseline 3x-5x gain.)\n\n### 2) Supercharged querying \u0026 eager loading\n\n* Customize the SQL used to eager load associations (order them, apply filters, etc)\n* Use cursors (Postgres only)\n* Use `ORDER BY` with `find_each`/`find_in_batches`\n* Use `find_each`/`find_in_batches` with raw SQL\n* Eager load associations off of raw SQL queries\n* Use `pluck` with raw SQL queries\n\n### How does OccamsRecord do all this?\n[Look over the speed and memory measurements yourself!](https://github.com/jhollinger/occams-record/wiki/Measurements) OccamsRecord achieves all of this by making some **very specific trade-offs:**\n\n* OccamsRecord results are *read-only*.\n* OccamsRecord results are *purely database rows* - they don't have any instance methods from your Rails models.\n* You *must eager load* each assocation you intend to use. If you try to use one you didn't eager load, an exception will be raised.\n\n# Overview\n\nFull documentation is available at [rubydoc.info/gems/occams-record](http://www.rubydoc.info/gems/occams-record). Code lives at at [github.com/jhollinger/occams-record](https://github.com/jhollinger/occams-record). Contributions welcome!\n\nSimply add `occams-record` to your `Gemfile`:\n\n```ruby\ngem 'occams-record'\n```\n\nBuild your queries like normal, using ActiveRecord's excellent query builder. Then pass them off to Occams Record.\n\n```ruby\nq = Order\n  .completed\n  .where(\"order_date \u003e ?\", 30.days.ago)\n  .order(\"order_date DESC\")\n\norders = OccamsRecord\n  .query(q)\n  .run\n````\n\n`each`, `map`, `reduce`, and other Enumerable methods may be used instead of *run*. `find_each` and `find_in_batches` are also supported, and unlike in ActiveRecord, `ORDER BY` works as you'd expect.\n\nOccams Record has great support for raw SQL queries too, but we'll get to those later.\n\n## Basic eager loading\n\nEager loading is similiar to ActiveRecord's `preload`: each association is loaded in a separate query. Unlike ActiveRecord, nested associations use blocks instead of Hashes. More importantly, if you try to use an association you didn't eager load *an exception will be raised*. In other words, the N+1 query problem simply doesn't exist.\n\n```ruby\nOccamsRecord\n  .query(q)\n  .eager_load(:customer)\n  .eager_load(:line_items) { |l|\n    l.eager_load(:product)\n    l.eager_load(:something_else)\n  }\n  .find_each { |order|\n    puts order.customer.name\n    order.line_items.each { |line_item|\n      puts line_item.product.name\n      puts line_item.product.category.name\n      OccamsRecord::MissingEagerLoadError: Association 'category' is unavailable on Product because it was not eager loaded! Found at root.line_items.product\n    }\n  }\n```\n\n## Advanced eager loading\n\nOccams Record allows you to tweak the SQL of any eager load. Pull back only the columns you need, change the order, add a `WHERE` clause, etc.\n\n```ruby\norders = OccamsRecord\n  .query(q)\n  # Only SELECT the columns you need. Your DBA will thank you.\n  .eager_load(:customer, select: \"id, name\")\n\n  # Or use 'scope' to access the full power of ActiveRecord's query builder.\n  # Here, only 'active' line items will be returned, and in a specific order.\n  .eager_load(:line_items) { |l|\n    l.scope { |q| q.active.order(\"created_at\") }\n\n    l.eager_load(:product)\n    l.eager_load(:something_else)\n  }\n  .run\n```\n\nOccams Record also supports loading ad hoc associations using raw SQL. We'll get to that in a later section.\n\n## Query with cursors\n\n`find_each_with_cursor`/`find_in_batches_with_cursor` work like `find_each`/`find_in_batches`, except they use cursors. For large data sets, cursors offer a noticible speed boost. Postgres only.\n\n```ruby\nOccamsRecord\n  .query(q)\n  .eager_load(:customer)\n  .find_each_with_cursor do |order|\n    ...\n  end\n```\n\nThe `cursor.open` method allows lower level access to cursor behavior. See `OccamsRecord::Cursor` for more info.\n\n```ruby\norders = OccamsRecord\n  .query(q)\n  .eager_load(:customer)\n  .cursor\n  .open do |cursor|\n    cursor.move(:forward, 300)\n    cursor.fetch(:forward, 100)\n  end\n```\n\n## Raw SQL queries\n\nActiveRecord has raw SQL escape hatches like `find_by_sql` and `exec_query`, but they give up critical features like eager loading and `find_each`/`find_in_batches`. Occams Record's escape hatches don't make you give up anything.\n\n**Query params**\n\n```ruby\n# Supported in all versions of OccamsRecord\nOccamsRecord.sql(\"SELECT * FROM orders WHERE user_id = %{user_id}\", {user_id: user.id}).run\n\n# Supported in OccamsRecord 1.9+\nOccamsRecord.sql(\"SELECT * FROM orders WHERE user_id = :user_id\", {user_id: user.id}).run\nOccamsRecord.sql(\"SELECT * FROM orders WHERE user_id = ?\", [user.id]).run\nOccamsRecord.sql(\"SELECT * FROM orders WHERE user_id = %s\", [user.id]).run\n```\n\n**Batched loading with cursors**\n\n`find_each_with_cursor`, `find_in_batches_with_cursor`, and `cursor.open` are all available.\n\n```ruby\nOccamsRecord\n  .sql(\"\n    SELECT * FROM orders\n    WHERE order_date \u003e %{date}\n    ORDER BY order_date DESC, id\n  \", {\n    date: 10.years.ago\n  })\n  .find_each_with_cursor(batch_size: 1000) do |order|\n    ...\n  end\n```\n\n**Batched loading without cursors**\n\nIf your database doesn't support cursors, you can use `find_each`/`find_in_batches`. Just provide `LIMIT` and `OFFSET` (see below), and Occams will plug in the right numbers.\n\n```ruby\nOccamsRecord\n  .sql(\"\n    SELECT * FROM orders\n    WHERE order_date \u003e %{date}\n    ORDER BY order_date DESC, id\n    LIMIT %{batch_limit}\n    OFFSET %{batch_offset}\n  \", {\n    date: 10.years.ago\n  })\n  .find_each(batch_size: 1000) do |order|\n    ...\n  end\n```\n\n**Eager loading**\n\nTo use `eager_load` with a raw SQL query you must tell Occams what the base model is. (That doesn't apply if you're loading an ad hoc, raw SQL association. We'll get to those next.)\n\n```ruby\norders = OccamsRecord\n  .sql(\"\n    SELECT * FROM orders\n    WHERE order_date \u003e %{date}\n    ORDER BY order_date DESC, id\n  \", {\n    date: 30.days.ago\n  })\n  .model(Order)\n  .eager_load(:customer)\n  .run\n```\n\n## Raw SQL eager loading\n\nLet's say we want to load each product with an array of all customers who've ordered it. We *could* do that by loading various nested associations:\n\n```ruby\nproducts_with_orders = OccamsRecord\n  .query(Product.all)\n  .eager_load(:line_items) { |l|\n    l.eager_load(:order) { |l|\n      l.eager_load(:customer)\n    }\n  }\n  .map { |product|\n    customers = product.line_items.map(\u0026:order).map(\u0026:customer).uniq\n    [product, customers]\n  }\n```\n\nBut that's very wasteful. Occams gives us better options: `eager_load_many` and `eager_load_one`.\n\n```ruby\nproducts = OccamsRecord\n  .query(Product.all)\n  .eager_load_many(:customers, {:id =\u003e :product_id}, \"\n    SELECT DISTINCT product_id, customers.*\n    FROM line_items\n      INNER JOIN orders ON line_items.order_id = orders.id\n      INNER JOIN customers on orders.customer_id = customers.id\n    WHERE line_items.product_id IN (%{ids})\n  \", binds: {\n    # additional bind values (ids will be passed in for you)\n  })\n  .run\n```\n\n`eager_load_many` is declaring an ad hoc *has_many* association called *customers*. The `{:id =\u003e :product_id}` Hash defines the mapping: *id* in the parent record maps to *product_id* in the child records.\n\nThe SQL string and binds should be familiar. `%{ids}` will be provided for you - just stick it in the right place. Note that it won't always be called *ids*; the name will be the plural version of the key in your mapping.\n\n`eager_load_one` defines an ad hoc `has_one`/`belongs_to` association. It and `eager_load_many` are available with both `OccamsRecord.query` and `OccamsRecord.sql`.\n\n## Injecting instance methods\n\nOccams Records results are just plain rows; there are no methods from your Rails models. (Separating your persistence layer from your domain is good thing!) But sometimes you need a few methods. Occams Record provides two ways of accomplishing this.\n\n### Include custom modules\n\nYou may also specify one or more modules to be included in your results:\n\n```ruby\nmodule MyOrderMethods\n  def description\n    \"#{order_number} - #{date}\"\n  end\nend\n\nmodule MyProductMethods\n  def expensive?\n    price \u003e 100\n  end\nend\n\norders = OccamsRecord\n  .query(Order.all, use: MyOrderMethods)\n  .eager_load(:line_items) {\n    eager_load(:product, use: [MyProductMethods, OtherMethods])\n  }\n  .run\n```\n\n### ActiveRecord method fallback\n\nThis is an ugly hack of last resort if you can't easily extract a method from your model into a shared module. Plugins, like `carrierwave`, are a good example. When you call a method that doesn't exist on an Occams Record result, it will initialize an ActiveRecord object and forward the method call to it.\n\nThe `active_record_fallback` option must be passed either `:lazy` or `:strict` (recommended). `:strict` enables ActiveRecord's strict loading option, helping you avoid N+1 queries. :lazy allows them. Note that `:strict` is only available for ActiveRecord 6.1 and later.\n\nThe following will forward any nonexistent methods for `Order` and `Product` records:\n\n```ruby\norders = OccamsRecord\n  .query(Order.all, active_record_fallback: :strict)\n  .eager_load(:line_items) {\n    eager_load(:product, active_record_fallback: :strict)\n  }\n  .run\n```\n\n---\n\n# Unsupported features\n\nThe following ActiveRecord features are under consideration, but not high priority. Pull requests welcome!\n\n* Eager loading `through` associations that involve a `has_and_belongs_to_many`.\n\nThe following ActiveRecord features are not supported, and likely never will be. Pull requests are still welcome, though.\n\n* Eager loading `through` associations that involve a polymorphic association.\n* ActiveRecord serialized types\n\n---\n\n# Benchmarking\n\n`bundle exec rake bench` will run a suite of speed and memory benchmarks comparing Occams Record to Active Record. [You can find an example of a typical run here.](https://github.com/jhollinger/occams-record/wiki/Measurements) These are primarily used during development to prevent performance regressions. An in-memory Sqlite database is used.\n\nIf you run your own benchmarks, keep in mind exactly what you're measuring. For example if you're benchmarking a report written in AR vs OR, there are many constants in that measurement: the time spent in the database, the time spent sending the database results over the network, any calculations you're doing in Ruby, and the time spent building your html/json/csv/etc. So if OR is 3x faster than AR, the total runtime of said report *won't* improve by 3x.\n\nOn the other hand, Active Record makes it *very* easy to forget to eager load associations (the N+1 query problem). Occams Record fixes that. So if your report was missing some associations you could see easily see performance improvements well over 3x.\n\n# Testing\n\nTests are run with `appraisal` in Docker Compose using the `bin/test` or `bin/testall` scripts. See [test/matrix](./test/matrix) for the full list of Ruby, ActiveRecord, and database versions that are tested against.\n\n```bash\n# Run tests against all supported ActiveRecord versions, Ruby versions, and databases\nbin/testall\n\n# Run tests only for Ruby 3.1\nbin/testall ruby-3.1\n\n# Run tests only for Ruby 3.1 and ActiveRecored 6.1\nbin/testall ruby-3.1 ar-6.1\n\n# Run tests against a specific database\nbin/testall sqlite3|postgres-14|mysql-8\n\n# Run exactly one set of tests\nbin/test ruby-3.1 ar-7.0 postgres-14\n\n# Use Podman Compose\nOCCAMS_PODMAN=1 bin/testall\n\n# If all tests complete successfully, you'll be rewarded by an ASCII Nyancat!\n\n+      o     +              o\n    +             o     +       +\no          +\n    o  +           +        +\n+        o     o       +        o\n-_-_-_-_-_-_-_,------,      o\n_-_-_-_-_-_-_-|   /\\_/\\\n-_-_-_-_-_-_-~|__( ^ .^)  +     +\n_-_-_-_-_-_-_-\"\"  \"\"\n    +      o         o   +       o\n    +         +\no        o         o      o     +\n    o           +\n+      +     o        o      +\n```\n\n## Testing without Docker\n\nIt's possible to run tests without Docker Compose, but you'll be limited by the Ruby version(s) and database(s) you have on your system.\n\n```bash\nbundle install\nbundle exec appraisal ar-8.0 bundle install\nbundle exec appraisal ar-8.0 rake test\n```\n\n# License\n\nMIT License. See LICENSE for details.\n\n# Copyright\n\nCopyright (c) 2019 Jordan Hollinger.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjhollinger%2Foccams-record","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjhollinger%2Foccams-record","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjhollinger%2Foccams-record/lists"}