{"id":52089050,"url":"https://github.com/rameerez/rails-sqlite-partial-index-dumper-bug","last_synced_at":"2026-08-04T04:30:39.267Z","repository":{"id":372569719,"uuid":"1308276186","full_name":"rameerez/rails-sqlite-partial-index-dumper-bug","owner":"rameerez","description":"Executable proof of a Rails SQLite partial/expression index schema-dumper newline bug","archived":false,"fork":false,"pushed_at":"2026-07-22T02:49:10.000Z","size":83,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-22T04:19:08.879Z","etag":null,"topics":["activerecord","bug-reproduction","rails","ruby-on-rails","schema-dumper","sqlite"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rameerez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-22T01:49:03.000Z","updated_at":"2026-07-22T02:49:15.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/rameerez/rails-sqlite-partial-index-dumper-bug","commit_stats":null,"previous_names":["rameerez/rails-sqlite-partial-index-dumper-bug"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/rameerez/rails-sqlite-partial-index-dumper-bug","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rameerez%2Frails-sqlite-partial-index-dumper-bug","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rameerez%2Frails-sqlite-partial-index-dumper-bug/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rameerez%2Frails-sqlite-partial-index-dumper-bug/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rameerez%2Frails-sqlite-partial-index-dumper-bug/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rameerez","download_url":"https://codeload.github.com/rameerez/rails-sqlite-partial-index-dumper-bug/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rameerez%2Frails-sqlite-partial-index-dumper-bug/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36260999,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-08-04T02:00:06.901Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["activerecord","bug-reproduction","rails","ruby-on-rails","schema-dumper","sqlite"],"created_at":"2026-08-04T04:30:38.683Z","updated_at":"2026-08-04T04:30:39.251Z","avatar_url":"https://github.com/rameerez.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Rails SQLite index introspection loses multiline partial predicates\n\n[![Verification](https://github.com/rameerez/rails-sqlite-partial-index-dumper-bug/actions/workflows/ci.yml/badge.svg)](https://github.com/rameerez/rails-sqlite-partial-index-dumper-bug/actions/workflows/ci.yml)\n\nExecutable reproduction and source-backed audit of an Active Record\nSQLite3Adapter bug. A terminal newline—or a newline inside one of the parser's\ndot-based captures—can make Rails forget a partial index's `WHERE` clause. An\nexpression index can instead break introspection and cause `schema.rb` to omit\nthe entire table.\n\nStatus: **confirmed on Active Record 7.1.6, 7.2.3.1, 8.0.5, 8.1.3, and Rails\nmain at [`d9e67f6`](https://github.com/rails/rails/commit/d9e67f6268fc6793ecc7bbfa6c71e145a6dc8096).**\nThis is an ordinary correctness bug, not a security vulnerability. It is filed\nupstream as [rails/rails#58200](https://github.com/rails/rails/issues/58200),\nwith the tested fix in\n[rails/rails#58201](https://github.com/rails/rails/pull/58201).\n\n## Upstream issue and native fix\n\nThe upstream artifacts are deliberately cross-linked and independently\nreviewable:\n\n- [Issue #58200](https://github.com/rails/rails/issues/58200) contains the\n  compact executable reproduction and exact causal chain.\n- [Pull request #58201](https://github.com/rails/rails/pull/58201) contains the\n  one-line adapter fix, four native Minitest regressions, and the Active Record\n  changelog entry.\n- [Commit `a4f3c229b8`](https://github.com/rails/rails/pull/58201/commits/a4f3c229b864c6ebbe7e2c03d3a85482ef23c9eb)\n  is based on the same pinned Rails revision used by this audit.\n- The exact submitted [parser line](https://github.com/rameerez/rails/blob/a4f3c229b864c6ebbe7e2c03d3a85482ef23c9eb/activerecord/lib/active_record/connection_adapters/sqlite3/schema_statements.rb#L24)\n  and [regression tests](https://github.com/rameerez/rails/blob/a4f3c229b864c6ebbe7e2c03d3a85482ef23c9eb/activerecord/test/cases/adapters/sqlite3/sqlite3_adapter_test.rb#L732-L809)\n  are immutable links into the submitted commit.\n- Every upstream check is green: the [Rails Buildkite suite](https://buildkite.com/rails/rails/builds/131320),\n  [Docker smoke test](https://github.com/rails/rails/actions/runs/29885637092),\n  [docs preview](https://buildkite.com/rails/docs-preview/builds/20921), and\n  [labeler](https://github.com/rails/rails/actions/runs/29885636090).\n\nThe native tests were run without the fix first. They reproduced predicate\nloss, expression-index `NoMethodError`, and schema-dump table omission. With the\nfix applied, validation on Ruby 3.4.2 and sqlite3 2.9.5 produced:\n\n```text\nFocused regressions:          4 runs,    11 assertions, 0 failures, 0 errors\nsqlite3_adapter_test.rb:    101 runs,   253 assertions, 0 failures, 0 errors\nFull Active Record SQLite: 9,643 runs, 32,864 assertions, 0 failures, 0 errors, 38 skips\nRuboCop:                       3 files inspected, no offenses detected\n```\n\n## The shortest convincing proof\n\nThis is not limited to raw `execute` SQL. The documented public API triggers it:\n\n```ruby\nconnection.add_index :join_requests, [:organization_id, :user_id],\n  unique: true,\n  name: :idx_pending,\n  where: \u003c\u003c~SQL\n    status = 'pending'\n  SQL\n```\n\nAfter that call:\n\n- SQLite's structured `PRAGMA index_list` metadata says the index is both unique\n  and partial (`partial=1`). [SQLite documents that field\n  exactly](https://www.sqlite.org/pragma.html#pragma_index_list).\n- SQLite's stored CREATE text ends with the heredoc's LF for this input.\n- `connection.indexes(:join_requests)` returns the index with `unique=true` but\n  `where=nil`.\n- `schema.rb` consequently emits a full unique index. Loading it into a fresh\n  database changes the constraint from “one pending request per key” to “one\n  request per key regardless of status.”\n\nThe round-trip test proves the semantic payload: a rejected row followed by a\npending row succeeds in the migrated database, while the same writes against a\nschema-loaded database raise `ActiveRecord::RecordNotUnique`.\n\n[`repro.rb`](repro.rb) makes every step executable. The complete reasoning,\ntrigger map, raw outputs, corrections to earlier hypotheses, and fix analysis\nare in [EVIDENCE.md](EVIDENCE.md). Every external claim is mapped to an exact\nprimary source in [SOURCES.md](SOURCES.md).\n\n## Run it\n\nRuby 3.4.2 is recorded in [`.ruby-version`](.ruby-version). The scripts use\n`bundler/inline` and in-memory SQLite; there is no application setup.\n\n```bash\n# Intentionally red: tests assert the correct Rails behavior.\nruby repro.rb\n\n# Green only when the complete known upstream failure signature is present.\nruby script/verify_bug.rb\n\n# Green validation of the candidate parser change.\nruby fix_validation.rb\n```\n\nTest exact releases or current Rails main:\n\n```bash\nAR_VERSION=7.1.6 ruby script/verify_bug.rb\nAR_VERSION=7.2.3.1 ruby script/verify_bug.rb\nAR_VERSION=8.0.5 ruby script/verify_bug.rb\nAR_VERSION=8.1.3 ruby script/verify_bug.rb\nAR_SOURCE=edge ruby script/verify_bug.rb\n\nAR_VERSION=7.1.6 ruby fix_validation.rb\nAR_SOURCE=edge ruby fix_validation.rb\n```\n\nThe direct repro is supposed to exit nonzero with:\n\n```text\n7 runs, 8 assertions, 4 failures, 2 errors, 0 skips\n```\n\nThe wrapper rejects partial, incidental, or changed failures; it checks all six\naffected test names, `RecordNotUnique`, the expression-index `NoMethodError`, the\nschema-dumper diagnostic, and the exact summary. The candidate validator exits\nzero with:\n\n```text\n14 runs, 25 assertions, 0 failures, 0 errors, 0 skips\n```\n\n## What actually fails\n\n| Case | Current upstream behavior |\n|---|---|\n| Single-line `where:` control | Correctly returns the predicate |\n| Public `add_index` with a heredoc predicate | SQLite says partial; Rails returns `where=nil` |\n| Otherwise single-line raw SQL plus one final LF | Rails returns `where=nil` |\n| Semicolon-free multiline migration heredoc | Rails returns `where=nil` |\n| Dump/load of the partial unique index | Valid second insert raises `RecordNotUnique` |\n| Expression index plus final LF | `connection.indexes` raises `NoMethodError` |\n| Schema dump with that expression index | Dumper catches the error, writes `Could not dump table`, omits the entire table, and returns normally |\n\nThat final row is important. The expression-index case does **not** necessarily\nmake `db:schema:dump` exit nonzero: Rails rescues at table scope in\n[`SchemaDumper#table`](https://github.com/rails/rails/blob/d9e67f6268fc6793ecc7bbfa6c71e145a6dc8096/activerecord/lib/active_record/schema_dumper.rb#L225-L248).\nThe reproducible result is a diagnostic comment and a missing table.\n\n## Root cause\n\nRails main parses `sqlite_master.sql` here:\n[`sqlite3/schema_statements.rb:7-52`](https://github.com/rails/rails/blob/d9e67f6268fc6793ecc7bbfa6c71e145a6dc8096/activerecord/lib/active_record/connection_adapters/sqlite3/schema_statements.rb#L7-L52).\nThe exact line is:\n\n```ruby\n/\\bON\\b\\s*\"?(\\w+?)\"?\\s*\\((?\u003cexpressions\u003e.+?)\\)(?:\\s*WHERE\\b\\s*(?\u003cwhere\u003e.+))?(?:\\s*\\/\\*.*\\*\\/)?\\z/i =~ index_sql\n```\n\nTwo Ruby regex rules interact:\n\n- Dot excludes newline without `/m`; [Ruby documents `/./` versus\n  `/./m`](https://docs.ruby-lang.org/en/3.4/Regexp.html#class-Regexp-label-Shorthand+Character+Classes).\n- `\\z` means the exact end of the string and does not tolerate a final LF;\n  [Ruby distinguishes `\\Z` and `\\z`\n  explicitly](https://docs.ruby-lang.org/en/3.4/Regexp.html#class-Regexp-label-Boundary+Anchors).\n\nRuby also documents that [a heredoc result includes its ending\nnewline](https://docs.ruby-lang.org/en/3.4/syntax/literals_rdoc.html#label-Here+Document+Literals).\nRails carries a public `where:` string through index definition and SQL creation\nwithout stripping it:\n[`add_index_options`](https://github.com/rails/rails/blob/d9e67f6268fc6793ecc7bbfa6c71e145a6dc8096/activerecord/lib/active_record/connection_adapters/abstract/schema_statements.rb#L1601-L1623)\nand\n[`visit_CreateIndexDefinition`](https://github.com/rails/rails/blob/d9e67f6268fc6793ecc7bbfa6c71e145a6dc8096/activerecord/lib/active_record/connection_adapters/abstract/schema_creation.rb#L106-L123).\n\nWhen the complete match fails, both named captures are nil. Ordinary columns\nstill come from `PRAGMA index_info`, so predicate loss is silent. SQLite returns\na NULL name for an expression term; Rails then substitutes the nil\n`expressions` capture and `IndexDefinition` calls `columns.size`, explaining the\nsecond symptom exactly:\n[`schema_statements.rb:26-50`](https://github.com/rails/rails/blob/d9e67f6268fc6793ecc7bbfa6c71e145a6dc8096/activerecord/lib/active_record/connection_adapters/sqlite3/schema_statements.rb#L26-L50),\n[`schema_definitions.rb:64-71`](https://github.com/rails/rails/blob/d9e67f6268fc6793ecc7bbfa6c71e145a6dc8096/activerecord/lib/active_record/connection_adapters/abstract/schema_definitions.rb#L64-L71),\nand [SQLite's `PRAGMA index_info`\ncontract](https://www.sqlite.org/pragma.html#pragma_index_info).\n\n## Precise trigger scope\n\nThe old shorthand “any newline” was too broad. The green characterization probe\n[`script/trigger_matrix.rb`](script/trigger_matrix.rb) establishes:\n\n- terminal LF, repeated terminal LFs, and CRLF fail;\n- terminal space, tab, and bare CR match but are currently retained inside the\n  returned `where` string;\n- newline inside the expression/column capture or predicate capture fails;\n- leading newline, newline before `ON`, and newline immediately before `WHERE`\n  can survive because SQLite normalization or an existing `\\s*` handles them;\n- a semicolon-terminated input is not the same trigger in this setup because\n  SQLite stores it without the semicolon and trailing LF.\n\nLikewise, SQLite's precise contract is not “verbatim storage.” It stores a copy\nof CREATE text with [documented\nnormalization](https://www.sqlite.org/schematab.html#interpretation_of_the_schema_table).\nOur narrower empirical claim—this affected SQL retains its final LF byte—is\nasserted across sqlite3 gems 1.7.3, 2.5.0, and 2.9.5 by\n[`script/sqlite_storage_probe.rb`](script/sqlite_storage_probe.rb).\n\n## Why this is unambiguously a Rails bug\n\n- Rails [documents partial indexes for PostgreSQL and\n  SQLite](https://github.com/rails/rails/blob/d9e67f6268fc6793ecc7bbfa6c71e145a6dc8096/activerecord/lib/active_record/connection_adapters/abstract/schema_statements.rb#L909-L917)\n  and reports both partial- and expression-index support in the\n  [SQLite adapter](https://github.com/rails/rails/blob/d9e67f6268fc6793ecc7bbfa6c71e145a6dc8096/activerecord/lib/active_record/connection_adapters/sqlite3_adapter.rb#L209-L214).\n- The strongest reproducer uses that documented public `add_index` API.\n- Rails' generic dumper test expects `where:` whenever an adapter reports\n  support: [`schema_dumper_test.rb:250-257`](https://github.com/rails/rails/blob/d9e67f6268fc6793ecc7bbfa6c71e145a6dc8096/activerecord/test/cases/schema_dumper_test.rb#L250-L257).\n- The database's independent metadata contradicts the Rails object.\n- The dump/load changes actual uniqueness semantics, not merely formatting.\n- Rails says `schema.rb` attempts to capture the database and is the default\n  source used to build new databases: [migration guide,\n  lines 1552-1609](https://github.com/rails/rails/blob/d9e67f6268fc6793ecc7bbfa6c71e145a6dc8096/guides/source/active_record_migrations.md#L1552-L1609).\n- The bug remains on current main, and the exact parser line is byte-identical\n  across every tested version.\n\nUsing `structure.sql` is an application workaround, not a rebuttal: partial\nindexes are already representable by `schema.rb`, and the dumper handles the\nsame index correctly when formatting avoids the parser defect.\n\n## Implemented and validated fix\n\nThe narrow fix submitted in\n[rails/rails#58201](https://github.com/rails/rails/pull/58201) is:\n\n```ruby\n/\\bON\\b\\s*\"?(\\w+?)\"?\\s*\\((?\u003cexpressions\u003e.+?)\\)(?:\\s*WHERE\\b\\s*(?\u003cwhere\u003e.+?))?(?:\\s*\\/\\*.*\\*\\/)?\\s*\\z/im\n```\n\nIt adds `/m`, makes the predicate lazy, and consumes final whitespace outside\nthe capture. [`fix_validation.rb`](fix_validation.rb) applies that regex to a\ncopy of `#indexes` and passes all 14 positive/regression cases across the entire\nrelease/edge matrix. The upstream commit applies the same change directly to\nthe adapter and adds native coverage beside Rails' existing SQLite index tests.\n\nIt intentionally does not normalize the whole SQL with `squish` or\n`gsub(/\\s+/, \" \")`. Rails' [`String#squish`\nimplementation](https://github.com/rails/rails/blob/d9e67f6268fc6793ecc7bbfa6c71e145a6dc8096/activesupport/lib/active_support/core_ext/string/filters.rb#L3-L26)\ncollapses internal whitespace, which can change a predicate such as\n`WHERE note = 'two  spaces'`. The validator has an exact regression guard for\nthat case.\n\nThe native Rails patch was also proven red before green and passed the complete\nActive Record SQLite suite. Its [submitted tests](https://github.com/rameerez/rails/blob/a4f3c229b864c6ebbe7e2c03d3a85482ef23c9eb/activerecord/test/cases/adapters/sqlite3/sqlite3_adapter_test.rb#L732-L809)\nexercise the public-API predicate, terminal-newline expression, combined\nmultiline expression/predicate, literal whitespace, and full schema-dump paths.\nThe exact local and hosted results are in\n[UPSTREAM_VALIDATION.md](UPSTREAM_VALIDATION.md).\n\n## Prior art checked\n\n- [#31603](https://github.com/rails/rails/issues/31603), fixed by\n  [#31607](https://github.com/rails/rails/pull/31607): same broad partial-index\n  loss, different SQLite table-recreation path.\n- [#53570](https://github.com/rails/rails/pull/53570): merged fix to this parser\n  for trailing comments, reinforcing that accurate introspection is intended.\n- [#55627](https://github.com/rails/rails/issues/55627): same regex, different\n  unconventional-table-name trigger; closed stale without a fix.\n- [#58136](https://github.com/rails/rails/pull/58136): open multiline SQLite\n  parser work for virtual tables, generated columns, and collations; it does not\n  modify index introspection.\n\nTargeted issue/PR searches found no direct report of this newline trigger as of\n2026-07-22. Full query details and exact states are in [SOURCES.md](SOURCES.md).\n\n## Repository map\n\n- [`repro.rb`](repro.rb): intentionally failing, public-API-first bug report.\n- [`issue_repro.rb`](issue_repro.rb): compact, edge-only version for the eventual\n  upstream issue body.\n- [`script/verify_bug.rb`](script/verify_bug.rb): green CI oracle for the exact\n  upstream failure signature.\n- [`fix_validation.rb`](fix_validation.rb): all-green candidate-fix validation.\n- [`script/trigger_matrix.rb`](script/trigger_matrix.rb): exact newline placement\n  characterization.\n- [`script/sqlite_storage_probe.rb`](script/sqlite_storage_probe.rb): storage and\n  structured-metadata probe across SQLite binding versions.\n- [`script/parser_comparison.rb`](script/parser_comparison.rb): proves unchanged\n  captures for Rails' existing single-line SQLite index corpus.\n- [`EVIDENCE.md`](EVIDENCE.md): full proof, corrections, causal chain, candidate\n  design, and explicit boundaries.\n- [`UPSTREAM_VALIDATION.md`](UPSTREAM_VALIDATION.md): exact native Rails\n  red/green commands, outputs, full-suite result, and process traceability.\n- [`SOURCES.md`](SOURCES.md): immutable source ledger for every material claim.\n- [`ISSUE_DRAFT.md`](ISSUE_DRAFT.md): preserved source for the filed\n  [upstream issue](https://github.com/rails/rails/issues/58200).\n- [`PR_DRAFT.md`](PR_DRAFT.md): preserved source for the filed\n  [upstream fix](https://github.com/rails/rails/pull/58201).\n- [`results/`](results): deterministic raw logs for releases, edge, candidate\n  validation, trigger mapping, and storage probes.\n- [`.github/workflows/ci.yml`](.github/workflows/ci.yml): public CI matrix.\n\n## Regenerate all checked-in evidence\n\n```bash\nruby script/capture_results.rb\n```\n\nThis rewrites the release/edge, fix, trigger, and SQLite storage logs only after\neach expected result is verified.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frameerez%2Frails-sqlite-partial-index-dumper-bug","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frameerez%2Frails-sqlite-partial-index-dumper-bug","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frameerez%2Frails-sqlite-partial-index-dumper-bug/lists"}