{"id":13879812,"url":"https://github.com/ankane/blind_index","last_synced_at":"2025-11-17T14:03:23.493Z","repository":{"id":37819168,"uuid":"114589573","full_name":"ankane/blind_index","owner":"ankane","description":"Securely search encrypted database fields","archived":false,"fork":false,"pushed_at":"2025-10-22T05:14:04.000Z","size":259,"stargazers_count":699,"open_issues_count":1,"forks_count":27,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-11-14T02:17:25.472Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ankane.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2017-12-18T02:59:36.000Z","updated_at":"2025-11-12T13:07:19.000Z","dependencies_parsed_at":"2023-01-31T00:30:48.031Z","dependency_job_id":"f33dae45-ddf3-4fea-9ba4-9f2401f8d130","html_url":"https://github.com/ankane/blind_index","commit_stats":{"total_commits":289,"total_committers":5,"mean_commits":57.8,"dds":0.01384083044982698,"last_synced_commit":"342605bba2c148d1ed86303f347a3106a9bf3977"},"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"purl":"pkg:github/ankane/blind_index","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fblind_index","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fblind_index/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fblind_index/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fblind_index/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ankane","download_url":"https://codeload.github.com/ankane/blind_index/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fblind_index/sbom","scorecard":{"id":197141,"data":{"date":"2025-08-11","repo":{"name":"github.com/ankane/blind_index","commit":"986cffd084a3df01273d789a6c5d44f8e4d408aa"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.4,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/build.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build.yml:25: update your workflow using https://app.stepsecurity.io/secureworkflow/ankane/blind_index/build.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/build.yml:26: update your workflow using https://app.stepsecurity.io/secureworkflow/ankane/blind_index/build.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/build.yml:31: update your workflow using https://app.stepsecurity.io/secureworkflow/ankane/blind_index/build.yml/master?enable=pin","Info:   0 out of   1 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   2 third-party GitHubAction dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE.txt:0","Info: FSF or OSI recognized license: MIT License: LICENSE.txt:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-16T22:01:00.884Z","repository_id":37819168,"created_at":"2025-08-16T22:01:00.884Z","updated_at":"2025-08-16T22:01:00.884Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":284572988,"owners_count":27028276,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-15T02:00:06.050Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-06T08:02:34.244Z","updated_at":"2025-11-17T14:03:23.475Z","avatar_url":"https://github.com/ankane.png","language":"Ruby","funding_links":[],"categories":["Ruby","Gems","Encryption"],"sub_categories":["Caching and Indexing"],"readme":"# Blind Index\n\nSecurely search encrypted database fields\n\nWorks with [Lockbox](https://github.com/ankane/lockbox) ([full example](https://ankane.org/securing-user-emails-lockbox)) and [attr_encrypted](https://github.com/attr-encrypted/attr_encrypted) ([full example](https://ankane.org/securing-user-emails-in-rails))\n\nLearn more about [securing sensitive data in Rails](https://ankane.org/sensitive-data-rails)\n\n[![Build Status](https://github.com/ankane/blind_index/actions/workflows/build.yml/badge.svg)](https://github.com/ankane/blind_index/actions)\n\n## How It Works\n\nWe use [this approach](https://paragonie.com/blog/2017/05/building-searchable-encrypted-databases-with-php-and-sql) by Scott Arciszewski. To summarize, we compute a keyed hash of the sensitive data and store it in a column. To query, we apply the keyed hash function to the value we’re searching and then perform a database search. This results in performant queries for exact matches. Efficient `LIKE` queries are [not possible](#like-ilike-and-full-text-searching), but you can index expressions.\n\n## Leakage\n\nAn important consideration in searchable encryption is leakage, which is information an attacker can gain. Blind indexing leaks that rows have the same value. If you use this for a field like last name, an attacker can use frequency analysis to predict the values. In an active attack where an attacker can control the input values, they can learn which other values in the database match.\n\nHere’s a [great article](https://blog.cryptographyengineering.com/2019/02/11/attack-of-the-week-searchable-encryption-and-the-ever-expanding-leakage-function/) on leakage in searchable encryption. Blind indexing has the same leakage as [deterministic encryption](#alternatives).\n\n## Installation\n\nAdd this line to your application’s Gemfile:\n\n```ruby\ngem \"blind_index\"\n```\n\n## Prep\n\nYour model should already be set up with Lockbox or attr_encrypted. The examples are for a `User` model with `has_encrypted :email` or `attr_encrypted :email`. See the full examples for [Lockbox](https://ankane.org/securing-user-emails-lockbox) and [attr_encrypted](https://ankane.org/securing-user-emails-in-rails) if needed.\n\nAlso, if you use attr_encrypted, [generate a key](#key-generation).\n\n## Getting Started\n\nCreate a migration to add a column for the blind index\n\n```ruby\nadd_column :users, :email_bidx, :string\nadd_index :users, :email_bidx # unique: true if needed\n```\n\nAdd to your model\n\n```ruby\nclass User \u003c ApplicationRecord\n  blind_index :email\nend\n```\n\nFor more sensitive fields, use\n\n```ruby\nclass User \u003c ApplicationRecord\n  blind_index :email, slow: true\nend\n```\n\nBackfill existing records\n\n```ruby\nBlindIndex.backfill(User)\n```\n\nAnd query away\n\n```ruby\nUser.where(email: \"test@example.org\")\n```\n\n## Expressions\n\nYou can apply expressions to attributes before indexing and searching. This gives you the the ability to perform case-insensitive searches and more.\n\n```ruby\nclass User \u003c ApplicationRecord\n  blind_index :email, expression: -\u003e(v) { v.downcase }\nend\n```\n\n## Validations\n\nYou can use blind indexes for uniqueness validations.\n\n```ruby\nclass User \u003c ApplicationRecord\n  validates :email, uniqueness: true\nend\n```\n\nWe recommend adding a unique index to the blind index column through a database migration.\n\n```ruby\nadd_index :users, :email_bidx, unique: true\n```\n\nFor `allow_blank: true`, use:\n\n```ruby\nclass User \u003c ApplicationRecord\n  blind_index :email, expression: -\u003e(v) { v.presence }\n  validates :email, uniqueness: {allow_blank: true}\nend\n```\n\nFor `case_sensitive: false`, use:\n\n```ruby\nclass User \u003c ApplicationRecord\n  blind_index :email, expression: -\u003e(v) { v.downcase }\n  validates :email, uniqueness: true # for best performance, leave out {case_sensitive: false}\nend\n```\n\n## Multiple Indexes\n\nYou may want multiple blind indexes for an attribute. To do this, add another column:\n\n```ruby\nadd_column :users, :email_ci_bidx, :string\nadd_index :users, :email_ci_bidx\n```\n\nUpdate your model\n\n```ruby\nclass User \u003c ApplicationRecord\n  blind_index :email\n  blind_index :email_ci, attribute: :email, expression: -\u003e(v) { v.downcase }\nend\n```\n\nBackfill existing records\n\n```ruby\nBlindIndex.backfill(User, columns: [:email_ci_bidx])\n```\n\nAnd query away\n\n```ruby\nUser.where(email_ci: \"test@example.org\")\n```\n\n## Index Only\n\nIf you don’t need to store the original value (for instance, when just checking duplicates), use a virtual attribute:\n\n```ruby\nclass User \u003c ApplicationRecord\n  attribute :email, :string\n  blind_index :email\nend\n```\n\n## Multiple Columns\n\nYou can also use virtual attributes to index data from multiple columns:\n\n```ruby\nclass User \u003c ApplicationRecord\n  attribute :initials, :string\n  blind_index :initials\n\n  before_validation :set_initials, if: -\u003e { changes.key?(:first_name) || changes.key?(:last_name) }\n\n  def set_initials\n    self.initials = \"#{first_name[0]}#{last_name[0]}\"\n  end\nend\n```\n\n## Migrating Data\n\nIf you’re encrypting a column and adding a blind index at the same time, use the `migrating` option.\n\n```ruby\nclass User \u003c ApplicationRecord\n  blind_index :email, migrating: true\nend\n```\n\nThis allows you to backfill records while still querying the unencrypted field.\n\n```ruby\nBlindIndex.backfill(User)\n```\n\nOnce that completes, you can remove the `migrating` option.\n\n## Key Rotation\n\nTo rotate keys without downtime, add a new column:\n\n```ruby\nadd_column :users, :email_bidx_v2, :string\nadd_index :users, :email_bidx_v2\n```\n\nAnd add to your model\n\n```ruby\nclass User \u003c ApplicationRecord\n  blind_index :email, rotate: {version: 2, master_key: ENV[\"BLIND_INDEX_MASTER_KEY_V2\"]}\nend\n```\n\nThis will keep the new column synced going forward. Next, backfill the data:\n\n```ruby\nBlindIndex.backfill(User, columns: [:email_bidx_v2])\n```\n\nThen update your model\n\n```ruby\nclass User \u003c ApplicationRecord\n  blind_index :email, version: 2, master_key: ENV[\"BLIND_INDEX_MASTER_KEY_V2\"]\nend\n```\n\nFinally, drop the old column.\n\n## Key Separation\n\nThe master key is used to generate unique keys for each blind index. This technique comes from [CipherSweet](https://ciphersweet.paragonie.com/internals/key-hierarchy). The table name and blind index column name are both used in this process.\n\nYou can get an individual key with:\n\n```ruby\nBlindIndex.index_key(table: \"users\", bidx_attribute: \"email_bidx\")\n```\n\nTo rename a table with blind indexes, use:\n\n```ruby\nclass User \u003c ApplicationRecord\n  blind_index :email, key_table: \"original_table\"\nend\n```\n\nTo rename a blind index column, use:\n\n```ruby\nclass User \u003c ApplicationRecord\n  blind_index :email, key_attribute: \"original_column\"\nend\n```\n\n## Algorithm\n\nArgon2id is used for best security. The default cost parameters are 3 iterations and 4 MB of memory. For `slow: true`, the cost parameters are 4 iterations and 32 MB of memory.\n\nA number of other algorithms are [also supported](docs/Other-Algorithms.md). Unless you have specific reasons to use them, go with Argon2id.\n\n## Fixtures\n\nYou can use blind indexes in fixtures with:\n\n```yml\ntest_user:\n  email_bidx: \u003c%= User.generate_email_bidx(\"test@example.org\").inspect %\u003e\n```\n\nBe sure to include the `inspect` at the end or it won’t be encoded properly in YAML.\n\n## Mongoid\n\nFor Mongoid, use:\n\n```ruby\nclass User\n  field :email_bidx, type: String\n  index({email_bidx: 1})\nend\n```\n\n## Key Generation\n\nThis is optional for Lockbox, as its master key is used by default.\n\nGenerate a key with:\n\n```ruby\nBlindIndex.generate_key\n```\n\nStore the key with your other secrets. This is typically Rails credentials or an environment variable ([dotenv](https://github.com/bkeepers/dotenv) is great for this). Be sure to use different keys in development and production. Keys don’t need to be hex-encoded, but it’s often easier to store them this way.\n\nSet the following environment variable with your key (you can use this one in development)\n\n```sh\nBLIND_INDEX_MASTER_KEY=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\n```\n\nor create `config/initializers/blind_index.rb` with something like\n\n```ruby\nBlindIndex.master_key = Rails.application.credentials.blind_index_master_key\n```\n\n## LIKE, ILIKE, and Full-Text Searching\n\nUnfortunately, blind indexes can’t be used for `LIKE`, `ILIKE`, or full-text searching. Instead, records must be loaded, decrypted, and searched in memory.\n\nFor `LIKE`, use:\n\n```ruby\nUser.select { |u| u.email.include?(\"value\") }\n```\n\nFor `ILIKE`, use:\n\n```ruby\nUser.select { |u| u.email =~ /value/i }\n```\n\nFor full-text or fuzzy searching, use a gem like [FuzzyMatch](https://github.com/seamusabshere/fuzzy_match):\n\n```ruby\nFuzzyMatch.new(User.all, read: :email).find(\"value\")\n```\n\nIf the number of records is large, try to find a way to narrow it down. An [expression index](#expressions) is one way to do this, but leaks which records have the same value of the expression, so use it carefully.\n\n## Reference\n\nSet default options in an initializer with:\n\n```ruby\nBlindIndex.default_options = {algorithm: :pbkdf2_sha256}\n```\n\nBy default, blind indexes are encoded in Base64. Set a different encoding with:\n\n```ruby\nclass User \u003c ApplicationRecord\n  blind_index :email, encode: -\u003e(v) { [v].pack(\"H*\") }\nend\n```\n\nBy default, blind indexes are 32 bytes. Set a smaller size with:\n\n```ruby\nclass User \u003c ApplicationRecord\n  blind_index :email, size: 16\nend\n```\n\nSet a key directly for an index with:\n\n```ruby\nclass User \u003c ApplicationRecord\n  blind_index :email, key: ENV[\"USER_EMAIL_BLIND_INDEX_KEY\"]\nend\n```\n\n## Compatibility\n\nYou can generate blind indexes from other languages as well. For Python, you can use [argon2-cffi](https://github.com/hynek/argon2-cffi).\n\n```python\nfrom argon2.low_level import Type, hash_secret_raw\nfrom base64 import b64encode\n\nkey = '289737bab72fa97b1f4b081cef00d7b7d75034bcf3183c363feaf3e6441777bc'\nvalue = 'test@example.org'\n\nbidx = b64encode(hash_secret_raw(\n    secret=value.encode(),\n    salt=bytes.fromhex(key),\n    time_cost=3,\n    memory_cost=2**12,\n    parallelism=1,\n    hash_len=32,\n    type=Type.ID\n))\n```\n\n## Alternatives\n\nOne alternative to blind indexing is to use a deterministic encryption scheme, like [AES-SIV](https://github.com/miscreant/miscreant). In this approach, the encrypted data will be the same for matches. We recommend blind indexing over deterministic encryption because:\n\n1. You can keep encryption consistent for all fields (both searchable and non-searchable)\n2. Blind indexing supports expressions\n\n## History\n\nView the [changelog](https://github.com/ankane/blind_index/blob/master/CHANGELOG.md)\n\n## Contributing\n\nEveryone is encouraged to help improve this project. Here are a few ways you can help:\n\n- [Report bugs](https://github.com/ankane/blind_index/issues)\n- Fix bugs and [submit pull requests](https://github.com/ankane/blind_index/pulls)\n- Write, clarify, or fix documentation\n- Suggest or add new features\n\nTo get started with development and testing:\n\n```sh\ngit clone https://github.com/ankane/blind_index.git\ncd blind_index\nbundle install\nbundle exec rake test\n```\n\nFor security issues, send an email to the address on [this page](https://github.com/ankane).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fankane%2Fblind_index","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fankane%2Fblind_index","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fankane%2Fblind_index/lists"}