{"id":20812043,"url":"https://github.com/kanidm/idlset","last_synced_at":"2026-03-15T11:43:16.475Z","repository":{"id":43037016,"uuid":"153976977","full_name":"kanidm/idlset","owner":"kanidm","description":"IDLSet - Fast u64 integer set operations","archived":false,"fork":false,"pushed_at":"2024-05-30T05:44:22.000Z","size":213,"stargazers_count":13,"open_issues_count":4,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-27T02:38:33.327Z","etag":null,"topics":["database","indexing"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kanidm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-10-21T05:36:26.000Z","updated_at":"2025-05-30T09:57:40.000Z","dependencies_parsed_at":"2025-05-07T10:12:13.146Z","dependency_job_id":"8f2f0cbe-7fde-40fa-83fc-291b94c2fa93","html_url":"https://github.com/kanidm/idlset","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/kanidm/idlset","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanidm%2Fidlset","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanidm%2Fidlset/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanidm%2Fidlset/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanidm%2Fidlset/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kanidm","download_url":"https://codeload.github.com/kanidm/idlset/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanidm%2Fidlset/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268675515,"owners_count":24288285,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-04T02:00:09.867Z","response_time":79,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","indexing"],"created_at":"2024-11-17T20:49:56.188Z","updated_at":"2026-03-15T11:43:16.439Z","avatar_url":"https://github.com/kanidm.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"IDLSet\n======\n\nIDLSet - Fast u64 integer set operations\n\nIDLSet is a specialised library for fast logical set operations on\nu64. For example, this means union (or), intersection (and) and not\noperations on sets. In the best case, speed ups of 15x have been observed\nwith the general case performing approximately 4x faster that a Vec\u003cu64\u003e\nbased implementation.\n\nThese operations are heavily used in low-level implementations of databases\nfor their indexing logic, but has applications with statistical analysis and\nother domains that require logical set operations.\n\nHow Does It Work?\n-----------------\n\nEach set initially is \"sparse\". This is stored in the manner you expect historically,\nusing a `Vec\u003cu64\u003e` internally.\n\n::\n\n    [ 0, 1, 2, 3, ... , 1024 ]\n\nYou can then call `maybe_compress` on the set, which will look at the content and determine\nif it would be beneficial to compress this. When compressed, each value is transformed into\na tuple pair of `range` and `mask`. The range represents the starting value of this set of\n64 values, and the mask determines if a value of that range is present. For example:\n\n\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"https://raw.githubusercontent.com/Firstyear/idlset/master/static/idl_4.png\" width=\"60%\" height=\"auto\" /\u003e\n\u003c/p\u003e\n\nAs these now contain a bit mask, we can use CPU operations for logical operations like `AND`, `OR` and\n`AND NOT`. This example demonstrates an `AND` operation.\n\n\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"https://raw.githubusercontent.com/Firstyear/idlset/master/static/idl_5.png\" width=\"60%\" height=\"auto\" /\u003e\n\u003c/p\u003e\n\nDue to this compression, on high density sets, memory is reduced, as is improvements to CPU cache\nbehaviour due to lower pressure on the caches. It also allows faster seeking through sets to determine\nvalue presence.\n\n\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"https://raw.githubusercontent.com/Firstyear/idlset/master/static/idl_6.png\" width=\"60%\" height=\"auto\" /\u003e\n\u003c/p\u003e\n\nDuring operations between compressed and uncompressed sets, the \"better\" choice of compressed or\nuncompressed is preserved for the result set based on the inputs and operation performed.\nIn other words, the result set may be compressed or uncompressed\ndepending on the operation and it's interactions, to improve performance of subsequent operations.\nThis helps to carry forward these optimisation choices to result sets meaning that chained and\nmany operations over sets, and reduces memory consumption of intermediate set results during\noperations.\n\nContributing\n------------\n\nPlease open an issue, pr or contact me directly by email (see github)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkanidm%2Fidlset","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkanidm%2Fidlset","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkanidm%2Fidlset/lists"}