{"id":16572803,"url":"https://github.com/lemire/fastscancount","last_synced_at":"2025-03-23T14:31:01.250Z","repository":{"id":66054867,"uuid":"205433415","full_name":"lemire/fastscancount","owner":"lemire","description":"Fast implementations of the scancount algorithm: C++ header-only library","archived":false,"fork":false,"pushed_at":"2019-10-07T14:48:13.000Z","size":64,"stargazers_count":26,"open_issues_count":8,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-18T20:54:26.277Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lemire.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-08-30T18:05:35.000Z","updated_at":"2024-12-26T17:16:06.000Z","dependencies_parsed_at":"2023-07-09T12:47:49.754Z","dependency_job_id":null,"html_url":"https://github.com/lemire/fastscancount","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Ffastscancount","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Ffastscancount/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Ffastscancount/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Ffastscancount/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lemire","download_url":"https://codeload.github.com/lemire/fastscancount/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245115678,"owners_count":20563209,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T21:28:37.191Z","updated_at":"2025-03-23T14:31:00.951Z","avatar_url":"https://github.com/lemire.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fastscancount\nFast implementations of the scancount algorithm\n\n\nGiven a set of arrays of integers, we seek to identify \nall values that occur more than 'threshold' times. We do so using the\n'scancount' algorithm. It is assumed\nthat you have fewer than 256 arrays of integers and that the threshold is no larger than 254.\n\nWe are effectively providing optimized versions of the following function:\n\n```C++\nvoid scancount(std::vector\u003cstd::vector\u003cuint32_t\u003e\u003e \u0026data, std::vector\u003cuint32_t\u003e \u0026out, uint8_t threshold) {\n  std::fill(counters.begin(), counters.end(), 0);\n  out.clear();\n  for (size_t c = 0; c \u003c data.size(); c++) {\n    std::vector\u003cuint32_t\u003e \u0026v = data[c];\n    for (size_t i = 0; i \u003c v.size(); i++) {\n      counters[v[i]]++;\n    }\n  }\n  for (uint32_t i = 0; i \u003c counters.size(); i++) {\n    if (counters[i] \u003e threshold)\n      out.push_back(i);\n  }\n}\n```\n\nOur optimized versions assume that your arrays are made of sorted integers.\n\nThere are two headers, `fastscancount.h` uses plain C++ and should\nbe portable. It has one main function in the fastscancount namespace.\nWe always write the result to 'out'.\n\n```C++\nvoid fastscancount(std::vector\u003cstd::vector\u003cuint32_t\u003e\u003e \u0026data,\n    std::vector\u003cuint32_t\u003e \u0026out, uint8_t threshold)\n```\n\nThere is another header `fastscancount_avx2.h`\nwhich expects an x64 processor supporting the AVX2 instruction set.  \nIt has a similar function signature:\n\n```C++\nvoid fastscancount_avx2(std::vector\u003cstd::vector\u003cuint32_t\u003e\u003e \u0026data,\n    std::vector\u003cuint32_t\u003e \u0026out, uint8_t threshold)\n```\n\nThe AVX2 version assumes that you have fewer than 128 arrays of integers.\n\nBecause this library is made solely of headers, there is no\nneed for a build system.\n\n## Linux benchmark\n\nIf you have bare metal access to a Linux box, you can run cycle-accurate benchmarks.\n\n```\nmake\n./counter\n```\n\nSample output with GNU GCC 8.3:\n\n```\n$ ./counter\nGot 2497 hits\noptimized cache-sensitive scancount\n4.01381 cycles/element\nAVX2-based scancount\n3.58494 cycles/element\n```\n\nWith LLVM clang, we seem to get better results:\n\n```\n$ ./counter\nGot 2497 hits\noptimized cache-sensitive scancount\n3.54267 cycles/element\n2.8908 instructions/cycles\n0.0134279 miss/element\nAVX2-based scancount\n3.57374 cycles/element\n2.03391 instructions/cycles\n0.0109755 miss/element\n```\n\n## Blog post\n\n[How fast can scancount be?](http://lemire.me/blog/2019/08/30/how-fast-can-scancount-be/ )\n\n## Using actual data\n\n```\n./counter --postings data/postings.bin --queries data/queries.bin --threshold 3\n```\n\n## Credit\n\nThe AVX2 version was designed and implemented by Travis Downs.\nThe scalar version was designed and implemented by Daniel Lemire based on ideas by Nathan Kurz,  Travis Downs and others.\n\n## Reference\n\n\nOwen Kaser, Daniel Lemire, [Compressed bitmap indexes: beyond unions and intersections](https://arxiv.org/abs/1402.4466), Software: Practice and Experience 46 (2), 2016\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flemire%2Ffastscancount","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flemire%2Ffastscancount","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flemire%2Ffastscancount/lists"}