{"id":19591860,"url":"https://github.com/ralgond/sortedlistintersection","last_synced_at":"2025-07-17T01:33:55.831Z","repository":{"id":178252816,"uuid":"326561659","full_name":"ralgond/SortedListIntersection","owner":"ralgond","description":"使用SIMD的有序列表求交集算法","archived":false,"fork":false,"pushed_at":"2021-01-17T09:26:41.000Z","size":1171,"stargazers_count":7,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-27T14:38:56.892Z","etag":null,"topics":["algorithm","intersection","simd","sorted-lists"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ralgond.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-04T03:42:52.000Z","updated_at":"2024-11-19T00:40:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"a494d606-04c1-4b40-bee8-f490556b4f1a","html_url":"https://github.com/ralgond/SortedListIntersection","commit_stats":null,"previous_names":["ralgond/sortedlistintersection"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ralgond/SortedListIntersection","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ralgond%2FSortedListIntersection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ralgond%2FSortedListIntersection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ralgond%2FSortedListIntersection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ralgond%2FSortedListIntersection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ralgond","download_url":"https://codeload.github.com/ralgond/SortedListIntersection/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ralgond%2FSortedListIntersection/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265558614,"owners_count":23787950,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["algorithm","intersection","simd","sorted-lists"],"created_at":"2024-11-11T08:31:20.754Z","updated_at":"2025-07-17T01:33:55.812Z","avatar_url":"https://github.com/ralgond.png","language":"C++","readme":"# SortedListIntersection\n有序列表求交集算法\n\n# 如何运行\n首先要确保你的CPU在指令集方面支持AVX2, AVX512F和SSE4.2的STTNI\n\nmake\n\n./main data/docid_list_s.txt data/docid_list_c.txt\n\n# 性能\n### CPU配置\nIntel(R) Xeon(R) Gold 6146 CPU @ 3.20GHz\n\n### 数据集1（数量为2w*2w，交集数为6）\n./main data/docid_list_s.txt data/docid_list_c.txt\n\n结果有4列，第一列是算法名，第二列是花费时间，第三列是QPS，第四列是交集大小。比如下面：\n```\nv1_avx2 185ms   54054.1 6\nv3_avx2 365ms   27397.3 6\nhighlyscalable_SIMD     157ms   63694.3 6\nhighlyscalable_avx2     161ms   62111.8 6\nhighlyscalable_avx512   243ms   41152.3 6\nlemire_highlyscalable_SIMD      179ms   55865.9 6\nscalar  136ms   73529.4 6\nblock_merge     488ms   20491.8 6\nsttni   199ms   50251.3 6\nbisearch\t4002ms\t2498.75\t6\nbloom\t3931ms\t2543.88\t6\n```\n可以看出scalar算法(最简单的算法)在众算法中最快，而bloom过滤器慢得出乎意料。\n\n### 数据集2（数量为10w*10w，交集数为8944）\n./main data/10w_1.txt data/10w_2.txt \n```\nv1_avx2 1889ms  5293.81 8944\nv3_avx2 3122ms  3203.07 8944\nhighlyscalable_SIMD     2053ms  4870.92 8944\nhighlyscalable_avx2     1169ms  8554.32 8944\nhighlyscalable_avx512   1413ms  7077.14 8944\nlemire_highlyscalable_SIMD      2188ms  4570.38 8944\nscalar  6890ms  1451.38 8944\nblock_merge     4341ms  2303.62 8944\nsttni   2298ms  4351.61 8944\n```\n可以看出highlyscalable_avx2最快，它是[highlyscalable_SIMD](http://highlyscalable.wordpress.com/2012/06/05/fast-intersection-sorted-lists-sse/)算法的升级版。\n\n### 数据集3（数量为10w*10w，交集数为956）\n./main data/10w_3.txt data/10w_4.txt\n```\nv1_avx2 1942ms  5149.33 956\nv3_avx2 2470ms  4048.58 956\nhighlyscalable_SIMD     2182ms  4582.95 956\nhighlyscalable_avx2     1278ms  7824.73 956\nhighlyscalable_avx512   1500ms  6666.67 956\nlemire_highlyscalable_SIMD      2325ms  4301.08 956\nscalar  6792ms  1472.32 956\nblock_merge     4299ms  2326.12 956\nsttni   1543ms  6480.88 956\n```\n结合数据集2的结果可以看出，除了基于sttni的算法，交集的大小对其他算法的性能影响很小。而对于基于sttni的算法，交集越小速度越快。\n\n### 数据集4（数量为100w*100w，交集数为9792）\n```\nv1_avx2 20172ms 495.737 9792\nv3_avx2 24989ms 400.176 9792\nhighlyscalable_SIMD     22872ms 437.216 9792\nhighlyscalable_avx2     13892ms 719.839 9792\nhighlyscalable_avx512   15337ms 652.018 9792\nlemire_highlyscalable_SIMD      24527ms 407.714 9792\nscalar  69943ms 142.974 9792\nblock_merge     44386ms 225.296 9792\nsttni   16300ms 613.497 9792\n```\n当数量级达到100w时，highlyscalable_avx2算法速度依然是最快的。\n\n### 结论\n1. 推荐使用highlyscalable_avx2算法，它简单、不依赖额外内存空间且速度最快。\n2. 使用和highlyscalable_avx2相同思路，用AVX512实现的算法highlyscalable_avx512的速度并不比highlyscalable_avx2快，反而是更慢了。\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fralgond%2Fsortedlistintersection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fralgond%2Fsortedlistintersection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fralgond%2Fsortedlistintersection/lists"}