{"id":21064052,"url":"https://github.com/luochen1990/top-count","last_synced_at":"2025-03-14T01:24:56.844Z","repository":{"id":71923195,"uuid":"177961793","full_name":"luochen1990/top-count","owner":"luochen1990","description":"Find the k most frequently appearing items in a big file","archived":false,"fork":false,"pushed_at":"2019-10-08T10:34:48.000Z","size":49,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-20T20:49:52.123Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/luochen1990.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-27T09:32:54.000Z","updated_at":"2019-10-08T10:34:50.000Z","dependencies_parsed_at":"2023-03-11T11:36:01.929Z","dependency_job_id":null,"html_url":"https://github.com/luochen1990/top-count","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luochen1990%2Ftop-count","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luochen1990%2Ftop-count/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luochen1990%2Ftop-count/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luochen1990%2Ftop-count/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/luochen1990","download_url":"https://codeload.github.com/luochen1990/top-count/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243506833,"owners_count":20301760,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T17:48:08.545Z","updated_at":"2025-03-14T01:24:56.814Z","avatar_url":"https://github.com/luochen1990.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"Top Count\n=========\n\n编译:\n\n```\nstack build\n```\n\n安装可执行文件:\n\n```\nstack install\n```\n\n可执行文件:\n\n- gen-test-file: 生成测试文件, 如 `u100` 表示生成100个URL, 文件名形如 `u100.in`\n- ext-sort: 外部排序, 输入文件名, 输出结果会写到相应文件, 文件名形如 `u100.in.sort.out`\n- top-count: 找出现次数最多的100项, 输入文件名, 输出结果写到文件, 文件名形如 `u100.in.top-count.out`\n\nTips:\n\n- 命令行参数后面加 `+RTS -s` 可以查看用时及性能分析数据, 如 `top-count u100.in +RTS -s`\n- 命令行参数后面加 `+RTS -Nx` 可以指定CPU核心数, 如 `top-count u100.in +RTS -s -N4`\n\n运行单元测试:\n\n```\nstack test\n```\n\n复杂度分析\n----------\n\n设文件共有 `n` 行, 每行平均大小为 `l`, 需要取出现次数最多的 Top `m` 项.\n记 `k` 为所有行的去重计数, 即不同行的总数量 (该值可能无法预先知道).\n\n另有以下记号:\n\n- `Time` 表示CPU时间代价\n- `Mem` 表示内存峰值\n- `Read` 表示连续读盘代价\n- `Read?` 表示不连续读盘代价, 通过适当的缓存策略可优化为连续读盘代价\n- `Write` 表示写盘代价\n- `X + Y` 先后发生的资源占用\n- `X | Y` 同时发生的对不同设备时间的占用, 在有适当缓存策略的情况下可以重叠, 此时两者取最大值.\n\n### 阶段1\n\n- 分块读取所有数据到内存 `(n * l) Read` (其中 `n` 为总行数, `l` 为每行的平均大小)\n- 对每块进行内存中排序 `(b * log b * l * a) Time; (b * l) Mem` (其中 `b` 为分块大小, `a` 为块数量, 可知 `n = a * b`)\n- 将每块的排序结果落盘 `(n * l) Write`\n\n小计: `( (n * l) * (Read + Write) | (b * log b * l * a) Time ) ; (b * l) Mem`\n\n### 阶段2\n\n- 对已排序块执行合并有序流 (会有一次读盘) 并进行连续项计数 `(n * l) Read? | (n * log a * l) Time`\n- 对计数结果 (设记录数为 `k`, 按 `kb` 分块, 一共 `ka` 块) 进行分块并在内存中排序 `(kb * log kb * l * ka) Time ; (kb * l) Mem`\n- 将每块的排序结果落盘 `(k * l) Write`\n\n小计: `((n * l) Read? + (k * l) Write) | (n * log a * l + kb * log kb * l * ka) Time ; (kb * l) Mem`\n\n### 阶段3\n\n- 对已排序块执行合并有序流 (会有一次读盘) 并取 Top m 项 `(m * l) Read? + (m * log ka * l) Time`\n- 将最后结果落盘 `(m * l) Write`\n\n小计: `(m * l) * (Read? + Write) | (m * log ka * l) Time`\n\n### 总计\n\n#### 内存\n\n内存占用主要取决于两次外排序的分块大小, 忽略常数为 `b * l`\n\n#### 时间\n\n```\n( (n * l) * (Read + Write) | (b * log b * l * a) Time )\n+\n( ( (n * l) Read? + (k * l) Write ) | (n * log a * l + kb * log kb * l * ka) Time )\n+\n( (m * l) * (Read? + Write) | (m * log ka * l) Time )\n```\n\n若假设磁盘时间远大于CPU时间, 则以上结果可简化为\n\n```\n(n * l) * (Read + Read? + Write) + (k * l) Write + (m * l) * (Read? + Write)\n```\n\n其中 `n \u003e= k \u003e= m`\n\n极端情况下有 `n = k = m`, 则以上可简化为\n\n```\n(n * l) * (Read + Read? * 2 + Write * 3)\n```\n\n较好情况下, 若数据中存在大量重复项, 即有 `n \u003e\u003e k` (n 远大于 k), 则以上可简化为\n\n```\n(n * l) * (Read + Read? + Write)\n```\n\n一般情况下有 `n ~= k \u003e\u003e m` (k 与 n 接近, 并远大于 m), 则以上可简化为\n\n```\n(n * l) * (Read + Read? + Write * 2)\n```\n\n该算法实现及优化的关键是:\n\n- 使用流式操作, 确保数据不会堆积在内存中导致OOM\n- 尽量对多个阶段进行融合, 减少磁盘读写次数\n- 确保用适当的缓存策略将磁盘随机读 (Read?) 优化到磁盘连续读 (Read)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluochen1990%2Ftop-count","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluochen1990%2Ftop-count","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluochen1990%2Ftop-count/lists"}