{"id":19706436,"url":"https://github.com/superioone/1brc","last_synced_at":"2025-07-26T15:11:04.434Z","repository":{"id":246347319,"uuid":"820787422","full_name":"SuperioOne/1brc","owner":"SuperioOne","description":"One billion rows challenge with Rust","archived":false,"fork":false,"pushed_at":"2024-10-15T11:38:54.000Z","size":40,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-24T07:08:45.650Z","etag":null,"topics":["1brc","1brc-rs","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SuperioOne.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-27T07:18:00.000Z","updated_at":"2024-10-15T11:38:57.000Z","dependencies_parsed_at":"2024-06-27T09:55:40.296Z","dependency_job_id":"33fa7f5f-190a-4bdb-b73e-16603df60e7e","html_url":"https://github.com/SuperioOne/1brc","commit_stats":null,"previous_names":["superioone/1brc"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/SuperioOne/1brc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SuperioOne%2F1brc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SuperioOne%2F1brc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SuperioOne%2F1brc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SuperioOne%2F1brc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SuperioOne","download_url":"https://codeload.github.com/SuperioOne/1brc/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SuperioOne%2F1brc/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267184069,"owners_count":24049126,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-26T02:00:08.937Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["1brc","1brc-rs","rust"],"created_at":"2024-11-11T21:35:31.192Z","updated_at":"2025-07-26T15:11:04.386Z","avatar_url":"https://github.com/SuperioOne.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 1 Billion Rows Challenge - Rust\n\n## What's new\n\nI find the AVX2 line iterator pretty useful, and I decided to move it to my own [algorithms repo](https://github.com/SuperioOne/algorithms) for future uses.\nAlso, it now has a fallback algorithm based on SWAR for other CPU architectures and older x86-64 CPUs.\n\nFor previous inlined AVX2 only version, see [`avx2_only` tag](https://github.com/SuperioOne/1brc/tree/avx2_only)\n\n## x86-64-v3 with AVX2 (0.95855 seconds)\n\nSIMD instructions, libc mmap, CityHash and Rust standard library.\n\nRUSTFLAGS: \"-O -Ctarget-cpu=x86-64-v3\"\n\n**Test Environment:**\n- Ryzen 9 5950x CPU (16C/32T)(x86-64-v3)\n- 64GB Memory\n- Evo 970 SSD (3400 MB/s Seq. Read)\n- Rust 1.80.1\n\nBest case scenario - measurements.txt is already cached on memory.\n```\n Performance counter stats for './target/release/challenge_1brc ./measurements.txt' (20 runs):\n\n        36,300,329      cache-misses                                                            ( +-  0.53% )  (83.23%)\n    97,810,561,440      cycles                           #    4.452 GHz                         ( +-  0.05% )  (83.25%)\n   190,712,123,166      instructions                     #    1.95  insn per cycle              ( +-  0.01% )  (83.29%)\n       542,035,366      branch-misses                                                           ( +-  0.04% )  (83.39%)\n    37,941,771,530      all_data_cache_accesses          #    1.727 G/sec                       ( +-  0.01% )  (83.53%)\n         21,969.19 msec cpu-clock                        #   22.919 CPUs utilized               ( +-  0.09% )\n       542,153,328      branch-misses                                                           ( +-  0.05% )  (83.51%)\n               144      context-switches                 #    6.555 /sec                        ( +-  3.24% )\n\n           # Table of individual measurements:\n           0.94001 (-0.01854) #\n           0.95140 (-0.00715) #\n           0.94972 (-0.00883) #\n           0.94955 (-0.00900) #\n           0.95507 (-0.00348) #\n           0.94923 (-0.00932) #\n           0.95880 (+0.00025) #\n           0.96161 (+0.00306) #\n           0.96030 (+0.00175) #\n           0.95484 (-0.00370) #\n           0.95875 (+0.00021) #\n           0.96139 (+0.00284) #\n           0.99126 (+0.03271) #\n           0.96414 (+0.00559) #\n           0.95875 (+0.00021) #\n           0.95207 (-0.00648) #\n           0.95972 (+0.00117) #\n           0.96678 (+0.00823) #\n           0.96704 (+0.00849) #\n           0.96054 (+0.00199) #\n\n           # Final result:\n           0.95855 +- 0.00228 seconds time elapsed  ( +-  0.24% )\n\n```\n\nWorst case scenario - Cold cache, nothing cached on memory.\n```\n Performance counter stats for './target/release/challenge_1brc ./measurements.txt':\n\n       277,784,366      cache-misses                                                            (82.66%)\n    83,212,123,901      cycles                           #    4.430 GHz                         (83.42%)\n   204,322,015,153      instructions                     #    2.46  insn per cycle              (83.69%)\n       826,555,011      branch-misses                                                           (83.75%)\n    44,637,535,477      all_data_cache_accesses          #    2.376 G/sec                       (82.61%)\n         18,783.41 msec cpu-clock                        #    4.619 CPUs utilized\n       827,751,960      branch-misses                                                           (83.90%)\n           100,820      context-switches                 #    5.368 K/sec\n\n       4.066922345 seconds time elapsed\n\n      14.087471000 seconds user\n       4.297025000 seconds sys\n```\n\n## x86-64 without any SIMD tricks (1.07852)\n\nlibc mmap, CityHash and Rust standard library.\n\nRUSTFLAGS: \"-O -Ctarget-cpu=x86-64\"\n\n**Test Environment:**\n- Ryzen 9 5950x CPU (16C/32T)\n- 64GB Memory\n- Evo 970 SSD (3400 MB/s Seq. Read)\n- Rust 1.80.1\n\nBest case scenario - measurements.txt is already cached on memory.\n```\n Performance counter stats for './target/release/challenge_1brc ./measurements.txt' (20 runs):\n\n        31,335,865      cache-misses                                                            ( +-  0.44% )  (83.26%)\n   114,884,685,620      cycles                           #    4.446 GHz                         ( +-  0.06% )  (83.20%)\n   230,630,440,560      instructions                     #    2.01  insn per cycle              ( +-  0.01% )  (83.28%)\n       783,966,172      branch-misses                                                           ( +-  0.01% )  (83.40%)\n    44,204,291,969      all_data_cache_accesses          #    1.711 G/sec                       ( +-  0.02% )  (83.48%)\n         25,840.46 msec cpu-clock                        #   23.959 CPUs utilized               ( +-  0.09% )\n       784,057,687      branch-misses                                                           ( +-  0.01% )  (83.56%)\n               160      context-switches                 #    6.192 /sec                        ( +-  3.40% )\n\n           # Table of individual measurements:\n           1.06180 (-0.01671) #\n           1.08275 (+0.00423) #\n           1.06213 (-0.01638) #\n           1.07163 (-0.00689) #\n           1.07277 (-0.00575) #\n           1.06830 (-0.01022) #\n           1.06949 (-0.00902) #\n           1.07045 (-0.00807) #\n           1.06786 (-0.01066) #\n           1.07656 (-0.00195) #\n           1.07785 (-0.00066) #\n           1.07309 (-0.00543) #\n           1.07582 (-0.00269) #\n           1.07799 (-0.00052) #\n           1.07297 (-0.00555) #\n           1.11058 (+0.03207) #\n           1.09281 (+0.01430) #\n           1.12555 (+0.04703) #\n           1.07851 (-0.00000) #\n           1.08138 (+0.00287) #\n\n           # Final result:\n           1.07852 +- 0.00345 seconds time elapsed  ( +-  0.32% )\n```\n\nWorst case scenario - Cold cache, nothing cached on memory.\n```\n Performance counter stats for './target/release/challenge_1brc ./measurements.txt':\n\n       274,598,290      cache-misses                                                            (82.63%)\n    98,389,092,251      cycles                           #    4.438 GHz                         (82.68%)\n   244,232,067,759      instructions                     #    2.48  insn per cycle              (83.54%)\n     1,074,358,436      branch-misses                                                           (84.12%)\n    53,795,941,392      all_data_cache_accesses          #    2.426 G/sec                       (83.44%)\n         22,171.22 msec cpu-clock                        #    5.438 CPUs utilized\n     1,074,669,965      branch-misses                                                           (83.62%)\n            99,908      context-switches                 #    4.506 K/sec\n\n       4.076780812 seconds time elapsed\n\n      17.503729000 seconds user\n       4.241538000 seconds sys\n```\n\n## Testing on your machine\n\n1. Generate `measurements.txt`. See [https://github.com/gunnarmorling/1brc](https://github.com/gunnarmorling/1brc)\n2. Set `RUSTFLAGS` ENV variable to `-Ctarget-cpu=native`. Linux example; `export RUSTFLAGS=\"-Ctarget-cpu=native\"`\n3. Run cargo release build by `cargo build --release`.\n4. Test it. `./target/release/challenge_1brc \u003cPATH_TO_MEASUREMENTS_FILE\u003e`\n\nIf you want to use Linux only `./perf_measure.sh` script to generate stats:\n\n5. Audit the script, do not run random scripts from internet. It's only 15 lines.\n6. Make sure Linux `perf` tool is installed.\n7. Run the script `sudo ./perf_measure.sh \u003cPATH_TO_MEASUREMENTS_FILE\u003e` \n\n\u003e Dropping memory caches and performance counters requires `sudo` privilages.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuperioone%2F1brc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsuperioone%2F1brc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuperioone%2F1brc/lists"}