https://github.com/timelessnesses/1brc-timeless
one billion row challenge in rust by a shitty coder (me)
https://github.com/timelessnesses/1brc-timeless
Last synced: 10 months ago
JSON representation
one billion row challenge in rust by a shitty coder (me)
- Host: GitHub
- URL: https://github.com/timelessnesses/1brc-timeless
- Owner: timelessnesses
- License: wtfpl
- Created: 2024-06-22T04:19:45.000Z (almost 2 years ago)
- Default Branch: master
- Last Pushed: 2025-03-25T15:48:17.000Z (about 1 year ago)
- Last Synced: 2025-03-25T16:43:51.681Z (about 1 year ago)
- Language: Rust
- Size: 3.38 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# 1brc-timeless
This repository contains rust code that can be used with 1brc (One Billion Row Challenge) dataset. *how the fuck did java #1 did it in 1 seconds*
## Running
Get yourself a `measurements.txt` and just run `cargo run -r`. It will spit out progress on reading file and time it take to parse file and do computation and a `out.json` file
## Getting yourself a `measurements.txt`
Inside `scripts/` folders contains 2 versions of `create_measurements` scripts that you can use. Python implementation is stolen from [`1brc`](https://github.com/gunnarmorling/1brc) repository and there is Rust's port of the Python implementation with multi-threading for fast speeds.
Executing them is simply just
```sh
python scripts/create_measurements.py 1000000000
# or
cd scripts/create_measurements
cargo run -r -- 1000000000
```
## Perf
ran on my laptop (AMD Ryzen 7 7730U with 16 Gigabytes of RAM with 8 Cores and 16 Logical Processors) on Windows
| stuff | my implementation in rust | #1 java 1brc |
|---------------------- |--------------------------- |------------------------------ |
| RAM Usage (Peak) | 7 Megabytes | 80+ Megabytes |
| Time to process | 32 seconds (pretty close) | 56 seconds |
| Peak Disk Activities | 460+ MB/s | 280+ MB/s |
| Avg Disk Activites | 230-420+ MB/S | 230-280+ MB/S
## Special Thanks
- [0/0](https://github.com/tazz4843)
- acrimon (discord)
- erk (discord)
- and everyone else in Serenity Discord Bot Library Guild!