{"id":22109209,"url":"https://github.com/kyagara/1brc","last_synced_at":"2025-03-24T03:46:34.776Z","repository":{"id":229920333,"uuid":"778012492","full_name":"Kyagara/1brc","owner":"Kyagara","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-01T19:53:27.000Z","size":473,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-29T10:16:57.600Z","etag":null,"topics":["1brc","1brc-go"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Kyagara.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-26T23:08:37.000Z","updated_at":"2024-06-11T11:31:43.000Z","dependencies_parsed_at":"2024-12-02T07:55:18.548Z","dependency_job_id":null,"html_url":"https://github.com/Kyagara/1brc","commit_stats":null,"previous_names":["kyagara/1brc"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kyagara%2F1brc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kyagara%2F1brc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kyagara%2F1brc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kyagara%2F1brc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Kyagara","download_url":"https://codeload.github.com/Kyagara/1brc/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245206534,"owners_count":20577582,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["1brc","1brc-go"],"created_at":"2024-12-01T09:29:51.018Z","updated_at":"2025-03-24T03:46:34.740Z","avatar_url":"https://github.com/Kyagara.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 1brc\n\nHaving fun with the [1brc](https://github.com/gunnarmorling/1brc) challenge in go. Im not attempting to upload this somewhere, just trying different things and see what I can do.\n\n~Development and benchmarking on Windows because I don't want to accidentaly kill my WSL instance with a OOM that is bound to occur at some point.~ I hate Windows.\n\nTests are not currently being done to validate the correctness of the solutions.\n\nOnly the `README.md` from the main branch will be up-to-date.\n\n## Challenge excerpts\n\nCopy and paste of some of the rules and how the output should look like.\n\n### Rules and limits\n\n- No external library dependencies may be used\n\n- The computation must happen at application _runtime_, i.e. you cannot process the measurements file at _build time_ and just bake the result into the binary\n\n- Input value ranges are as follows:\n\n  - Station name: non null UTF-8 string of min length 1 character and max length 100 bytes, containing neither `;` nor `\\n` character (i.e. this could be 100 one-byte characters, or 50 two-byte characters, etc.)\n  - Temperature value: non null double between -99.9 (inclusive) and 99.9 (inclusive), always with one fractional digit\n\n- Implementations must not rely on specifics of a given data set, e.g. any valid station name as per the constraints above and any data distribution (number of measurements per station) must be supported\n\n- The rounding of output values must be done using the semantics of IEEE 754 rounding-direction \"roundTowardPositive\"\n\n### Output\n\nThe task is to write a ~Java~ go program which reads the file, calculates the min, mean, and max temperature value per weather station, and emits the results on stdout like this (i.e. sorted alphabetically by station name, and the result values per station in the format \u003cmin\u003e/\u003cmean\u003e/\u003cmax\u003e, rounded to one fractional digit):\n\n```\n{Abha=-23.0/18.0/59.2, Abidjan=-16.2/26.0/67.3, Abéché=-10.0/29.4/69.0, Accra=-10.1/26.4/66.4, Addis Ababa=-23.7/16.0/67.0, Adelaide=-27.8/17.3/58.5, ...}\n```\n\n## Commands\n\nSaving some commands I use here.\n\n```bash\ngo test ./calculate -bench=Benchmark100M -benchtime=1x -benchmem -memprofile mem.out -cpuprofile cpu.out # create pprof profiles\ngo tool pprof -http :3000 cpu.out\ngo run . path # add '-s' to print only stats, '-d' to print the output and stats\n```\n\n## Benchmark Results\n\nBenchmarking the process of reading, calculating and sorting the data, basically, everything that happens inside the `calculate.Run()` function.\n\nAllocs in the new way of getting results (just using runtime package) from v2 and above is not a 1:1 from the old way, I don't know how to get the same allocs/op from the go benchmark with the `-benchtime=1x -benchmem` flags.\n\nV2 and before had `-memprofile mem.out -cpuprofile cpu.out` flags set in the benchmark command, usually adding around 40 allocs/op.\n\nThere was a 1gb buffer set before v2.1 when reading the file which is why the memory usage up until v2.1 are pretty similar.\n\n### v4.1\n\nUsing more for ranges. Forgot to use 1 in AppendFloat for the output at the Min and Max fields. Merging read and process functions, this improved my results by around 2s.\n\nFor now this is it, bottlenecks looks like to be the hashmap (cache miss when looking up a station) and the temperature calculation/parsing. A smaller struct and maybe a different approach on the hashmap itself might be the next step. There is a lack of unsafe, which is good thing for me.\n\nOther than those things and the use of the experimental mmap package, I am happy with the progress, learned more about go and other things and had a lot of fun optimizing.\n\n```\ngo build\n\n./brc ./data/1b.txt -d\n{Abha=-33.3/18.0/68.9, Abidjan=-20.6/26.0/73.5, ...}\nTime: 19.81s    Memory: 216mb   Stations: 413\nMallocs: 8460   Frees: 7154     GC cycles: 137\n\ntime ./brc ./data/1b.txt \u003e /dev/null\nreal    0m19.577s\nuser    1m9.453s\nsys     0m5.433s\n```\n\n### v4\n\nUsing goroutines and channels. Hashmap and process workers using shards to avoid using a lock. Changed buffer size, again, now at 2gb, works well for me.\n\nHow can I avoid creating a new buffer for every read? Increasing the buffer size helps with gc since there will be less allocations and less garbage to collect.\n\nI will probably have to start using unsafe, temperature conversion and updating the station is taking some cpu time.\n\n```\nTime: 22.85s Memory: 237mb Stations: 413\nMallocs: 6980 Frees: 6554 GC cycles: 121\n```\n\n### v3\n\nUsing mmap, from the golang experimental package, that is technically a external library, I don't care, fow now.\n\nCustom hash map, incomplete, theres collision issues, size had to be increased to avoid it, which is why the memory usage is 4mb higher.\n\nSome functions either moved or rewritten to accommodate the above, this will also help when adding goroutines and channels next version.\n\nThe writing of the results (from creation of the bytes.Buffer to copying it to stdout), is probably really slow, it works for now.\n\n```\nTime: 77.33s Memory: 10mb Stations: 413\nMallocs: 162 Frees: 16 GC cycles: 1\n```\n\n### v2.1\n\nAdded a len(stations) to stats. Read buffer is now a fixed 65kb size. Some micro optimizations to functions. Rework on readLines and hash.\n\nMore importantly, made new datasets, all have 413 unique stations, I will only include 1b benchmark results from now on.\n\nUsing Linux from now on.\n\n```\nTime: 97.12s    Memory: 6mb     Stations: 413\nMallocs: 125    Frees: 3        GC cycles: 0\n```\n\n### v2\n\nNo unsafe, for now. Dealing with min, mean and max in separated `Station` fields, instead of having a single []float32, that was a horrible idea. Station.Name is now a [100]byte. When using []byte, there usually needs to have a copy of the data to avoid garbage.\n\nFirst time 1B works!\n\nIncluding the old way of benchmark results and the new one.\n\n```\nResults for '1m':\nTime: 0.14s     System Memory: 10mb\nMallocs: 235    Frees: 7        GC cycles: 0\n\nResults for '100m':\nTime: 11.72s    System Memory: 987mb\nMallocs: 273    Frees: 45       GC cycles: 1\n\nResults for '1b':\nTime: 156.70s   System Memory: 987mb\nMallocs: 277    Frees: 52       GC cycles: 2\n\nBenchmark1M-16       144884800 ns/op    3257648 B/op 68 allocs/op\n0.355s\n\nBenchmark100M-16   12598715300 ns/op 1027298936 B/op 78 allocs/op\n12.805s\n\nBenchmark1B-16    168614731300 ns/op 1027350960 B/op 96 allocs/op\n168.857s\n```\n\n### v1.1\n\nI was, for some reason, sorting the temperatures. Calculating the min, mean and max is now done in a single loop. I also added a custom parseFloat32 function, no more anxiety.\n\n```\nBenchmark1M-16     140032600 ns/op   23945352 B/op 10068 allocs/op\n0.338s\n\nBenchmark100M-16 14905148400 ns/op 2613198056 B/op 73213 allocs/op\n15.203s\n\nBenchmark1B-can you guess?\n```\n\n### v1\n\nJesus christ. The amount of allocs upsets me, the conversion of []byte -\u003e string -\u003e float32 gives me anxiety and the fact that I used unsafe in my first attempt is a sign of things to come.\n\n```\nBenchmark1M-16     216147500 ns/op   23946216 B/op 10073 allocs/op\n0.339s\n\nBenchmark100M-16 24891124400 ns/op 2613286360 B/op 73236 allocs/op\n25.289s\n\ngo test ./calculate -bench=Benchmark1B -benchtime=1x -benchmem -memprofile mem.out -cpuprofile cpu.out\nruntime: VirtualAlloc of 524288 bytes failed with errno=1455\nfatal error: out of memory\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyagara%2F1brc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkyagara%2F1brc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyagara%2F1brc/lists"}