{"id":14991990,"url":"https://github.com/jinyus/related_post_gen","last_synced_at":"2025-06-23T19:09:35.374Z","repository":{"id":196248694,"uuid":"695311175","full_name":"jinyus/related_post_gen","owner":"jinyus","description":"Data Processing benchmark featuring Rust, Go, Swift, Zig, Julia etc.","archived":false,"fork":false,"pushed_at":"2025-05-28T04:25:19.000Z","size":12136,"stargazers_count":358,"open_issues_count":25,"forks_count":102,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-05-28T04:49:53.216Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jinyus.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-22T20:39:07.000Z","updated_at":"2025-05-28T04:25:23.000Z","dependencies_parsed_at":"2023-10-10T14:29:18.965Z","dependency_job_id":"1a89db22-0018-49ce-873f-b81294ba73b6","html_url":"https://github.com/jinyus/related_post_gen","commit_stats":null,"previous_names":["jinyus/related_post_gen"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jinyus/related_post_gen","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jinyus%2Frelated_post_gen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jinyus%2Frelated_post_gen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jinyus%2Frelated_post_gen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jinyus%2Frelated_post_gen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jinyus","download_url":"https://codeload.github.com/jinyus/related_post_gen/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jinyus%2Frelated_post_gen/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261539362,"owners_count":23174139,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T15:00:37.951Z","updated_at":"2025-06-23T19:09:30.348Z","avatar_url":"https://github.com/jinyus.png","language":"C++","funding_links":[],"categories":["Performance Benchmark"],"sub_categories":[],"readme":"### Problem:\n\nGiven a list of posts, compute the top 5 related posts for each post based on the number of shared tags.\n\n\u003cdetails\u003e\n\u003csummary\u003e Steps \u003c/summary\u003e\n\n- Read the posts JSON file.\n- Iterate over the posts and populate a map containing: `tag -\u003e List\u003cint\u003e`, with the int representing the post index of each post with that tag.\n- Iterate over the posts and for each post:\n - Create a map: `PostIndex -\u003e int` to track the number of shared tags\n - For each tag, Iterate over the posts that have that tag\n - For each post, increment the shared tag count in the map.\n- Sort the related posts by the number of shared tags.\n- Write the top 5 related posts for each post to a new JSON file.\n\u003c/details\u003e\n\n### Run Benchmark\n\n```bash\n./run.sh go | rust | python | all\n\n# windows (powershell)\n./run.ps1 go | rust | python | all\n# OR\npwsh ./run.ps1 go | rust | python | all\n\n# Docker (check the dockerfile for available variables)\ndocker build -t databench .\n# OR\ndocker pull ghcr.io/jinyus/databench:latest\n# THEN\ndocker run -e TEST_NAME=all -it --rm databench\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e Rules \u003c/summary\u003e\n\n\u003ch3\u003eNo:\u003c/h3\u003e\n\n- FFI (including assembly inlining)\n- Unsafe code blocks\n- Custom benchmarking\n- Disabling runtime checks (bounds etc)\n- Specific hardware targeting\n- SIMD for single threaded solutions\n- Hardcoding number of posts\n- Lazy evaluation (Unless results are computed at runtime and timed)\n- Computation Caching\n\n\u003ch3\u003eMust:\u003c/h3\u003e\n\n- Support up to 100,000 posts\n- Support UTF8 strings\n- Parse json at runtime\n- Support up to 100 tags\n- Represent tags as strings\n- Be production ready\n- Use less than 8GB of memory\n\u003c/details\u003e\n\n### Updated Results from github workflow ([raw data](https://github.com/jinyus/related_post_gen/blob/main/raw_results.md))\n\n##### VM Specs ( AWS c7a.xlarge-4vCPU-8GB-RAM-Ubuntu 22.04 )\n\n| Language | Time (5k posts) | 20k posts | 60k posts | Total |\n| ------------------ | ------------------------------------- | -------------------------------------- | -------------------------------------- | --------- |\n| _Julia HO_[^1] | 5.80 ms | 20.00 ms | 89.33 ms | 115.13 ms |\n| D (v2) | $\\textsf{\\color{lightgreen}11.73 ms}$ | $\\textsf{\\color{lightgreen}123.60 ms}$ | $\\textsf{\\color{lightgreen}972.19 ms}$ | 1.11 s |\n| C++ | 13.80 ms | 170.67 ms | 1.45 s | 1.63 s |\n| Rust | 14.18 ms | 193.88 ms | 1.68 s | 1.89 s |\n| Zig | 14.00 ms | 195.67 ms | 1.68 s | 1.89 s |\n| D | 18.44 ms | 245.60 ms | 2.09 s | 2.35 s |\n| Go | 16.87 ms | 248.89 ms | 2.15 s | 2.42 s |\n| Neat | 19.21 ms | 260.15 ms | 2.14 s | 2.42 s |\n| Java (JIT) | 25.60 ms | 260.33 ms | 2.20 s | 2.48 s |\n| C# (AOT) | 18.81 ms | 267.93 ms | 2.35 s | 2.63 s |\n| C# (JIT) | 19.10 ms | 270.89 ms | 2.34 s | 2.63 s |\n| Nim | 19.39 ms | 291.69 ms | 2.52 s | 2.84 s |\n| F# (AOT) | 24.78 ms | 366.04 ms | 3.23 s | 3.62 s |\n| Julia | 27.37 ms | 407.72 ms | 3.62 s | 4.05 s |\n| Java (GraalVM) | 28.10 ms | 416.00 ms | 3.63 s | 4.07 s |\n| Vlang | 28.81 ms | 429.55 ms | 3.74 s | 4.20 s |\n| F# (JIT) | 26.25 ms | 410.77 ms | 3.92 s | 4.36 s |\n| Swift | 34.58 ms | 496.38 ms | 4.17 s | 4.70 s |\n| Haskell | 36.30 ms | 508.33 ms | 4.42 s | 4.96 s |\n| Odin | 34.53 ms | 510.76 ms | 4.47 s | 5.02 s |\n| Crystal | 39.44 ms | 582.76 ms | 5.13 s | 5.75 s |\n| JS (Bun) | 64.60 ms | 763.33 ms | 6.32 s | 7.15 s |\n| LuaJIT | 63.53 ms | 824.24 ms | 6.72 s | 7.61 s |\n| Pypy | 61.62 ms | 748.57 ms | 6.83 s | 7.64 s |\n| Dart VM | 50.30 ms | 969.67 ms | 7.51 s | 8.53 s |\n| Common Lisp (SBCL) | 130.00 ms | 995.00 ms | 8.37 s | 9.49 s |\n| Dart AOT | 66.20 ms | 1.02 s | 8.97 s | 10.05 s |\n| JS (Deno) | 92.30 ms | 1.08 s | 10.19 s | 11.36 s |\n| Ocaml | 77.40 ms | 1.22 s | 10.14 s | 11.43 s |\n| JS (Node) | 104.30 ms | 1.32 s | 12.39 s | 13.81 s |\n| Racket | 114.77 ms | 1.77 s | 14.12 s | 16.01 s |\n| Typed Racket | 117.18 ms | 1.73 s | 14.18 s | 16.02 s |\n| Lobster (C++) | 123.76 ms | 1.91 s | 16.66 s | 18.69 s |\n| Scala Native | 254.40 ms | 3.14 s | 25.70 s | 29.10 s |\n| Inko[^2] | 435.70 ms | 6.67 s | 58.94 s | 66.05 s |\n| LuaJIT (JIT OFF) | 538.14 ms | 8.47 s | 66.93 s | 75.94 s |\n| Erlang | 643.30 ms | 10.46 s | 94.95 s | 106.06 s |\n| Lua | 827.92 ms | 12.78 s | 113.92 s | 127.53 s |\n| Python | 1.20 s | 18.93 s | 169.05 s | 189.18 s |\n| Lobster (JIT) | 1.35 s | 20.78 s | 195.33 s | 217.46 s |\n| Ruby | 1.50 s | 22.91 s | 213.64 s | 238.05 s |\n| Numpy | 216.51 ms | OOM | 3.27 s | Infinity s |\n\n### Multicore Results\n\n| Language | Time (5k posts) | 20k posts | 60k posts | Total |\n| -------------- | --------------- | ---------------- | ---------------- | --------- |\n| D Concurrent (v2) | 5.83 ms | $\\textsf{\\color{lightgreen}44.35 ms}$ | $\\textsf{\\color{lightgreen}265.80 ms}$ | 315.98 ms |\n| C# Concurrent (JIT) | 6.56 ms | 47.71 ms | 378.32 ms | 432.59 ms |\n| C++ Concurrent | 5.00 ms | 48.33 ms | 390.00 ms | 443.33 ms |\n| C# Concurrent (AOT) | 4.97 ms | 56.80 ms | 458.21 ms | 519.98 ms |\n| D Concurrent | 7.41 ms | 63.79 ms | 465.82 ms | 537.03 ms |\n| Nim Concurrent | $\\textsf{\\color{lightgreen}4.64 ms}$ | 59.38 ms | 497.93 ms | 561.95 ms |\n| Rust Concurrent | 5.05 ms | 67.47 ms | 541.40 ms | 613.93 ms |\n| Go Concurrent | 5.84 ms | 77.61 ms | 642.69 ms | 726.14 ms |\n| Julia Concurrent | 5.89 ms | 77.39 ms | 660.25 ms | 743.54 ms |\n| F# Concurrent | 7.50 ms | 104.33 ms | 904.33 ms | 1.02 s |\n| Swift Concurrent | 10.86 ms | 125.49 ms | 1.01 s | 1.15 s |\n| F# Concurrent (AOT) | 8.50 ms | 128.33 ms | 1.14 s | 1.28 s |\n| Java Concurrent (JIT) | 66.60 ms | 186.67 ms | 1.16 s | 1.41 s |\n| Zig Concurrent | 14.96 ms | 190.59 ms | 1.59 s | 1.79 s |\n| Java (GraalVM) Concurrent | 12.40 ms | 252.67 ms | 1.84 s | 2.11 s |\n\n\u003cdetails\u003e\n\u003csummary\u003e Old Results with details (on my machine) \u003c/summary\u003e\n\n| Language | Processing Time | Total (+ I/O) | Details |\n| ---------- | --------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Rust | - | 4.5s | Initial |\n| Rust v2 | - | 2.60s | Replace std HashMap with fxHashMap by [phazer99](https://www.reddit.com/r/rust/comments/16plgok/comment/k1rtr4x/?utm_source=share\u0026utm_medium=web2x\u0026context=3) |\n| Rust v3 | - | 1.28s | Preallocate and reuse map and unstable sort by [vdrmn](https://www.reddit.com/r/rust/comments/16plgok/comment/k1rzo7g/?utm_source=share\u0026utm_medium=web2x\u0026context=3) and [Darksonn](https://www.reddit.com/r/rust/comments/16plgok/comment/k1rzwdx/?utm_source=share\u0026utm_medium=web2x\u0026context=3) |\n| Rust v4 | - | 0.13s | Use Post index as key instead of Pointer and Binary Heap by [RB5009](https://www.reddit.com/r/rust/comments/16plgok/comment/k1s5ea0/?utm_source=share\u0026utm_medium=web2x\u0026context=3) |\n| Rust v5 | 38ms | 52ms | Rm hashing from loop and use vec[count] instead of map[index]count by RB5009 |\n| Rust v6 | 23ms | 36ms | Optimized Binary Heap Ops by [scottlamb](https://github.com/jinyus/related_post_gen/pull/12) |\n| Rust Rayon | 9ms | 22ms | Parallelize by [masmullin2000](https://github.com/jinyus/related_post_gen/pull/4) |\n| Rust Rayon | 8ms | 22ms | Remove comparison out of hot loop |\n| ⠀ | ⠀ | ⠀ | ⠀ |\n| Go | - | 1.5s | Initial |\n| Go v2 | - | 80ms | Add rust optimizations |\n| Go v3 | 56ms | 70ms | Use goccy/go-json |\n| Go v3 | 34ms | 55ms | Use generic binaryheap by [DrBlury](https://github.com/jinyus/related_post_gen/pull/7) |\n| Go v4 | 26ms | 50ms | Replace binary heap with custom priority queue |\n| Go v5 | 20ms | 43ms | Remove comparison out of hot loop |\n| Go Con | 10ms | 33ms | Go concurrency by [tirprox](https://github.com/jinyus/related_post_gen/pull/17) and [DrBlury](https://github.com/jinyus/related_post_gen/pull/8) |\n| Go Con v2 | 5ms | 29ms | Use arena, use waitgroup, rm binheap by [DrBlury](https://github.com/jinyus/related_post_gen/pull/20) |\n| ⠀ | ⠀ | ⠀ | ⠀ |\n| Python | - | 7.81s | Initial |\n| Python v2 | 1.35s | 1.53s | Add rust optimizations by [dave-andersen](https://github.com/jinyus/related_post_gen/pull/10) |\n| Numpy | 0.57s | 0.85s | Numpy implementation by [Copper280z](https://github.com/jinyus/related_post_gen/pull/11) |\n| ⠀ | ⠀ | ⠀ | ⠀ |\n| Crystal | 50ms | 96ms | Inital w/ previous optimizations |\n| Crystal v2 | 33ms | 72ms | Replace binary heap with custom priority queue |\n| ⠀ | ⠀ | ⠀ | ⠀ |\n| Odin | 110ms | 397ms | Ported from golang code |\n| Odin v2 | 104ms | 404ms | Remove comparison out of hot loop |\n| ⠀ | ⠀ | ⠀ | ⠀ |\n| Dart VM | 125ms | 530ms | Ported from golang code |\n| Dart bin | 274ms | 360ms | Compiled executable |\n| ⠀ | ⠀ | ⠀ | ⠀ |\n| Vlang | 339ms | 560ms | Ported from golang code |\n| ⠀ | ⠀ | ⠀ | ⠀ |\n| Zig | 80ms | 110ms | Provided by [akhildevelops](https://github.com/jinyus/related_post_gen/pull/30) |\n\n\u003c/details\u003e\n\n[^1]: Uses specialized datastructures meant for demonstration purposes: [more](https://github.com/LilithHafner/Jokes/tree/main/SuperDataStructures.jl)\n[^2]: Inko is currently in beta and optimizations haven't been applied. [more](https://github.com/jinyus/related_post_gen/pull/440#issuecomment-1816583612)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjinyus%2Frelated_post_gen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjinyus%2Frelated_post_gen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjinyus%2Frelated_post_gen/lists"}