{"id":13757230,"url":"https://github.com/axsaucedo/rust-io-file-benchmark","last_synced_at":"2025-05-13T23:35:47.982Z","repository":{"id":150594596,"uuid":"173961371","full_name":"axsaucedo/rust-io-file-benchmark","owner":"axsaucedo","description":null,"archived":false,"fork":false,"pushed_at":"2019-03-10T11:01:11.000Z","size":2007,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-16T22:42:56.345Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/axsaucedo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-03-05T14:26:15.000Z","updated_at":"2022-03-14T13:20:55.000Z","dependencies_parsed_at":"2023-04-29T13:00:30.849Z","dependency_job_id":null,"html_url":"https://github.com/axsaucedo/rust-io-file-benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/axsaucedo%2Frust-io-file-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/axsaucedo%2Frust-io-file-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/axsaucedo%2Frust-io-file-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/axsaucedo%2Frust-io-file-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/axsaucedo","download_url":"https://codeload.github.com/axsaucedo/rust-io-file-benchmark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254043218,"owners_count":22004912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T12:00:30.113Z","updated_at":"2025-05-13T23:35:47.935Z","avatar_url":"https://github.com/axsaucedo.png","language":"Rust","funding_links":[],"categories":["IO"],"sub_categories":["benchmark"],"readme":"# Rust benchmarking on Large File IO\n\nThis is a short report on the performance metrics obtained processing large files with a small rust/python script.\n\nIn this case the definition of \"Large\" is files that won't fit in memory easily (e.g. 100GB \u003e) and require streaming / buffers.\n\n## Experiment overview\n\nThe experiment is as follows:\n\n* 2GB text file containing text information about objects\n* Each 3 consecutive lines has information about one object (i.e. 1 line = one attribute)\n* Each object is separated by one blank line\n\n## Objective\n\nObjective is to read file, iterate through lines and write results to CSV/TSV\n\n## Example\n\n#### input file example\n\nOBJECT 1 ATTR 1: CONTENT\nOBJECT 1 ATTR 2: CONTENT\nOBJECT 1 ATTR 3: CONTENT\n\nOBJECT 2 ATTR 1: CONTENT\nOBJECT 2 ATTR 2: CONTENT\nOBJECT 2 ATTR 3: CONTENT\n\n...etc\n\n#### expected output file example\n\nOBJECT 1 ATTR1, OBJECT 1 ATTR 2, OBJECT 1 ATTR 3 \u003cbr\u003e\nOBJECT 2 ATTR1, OBJECT 2 ATTR 2, OBJECT 2 ATTR 3\n... etc\n\n\n# Improvements\n\nCurrently there are clear optimisations required for the Rust code, as there are several string operations.\n\nIdeally it would be possible to process the files in rust as u8 (byte) format to save time, which would accelerate the processing, but unfortunately the BufReader class doesn't seem to provide functionality to read the files as bytes directly.\n\n# Results\n\nThe results are provided below.\n \n\n## Python:\n\nSimple python implementation without any buffering, using the native python file IO read_line / write.\n\nreal    2m16.087s\n\nuser    1m4.397s\n\nsys     0m4.352s\n\n\n## Rust 1.33 main.rs:\n\nRust implementation using the BufReader and BufWriter converting to string, appending and writing bytes.\n\nreal    7m28.602s\n\nuser    7m19.379s\n\nsys     0m5.094s\n\n\n## Rust 1.33 main-vec.rs:\n\nRust implementation using BufReader and Bufwriter, and using a vector to attempt single string concat.\n\nreal    8m35.463s                                                                                        \n\nuser    8m24.227s                                                                                        \n\nsys     0m5.918s\n\n## Rust 1.33 main-copy.rs:\n\nRust implementation using plain File reading in bytes and copying it to another location without performing and processing.\n\nreal    41m12.918s\n\nuser    22m55.845s\n\nsys     18m15.949s\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faxsaucedo%2Frust-io-file-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faxsaucedo%2Frust-io-file-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faxsaucedo%2Frust-io-file-benchmark/lists"}