{"id":17222894,"url":"https://github.com/rush/r2-benchmark","last_synced_at":"2025-03-25T16:22:15.465Z","repository":{"id":198814617,"uuid":"701595645","full_name":"Rush/r2-benchmark","owner":"Rush","description":"Benchmark R2 storage","archived":false,"fork":false,"pushed_at":"2023-10-07T03:29:51.000Z","size":36,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-19T20:53:05.030Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Rush.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-10-07T02:34:31.000Z","updated_at":"2023-10-07T03:27:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"8d3daef2-0e94-4480-b404-2afa0ee7acc5","html_url":"https://github.com/Rush/r2-benchmark","commit_stats":null,"previous_names":["rush/r2-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rush%2Fr2-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rush%2Fr2-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rush%2Fr2-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rush%2Fr2-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Rush","download_url":"https://codeload.github.com/Rush/r2-benchmark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245496309,"owners_count":20624871,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T04:06:39.740Z","updated_at":"2025-03-25T16:22:15.440Z","avatar_url":"https://github.com/Rush.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# r2-benchmark\n\nBenchmarking tool for Cloudflare's R2 storage system. Assess the maximum number of operations per second (ops/s) for various operations like write, read, remove, and stat, either on a single bucket or spanning multiple buckets.\n\n## Background\n\nCloudflare R2 storage has certain performance limits. As per a discussion with a Cloudflare R2 engineer:\n- Expected writes per second: ~3000 ops/s\n- Expected reads per second: ~20000 ops/s\n\nMy benchmarks yielded the following results over 3 different buckets, executed from different hosts:\n- **Write**: 5439 ops/s\n- **Read**: 3111 ops/s\n- **Stat**: 2409 ops/s\n- **Remove**: 2798 ops/s\n\n## Installation\n\n```bash\nnpm install\n```\n\n## Configuration\n\nBegin by creating a `.env` file using the template provided in `.env.example`:\n\n```plaintext\n# Specify one or more buckets, separated by a comma.\n# More buckets can theoretically provide higher performance if we approach the I/O limits of a single bucket.\nS3_BUCKET=bucket1,bucket2,bucket3\nS3_HOSTNAME=XXX.r2.cloudflarestorage.com\nS3_ACCESS_KEY=some_access_key\nS3_SECRET_KEY=some_secret_key\n```\n\n## Usage\n\nRun the benchmark using:\n\n```bash\nnode r2-benchmark.js \u003cnumber of objects\u003e \u003cnumber of threads\u003e\n```\n\n- **Number of objects**: Total objects the program will write to the bucket.\n- **Number of threads**: Threads that will use separate keep-alive TCP connections from an HTTP agent. A high number of threads are necessary for sustaining top throughput.\n\nThe benchmark will execute in stages:\n1. **Writing**: The program writes the specified number of objects using the provided threads.\n2. **Reading**: Validates the contents of each stored object by reading and measuring read performance.\n3. **Stat**: Measures the stat performance.\n4. **Removal**: Cleans up all objects that were written and measures removal performance.\n\n### Example:\n\n```bash\n\u003e node r2-benchmark.js 10000 1000\n```\n\nOutput:\n\n```plaintext\n\u003e node r2-benchmark.js 10000 1000\nWrite: scheduling for 10000 objects in 1000 threads and 1 buckets\n03:10:02: Write: Ops 0.0/s Throughput 0.0MB/s\n03:10:03: Write: Ops 0.0/s Throughput 0.0MB/s\n03:10:04: Write: Ops 642.2/s Throughput 10.0MB/s\n03:10:05: Write: Ops 931.1/s Throughput 14.5MB/s\n03:10:06: Write: Ops 1162.0/s Throughput 18.2MB/s\nRetrying on TRANSIENT error\nRetrying on TRANSIENT error\n03:10:07: Write: Ops 1213.3/s Throughput 19.0MB/s\nRetrying on TRANSIENT error\n03:10:08: Write: Ops 1127.9/s Throughput 17.6MB/s\nRetrying on TRANSIENT error\n03:10:09: Write: Ops 1284.2/s Throughput 20.1MB/s\n03:10:10: Write: Ops 1251.9/s Throughput 19.6MB/s\nRetrying on TRANSIENT error\nRetrying on TRANSIENT error\n03:10:11: Write: Ops 1183.1/s Throughput 18.5MB/s\nRetrying on TRANSIENT error\nRetrying on TRANSIENT error\n03:10:12: Write: Ops 867.6/s Throughput 13.6MB/s\nRetrying on TRANSIENT error\n03:10:13: Write: Ops 23.0/s Throughput 0.4MB/s\nWrite took 13477ms. Peak ops 1284.2/s Avg ops 742.0/s  Total size 156.3MB  Recoverable errors 9\nRead: scheduling for 10000 objects in 1000 threads and 1 buckets\n03:10:15: Read: Ops 0.0/s Throughput 0.0MB/s\n03:10:16: Read: Ops 741.6/s Throughput 11.6MB/s\n03:10:18: Read: Ops 1109.1/s Throughput 17.3MB/s\n03:10:19: Read: Ops 1346.1/s Throughput 21.0MB/s\n03:10:20: Read: Ops 1435.3/s Throughput 22.4MB/s\nRetrying on TRANSIENT error\n03:10:22: Read: Ops 1429.4/s Throughput 22.3MB/s\n03:10:23: Read: Ops 2057.7/s Throughput 32.2MB/s\nRead took 8985ms. Peak ops 2057.7/s Avg ops 1113.0/s  Total size 156.3MB  Recoverable errors 1\nStat: scheduling for 10000 objects in 1000 threads and 1 buckets\n03:10:24: Stat: Ops 0.0/s Throughput 0.0MB/s\n03:10:25: Stat: Ops 1108.4/s Throughput 17.3MB/s\n03:10:26: Stat: Ops 1575.2/s Throughput 24.6MB/s\n03:10:28: Stat: Ops 1625.4/s Throughput 25.4MB/s\n03:10:29: Stat: Ops 1634.1/s Throughput 25.5MB/s\nRetrying on TRANSIENT error\n03:10:30: Stat: Ops 2418.3/s Throughput 37.8MB/s\nStat took 7916ms. Peak ops 2418.3/s Avg ops 1263.3/s  Total size 156.3MB  Recoverable errors 1\nRemove: scheduling for 10000 objects in 1000 threads and 1 buckets\n03:10:32: Remove: Ops 0.0/s Throughput 0.0MB/s\n03:10:33: Remove: Ops 847.1/s Throughput 13.2MB/s\n03:10:34: Remove: Ops 1570.5/s Throughput 24.5MB/s\n03:10:35: Remove: Ops 1614.0/s Throughput 25.2MB/s\nRetrying on TRANSIENT error\nRetrying on TRANSIENT error\n03:10:36: Remove: Ops 1599.6/s Throughput 25.0MB/s\nRetrying on TRANSIENT error\n03:10:37: Remove: Ops 1648.8/s Throughput 25.8MB/s\nRetrying on TRANSIENT error\nRetrying on TRANSIENT error\nRetrying on TRANSIENT error\nRetrying on TRANSIENT error\nRetrying on TRANSIENT error\nRetrying on TRANSIENT error\n03:10:38: Remove: Ops 1749.2/s Throughput 27.3MB/s\nRetrying on TRANSIENT error\n03:10:39: Remove: Ops 485.0/s Throughput 7.6MB/s\n03:10:40: Remove: Ops 14.0/s Throughput 0.2MB/s\nRetrying on TRANSIENT error\nRetrying on TRANSIENT error\n03:10:41: Remove: Ops 2.0/s Throughput 0.0MB/s\nRemove took 11403ms. Peak ops 1749.2/s Avg ops 877.0/s  Total size 156.3MB  Recoverable errors 12\n```\n\nFrom this execution, we observed:\n- Peak **Write** ops/s: 1284\n- Peak **Read** ops/s: 2057\n- Peak **Stat** ops/s: 2418\n- Peak **Remove** ops/s: 1749\n\n## Running on Multiple Hosts\n\nTo fully evaluate the maximum operations per second (ops/s) achievable, it's advantageous to run this script across several Node processes and on different machines.\n\n### Setup\n\n1. Install and configure this repository on multiple SSH hosts.\n2. Ensure you have a terminal feature, like the one in Konsole, that can simultaneously run commands across multiple terminals.\n\n### Execution\n\nOn each host, run:\n\n```bash\nnode r2-benchmark.js 10000 1000 | tee log-$HOST.txt\n```\n\nThis command will execute the benchmark and save the logs with the hostname as a part of the filename.\n\nAfter collecting data from all hosts, gather all log files in a single location and analyze the peak performance:\n\n```bash\nnode peak-performance.js log*.txt\n```\n\nOutput:\n\n```plaintext\nPeak Performance: {\n  Write: 5439.7,\n  Read: 3111.2000000000003,\n  Stat: 2409.5,\n  Remove: 2798.0000000000005\n}\n```\n\n## License\n\nCopyright (c) 2023, Code Charm, Inc.\n\nLicense: MIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frush%2Fr2-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frush%2Fr2-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frush%2Fr2-benchmark/lists"}