{"id":24228780,"url":"https://github.com/blacha/s3-client-benchmark","last_synced_at":"2025-09-15T02:09:25.076Z","repository":{"id":146025609,"uuid":"347781629","full_name":"blacha/s3-client-benchmark","owner":"blacha","description":"Benchmarking AWSCLI vs SDK vs S5cmd","archived":false,"fork":false,"pushed_at":"2021-04-03T21:33:18.000Z","size":53,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-04T07:42:17.415Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blacha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-14T23:38:56.000Z","updated_at":"2021-04-03T21:33:20.000Z","dependencies_parsed_at":"2023-04-11T15:46:27.448Z","dependency_job_id":null,"html_url":"https://github.com/blacha/s3-client-benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/blacha/s3-client-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blacha%2Fs3-client-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blacha%2Fs3-client-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blacha%2Fs3-client-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blacha%2Fs3-client-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blacha","download_url":"https://codeload.github.com/blacha/s3-client-benchmark/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blacha%2Fs3-client-benchmark/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259297803,"owners_count":22836434,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-14T11:32:48.173Z","updated_at":"2025-06-11T16:32:42.902Z","avatar_url":"https://github.com/blacha.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Benchmarking S3 Transfer tools\n\n- awscli v1.19.27\n- [s5cmd](https://github.com/peak/s5cmd)\n- [aws-sdk (js)](https://github.com/aws/aws-sdk-js) v2.862.0\n\n\n# Setup\n\nRandom data can be created with `./make-data.sh` which is generated into `data/`\n\n|Name|File Count|File Size|Total Size|\n|-|-|-|-|\n|data/128|32|256MB|4GB|\n|data/256|16|256MB|4GB|\n|data/2048|2|2GB|4GB\n|data/4096|1|4GB|4GB\n\n## Mock s3 API\nMock server server with 4 worker threads\n\n```\nWORKER_COUNT=4 node src/server.js\n```\n\nOn a high end desktop CPU a single nodejs thread can process \u003e1GB/s\n\n## Client Connection\n\nConnect client to a local s3 mock server\n\n```\nnode src/client http://localhost:8080\n```\n\n\n# Results\n\n## Remote\n\nTwo computers were directly connected with a 8M cat6 networking cable.\n\n\n### Client\n- CPU: 16 Cores/32 Threads AMD Ryzen 9 5950X\n- OS: Ubuntu 21.04\n- Nic: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller \\[AQtion\\] (rev 02)\n\n### Server\n- CPU 8 Cores/16 Threads Intel Xeon D-2141I CPU @ 2.20GHz\n- OS: Ubuntu 20.10\n- Nic: Intel Corporation Ethernet Connection X722 for 10GBASE-T\n\n### Iperf3\niperf3 was run across the link for 1 minute and showed a average transfer rate of around 700MB/s\n\n```\nConnecting to host 192.168.0.250, port 5201\n[  5] local 192.168.0.200 port 59630 connected to 192.168.0.250 port 5201\n[ ID] Interval           Transfer     Bitrate         Retr  Cwnd\n[  5]   0.00-1.00   sec   730 MBytes  6.12 Gbits/sec   91   1.34 MBytes       \n[  5]   1.00-2.00   sec   729 MBytes  6.11 Gbits/sec    0   1.35 MBytes       \n[  5]   2.00-3.00   sec   730 MBytes  6.12 Gbits/sec    0   1.35 MBytes       \n[  5]   3.00-4.00   sec   729 MBytes  6.11 Gbits/sec    0   1.36 MBytes       \n[  5]   4.00-5.00   sec   730 MBytes  6.12 Gbits/sec    0   1.36 MBytes       \n[  5]   5.00-6.00   sec   730 MBytes  6.12 Gbits/sec    0   1.38 MBytes       \n[  5]   6.00-7.00   sec   729 MBytes  6.11 Gbits/sec    0   1.38 MBytes       \n[  5]   7.00-8.00   sec   730 MBytes  6.12 Gbits/sec    0   1.53 MBytes       \n[  5]   8.00-9.00   sec   729 MBytes  6.11 Gbits/sec    0   1.53 MBytes       \n[  5]   9.00-10.00  sec   730 MBytes  6.12 Gbits/sec    0   1.53 MBytes       \n\n...\n\n[ ID] Interval           Transfer     Bitrate         Retr\n[  5]   0.00-60.00  sec  42.7 GBytes  6.12 Gbits/sec  198             sender\n[  5]   0.00-60.00  sec  42.7 GBytes  6.12 Gbits/sec                  receiver\n```\n\n\nClient|Threads|Files|FileSize (MB)|TotalTransferred (MB)|Duration (s)|Speed (MB/s)\n-|-|-|-|-|-|-\naws-sdk:js|1|32|128|4096|6.319|648.2\ns5-cmd|1|32|128|4096|9.363|437.47\naws-sdk:js|2|64|128|8192|12.207|671.09\ns5-cmd|2|64|128|8192|13.238|618.82\naws-sdk:js|4|128|128|16384|23.964|683.69\ns5-cmd|4|128|128|16384|23.122|708.59\naws-sdk:js|1|16|256|4096|6.137|667.43\ns5-cmd|1|16|256|4096|8.258|496\naws-sdk:js|2|32|256|8192|11.825|692.77\ns5-cmd|2|32|256|8192|13.208|620.23\naws-sdk:js|4|64|256|16384|23.469|698.11\ns5-cmd|4|64|256|16384|23.396|700.29\naws-sdk:js|1|8|512|4096|5.955|687.83\ns5-cmd|1|8|512|4096|7.454|549.5\naws-sdk:js|2|16|512|8192|11.672|701.85\ns5-cmd|2|16|512|8192|12.299|666.07\naws-sdk:js|4|32|512|16384|23.162|707.37\ns5-cmd|4|32|512|16384|23.085|709.72\naws-sdk:js|1|2|2048|4096|5.907|693.41\ns5-cmd|1|2|2048|4096|6.422|637.81\naws-sdk:js|2|4|2048|8192|11.579|707.49\ns5-cmd|2|4|2048|8192|11.581|707.37\naws-sdk:js|4|8|2048|16384|22.976|713.09\ns5-cmd|4|8|2048|16384|22.791|718.88\naws-sdk:js|1|1|4098|4098|6.275|653.07\ns5-cmd|1|1|4098|4098|5.821|704\naws-sdk:js|2|2|4098|8196|11.624|705.09\ns5-cmd|2|2|4098|8196|11.474|714.31\naws-sdk:js|4|4|4098|16392|22.882|716.37\ns5-cmd|4|4|4098|16392|22.807|718.73\n\n\n## Localhost \nTesting with Both Client and Server on the same machine\n\n### Linux Laptop (HP 840 G7)\n- Server running on http://localhost:8080\n- CPU: 3 Cores/6 Threads Intel i7-10810U \n- OS: Ubuntu 20.04\n\n|Client|Threads|Files|FileSize (MB)|TotalTransferred (MB)|Duration (s)|Speed (MB/s)|\n|-|-|-|-|-|-|-|\naws-cli|1|16|256|4096|21.168|193.5\naws-sdk:js|1|16|256|4096|13.923|294.19\ns5-cmd|1|16|256|4096|7.288|562.02\naws-cli|2|16|256|8192|38.174|214.6\naws-sdk:js|2|16|256|8192|18.253|448.8\ns5-cmd|2|16|256|8192|11.987|683.41\naws-cli|4|16|256|16384|64.235|255.06\naws-sdk:js|4|16|256|16384|21.03|779.08\ns5-cmd|4|16|256|16384|19.36|846.28\n\n### Linux Desktop\n\n- Server running on http://localhost:8080\n- CPU: 16 Cores/32 Threads AMD Ryzen 9 5950X\n- OS: Ubuntu 21.04\n\n#### 32 x 128MB\nClient|Threads|Files|FileSize (MB)|TotalTransferred (MB)|Duration (s)|Speed (MB/s)\n-|-|-|-|-|-|-\naws-cli|1|32|128|4096|20.455|200.24\naws-sdk:js|1|32|128|4096|4.499|910.42\ns5-cmd|1|32|128|4096|6.418|638.21\naws-cli|2|64|128|8192|17.885|458.04\naws-sdk:js|2|64|128|8192|5.318|1540.43\ns5-cmd|2|64|128|8192|6.458|1268.5\naws-cli|4|128|128|16384|17.081|959.19\naws-sdk:js|4|128|128|16384|7.581|2161.19\ns5-cmd|4|128|128|16384|6.86|2388.34\naws-cli|8|256|128|32768|24.684|1327.5\naws-sdk:js|8|256|128|32768|15.424|2124.48\ns5-cmd|8|256|128|32768|9.849|3327.04\n\n#### 16 x 256MB\nClient|Threads|Files|FileSize (MB)|TotalTransferred (MB)|Duration (s)|Speed (MB/s)\n-|-|-|-|-|-|-\naws-cli|1|16|256|4096|16.201|252.82\naws-sdk:js|1|16|256|4096|4.097|999.76\ns5-cmd|1|16|256|4096|3.493|1172.63\naws-cli|2|32|256|8192|14.173|578\naws-sdk:js|2|32|256|8192|4.71|1739.28\ns5-cmd|2|32|256|8192|4.184|1957.93\naws-cli|4|64|256|16384|13.797|1187.5\naws-sdk:js|4|64|256|16384|7.255|2258.3\ns5-cmd|4|64|256|16384|6.086|2692.08\naws-cli|8|128|256|32768|20.039|1635.21\naws-sdk:js|8|128|256|32768|14.8|2214.05\ns5-cmd|8|128|256|32768|11.269|2907.8\n\n#### 8 x 512MB\nClient|Threads|Files|FileSize (MB)|TotalTransferred (MB)|Duration (s)|Speed (MB/s)\n-|-|-|-|-|-|-\naws-cli|1|8|512|4096|13.727|298.39\naws-sdk:js|1|8|512|4096|3.984|1028.11\ns5-cmd|1|8|512|4096|3.336|1227.82\naws-cli|2|16|512|8192|11.689|700.83\naws-sdk:js|2|16|512|8192|4.529|1808.79\ns5-cmd|2|16|512|8192|3.632|2255.51\naws-cli|4|32|512|16384|11.273|1453.38\naws-sdk:js|4|32|512|16384|7.025|2332.24\ns5-cmd|4|32|512|16384|5.255|3117.79\naws-cli|8|64|512|32768|17.511|1871.28\naws-sdk:js|8|64|512|32768|14.453|2267.21\ns5-cmd|8|64|512|32768|9.202|3560.97\n\n#### 2 x 2048MB\nClient|Threads|Files|FileSize (MB)|TotalTransferred (MB)|Duration (s)|Speed (MB/s)\n-|-|-|-|-|-|-\naws-cli|1|2|2048|4096|12.854|318.66\naws-sdk:js|1|2|2048|4096|3.956|1035.39\ns5-cmd|1|2|2048|4096|3.145|1302.38\naws-cli|2|4|2048|8192|10.833|756.21\naws-sdk:js|2|4|2048|8192|4.416|1855.07\ns5-cmd|2|4|2048|8192|3.45|2374.49\naws-cli|4|8|2048|16384|10.54|1554.46\naws-sdk:js|4|8|2048|16384|7.008|2337.9\ns5-cmd|4|8|2048|16384|4.668|3509.85\naws-cli|8|16|2048|32768|16.051|2041.49\naws-sdk:js|8|16|2048|32768|14.265|2297.09\ns5-cmd|8|16|2048|32768|9.365|3498.99\n\n#### 1 x 4096MB\nClient|Threads|Files|FileSize (MB)|TotalTransferred (MB)|Duration (s)|Speed (MB/s)\n-|-|-|-|-|-|-\naws-cli|1|1|4098|4098|12.217|335.43\naws-sdk:js|1|1|4098|4098|3.797|1079.27\ns5-cmd|1|1|4098|4098|2.97|1379.8\naws-cli|2|2|4098|8196|10.64|770.3\naws-sdk:js|2|2|4098|8196|4.478|1830.28\ns5-cmd|2|2|4098|8196|3.095|2648.14\naws-cli|4|4|4098|16392|9.827|1668.06\naws-sdk:js|4|4|4098|16392|6.845|2394.74\ns5-cmd|4|4|4098|16392|4.298|3813.87\naws-cli|8|8|4098|32784|15.377|2132.02\naws-sdk:js|8|8|4098|32784|14.163|2314.76\ns5-cmd|8|8|4098|32784|8.958|3659.75\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblacha%2Fs3-client-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblacha%2Fs3-client-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblacha%2Fs3-client-benchmark/lists"}