{"id":19281759,"url":"https://github.com/scality/s3blaster","last_synced_at":"2026-03-05T00:36:20.528Z","repository":{"id":89559581,"uuid":"60217924","full_name":"scality/s3blaster","owner":"scality","description":"S3 Blaster","archived":false,"fork":false,"pushed_at":"2022-07-22T03:18:32.000Z","size":229,"stargazers_count":10,"open_issues_count":3,"forks_count":2,"subscribers_count":58,"default_branch":"development/8.0","last_synced_at":"2025-04-22T01:38:37.725Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scality.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-06-01T23:38:00.000Z","updated_at":"2022-11-30T17:47:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"f4a67a44-7504-4c76-815b-953facb584a0","html_url":"https://github.com/scality/s3blaster","commit_stats":null,"previous_names":[],"tags_count":256,"template":false,"template_full_name":null,"purl":"pkg:github/scality/s3blaster","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fs3blaster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fs3blaster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fs3blaster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fs3blaster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scality","download_url":"https://codeload.github.com/scality/s3blaster/tar.gz/refs/heads/development/8.0","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fs3blaster/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30102482,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T23:59:36.199Z","status":"ssl_error","status_checked_at":"2026-03-04T23:56:48.556Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T21:24:04.033Z","updated_at":"2026-03-05T00:36:20.475Z","avatar_url":"https://github.com/scality.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# S3Blaster program\n\nS3Blaster program measures performance of S3 servers. For a given test scenario,\nS3Blaster emits requests to S3 servers. By collecting responses, S3Blaster\ncomputes different metrics to measure performance of S3.\n\n## Performance metrics\n\nS3Blaster computes two performance metrics:\n\n* Latency of a request\n\n    Latency of a request is the interval between the time the request is sent to\nS3 and the time S3Blaster receives the request's response. S3Blaster\ncomputes different statistics on request latencies: average, standard-deviation,\npercentiles.\n\n* Number of operations per second\n\n    This is the number of responses that S3Blaster receives per second.\nS3Blaster computes average number of operations/s.\n\n## Working schema\n\n![Schema](./img/s3blaster.png)\n\nFigure 1: Working schema of S3Blaster program\n\nA working schema of S3Blaster is shown in Figure 1. Note, if there is no fork\noperation (clusters), S3Blaster workers in the schema are replaced by the\nS3Blaster Master. Roles of each component are described below.\n\n### S3Blaster Master\n\nBesides forking workers (if relevant), S3Blaster Master does following jobs:\n\n* Monitoring resources' consuming (CPU \u0026 Memory usage) of running processes on\n    all servers.\n\n* Gathering statistics results computed from all workers.\n\n* Forwarding obtained results to `Plotter` that plots appropriated graphs.\n\n### S3Blaster Worker\n\n* Based on programmed scenarios, each worker sends requests to S3 Servers and\n    then computes performance metrics based on responses received from the\n    servers.\n\n* Forwarding obtained results to `Plotter` that plots appropriated graphs.\n\n### Plotter\n\nThe component displays performance metrics and monitored resources' consuming\non graphs.\n\n## Installation\n\nSimple as `npm install`\n\n### Plot dependencies\n\nOutput graphs would be created using `gnuplot` program. To install it on Ubuntu\n\n```\nsudo apt-get install libcairo2-dev\nsudo apt-get install libpango1.0-dev\nsudo apt-get install gnuplot\n```\n\nFor other OS, see Cairo (https://cairographics.org/download/) and\ngnuplot (http://www.gnuplot.info/download.html)\n\n## Program structure\n\nThe program consists of three main files:\n\n* `s3blaster.js`\n    is the **core** of S3Blaster. S3Blaster master does the following jobs:\n    * Forking S3Blaster workers (if relevant)\n    * Gathering statistics results computed from all workers\n    * Storing statistics in files that are used for plotting graphs. There are\n    2 files `.txt` containing statistics whose suffix names are\n    *_ssm* (for system monitoring) and *_summary* (for summary results).\n\n    Whereas, S3Blaster worker does the following jobs:\n    * Sending/receiving requests to/from S3\n    * Calculating statistics on request latency, number of operations/s\n    * Storing statistics in files that are used for plotting graphs. There are\n    5 files `.txt` containing statistics whose suffix names are\n    *_live*, *_stats*, *_thread*, *_pdf_func*, *_cdf_func*.\n\n* `config.js`\n    get all input parameters for S3Blaster. All input parameters for s3blaster\n    can be assigned via the following ways with a decreasing priority\n    level, i.e. parameter will get value given by the highest priority one:\n\n    - command line\n    - a JSON file defined by `--config` command, `config.json` by default\n    - initialized in `config.js` file\n\n    For credential keys `accessKeyId` and `secretAccessKey`, they can be\n    assigned by the following ways with a decreasing priority level:\n    - command line:\n\n    ```\n    --accessKeyId = your_access_key_id\n    --secretAccessKey = your_secret_access_key\n    ```\n\n    - config JSON file, in \"acc\" sub-object:\n\n    ```\n    \"accessKeyId\": \"your_access_key_id\"\n    \"secretAccessKey\": \"your_secret_access_key\"\n    ```\n\n    - environment variable:\n\n    ```\n    ACCESSKEYID=your_access_key_id\n    SECRETACCESSKEY=your_secret_access_key\n    ```\n\n    - credential file defined by `--accFile`, default file `~/.s3cfg`\n\n    ```\n    access_key = your_access_key_id\n    secret_key = your_secret_access_key\n    ```\n\n* `plotter.js`\n    plots graphs showing statistics measured by S3Blaster. Plotter generates\n    two graphs for *_ssm* and *_summary* files sent from S3Blaster Master.\n    For each S3Blaster worker, Plotter could generate up to 5 graph files\n    whose suffix names are *_stats*, *_thread*, *_size*, *_pdf*, *_cdf*.\n\nAn additional file `runS3Blaster.js` allows executing S3Blaster with input\nparameters stored in an object that is used for pre-programmed tests (see\n    [Pre-programmed tests](#pre-programmed-tests))\n\nBesides, monitoring of resources' consuming of processes on servers and\nmonintoring storage size of Metadata store are implemented in files in `ssm`\nfolder.\n\n## How to run S3Blaster\n\nS3Blaster is executed with default input parameters with: `npm start`\n\nBy default, output results are stored in a sub-folder of the `results`\ndirectory. A default name for the sub-folder is the current date. A specified\noutput folder can be defined by using the environment variable FOLDERNAME, e.g.\n\n`FOLDERNAME=release/today npm start`\n\nThen, all output files will be stored in the `./results/release/today`\ndirectory.\n\n### Plot manipulation\n\nThere are two plot actions applicable for simulation results. Configuration\ninfo and results of a simulation is stored in an `.id` file whose prefix is\nsame as stats files.\n\n* Re-plot graphs\n\n    All graphs of a simulation can be re-plotted with modifying on\n\n    - Prefix for output graph files\n    - Type of output files (pdf or png)\n    - Font size\n    - Font type.\n\n    CLI for re-plot:\n\n    ```\n    node lib/plotter.js --replot path_to_id_file --outputPrefixName \u003cname\u003e\n    --outputType \u003ctype\u003e --fontType \u003ctype\u003e --fontSize \u003csize\u003e\n    ```\n\n* Performance comparison\n\n    Performance of different tests are shown in a same file. The graph file\n    contains two pages for latency and #operations/s performance metrics.\n\n    In each page, graphs are placed in a layout where each column corresponds\n    to a request type, e.g. put, get\n\n    Each graph displays performances for an object size. The x-axis is number of\n    parallel requests. Each curve in the graph shows performance of each test.\n\n    CLI for comparison:\n\n    ```\n    node lib/plotter.js --compare\n    path_to_id_file_test_1,path_to_id_file_test_2,path_to_id_file_test_3\n    ```\n\n### Main input parameters\n\nThere are multiple configurable input parameters as shown in\n`config.json` file. However, some of them should be correctly\nassigned for an appropriated measurement:\n\nName | Related keys\n--- | ---\nCredential keys | `accessKeyId`, `secretAccessKey`\nS3 servers' address | `host`, `port`\nRequests for measuring | `requests`\nSchedule of requests | `schedule`\nBuckets | `bucketPrefix`, `bucketsNb`\nObjects | `objectsNb`, `sizes`, `unit`\nMeasurement time | `runTime`, `observationsNb`\nMonitoring | `ssm`, `servers`, `user`, `publicKey`, `privateKey`, `mdPath`\nMultipart Upload | `partSizes`\n\nTo run S3Blaster with other specified parameters, please see [configuration]\n(./CONFIGURATION.md).\n\n## Output: statistics and graphs\n\nFor each test, s3blaster generates files containing statistics results (`.txt`)\nand graph files (`.pdf` or `.png`). Main configuration  information is shown on\ntop of each statistics/graph file.\n\n* Summary results\n\n    Summary results are shown in *_summary* text/graph files. It contains\nperformance metrics that are computed based on results from all workers (if\nrelevant). Moreover, if the monitoring of servers was active, the file show\namount of storage using by metadata. The amount is computed as the difference\nof meta-stores between before and after test.\n\n* Detailed result files\n\n    For each S3Blaster worker, there are five `.txt` files and five graph files.\n\n    * *_live* file contains real-time performance metrics. The 'real-time' means\n    that the metrics are calculated from requests/responses received within the\n    last duration time. The time duration is determined by `rate` in the\n    `lib/config.js` file. Results of the file will be used to generate *_live*\n    graph.\n\n    * *_stats* file contains performance metrics computed after some given\n    number of requests successes. A graph file is generated for each such file.\n\n    * *_thread* file contains statistics vs. number of parallel requests and\n        number of object sizes. A graph file is generated for each such file.\n\n    * *_pdf_func* file contains estimated probability distribution function. A\n    graph file is generated for each such file.\n\n    * *_cdf_func* file contains estimated cumulative distribution function. A\n    graph file is generated for each such file.\n\nIn each graph file, there could be some figures that are located in rows and\ncolumns layout. Graphs on a row/column have a same configuration parameter,\ne.g. same type of request, same object size etc.\n\n* Monitored resources' consuming\n\n    If the monitoring of resources' consuming by running processes was active,\ncollected  are stored in *_ssm.txt* file. These results combined with *_live*\nstatistics from each worker will be shown in a single graph *_live* file. In\nthat file, measured performance and resources' consuming are shown versus\nelapsed time.\n\n## Examples\n\n* Example 1\n\nSingle connector, single bucket with 1000 objects, mixed requests, object 8KB,\nrun in 100 seconds\n\n```javascript\nnode index.js --host localhost --port 8000\n--bucketsNb 1 --objectsNb 1000 --sizes 8 --unit KB --requests\nput,get,delete --schedule mixed --runTime 100 --output example1\n```\n\n* Example 2\n\nHaproxy connector with port 80, multiple buckets, sequential requests, object\n8KB \u0026 100KB\n\n```javascript\nnode index.js --host localhost --port 80\n--bucketsNb 100 --objectsNb 1000 --sizes 8,100 --unit KB --requests\nput,get,delete --schedule each --runTime 100 --output example2\n```\n\n* Example 3\n\n*Balancing* connectors, 10 forks, monitoring servers, multiple buckets,\nmixed requests with their proportion and range\n\n```javascript\nnode index.js --host balancing --port 8000 --forksNb 10\n--bucketsNb 100 --objectsNb 1000 --sizes 8 --unit KB --requests\nput,get --proprReqs 2,8 --range 0:500,500:1000 --schedule each --ssm true\n--runTime 100 --output example3\n```\n\n## Pre-programmed tests\n\nThere are six pre-programmed tests as below.\n\n Parameters | S3Simple | S3Standard | S3Full | S3Sizes | S3Parallel | S3MPU\n --- | --- | --- | --- | --- | --- | ---\n S3 Connector | single | single, balancing | single, haproxy, balancing | single | single | single\n #S3blaster workers  | 1 | 1 \u0026 multiple | 1 \u0026 multiple | 1 | 1 \u0026 multiple | 1\n Bucket | single \u0026 multiple |  single \u0026 multiple | single \u0026 multiple | single \u0026 multiple | single \u0026 multiple | single \u0026 multiple\n Requests | `put`, `get`, `delete` | `put`, `get`, `delete` | `put`, `list`, `get`, `get ACL`, `delete`, `combine` | `put`, `get`, `delete` | `put`, `get`, `delete` | `multi-upload`, `get`, `delete`\n Schedule | `sequential` \u0026 `mixed` | `sequential`| `sequential` \u0026 `mixed` | `sequential` \u0026 `mixed` | `sequential` | `sequential` \u0026 `mixed`\n Object size | `OB`, `10KB` | `OB`, `10KB`, `1MB` | `OB`, `10KB`, `512KB`, `1MB`, `10MB` | `0, 10, .., 90KB` \u0026 `100, 200, .., 900KB` \u0026 `1, 2, .., 10MB` | `OB`, `10KB`, `1MB` | `1GB` (part size `5MB`, `20MB`)\n #parallel requests | 64 | 1, 32, 64, 128, .., 2048 | 1, 8, 16, 32, .., 2048 | 1, 50 | 32, 64, 128, .., 10240 | 1, 2, 4, 8, 16\n Run time (s) | 60 | 120 | 240 | 600 | 600 | 1200\n |\n **Execution cmd** | `npm run s3simple` | `npm run s3standard` | `npm run s3full` | `npm run s3sizes` | `npm run s3parallel` | `npm run s3mpu`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscality%2Fs3blaster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscality%2Fs3blaster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscality%2Fs3blaster/lists"}