{"id":27962238,"url":"https://github.com/perbu/ostresser","last_synced_at":"2025-05-07T19:15:10.158Z","repository":{"id":285564741,"uuid":"958566607","full_name":"perbu/ostresser","owner":"perbu","description":"Object store stress test.","archived":false,"fork":false,"pushed_at":"2025-05-06T07:01:06.000Z","size":61,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-07T19:15:06.346Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/perbu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-04-01T12:07:49.000Z","updated_at":"2025-05-06T07:01:09.000Z","dependencies_parsed_at":"2025-04-01T13:30:26.789Z","dependency_job_id":"46129b13-058a-4d3d-acdc-c6897b7f551b","html_url":"https://github.com/perbu/ostresser","commit_stats":null,"previous_names":["perbu/ostresser"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perbu%2Fostresser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perbu%2Fostresser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perbu%2Fostresser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perbu%2Fostresser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/perbu","download_url":"https://codeload.github.com/perbu/ostresser/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252940904,"owners_count":21828769,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-07T19:15:09.349Z","updated_at":"2025-05-07T19:15:10.136Z","avatar_url":"https://github.com/perbu.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Object Store Stress Tester - Developer Guide\n\n[![License](https://img.shields.io/badge/License-BSD_2--Clause-orange.svg)](https://opensource.org/licenses/BSD-2-Clause)\n\nThis document provides developer-specific information about the Go object store stress tester, focusing on its\nconfiguration, programmatic usage, and internal design.\n\n## Install\n```bash\ngo install github.com/perbu/ostresser@latest\n```\n\n\n## Example output.\n\n```text\n--- Stress Test Summary --- (7.556s) ---\nOverall:\n  Total Requests: 10000 (1323.46 req/s)\n  Total Success:  10000\n  Total Errors:   0\n\nGET Operations (0 total):\n  Success:        0\n  Bytes D/L:      0 (0.00 MiB)\n  Avg Throughput: 0.00 MiB/s\n  No successful GETs to calculate latency.\n\nPUT Operations (10000 total):\n  Success:        10000\n  Bytes U/L:      10485760000 (10000.00 MiB)\n  Avg Throughput: 1323.46 MiB/s\n  Latency (ms): |   Min  |   Avg  |   P50  |   P90  |   P99  |   Max\n  --------------|--------|--------|--------|--------|--------|--------\n  TTLB (total)  |   3.21 |   7.50 |   6.18 |  11.59 |  27.16 |  84.92\n----------------------------------------\nDetailed results written to stress_results.csv\ntime=2025-04-01T14:00:16.933+02:00 level=INFO msg=\"Stress test completed successfully\"\n```\n\n## Manifest File (manifest.txt)\n\nThe manifest file provides a list of object keys that the stress tester will interact with during 'read' or 'mixed'\noperations.\n\nFormat:\n\n* Plain text file.\n* Each line contains one object key.\n* Leading/trailing whitespace around keys is automatically trimmed.\n* Empty lines are ignored.\n* Lines containing only whitespace are ignored.\n\nExample (manifest.txt):\n\n```text\n\n\npath/to/object1.dat\nimages/archive.zip\nanother/key/with/leading/space.txt\nvideos/important_video.mp4\n\nbackup-2024-01-15.tar.gz\ntrailing-space-key.log \n```\n\n### Write Mode and Manifest File\n\nIn write mode, there are two ways to use the manifest file:\n\n1. **Continuous Generation**: The stress tester will continuously generate and upload objects with random keys for the\n   duration of the test.\n    - The manifest file path is still required as a command-line argument.\n    - By default, it will write all successfully uploaded object keys to the manifest file.\n    - Use `-genmf=false` to disable writing to the manifest file.\n\n2. **Fixed File Count Generation**: You can generate a specific number of files using the `-files` flag.\n    - Example: `-files 1000` will generate 1000 files with random keys.\n    - By default, all successfully uploaded keys will be written to the manifest file.\n    - The test will exit after all files have been generated and uploaded.\n    - File size is controlled with the `-putsize` flag (in KB).\n\n## Configuration options\n\n### 1. S3 Connection Details\n\nThese parameters define how to connect to the S3-compatible object storage service.\n\n* **`endpoint` (YAML) / `AWS_ENDPOINT_URL` (Env)**\n   * **Description:** The full URL of the S3-compatible endpoint (e.g., `http://localhost:9000` or `https://s3.amazonaws.com`).\n   * **Required:** Yes (must be set via YAML or Environment Variable).\n   * **Type:** `string`\n\n* **`region` (YAML) / `AWS_REGION` (Env)**\n   * **Description:** The AWS region associated with the endpoint. This is often required by the AWS SDK for proper signing and functioning, even when using a non-AWS S3-compatible endpoint.\n   * **Required:** No (Defaults to `us-east-1` if not set).\n   * **Type:** `string`\n   * **Default:** `us-east-1`\n\n* **`bucket` (YAML) / `S3_BUCKET` (Env)**\n   * **Description:** The name of the S3 bucket to target for the stress test operations. Note the specific environment variable `S3_BUCKET` is used.\n   * **Required:** Yes (must be set via YAML or Environment Variable).\n   * **Type:** `string`\n\n* **`accessKey` (YAML) / `AWS_ACCESS_KEY_ID` (Env)**\n   * **Description:** The access key credential for authenticating with the S3 service.\n   * **Required:** No (Optional. If not provided, the SDK may attempt to use other credential sources like IAM instance profiles or shared credential files).\n   * **Type:** `string`\n\n* **`secretKey` (YAML) / `AWS_SECRET_ACCESS_KEY` (Env)**\n   * **Description:** The secret key credential for authenticating with the S3 service.\n   * **Required:** No (Optional, typically provided alongside `accessKey`).\n   * **Type:** `string`\n\n* **`insecureSkipVerify` (YAML) / `STRESSER_INSECURE_SKIP_VERIFY` (Env)**\n   * **Description:** If set to `true`, TLS certificate verification for the S3 endpoint will be skipped. Use with caution, primarily for testing with self-signed certificates. The environment variable must be set to the string `\"true\"` or `\"false\"`.\n   * **Required:** No (Defaults to `false`).\n   * **Type:** `bool`\n   * **Default:** `false`\n\n---\n\n### 2. Test Parameters\n\nThese parameters control the execution of the stress test itself. Most are primarily set via command-line flags/arguments.\n\n* **`Duration` (Flag `-d`)**\n   * **Description:** Specifies the total duration the stress test should run (e.g., \"10s\", \"5m\", \"1h\").\n   * **Required:** Yes (must be set via flag).\n   * **Type:** `string` (parsed into a duration)\n   * **Source:** Command-line flag (`-d`) only.\n\n* **`Concurrency` (Flag `-c`)**\n   * **Description:** The number of concurrent workers (goroutines) performing S3 operations.\n   * **Required:** Yes (must be set via flag and be \u003e 0).\n   * **Type:** `int`\n   * **Source:** Command-line flag (`-c`) only.\n\n* **`Randomize` (Flag `--randomize`)**\n   * **Description:** If set, the order of keys read from the manifest file will be randomized for each worker. If false, each worker processes a distinct, sequential chunk of the manifest.\n   * **Required:** No (Defaults to `false` likely, based on typical flag handling).\n   * **Type:** `bool`\n   * **Source:** Command-line flag (`--randomize`) only.\n\n* **`ManifestPath` (Positional Argument)**\n   * **Description:** The path to the input manifest file. This file should contain a list of object keys (one per line) to be used for `read` or `mixed` operations. In `write` mode, if `generateManifest` is true, this is the *output* path where generated keys will be written.\n   * **Required:** Yes (must be provided as a command-line argument).\n   * **Type:** `string`\n   * **Source:** Command-line argument only.\n\n* **`OutputFile` (Flag `-o`)**\n   * **Description:** The path to the CSV file where the results (performance metrics) of the stress test will be written.\n   * **Required:** Yes (must be set via flag).\n   * **Type:** `string`\n   * **Source:** Command-line flag (`-o`) only.\n\n* **`OperationType` (Flag `-op`, YAML `operationType`, Env `STRESSER_OPERATION_TYPE`)**\n   * **Description:** Specifies the type of S3 operations to perform. Valid values are `\"read\"` (GET objects), `\"write\"` (PUT objects), or `\"mixed\"` (both GET and PUT objects). Values are case-insensitive but normalized to lowercase.\n   * **Required:** No (Defaults to `read`).\n   * **Type:** `string`\n   * **Valid Values:** `read`, `write`, `mixed`\n   * **Default:** `read`\n\n* **`PutObjectSizeKB` (Flag `-putsize`, YAML `putObjectSizeKB`, Env `STRESSER_PUT_SIZE_KB`)**\n   * **Description:** The size (in Kilobytes) of the objects to create when the `operationType` is `\"write\"` or `\"mixed\"`. Must be greater than 0 in these modes.\n   * **Required:** Yes, if `operationType` is `write` or `mixed`.\n   * **Type:** `int`\n   * **Default:** `1024` (1 MiB)\n\n---\n\n### 3. File Generation Parameters (Write Mode)\n\nThese parameters are specifically used when `operationType` is set to `write`.\n\n* **`FileCount` (Flag `-filecount`, YAML `fileCount`, Env `STRESSER_FILE_COUNT`)**\n   * **Description:** The number of unique object keys (and thus files) to generate and potentially upload if running in `write` mode. This is used to determine how many PUT operations to attempt if generating data.\n   * **Required:** No (Defaults to `1000`).\n   * **Type:** `int`\n   * **Default:** `1000`\n\n* **`GenerateManifest` (Flag `--generate-manifest / --no-generate-manifest` , YAML `generateManifest`, Env `STRESSER_GENERATE_MANIFEST`)**\n   * **Description:** Controls whether the list of generated object keys (determined by `FileCount`) should be written to the file specified by `ManifestPath` when running in `write` mode. The environment variable must be set to the string `\"true\"` or `\"false\"`. Flags likely control this boolean directly.\n   * **Required:** No (Defaults to `true`).\n   * **Type:** `bool`\n   * **Default:** `true`\n\n---\n\n### 4. Logging Configuration\n\n* **`LogLevel` (Flag `-loglevel`, YAML `logLevel`, Env `STRESSER_LOG_LEVEL`)**\n   * **Description:** Controls the verbosity of the application's logging output. Valid values are `\"debug\"`, `\"info\"`, `\"warn\"`, `\"error\"`. Values are case-insensitive but normalized to lowercase.\n   * **Required:** No (Defaults to `info`).\n   * **Type:** `string`\n   * **Valid Values:** `debug`, `info`, `warn`, `error`\n   * **Default:** `info`\n\n\n\n## Programmatic Usage (within the same module)\n\nWhile the tool is primarily designed as a command-line application, its core logic in the internal/stresser package can\nbe invoked programmatically from other Go code within the same module.\n\nSteps:\n\n* Create Configuration: Instantiate and populate the stresser.Config struct. You can manually set fields instead of\n  relying on file/env/flag parsing.\n* Create Context: Set up a context.Context, potentially with a timeout or cancellation signal.\n* Call RunStressTest: Invoke the main execution function, passing the context and config.\n* Process Results: Handle the returned results slice ([]stresser.Result) and statistics (*stresser.Stats).\n\nExample Snippet:\n\n```go\npackage main // Or your package\n\nimport (\n   \"context\"\n   \"log\"\n   \"os\"\n   \"time\"\n\nAdjust import path\n\n)\n\nfunc runProgrammatically() {\n   // 1. Configure the test manually\n   cfg := \u0026stresser.Config{\n      // Connection Details\n      Endpoint:           \"http://localhost:9000\", // Example endpoint\n      Region:             \"us-east-1\",\n      Bucket:             \"my-test-bucket\",\n      AccessKey:          \"minioadmin\", // Example credentials\n      SecretKey:          \"minioadmin\",\n      InsecureSkipVerify: true, // Example: Allow self-signed certs\n\n      // Test Parameters (set directly)\n      Duration:        \"15s\", // Use string format expected by time.ParseDuration internally\n      Concurrency:     5,\n      Randomize:       true,\n      ManifestPath:    \"path/to/your/manifest.txt\", // Still needed for read/mixed\n      OutputFile:      \"programmatic_results.csv\",  // Where to save CSV\n      OperationType:   \"mixed\",                     // \"read\", \"write\", or \"mixed\"\n      PutObjectSizeKB: 256,                         // 256 KB uploads\n\n   }\n\n   // Basic validation after manual setup\n   if err := cfg.Validate(); err != nil {\n      log.Fatalf(\"Manual configuration validation failed: %v\", err)\n   }\n\n   // 2. Create a context with timeout\n   ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) // Overall timeout slightly longer than test\n   duration\n   defer cancel()\n\n   log.Println(\"Starting programmatic stress test run...\")\n\n   // 3. Run the stress test\n   results, stats, err := stresser.RunStressTest(ctx, cfg)\n   if err != nil {\n      // Handle potential errors (excluding expected context cancellations)\n      if ctx.Err() == context.Canceled || ctx.Err() == context.DeadlineExceeded {\n         log.Printf(\"Test run ended via context: %v\", ctx.Err())\n      } else {\n         log.Fatalf(\"Stress test execution failed: %v\", err)\n      }\n   }\n\n   log.Printf(\"Programmatic run finished. Collected %d results.\", len(results))\n\n   // 4. Process results and stats (example: print summary)\n   if stats != nil {\n      stats.PrintSummary(os.Stdout) // Print summary to console\n   } else {\n      log.Println(\"Stats object is nil.\")\n   }\n\n   // You can also iterate through 'results' or save the CSV manually if needed\n   if len(results) \u003e 0 {\n      // Example: Manually trigger CSV write if needed (RunStressTest usually does this)\n      // if err := stresser.WriteResultsCSV(results, cfg.OutputFile); err != nil {\n      // log.Printf(\"Error writing results CSV: %v\", err)\n      // }\n      // log.Printf(\"First result TTFB (ms): %.3f\", stresser.ms(results[0].TTFB)) // Requires helper func\n   } else {\n      log.Println(\"No results collected.\")\n   }\n\n}\n\n``` \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperbu%2Fostresser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperbu%2Fostresser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperbu%2Fostresser/lists"}