{"id":30029894,"url":"https://github.com/manticoresoftware/manticore-load","last_synced_at":"2026-03-05T01:02:12.449Z","repository":{"id":269858737,"uuid":"908659669","full_name":"manticoresoftware/manticore-load","owner":"manticoresoftware","description":"Manticore Load Emulator","archived":false,"fork":false,"pushed_at":"2025-12-19T18:22:17.000Z","size":204,"stargazers_count":6,"open_issues_count":3,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-12-22T00:35:38.539Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/manticoresoftware.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-26T16:14:22.000Z","updated_at":"2025-12-19T18:20:57.000Z","dependencies_parsed_at":"2024-12-26T18:18:08.624Z","dependency_job_id":"536eca44-7968-487e-8ac2-3414ef66f2d9","html_url":"https://github.com/manticoresoftware/manticore-load","commit_stats":null,"previous_names":["manticoresoftware/manticore-load"],"tags_count":41,"template":false,"template_full_name":null,"purl":"pkg:github/manticoresoftware/manticore-load","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manticoresoftware%2Fmanticore-load","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manticoresoftware%2Fmanticore-load/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manticoresoftware%2Fmanticore-load/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manticoresoftware%2Fmanticore-load/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/manticoresoftware","download_url":"https://codeload.github.com/manticoresoftware/manticore-load/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manticoresoftware%2Fmanticore-load/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30104218,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T00:38:46.881Z","status":"ssl_error","status_checked_at":"2026-03-05T00:38:45.829Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-06T18:03:11.544Z","updated_at":"2026-03-05T01:02:12.421Z","avatar_url":"https://github.com/manticoresoftware.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# manticore-load\n\nManticore Load Emulator is a powerful tool for testing and benchmarking Manticore Search by simulating various workloads. It supports high concurrency and custom query generation, enabling users to assess performance and optimize configurations.\n\n[![asciicast](https://asciinema.org/a/oKBkXMLGSUHEKCm99IE9LHNhX.svg)](https://asciinema.org/a/oKBkXMLGSUHEKCm99IE9LHNhX)\n\n---\n\n## Features\n\n- **High Concurrency:** Simulates high-load scenarios using multiple threads and processes.\n- **Custom Query Generation:** Allows creation of dynamic queries with configurable patterns.\n- **Helpful Patterns:** Provides helpful patterns for generating random data.\n- **Batch Loading:** Efficiently handles large data insertions or replacements in batches.\n- **Progress Monitoring:** Displays real-time progress and detailed statistics.\n- **Flexible Configuration:** Configurable via command-line arguments for convenience.\n- **Latency and QPS Tracking:** Tracks latency percentiles and queries-per-second (QPS) for performance insights.\n- **Multi-Process Support:** Runs different workloads simultaneously for comprehensive testing.\n- **Histogram-Based Latency Tracking:** Uses histogram-based latency tracking for memory-efficient but approximate percentiles.\n- **Precise Latency Tracking:** Uses precise latency tracking for exact percentiles.\n- **Graceful Shutdown:** Handles CTRL-C interruption gracefully, ensuring clean termination of all processes and threads.\n- **Hyperparameter Testing:** Supports testing multiple thread counts and batch sizes in a single run using comma-separated values (e.g., `--threads=1,2,4,8 --batch-size=1000,10000`).\n- **Spreadsheet-Friendly Output:** In quiet mode, produces semicolon-separated output that can be easily copied to spreadsheet applications for charting and analysis.\n\n---\n\n## Requirements\n\n- `manticore-extra` dev package or release version higher than 6.3.8 installed\n- Linux or MacOS\n\n## Installation\n\nJust install `manticore-extra` dev package or release version higher than 6.3.8. More details here:\n- Release: https://manticoresearch.com/install\n- Dev: https://mnt.cr/dev/nightly\n\n### Installation from source\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/manticoresoftware/manticore-load.git\n   ```\n2. Run:\n   ```\n   cd manticore-load\n   ./manticore-load --help\n   ```\n\n---\n\n## Usage\n\nRun `manticore-load` with a variety of options to simulate workloads:\n\n### Basic Examples\n\n#### Write Example\n\nInsert 1,000,000 documents in batches of 1,000:\n\n```bash\nmanticore-load \\\n  --batch-size=1000 \\\n  --threads=4 \\\n  --total=1000000 \\\n  --init=\"CREATE TABLE test(id bigint, name text, type int)\" \\\n  --load=\"INSERT INTO test(id,name,type) VALUES(\u003cincrement\u003e,'\u003ctext/10/100\u003e',\u003cint/1/100\u003e)\"\n```\n\n#### SELECT Example\n\nRun 1,000 search queries:\n\n```bash\nmanticore-load \\\n  --threads=4 \\\n  --total=1000 \\\n  --load=\"SELECT * FROM test WHERE MATCH('\u003ctext/1/2\u003e')\"\n```\n\n### Advanced Examples\n\n#### Mixed Workload Example\n\nRun a single process with multiple load statements using a weighted distribution:\n\n```bash\nmanticore-load \\\n  --threads=4 \\\n  --total=10000 \\\n  --load=\"SELECT * FROM test WHERE MATCH('\u003ctext/1/2\u003e')\" \\\n  --load=\"SELECT * FROM test WHERE MATCH('\u003ctext/10/20\u003e')\" \\\n  --load-distribution=0.3,0.7\n```\n\nThis runs a single workload that has multiple `--load` templates. For each of the `--total` iterations, `manticore-load` picks one of the templates using `--load-distribution` (defaults to an even split if you omit it), so roughly 30% of the queries come from the first template and 70% from the second. If your templates use `\u003cincrement\u003e`, each template keeps its own counter.\n\nIf you use `--batch-size \u003e 1` with INSERT/REPLACE templates, `--total` still controls how many rows are generated, but the tool buffers rows per template and only emits a batched statement once a template has accumulated `--batch-size` rows. This means the distribution applies to generated rows (not necessarily to the number of SQL statements), and you may see a smaller “final” batch per template at the end of the run.\n\nExample with batching and mixed templates (INSERT + UPDATE):\n\n```bash\nmanticore-load \\\n  --drop \\\n  --init=\"create table t(id int, a int)\" \\\n  --total=10 \\\n  --batch-size=2 \\\n  --load=\"insert into t values(\u003cincrement\u003e, \u003cint/1/10\u003e)\" \\\n  --load=\"update t set a = \u003cint/100/200\u003e where a = \u003cint/1/5\u003e\"\n```\n\nOne possible set of prepared queries (showing how INSERT rows are batched and UPDATEs are interleaved):\n\n```sql\nupdate t set a = 102 where a = 5;\nupdate t set a = 112 where a = 1;\ninsert into t values(1, 8),(2, 4);\ninsert into t values(3, 5),(4, 5);\nupdate t set a = 143 where a = 3;\nupdate t set a = 106 where a = 5;\nupdate t set a = 152 where a = 2;\ninsert into t values(5, 8)\n```\n\n#### Multi-Process Example\n\nRun multiple workloads simultaneously:\n\n```bash\nmanticore-load \\\n  --host=127.0.0.1 \\\n  --port=9306 \\\n  --batch-size=10000 \\\n  --threads=5 \\\n  --total=1000000 \\\n  --drop \\\n  --init=\"CREATE TABLE t(f text, age int, color string, j json)\" \\\n  --load=\"INSERT INTO t(id, f, age, color, j) VALUES(0, '\u003ctext/10/100\u003e', \u003cint/5/90\u003e, '\u003cstring/3/10\u003e', '{\\\"a\\\": \u003cint/1/100\u003e}')\" \\\n  \\\n  --together \\\n  --threads=10 \\\n  --total=10000 \\\n  --load=\"SELECT * FROM t WHERE MATCH('\u003ctext/1/2\u003e')\" \\\n  --together \\\n  \\\n  --threads=10 \\\n  --total=10000 \\\n  --load=\"SELECT * FROM t WHERE MATCH('\u003ctext/10/20\u003e')\"\n```\n\n#### Testing Different Threads\n\nTest different thread counts in a single run:\n\n```bash\nmanticore-load \\\n  --threads=1,2,4,8 \\\n  --total=100000 \\\n  --drop \\\n  --init=\"CREATE TABLE test(name text)\" \\\n  --load=\"SELECT * FROM test WHERE MATCH('\u003ctext/1/2\u003e')\"\n```\n\n#### Comparing Batch Sizes\n\nCompare different batch sizes for inserts:\n\n```bash\nmanticore-load \\\n  --batch-size=100,1000,10000 \\\n  --threads=4 \\\n  --total=1000000 \\\n  --drop \\\n  --init=\"CREATE TABLE test(name text)\" \\\n  --load=\"INSERT INTO test(id,name) VALUES(0,'\u003ctext/10/100\u003e')\"\n```\n\n#### Combined Parameters with Quiet Mode\n\nCombine different threads and batch sizes in a single run and run in --quiet mode to get only the final report in compact format:\n\n```bash\nmanticore-load \\\n  --threads=1,2,4,8 \\\n  --batch-size=100,1000,10000 \\\n  --total=1000000 \\\n  --drop \\\n  --init=\"CREATE TABLE test(name text)\" \\\n  --load=\"INSERT INTO test(id,name) VALUES(0,'\u003ctext/10/100\u003e')\" \\\n  --quiet\n```\n\n#### Quiet Mode with Custom Column\n\nRun load test with a custom column showing batch size:\n\n```bash\nmanticore-load \\\n  --quiet \\\n  --column=custom/abc \\\n  --batch-size=1000 \\\n  --threads=4 \\\n  --total=1000000 \\\n  --drop \\\n  --init=\"CREATE TABLE test(name text)\" \\\n  --load=\"INSERT INTO test(id,name) VALUES(\u003cincrement\u003e,'\u003ctext/10/100\u003e')\"\n```\n\nThis will add a \"batch\" column with value \"1000\" at the beginning of the output table:\n\n```\ncustom      ; Threads ; Batch     ; Time        ; Total Docs  ; Docs/Sec    ; Avg QPS     ; p99 QPS     ; p95 QPS     ; p5 QPS      ; p1 QPS      ; Lat Avg     ; Lat p50     ; Lat p95     ; Lat p99     ;\nabc         ; 4       ; 1000      ; 00:15       ; 1000000     ; 64305       ; 66          ; 171         ; 171         ; 39          ; 39          ; 76.7        ; 77.5        ; 135.0       ; 245.0       ;\n```\n\n#### Query with Delay\n\nRun queries with a 1s delay between them:\n\n```bash\nmanticore-load \\\n  --threads=4 \\\n  --total=1000 \\\n  --delay=1 \\\n  --load=\"SELECT * FROM test WHERE MATCH('\u003ctext/1/2\u003e')\"\n```\n\n#### Quiet Mode with JSON Output\n\nRun load test with JSON output format:\n\n```bash\nmanticore-load \\\n  --quiet \\\n  --json \\\n  --batch-size=1000 \\\n  --threads=4 \\\n  --total=1000000 \\\n  --drop \\\n  --init=\"CREATE TABLE test(name text)\" \\\n  --load=\"INSERT INTO test(id,name) VALUES(\u003cincrement\u003e,'\u003ctext/10/100\u003e')\"\n```\n\nThis will output statistics in JSON format:\n\n```json\n{\n    \"threads\": 4,\n    \"batch_size\": 1000,\n    \"time\": \"00:15\",\n    \"total_operations\": 1000000,\n    \"operations_per_second\": 64305,\n    \"qps\": {\n        \"avg\": 66,\n        \"p99\": 171,\n        \"p95\": 171,\n        \"p5\": 39,\n        \"p1\": 39\n    },\n    \"latency\": {\n        \"avg\": 76.7,\n        \"p50\": 77.5,\n        \"p95\": 135.0,\n        \"p99\": 245.0\n    }\n}\n```\n\nNote: The `--json` option can only be used with `--quiet`. If used without `--quiet`, an error will be shown.\n\n---\n\n## Configuration Options\n\n### Global Options\n\n| Option        | Description                             |\n|---------------|-----------------------------------------|\n| `--host`      | Manticore host (default: `127.0.0.1`)   |\n| `--port`      | Manticore port (default: `9306`)        |\n| `--quiet`     | Suppress output                        |\n| `--verbose`   | Show detailed progress and queries     |\n| `--no-color`  | Disable terminal colorization          |\n| `--column`    | Add custom column in quiet mode output (format: name/value) |\n| `--json`      | Output statistics in JSON format (requires --quiet) |\n\n### Workload-Specific Options\n\n| Option          | Description                                         |\n|------------------|-----------------------------------------------------|\n| `--threads`      | Number of concurrent threads (single value or comma-separated list, e.g., \"1,2,4,8\") |\n| `--total`        | Total queries or documents to process               |\n| `--batch-size`   | Batch size for inserts or replacements (single value or comma-separated list, e.g., \"100,1000,10000\") |\n| `--iterations`   | Number of times to repeat data generation           |\n| `--load`         | SQL command template for the workload (repeatable)  |\n| `--load-distribution` | Comma-separated weights for multiple `--load` values (default: even split) |\n| `--init`         | Initial SQL commands (e.g., `CREATE TABLE`)         |\n| `--drop`         | Drop the table before starting (applies to target table only) |\n| `--delay`        | Delay in seconds between queries (default: 0)  |\n| `--cache-gen-workers` | Number of worker processes for cache generation (default: 1) |\n| `--cache-from-disk` | Stream cache from disk instead of loading into memory |\n\n### Advanced Patterns\n\n| Pattern                    | Description                                      |\n|---------------------------|--------------------------------------------------|\n| `\u003cincrement/N\u003e`           | Auto-incrementing value starting from N (default: 1) |\n| `\u003cstring/N/M\u003e`            | Random string with length between N and M         |\n| `\u003ctext/N/M\u003e`              | Random text with N to M words                    |\n| `\u003ctext/{path}/MIN/MAX\u003e`   | Random text using words from file, MIN to MAX words |\n| `\u003cint/N/M\u003e`               | Random integer between N and M                   |\n| `\u003carray/size_min/size_max/val_min/val_max\u003e` | Array of random integers, size between size_min and size_max, values between val_min and val_max |\n| `\u003carray_float/size_min/size_max/val_min/val_max\u003e` | Array of random floats, size between size_min and size_max, values between val_min and val_max |\n\n---\n\n## Monitoring and Statistics\n\nThe tool provides real-time feedback, including:\n\n- **Time:** Elapsed time since the workload started.\n- **Progress:** Percentage of completed queries or documents.\n- **QPS:** Queries per second.\n- **Latency:** Percentiles and average latencies.\n- **System Metrics:** Threads, CPU usage, and more.\n\n---\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](./LICENSE) file for more information.\n\n---\n\n## Contributing\n\nContributions are welcome! Please open issues or submit pull requests on GitHub.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanticoresoftware%2Fmanticore-load","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmanticoresoftware%2Fmanticore-load","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanticoresoftware%2Fmanticore-load/lists"}