{"id":27351824,"url":"https://github.com/serverlessllm/tracestorm","last_synced_at":"2025-04-12T20:41:27.415Z","repository":{"id":268897991,"uuid":"905795673","full_name":"ServerlessLLM/TraceStorm","owner":"ServerlessLLM","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-21T14:23:13.000Z","size":101,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-26T14:55:49.526Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ServerlessLLM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-19T14:37:36.000Z","updated_at":"2025-03-08T23:35:47.000Z","dependencies_parsed_at":"2024-12-26T17:19:20.795Z","dependency_job_id":"eb77fd3f-e02c-4cb9-9895-77efe1153127","html_url":"https://github.com/ServerlessLLM/TraceStorm","commit_stats":null,"previous_names":["future-xy/llm_load_test","future-xy/tracestorm","serverlessllm/tracestorm"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ServerlessLLM%2FTraceStorm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ServerlessLLM%2FTraceStorm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ServerlessLLM%2FTraceStorm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ServerlessLLM%2FTraceStorm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ServerlessLLM","download_url":"https://codeload.github.com/ServerlessLLM/TraceStorm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248631669,"owners_count":21136554,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-12T20:41:26.454Z","updated_at":"2025-04-12T20:41:27.409Z","avatar_url":"https://github.com/ServerlessLLM.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TraceStorm\n\nTraceStorm is a tool for generating and replaying traces of requests to OpenAI API endpoints. It allows users to simulate load testing by generating requests based on specified patterns and configurations.\n\n## Features\n\n- Generate synthetic traces using various patterns (e.g., uniform, poisson, etc.)\n- Load traces from public datasets (e.g., [Azure LLM Inference dataset](https://github.com/Azure/AzurePublicDataset/blob/master/AzureLLMInferenceDataset2023.md))\n- Replay requests to any OpenAI-compatible LLM service\n- Analyze results and visualize performance metrics\n\n## Installation\n\n```bash\npip install tracestorm\n```\n\n## Usage\n\n### Start an OpenAI-Compatible Server\n\nBefore running the load test, ensure you have an OpenAI-compatible server running. If you haven't already installed `vllm`, you can do so with the following command:\n\n```bash\n# Install vllm if you haven't already\npip install vllm\n\n# Start the server with the desired model\nvllm serve Qwen/Qwen2.5-1.5B-Instruct\n```\n\n### Run the Load Test\n\nOnce the server is running, you can execute the load test using the `tracestorm` command. Here's how to run it with different options:\n\n#### Example Command for Synthetic Trace\n\n```bash\ntracestorm --model \"Qwen/Qwen2.5-1.5B-Instruct\" --rps 5 --pattern uniform --duration 10\n```\n\n#### Example Command for Azure Trace\n\nTo load a trace from the Azure dataset, use:\n\n```bash\ntracestorm --model \"Qwen/Qwen2.5-1.5B-Instruct\" --pattern azure_code\n```\n\n#### Example Command for Loading Prompts from Datasets\n\n```bash\ntracestorm --model \"Qwen/Qwen2.5-1.5B-Instruct\" --duration 30 --datasets-config ./examples/datasets_config_hf.json\n```\n\n\n**Supported Dataset Sources**:\n\n1. Locally stored, pre-processed datasets\n- Refer to `./examples/datasets_config_local.json` for an example configuration.\n- If you want to test loading from local files, please run `./examples/test_data_loader.py` first to download and save two datasets.\n\n2. Remote datasets from Hugging Face\n- Refer to `./examples/datasets_config_hf.json` for an example configuration.\n\n**Sorting Strategy**: Defines how prompts from multiple datasets are ordered\n- random (default): Shuffles prompts randomly.\n- original: Maintains the original order of prompts.\n\nPlease check `./examples/datasets_config_default.json` for required fields in `datasets-config-file`. This file contains placeholders for necessary configurations.\n\n\n### Command Options\n\n- `--model`: Required. The name of the model to use.\n- `--rps`: Optional. Requests per second (default is 1, only used for synthetic patterns).\n- `--pattern`: Optional. Pattern for generating trace. Valid patterns include:\n  - `uniform`: Distributes requests evenly across the duration.\n  - `poisson`: Generates request timings based on a Poisson process.\n  - `random`: Generates requests at random intervals within the duration.\n  - `azure_code`: Loads the Azure inference dataset for code.\n  - `azure_conv`: Loads the Azure inference dataset for conversation.\n- `--duration`: Optional. Duration in seconds (default is 10, only used for synthetic patterns).\n- `--subprocesses`: Optional. Number of subprocesses to use (default is 1).\n- `--base-url`: Optional. OpenAI Base URL (default is `http://localhost:8000/v1`).\n- `--api-key`: Optional. OpenAI API Key (default is `none`).\n- `--seed`: Optional. Random seed for trace pattern reproducibility (default is `none`).\n- `--datasets-config`: Optional. Configuration file for loading prompt messages from provided datasets. Uses `DEFAULT_MESSAGES` is not specified.\n\nMake sure to adjust the parameters according to your testing needs!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserverlessllm%2Ftracestorm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fserverlessllm%2Ftracestorm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserverlessllm%2Ftracestorm/lists"}