{"id":34524878,"url":"https://github.com/dpguthrie/scale-test","last_synced_at":"2026-06-01T05:31:55.411Z","repository":{"id":326625345,"uuid":"1106348416","full_name":"dpguthrie/scale-test","owner":"dpguthrie","description":null,"archived":false,"fork":false,"pushed_at":"2025-11-29T21:18:15.000Z","size":201,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-25T16:28:47.925Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpguthrie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-29T04:24:30.000Z","updated_at":"2025-11-29T21:18:19.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dpguthrie/scale-test","commit_stats":null,"previous_names":["dpguthrie/scale-test"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dpguthrie/scale-test","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fscale-test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fscale-test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fscale-test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fscale-test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpguthrie","download_url":"https://codeload.github.com/dpguthrie/scale-test/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fscale-test/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33762215,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-01T02:00:06.963Z","response_time":115,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-24T05:07:01.668Z","updated_at":"2026-06-01T05:31:55.389Z","avatar_url":"https://github.com/dpguthrie.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AI Observability Scale Test Framework\n\nScale testing framework for AI observability platforms using realistic OpenTelemetry traces.\n\n## Features\n\n- **Framework-free**: Pure Python + OTEL SDK, no agent frameworks\n- **Multi-platform**: Braintrust, LangSmith, OTLP-compatible backends\n- **Configurable scale**: 10-100 req/sec with async execution\n- **Production-ready**: OpenTelemetry Collector support for high-throughput testing\n\n## Quick Start\n\n### Installation\n\n```bash\n# Install dependencies\nuv sync\n\n# Copy configuration file\ncp .env.example .env\n\n# Edit .env with your API key\nnano .env\n```\n\n### Configuration\n\n**All configuration is in a single `.env` file**. This file configures both your test script AND the OTel Collector (if using).\n\n```bash\n# Edit .env file:\n# 1. Add your API key to BRAINTRUST_API_KEY or LANGSMITH_API_KEY\n# 2. Choose deployment mode (see below)\n# 3. Adjust test parameters (concurrency, duration, etc.)\n```\n\nSee `.env.example` for full configuration options with comments.\n\n## Deployment Modes\n\n### Option 1: Direct Export (Simple, Low Scale)\n\nFor quick tests with \u003c 50 traces. Your script connects directly to Braintrust/LangSmith.\n\n**When to use:** Quick validation, low concurrency (\u003c 10 workers), short duration (\u003c 5 minutes)\n\n**Setup (.env):**\n```bash\nOTEL_PLATFORM=braintrust\nOTEL_EXPORTER_OTLP_ENDPOINT=https://api.braintrust.dev/otel/v1/traces\nOTEL_EXPORTER_OTLP_HEADERS=\"Authorization=Bearer sk-YOUR_KEY, x-bt-parent=project_name:YOUR_PROJECT\"\n```\n\n**Run:**\n```bash\nuv run python scripts/run_scale_test.py\n```\n\n**Limitations:**\n- ❌ Script hangs 5-10 minutes on shutdown waiting for spans to export\n- ❌ Timeouts on large tests (500+ traces)\n\n---\n\n### Option 2: Collector Export (Production, High Scale)\n\nFor serious scale testing with 100+ traces. Your script sends spans to a local OpenTelemetry Collector that handles export in the background.\n\n**When to use:** Scale testing (100+ traces), high concurrency (10+ workers), production workloads\n\n**Architecture:**\n```\nYour Script → localhost:4318 (fast) → Collector → Braintrust/LangSmith\n   ↓\nExits in ~5 seconds\n(Collector continues exporting in background)\n```\n\n**Setup:**\n\n1. **Configure .env file:**\n```bash\n# Deployment mode (send to local collector)\nOTEL_PLATFORM=otlp\nOTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces\n# No OTEL_EXPORTER_OTLP_HEADERS needed!\n\n# API keys (for collector to authenticate with platform)\nBRAINTRUST_API_KEY=sk-YOUR_ACTUAL_KEY\nBRAINTRUST_PROJECT=scale-test\n\n# Enable fast shutdown\nOTEL_SKIP_SHUTDOWN=true\n```\n\n2. **Edit collector pipeline** (`otel-collector-config.yaml`):\n\n**IMPORTANT:** Enable ONLY ONE platform pipeline. Having both active sends duplicate spans and causes errors.\n\n```yaml\nservice:\n  pipelines:\n    # For Braintrust (enabled by default)\n    traces/braintrust:\n      receivers: [otlp]\n      processors: [memory_limiter, batch]\n      exporters: [otlphttp/braintrust]\n\n    # For LangSmith (comment out Braintrust above first!)\n    # traces/langsmith:\n    #   receivers: [otlp]\n    #   processors: [memory_limiter, batch]\n    #   exporters: [otlphttp/langsmith]\n```\n\n3. **Start collector:**\n```bash\ndocker-compose up -d\n```\n\n4. **Run test:**\n```bash\nuv run python scripts/run_scale_test.py\n# Script exits in ~5 seconds\n# Collector continues exporting in background\n```\n\n5. **Monitor collector:**\n```bash\ndocker-compose logs -f otel-collector\n```\n\n**Benefits:**\n- ✅ Script exits immediately (\u003c 5 seconds)\n- ✅ Automatic retries with exponential backoff\n- ✅ Built-in rate limiting and backpressure\n- ✅ Can handle 100+ req/s sustained\n- ✅ Spans continue exporting after script exits\n- ✅ 10x faster iteration\n\n---\n\n## Configuration Details\n\n### Single .env File Approach\n\n**One file configures everything:**\n- Your test script reads: `OTEL_PLATFORM`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `SCALE_TEST_*`\n- Docker Compose reads: `BRAINTRUST_API_KEY`, `LANGSMITH_API_KEY` (via `env_file: .env`)\n- Collector reads: API keys passed from Docker Compose as environment variables\n\n### Key Configuration Variables\n\n#### Deployment Mode\n```bash\n# Direct export to platform\nOTEL_PLATFORM=braintrust  # or langsmith\nOTEL_EXPORTER_OTLP_ENDPOINT=https://api.braintrust.dev/otel/v1/traces\nOTEL_EXPORTER_OTLP_HEADERS=\"Authorization=Bearer sk-..., x-bt-parent=project_name:...\"\n\n# OR collector export\nOTEL_PLATFORM=otlp\nOTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces\n# No headers needed!\n```\n\n#### API Keys (for Collector)\n```bash\n# Braintrust\nBRAINTRUST_API_KEY=sk-...\nBRAINTRUST_PROJECT=scale-test\n\n# LangSmith\nLANGSMITH_API_KEY=lsv2_...\nLANGSMITH_PROJECT=scale-test\n```\n\n#### Test Parameters\n```bash\nSCALE_TEST_CONCURRENCY=50     # Number of concurrent workers\nSCALE_TEST_DURATION=300       # Test duration in seconds\nSCALE_TEST_RATE_LIMIT=50      # Max requests per second\n\n# Query mix (must sum to 1.0)\nSCALE_TEST_MIX_SIMPLE=0.40    # Simple queries\nSCALE_TEST_MIX_SEARCH=0.30    # Service searches\nSCALE_TEST_MIX_BOOKING=0.20   # Delegated bookings\nSCALE_TEST_MIX_COMPLEX=0.10   # Complex workflows\n```\n\n#### OTEL BatchSpanProcessor Tuning\n```bash\nOTEL_BSP_SCHEDULE_DELAY=1000         # Export every 1s (for collector)\nOTEL_BSP_MAX_EXPORT_BATCH_SIZE=100   # Large batches for local network\nOTEL_BSP_MAX_QUEUE_SIZE=8192         # Large queue for high concurrency\n```\n\n#### Shutdown Behavior\n```bash\nOTEL_SKIP_SHUTDOWN=false  # Wait for spans to export (direct mode)\nOTEL_SKIP_SHUTDOWN=true   # Exit immediately (recommended for collector mode)\n```\n\n### Understanding the Queue\n\nWhen you see `\"Queue is full, likely spans will be dropped\"`:\n\n- **What queue?** The BatchSpanProcessor queue in YOUR app (not the platform)\n- **Where?** In-memory on your machine\n- **Size:** Configured by `OTEL_BSP_MAX_QUEUE_SIZE` (default: 8192)\n- **Memory usage:** ~4KB per span × queue size (8192 spans ≈ 32MB RAM)\n\n**Solution:** Increase `OTEL_BSP_MAX_QUEUE_SIZE` or use collector mode\n\n### Tuning Guidelines\n\n| Concurrency | Mode      | Schedule Delay | Queue Size | Notes |\n|-------------|-----------|----------------|------------|-------|\n| 1-10        | Direct    | 5000ms         | 4096       | Works for small tests |\n| 10-50       | Direct    | 3000ms         | 8192       | Will timeout on shutdown |\n| 50-100      | Collector | 1000ms         | 8192       | Recommended |\n| 100+        | Collector | 1000ms         | 16384      | High throughput |\n\n---\n\n## Usage\n\n### Running Tests\n\n```bash\n# Basic run\nuv run python scripts/run_scale_test.py\n\n# Quick test\nSCALE_TEST_DURATION=10 SCALE_TEST_CONCURRENCY=5 uv run python scripts/run_scale_test.py\n\n# With debug logging\nLOG_LEVEL=DEBUG uv run python scripts/run_scale_test.py\n```\n\n### Logging Levels\n\n```bash\nLOG_LEVEL=ERROR    # Only errors\nLOG_LEVEL=WARNING  # Warnings and errors\nLOG_LEVEL=INFO     # High-level progress (default)\nLOG_LEVEL=DEBUG    # Detailed execution (every scenario, iteration)\n```\n\n**Troubleshooting tip:** If the script seems stuck, run with `LOG_LEVEL=DEBUG` to see where it's hanging.\n\n### Monitoring Collector\n\n#### Real-time Logs\n```bash\n# View real-time logs\ndocker-compose logs -f otel-collector\n\n# Check container status\ndocker-compose ps\n\n# Restart collector (after editing .env or config)\ndocker-compose restart\n\n# Stop collector\ndocker-compose down\n```\n\n#### Drop Analysis\n\nAfter running tests, analyze dropped spans and platform rejections:\n\n```bash\n./scripts/analyze_collector_drops.sh\n```\n\n**Example Output:**\n```\n🔍 OpenTelemetry Collector Drop Analysis\n==========================================\n\n📉 Dropped Spans Summary:\n------------------------\nTotal spans dropped: 1812\n\n🚫 Platform Rejection Breakdown:\n--------------------------------\nHTTP 413 (Payload Too Large):\n  Occurrences: 1\n  Spans dropped: 112\n\nHTTP 429 (Rate Limited):\n  Occurrences: 172\n  Spans dropped: 0\n\nHTTP 409 (Conflict):\n  Occurrences: 17\n  Spans dropped: 1700\n\n💡 Recommendations:\n⚠️  HTTP 413: Reduce OTEL_BSP_MAX_EXPORT_BATCH_SIZE in collector config\n```\n\n**What the metrics mean:**\n\n- **HTTP 413 (Payload Too Large)**: Batch size exceeds platform's limit (10MB for Braintrust)\n  - **Fix**: Reduce `max_export_batch_size` in `otel-collector-config.yaml` batch processor\n\n- **HTTP 429 (Rate Limited)**: Platform is rate limiting your requests\n  - **Fix**: Reduce test concurrency, increase retry intervals, or contact platform for higher limits\n  - **Note**: Retries usually succeed, so drops are rare unless sustained\n\n- **HTTP 503 (Service Unavailable)**: Platform temporarily unavailable\n  - **Fix**: Usually transient, retries handle it. If persistent, check platform status\n\n- **HTTP 409 (Conflict)**: Duplicate trace IDs or pipeline conflicts\n  - **Fix**: Ensure only ONE platform pipeline is active in collector config\n\n**Comparing Platform Ingestion:**\n\nTo compare ingestion capabilities between Braintrust and LangSmith:\n\n1. Run identical test with Braintrust (enable only `traces/braintrust` pipeline)\n2. Run drop analysis and save results\n3. Switch to LangSmith (disable Braintrust, enable `traces/langsmith` pipeline)\n4. Restart collector: `docker-compose restart`\n5. Run identical test with LangSmith\n6. Run drop analysis and compare:\n   - Total spans dropped\n   - HTTP error distribution (413 vs 429 vs 503)\n   - Memory warnings\n   - Retry success rate\n\n#### Prometheus Metrics (Optional)\n\nThe collector exposes Prometheus metrics on port 8888. To enable detailed telemetry, add this to `otel-collector-config.yaml`:\n\n```yaml\nservice:\n  telemetry:\n    metrics:\n      level: detailed\n      address: :8888\n```\n\nThen query metrics:\n\n```bash\n# View all metrics\ncurl http://localhost:8888/metrics\n\n# Filter for exporter metrics\ncurl http://localhost:8888/metrics | grep exporter\n\n# Filter for span counts\ncurl http://localhost:8888/metrics | grep spans\n```\n\n**Key metrics:**\n- `otelcol_exporter_sent_spans`: Successfully exported spans\n- `otelcol_exporter_send_failed_spans`: Failed span exports\n- `otelcol_processor_batch_batch_send_size`: Batch sizes being sent\n- `otelcol_processor_batch_timeout_trigger`: How often batches timeout\n\n**Note:** The drop analysis script (`analyze_collector_drops.sh`) provides more actionable insights than raw Prometheus metrics.\n\n---\n\n## Built-in Scenarios\n\n1. **simple_query** (40% default): Simple questions (~5K tokens, 5 spans, ~35KB)\n2. **single_service_search** (30% default): Service searches (~18K tokens, 12 spans, ~270KB)\n3. **delegated_booking** (20% default): Specialist delegation (~40K tokens, 25 spans, ~470KB)\n4. **multi_service_complex** (10% default): Multi-service workflows (~185K tokens, 80 spans, ~500KB)\n\n---\n\n## Example Output\n\n```\n📊 Scale Test Results:\n   Duration: 300.0s\n   Total requests: 15,000\n   Success rate: 99.8%\n   Throughput: 50.0 req/s\n\n⏱️  Latency:\n   P50: 45ms\n   P95: 120ms\n   P99: 250ms\n\n📦 Data Volume:\n   Total: 11.2 GB\n   Avg per request: 747 KB\n\n📈 Query Mix Breakdown:\n   simple_query: 6000 traces, P50=30ms, 35KB avg\n   single_service_search: 4500 traces, P50=250ms, 270KB avg\n   delegated_booking: 3000 traces, P50=1200ms, 470KB avg\n   multi_service_complex: 1500 traces, P50=4500ms, 500KB avg\n```\n\n---\n\n## Architecture\n\nThree-layer design:\n1. **Trace Generation**: Declarative scenario definitions with workflow steps\n2. **OTEL Instrumentation**: Automatic span creation with GenAI semantic conventions\n3. **Workload Execution**: Async executor with rate limiting and metrics collection\n\n### Code Structure\n\n```\nsrc/\n├── workflow.py          # Workflow step definitions (LLM, Tool, Delegation, etc.)\n├── scenarios.py         # Pre-defined trace scenarios\n├── instrumentation.py   # OTEL span creation with GenAI conventions\n├── platforms.py         # Platform configurations (Braintrust, LangSmith, etc.)\n├── payloads.py          # Realistic payload generation\n├── metrics.py           # Metrics collection and reporting\n└── executor.py          # Async workload execution with rate limiting\n\nscripts/\n├── run_scale_test.py             # CLI entry point\n└── analyze_collector_drops.sh    # Collector drop analysis tool\n\ntests/\n└── test_*.py            # Comprehensive test suite\n```\n\n---\n\n## Troubleshooting\n\n### Script hangs at shutdown\n**Problem:** \"Shutting down tracer provider...\" hangs for 5+ minutes\n\n**Solution:**\n1. Use collector mode (recommended)\n2. Or set `OTEL_SKIP_SHUTDOWN=true` (may lose some spans)\n3. Or reduce test duration/concurrency\n\n### \"Queue is full\" warnings\n**Problem:** `\"Queue is full, likely spans will be dropped\"`\n\n**Solution:** Increase queue size:\n```bash\nOTEL_BSP_MAX_QUEUE_SIZE=16384\n```\n\n### Collector \"connection refused\"\n**Problem:** Cannot connect to localhost:4318\n\n**Solution:**\n```bash\n# Check collector is running\ndocker-compose ps\n\n# Start if not running\ndocker-compose up -d\n```\n\n### Spans not appearing in platform\n**Problem:** Spans aren't showing up in Braintrust/LangSmith\n\n**Solution:**\n1. Check API key is set in `.env`\n2. Restart collector: `docker-compose restart`\n3. Check collector logs: `docker-compose logs otel-collector`\n4. Look for \"401 Unauthorized\" or \"429 Rate Limit\" errors\n5. Run drop analysis: `./scripts/analyze_collector_drops.sh` to see if spans are being rejected\n\n### Collector dropping spans\n**Problem:** Collector logs show \"Exporting failed. Dropping data\" messages\n\n**Solution:**\n1. Run drop analysis to identify the root cause:\n   ```bash\n   ./scripts/analyze_collector_drops.sh\n   ```\n\n2. Address based on HTTP status code:\n   - **HTTP 413**: Batch size too large\n     - Reduce `max_export_batch_size` in `otel-collector-config.yaml` (try 50 or 10)\n   - **HTTP 429**: Rate limiting\n     - Reduce test concurrency (`SCALE_TEST_CONCURRENCY`)\n     - Increase batch schedule delay in collector config\n   - **HTTP 409**: Multiple pipelines active\n     - Ensure only ONE platform pipeline is uncommented in collector config\n     - Restart collector after changing config\n\n3. For persistent issues, check platform-specific limits:\n   - Braintrust: 10MB per batch, rate limits vary by plan\n   - LangSmith: Contact support for specific limits\n\n### Collector errors on startup\n**Problem:** Collector crashes with config errors\n\n**Solution:**\n1. Check `.env` has API keys set\n2. Verify pipeline is uncommented in `otel-collector-config.yaml`\n3. Check logs: `docker-compose logs otel-collector`\n\n---\n\n## Development\n\n```bash\n# Run all tests\nuv run pytest\n\n# Run specific test file\nuv run pytest tests/test_scenarios.py -v\n\n# Run with output\nuv run pytest -v -s\n```\n\n---\n\n## Performance Comparison\n\n### Direct Export (549 traces, concurrency=50)\n- Test duration: 100s\n- Shutdown wait: 300s (timed out!)\n- **Total time: 400s+**\n- Spans lost: ~10-20%\n\n### Collector Export (same test)\n- Test duration: 100s\n- Shutdown wait: \u003c 5s\n- **Total time: 105s**\n- Spans lost: 0%\n\n**10x faster iteration, 100% data delivery!**\n\n---\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fscale-test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpguthrie%2Fscale-test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fscale-test/lists"}