{"id":48955656,"url":"https://github.com/opencv/cool-benchmark","last_synced_at":"2026-04-18T00:00:30.357Z","repository":{"id":351796102,"uuid":"1178918367","full_name":"opencv/COOL-Benchmark","owner":"opencv","description":"Benchmark for Graviton-based AWS Cloud Optimized OpenCV Library (COOL)","archived":false,"fork":false,"pushed_at":"2026-04-16T13:34:33.000Z","size":104,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-16T15:33:00.451Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opencv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-11T13:57:46.000Z","updated_at":"2026-04-16T13:34:37.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/opencv/COOL-Benchmark","commit_stats":null,"previous_names":["opencv/cool-benchmark"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/opencv/COOL-Benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencv%2FCOOL-Benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencv%2FCOOL-Benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencv%2FCOOL-Benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencv%2FCOOL-Benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opencv","download_url":"https://codeload.github.com/opencv/COOL-Benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencv%2FCOOL-Benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31950891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-17T17:29:20.459Z","status":"ssl_error","status_checked_at":"2026-04-17T17:28:47.801Z","response_time":62,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-18T00:00:29.366Z","updated_at":"2026-04-18T00:00:30.312Z","avatar_url":"https://github.com/opencv.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# COOL Benchmark : OpenCV on AWS Graviton\n\nA benchmarking system that measures OpenCV image-processing performance across AWS Graviton (ARM64) EC2 instances. It spins up real EC2 instances, runs a configurable image pipeline in volatile memory (no S3/disk bottlenecks), and reports throughput, latency, and cost side-by-side.\n\n---\n\n## How It Works\n\n1. The **orchestrator** (`shared/benchmark-orchestrator.py`) exposes a REST API on port `8080`. It receives benchmark requests from the frontend, launches EC2 instances via the AWS SDK, runs the OpenCV pipeline over SSH/SSM, collects results, and terminates instances when done.\n2. The **frontend** (`frontend/serve.py`) serves a static web UI on port `3000` that lets you configure and trigger benchmarks, watch live progress, and compare results.\n3. The **image search agent** (`agents/image-search-agent.py`) is an MCP server that fetches public images on demand (NASA, medical datasets, etc.) into volatile memory for the benchmark workload.\n\n---\n\n## Prerequisites\n\n- Python 3.10+\n- An AWS account with permission to launch EC2 instances (e.g. :`t4g`, `m7g`, `c7g`, `m6i` families)\n- An EC2 key pair in your target region\n- An [Anthropic API key](https://console.anthropic.com/) for the image search agent\n- The AWS Marketplace OpenCV Graviton AMI ID for your region\n\n---\n\n## Setup\n\n### 1. Clone the repository\n\n```bash\ngit clone https://github.com/opencv/COOL-Benchmark\ncd COOL-Benchmark\n```\n\n### 2. Create and activate a virtual environment\n\n```bash\npython3 -m venv cool\nsource cool/bin/activate        # Linux / macOS\n# cool\\Scripts\\activate         # Windows\n```\n\n### 3. Install dependencies\n\n```bash\npip install -r requirements.txt\n```\n\n### 4. Create your marketplace config\n\n```bash\ncp config-marketplace.example.json config-marketplace.json\n```\n\nOpen `config-marketplace.json` and replace `ami-xxxxxxxxxxxxxxxxx` with the OpenCV Graviton AMI ID from AWS Marketplace for your region. You can also set this later through the UI\n\n---\n\n## Export Environment Variables\n\nSet these before starting either process. Open a terminal and run:\n\n```bash\n# AWS credentials — used by the orchestrator to launch EC2 instances\nexport AWS_ACCESS_KEY_ID=\"YOUR_AWS_ACCESS_KEY_ID\"\nexport AWS_SECRET_ACCESS_KEY=\"YOUR_AWS_SECRET_ACCESS_KEY\"\nexport AWS_DEFAULT_REGION=\"us-east-1\"          # change to your preferred region\n\n# EC2 key pair name — must already exist in the region above\nexport EC2_KEY_PAIR_NAME=\"your-ec2-key-pair-name\"\n\n# Anthropic API key — used by the auto-retry feature to analyse build errors\nexport ANTHROPIC_API_KEY=\"sk-ant-api03-...\"\n\n# (Optional) Pre-configure the Marketplace AMI ID instead of entering it in the UI\n# export MARKETPLACE_AMI_ID=\"ami-xxxxxxxxxxxxxxxxx\"\n```\n\n---\n\n## Run\n\nOpen **two terminals**, both with the environment variables above already exported.\n\n### Terminal 1 : Start the backend orchestrator\n\n```bash\ncd COOL-Benchmark\npython shared/benchmark-orchestrator.py\n```\n\nExpected output:\n```\nINFO  benchmark-orchestrator - Benchmark orchestrator started on http://0.0.0.0:8080\n```\n\n### Terminal 2 : Start the frontend server\n\n```bash\ncd COOL-Benchmark\npython frontend/serve.py\n```\n\nExpected output:\n```\nFrontend server running at http://localhost:3000/\nPress Ctrl+C to stop\n```\n\n### Open the UI\n\nNavigate to [http://localhost:3000](http://localhost:3000) in your browser.\n\n---\n\n## Project Structure\n\n```\nCOOL-Benchmark/\n├── shared/\n│   ├── benchmark-orchestrator.py       # Main backend — REST API + EC2 orchestration\n│   ├── benchmark_executor.py           # Per-instance benchmark runner\n│   ├── build_manager.py                # OpenCV build/install logic on remote instances\n│   ├── auto_retry_manager.py           # LLM-powered auto-retry for build failures\n│   └── shared_instance_benchmark.py\n├── agentcore/\n│   └── instance-manager.py             # EC2 lifecycle management (launch, pool, terminate)\n├── opencv-ami/\n│   ├── opencv-mcp-server.py            # MCP server deployed to every EC2 instance at runtime\n│   └── install-opencv-optimized.sh     # Graviton-optimised OpenCV build script (used by compile-from-source option)\n├── agents/\n│   ├── image-search-agent.py           # MCP server — fetches public images into memory\n│   └── requirements.txt\n├── frontend/\n│   ├── serve.py                        # Static file server (port 3000)\n│   ├── index.html                      # Main UI\n│   ├── app.js                          # Frontend logic\n│   └── styles.css\n├── requirements.txt                    # All Python dependencies\n├── config-marketplace.example.json     # Template — copy to config-marketplace.json and fill in your AMI ID\n└── README.md\n```\n\n---\n\n## Performance Metrics Reported\n\n- **Processing duration** (seconds per scenario)\n- **Throughput** (images per second)\n- **Cost estimate** (USD, live AWS pricing)\n- **Instance count** (active at peak)\n- **Real-time image preview** (cycles through processed results)\n\n---\n\n## Troubleshooting\n\n**Orchestrator fails to launch instances**\n\n- Confirm `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_DEFAULT_REGION` are exported.\n- Verify your IAM user/role has `ec2:RunInstances`, `ec2:DescribeInstances`, `ec2:TerminateInstances`, and `ssm:SendCommand` permissions.\n- Check that `EC2_KEY_PAIR_NAME` matches an existing key pair in the configured region.\n\n**Auto-retry does not start**\n\n- Ensure `ANTHROPIC_API_KEY` is set before submitting a benchmark run.\n\n**Frontend shows \"Cannot connect to backend\"**\n\n- Make sure the orchestrator is running and listening on port `8080` before opening the UI.\n- Both processes must be in the same network context (local machine or same EC2 instance).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopencv%2Fcool-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopencv%2Fcool-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopencv%2Fcool-benchmark/lists"}