{"id":31112074,"url":"https://github.com/comfy-org/comfyui-benchmark","last_synced_at":"2025-09-17T08:52:34.489Z","repository":{"id":313118332,"uuid":"1049982814","full_name":"Comfy-Org/comfyui-benchmark","owner":"Comfy-Org","description":"Custom node pack for benchmarking ComfyUI performance","archived":false,"fork":false,"pushed_at":"2025-09-11T02:34:43.000Z","size":199,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-11T05:50:10.066Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Comfy-Org.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-03T19:23:16.000Z","updated_at":"2025-09-11T02:34:46.000Z","dependencies_parsed_at":"2025-09-04T03:10:16.864Z","dependency_job_id":null,"html_url":"https://github.com/Comfy-Org/comfyui-benchmark","commit_stats":null,"previous_names":["comfy-org/comfyui-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Comfy-Org/comfyui-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Comfy-Org%2Fcomfyui-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Comfy-Org%2Fcomfyui-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Comfy-Org%2Fcomfyui-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Comfy-Org%2Fcomfyui-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Comfy-Org","download_url":"https://codeload.github.com/Comfy-Org/comfyui-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Comfy-Org%2Fcomfyui-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275563723,"owners_count":25487542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-17T02:00:09.119Z","response_time":84,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-17T08:52:31.610Z","updated_at":"2025-09-17T08:52:34.483Z","avatar_url":"https://github.com/Comfy-Org.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# comfyui-benchmark\nCustom node pack for benchmarking ComfyUI performance\n\n## Output\n\nBenchmark results are saved to the `ComfyUI/outputs/benchmarks` directory as JSON files with timestamps in their filenames.\n\n## Configuration\n\nThe benchmark behavior can be configured using `config.yaml`. \n\n### Setting up config.yaml\n\n- On first run, a `config.yaml` file is automatically generated with default settings\n- Alternatively, you can rename `config.yaml.example` to `config.yaml` before the first run to use your preferred settings\n- The configuration is read at the start of each workflow execution, so you can modify settings without restarting ComfyUI\n\n### Configuration Options\n\nThe `config.yaml` file contains the following settings to control benchmark data collection:\n\n- `iteration_times` (default: true) - When set to false, disables logging of individual step execution times\n- `enable_thread` (default: true) - When set to false, disables nvidia-smi data collection during workflow execution (no VRAM, GPU Utilization, or Power Usage data will be recorded)\n- `check_interval` (default: 0.25) - The time interval in seconds between nvidia-smi data collection calls during workflow execution\n\n## Visualization\n\nThe benchmark results can be visualized using the included `visualize_benchmark.py` script.\n\n\u003cimg width=\"3840\" height=\"2160\" alt=\"use-in-readmy\" src=\"https://github.com/user-attachments/assets/7f71f195-5032-4280-9494-2501237b98cc\" /\u003e\n\n\n### Usage\n\n1. Install the required dependency:\n   ```bash\n   pip install -r visualize_requirements.txt\n   ```\n\n2. Run the visualization script with a benchmark file:\n   ```bash\n   python visualize_benchmark.py \u003cbenchmark_file.json\u003e\n   ```\n   \n   For example:\n   ```bash\n   python visualize_benchmark.py benchmark_20250904_140902.json\n   ```\n\n   The benchmark file can be specified with any path - it doesn't need to follow a specific naming format.\n\n3. **Comparing Multiple Benchmarks**: To compare multiple benchmark results, you have several options:\n   \n   Provide multiple JSON files:\n   ```bash\n   python visualize_benchmark.py benchmark1.json benchmark2.json benchmark3.json\n   ```\n   \n   Compare all benchmarks in a directory:\n   ```bash\n   # Current directory\n   python visualize_benchmark.py .\n   \n   # Specific directory\n   python visualize_benchmark.py /path/to/benchmarks/\n   ```\n   \n   Mix directories and individual files:\n   ```bash\n   python visualize_benchmark.py benchmarks_dir/ extra_benchmark.json\n   ```\n   \n   This creates a comparison view showing only the Workflow Operations Timeline for each benchmark, making it easy to compare performance across different runs or configurations. When providing a directory, the script automatically finds and validates all JSON files with proper benchmark structure.\n\n### Command-line Arguments\n\nThe visualization script supports the following optional arguments:\n\n- `--save-html [PATH]` - Save the visualization as an interactive HTML file. If no path is provided, saves in the current directory with `_visualization.html` suffix.\n- `--save-image [PATH]` - Save the visualization as a static image (PNG, JPG, SVG, etc.). If no path is provided, saves in the current directory with `_visualization.png` suffix. **Note: Requires the `kaleido` package (`pip install kaleido`).**\n- `--no-show` - Skip opening the visualization in the browser (useful for batch processing or headless environments).\n\nExamples:\n```bash\n# Open in browser and save as HTML in current directory\npython visualize_benchmark.py benchmark_20250904_140902.json --save-html\n\n# Save as PNG with custom path without opening browser\npython visualize_benchmark.py benchmark_20250904_140902.json --save-image results/benchmark.png --no-show\n\n# Save both HTML and image formats\npython visualize_benchmark.py benchmark_20250904_140902.json --save-html report.html --save-image report.svg\n\n# Compare multiple benchmarks and save as HTML\npython visualize_benchmark.py benchmark1.json benchmark2.json --save-html comparison.html\n\n# Compare all benchmarks in current directory\npython visualize_benchmark.py . --save-html all_benchmarks.html\n\n# Compare benchmarks from a specific directory\npython visualize_benchmark.py ./outputs/benchmarks/ --save-image comparison.png\n```\n\n### Visualization Features\n\nThe script creates an interactive Plotly graph showing:\n\n**Single Benchmark Mode:**\n- **Device Information Table** - GPU details, PyTorch version, startup arguments\n- **VRAM Usage** - Timeline showing memory consumption with initial and maximum VRAM lines (NVIDIA only*)\n- **RAM Usage** - System memory consumption over time (when psutil data is available)\n- **GPU Utilization** - GPU usage percentage over time (NVIDIA only*)\n- **Power Usage** - Power draw with both average and instantaneous values (NVIDIA only*)\n- **Workflow Operations Timeline** - Color-coded bars showing when different operations occurred (model loading, sampling, VAE encode/decode, etc.) with total workflow duration indicator\n\n**Comparison Mode (Multiple Benchmarks):**\n- **Stacked Workflow Operations Timelines** - Each benchmark's operations are displayed in a separate row with:\n  - Shared time axis for easy comparison\n  - Total duration indicators for each benchmark\n  - Consistent color coding across all benchmarks\n  - Aligned to the longest benchmark's duration\n\n*Note: VRAM usage, GPU utilization, and power usage graphs are only available on NVIDIA devices with the `nvidia-smi` command working. The workflow operations timeline will still be displayed for all devices.\n\nThe visualization opens in your default web browser and supports interactive zooming, panning, and hovering for detailed information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcomfy-org%2Fcomfyui-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcomfy-org%2Fcomfyui-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcomfy-org%2Fcomfyui-benchmark/lists"}