{"id":29194759,"url":"https://github.com/paradite/model-benchmark-viz","last_synced_at":"2026-02-03T06:01:59.082Z","repository":{"id":289058231,"uuid":"969597691","full_name":"paradite/model-benchmark-viz","owner":"paradite","description":"Model Benchmark Viz","archived":false,"fork":false,"pushed_at":"2025-06-08T07:39:46.000Z","size":6142,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-08T08:26:07.688Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://eval.16x.engineer/","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paradite.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-20T14:14:03.000Z","updated_at":"2025-06-08T07:39:51.000Z","dependencies_parsed_at":"2025-06-08T08:32:20.199Z","dependency_job_id":null,"html_url":"https://github.com/paradite/model-benchmark-viz","commit_stats":null,"previous_names":["paradite/model-benchmark-viz"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/paradite/model-benchmark-viz","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paradite%2Fmodel-benchmark-viz","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paradite%2Fmodel-benchmark-viz/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paradite%2Fmodel-benchmark-viz/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paradite%2Fmodel-benchmark-viz/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paradite","download_url":"https://codeload.github.com/paradite/model-benchmark-viz/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paradite%2Fmodel-benchmark-viz/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263076973,"owners_count":23410164,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-02T04:38:04.024Z","updated_at":"2026-02-03T06:01:59.077Z","avatar_url":"https://github.com/paradite.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Model Benchmark Viz\n\nA project by [16x Eval](https://eval.16x.engineer/) to evaluate different models on their ability to generate visualizations of model benchmarks for coding.\n\nFor evaluation rubrics, check the [eval-data](https://github.com/paradite/eval-data) repo. [Direct link](https://github.com/paradite/eval-data/blob/main/projects/visualization/RUBRICS.md).\n\n## Results\n\n### Claude Opus 4\n\n\u003e Human rating: 9.25/10\n\nClaude 4 Opus generated a great looking visualization, with side-by-side comparison colored by benchmark and a legend. But it lacked labels on the bar chart:\n\n![claude-4-opus](output/claude-opus-4.png)\n\nSource code: [claude-opus-4.html](output/claude-opus-4.html)\n\n### Claude Sonnet 4\n\n\u003e Human rating: 9.25/10\n\nClaude Sonnet 4 generated a visually pleasing visualization that is very similar to Claude 4 Opus:\n\n![claude-sonnet-4](output/claude-sonnet-4.png)\n\nSource code: [claude-sonnet-4.html](output/claude-sonnet-4.html)\n\n### Grok-4\n\n\u003e Human rating: 9.25/10\n\nGrok-4 generated a visually pleasing visualization with side-by-side comparison of two benchmarks for each model, with clear labels on the ranking.\n\nThe colors were by model instead of by benchmark.\n\n![grok-4](output/grok-4.png)\n\nSource code: [grok-4.html](output/grok-4.html)\n\n### Kimi K2\n\n\u003e Human rating: 9.25/10\n\nKimi K2 generated a visually pleasing side-by-side visualization without labels on the bar chart.\n\nAlthough the bars are colored by model instead of by benchmark, different benchmarks have different alpha intensity, which is a nice touch.\n\n![kimi-k2](output/kimi-k2-0711-preview.png)\n\nSource code: [kimi-k2-0711-preview.html](output/kimi-k2-0711-preview.html)\n\n### GPT-5 (High)\n\n\u003e Human rating: 9.25/10\n\nGPT-5 (High) generated a visually pleasing visualization with clear labels on the bar chart and legend.\n\nThe graph was also interactive. Upon hover of the bar, it actually highlights the current model across two different benchmarks for easy comparison.\n\n![gpt-5-high-1](output/gpt-5-high-1.png)\n\n![gpt-5-high-2](output/gpt-5-high-2.png)\n\nSource code: [gpt-5-high.html](output/gpt-5-high.html)\n\n### GPT-4.1\n\n\u003e Human rating: 8.75/10\n\nGPT-4.1 generated a visually pleasing visualization with clear labels on the bar chart and legend:\n\n![GPT-4.1](output/gpt-4.1.png)\n\nSource code: [gpt-4.1.html](output/gpt-4.1.html)\n\n### Gemini 2.5 Pro (GA version)\n\n\u003e Human rating: 8.5/10\n\nGemini 2.5 Pro (GA version) generated a visually pleasing visualization that is great looking with clear labels on the bar chart.\n\n![gemini-2.5-pro-ga](output/gemini-2.5-pro-ga.png)\n\n### GPT OSS 120B\n\n\u003e Human rating: 8.5/10\n\nGPT OSS 120B (via Cerebras) generated a visualization with clear labels on the bar chart.\n\n![gpt-oss-120b-cerebras-1](output/gpt-oss-120b-cerebras-1.png)\n\nSource code: [gpt-oss-120b-cerebras-1.html](output/gpt-oss-120b-cerebras-1.html)\n\n### GPT-5 (Medium)\n\n\u003e Human rating: 8.5/10\n\nGPT-5 (Medium) generated a visualization with clear labels on the bar chart.\n\n![gpt-5-medium](output/gpt-5.png)\n\nSource code: [gpt-5-medium.html](output/gpt-5.html)\n\n### Grok Grok Code Fast 1\n\n\u003e Human rating: 8.25/10\n\nGrok Grok Code Fast 1 generated two completely separate bar charts, one for Aider and one for KCORES, which are visually pleasing.\n\n![grok-code-fast-1](output/grok-code-fast-1.png)\n\nSource code: [grok-code-fast-1.html](output/grok-code-fast-1.html)\n\n### o3\n\n\u003e Human rating: 8/10\n\no3 generated a visualization with clear labels on the bar chart. However, the colors were poorly chosen:\n\n![o3](output/o3.png)\n\nSource code: [o3.html](output/o3.html)\n\n### Gemini 2.5 Pro Preview (06-05)\n\n\u003e Human rating: 8/10\n\nThe Gemini 2.5 Pro Preview [06-05] generated a visualization that is similar to o3, but with better labels on the bar chart. The colors were also pretty bad:\n\n![gemini-2.5-pro-preview-06-05](output/gemini-2.5-pro-preview-06-05.png)\n\nSource code: [gemini-2.5-pro-preview-06-05.html](output/gemini-2.5-pro-preview-06-05.html)\n\n### Claude 3.7\n\n\u003e Human rating: 7.5/10\n\nClaude 3.7 generated scatter plots to compare the results of the models which is a big plus. However it has some formatting issues so the output doesn't look as good as the other visualizations:\n\n![claude-3.7](output/claude-3.7.png)\n\nSource code: [claude-3.7.html](output/claude-3.7.html)\n\nI also ran Claude 3.7 again with the same prompt, but it produced a bar chart visualization:\n\n![claude-3.7-2](output/claude-3.7-2.png)\n\nSource code: [claude-3.7-2.html](output/claude-3.7-2.html)\n\n### Qwen3 Coder (via OpenRouter Alibaba Plus)\n\n\u003e Human rating: 7/10\n\nQwen3 Coder (via OpenRouter Alibaba Plus) generated a visualization with horizontal bars (which is cannot be fit in the page). The legends overlaps with the bar chart, causing minor formatting issues.\n\nThe first run generation was missing a chart, so we re-ran the prompt again.\n\n![qwen3-coder-or-alibaba-plus](output/qwen3-coder-or-alibaba-plus-2.png)\n\nSource code: [qwen3-coder-or-alibaba-plus.html](output/qwen3-coder-or-alibaba-plus-2.html)\n\n### Gemini 2.5 Pro Experimental\n\n\u003e Human rating: 7/10\n\nGemini 2.5 Pro Experimental, despite being ranked as the top model, produced a visualization that is not as good as the other models, with no labels on the bar chart.\n\n![gemini-2.5-pro-exp](output/gemini-2.5-pro-exp.png)\n\nSource code: [gemini-2.5-pro-exp.html](output/gemini-2.5-pro-exp.html)\n\nRe-running the prompt again with Gemini 2.5 Pro Experimental did not produce a better visualization.\n\n### Gemini 2.5 Pro Preview 05-06\n\n\u003e Human rating: 7/10\n\nGemini 2.5 Pro Preview 05-06 generated a visualization that is very similar to the Gemini 2.5 Pro Experimental visualization.\n\n![gemini-2.5-pro-preview-05-06](output/gemini-2.5-pro-preview-05-06.png)\n\nSource code: [gemini-2.5-pro-preview-05-06.html](output/gemini-2.5-pro-preview-05-06.html)\n\n### DeepSeek V3\n\n\u003e Human rating: 7/10\n\nDeepSeek V3 generated a similar visualization to Gemini 2.5 Pro.\n\n![deepseek-v3](output/deepseek-v3.png)\n\nSource code: [deepseek-v3.html](output/deepseek-v3.html)\n\nSource code: [mercury-coder-small.html](output/mercury-coder-small.html)\n\n### Mistral Medium 3\n\n\u003e Human rating: 7/10\n\nMistral Medium 3 generated a typical bar chart visualization.\n\n![mistral-medium-3](output/mistral-medium-3.png)\n\nSource code: [mistral-medium-3.html](output/mistral-medium-3.html)\n\n### Devstral Small\n\n\u003e Human rating: 7/10\n\nDevstral Small generated a visualization that is surprisingly good considering its small model size.\n\n![devstral-small](output/devstral-small.png)\n\nSource code: [devstral-small.html](output/devstral-small.html)\n\n### Gemini 2.5 Pro Preview 03-25\n\n\u003e Human rating: 6/10\n\nGemini 2.5 Pro Preview 03-25 generated a visualization that is very similar to the Gemini 2.5 Pro Experimental visualization, with a minor bug and no labels on the bar chart.\n\n![gemini-2.5-pro-preview-03-25](output/gemini-2.5-pro-preview-03-25.png)\n\nSource code: [gemini-2.5-pro-preview-03-25.html](output/gemini-2.5-pro-preview-03-25.html)\n\n### Horizon Alpha\n\n\u003e Human rating: 5.5/10\n\nHorizon Alpha generated generated an unusual visualzation with a vertical line and two ends representing two benchmarks. The visualization also had some formatting issues that makes it hard to read.\n\nWe gave it a second try, but it generated something very similar again.\n\n![horizon-alpha](output/horizon-alpha.png)\n\nSource code: [horizon-alpha.html](output/horizon-alpha.html)\n\n### DeepSeek-V3.1\n\nDeepSeek-V3.1 generated generated an unusual visualzation with a vertical line and two ends representing two benchmarks. The visualization also had some formatting issues that makes it hard to read.\n\nWe gave it a second try, but it generated something very similar again.\n\n![deepseek-v3.1](output/deepseek-v3.1.png)\n\nSource code: [deepseek-v3.1.html](output/deepseek-v3.1.html)\n\n### Qwen3 235B A22B\n\n\u003e Human rating: 5/10\n\nQwen3 235B A22B (thinking, via Parasail on OpenRouter) generated a visualization that is very small and hard to read.\n\n![qwen3](output/qwen3.png)\n\nSource code: [qwen3.html](output/qwen3.html)\n\nI also tried the non-thinking version of Qwen3 (by adding `/no_think` to the end of the prompt, via Parasail on OpenRouter), it produced a nice radar chart visualization, which is interesting, but not suitable for the task of comparing the performance of the models:\n\n![qwen3-no-think](output/qwen3-no-think.png)\n\nSource code: [qwen3-no-think.html](output/qwen3-no-think.html)\n\n### Mercury Coder Small\n\n\u003e Human rating: 5/10\n\nMercury Coder Small generated a visualization that is buggy and not very readable.\n\nIt does not seem to show both benchmarks, and the labels are not very readable.\n\n![mercury-coder-small](output/mercury-coder-small.png)\n\n### DeepSeek R1-0528-Qwen3-8B\n\n\u003e Human rating: 1/10\n\nDeepSeek R1-0528-Qwen3-8B did not manage to generate a runnable visualization. The code contains bugs that are not easy to fix.\n\nOn second attempt, the model just generated a json file instead of an HTML file. There were some HTML code in the thinking process, but the final output was just a json file.\n\nSource code: [deepseek-r1-0528-qwen3-8b.html](output/deepseek-r1-0528-qwen3-8b.html)\n\n## Evaluation Results\n\nEvaluations are carried out using [16x Eval](https://eval.16x.engineer/). Here is the screenshot:\n\n![16x Eval table](16x-eval-table.png)\n\n## Prompt\n\nThe prompt used to generate the visualization is [PROMPT.md](PROMPT.md).\n\nContext used:\n\n- [results.js](results.js)\n- [sample-index.html](sample-index.html)\n\n## Benchmark Data\n\nThe data is stored in `results.js`.\n\nBenchmark Data Sources:\n\n- [Aider polyglot coding leaderboard](https://aider.chat/docs/leaderboards/)\n- [KCORES LLM Arena](https://github.com/KCORES/kcores-llm-arena)\n\n## Model Inclusion Criteria\n\nTo avoid too much clutter, we employ the following criteria for model inclusion:\n\n- The model must not be superceded by a newer model, where \"newer\" is defined as a model released by the same company with strictly higher performance and similar cost.\n  - For example, o3-mini is superceded by o4-mini by OpenAI, hence it is not included.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparadite%2Fmodel-benchmark-viz","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fparadite%2Fmodel-benchmark-viz","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparadite%2Fmodel-benchmark-viz/lists"}