{"id":46533647,"url":"https://github.com/nuxt/nuxt-evals","last_synced_at":"2026-03-06T23:01:51.498Z","repository":{"id":342112106,"uuid":"1090564103","full_name":"nuxt/nuxt-evals","owner":"nuxt","description":"Evals for Nuxt to test AI model competency at Nuxt.","archived":false,"fork":false,"pushed_at":"2026-03-03T13:29:30.000Z","size":674,"stargazers_count":37,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-05T01:47:46.482Z","etag":null,"topics":["ai","evals","nuxt"],"latest_commit_sha":null,"homepage":"https://nuxt.com/evals","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nuxt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-05T20:49:02.000Z","updated_at":"2026-03-04T10:11:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"95231325-ca22-46b2-9fe3-cc9ecee01823","html_url":"https://github.com/nuxt/nuxt-evals","commit_stats":null,"previous_names":["nuxt/nuxt-evals"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/nuxt/nuxt-evals","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nuxt%2Fnuxt-evals","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nuxt%2Fnuxt-evals/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nuxt%2Fnuxt-evals/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nuxt%2Fnuxt-evals/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nuxt","download_url":"https://codeload.github.com/nuxt/nuxt-evals/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nuxt%2Fnuxt-evals/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30202507,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T19:07:06.838Z","status":"ssl_error","status_checked_at":"2026-03-06T18:57:34.882Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","evals","nuxt"],"created_at":"2026-03-06T23:00:43.840Z","updated_at":"2026-03-06T23:01:51.472Z","avatar_url":"https://github.com/nuxt.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Nuxt Evals\n\nAgent evaluations for Nuxt coding tasks, powered by [`@vercel/agent-eval`](https://www.npmjs.com/package/@vercel/agent-eval).\n\n## Setup\n\n```bash\npnpm install\ncp .env.example .env   # requires VERCEL_OIDC_TOKEN and AI_GATEWAY_API_KEY\n```\n\n## Scripts\n\n### `pnpm run eval`\n\nRuns agent evaluations.\n\n```bash\npnpm run eval                          # Run all experiments\npnpm run eval -- claude-opus-4.6       # Run a specific experiment\npnpm run eval:smoke                    # Run smoke test (1 eval per experiment)\npnpm run eval:dry                      # Preview what would run\n```\n\n### `pnpm run export-results`\n\nExports clean results to `agent-results.json`.\n\n```bash\npnpm run export-results                          # Export from all experiments\npnpm run export-results -- claude-opus-4.6       # Export specific experiment\n```\n\n## Models\n\n| Experiment | Agent | Model |\n|------------|-------|-------|\n| `claude-opus-4.6` | `claude-code` | `claude-opus-4-6` |\n| `claude-sonnet-4.5` | `claude-code` | `claude-sonnet-4-5` |\n| `claude-sonnet-4.6` | `claude-code` | `claude-sonnet-4-6` |\n| `cursor-composer-1.5` | `cursor` | `composer-1.5` |\n| `gemini-3-pro-preview` | `gemini` | `gemini-3-pro-preview` |\n| `gemini-3.1-pro-preview` | `gemini` | `gemini-3.1-pro-preview` |\n| `devstral-2` | `opencode` | `vercel/mistral/devstral-2` |\n| `gpt-5.3-codex-xhigh` | `codex` | `gpt-5.3-codex-api-preview?reasoningEffort=xhigh` |\n\n## Eval structure\n\nEach eval is a self-contained Nuxt project in `evals/`. Most evals provide broken or suboptimal starter code that the agent must fix — the prompt describes a symptom without revealing the solution.\n\n```\nevals/nuxt-000-fix-data-fetching/\n├── PROMPT.md          # task given to the agent\n├── EVAL.ts            # vitest assertions (withheld from the agent)\n├── package.json       # Nuxt project manifest\n├── nuxt.config.ts\n├── tsconfig.json\n├── eslint.config.mjs\n├── server/\n│   └── api/\n│       └── greeting.ts\n└── app/\n    ├── app.vue\n    └── pages/\n        └── index.vue  # broken starter code the agent must fix\n```\n\n| File | Purpose |\n|------|---------|\n| `PROMPT.md` | The task prompt sent to the agent |\n| `EVAL.ts` | Test file run after the agent finishes (withheld from agent) |\n| `package.json` | Must have `\"type\": \"module\"` and a `\"build\"` script |\n| Everything else | Source files the agent can see and modify |\n\n## Adding a new eval\n\n1. Create a directory under `evals/` (e.g., `evals/nuxt-015-my-eval/`)\n2. Add `PROMPT.md` with a vague, symptom-based task description (don't reveal the solution)\n3. Add broken or suboptimal starter code in `app/` for the agent to fix\n4. Add `EVAL.ts` with vitest assertions that check for the correct fix and reject anti-patterns\n5. Add `package.json` with `\"type\": \"module\"` and `\"build\": \"nuxt build\"`\n6. Run `pnpm run eval` — it will automatically run the new eval for all models\n\n## Adding a new model\n\n1. Create a config in `experiments/` (e.g., `experiments/my-model.ts`)\n2. Add the display name to `MODEL_NAMES` in `scripts/export-results.ts`\n3. Run `pnpm run eval` — it will automatically run all evals for the new model\n\n## Current evals\n\n### Nuxt (15)\n\n| Eval | Type | Tests |\n|------|------|-------|\n| nuxt-000-fix-data-fetching | fix | Replace onMounted + $fetch with useFetch |\n| nuxt-001-prefer-nuxt-link | fix | Replace `\u003ca href\u003e` with `\u003cNuxtLink to\u003e` |\n| nuxt-002-state-composables | build | State management with useState composable |\n| nuxt-003-page-meta | build | Page meta, useHead, and custom layouts |\n| nuxt-004-error-handling | build | Error handling with NuxtErrorBoundary |\n| nuxt-005-fix-seo-meta | fix | Replace useHead meta arrays with useSeoMeta |\n| nuxt-006-runtime-config | build | Runtime config with public vs private keys |\n| nuxt-007-avoid-redundant-ref | fix | Replace ref + watch with computed for derived state |\n| nuxt-008-fix-exposed-secret | fix | Move private runtimeConfig access to server API route |\n| nuxt-009-cache-api-response | fix | Replace defineEventHandler with defineCachedEventHandler |\n| nuxt-010-fix-watch-fetch | fix | Replace watch + $fetch with useFetch reactive URL |\n| nuxt-011-fix-sequential-fetching | fix | Parallelize sequential await useFetch with Promise.all |\n| nuxt-012-nuxt3-to-nuxt4-migration | fix | Migrate Nuxt 3 directory structure to Nuxt 4 |\n| nuxt-013-prefer-nuxt-image | fix | Replace raw `\u003cimg\u003e` with NuxtImg + @nuxt/image |\n| nuxt-014-prefer-use-cookie | fix | Replace document.cookie with useCookie composable |\n\n### Nuxt Content (2)\n\n| Eval | Type | Tests |\n|------|------|-------|\n| nuxt-content-000-navigation | build | Documentation site with queryCollectionNavigation sidebar |\n| nuxt-content-001-data-collection | build | Data collection (type \"data\") with JSON files |\n\n### Nuxt UI (8)\n\n| Eval | Type | Tests |\n|------|------|-------|\n| nuxt-ui-000-theming | build | Theming with app.config.ts colors and semantic utilities |\n| nuxt-ui-001-fix-raw-html-page | fix | Replace raw HTML with UHeader/UFooter/UPageHero/UPageSection |\n| nuxt-ui-002-dashboard-layout | build | Dashboard with UDashboardGroup/Sidebar/Panel |\n| nuxt-ui-003-fix-raw-form | fix | Replace raw form with UForm + Zod validation |\n| nuxt-ui-004-table | build | Data table with UTable, columns, and search |\n| nuxt-ui-005-modal | build | Modal overlay with UModal and v-model:open |\n| nuxt-ui-006-command-palette | build | Command palette with UCommandPalette and keyboard shortcuts |\n| nuxt-ui-007-dropdown-menu | build | Dropdown menu with grouped items, icons, and onSelect |\n\n## License\n\nSee [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnuxt%2Fnuxt-evals","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnuxt%2Fnuxt-evals","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnuxt%2Fnuxt-evals/lists"}