{"id":44791450,"url":"https://github.com/raphaelpor/katt","last_synced_at":"2026-02-19T14:01:04.576Z","repository":{"id":338686126,"uuid":"1156540117","full_name":"raphaelpor/katt","owner":"raphaelpor","description":"Katt is a lightweight testing framework for running AI Evals.","archived":false,"fork":false,"pushed_at":"2026-02-18T00:58:49.000Z","size":446,"stargazers_count":1,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-18T15:43:35.477Z","etag":null,"topics":["agent","ai","cli","copilot","evals"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/raphaelpor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-02-12T19:06:42.000Z","updated_at":"2026-02-17T23:10:37.000Z","dependencies_parsed_at":"2026-02-17T12:00:51.440Z","dependency_job_id":"42eaf336-067e-43e8-8a25-5653fdc1fbc1","html_url":"https://github.com/raphaelpor/katt","commit_stats":null,"previous_names":["raphaelpor/katt"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/raphaelpor/katt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raphaelpor%2Fkatt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raphaelpor%2Fkatt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raphaelpor%2Fkatt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raphaelpor%2Fkatt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/raphaelpor","download_url":"https://codeload.github.com/raphaelpor/katt/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raphaelpor%2Fkatt/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29616953,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-19T13:04:20.082Z","status":"ssl_error","status_checked_at":"2026-02-19T13:03:33.775Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","ai","cli","copilot","evals"],"created_at":"2026-02-16T11:20:32.980Z","updated_at":"2026-02-19T14:01:04.535Z","avatar_url":"https://github.com/raphaelpor.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Katt\n[![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/raphaelpor/katt/blob/main/LICENSE) [![npm version](https://img.shields.io/npm/v/katt.svg?style=flat)](https://www.npmjs.com/package/katt)\n\nKatt is a lightweight testing framework for running AI Evals, inspired by [Jest](https://github.com/jestjs/jest).\n\n\u003cimg src=\"https://raw.githubusercontent.com/raphaelpor/katt/main/docs/logo.png\" alt=\"Katt logo\" width=\"250\" /\u003e\n\n## Table of Contents\n\n- [Overview](#overview)\n- [API Documentation](#api-documentation)\n- [Hello World - Example](#hello-world---example)\n- [Main Features](#main-features)\n- [Usage](#usage)\n- [Installation](#installation)\n- [Basic Usage](#basic-usage)\n- [Using promptFile](#using-promptfile)\n- [Specifying AI Models](#specifying-ai-models)\n- [Development](#development)\n- [Setup](#setup)\n- [Available Scripts](#available-scripts)\n- [Verification Process](#verification-process)\n- [Project Structure](#project-structure)\n- [How It Works](#how-it-works)\n- [Requirements](#requirements)\n- [License](#license)\n- [Contributing](#contributing)\n\n## Overview\n\nKatt is designed to evaluate and validate the behavior of AI agents like **Claude Code**, **GitHub Copilot**, **OpenAI Codex** and more. It provides a simple, intuitive API for writing tests that interact with AI models and assert their responses.\n\n## API Documentation\n\nFor a complete list of features and usage examples, see [docs/api-documentation.md](https://github.com/raphaelpor/katt/blob/main/docs/api-documentation.md).\n\n## Hello World - Example\n\n```typescript\nimport { expect, prompt } from \"katt\";\n\nconst result = await prompt(\"If you read this just say 'hello world'\");\nexpect(result).toContain(\"hello world\");\n```\n\nIt also supports the familiar `describe` and `it` syntax for organizing tests:\n\n```typescript\nimport { describe, expect, it, prompt } from \"katt\";\n\ndescribe(\"Greeting agent\", () =\u003e {\n  it(\"should say hello world\", async () =\u003e {\n    const result = await prompt(\"If you read this just say 'hello world'\");\n    expect(result).toContain(\"hello world\");\n  });\n});\n```\n\n## Main Features\n\n- **Simple Testing API**: Familiar `describe` and `it` syntax for organizing tests\n- **AI Interaction and Verification**: Built-in `prompt()`, `promptFile()` and `promptCheck()` functions for running and analyzing prompts to AI agents\n- **Classification Matcher**: Built-in `toBeClassifiedAs()` matcher to grade a response against a target label on a 1-5 scale\n- **Concurrent Execution**: Runs eval files concurrently for faster test execution\n- **Model Selection**: Support for specifying custom AI models\n- **Configurable Timeouts**: Override prompt wait time per test or via `katt.json`\n\n## Usage\n\n### Installation\n\n```bash\nnpm install -g katt\n```\n\n### Basic Usage\n\n1. Create a file with the `.eval.ts` or `.eval.js` extension and write your tests.\n```typescript\nimport { expect, prompt } from \"katt\";\n\nconst result = await prompt(\"If you read this just say 'hello world'\");\nexpect(result).toContain(\"hello world\");\n```\n\n2. Run Katt from your project directory:\n\n```bash\nkatt\n```\n\n### Using promptFile\n\nLoad prompts from external files:\n\n```javascript\n// test.eval.js\nimport { describe, expect, it, promptFile } from \"katt\";\n\ndescribe(\"Working with files\", () =\u003e {\n  it(\"should load the file and respond\", async () =\u003e {\n    const result = await promptFile(\"./myPrompt.md\");\n    expect(result).toContain(\"expected response\");\n  });\n});\n```\n\n### Specifying AI Models\n\nYou can specify a custom model for your prompts:\n\n```javascript\nimport { describe, expect, it, prompt } from \"katt\";\n\ndescribe(\"Model selection\", () =\u003e {\n  it(\"should use a specific model\", async () =\u003e {\n    const promptString = \"You are a helpful agent. Say hi and ask what you could help the user with.\";\n    const result = await prompt(promptString, { model: \"gpt-5.2\" });\n\n    expect(result).promptCheck(\"It should be friendly and helpful\");\n  });\n});\n```\n\nYou can also set a default model for the project by adding a `katt.json` file in the project root:\n\n```json\n{\n  \"copilot\": {\n    \"model\": \"gpt-5-mini\"\n  },\n  \"prompt\": {\n    \"timeoutMs\": 240000\n  }\n}\n```\n\nWhen this file exists:\n\n- `prompt(\"...\")` and `promptFile(\"...\")` use `copilot.model` by default\n- `prompt(\"...\", { model: \"...\" })` still overrides the config value\n- `prompt.timeoutMs` sets the default wait timeout for long-running prompts\n\n## Development\n\n### Setup\n\n```bash\nnpm install\n```\n\n### Available Scripts\n\n- `npm run dev` - Run the CLI in development mode\n- `npm run build` - Build the project\n- `npm run test` - Run tests\n- `npm run typecheck` - Run TypeScript type checking\n- `npm run format` - Format code using Biome\n- `npm run lint` - Lint code using Biome\n- `npm run test:build` - Test the built CLI\n\n### Verification Process\n\nAfter making changes, run the following sequence:\n\n1. `npm run format`\n2. `npm run typecheck`\n3. `npm run test`\n4. `npm run build`\n5. `npm run test:build`\n\n## Project Structure\n\n```\nkatt/\n├── src/              # Source code\n│   ├── cli/          # CLI implementation\n│   ├── lib/          # Core libraries (describe, it, expect, prompt)\n│   └── types/        # TypeScript type definitions\n├── examples/         # Example eval files\n├── specs/            # Markdown specifications\n├── package.json      # Package configuration\n└── tsconfig.json     # TypeScript configuration\n```\n\n## How It Works\n\n1. Katt searches the current directory recursively for `*.eval.js` and `*.eval.ts` files\n2. It skips `.git` and `node_modules` directories\n3. Found eval files are imported and executed concurrently\n4. Tests registered with `describe()` and `it()` are collected and run\n5. Each test duration is printed after execution\n6. A summary is displayed showing passed/failed tests and total duration\n7. Katt exits with code `0` on success or `1` on failure\n\n## Requirements\n\n- Node.js\n- GitHub Copilot CLI installed (see [GitHub Copilot CLI installation docs](https://docs.github.com/en/copilot/how-tos/copilot-cli/install-copilot-cli))\n- Access to AI models (e.g., OpenAI API key for Codex)\n\n## License\n\nMIT\n\n## Contributing\n\nWe welcome contributions from the community! Please see our [CONTRIBUTING.md](https://github.com/raphaelpor/katt/blob/main/CONTRIBUTING.md) guide for detailed information on how to contribute to Katt.\n\nQuick start:\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Run the verification process\n5. Submit a pull request\n\nFor detailed guidelines, development setup, coding standards, and more, check out our [contribution guide](https://github.com/raphaelpor/katt/blob/main/CONTRIBUTING.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraphaelpor%2Fkatt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraphaelpor%2Fkatt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraphaelpor%2Fkatt/lists"}