{"id":47635437,"url":"https://github.com/mondaycom/sensei","last_synced_at":"2026-04-02T00:03:09.751Z","repository":{"id":344669865,"uuid":"1182655226","full_name":"mondaycom/sensei","owner":"mondaycom","description":"Open-source AI agent qualification engine. Test, evaluate, and certify AI agents across professional skills.","archived":false,"fork":false,"pushed_at":"2026-03-19T19:18:18.000Z","size":19083,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-20T02:51:48.267Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mondaycom.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-15T20:00:04.000Z","updated_at":"2026-03-19T19:18:23.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mondaycom/sensei","commit_stats":null,"previous_names":["nymeria-ai/sensei","mondaycom/sensei"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/mondaycom/sensei","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mondaycom%2Fsensei","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mondaycom%2Fsensei/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mondaycom%2Fsensei/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mondaycom%2Fsensei/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mondaycom","download_url":"https://codeload.github.com/mondaycom/sensei/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mondaycom%2Fsensei/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31293159,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T21:15:39.731Z","status":"ssl_error","status_checked_at":"2026-04-01T21:15:34.046Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-02T00:03:08.238Z","updated_at":"2026-04-02T00:03:09.735Z","avatar_url":"https://github.com/mondaycom.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sensei\n\n![CI](https://github.com/mondaycom/sensei/actions/workflows/ci.yml/badge.svg)\n[![npm](https://img.shields.io/npm/v/@mondaycom/sensei-engine)](https://www.npmjs.com/package/@mondaycom/sensei-engine)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n\n**Open-source AI agent qualification engine.**\n\nTest, evaluate, and certify AI agents across professional skills with standardized benchmarks, real-world scenarios, and measurable KPIs.\n\n\u003e *\"Before you hire an agent, ask the Sensei.\"*\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/images/sensei-hero.jpg\" alt=\"Sensei — The open-source qualification engine for AI agents\" width=\"800\" /\u003e\n\u003c/p\u003e\n\n## What is Sensei?\n\nSensei is an open-source framework for evaluating AI agents on real-world professional tasks. It provides:\n\n- **Standardized test suites** for common agent roles (SDR, Support, QA, Content, Data Analysis, etc.)\n- **Three-layer evaluation** — Task execution, Reasoning, Self-improvement\n- **Professional-grade KPIs** — not toy benchmarks, but metrics that matter in production\n- **Pluggable architecture** — bring your own agent, any framework, any model\n- **Machine-readable results** — JSON reports, scores, badges, CI/CD integration\n\n## Quick Start\n\n```bash\n# Install\nnpm install @mondaycom/sensei-engine\n\n# Or use the CLI\nnpm install -g @mondaycom/sensei-cli\n```\n\n### Programmatic Usage\n\n```typescript\nimport { SuiteLoader, Runner, Judge, Comparator, createAdapter } from '@mondaycom/sensei-engine';\n\n// Load a test suite\nconst loader = new SuiteLoader();\nconst suite = await loader.loadFile('./suites/sdr-qualification/suite.yaml');\n\n// Create adapter from suite config (or override)\nconst adapter = createAdapter(suite.agent!);\n\n// Create LLM judge for quality evaluation\nconst judge = new Judge(suite.judge!);\nconst comparator = new Comparator(suite.judge!);\n\n// Run against your agent\nconst runner = new Runner(adapter, {\n  retries: 2,\n  judgeScorer: async (kpi, agentOutput, scenarioInput) =\u003e {\n    const verdict = await judge.evaluate({ kpi, scenarioInput: { prompt: scenarioInput }, agentOutput });\n    return {\n      kpi_id: kpi.id, kpi_name: kpi.name,\n      score: (verdict.score / verdict.max_score) * 100,\n      raw_score: verdict.score, max_score: verdict.max_score,\n      weight: kpi.weight, method: kpi.method, evidence: verdict.reasoning,\n    };\n  },\n});\n\nconst result = await runner.run(suite);\n\n// Output results\nimport { Reporter } from '@mondaycom/sensei-engine';\nconst reporter = new Reporter();\nconsole.log(reporter.toTerminal(result));   // Pretty terminal output\nconsole.log(reporter.toJSON(result));       // Machine-readable JSON\n```\n\n### SDK Usage (Programmatic Suite Building)\n\n```typescript\nimport { SuiteBuilder, scenario, kpi } from '@mondaycom/sensei-sdk';\n\nconst suite = new SuiteBuilder()\n  .id('my-eval')\n  .name('My Agent Evaluation')\n  .version('1.0.0')\n  .agent({ adapter: 'http', endpoint: 'http://localhost:3000' })\n  .judge({ provider: 'openai', model: 'gpt-4o' })\n  .addScenario(scenario('write-email', {\n    layer: 'execution',\n    input: { prompt: 'Write a professional cold email to Sarah Chen, VP Eng at TechCorp.' },\n    kpis: [\n      kpi('personalization', { weight: 0.6, method: 'llm-judge', config: { rubric: '5: Excellent personalization\\n1: Generic', max_score: 5 } }),\n      kpi('length', { weight: 0.4, method: 'automated', config: { type: 'word-count', expected: { min: 80, max: 200 } } }),\n    ],\n  }))\n  .build();\n```\n\n### CLI Usage\n\n```bash\n# Run a full suite against your agent\nsensei run --suite ./suites/sdr-qualification/suite.yaml --target http://localhost:3000\n\n# Run with a specific judge model\nsensei run --suite ./my-suite.yaml --target http://localhost:3000 --judge-model gpt-4o\n\n# Validate a custom suite definition\nsensei validate ./my-suite.yaml\n\n# Generate a new suite template\nsensei init my-suite\n\n# Render a report from a previous JSON result\nsensei report --input ./result.json\n```\n\n## Suite Marketplace\n\nThe [Sensei Suite Marketplace](https://sensei.sh/marketplace) is a community hub for discovering, sharing, and installing evaluation suites.\n\n### Search for suites\n\n```bash\n# Search by keyword\nsensei search \"sdr\"\n\n# Filter by category and sort by rating\nsensei search \"sales\" --category sales --sort rating --limit 5\n```\n\n### Install a suite\n\n```bash\n# Install to local ./suites/ directory\nsensei install sdr-qualification\n\n# Install globally to ~/.sensei/suites/\nsensei install sdr-qualification --global\n\n# Install to a custom path\nsensei install sdr-qualification --output ./my-suites/sdr.yaml\n```\n\nAfter installing, run the suite against your agent:\n\n```bash\nsensei run --suite ./suites/sdr-qualification/suite.yaml --target http://localhost:3000\n```\n\n### Publish a suite\n\n```bash\n# Publish suite.yaml from current directory\nsensei publish --api-key \u003cyour-key\u003e\n\n# Publish a specific file with metadata overrides\nsensei publish --file ./my-suite.yaml --name \"My Suite\" --category sales --tags \"sdr,cold-email\"\n```\n\nYou can also set the `SENSEI_API_KEY` environment variable instead of passing `--api-key` each time.\n\n## Three-Layer Evaluation\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/images/sensei-see-it-in-action.jpg\" alt=\"Sensei — See It In Action\" width=\"800\" /\u003e\n\u003c/p\u003e\n\n### Layer 1: Task Execution (50%)\n*\"Can the agent do the job?\"*\n\nFeed the agent realistic scenarios with clear success criteria. Measure output quality, accuracy, completeness, and speed.\n\n### Layer 2: Conversational Reasoning (30%)\n*\"Can the agent explain its decisions?\"*\n\nAfter task completion, the agent is questioned about its approach. Why did it choose this strategy? What tradeoffs did it consider?\n\n### Layer 3: Self-Improvement (20%)\n*\"Can the agent learn from feedback?\"*\n\nGive the agent specific feedback. Re-run the test. Compare before/after using a comparative judge. Agents that improve score higher.\n\n## Scoring\n\n```\nScenario Score = weighted average of KPI scores\nLayer Score    = average of scenario scores in that layer\nOverall Score  = execution × 0.50 + reasoning × 0.30 + self_improvement × 0.20\n```\n\nNote: If a suite only defines some layers (e.g., only execution), missing layers are excluded and the remaining weights are re-normalized. A suite with only execution scenarios can still achieve 100%.\n\n### Badge Levels\n\n| Badge | Score | Meaning |\n|-------|-------|---------|\n| 🥇 Gold | 90+ | Exceptional, top-tier agent |\n| 🥈 Silver | 75-89 | Solid professional performance |\n| 🥉 Bronze | 60-74 | Meets minimum qualification |\n\n### KPI Scoring Methods\n\n- **Automated** — deterministic checks:\n  - `contains` — output includes expected string\n  - `regex` — output matches regex pattern\n  - `json-schema` — output validates against JSON Schema (via Ajv)\n  - `json-parse` — output is valid JSON\n  - `numeric-range` — output parses to number within range\n  - `word-count` — output word count within range\n  - `function` — custom scoring function via SDK `registerKPI()`\n- **LLM Judge** — an LLM evaluates output quality against a rubric\n- **Comparative Judge** — compares before/after outputs for self-improvement scoring\n\n## Suite Definition (YAML)\n\n```yaml\nid: my-suite\nname: My Test Suite\nversion: \"1.0.0\"\n\nagent:\n  adapter: http\n  endpoint: http://localhost:3000\n  timeout_ms: 60000\n\njudge:\n  provider: openai\n  model: gpt-4o\n  temperature: 0.0\n\ndefaults:\n  timeout_ms: 60000\n  judge_model: gpt-4o\n\nscenarios:\n  - id: basic-task\n    name: Basic Task\n    layer: execution\n    input:\n      prompt: \"Write a professional email\"\n    kpis:\n      - id: quality\n        name: Output Quality\n        weight: 0.5\n        method: llm-judge\n        config:\n          rubric: |\n            5: Excellent — clear, professional, compelling\n            3: Adequate — gets the point across\n            1: Poor — unclear or unprofessional\n          max_score: 5\n      - id: has-subject\n        name: Has Subject Line\n        weight: 0.3\n        method: automated\n        config:\n          type: regex\n          expected: \"^Subject:\"\n      - id: length\n        name: Email Length\n        weight: 0.2\n        method: automated\n        config:\n          type: word-count\n          expected: { min: 50, max: 300 }\n\n  - id: explain-approach\n    name: Explain Approach\n    layer: reasoning\n    depends_on: basic-task\n    input:\n      prompt: \"Explain your approach to the previous task.\"\n    kpis:\n      - id: clarity\n        name: Reasoning Clarity\n        weight: 1.0\n        method: llm-judge\n        config:\n          rubric: |\n            5: Clear, structured, insightful reasoning\n            3: Adequate explanation\n            1: Vague or missing reasoning\n          max_score: 5\n\n  - id: improve-after-feedback\n    name: Improve After Feedback\n    layer: self-improvement\n    depends_on: basic-task\n    input:\n      prompt: \"Redo the original task incorporating this feedback.\"\n      feedback: \"Be more specific and provide concrete examples.\"\n    kpis:\n      - id: improvement\n        name: Improvement Over Original\n        weight: 1.0\n        method: comparative-judge\n        config:\n          comparison_type: improvement\n```\n\n## Packages\n\n| Package | Description |\n|---------|-------------|\n| `@mondaycom/sensei-engine` | Core evaluation engine — loader, runner, scorer, judge, comparator, reporter, adapters |\n| `@mondaycom/sensei-cli` | Command-line interface — `run`, `validate`, `init`, `report`, `install`, `search`, `publish` |\n| `@mondaycom/sensei-sdk` | SDK for building custom suites programmatically + custom KPI functions |\n\n## Architecture\n\n```\npackages/\n├── engine/src/\n│   ├── types.ts          # Core type definitions + constants\n│   ├── schema.ts         # Zod validation schemas\n│   ├── loader.ts         # YAML suite parser + fixture resolution\n│   ├── runner.ts         # Scenario execution orchestrator\n│   ├── scorer.ts         # KPI scoring + layer aggregation\n│   ├── judge.ts          # LLM-as-judge (single + multi-judge)\n│   ├── comparator.ts     # Before/after comparative evaluation\n│   ├── reporter.ts       # JSON + ANSI terminal output\n│   ├── llm-client.ts     # Shared OpenAI-compatible client factory\n│   ├── registry-client.ts # Marketplace registry API client\n│   └── adapters/\n│       ├── types.ts      # Adapter registry + factory\n│       ├── http.ts       # HTTP POST adapter\n│       ├── stdio.ts      # Stdin/stdout JSON-line adapter\n│       ├── openai-compat.ts  # OpenAI-compatible adapter (openai, openclaw aliases)\n│       └── langserve.ts      # LangServe adapter\n├── cli/src/\n│   ├── index.ts          # CLI entry point (commander)\n│   ├── loader.ts         # Suite file loader (YAML + JSON with Zod)\n│   ├── format.ts         # Terminal + HTML report formatting\n│   ├── html-report.ts    # Self-contained dark-theme HTML reports\n│   ├── output.ts         # File output utility\n│   └── commands/\n│       ├── run.ts        # sensei run — execute suite against agent\n│       ├── validate.ts   # sensei validate — check suite YAML\n│       ├── init.ts       # sensei init — scaffold new suite\n│       ├── report.ts     # sensei report — render from JSON result\n│       ├── install.ts    # sensei install — download suite from marketplace\n│       ├── search.ts     # sensei search — search marketplace suites\n│       └── publish.ts    # sensei publish — publish suite to marketplace\n└── sdk/src/\n    ├── index.ts          # Public API exports\n    ├── builder.ts        # SuiteBuilder fluent API + helpers\n    ├── custom-kpi.ts     # Custom KPI function registry\n    └── result-utils.ts   # Filter, compare, summarize results\n```\n\n## Built-In Test Suites\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/images/sensei-test-suites.jpg\" alt=\"Sensei — Built-In Test Suites\" width=\"800\" /\u003e\n\u003c/p\u003e\n\n## Adapters\n\nSensei communicates with agents through adapters:\n\n```typescript\ninterface AgentAdapter {\n  name: string;\n  connect(): Promise\u003cvoid\u003e;\n  healthCheck(): Promise\u003cboolean\u003e;\n  send(input: AdapterInput): Promise\u003cAdapterOutput\u003e;\n  disconnect(): Promise\u003cvoid\u003e;\n}\n\ninterface AdapterInput {\n  prompt: string;\n  context?: Record\u003cstring, unknown\u003e;\n  timeout_ms?: number;\n}\n\ninterface AdapterOutput {\n  response: string;\n  duration_ms: number;\n  metadata?: Record\u003cstring, unknown\u003e;\n  error?: string;\n}\n```\n\nBuilt-in adapters:\n- **HTTP** — POST JSON to an endpoint, get JSON response\n- **Stdio** — Spawn a child process, communicate via stdin/stdout JSON lines\n- **OpenAI-Compatible** (`openai-compat` / `openai` / `openclaw`) — Universal adapter for any OpenAI-compatible `/v1/chat/completions` endpoint (OpenAI, Azure, vLLM, Ollama, OpenClaw, etc.)\n- **LangServe** — Integration with LangChain LangServe deployments via `/invoke` protocol\n\n## Roadmap\n\n- [x] Architecture \u0026 specification\n- [x] Core engine (runner, scorer, loader, reporter)\n- [x] Zod schema validation\n- [x] LLM Judge integration (single + multi-judge)\n- [x] Comparative Judge (before/after self-improvement)\n- [x] HTTP, Stdio, OpenAI-Compatible, LangServe adapters\n- [x] CLI commands (`run`, `validate`, `init`, `report`)\n- [x] SDR test suite with fixtures\n- [x] HTML reporter (dark theme)\n- [x] Terminal reporter (ANSI colors)\n- [x] SDK with fluent SuiteBuilder API\n- [x] Custom KPI function registry\n- [x] 173 unit + integration tests\n- [x] CI/CD workflows\n- [ ] Additional test suites (Support, Content, QA, Data, Developer)\n- [ ] Web dashboard\n- [x] Community suite marketplace (`install`, `search`, `publish` CLI commands)\n- [ ] npm publish to registry\n\n## Contributing\n\nWe welcome contributions! Whether it's new test suites, scoring improvements, or framework adapters — Sensei gets better when the community builds together.\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n## License\n\nMIT — use it, fork it, improve it.\n\n---\n\n*Built by [AgentTalent.ai](https://agentalent.ai) — The managed marketplace for AI agent labor.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmondaycom%2Fsensei","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmondaycom%2Fsensei","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmondaycom%2Fsensei/lists"}