{"id":49567329,"url":"https://github.com/javierdejesusda/checkllm","last_synced_at":"2026-05-03T12:02:43.599Z","repository":{"id":349625886,"uuid":"1194783868","full_name":"javierdejesusda/checkllm","owner":"javierdejesusda","description":"The pytest of LLM testing. Test LLM-powered applications with the same rigor as traditional software.","archived":false,"fork":false,"pushed_at":"2026-04-18T11:07:44.000Z","size":1909,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-18T12:25:16.351Z","etag":null,"topics":["ai-compliance","ai-safety","ai-testing","anthropic","hallucination","llm","llm-evaluation","openai","prompt-engineering","pytest","rag","red-teaming"],"latest_commit_sha":null,"homepage":"https://javierdejesusda.github.io/checkllm/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/javierdejesusda.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-28T20:07:20.000Z","updated_at":"2026-04-18T11:07:47.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/javierdejesusda/checkllm","commit_stats":null,"previous_names":["javierdejesusda/checkllm"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/javierdejesusda/checkllm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/javierdejesusda%2Fcheckllm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/javierdejesusda%2Fcheckllm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/javierdejesusda%2Fcheckllm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/javierdejesusda%2Fcheckllm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/javierdejesusda","download_url":"https://codeload.github.com/javierdejesusda/checkllm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/javierdejesusda%2Fcheckllm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32568036,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T06:36:36.687Z","status":"ssl_error","status_checked_at":"2026-05-03T06:36:09.306Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-compliance","ai-safety","ai-testing","anthropic","hallucination","llm","llm-evaluation","openai","prompt-engineering","pytest","rag","red-teaming"],"created_at":"2026-05-03T12:02:42.837Z","updated_at":"2026-05-03T12:02:43.584Z","avatar_url":"https://github.com/javierdejesusda.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CheckLLM: Reproducible Agent-Trajectory Evaluation\n\nA deterministic, judge-free metric for scoring agent tool-call trajectories -- with AUROC 0.93 against synthetic ground truth and ~1500x faster than DeepEval's `ToolCorrectnessMetric`.\n\n[![PyPI](https://img.shields.io/pypi/v/checkllm)](https://pypi.org/project/checkllm/) [![Python](https://img.shields.io/pypi/pyversions/checkllm)](https://pypi.org/project/checkllm/) [![License](https://img.shields.io/pypi/l/checkllm)](https://github.com/javierdejesusda/checkllm/blob/main/LICENSE) [![CI](https://github.com/javierdejesusda/checkllm/actions/workflows/ci.yml/badge.svg)](https://github.com/javierdejesusda/checkllm/actions/workflows/ci.yml) [![arXiv](https://img.shields.io/badge/arXiv-XXXX.YYYYY-b31b1b.svg)](https://arxiv.org/abs/XXXX.YYYYY) [![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.PLACEHOLDER-blue)](https://doi.org/10.5281/zenodo.PLACEHOLDER) [![Benchmark](https://img.shields.io/badge/leaderboard-rank%201-brightgreen)](docs/benchmarks/competitor-comparison.md)\n\n```bash\npip install checkllm\n```\n\n```python\nfrom checkllm.metrics.trajectory_metric import TrajectoryMetric\n\n# Expected plan vs. what the agent actually did\nexpected = [\"search\", \"fetch\", \"parse\", \"respond\"]\nactual = [\"search\", \"fetch\", \"parse\", \"fetch\", \"respond\"]\n\nmetric = TrajectoryMetric(expected_trajectory=expected)\nsub = metric.compute_subscores(actual)\n\nprint(f\"ordering   {sub.ordering:.2f}\")   # 0.80\nprint(f\"loops      {sub.loops:.2f}\")      # 1.00\nprint(f\"coverage   {sub.coverage:.2f}\")   # 1.00\nprint(f\"unexpected {sub.unexpected:.2f}\") # 1.00\nprint(f\"overall    {sub.overall:.2f}\")    # 0.92\n```\n\nNo judge LLM. No API key. Bit-identical scores across runs. See [the 10-minute tutorial](docs/tutorials/evaluating-agents-in-10-minutes.md) for the full walkthrough.\n\n## Why CheckLLM?\n\n- **Deterministic** -- no judge LLM, no API cost, bit-identical scores across runs and machines.\n- **Composite** -- 4-axis trajectory scoring (ordering, loops, coverage, unexpected), AUROC 0.93 [0.91, 0.94] on 500 trajectories.\n- **OTel-compatible** -- ingest traces from any agent framework via OpenTelemetry GenAI semantic conventions.\n\nBeyond trajectory evaluation, CheckLLM also ships the broader testing suite the project has always provided:\n\n- **Zero learning curve** -- if you know pytest, you know checkllm. Just add a `check` parameter.\n- **39 free deterministic checks** run instantly with zero API calls. No API key needed to start.\n- **72 LLM-as-judge metrics** -- hallucination, faithfulness, trajectory, per-turn, dual-judge, and more.\n- **151 red team vulnerability types** with 25 attack strategies -- the most comprehensive adversarial testing suite available.\n- **17 compliance frameworks** -- OWASP LLM/API/Agentic Top 10, MITRE ATLAS, EU AI Act, ISO 42001, HIPAA, GDPR, and more.\n- **Same checks everywhere** -- use them in tests, CI, and production guardrails.\n\n## Quickstart\n\n### Install\n\n```bash\npip install checkllm\ncheckllm init --use-case rag  # generates a tailored test file\n```\n\n### 1. Deterministic checks (free, instant)\n\n```python\ndef test_basic_quality(check):\n    output = my_llm(\"Summarize this article.\")\n\n    check.contains(output, \"key finding\")\n    check.max_tokens(output, limit=200)\n    check.no_pii(output)\n    check.is_json(output)\n    check.gleu(output, reference=\"Expected summary text.\", threshold=0.5)\n    check.chrf(output, reference=\"Expected summary text.\", threshold=0.4)\n    check.latency_check(start_time, end_time, max_ms=3000)\n    check.cost_check(input_tokens=500, output_tokens=200, model=\"gpt-4o\", max_cost=0.05)\n```\n\n### 2. LLM-as-judge (deeper evaluation)\n\n```python\ndef test_rag_quality(check):\n    output = my_rag(\"What causes climate change?\")\n    context = retrieve_context(\"climate change\")\n\n    check.hallucination(output, context=context)\n    check.faithfulness(output, context=context)\n    check.relevance(output, query=\"What causes climate change?\")\n    check.toxicity(output)\n```\n\n### 3. Fluent chaining\n\n```python\ndef test_with_chaining(check):\n    output = my_llm(\"Explain quantum physics simply.\")\n\n    check.that(output) \\\n        .contains(\"quantum\") \\\n        .max_tokens(200) \\\n        .has_no_pii() \\\n        .scores_above(\"relevance\", 0.8, query=\"quantum physics\")\n```\n\n### 4. Production guardrails\n\n```python\nfrom checkllm import Guard, CheckSpec\n\nguard = Guard(checks=[\n    CheckSpec(check_type=\"no_pii\"),\n    CheckSpec(check_type=\"max_tokens\", params={\"limit\": 500}),\n    CheckSpec(check_type=\"toxicity\"),\n])\n\nresult = guard.validate(llm_output)\nif not result.valid:\n    result.raise_on_failure()\n```\n\n### 5. YAML-based evaluation\n\n```yaml\n# checkllm.yaml\ndescription: \"Customer support chatbot evaluation\"\njudge:\n  backend: openai\n  model: gpt-4o\n\nprompts:\n  - \"You are a helpful support agent. Answer: {{query}}\"\n\ntests:\n  - vars:\n      query: \"How do I return an item?\"\n    assert:\n      - type: contains\n        value: \"return policy\"\n      - type: relevance\n        threshold: 0.8\n      - type: no_pii\n      - type: max_tokens\n        value: 500\n\nsettings:\n  budget: 5.0\n```\n\n```bash\ncheckllm eval-yaml checkllm.yaml\n```\n\n## How checkllm compares\n\n\u003e **Independent benchmark, not just feature counts.** On the public competitor leaderboard\n\u003e ([docs/benchmarks/competitor-comparison.md](docs/benchmarks/competitor-comparison.md))\n\u003e checkllm holds **rank 1 on every published row** against DeepEval and promptfoo:\n\u003e halubench/hallucination 0.783, ragtruth/hallucination 0.663,\n\u003e ragtruth/faithfulness 0.754, ragtruth/context_relevance 0.565, and\n\u003e truthfulqa/answer_relevancy 0.546 (ROC-AUC, gpt-4o-mini judge,\n\u003e 200 source rows per slice). Methodology is in\n\u003e [docs/benchmarks/methodology.md](docs/benchmarks/methodology.md);\n\u003e raw scores ship in `benchmarks/competitor_comparison/`.\n\n### Feature comparison\n\n| Feature | checkllm | DeepEval | Ragas | promptfoo |\n|---------|----------|----------|-------|-----------|\n| pytest native | Yes | Wrapper | No | No |\n| Free deterministic checks | **39** | Limited | Limited | Yes |\n| LLM-as-judge metrics | **72** | ~50 | ~40 | Custom |\n| Red team vulnerability types | **151** | 40+ | 0 | 100+ |\n| Attack strategies | **25** | 10+ | 0 | 30+ |\n| Compliance frameworks | **17** | 3 | 0 | 10+ |\n| Multi-provider judges | **15+ backends** | 13+ | ~6 | 50+ |\n| Consensus judging | **7 strategies** | No | Dual-judge | No |\n| Production guardrails | **Built-in** | No | No | API |\n| Cost control \u0026 budgets | **Built-in** | No | No | Caching |\n| Knowledge Graph synthesis | **Full pipeline** | No | Yes | No |\n| Multilingual prompts | **20 languages** | No | Yes | No |\n| Prompt optimization | **4 algorithms** | 4 | 2 | No |\n| YAML config evaluation | **Yes** | No | No | Yes |\n| Streaming evaluation | **Token-by-token** | No | No | No |\n| Regression detection | **Statistical (p-values)** | No | No | No |\n| DPO export | **Yes** | No | No | No |\n| Telemetry / phoning home | **None** | PostHog + Sentry | None | Telemetry |\n| Independence | **Fully independent** | YC-backed | YC-backed | OpenAI-owned |\n\n## All metrics by category\n\n### RAG Evaluation (14 metrics)\n`hallucination` `faithfulness` `faithfulness_hhem` `context_relevance` `context_entity_recall` `contextual_precision` `contextual_recall` `answer_completeness` `groundedness` `nonllm_context_precision` `nonllm_context_recall` `quoted_spans_alignment` `nv_context_relevance` `nv_response_groundedness`\n\n### General Quality (12 metrics)\n`relevance` `coherence` `fluency` `consistency` `correctness` `factual_correctness` `sentiment` `toxicity` `bias` `summarization` `nv_answer_accuracy` `prompt_alignment`\n\n### Completeness \u0026 Instruction Following (5 metrics)\n`response_completeness` `instruction_following` `instruction_completeness` `conversation_completeness` `topic_adherence`\n\n### Agent \u0026 Tool Evaluation (12 metrics)\n`task_completion` `tool_accuracy` `tool_call_f1` `plan_adherence` `plan_quality` `step_efficiency` `knowledge_retention` `goal_accuracy` `trajectory_goal_success` `trajectory_tool_sequence` `trajectory_step_count` `trajectory_tool_args_match`\n\n### Per-Turn Conversation (3 metrics)\n`turn_relevancy` `turn_faithfulness` `turn_coherence`\n\n### Multimodal (6 metrics)\n`image_relevance` `image_helpfulness` `image_coherence` `text_to_image` `image_editing` `image_reference`\n\n### Structured Output (4 metrics)\n`code_correctness` `sql_equivalence` `comparative_quality` `datacompy_score`\n\n### Role \u0026 Safety (3 metrics)\n`role_adherence` `role_violation` `non_advice`\n\n### MCP \u0026 Tool-Specific (3 metrics)\n`mcp_use` `mcp_task_completion` `multi_turn_mcp_use`\n\n### Specialized (3 metrics)\n`g_eval` `noise_sensitivity` `rubric`\n\n### Deterministic Checks (39, zero API cost)\n`contains` `not_contains` `starts_with` `ends_with` `regex` `exact_match` `exact_match_strict` `min_tokens` `max_tokens` `min_words` `max_words` `min_chars` `max_chars` `min_sentences` `max_sentences` `is_json` `json_schema` `is_xml` `is_yaml` `is_html` `no_pii` `language` `readability` `similarity` `bleu` `rouge_l` `meteor` `gleu` `chrf` `latency_check` `cost_check` `string_distance` `perplexity` `is_valid_python` `is_url` `has_url` `word_count` `char_count` `sentence_count`\n\n## Red teaming \u0026 adversarial testing\n\n```python\nfrom checkllm.redteam import RedTeamer, VulnerabilityType\nfrom checkllm.redteam_strategies import StrategyType\n\nred = RedTeamer()\nreport = await red.scan(\n    target=my_llm_function,\n    vulnerability_types=[\n        VulnerabilityType.PROMPT_INJECTION,\n        VulnerabilityType.JAILBREAK,\n        VulnerabilityType.PII_LEAKAGE,\n        VulnerabilityType.DATA_EXFILTRATION,\n    ],\n    strategies=[StrategyType.BASE64, StrategyType.CRESCENDO, StrategyType.PERSONA],\n    attacks_per_type=5,\n)\nprint(report.summary())\nprint(report.risk_summary())  # CVSS severity breakdown\n```\n\n**151 vulnerability types** across 12 categories: prompt injection, jailbreak, PII leakage, harmful content, encoding attacks, privilege escalation, agentic AI attacks, brand \u0026 reputation, industry compliance, and more.\n\n**25 attack strategies**: BASE64, ROT13, HEX, LEETSPEAK, MORSE, HOMOGLYPH, CRESCENDO (multi-turn escalation), JAILBREAK_TREE, JAILBREAK_META, JAILBREAK_COMPOSITE, BEST_OF_N, PERSONA, HYPOTHETICAL, ROLEPLAY, LAYER (composable chaining), and more.\n\n### Coding agent security\n\n```python\nfrom checkllm.redteam_coding_agents import CodingAgentScanner\n\nscanner = CodingAgentScanner(judge=judge)\nreport = await scanner.scan(target=my_coding_agent)\n# Tests: repo prompt injection, sandbox escape, secret leakage, verifier sabotage\n```\n\n## Compliance frameworks\n\n```python\nfrom checkllm.compliance_frameworks import ComplianceScanner, ComplianceFramework\n\nscanner = ComplianceScanner(judge=judge)\nreport = await scanner.scan(\n    target=my_llm,\n    frameworks=[\n        ComplianceFramework.OWASP_LLM_TOP10,\n        ComplianceFramework.OWASP_AGENTIC_TOP10,\n        ComplianceFramework.EU_AI_ACT,\n        ComplianceFramework.HIPAA,\n    ],\n)\nprint(report.summary())\n```\n\n**17 frameworks**: OWASP LLM Top 10, OWASP API Top 10, OWASP Agentic Top 10, MITRE ATLAS, EU AI Act, ISO 42001, NIST AI RMF, NIST CSF, HIPAA, GDPR, PCI-DSS, SOC2, ISO 27001, COPPA, FERPA, CCPA, DoD AI Ethics.\n\n## Knowledge Graph test generation\n\n```python\nfrom checkllm.knowledge_graph import KGTestGenerator, EntityExtractor, SimilarityBuilder\n\ngen = KGTestGenerator(judge=judge)\nsamples = await gen.generate(\n    documents=[\"doc1 text...\", \"doc2 text...\"],\n    num_samples=50,\n    synthesizers={\"single_hop\": 0.4, \"multi_hop_abstract\": 0.3, \"multi_hop_specific\": 0.3},\n    personas=5,\n)\ncases = gen.to_cases(samples)\n```\n\nBuild a knowledge graph from your documents, then generate diverse test cases with single-hop, multi-hop abstract, and multi-hop specific queries. Supports persona variation, query styles (web search, misspelled, conversational), and configurable complexity.\n\n## Multilingual evaluation\n\n```python\nfrom checkllm.multilingual import PromptAdapter, detect_language\n\nadapter = PromptAdapter(judge=judge)\ntranslated = await adapter.adapt(template=my_prompt, target_language=\"ja\")\nadapter.save_translations(\"translations/ja.json\")\n\nlang = detect_language(\"Esto es un texto en espanol.\")  # \"es\"\n```\n\nSupports 20+ languages with automatic prompt adaptation. Language detection uses Unicode character-range analysis with LLM fallback.\n\n## Prompt optimization\n\n```python\nfrom checkllm.optimize import create_optimizer\n\noptimizer = create_optimizer(\"miprov2\", judge=judge)  # or \"genetic\", \"copro\", \"simba\"\nresult = await optimizer.optimize(\n    prompt=\"Summarize this document.\",\n    test_cases=my_test_cases,\n    metric_fn=my_metric,\n    num_candidates=10,\n)\nprint(f\"Improved from {result.initial_score:.2f} to {result.best_score:.2f}\")\n```\n\nFour optimization algorithms: Genetic (evolutionary), MIPROv2 (instruction + demonstration), COPRO (failure-driven iterative), SIMBA (similarity-based adaptation).\n\n## Multi-provider judges\n\n```python\nfrom checkllm import create_judge\n\njudge = create_judge(\"openai\", model=\"gpt-4o\")\njudge = create_judge(\"anthropic\", model=\"claude-sonnet-4-6\")\njudge = create_judge(\"gemini\", model=\"gemini-2.0-flash\")\njudge = create_judge(\"ollama\", model=\"llama3.1\")       # Free, local\njudge = create_judge(\"litellm\", model=\"any-model\")     # 100+ models\njudge = create_judge(\"deepseek\")\njudge = create_judge(\"groq\")\njudge = create_judge(\"fireworks\")\n```\n\nAuto-detection: set `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or have Ollama running -- checkllm picks the best judge automatically.\n\n## Consensus judging\n\n```python\nfrom checkllm import ConsensusJudge\n\njudges = [(\"gpt4\", gpt4_judge), (\"claude\", claude_judge), (\"gemini\", gemini_judge)]\nconsensus = ConsensusJudge(judges, strategy=\"majority\")  # or mean, median, unanimous, min, max, weighted\n```\n\n## Cost control\n\n```bash\ncheckllm estimate tests/              # See costs before running\ncheckllm run tests/ --budget 5.0      # Cap spend at $5\ncheckllm run tests/ --dry-run         # Estimate without executing\n```\n\n## Configuration\n\n```toml\n# pyproject.toml\n[tool.checkllm]\njudge_backend = \"auto\"\njudge_model = \"gpt-4o\"\ndefault_threshold = 0.8\nbudget = 10.0\ncache_enabled = true\nengine = \"auto\"\n```\n\n## CLI\n\n| Command | Description |\n|---------|-------------|\n| `checkllm init` | Scaffold a project (`--use-case`, `--ci`) |\n| `checkllm run` | Run tests (`--budget`, `--dry-run`, `--snapshot`) |\n| `checkllm eval-yaml` | Run YAML-based evaluation |\n| `checkllm estimate` | Estimate costs before running |\n| `checkllm watch` | Re-run on file changes |\n| `checkllm report` | Generate HTML report |\n| `checkllm snapshot` | Save baseline for regression detection |\n| `checkllm diff` | Compare snapshots |\n| `checkllm history` | View run history and trends |\n| `checkllm list-metrics` | Show all available checks and metrics |\n| `checkllm cache` | Manage judge response cache |\n| `checkllm dashboard` | Launch web dashboard |\n\n## Framework integrations\n\n```python\n# LangChain\nfrom checkllm.integrations.langchain import CheckllmCallbackHandler\nchain.invoke(input, config={\"callbacks\": [CheckllmCallbackHandler(checks=[\"no_pii\"])]})\n\n# CrewAI\nfrom checkllm.integrations.crewai import CheckllmCrewCallback\n\n# OpenAI Agents SDK\nfrom checkllm.integrations.openai_agents import CheckllmRunHandler\n\n# Claude Agent SDK\nfrom checkllm.integrations.claude_agents import CheckllmAgentHandler\n\n# PydanticAI\nfrom checkllm.integrations.pydantic_ai import CheckllmResultValidator\n\n# LlamaIndex\nfrom checkllm.integrations.llama_index import CheckllmCallbackHandler\n```\n\n## Custom metrics\n\n```python\nfrom checkllm import metric, CheckResult\n\n@metric(\"brevity\")\ndef brevity_check(output: str, max_words: int = 50, **kwargs) -\u003e CheckResult:\n    words = len(output.split())\n    return CheckResult(\n        passed=words \u003c= max_words,\n        score=min(1.0, max_words / max(words, 1)),\n        reasoning=f\"{words} words (limit: {max_words})\",\n        cost=0.0, latency_ms=0, metric_name=\"brevity\",\n    )\n```\n\n## Citing CheckLLM\n\nIf you use CheckLLM's trajectory metric in academic work, please cite the companion paper:\n\n```bibtex\n@article{dejesus2026checkllm,\n  title        = {{CheckLLM}: Reproducible Agent-Trajectory Evaluation at Scale},\n  author       = {de Jesus, Javier},\n  journal      = {arXiv preprint arXiv:XXXX.YYYYY},\n  year         = {2026},\n  doi          = {10.5281/zenodo.PLACEHOLDER},\n  url          = {https://github.com/javierdejesusda/checkllm}\n}\n```\n\nThe arXiv ID and Zenodo DOI placeholders will be replaced once the paper-v1 tag is cut. See [`CITATION.cff`](CITATION.cff) for the canonical citation metadata.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjavierdejesusda%2Fcheckllm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjavierdejesusda%2Fcheckllm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjavierdejesusda%2Fcheckllm/lists"}