{"id":50705169,"url":"https://github.com/Virtue-Research/guard-eval-harness","last_synced_at":"2026-06-26T11:00:33.556Z","repository":{"id":359462768,"uuid":"1204165073","full_name":"Virtue-Research/guard-eval-harness","owner":"Virtue-Research","description":"One command to benchmark AI guardrails and coding agents across safety, security, jailbreak, prompt-injection, and secure-code tasks.","archived":false,"fork":false,"pushed_at":"2026-06-16T22:45:20.000Z","size":4452,"stargazers_count":11,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-17T00:25:41.632Z","etag":null,"topics":["ai-safety","benchmark","cli","eval-harness","guardrails","llm-evaluation","llm-safety","safety-evaluation"],"latest_commit_sha":null,"homepage":"https://virtue-research.github.io/guard-eval-harness/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Virtue-Research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-07T18:49:37.000Z","updated_at":"2026-06-16T22:45:25.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Virtue-Research/guard-eval-harness","commit_stats":null,"previous_names":["virtue-research/guard-eval-harness"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/Virtue-Research/guard-eval-harness","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Virtue-Research%2Fguard-eval-harness","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Virtue-Research%2Fguard-eval-harness/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Virtue-Research%2Fguard-eval-harness/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Virtue-Research%2Fguard-eval-harness/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Virtue-Research","download_url":"https://codeload.github.com/Virtue-Research/guard-eval-harness/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Virtue-Research%2Fguard-eval-harness/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34813782,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-26T02:00:06.560Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-safety","benchmark","cli","eval-harness","guardrails","llm-evaluation","llm-safety","safety-evaluation"],"created_at":"2026-06-09T11:00:25.622Z","updated_at":"2026-06-26T11:00:33.548Z","avatar_url":"https://github.com/Virtue-Research.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["Evaluation \\\u0026 Analysis"],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/Virtue-Research/guard-eval-harness/main/assets/polished.svg\" alt=\"guard-eval-harness\" width=\"550\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003eCLI-first harness for benchmarking guardrail, moderation, and safety classification models.\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/Virtue-Research/guard-eval-harness/main/assets/demo.gif\" alt=\"geh demo — run a benchmark pack and export the results as a table\" width=\"900\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://pypi.org/project/geh/\"\u003e\u003cimg alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/geh.svg?style=flat-square\u0026color=4c1\u0026logo=pypi\u0026logoColor=white\u0026logoSize=auto\u0026label=pypi\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/geh/\"\u003e\u003cimg alt=\"Python\" src=\"https://img.shields.io/badge/python-3.10%2B-2d3748?style=flat-square\u0026logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNTYgMjU1Ij48ZGVmcz48bGluZWFyR3JhZGllbnQgaWQ9ImEiIHgxPSIxMi45NTklIiB4Mj0iNzkuNjM5JSIgeTE9IjEyLjAzOSUiIHkyPSI3OC4yMDElIj48c3RvcCBvZmZzZXQ9IjAiIHN0b3AtY29sb3I9IiMzODdFQjgiLz48c3RvcCBvZmZzZXQ9IjEiIHN0b3AtY29sb3I9IiMzNjY5OTQiLz48L2xpbmVhckdyYWRpZW50PjxsaW5lYXJHcmFkaWVudCBpZD0iYiIgeDE9IjE5LjEyOCUiIHgyPSI5MC43NDIlIiB5MT0iMjAuNTc5JSIgeTI9Ijg4LjQyOSUiPjxzdG9wIG9mZnNldD0iMCIgc3RvcC1jb2xvcj0iI0ZGRTA1MiIvPjxzdG9wIG9mZnNldD0iMSIgc3RvcC1jb2xvcj0iI0ZGQzMzMSIvPjwvbGluZWFyR3JhZGllbnQ+PC9kZWZzPjxwYXRoIGZpbGw9InVybCgjYSkiIGQ9Ik0xMjYuOTE2LjA3MmMtNjQuODMyIDAtNjAuNzg0IDI4LjExNS02MC43ODQgMjguMTE1bC4wNzIgMjkuMTI4aDYxLjg2OHY4Ljc0NUg0MS42MzFTLjE0NSA2MS4zNTUuMTQ1IDEyNi43N2MwIDY1LjQxNyAzNi4yMSA2My4wOTcgMzYuMjEgNjMuMDk3aDIxLjYxdi0zMC4zNTZzLTEuMTY1LTM2LjIxIDM1LjYzMi0zNi4yMWg2MS4zNjJzMzQuNDc1LjU1NyAzNC40NzUtMzMuMzE5VjMzLjk3UzE5NC42Ny4wNzIgMTI2LjkxNi4wNzJ6TTkyLjgwMiAxOS42NmExMS4xMiAxMS4xMiAwIDAgMSAxMS4xMyAxMS4xMyAxMS4xMiAxMS4xMiAwIDAgMS0xMS4xMyAxMS4xMyAxMS4xMiAxMS4xMiAwIDAgMS0xMS4xMy0xMS4xMyAxMS4xMiAxMS4xMiAwIDAgMSAxMS4xMy0xMS4xM3oiLz48cGF0aCBmaWxsPSJ1cmwoI2IpIiBkPSJNMTI4Ljc1NyAyNTQuMTI2YzY0LjgzMiAwIDYwLjc4NC0yOC4xMTUgNjAuNzg0LTI4LjExNWwtLjA3Mi0yOS4xMjdIMTI3LjZ2LTguNzQ1aDg2LjQ0MXM0MS40ODYgNC43MDUgNDEuNDg2LTYwLjcxMmMwLTY1LjQxNi0zNi4yMS02My4wOTYtMzYuMjEtNjMuMDk2aC0yMS42MXYzMC4zNTVzMS4xNjUgMzYuMjEtMzUuNjMyIDM2LjIxaC02MS4zNjJzLTM0LjQ3NS0uNTU3LTM0LjQ3NSAzMy4zMnY1Ni4wMTNzLTUuMjM1IDMzLjg5NyA2Mi41MTggMzMuODk3em0zNC4xMTQtMTkuNTg2YTExLjEyIDExLjEyIDAgMCAxLTExLjEzLTExLjEzIDExLjEyIDExLjEyIDAgMCAxIDExLjEzLTExLjEzMSAxMS4xMiAxMS4xMiAwIDAgMSAxMS4xMyAxMS4xMyAxMS4xMiAxMS4xMiAwIDAgMS0xMS4xMyAxMS4xM3oiLz48L3N2Zz4K\u0026logoSize=auto\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/Virtue-Research/guard-eval-harness/blob/main/LICENSE\"\u003e\u003cimg alt=\"License\" src=\"https://img.shields.io/badge/license-MIT-3DA639?style=flat-square\u0026logo=opensourceinitiative\u0026logoColor=white\u0026logoSize=auto\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/Virtue-Research/guard-eval-harness/actions/workflows/ci.yml\"\u003e\u003cimg alt=\"CI\" src=\"https://img.shields.io/github/actions/workflow/status/Virtue-Research/guard-eval-harness/ci.yml?branch=main\u0026label=CI\u0026style=flat-square\u0026logo=githubactions\u0026logoColor=white\u0026logoSize=auto\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://virtue-research.github.io/guard-eval-harness/\"\u003e\u003cimg alt=\"Docs\" src=\"https://img.shields.io/badge/docs-online-526CFE?style=flat-square\u0026logo=materialformkdocs\u0026logoColor=white\u0026logoSize=auto\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pepy.tech/projects/geh\"\u003e\u003cimg alt=\"PyPI Downloads\" src=\"https://static.pepy.tech/personalized-badge/geh?period=total\u0026units=INTERNATIONAL_SYSTEM\u0026left_color=BLACK\u0026right_color=GREEN\u0026left_text=downloads\"\u003e\u003c/a\u003e\n  \u003cimg alt=\"Made with love\" src=\"https://img.shields.io/badge/made%20with-%E2%9D%A4-FF1493?style=flat-square\"\u003e\n\u003c/p\u003e\n\nEvaluate any safety model — local HuggingFace, vLLM, OpenAI, Anthropic, or custom API — against 80+ built-in safety benchmarks with a single command.\n\n## Quickstart\n\n```bash\npip install geh\n\n# Run a quick eval\ngeh run --dataset xstest --model mock --limit 50\n\n# Run multiple datasets\ngeh run --dataset xstest,toxic_chat,harmful_qa --model hf \\\n    --model-name meta-llama/Llama-Guard-3-8B\n\n# Run from a YAML config\ngeh run --config examples/run-mock-jsonl.yaml\n\n# Use benchmark packs\ngeh run --pack core --model mock\n```\n\n## Installation\n\nRequires Python 3.10+.\n\n```bash\n# Base install\npip install geh\n\n# With HuggingFace model support\npip install \"geh[hf]\"\n\n# With vLLM support\npip install \"geh[vllm]\"\n\n# With API model support (OpenAI, Anthropic)\npip install \"geh[api]\"\n```\n\nFrom source (for development):\n\n```bash\ngit clone https://github.com/Virtue-Research/guard-eval-harness.git\ncd guard-eval-harness\npip install -e \".[dev]\"\n```\n\nCopy `.env.example` to `.env` and fill in the API keys you need.\n\n## Usage\n\n### Inline mode\n\nThe fastest way to run evals — no config files needed:\n\n```bash\ngeh run --dataset \u003cdataset\u003e --model \u003cadapter\u003e [--model-name \u003cname\u003e] [options]\n```\n\n```bash\n# HuggingFace model on XSTest\ngeh run --dataset xstest --model hf --model-name meta-llama/Llama-Guard-3-8B\n\n# OpenAI moderation\ngeh run --dataset xstest,toxic_chat --model openai_moderation\n\n# vLLM serving\ngeh run --dataset harmbench_behaviors --model vllm \\\n    --model-name meta-llama/Llama-Guard-3-8B --batch-size 32\n\n# Limit samples for quick smoke tests\ngeh run --dataset xstest --model mock --limit 10\n```\n\n### YAML config mode\n\nFor full control over model args, dataset options, execution tuning, and output:\n\n```bash\ngeh run --config examples/run-mock-jsonl.yaml\n```\n\nSee [`examples/`](examples/) for sample configs.\n\n### Benchmark packs\n\nCurated dataset bundles for common evaluation scenarios:\n\n```bash\ngeh list packs\ngeh run --pack core --model mock\ngeh run --pack jailbreak --model hf --model-name meta-llama/Llama-Guard-3-8B\n```\n\n### Discovery\n\n```bash\ngeh list datasets    # 80+ built-in safety benchmarks\ngeh list backends    # Available model adapters\ngeh list packs       # Curated benchmark bundles\ngeh list metrics     # Supported metrics\n```\n\n### Inspecting results\n\n```bash\ngeh inspect --run-dir out/my-run       # View manifest, summary, artifacts\ngeh report --run-dir out/my-run        # Rebuild HTML report\ngeh compare --run-a out/run1 --run-b out/run2  # Diff two runs\ngeh export --run-dir out/my-run --format csv --output results.csv\n```\n\n## Run artifacts\n\nEach run writes a self-contained directory:\n\n```\nout/my-run/\n  manifest.json              # Run metadata\n  resolved-config.json       # Exact config snapshot\n  summary.json               # Aggregated metrics\n  report.html                # Static HTML report\n  datasets/\n    \u003cdataset\u003e/\n      predictions.jsonl      # Per-sample predictions\n      metrics.json           # Dataset-level metrics\n      dataset-manifest.json  # Dataset metadata\n```\n\n## Model adapters\n\n| Adapter | Description |\n|---------|-------------|\n| `mock` | Deterministic mock for testing |\n| `hf` | HuggingFace Transformers (local GPU) |\n| `vllm` | vLLM inference server |\n| `openai_compatible` | OpenAI-compatible APIs |\n| `openai_moderation` | OpenAI Moderation endpoint |\n| `anthropic` | Anthropic Claude API |\n| `http` | Generic HTTP endpoint |\n\n## Datasets\n\n80+ built-in safety benchmarks spanning two modalities:\n\n### Text\n\nThe core modality — evaluate text-based guardrails and moderation models across a range of safety dimensions:\n\n- **Jailbreak / adversarial**: XSTest, HarmBench, JBB Behaviors, AdvBench, Do-Anything-Now, StrongREJECT, MaliciousInstruct, WildGuardMix\n- **Toxicity**: ToxicChat, ToxiGen, Jigsaw Toxicity, Civil Comments, RealToxicityPrompts, OR-Bench\n- **Hate \u0026 harassment**: HateCheck, DynaHate, ETHOS, HatExplain, Implicit Hate, Measuring Hate Speech, Social Bias Frames, ConvAbuse\n- **General safety**: BeaverTails 330k, Do-Not-Answer, OpenAI Moderation (via API), GuardBench, CircleGuardBench\n- **Prompt injection**: Dedicated prompt-injection benchmarks for testing input-filtering guardrails\n\n### Image\n\nEvaluate multimodal safety models that process image+text inputs. The harness handles image downloading, caching, and normalization automatically:\n\n- **Unsafe content detection**: UnsafeBench (8k+ images across safety categories), HoliSafeBench (holistic image safety with fine-grained risk types)\n- **Visual jailbreaks**: JailbreakV (adversarial images designed to bypass vision-language model safeguards)\n- **Image edit safety**: Safe-vs-Unsafe Image Edits (detecting harmful image manipulation requests)\n- **Cross-modal attacks**: VLSBench, MSTS (text+image multimodal safety evaluation)\n- **Benign baselines**: ImageNet-1k safe subset (measuring false positive rates on benign images)\n- **Local image data**: Load from local directories or JSONL manifests with image paths/URLs\n\n### Local files\n\nBring your own data in any modality:\n\n- `local_jsonl` — text samples from a JSONL file\n- `local_csv` — text samples from a CSV file\n- `local_image_jsonl` — image+text samples from a JSONL manifest with image paths/URLs\n- `local_image_dir` — image samples from a directory of images\n\nRun `geh list datasets` for the full list.\n\n### Secure-coding agents\n\nBeyond classification, the harness also runs repository-level secure-coding\nbenchmarks under `geh vibe`: a coding agent writes or completes real code, and\nan out-of-process oracle builds it in a container to score functional\ncorrectness and security. See the [VibeCoding Bench guide](docs/vibecoding.md)\nand `geh vibe datasets`.\n\n## About\n\n`guard-eval-harness` is built and maintained by the research team at\n**[Virtue AI](https://www.virtueai.com)** — one security solution for your entire AI stack.\n\n## License\n\n[MIT](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVirtue-Research%2Fguard-eval-harness","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FVirtue-Research%2Fguard-eval-harness","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVirtue-Research%2Fguard-eval-harness/lists"}