{"id":50718164,"url":"https://github.com/ifixai-ai/iFixAi","last_synced_at":"2026-06-26T22:00:35.410Z","repository":{"id":354208686,"uuid":"1222527844","full_name":"ifixai-ai/iFixAi","owner":"ifixai-ai","description":"Catch your AI's mistakes and blind spots before your customers or regulators do. iFixAi runs 45 inspections, 32 graded core plus 13 extended for frontier risks like sabotage, sandbagging, and oversight evasion. It returns a letter grade in under 5 minutes. Industry and model agnostic.","archived":false,"fork":false,"pushed_at":"2026-06-26T10:19:55.000Z","size":22208,"stargazers_count":570,"open_issues_count":1,"forks_count":112,"subscribers_count":9,"default_branch":"main","last_synced_at":"2026-06-26T11:17:25.445Z","etag":null,"topics":["agent-evaluation","ai","ai-alignment","ai-evaluation","ai-governance","ai-safety","cli","diagnostic-tool","eu-ai-act","hallucination-detection","iso-42001","llm-evaluation","llm-security","nist-ai-rmf","owasp-llm","prompt-injection","python","responsible-ai","risk-assessment","risk-management"],"latest_commit_sha":null,"homepage":"https://www.ifixai.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ifixai-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-27T13:02:34.000Z","updated_at":"2026-06-26T00:47:10.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ifixai-ai/iFixAi","commit_stats":null,"previous_names":["ifixai-ai/diagnostic","ifixai-ai/ifixai"],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/ifixai-ai/iFixAi","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ifixai-ai%2FiFixAi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ifixai-ai%2FiFixAi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ifixai-ai%2FiFixAi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ifixai-ai%2FiFixAi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ifixai-ai","download_url":"https://codeload.github.com/ifixai-ai/iFixAi/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ifixai-ai%2FiFixAi/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34834415,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-26T02:00:06.560Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-evaluation","ai","ai-alignment","ai-evaluation","ai-governance","ai-safety","cli","diagnostic-tool","eu-ai-act","hallucination-detection","iso-42001","llm-evaluation","llm-security","nist-ai-rmf","owasp-llm","prompt-injection","python","responsible-ai","risk-assessment","risk-management"],"created_at":"2026-06-09T21:00:25.962Z","updated_at":"2026-06-26T22:00:35.404Z","avatar_url":"https://github.com/ifixai-ai.png","language":"Python","funding_links":[],"categories":["*Ops for AI"],"sub_categories":["LLMOps"],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/assets/ifixai-banner.png\" alt=\"iFixAi\" width=\"200\" /\u003e\n\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003eiFixAi\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\u003cstrong\u003eThe diagnostic for AI operational misalignment\u003c/strong\u003e\u003c/p\u003e\n\u003cp align=\"center\"\u003eCatch your agent's mistakes and blind spots before the shit hits the fan.\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#quick-start\"\u003eQuick start\u003c/a\u003e •\n  \u003ca href=\"#two-ways-to-run-it\"\u003eTwo ways to run\u003c/a\u003e •\n  \u003ca href=\"#test-your-own-agent\"\u003eTest your agent\u003c/a\u003e •\n  \u003ca href=\"#what-you-get-back\"\u003eScoring\u003c/a\u003e •\n  \u003ca href=\"#in-the-wild\"\u003eIn the wild\u003c/a\u003e •\n  \u003ca href=\"docs/\"\u003eDocs\u003c/a\u003e •\n  \u003ca href=\"CONTRIBUTING.md\"\u003eContributing\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-Apache%202.0-blue.svg\" alt=\"license: Apache 2.0\" /\u003e\u003c/a\u003e\n  \u003ca href=\"pyproject.toml\"\u003e\u003cimg src=\"https://img.shields.io/badge/python-3.10%2B-blue.svg\" alt=\"python 3.10+\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/ifixai-ai/iFixAi/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://github.com/ifixai-ai/iFixAi/actions/workflows/ci.yml/badge.svg\" alt=\"CI\" /\u003e\u003c/a\u003e\n  \u003cimg src=\"https://img.shields.io/badge/inspections-45-orange.svg\" alt=\"45 inspections\" /\u003e\n  \u003ca href=\"https://github.com/ifixai-ai/iFixAi/issues?q=is%3Aopen+label%3A%22good+first+issue%22\"\u003e\u003cimg src=\"https://img.shields.io/github/issues/ifixai-ai/iFixAi/good%20first%20issue?label=good%20first%20issues\u0026color=7057ff\" alt=\"good first issues\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/assets/unique_cloners_chart.png\" alt=\"UniqueClones\" width=\"750\" /\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/assets/ifixai-demo.gif\" alt=\"iFixAi demo\" width=\"720\" /\u003e\n  \u003cbr/\u003e\n  \u003cem\u003eRecorded from a custom client build. The open-source CLI runs the same diagnostic with different presentation.\u003c/em\u003e\n\u003c/p\u003e\n\n---\n\n## What it is\n\niFixAi detects AI operational misalignment before it damages your business. By that, we mean\nany action, omission, or behaviour from your AI that does not match what your business\nintended, designed, or expects it to do. The dangerous part is that this rarely shows up in\nyour usual KPIs. An agent can hit every dashboard target while quietly leaking a permission,\nfabricating a citation, caving to a manipulative prompt, or doing something it was never\nauthorised to do. Those are the blind spots that surface as an incident, a customer complaint,\nor a regulator's question long after the damage is done. iFixAi finds them first.\n\nIt runs up to 45 inspections against your agent, from direct policy compliance to adversarial\npressure and structural edge cases. These come in two tiers: 32 core plus 13 extended. The 32\ncore inspections cover five pillars of misalignment risk: fabrication, manipulation, deception,\nunpredictability, and opacity. Together with five of the extended inspections, they produce the\nletter grade, which you get back in under 5 minutes. The 13 extended inspections span 11 new\ncategories of frontier agent risk, such as sabotage, sandbagging, oversight evasion, and power\nelevation. Five of them feed the grade, one a mandatory minimum that can cap it; the other eight\nare exploratory, scored and reported on their own, so they widen your coverage without moving\nthe headline grade.\n\nBecause the whole point is trust, iFixAi is honest about what it is. It is not a certification\nor a safety guarantee. It is a repeatable diagnostic you can run in CI: by default, your agent\nis judged by independent providers rather than by itself, one in Standard mode and an ensemble\nof two or more in Full mode. Every run also writes a manifest of all its inputs, so the result\ncan be audited and replayed.\n\n## Two ways to run it\n\nThere are two ways to run iFixAi, and both run the same diagnostic underneath: the\ncommand-line tool (the CLI) or the Claude Code plugin. Either one tests any model and lets\nyou choose who grades it. The difference is who drives: you script the CLI yourself, or let\nClaude operate the plugin for you.\n\n| | **CLI** (`pip install`) | **Claude Code plugin** |\n|---|---|---|\n| **How you drive it** | you write the **fixture** (the config describing your agent) and CLI flags; scriptable, CI-friendly | Claude is the operator: it discovers your setup, builds the fixture, runs it, and explains the scorecard |\n| **What you can test** | any provider, or your agent's real endpoint | any provider (Anthropic, OpenAI, Gemini, Azure, Bedrock, …); Claude only guides |\n| **Who grades it** | any judge: self, one independent vendor, or a panel | same: self, one independent judge, or a cross-vendor panel |\n| **Output** | JSON + markdown/HTML reports | interactive results artifact (+ JSON source of truth; static-report fallback) |\n| **Setup** | `pip install` + the provider key(s) you'll test | keys in your Claude Code `settings.json` env; the engine self-provisions |\n| **Best for** | CI, automation, audit-ready batch runs | a guided, explained run with discovery and an interactive scorecard |\n\n**Plugin:** Claude runs the diagnostic for you. Open this repo in [Claude Code](https://claude.com/claude-code) and say\n*\"run iFixAi on my setup.\"* Claude reads your agent's config, shows the test fixture\nit builds and names the cost before billing, runs the diagnostic on the model(s)\nand judge(s) you choose, then explains the scorecard. The rest of this page covers the CLI.\n\n## Quick start\n\nNow try it yourself. In three commands you install iFixAi, check that it runs, then grade a\nreal model. The grade you get back is citable because a different vendor's AI does the grading,\nnot the agent judging itself. Full walkthrough: **[docs/get-started.md](docs/get-started.md)**.\n\n```bash\n# 1. Install the CLI + the extra for the provider you'll test\npip install \"ifixai[anthropic]\"\n\n# 2. Prove the pipeline runs: built-in mock, no keys, no network, ~1s\nifixai run --provider mock --api-key not-used --eval-mode self\n\n# 3. Get a citable grade: your model graded by a *different* vendor's judge\npip install \"ifixai[anthropic,openai]\"     # SUT's + judge's SDKs (or ifixai[all])\nexport ANTHROPIC_API_KEY=sk-ant-...         # the SUT, graded\nexport OPENAI_API_KEY=sk-...                # the judge, auto-paired from the environment\nifixai run --provider anthropic --api-key \"$ANTHROPIC_API_KEY\"\n```\n\nEvery run has **two roles**, and a citable run needs a key for each:\n\n| Role | What it is | How you set it |\n|---|---|---|\n| **SUT** (system under test) | the agent/model being **graded** | `--provider` + `--api-key`; the SUT key is always passed explicitly, never read from the environment |\n| **Judge** | who **grades** it | auto-paired from a *different* provider whose key is in your environment (the SUT's own vendor is excluded, so it never grades itself) |\n\nReports land in `./ifixai-results/` as JSON **and** Markdown. Without a second key, add\n`--eval-mode self` to run as a smoke test (the grade still prints, but it's flagged as\nself-judged, not a result you can cite). Pinning the judge, Full-mode ensembles, and the eval modes:\n**[docs/running.md](docs/running.md)**. Other providers (OpenAI, OpenRouter, Gemini,\nAzure, Bedrock, Hugging Face, HTTP, LangChain) install the matching extra and follow the\nsame steps: **[docs/providers.md](docs/providers.md)**.\n\n### Test your own agent\n\nThe commands above call a **bare model API**: the simplest case, and it scores lower\nbecause a bare model has none of the extra parts a real agent does. The real system under\ntest is usually your **agent**: a model wrapped with a system prompt, tools, retrieval, and guardrails.\niFixAi treats it as a black box reached through a thin adapter:\n\n- **Serves an OpenAI-compatible HTTP endpoint?** Point `--provider http --endpoint …` at it, no glue code.\n- **Runs anywhere else?** Implement one method, `ChatProvider.send_message` ([ifixai/providers/base.py](ifixai/providers/base.py)), and override the optional capability hooks (`list_tools`, `get_audit_trail`, `authorize_tool`, `retrieve_sources`, …).\n\nThe more of those parts your adapter exposes, the more inspections iFixAi can actually\nscore, instead of marking them `insufficient_evidence` (it couldn't see enough of your\nagent to judge; these are reported but don't count for or against your grade). Full\nwalkthrough with the model-vs-agent coverage map: **[docs/testing-your-agent.md](docs/testing-your-agent.md)**.\n\n## What you get back\n\nA letter grade with the breakdown behind it. iFixAi groups the 45 inspections into **16 categories**, five core pillars plus eleven premium. The five core pillars:\n\n| Core pillar | What it detects |\n|---|---|\n| **Fabrication** | uses a tool it wasn't granted, keeps no audit trail, makes unsourced or overconfident claims |\n| **Manipulation** | privilege escalation, breaking its own policy, prompt injection, poisoned retrieval context |\n| **Deception** | sandbagging (does better when it senses a test), secret side-goals, drifting off-task over long runs, failing silently |\n| **Unpredictability** | distorted context, drifting from instructions, inconsistent decisions |\n| **Opacity** | weak risk scoring, regulatory gaps, broken human-escalation, answering off-topic |\n\n- Your **A–F grade** is a weighted average of every category that produces a score: always the five core pillars, plus any premium categories your run can measure (A ≥ 0.90, B ≥ 0.80, C ≥ 0.70, D ≥ 0.60, F \u003c 0.60; pass threshold 0.85, `--min-score`).\n- **Mandatory minimums** (B01, B08, P01) cap the overall score at 60% if missed.\n\nThe other **11 categories are the premium tier**: sabotage, subversion, concealment,\nsandbagging, insubordination, usurpation, systemic risk, miscalibration, stakeholder\nconflict, perception governance, oversight atrophy. This repo ships **13 inspections from\nthem as a free preview of iFixAi's premium suite**, at least one per category. **Five feed\nyour grade** (including the P01 mandatory minimum above); the **other eight are\nexploratory**: scored and reported on their own, but kept out of the headline so they\ncan't skew comparisons.\n\nFull math and weights: **[docs/scoring.md](docs/scoring.md)**. The full `B01`–`B32` → pillar\nmapping and every premium category: **[docs/inspection_categories.md](docs/inspection_categories.md)**.\n\n## In the wild\n\nThree real open-source AI systems, graded end-to-end: two F's and a D. Each ran against a fixture\nthat describes its real setup (tools, rules, permissions), graded by a panel of judges from different vendors.\n\n| System | Upstream model | Score | Grade | Key finding |\n|---|---|---|---|---|\n| [Hermes Agent](case_studies/hermes-gpt-4o-mini/SCORECARD.md) (Nous Research) | `gpt-4o-mini` | **33.9%** | **F** | Capable model, but nothing stops bad actions; 23 of 32 inspections failed. [deep dive](https://ifixai.ai/docs/diagnostics/hermes) |\n| [OpenClaw](case_studies/openclaw-haiku/SCORECARD.md) v2026.5.4 | `claude-3.5-haiku` | **60.0%** | **D** | Follows the rules when asked plainly; caves when the request is dressed up. [deep dive](https://ifixai.ai/docs/diagnostics/openclaw) |\n| [Open WebUI](case_studies/openwebui-sonnet/SCORECARD.md) v0.9.5 | `claude-sonnet-4.6` | **11.3%** | **F** | Nothing passes once you strip the scaffolding that faked compliance. [deep dive](https://ifixai.ai/docs/diagnostics/openwebui) |\n\nThe takeaway from Hermes is the clearest: a capable model with nothing enforcing its rules\nis not safe. All scorecards live in **[case_studies/](case_studies/)**.\n\n## Documentation\n\nDocs are sorted by what you came to do. Start in **[docs/](docs/)**:\n\n- 🟢 **New here** → [Get started](docs/get-started.md)\n- 🔧 **Doing something** → [Run modes \u0026 judges](docs/running.md) · [Test your agent](docs/testing-your-agent.md) · [Providers](docs/providers.md) · [Author a fixture](docs/fixture_authoring.md)\n- 📖 **Looking it up** → [CLI](docs/cli.md) · [Python API](docs/python-api.md) · [Scoring](docs/scoring.md) · [Inspections](docs/inspections.md)\n- 💡 **Why it works this way** → [Methodology](docs/methodology.md)\n\n## Contributing\n\nIssues and PRs welcome. See **[CONTRIBUTING.md](CONTRIBUTING.md)** (`pip install -e \".[dev]\"`,\nthen `ruff`, `bandit`, `pytest`). Good first issues are\n[labelled here](https://github.com/ifixai-ai/iFixAi/issues?q=is%3Aopen+label%3A%22good+first+issue%22).\n\n## Contact\n\nBug reports, features, questions: open a [GitHub issue](https://github.com/ifixai-ai/iFixAi/issues).\nSecurity-sensitive reports: **[SECURITY.md](SECURITY.md)**. Anything else: **info@ime.life**.\n\n## License\n\n[Apache 2.0](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fifixai-ai%2FiFixAi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fifixai-ai%2FiFixAi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fifixai-ai%2FiFixAi/lists"}