https://github.com/arthur-ai/arthur-engine
Make AI work for Everyone - Monitoring and governing for your AI/ML
https://github.com/arthur-ai/arthur-engine
agentic benchmarking evaluation genai guardrails llm ml monitoring tracing
Last synced: 20 days ago
JSON representation
Make AI work for Everyone - Monitoring and governing for your AI/ML
- Host: GitHub
- URL: https://github.com/arthur-ai/arthur-engine
- Owner: arthur-ai
- License: mit
- Created: 2025-03-26T21:08:54.000Z (over 1 year ago)
- Default Branch: dev
- Last Pushed: 2026-06-15T12:14:48.000Z (23 days ago)
- Last Synced: 2026-06-15T12:20:19.527Z (23 days ago)
- Topics: agentic, benchmarking, evaluation, genai, guardrails, llm, ml, monitoring, tracing
- Language: Python
- Homepage: https://arthur.ai
- Size: 52 MB
- Stars: 82
- Watchers: 4
- Forks: 13
- Open Issues: 44
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
Make AI work for Everyone.
[](actions?query=workflow%3A%22%22GenAI+Engine+Workflow%22%22++)
[](https://discord.gg/tdfUAtaVHz)
[Website](https://arthur.ai?utm_source=github&utm_medium=readme) - [Documentation](https://docs.arthur.ai/?utm_source=github&utm_medium=readme) - [Talk to someone at Arthur](https://www.arthur.ai/book-demo?utm_source=github&utm_medium=readme)
# The Arthur Engine
The Arthur Engine provides a **complete service** for developing, monitoring, and governing your AI/ML workloads using popular open-source technologies and frameworks. It is a tool designed for:
- **Enforcing Guardrails in your LLM Applications and Generative AI Workflows**
- Configurable metrics for real-time detection of PII or Sensitive Data leakage, Hallucination, Prompt Injection attempts, Toxic language, and other quality metrics
- **Building, Evaluating, Observing and Governing Agentic Applications**
- Collect and analyze OpenInference OpenTelemetry (OTEL) traces from any agentic workflow or LLM application
- Run continuous evaluations on live traces to catch regressions and quality issues automatically
- Manage, version, and iterate on prompts across your applications
- Run experiments to compare prompt variants and measure their impact on quality metrics
- Evaluate and monitor Retrieval-Augmented Generation (RAG) pipelines end-to-end
- **Monitoring and Benchmarking Machine Learning models (requires the Arthur Platform)**
- Support for a wide range of evaluation metrics (e.g., drift, accuracy, precision, recall, F1, and AUC)
- Tools for comparing models, exploring feature importance, and identifying areas for optimization
- For LLMs/GenAI applications, measure and monitor response relevance, hallucination rates, token counts, latency, and more
- **Extensibility to fit into your application's architecture**
- Native support for custom metrics and extensible API
## Quickstart
**Claude Code users**
Paste this prompt directly into Claude Code that's running on your agent application:
```
For each skill name in this list — arthur-onboard-oss, arthur-onboard-oss-engine, arthur-onboard-task, arthur-onboard-analyze, arthur-onboard-instrument, arthur-onboard-prompts, arthur-onboard-verify, arthur-onboard-eval-provider, arthur-onboard-evals, arthur-skills-upgrade — fetch https://raw.githubusercontent.com/arthur-ai/arthur-engine/refs/heads/main/integrations/claude-code-skills/arthur-onboard//SKILL.md and save it to ~/.claude/skills//SKILL.md (create the directory if it doesn't exist). Also fetch https://raw.githubusercontent.com/arthur-ai/arthur-engine/refs/heads/main/integrations/claude-code-skills/arthur-onboard/arthur-onboard-instrument/EXAMPLES.md and save it to ~/.claude/skills/arthur-onboard-instrument/EXAMPLES.md. Once all files are saved, read ~/.claude/skills/arthur-onboard-oss/SKILL.md and follow its instructions.
```
**Codex users**
Paste this prompt directly into a Codex session that's running on your agent application:
```
Install the Arthur OSS onboarding Codex skills from arthur-ai/arthur-engine.
Install these skill folders into ${CODEX_HOME:-~/.codex}/skills:
- arthur-onboard-oss
- arthur-onboard-oss-engine
- arthur-onboard-task
- arthur-onboard-analyze
- arthur-onboard-instrument
- arthur-onboard-prompts
- arthur-onboard-verify
- arthur-onboard-eval-provider
- arthur-onboard-evals
Fetch each SKILL.md from:
https://raw.githubusercontent.com/arthur-ai/arthur-engine/main/integrations/claude-code-skills/arthur-onboard//SKILL.md
Create each directory as needed. After installation, remind me to restart Codex.
```
**Everyone else**
1. Run the engine installer with the below command:
Mac
```
bash <(curl -sSL https://get-genai-engine.arthur.ai/mac)
```
Windows
```
iex (iwr -Uri "https://get-genai-engine.arthur.ai/win" -UseBasicParsing).Content
```
2. Instrument your agents for evaluations and LLM guardrailing by referencing the examples:
https://github.com/arthur-ai/arthur-engine/tree/dev/genai-engine/examples
## Arthur Platform Free Version
To unlock the full capabilities of the Arthur Platform, [sign up](https://platform.arthur.ai/signup) and get started for free.
* Custom dashboards
* Alerts and notifications
* Configurable webhook that can trigger any workflow
* Agent discovery
* Governance
**Claude Code users**
Paste this prompt directly into a Claude Code session that's running on your agent application:
```
For each skill name in this list — arthur-onboard-platform, arthur-onboard-platform-access, arthur-onboard-platform-workspace, arthur-onboard-platform-engine, arthur-onboard-platform-model, arthur-onboard-platform-token, arthur-onboard-analyze, arthur-onboard-instrument, arthur-onboard-prompts, arthur-onboard-verify, arthur-onboard-eval-provider, arthur-onboard-evals — fetch https://raw.githubusercontent.com/arthur-ai/arthur-engine/refs/heads/main/integrations/claude-code-skills/arthur-onboard//SKILL.md and save it to ~/.claude/skills//SKILL.md (create the directory if it doesn't exist). Once all skills are saved, read ~/.claude/skills/arthur-onboard-platform/SKILL.md and follow its instructions.
```
**Codex users**
Install the Arthur Platform onboarding Codex skills from arthur-ai/arthur-engine.
```
Install these skill folders into ${CODEX_HOME:-~/.codex}/skills:
- arthur-onboard-platform
- arthur-onboard-platform-access
- arthur-onboard-platform-workspace
- arthur-onboard-platform-engine
- arthur-onboard-platform-model
- arthur-onboard-platform-token
- arthur-onboard-analyze
- arthur-onboard-instrument
- arthur-onboard-prompts
- arthur-onboard-verify
- arthur-onboard-eval-provider
- arthur-onboard-evals
Fetch each SKILL.md from:
https://raw.githubusercontent.com/arthur-ai/arthur-engine/main/integrations/claude-code-skills/arthur-onboard//SKILL.md
Create each directory as needed. After installation, remind me to restart Codex.
```
**Everyone else**
Onboard via the platform UI.
## Arthur Platform Enterprise Version
**The enterprise version of the Arthur Platform provides better performance, additional features, and capabilities**, including custom enterprise-ready guardrails + metrics, which can maximize the potential of AI for your organization.
Key features:
- State-of-the-art proprietary evaluation models trained by Arthur's world-class machine learning engineering team
- Air-gapped deployment of the Arthur Engine (no dependency to Hugging Face Hub)
- Optional on-premises deployment of the entire Arthur Platform
- Support from the world-class engineering teams at Arthur
To learn more about the enterprise version of the Arthur Platform, [reach out!](https://www.arthur.ai/book-demo)
## Contributing
- Join the Arthur community on [Discord](https://discord.gg/tdfUAtaVHz) to get help and share your feedback.
- To make a request for a bug fix or a new feature, please file a [GitHub issue](https://github.com/arthur-ai/arthur-engine/issues).
- To make code contributions, please review the [contributing guidelines](CONTRIBUTING.md).
- Thank you!