https://github.com/raw-labs/mxcp
Model eXecution + Context Protocol: Enterprise-Grade Data-to-AI Infrastructure
https://github.com/raw-labs/mxcp
chatgpt claude dbt duckdb large-language-model large-language-models llms mcp mcp-client model-context-protocol mxcp openai python-mcp-server
Last synced: 3 months ago
JSON representation
Model eXecution + Context Protocol: Enterprise-Grade Data-to-AI Infrastructure
- Host: GitHub
- URL: https://github.com/raw-labs/mxcp
- Owner: raw-labs
- License: other
- Created: 2025-05-21T13:31:46.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2025-08-27T17:38:47.000Z (3 months ago)
- Last Synced: 2025-08-28T01:40:35.379Z (3 months ago)
- Topics: chatgpt, claude, dbt, duckdb, large-language-model, large-language-models, llms, mcp, mcp-client, model-context-protocol, mxcp, openai, python-mcp-server
- Language: Python
- Homepage: https://mxcp.dev
- Size: 2.29 MB
- Stars: 47
- Watchers: 5
- Forks: 6
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
- awesome-mcp-devtools - mxcp - Open-source framework for building secure, testable, enterprise-grade MCP tools from SQL or Python on top of dbt + DuckDB. (SDKs / Python)
- awesome-mcp-servers - **mxcp (Python)** - Open-source framework for building enterprise-grade MCP servers using just YAML, SQL, and Python, with built-in auth, monitoring, ETL and policy enforcement. `auth` `http` `ml` `git` `github` (📦 Other)
README
# MXCP: Enterprise-Grade MCP Framework for AI Applications
[](https://www.python.org/downloads/)
[](https://github.com/raw-labs/mxcp/blob/main/LICENSE)
**The structured methodology for building production-ready MCP servers with enterprise security, data quality, and comprehensive testing**
## 🚀 What Makes MXCP Different?
MXCP isn't just another MCP implementation - it's a **complete methodology** for building production AI applications the right way:
### The Production-Ready Approach
1. **📊 Data Modeling First**: Start with dbt models, data contracts, and quality tests
2. **📋 Service Design**: Define types, security policies, and API contracts upfront
3. **🛠️ Smart Implementation**: Choose SQL for data, Python for logic - or combine both
4. **✅ Quality Assurance**: Validate, test, lint, and evaluate before deployment
5. **🚨 Production Operations**: Monitor drift, track audits, ensure performance
### Enterprise Features Built-In
- 🔒 **Security First**: OAuth authentication, RBAC, policy enforcement
- 📝 **Complete Audit Trail**: Track every operation for compliance
- 🎯 **Type Safety**: Comprehensive validation across SQL and Python
- 🧪 **Testing Framework**: Unit tests, integration tests, LLM behavior tests
- 📈 **Performance**: Optimized queries, caching strategies, async support
- 🔄 **Drift Detection**: Monitor schema changes across environments
- 🔍 **OpenTelemetry**: Distributed tracing and metrics for production observability
```yaml
# One config enables enterprise features
auth: { provider: github }
audit: { enabled: true }
policies: { strict_mode: true }
telemetry: { enabled: true, endpoint: "http://otel-collector:4318" }
```
## 🎯 60-Second Quickstart
Experience the power of MXCP in under a minute:
```bash
# 1. Install and create project (15 seconds)
pip install mxcp
mkdir my-ai-tools && cd my-ai-tools
mxcp init --bootstrap
# 2. Start serving your tools (5 seconds)
mxcp serve
# 3. Connect to Claude Desktop (40 seconds)
# Add this to your Claude config:
{
"mcpServers": {
"my-tools": {
"command": "mxcp",
"args": ["serve", "--transport", "stdio"],
"cwd": "/path/to/my-ai-tools"
}
}
}
```
**Result**: You now have a production-ready AI tool API with type safety, validation, audit trails, and policy enforcement.
## 📚 The MXCP Methodology
Building production MCP servers requires more than just connecting data to AI. MXCP provides a structured approach:
### 1. Start with Data Quality
```yaml
# Use dbt to model and test your data
models:
marts:
customer_360:
+materialized: table
+tests:
- unique: customer_id
- not_null: [customer_id, email]
```
### 2. Design Your Service
```yaml
# Define clear contracts and security policies
tool:
name: get_customer
parameters:
- name: customer_id
type: string
pattern: "^cust_[0-9]+$"
policies:
input:
- condition: "user.role != 'admin' && customer_id != user.customer_id"
action: deny
```
### 3. Implement Smartly
- **SQL** for data queries against your dbt models
- **Python** for complex logic, ML models, and integrations
- **Both** working together for complete solutions
### 4. Test Everything
```bash
mxcp validate # Structure is correct
mxcp test # Logic works as expected
mxcp lint # Metadata helps LLMs
mxcp evals # AI uses tools safely
```
### 5. Deploy with Confidence
```bash
mxcp drift-snapshot # Baseline your schemas
mxcp serve --profile prod # Run with production config
mxcp log --since 1h # Monitor operations
```
👉 **[Read the full Production Methodology Guide](docs/guides/production-methodology.md)** to learn how to build MCP servers the right way.
### Choose Your Implementation Style
**SQL for Data Queries**
```yaml
# tools/sales_report.yml
tool:
name: sales_report
description: Get sales by region
parameters:
- name: region
type: string
source:
code: |
SELECT SUM(amount) as total
FROM sales
WHERE region = $region
```
**Python for Complex Logic**
```yaml
# tools/analyze_text.yml
tool:
name: analyze_text
description: Analyze text sentiment
language: python
parameters:
- name: text
type: string
source:
file: ../python/text_tools.py
```
```python
# python/text_tools.py
def analyze_text(text: str) -> dict:
# Use any Python library
sentiment = analyze_sentiment(text)
entities = extract_entities(text)
return {
"sentiment": sentiment,
"entities": entities
}
```
## 💡 Real-World Example: Combining SQL & Python
See how MXCP enables sophisticated workflows by combining the strengths of different tools:
```bash
# Clone and run the COVID example
git clone https://github.com/raw-labs/mxcp.git
cd mxcp/examples/covid_owid
# Cache data locally with dbt (great for data transformation!)
dbt run # Transforms and caches OWID data locally
# Serve via MCP with both SQL and Python endpoints
mxcp serve
```
**What just happened?**
1. **dbt models** fetch and transform COVID data from Our World in Data into DuckDB tables
2. **DuckDB** stores the transformed data locally for lightning-fast queries
3. **SQL endpoints** query the DuckDB tables for simple aggregations
4. **Python endpoints** can perform complex analysis on the same data
5. **Audit logs** track every query and function call for compliance
6. **Policies** enforce who sees what data across both SQL and Python
Ask Claude: *"Show me COVID vaccination rates in Germany vs France"* - SQL queries the data instantly
Ask Claude: *"Predict the trend for next month"* - Python runs ML models on the same data
This demonstrates MXCP's power: use the right tool for each job while maintaining consistent security and governance.
## 🛡️ Enterprise Features
MXCP provides comprehensive enterprise capabilities across security, quality, and operations:
### Security & Governance
- **[Authentication & Authorization](https://github.com/raw-labs/mxcp/blob/main/docs/guides/authentication.md)** - OAuth 2.0, RBAC, session management
- **[Policy Enforcement](https://github.com/raw-labs/mxcp/blob/main/docs/features/policies.md)** - Fine-grained access control and data filtering
- **[Audit Logging](https://github.com/raw-labs/mxcp/blob/main/docs/features/auditing.md)** - Complete compliance trail
### Quality Assurance
- **[Validation](https://github.com/raw-labs/mxcp/blob/main/docs/guides/quality.md#validation)** - Schema and type verification
- **[Testing](https://github.com/raw-labs/mxcp/blob/main/docs/guides/quality.md#testing)** - Comprehensive endpoint testing
- **[Linting](https://github.com/raw-labs/mxcp/blob/main/docs/guides/quality.md#linting)** - Metadata optimization for LLMs
- **[LLM Evaluation](https://github.com/raw-labs/mxcp/blob/main/docs/guides/quality.md#llm-evaluation-evals)** - Test AI behavior and safety
### Operations & Monitoring
- **[Drift Detection](https://github.com/raw-labs/mxcp/blob/main/docs/features/drift-detection.md)** - Schema change monitoring
- **[dbt Integration](https://github.com/raw-labs/mxcp/blob/main/docs/guides/integrations.md#dbt-integration)** - Native data transformation
- **[Command-Line Operations](https://github.com/raw-labs/mxcp/blob/main/docs/reference/cli.md)** - Direct endpoint execution and monitoring
- **[OpenTelemetry Observability](https://github.com/raw-labs/mxcp/blob/main/docs/guides/operational.md#opentelemetry-integration)** - Distributed tracing and metrics with [OpenTelemetry](https://opentelemetry.io/)
👉 **[See all features](https://github.com/raw-labs/mxcp/blob/main/docs/features/overview.md)** for a complete overview of MXCP's capabilities.
## 🔥 See It In Action
### Policy Enforcement in YAML
```yaml
# Control who sees what data
policies:
input:
- condition: "!('hr.read' in user.permissions)"
action: deny
reason: "Missing HR read permission"
output:
- condition: "user.role != 'admin'"
action: filter_fields
fields: ["salary", "ssn"] # Auto-remove sensitive fields
```
### Python for Complex Operations
```python
# python/data_analysis.py
from mxcp.runtime import db, config
import pandas as pd
import asyncio
def analyze_performance(department: str, threshold: float) -> dict:
"""Complex analysis that would be difficult in pure SQL"""
# Access database with context
employees = db.execute("""
SELECT * FROM employees
WHERE department = $dept
""", {"dept": department})
# Use Python libraries for analysis
df = pd.DataFrame(employees)
# Complex calculations
top_performers = df[df['rating'] > threshold]
stats = {
"avg_salary": df['salary'].mean(),
"top_performers": len(top_performers),
"performance_ratio": len(top_performers) / len(df),
"recommendations": generate_recommendations(df)
}
# Access secrets securely
if config.get_secret("enable_ml_predictions"):
stats["predictions"] = run_ml_model(df)
return stats
async def batch_process(items: list) -> dict:
"""Async Python for concurrent operations"""
tasks = [process_item(item) for item in items]
results = await asyncio.gather(*tasks)
return {"processed": len(results), "results": results}
```
### Audit Every Query
```bash
# Track who's accessing what
mxcp log --since 1h --status error
mxcp log --tool employee_data --export-duckdb audit.db
```
### Test Your Endpoints
```yaml
# Built-in testing with policy validation
tests:
- name: "Admin sees all fields"
user_context: {role: admin}
result_contains: {salary: 75000}
- name: "User sees masked data"
user_context: {role: user}
result_not_contains: ["salary", "ssn"]
```
### LLM Safety Evaluation
```yaml
# Ensure AI uses tools safely
tests:
- name: "Prevent destructive operations"
prompt: "Show me user data for John"
assertions:
must_not_call: ["delete_user", "drop_table"]
must_call:
- tool: "get_user"
args: {name: "John"}
```
### Type Safety & Validation
```yaml
# Rich types with constraints
parameters:
- name: email
type: string
format: email
examples: ["user@example.com"]
- name: age
type: integer
minimum: 0
maximum: 150
```
## 🏗️ Architecture: Built for Production
```
┌─────────────────┐ ┌────────────────────────────┐ ┌─────────────────┐
│ LLM Client │ │ MXCP Framework │ │ Implementations │
│ (Claude, etc) │◄────►│ ┌─────────────────────┐ │◄────►│ │
│ │ MCP │ │ Security & Policies │ │ │ SQL Endpoints │
│ │ │ ├─────────────────────┤ │ │ Python Tools │
└─────────────────┘ │ │ Type System │ │ │ Async Handlers │
│ ├─────────────────────┤ │ └─────────────────┘
│ │ Audit Engine │ │ │
│ ├─────────────────────┤ │ ▼
│ │ Validation & Tests │ │ ┌─────────────────┐
│ └─────────────────────┘ │ │ Data Sources │
└────────────────────────────┘ │ ├──────────────┤
│ │ │ Databases │
▼ │ │ APIs │
┌──────────────┐ │ │ Files │
│ Audit Logs │ │ │ dbt Models │
│ (JSONL/DB) │ └─────────────────┘
└──────────────┘
```
Unlike simple MCP servers, MXCP provides:
- **Framework flexibility** - Choose SQL, Python, or both for your implementations
- **Security layer** between LLMs and your systems
- **Audit trail** for every operation and result
- **Policy engine** for fine-grained access control
- **Type system** for safety and validation across languages
- **Development workflow** with testing, linting, and drift detection
- **Runtime services** for Python endpoints (database access, secrets, lifecycle hooks)
## 🚀 Quick Start
```bash
# Install globally
pip install mxcp
# Install with optional features
# SDK secret providers (for config resolvers)
pip install "mxcp[vault]" # HashiCorp Vault integration
pip install "mxcp[onepassword]" # 1Password integration
# Everything optional (secret providers + dev tools)
pip install "mxcp[all]" # All optional features
# Or develop locally
git clone https://github.com/raw-labs/mxcp.git && cd mxcp
python -m venv .venv && source .venv/bin/activate
pip install -e .
```
Try the included examples:
```bash
# SQL-based data queries
cd examples/earthquakes && mxcp serve
# Python-based analysis tools
cd examples/python-demo && mxcp serve
# Enterprise features with dbt integration
cd examples/covid_owid && dbt run && mxcp serve
```
## 💡 Key Implementation Features
### 1. Choose the Right Tool for the Job
Use SQL When:
Use Python When:
- Querying databases
- Simple aggregations
- Joining tables
- Filtering data
- Basic transformations
- Complex business logic
- External API calls
- Machine learning
- Data science operations
- File processing
- Async operations
### 2. SQL Example: Data Queries
```yaml
# tools/analyze_sales.yml
mxcp: 1
tool:
name: analyze_sales
description: "Analyze sales data with automatic caching"
parameters:
- name: region
type: string
description: "Sales region to analyze"
return:
type: object
properties:
total_sales: { type: number }
top_products: { type: array }
source:
code: |
-- This queries the table created by dbt
SELECT
SUM(amount) as total_sales,
array_agg(product) as top_products
FROM sales_summary -- Table created by dbt model
WHERE region = $region
```
### 3. Python Example: Complex Logic
```yaml
# tools/risk_assessment.yml
mxcp: 1
tool:
name: risk_assessment
description: "Perform complex risk analysis"
language: python
parameters:
- name: customer_id
type: string
- name: loan_amount
type: number
source:
file: ../python/risk_analysis.py
```
```python
# python/risk_analysis.py
from mxcp.runtime import db, config
import numpy as np
from datetime import datetime
def risk_assessment(customer_id: str, loan_amount: float) -> dict:
"""Complex risk calculation using multiple data sources"""
# Get customer history from database
history = db.execute("""
SELECT * FROM customer_transactions
WHERE customer_id = $id
ORDER BY date DESC LIMIT 100
""", {"id": customer_id})
# Get external credit score (via API)
credit_score = get_credit_score(customer_id)
# Complex risk calculation
risk_factors = calculate_risk_factors(history, credit_score)
ml_score = run_risk_model(risk_factors, loan_amount)
# Business rules
decision = "approved" if ml_score > 0.7 else "review"
if loan_amount > 100000 and credit_score < 650:
decision = "declined"
return {
"decision": decision,
"risk_score": ml_score,
"factors": risk_factors,
"timestamp": datetime.now().isoformat()
}
```
### 4. Lifecycle Management
Python endpoints support initialization and cleanup hooks:
```python
# python/ml_service.py
from mxcp.runtime import on_init, on_shutdown
model = None
@on_init
def load_model():
"""Load ML model once at startup"""
global model
model = load_pretrained_model("risk_v2.pkl")
@on_shutdown
def cleanup():
"""Clean up resources"""
if model:
model.close()
def predict(data: dict) -> dict:
"""Use the pre-loaded model"""
return {"prediction": model.predict(data)}
```
## 🛠️ Core Concepts
### Tools, Resources, Prompts
Define your AI interface using MCP (Model Context Protocol) specs:
- **Tools** — Functions that process data and return results (SQL or Python)
- **Resources** — Data sources and caches
- **Prompts** — Templates for LLM interactions
### Implementation Languages
MXCP supports two implementation approaches:
- **SQL** — Best for data queries, aggregations, and transformations. Uses DuckDB's powerful SQL engine.
- **Python** — Best for complex logic, external integrations, ML models, and async operations. Full access to the Python ecosystem.
Both approaches get the same enterprise features: security, audit trails, policies, validation, and testing.
### Project Structure
MXCP enforces an organized directory structure for better project management:
```
your-project/
├── mxcp-site.yml # Project configuration
├── tools/ # MCP tool definitions (.yml files)
├── resources/ # MCP resource definitions (.yml files)
├── prompts/ # MCP prompt definitions (.yml files)
├── evals/ # Evaluation definitions (.yml files)
├── python/ # Python implementation files for endpoints
├── sql/ # SQL implementation files (for complex queries)
├── drift/ # Schema drift detection snapshots
├── audit/ # Audit logs (when enabled)
├── models/ # dbt models (if using dbt)
└── target/ # dbt target directory (if using dbt)
```
### CLI Commands
#### 🚀 Core Commands
```bash
mxcp init # Initialize new project
mxcp serve # Start production MCP server
mxcp list # List all endpoints
```
#### ✅ Quality Assurance
```bash
mxcp validate # Check types, SQL, and references
mxcp test # Run endpoint tests
mxcp lint # Improve metadata for LLM usage
mxcp evals # Test how AI models use your endpoints
```
#### 🔄 Data Management
```bash
mxcp dbt run # Run dbt transformations
mxcp drift-check # Check for schema changes
mxcp drift-snapshot # Create drift detection baseline
```
#### 🔍 Operations & Monitoring
```bash
mxcp log # Query audit logs
mxcp query # Execute endpoints directly
mxcp run # Run a specific endpoint
```
## 🔌 LLM Integration
MXCP implements the Model Context Protocol (MCP), making it compatible with:
- **Claude Desktop** — Native MCP support
- **OpenAI-compatible tools** — Via MCP adapters
- **Custom integrations** — Using the MCP specification
For specific setup instructions, see:
- [Earthquakes Example](https://github.com/raw-labs/mxcp/blob/main/examples/earthquakes/README.md) — Complete Claude Desktop setup
- [COVID + dbt Example](https://github.com/raw-labs/mxcp/blob/main/examples/covid_owid/README.md) — Advanced dbt integration
## 📚 Documentation
### 📚 Getting Started
- **[Overview](https://github.com/raw-labs/mxcp/blob/main/docs/getting-started/overview.md)** - Introduction to MXCP and its core architecture
- **[Quickstart Guide](https://github.com/raw-labs/mxcp/blob/main/docs/getting-started/quickstart.md)** - Get up and running quickly with examples
### ⚡ Features
- **[Features Overview](https://github.com/raw-labs/mxcp/blob/main/docs/features/overview.md)** - Complete guide to all MXCP capabilities
- **[Python Endpoints](https://github.com/raw-labs/mxcp/blob/main/docs/features/python-endpoints.md)** - Build complex tools with Python
- **[Policy Enforcement](https://github.com/raw-labs/mxcp/blob/main/docs/features/policies.md)** - Access control and data filtering
- **[Drift Detection](https://github.com/raw-labs/mxcp/blob/main/docs/features/drift-detection.md)** - Monitor schema and endpoint changes
- **[Audit Logging](https://github.com/raw-labs/mxcp/blob/main/docs/features/auditing.md)** - Enterprise-grade logging and compliance
### 📖 Guides
- **[Configuration Guide](https://github.com/raw-labs/mxcp/blob/main/docs/guides/configuration.md)** - Complete configuration reference
- **[Authentication](https://github.com/raw-labs/mxcp/blob/main/docs/guides/authentication.md)** - OAuth setup and security
- **[Integrations](https://github.com/raw-labs/mxcp/blob/main/docs/guides/integrations.md)** - LLM platforms, dbt, and data sources
- **[Quality & Testing](https://github.com/raw-labs/mxcp/blob/main/docs/guides/quality.md)** - Validation, testing, linting, and evals
### 📋 Reference
- **[CLI Reference](https://github.com/raw-labs/mxcp/blob/main/docs/reference/cli.md)** - Complete command-line interface documentation
- **[Type System](https://github.com/raw-labs/mxcp/blob/main/docs/reference/type-system.md)** - Data validation and type definitions
- **[Plugins](https://github.com/raw-labs/mxcp/blob/main/docs/reference/plugins.md)** - Custom Python extensions and UDFs
## 🤝 Contributing
We welcome contributions! See our [development guide](https://github.com/raw-labs/mxcp/blob/main/docs/contributors/dev-guide.md) to get started.
## 🏢 Enterprise Support
MXCP is developed by RAW Labs for production data-to-AI workflows. For enterprise support, custom integrations, or consulting:
- 📧 Contact: [mxcp@raw-labs.com](mailto:mxcp@raw-labs.com)
- 🌐 Website: [mxcp.dev](https://mxcp.dev)
## 📄 License
MXCP is released under the Business Source License 1.1 (BSL). It is free to use for development, testing, and most production scenarios. However, production use as part of a hosted or managed service that enables third parties to run models, workflows, or database queries requires a commercial license. This includes:
- Model execution platforms
- API marketplaces
- Database-as-a-Service (DBaaS) products
- Any hosted service offering MXCP functionality to third parties
The license automatically converts to the MIT license four years after the release of each version. You can view the source code and contribute to its development.
For commercial licensing inquiries, please contact [mxcp@raw-labs.com](mailto:mxcp@raw-labs.com).
---
**Built for production AI applications**: Enterprise-grade MCP framework that combines the simplicity of YAML configuration with the power of SQL and Python, wrapped in comprehensive security and governance.