{"id":49292395,"url":"https://github.com/keboola/agnes-the-ai-analyst","last_synced_at":"2026-06-13T11:01:23.809Z","repository":{"id":350180865,"uuid":"1205096772","full_name":"keboola/agnes-the-ai-analyst","owner":"keboola","description":"AI Data Analyst — open-source data distribution platform for AI analytical systems","archived":false,"fork":false,"pushed_at":"2026-06-11T13:11:34.000Z","size":23165,"stargazers_count":4,"open_issues_count":47,"forks_count":3,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-11T13:22:11.623Z","etag":null,"topics":["agnes"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/keboola.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-04-08T16:22:40.000Z","updated_at":"2026-06-11T12:26:41.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/keboola/agnes-the-ai-analyst","commit_stats":null,"previous_names":["keboola/agnes-the-ai-analyst"],"tags_count":2778,"template":false,"template_full_name":null,"purl":"pkg:github/keboola/agnes-the-ai-analyst","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keboola%2Fagnes-the-ai-analyst","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keboola%2Fagnes-the-ai-analyst/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keboola%2Fagnes-the-ai-analyst/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keboola%2Fagnes-the-ai-analyst/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/keboola","download_url":"https://codeload.github.com/keboola/agnes-the-ai-analyst/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keboola%2Fagnes-the-ai-analyst/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34281700,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-13T02:00:06.617Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agnes"],"created_at":"2026-04-26T01:01:30.996Z","updated_at":"2026-06-13T11:01:23.760Z","avatar_url":"https://github.com/keboola.png","language":"Python","funding_links":[],"categories":["🤖 AI \u0026 Machine Learning"],"sub_categories":[],"readme":"# Agnes — AI Data Analyst\n\nAgnes is an open-source data distribution platform for AI analytical systems. It extracts data from configured sources into DuckDB, serves it via a FastAPI backend, and distributes Parquet files to analysts who query them locally using Claude Code and DuckDB.\n\nEach data source produces a self-describing `extract.duckdb` file. The `SyncOrchestrator` attaches all extract databases into a master `analytics.duckdb`, making every table available through a unified view layer without copying data unnecessarily.\n\n## Architecture: extract.duckdb Contract\n\nEvery connector produces the same output structure:\n\n```\n/data/extracts/{source_name}/\n├── extract.duckdb          ← _meta table + views\n└── data/                   ← parquet files (local sources only)\n```\n\nThe orchestrator scans `/data/extracts/*/extract.duckdb`, attaches each into `analytics.duckdb`, and creates master views.\n\n```\n┌──────────────┐  ┌──────────────┐  ┌──────────────┐\n│   Keboola    │  │   BigQuery   │  │   Jira       │\n│  extractor   │  │  extractor   │  │  webhooks    │\n│ (DuckDB ext) │  │ (remote BQ)  │  │ (incremental)│\n└──────┬───────┘  └──────┬───────┘  └──────┬───────┘\n       │                 │                 │\n       ▼                 ▼                 ▼\n   extract.duckdb    extract.duckdb    extract.duckdb\n   + data/*.parquet  (views → BQ)      + data/*.parquet\n       │                 │                 │\n       └─────────────────┼─────────────────┘\n                         ▼\n              SyncOrchestrator.rebuild()\n              ATTACH → master views in analytics.duckdb\n                         │\n              ┌──────────┼──────────┐\n              ▼          ▼          ▼\n          FastAPI      CLI\n          (serve)    (agnes pull)\n```\n\n## Supported Data Sources\n\n| Mode | Distribution | Sources | Use when |\n|------|--------------|---------|----------|\n| **Batch pull** (`local`) | Parquet on disk, scheduled | Keboola | Source has a native bulk-export and the table fits on disk |\n| **Materialized SQL** (`materialized`) | Parquet on disk, scheduled query | BigQuery, Keboola | Source table is too large to mirror as-is; you want a curated subset / aggregate on disk |\n| **Remote attach** (`remote`) | View only, no download | BigQuery | Table is too large to materialize; latency cost of remote query is acceptable |\n| **Real-time push** | Incremental parquet | Jira | Source is event-driven and you need sub-minute freshness |\n\nThe first three modes are what `agnes pull` distributes to analysts. The fourth is server-side only — analysts query Jira data through the same `agnes pull`-distributed parquets.\n\nAdmins manage per-source registrations through the `/admin/tables` UI (per-connector tabs for BigQuery / Keboola / Jira) or the `agnes admin register-table` CLI; per-row \"Manage access\" deep-links to `/admin/access` for granting tables to user groups via `resource_grants(group, ResourceType.TABLE, table_id)`.\n\nAnalysts get a closed loop with Claude Code: `agnes init` writes `\u003cworkspace\u003e/.claude/settings.json` with SessionStart (`agnes pull --quiet`) and SessionEnd (`agnes push --quiet`) hooks so every Claude Code session starts with fresh RBAC-filtered parquets and ends with the session log uploaded back.\n\nAdding a new source means creating `connectors/\u003cname\u003e/extractor.py` that produces `extract.duckdb` with a `_meta` table (`table_name`, `description`, `rows`, `size_bytes`, `extracted_at`, `query_mode`). The orchestrator attaches it automatically.\n\n## Quick Start with Docker\n\n```bash\n# Clone the repository\ngit clone https://github.com/keboola/agnes-the-ai-analyst.git\ncd agnes-the-ai-analyst\n\n# Copy and edit configuration\ncp config/instance.yaml.example config/instance.yaml\ncp config/.env.template .env\n# Edit both files for your environment\n\n# Start the app and scheduler\ndocker compose up\n\n# Start with all optional services (Telegram bot, etc.)\ndocker compose --profile full up\n\n# Start with TLS (Caddy on :443 with corporate-CA certs from /data/state/certs)\ndocker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.tls.yml \\\n    --profile tls up -d\n```\n\nOnce running, the FastAPI app is available at `http://localhost:8000` (or `https://$DOMAIN` in TLS mode). See [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) for cert provisioning + auto-rotation via `scripts/ops/agnes-tls-rotate.sh`. Trigger a manual sync:\n\n```bash\ncurl -X POST http://localhost:8000/api/sync/trigger\n```\n\n## Local sync \u0026 auto-update\n\nAnalysts run Claude Code against a local DuckDB built from RBAC-filtered parquets pulled from the server. `agnes pull` is the distribution path:\n\n```bash\nagnes pull             # delta-pull: manifest → MD5 compare → download changed → rebuild views\nagnes pull --quiet     # same, no progress output (for hooks/cron)\nagnes push  # push session jsonl + CLAUDE.local.md back to the server\n```\n\n`agnes init` writes Claude Code lifecycle hooks into `\u003cworkspace\u003e/.claude/settings.json`:\n\n- `SessionStart` → `agnes pull --quiet` — fresh data on every session\n- `SessionEnd` → `agnes push --quiet` — uploads notes and session log\n\nHooks live at workspace level so they only fire in this analyst workspace, not in unrelated Claude Code sessions on the same machine.\n\n### Admin: which tables auto-sync to whom\n\nThe auto-sync set per analyst is the intersection of:\n\n1. Tables with `query_mode IN ('local', 'materialized')` — these have parquets on disk and end up in the manifest\n2. Tables granted to one of the analyst's groups via `resource_grants(group, ResourceType.TABLE, table_id)` (see [`docs/RBAC.md`](docs/RBAC.md))\n\nTo enroll a new table for auto-sync, register it (or update its `query_mode`) and grant it to the relevant groups in `/admin/access`. New analysts get the same set on their next `agnes pull`.\n\nFor BigQuery, register a `query_mode='materialized'` table with a SQL body:\n\n```bash\nagnes admin register-table orders_90d \\\n    --source-type bigquery \\\n    --query-mode materialized \\\n    --query @docs/queries/orders_90d.sql \\\n    --schedule \"every 6h\"\n```\n\nThe scheduler runs the query through the DuckDB BigQuery extension on each tick that's due, writes the result as a parquet, and the analyst picks it up on the next `agnes pull`. Cost guardrail: `data_source.bigquery.max_bytes_per_materialize` (default 10 GiB) — operations exceeding the BQ dry-run estimate are skipped.\n\n## Development Setup\n\n```bash\n# Create and activate virtual environment\npython3 -m venv .venv \u0026\u0026 source .venv/bin/activate\n\n# Install dependencies\nuv pip install \".[dev]\"\n\n# Run FastAPI locally with hot reload\nuvicorn app.main:app --reload\n\n# Run the test suite\npytest tests/ -v\n```\n\n## Project Structure\n\n```\n├── src/                    # Core engine\n│   ├── db.py               # DuckDB schema (system.duckdb, analytics.duckdb)\n│   ├── orchestrator.py     # SyncOrchestrator — ATTACHes extract.duckdb files\n│   ├── repositories/       # DuckDB-backed CRUD (sync_state, table_registry, users, etc.)\n│   ├── profiler.py         # Data profiling\n│   └── catalog_export.py   # OpenMetadata catalog export\n├── app/                    # FastAPI application\n│   ├── main.py             # App setup, router registration\n│   ├── api/                # REST API (sync, data, catalog, admin, auth)\n│   ├── auth/               # Auth providers (Google OAuth, email magic link, desktop JWT)\n│   └── web/                # HTML dashboard routes\n├── connectors/             # Data source connectors (extract.duckdb contract)\n│   ├── keboola/            # Keboola: extractor.py (DuckDB extension) + client.py (fallback)\n│   ├── bigquery/           # BigQuery: extractor.py (remote-only via DuckDB BQ extension)\n│   └── jira/               # Jira: webhook + incremental parquet → extract.duckdb\n├── cli/                    # CLI tool (`agnes pull`, `agnes query`, `agnes admin`)\n├── services/               # Standalone services (scheduler, telegram_bot, ws_gateway, etc.)\n├── scripts/                # Utility + migration scripts\n├── config/                 # Configuration templates (instance.yaml.example)\n├── docs/                   # Documentation + metric YAML definitions\n└── tests/                  # Test suite (633 tests)\n```\n\n## Configuration\n\n| File | Purpose |\n|------|---------|\n| `config/instance.yaml` | Instance-specific settings: branding, data source type, auth provider, Google domain |\n| `.env` | Secrets and environment variables — never committed |\n| `system.duckdb` `table_registry` table | Table definitions managed via `POST /api/admin/register-table` (or `PUT /api/admin/registry/{id}` to update) or the web UI |\n\nCopy the example to get started:\n\n```bash\ncp config/instance.yaml.example config/instance.yaml\n```\n\nSee `config/instance.yaml.example` for all available options.\n\n## Documentation\n\n**Full index: [docs/README.md](docs/README.md)** — every doc, organized by audience (analyst / operator / developer).\n\nKey entry points:\n\n- [Quickstart](docs/QUICKSTART.md) — local development setup\n- [Onboarding Guide](docs/ONBOARDING.md) — end-to-end Terraform deployment into a GCP project (recommended for production)\n- [Deployment Guide](docs/DEPLOYMENT.md) — chooses between Terraform and Docker Compose; covers OSS self-host\n- [Configuration Reference](docs/CONFIGURATION.md) — `instance.yaml`, env vars, per-instance options\n- [Architecture](ARCHITECTURE.md) — orchestrator, extractors, DB layout\n\n## Contributing\n\n1. Fork the repository and create a feature branch.\n2. Run `pytest tests/ -v` to verify all tests pass before opening a pull request.\n3. Keep commits focused and messages concise.\n4. Open a pull request against `main` with a clear description of the change.\n\nFor bugs and feature requests, open a GitHub issue.\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeboola%2Fagnes-the-ai-analyst","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkeboola%2Fagnes-the-ai-analyst","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeboola%2Fagnes-the-ai-analyst/lists"}