{"id":48299270,"url":"https://github.com/jmagar/axon","last_synced_at":"2026-06-28T03:00:41.127Z","repository":{"id":339265094,"uuid":"1160613618","full_name":"jmagar/axon","owner":"jmagar","description":"Self-hosted web crawling and RAG pipeline with MCP tooling for scrape, crawl, ingest, embed, query, and ask workflows.","archived":false,"fork":false,"pushed_at":"2026-06-21T06:14:42.000Z","size":68584,"stargazers_count":2,"open_issues_count":7,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-21T06:19:50.985Z","etag":null,"topics":["ai","claude-code","claude-code-plugins","cli","codex","gemini","homelab","llm","mcp","mcp-server","model-context-protocol","rag","rust","search","self-hosted","web-crawling"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jmagar.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"docs/contributing/checklist.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-02-18T06:43:46.000Z","updated_at":"2026-06-21T05:32:44.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jmagar/axon","commit_stats":null,"previous_names":["jmagar/axon_rust","jmagar/axon"],"tags_count":34,"template":false,"template_full_name":null,"purl":"pkg:github/jmagar/axon","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmagar%2Faxon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmagar%2Faxon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmagar%2Faxon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmagar%2Faxon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jmagar","download_url":"https://codeload.github.com/jmagar/axon/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmagar%2Faxon/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34875359,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-28T02:00:05.809Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","claude-code","claude-code-plugins","cli","codex","gemini","homelab","llm","mcp","mcp-server","model-context-protocol","rag","rust","search","self-hosted","web-crawling"],"created_at":"2026-04-04T23:43:09.418Z","updated_at":"2026-06-28T03:00:41.120Z","avatar_url":"https://github.com/jmagar.png","language":"Rust","funding_links":[],"categories":["Source Catalog"],"sub_categories":[],"readme":"# Axon\n\nVersion: 6.1.3\n\nAxon is a self-hosted RAG stack for crawling, scraping, ingesting, embedding, searching, and asking questions over indexed content. The production release is Docker Compose first: one Axon server container, Qdrant, Hugging Face TEI with `Qwen/Qwen3-Embedding-0.6B`, and Chrome for JS-heavy pages.\n\n## Production Contract\n\nSupported production runtime:\n\n- Docker Compose only.\n- Qdrant only for vector storage.\n- Hugging Face TEI only for embeddings.\n- `Qwen/Qwen3-Embedding-0.6B` as the production embedding model.\n- Gemini CLI is the default LLM synthesis path; OpenAI-compatible endpoints\n  such as llama.cpp are supported when configured with `AXON_LLM_BACKEND=openai-compat`.\n- Web search/research uses a self-hosted SearXNG instance (`AXON_SEARXNG_URL`) when\n  configured, falling back to Tavily (`TAVILY_API_KEY`) otherwise.\n- Local NVIDIA RTX 4070 target with NVIDIA Container Toolkit.\n- CLI and MCP run all actions in-process; deploy the `axon serve` container only when you need HTTP API access.\n- One shared config home: `~/.axon/.env`, `~/.axon/config.toml`, `~/.axon/jobs.db`, `~/.axon/output`, `~/.axon/logs`, `~/.axon/artifacts`, `~/.axon/screenshots`, and `~/.axon/tei`. Qdrant storage lives on the host running Qdrant; dookie points at tootie by default.\n\nNot supported in the production path:\n\n- systemd deployment of the Axon binary.\n- Postgres, Redis, RabbitMQ, AMQP, or external worker services.\n- OpenAI-compatible first-run LLM configuration. Configure\n  `AXON_LLM_BACKEND=openai-compat` manually after setup when using llama.cpp or\n  another OpenAI-compatible `/v1/chat/completions` endpoint.\n- Neo4j or graph retrieval.\n- Multiple competing `.env` or `config.toml` locations.\n\n## Install\n\n### Linux\n\nPrerequisites:\n\n- Linux x86_64.\n- Docker and Docker Compose.\n- NVIDIA driver, `nvidia-smi`, and NVIDIA Container Toolkit.\n- Gemini CLI installed and already authenticated, unless using a configured\n  OpenAI-compatible endpoint for LLM synthesis.\n- `curl`, `sha256sum`, and `install`.\n\nOne-line installer:\n\n```bash\ncurl -fsSL https://raw.githubusercontent.com/jmagar/axon/main/install.sh | sh\n```\n\nThe installer verifies the release checksum, installs the `axon` binary to `~/.local/bin`, then delegates setup to `axon setup`.\n\nUseful installer controls:\n\n```bash\nAXON_INSTALL_DRY_RUN=1 ./install.sh\nAXON_INSTALL_PREFIX=/opt/axon ./install.sh\nAXON_VERSION=vX.Y.Z ./install.sh   # pin a specific release; defaults to latest\nAXON_INSTALL_SKIP_SETUP=1 ./install.sh\n```\n\n### Windows\n\nPrerequisites:\n\n- Windows x86_64 (PowerShell 5.1+ or PowerShell Core).\n- Docker Desktop with the WSL 2 backend (for the full stack).\n- NVIDIA driver and NVIDIA Container Toolkit (for GPU acceleration).\n- Gemini CLI installed and already authenticated, or a configured\n  OpenAI-compatible endpoint for LLM synthesis.\n\nOne-line installer (PowerShell):\n\n```powershell\nirm https://raw.githubusercontent.com/jmagar/axon/main/install.ps1 | iex\n```\n\nThe installer verifies the release checksum, installs `axon.exe` to `%USERPROFILE%\\.local\\bin`, adds it to your user PATH, then delegates setup to `axon setup`.\n\nUseful installer controls:\n\n```powershell\n$env:AXON_INSTALL_DRY_RUN='1'; irm .../install.ps1 | iex\n$env:AXON_INSTALL_PREFIX='C:\\tools\\axon'; irm .../install.ps1 | iex\n$env:AXON_VERSION='vX.Y.Z'; irm .../install.ps1 | iex  # pin a version\n$env:AXON_INSTALL_SKIP_SETUP='1'; irm .../install.ps1 | iex\n```\n\nClaude Code plugin install:\n\n```bash\nclaude plugin install \u003cpath-to-this-repo\u003e\n```\n\nThe plugin ships no binary. Install `axon` first via `install.sh`, then install the plugin. Its `SessionStart` hook runs `scripts/plugin-setup.sh`, which syncs `CLAUDE_PLUGIN_OPTION_*` settings into process env and delegates to `axon setup plugin-hook`. That subcommand is **probe-only and never deploys**: it checks `/readyz` and exits silently when the stack is up, or prints a one-line `run /axon-deploy` advisory when it is down. `ConfigChange` runs the same script so updated plugin settings take effect immediately. Provisioning is the `/axon-deploy` slash command (or `axon setup` / `axon compose up`). No systemd unit is created.\n\n## Setup Flow\n\n`axon setup` is the convenience bootstrap path. It is idempotent and safe to rerun. It:\n\n1. Creates or refreshes `~/.axon`.\n2. Creates or preserves `~/.axon/config.toml`.\n3. Creates or preserves `~/.axon/.env`, filling only missing runtime values and preserving secrets.\n4. Writes Docker Compose assets under `~/.axon/compose`.\n5. Checks Docker, Docker Compose, `nvidia-smi`, Gemini CLI auth, and OAuth config when requested.\n6. Pulls and starts the Compose stack.\n7. Waits for Qdrant, TEI, Chrome, and Axon server health.\n\nFocused commands:\n\n```bash\naxon setup          # init + compose up + preflight\naxon setup init     # create ~/.axon, config.toml, .env, and compose assets\naxon preflight      # check prerequisites, auth config, and service readiness\naxon compose up       # pull/start services, then follow logs until Ctrl-C\naxon compose down     # stop services\naxon compose restart  # restart services\naxon compose rebuild  # rebuild the Axon image and start services\naxon smoke          # TEI prewarm + crawl/ask proof\naxon setup plugin-hook  # probe-only SessionStart path (never deploys; advises /axon-deploy when down)\naxon setup targets  # list SSH aliases discovered from ~/.ssh/config (informational)\n```\n\nFor local bearer-token operation, no manual env values are required. `setup init`\ndefaults to loopback MCP HTTP, writes `AXON_MCP_AUTH_MODE=bearer`, and generates\n`AXON_MCP_HTTP_TOKEN`. Optional features need credentials: Gemini auth under\n`~/.gemini` for default LLM features or `AXON_LLM_BACKEND=openai-compat` plus\n`AXON_OPENAI_BASE_URL` and `AXON_SYNTHESIS_OPENAI_MODEL` for OpenAI-compatible synthesis,\n`TAVILY_API_KEY` for search/research, `GITHUB_TOKEN` for higher-rate GitHub\ningest, and `REDDIT_CLIENT_ID` plus `REDDIT_CLIENT_SECRET` for Reddit ingest.\nOAuth mode also requires\n`AXON_MCP_PUBLIC_URL`, `AXON_MCP_GOOGLE_CLIENT_ID`,\n`AXON_MCP_GOOGLE_CLIENT_SECRET`, and `AXON_MCP_AUTH_ADMIN_EMAIL`.\n\nThe warm-path setup goal is under 2 minutes once images and model weights are cached. Cold starts that pull images and model weights can take longer; target-hardware timing still needs to be measured against published release artifacts.\n\n## Docker Stack\n\nThe production compose file starts:\n\n| Service | Purpose | Host bind |\n| --- | --- | --- |\n| `axon` | HTTP server, web panel, MCP HTTP, action API, in-process workers | `127.0.0.1:8001` |\n| `axon-qdrant` | vector storage, optional local service; production points at tootie by default | `127.0.0.1:53333`, `127.0.0.1:53334` |\n| `axon-tei` | Qwen3 embeddings through TEI | `127.0.0.1:52000` |\n| `axon-chrome` | browser rendering and CDP proxy | `127.0.0.1:6000`, `127.0.0.1:9222`, `127.0.0.1:9223` |\n\nStart manually:\n\n```bash\ndocker compose --env-file ~/.axon/.env -f ~/.axon/compose/docker-compose.yaml up -d\n```\n\nCheck:\n\n```bash\ndocker compose --env-file ~/.axon/.env -f ~/.axon/compose/docker-compose.yaml ps\naxon preflight\naxon doctor\n```\n\nStop:\n\n```bash\ndocker compose --env-file ~/.axon/.env -f ~/.axon/compose/docker-compose.yaml down\n```\n\nDevelopment stack:\n\n```bash\ncargo build --bin axon\ndocker compose --env-file .env.example -f docker-compose.yaml build axon\ndocker compose --env-file ~/.axon/.env -f docker-compose.yaml up -d axon\n```\n\nThe development stack uses the production infrastructure definitions but runs\n`axon` from the bind-mounted local debug binary under `target/debug`, inside the\nnewer `axon:dev-runtime` image.\n\n## Configuration\n\nAxon has two config layers:\n\n| File | Purpose |\n| --- | --- |\n| `~/.axon/.env` | URLs, secrets, auth, Docker interpolation, runtime bootstrap values |\n| `~/.axon/config.toml` | non-secret tuning and behavior |\n\nPrecedence:\n\n```text\nCLI flags \u003e environment variables \u003e ~/.axon/config.toml \u003e built-in defaults\n```\n\nKeep in `.env`:\n\n- URLs: `QDRANT_URL`, `TEI_URL`, `AXON_CHROME_REMOTE_URL`, `AXON_SEARXNG_URL`.\n- Secrets: `AXON_MCP_HTTP_TOKEN`, `TAVILY_API_KEY`, `GITHUB_TOKEN`, `GITLAB_TOKEN`, `GITEA_TOKEN`, Reddit credentials, OAuth credentials, `HF_TOKEN`.\n- Docker/runtime bootstrap: `AXON_HOME`, `AXON_DATA_DIR`, `AXON_IMAGE`, `AXON_MCP_HTTP_PUBLISH`, `TEI_HTTP_PORT`, GPU device values.\n- LLM runtime pointers when needed: `AXON_HEADLESS_GEMINI_CMD`,\n  `AXON_HEADLESS_GEMINI_HOME`, `AXON_LLM_BACKEND`, `AXON_OPENAI_BASE_URL`,\n  `AXON_SYNTHESIS_OPENAI_MODEL` (legacy alias: `AXON_OPENAI_MODEL`), and\n  optional `AXON_OPENAI_API_KEY`.\n\nPut in `config.toml`:\n\n- collection/search/ask tuning.\n- worker and job limits.\n- TEI client tuning.\n- Qdrant batch sizing.\n- logging behavior that is not a process-launch concern.\n- UI/output behavior.\n\n`~/.axon/.env` under the config home is never loaded through a symlink.\n\n## First Run\n\nAfter setup:\n\n```bash\naxon doctor\naxon crawl https://example.com --wait true\naxon ask \"What did we crawl?\"\n```\n\nCLI and MCP commands always run in-process against Qdrant and TEI. `axon serve` exposes the same operations over HTTP (`/v1/*`, MCP-over-HTTP) for clients that want API access to a deployed instance.\n\n## Notable Capabilities\n\n- **Hybrid search.** New Qdrant collections are created with named `dense` + `bm42` sparse vectors and queried with Reciprocal Rank Fusion (RRF). Legacy unnamed collections fall back to dense-only cosine search. Tune via the `[search]` section in `config.toml`; run `axon migrate --from \u003cold\u003e --to \u003cnew\u003e` to copy a legacy unnamed collection into a new named-mode one, then point `AXON_COLLECTION` at it.\n- **Normalized source planning.** Crawl manifests, scrape results, ingest sources, local files, and memory records are normalized into `SourceDocument` before chunking. The source-doc planner is the only layer that chooses markdown/plain-text/file chunking and emits `PreparedDoc` values for TEI/Qdrant. File origins (`GitFile`, `LocalFile`) get tree-sitter-aware code chunks where supported plus `code_*`, `symbol_*`, `chunk_locator`, `source_range`, and `chunk_content_kind` payload metadata; unsupported code falls back to prose chunks with fallback metadata instead of losing enrichment.\n- **Vertical extractors.** `scrape` (and the `scrape` MCP/REST action) auto-routes known URLs to structured per-site extractors (GitHub, PyPI, npm, crates.io, Reddit, YouTube, and more) instead of generic HTML→markdown. Vertical payloads keep `extractor_name` and bounded structured data through the same source-doc planner used by generic scrape. Disable with `AXON_ENABLE_VERTICALS=false` or the `[verticals]` config section.\n- **Freshness schedules.** The CLI can create recurring schedules with `--fresh \u003cNd\u003e` on `scrape`, `crawl`, `embed`, and `ingest`. `axon serve`/`axon mcp` run the scheduler from SQLite, replaying safe versioned snapshots through the normal service/job paths. Freshness management is CLI-only in v1: use `axon fresh list`, `axon fresh run-now \u003cid\u003e`, and `axon fresh history \u003cid\u003e`.\n- **Web panel.** `axon serve` hosts an Aurora-styled control panel at the bind address (default `http://127.0.0.1:8001`) with a first-run setup flow, config/stack inspection, and a command runner, alongside the `/v1/*` REST surface and OpenAPI docs at `/docs`.\n\n## CLI Map\n\nCore:\n\n- `scrape \u003curl\u003e...`\n- `crawl \u003curl\u003e...`\n- `map \u003curl\u003e`\n- `extract \u003curl\u003e...`\n- `embed [input]`\n- `query \u003ctext\u003e`\n- `retrieve \u003curl\u003e`\n- `ask \u003cquestion\u003e`\n- `summarize \u003curl\u003e...`\n- `evaluate \u003cquestion\u003e`\n- `diff \u003curl-a\u003e \u003curl-b\u003e` — show what changed between two URLs (content/metadata/links)\n- `brand \u003curl\u003e` — extract brand identity: colors, fonts, logos, favicon\n\nDiscovery and ingest:\n\n- `search \u003cquery\u003e` — web search via SearXNG (or Tavily), auto-queues crawl jobs\n- `research \u003cquery\u003e` — multi-source web research with LLM synthesis\n- `suggest [focus]`\n- `endpoints \u003curl\u003e` — discover API endpoints from page HTML and JavaScript bundles\n- `ingest \u003ctarget\u003e` — GitHub, GitLab, Gitea/Forgejo, generic Git, Reddit, or YouTube\n- `sessions` — index AI session exports (Claude, Codex, Gemini)\n\nOperations:\n\n- `setup`\n- `update` — install the latest GitHub Release binary and sync the local container\n- `doctor`\n- `debug`\n- `serve`\n- `mcp`\n- `status`\n- `monitor jobs` — stream job lifecycle events (start/complete/fail/cancel)\n- `fresh` — list, run, and inspect CLI-created freshness schedules\n- `sources`\n- `domains`\n- `stats`\n- `watch`\n- `refresh [filter]` — re-crawl/re-ingest previously indexed origins (full docs refresh)\n- `dedupe`\n- `migrate`\n- `screenshot`\n- `train` — collect human preference votes on retrieved RAG candidates\n- `config`\n- `completions`\n\nUse command-specific help:\n\n```bash\naxon --help\naxon setup --help\naxon crawl --help\naxon mcp --help\n```\n\nGraph flags are not part of the production CLI, MCP, or `/v1/ask` request contract.\n\n## MCP And Auth\n\nAxon exposes one MCP tool named `axon`; actions are routed by `action` and optional `subaction`.\n\nExamples:\n\n```json\n{ \"action\": \"doctor\" }\n{ \"action\": \"scrape\", \"url\": \"https://example.com\" }\n{ \"action\": \"ask\", \"query\": \"How does setup work?\" }\n{ \"action\": \"summarize\", \"url\": \"https://example.com\" }\n{ \"action\": \"crawl\", \"subaction\": \"status\", \"job_id\": \"\u003cuuid\u003e\" }\n```\n\nHTTP auth modes:\n\n- Static bearer token with `AXON_MCP_HTTP_TOKEN`.\n- OAuth/lab-auth with `AXON_MCP_AUTH_MODE=oauth`.\n\n`/mcp` and the `/v1/*` REST routes (e.g. `/v1/ask`, `/v1/scrape`, `/v1/query`) share the same auth policy. Tokenless HTTP is only for loopback development binds.\n\n## Development\n\nBuild:\n\n```bash\ncargo build --bin axon\ncargo build --release --bin axon\n```\n\nTest and lint:\n\n```bash\ncargo fmt --all -- --check\ncargo check --workspace --all-targets --features test-helpers\ncargo clippy --workspace --all-targets --features test-helpers -- -D warnings\ncargo test --workspace --features test-helpers\n```\n\nCommon focused checks:\n\n```bash\ncargo test --test cli_help_contract -- --nocapture\ncargo test parse_setup -- --nocapture\ncargo test load_dotenv -- --nocapture\npython3 scripts/generate_mcp_schema_doc.py --check\ndocker compose --env-file .env.example -f docker-compose.prod.yaml config --quiet\n```\n\nModule layout policy:\n\n- Do not add `mod.rs`.\n- Rust module roots live in `foo.rs`.\n- Submodules live in `foo/bar.rs`.\n\n## Release Gates\n\nRequired before production release:\n\n- CI fmt/check/clippy/test.\n- MCP schema doc sync.\n- CLI help contract tests.\n- Docker Compose config validation.\n- Docker image build and GHCR publish workflow.\n- Compose smoke workflow.\n- Self-hosted RTX 4070 smoke for Qwen3 TEI cold/warm setup timing.\n\n`cargo machete` should be run when installed; it is not vendored in this repo.\n\n## Troubleshooting\n\nFast checks:\n\n```bash\naxon preflight\naxon doctor\ndocker compose --env-file ~/.axon/.env -f ~/.axon/compose/docker-compose.yaml ps\ndocker compose --env-file ~/.axon/.env -f ~/.axon/compose/docker-compose.yaml logs --tail=100 axon axon-tei axon-qdrant axon-chrome\n```\n\nImportant paths:\n\n- `~/.axon/.env`\n- `~/.axon/config.toml`\n- `~/.axon/jobs.db`\n- `~/.axon/logs`\n- `~/.axon/artifacts`\n- `~/.axon/output`\n- `~/.axon/tei`\n- `~/.axon/qdrant` on the host running Qdrant, normally tootie\n\nCommon failures:\n\n- Docker missing: install Docker and Docker Compose, then rerun `axon setup`.\n- GPU unavailable: verify `nvidia-smi` and NVIDIA Container Toolkit.\n- Gemini unauthenticated: run Gemini CLI login outside Axon, then rerun setup,\n  or configure `AXON_LLM_BACKEND=openai-compat` for an OpenAI-compatible\n  endpoint.\n- TEI slow on first boot: model download/cache warmup is the cold path.\n- Auth failures: make sure Claude/plugin config uses the same token as `AXON_MCP_HTTP_TOKEN` in `~/.axon/.env`.\n\n## Related Files\n\n- `install.sh` — verified one-line installer bootstrapper.\n- `docker-compose.prod.yaml` — production Compose stack.\n- `docker-compose.yaml` — local development stack.\n- `.env.example` — production environment template.\n- `config.example.toml` — non-secret tuning template.\n- `plugins/axon/.claude-plugin/plugin.json` — Claude plugin manifest.\n- `scripts/plugin-setup.sh` — plugin hook delegating to shared setup.\n- `docs/reference/mcp/tool-schema.md` — generated MCP wire contract.\n- `docs/` — full documentation tree: guides, `reference/actions/`, architecture, and operations.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmagar%2Faxon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjmagar%2Faxon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmagar%2Faxon/lists"}