{"id":50870114,"url":"https://github.com/uts58/resilio","last_synced_at":"2026-06-15T04:31:10.266Z","repository":{"id":360138134,"uuid":"1076445974","full_name":"uts58/resilio","owner":"uts58","description":" AI cybersecurity advisor for SMBs - LangGraph ReAct agent with RAG over NIST CSF / SP 800-53 / CIS Controls v8, MCP HTTP server, Langfuse tracing, RAGAS eval.","archived":false,"fork":false,"pushed_at":"2026-05-25T05:59:23.000Z","size":841,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-25T07:26:48.692Z","etag":null,"topics":["ai-agent","ai-agents","chromadb","cis-controls","cybersecurity","groq","langchain","langfuse","langgraph","llm","llmops","mcp","mlops","model-context-protocol","nist-800-53","nist-csf","python","rag","ragas","retrieval-augmented-generation"],"latest_commit_sha":null,"homepage":"https://github.com/uts58","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uts58.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-14T21:48:41.000Z","updated_at":"2026-05-25T05:59:26.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/uts58/resilio","commit_stats":null,"previous_names":["uts58/resilio"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/uts58/resilio","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uts58%2Fresilio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uts58%2Fresilio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uts58%2Fresilio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uts58%2Fresilio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uts58","download_url":"https://codeload.github.com/uts58/resilio/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uts58%2Fresilio/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34348291,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-15T02:00:07.085Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agent","ai-agents","chromadb","cis-controls","cybersecurity","groq","langchain","langfuse","langgraph","llm","llmops","mcp","mlops","model-context-protocol","nist-800-53","nist-csf","python","rag","ragas","retrieval-augmented-generation"],"created_at":"2026-06-15T04:31:09.495Z","updated_at":"2026-06-15T04:31:10.258Z","avatar_url":"https://github.com/uts58.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cyber Resilience AI Agent\n\n![Resilio](https://socialify.git.ci/uts58/resilio/image?custom_language=Python\u0026font=Inter\u0026forks=1\u0026language=1\u0026name=1\u0026pattern=Solid\u0026stargazers=1\u0026theme=Light)\n\n[![CI](https://github.com/uts58/resilio/actions/workflows/ci.yml/badge.svg)](https://github.com/uts58/resilio/actions/workflows/ci.yml)\n\nAn AI-powered cybersecurity advisor for small and mid-sized businesses, providing intelligent guidance on security controls, risk assessment, and budget planning.\n\n---\n\n## Overview\n\nThe agent answers cybersecurity questions by combining:\n\n- **Security Guidance** — NIST and CIS control recommendations via semantic search\n- **Risk Calculations** — SLE, ARO, ALE, ROSI, and more\n- **Budget Planning** — IT budget estimation and safeguard value analysis\n\nIt exposes all tools as a **standalone MCP HTTP server**, making them available to any MCP-compatible client (Claude Desktop, CLI agents, etc.) in addition to the built-in Streamlit UI and CLI.\n\n---\n\n## Tech Stack\n\n| Component | Technology |\n|-----------|-----------|\n| **Language** | Python 3.11+ |\n| **UI** | Streamlit |\n| **LLM Orchestration** | LangChain + LangGraph (`create_react_agent`) |\n| **LLM Provider** | Groq (`llama-3.3-70b-versatile`) |\n| **Tool Protocol** | MCP (streamable-HTTP server, port 8001) |\n| **Embeddings** | HuggingFace TEI (`all-MiniLM-L6-v2`, OpenAI-compatible API, port 8002) |\n| **Vector Store** | ChromaDB |\n| **Observability** | Langfuse (self-hosted) — traces, spans, token counts |\n| **Trace Storage** | ClickHouse (columnar) + PostgreSQL (metadata) + Redis (queue) |\n| **Blob Storage** | MinIO (S3-compatible, backs Langfuse event uploads) |\n| **Package Manager** | uv |\n\n---\n\n## Project Structure\n\n```\nresilio/\n├── agent/\n│   └── agent.py              # MCPAgent — LangGraph ReAct agent over MCP HTTP\n├── mcp_server/\n│   └── server.py             # MCP server — all tools, embeddings, and retrieval in one place\n├── helper/\n│   └── helper.py             # Output rendering and text sanitization\n├── data/\n│   ├── knowledge_base.jsonl  # Security knowledge base (JSONL format)\n│   └── eval_dataset.jsonl    # 25 Q\u0026A pairs with ground truth for eval\n├── eval/\n│   └── run_ragas.py          # RAGAS scoring harness (faithfulness, recall, relevancy)\n├── main.py                   # Streamlit application entrypoint\n├── cli.py                    # CLI entrypoint\n├── mcp.json                  # MCP client config (Claude Desktop, etc.)\n├── docker-compose.yml        # Full stack — ChromaDB, TEI, Langfuse, MinIO, MCP server, app\n├── Dockerfile                # App container\n└── pyproject.toml            # Dependencies (managed by uv)\n```\n\n---\n\n## Prerequisites\n\n- **Python** 3.11+\n- **uv** — [install](https://docs.astral.sh/uv/getting-started/installation/)\n- **Docker** — for running the full stack\n- **Groq API key** — [get one](https://console.groq.com/)\n\n### Environment Variables\n\nCopy `.env.example` to `.env` and fill in your values:\n\n```bash\ncp .env.example .env\n```\n\n```env\n# Required\nGROQ_API_KEY=your_groq_api_key_here\n\n# Optional — defaults work for Docker Compose and local dev\nCHROMA_HOST=localhost\nCHROMA_PORT=8000\nMCP_SERVER_URL=http://localhost:8001/mcp\nTEI_URL=http://localhost:8002\nTEI_MODEL=sentence-transformers/all-MiniLM-L6-v2\n\n# Langfuse — pre-seeded on first startup, UI at http://localhost:3000\n# LANGFUSE_HOST is read by the Python SDK; Docker Compose injects\n# http://langfuse-server:3000 internally for the app container automatically\nLANGFUSE_HOST=http://localhost:3000\nLANGFUSE_PUBLIC_KEY=pk-lf-resilio-local\nLANGFUSE_SECRET_KEY=sk-lf-resilio-local\nLANGFUSE_USER_EMAIL=admin@resilio.local\nLANGFUSE_USER_PASSWORD=changeme123\n\n# MinIO — S3-compatible blob storage for Langfuse event uploads\n# Console at http://localhost:9091\nLANGFUSE_S3_ACCESS_KEY=minio\nLANGFUSE_S3_SECRET_KEY=miniosecret\n```\n\n---\n\n## Setup\n\n### Docker Compose\n\nThe stack is split into two compose files — core app and observability — so you can run them independently.\n\n**Core only** (ChromaDB, TEI, MCP server, Streamlit):\n```bash\ndocker compose up -d\n```\n\n**Full stack with Langfuse observability:**\n```bash\ndocker compose -f docker-compose.yml -f docker-compose.langfuse.yml up -d\n```\n\nFirst run takes a few minutes while TEI downloads the embedding model.\n\n| Service | URL | Credentials | Compose file |\n|---------|-----|-------------|--------------|\n| Streamlit UI | http://localhost:8501 | — | core |\n| MCP server | http://localhost:8001/mcp | — | core |\n| Langfuse UI | http://localhost:3000 | `admin@resilio.local` / `changeme123` | langfuse |\n| MinIO Console | http://localhost:9091 | `minio` / `miniosecret` | langfuse |\n\nThe Langfuse project is pre-seeded with API keys matching the defaults in `.env.example`. If you override `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY`, update the values in `.env` to match.\n\nTo tear down only the observability stack (preserving core app data):\n```bash\ndocker compose -f docker-compose.langfuse.yml down\n```\n\n### Local (without Docker)\n\nLangfuse tracing is optional when running locally — the agent skips it if `LANGFUSE_PUBLIC_KEY` or `LANGFUSE_SECRET_KEY` are absent from `.env`. To trace locally, run the full Docker Compose stack and point `LANGFUSE_HOST` at `http://localhost:3000`.\n\n**1. Start ChromaDB and TEI:**\n```bash\ndocker run -p 8000:8000 chromadb/chroma:1.4.4\ndocker run -p 8002:80 ghcr.io/huggingface/text-embeddings-inference:cpu-1.9.3 \\\n  --model-id sentence-transformers/all-MiniLM-L6-v2 --port 80\n```\n\n**2. Install dependencies:**\n```bash\nuv sync\n```\n\n**3. Start the MCP server:**\n```bash\nuv run python -m mcp_server\n```\n\n**4. Run the app** (in a separate terminal):\n```bash\nuv run streamlit run main.py\n```\n\nOr the CLI:\n```bash\nuv run python cli.py\n```\n\n---\n\n## MCP Server\n\nThe tools run as a standalone HTTP server (streamable-HTTP transport, port 8001). Any MCP-compatible client can connect to it directly.\n\n**Run locally:**\n```bash\nuv run python -m mcp_server\n# → listening on http://localhost:8001/mcp\n```\n\n**Connect Claude Desktop** — add to your `claude_desktop_config.json`:\n```json\n{\n  \"mcpServers\": {\n    \"resilio-tools\": {\n      \"url\": \"http://localhost:8001/mcp\",\n      \"transport\": \"streamable-http\"\n    }\n  }\n}\n```\n\n\u003e ChromaDB must be running before the MCP server starts.\n\n---\n\n## Available Tools\n\n| Tool | Description |\n|------|-------------|\n| `retrieve_cyber_context` | Semantic search over NIST/CIS knowledge base |\n| `calc_it_budget` | Estimate IT budget (~1.47% of revenue) |\n| `calc_sle` | Single Loss Expectancy (Asset Value × Exposure Factor) |\n| `calc_aro` | Annual Rate of Occurrence |\n| `calc_ale` | Annualized Loss Expectancy (SLE × ARO) |\n| `calc_rosi` | Return on Security Investment |\n| `calc_risk` | Basic risk score (Threat × Vulnerability × Impact) |\n| `calc_risk_reduction` | Risk reduction percentage after controls |\n| `calc_safeguard_value` | Value of a security control (ALE before − after) |\n| `calc_payback_period` | Investment payback in years |\n| `calc_it_risk_score` | Normalized IT risk score (0–100) |\n\n---\n\n## Observability\n\nEvery agent run is traced end-to-end in Langfuse — LLM calls, tool invocations, token counts, and latency. Open the Langfuse UI at `http://localhost:3000` and navigate to **Traces** to inspect runs.\n\nTracing is gated on `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` being set in `.env`. If either is missing, the agent runs without tracing — no errors.\n\n---\n\n## Evaluation\n\nA 25-question Q\u0026A set lives at `data/eval_dataset.jsonl` — 15 knowledge, 3 application, and 7 calculator questions across NIST CSF 2.0, NIST SP 800-53, and CIS Controls v8.\n\nThe RAGAS harness at `eval/run_ragas.py` scores the knowledge/application questions on three metrics:\n\n| Metric | What it measures |\n|--------|------------------|\n| `faithfulness` | Is the answer grounded in the retrieved contexts? |\n| `context_recall` | Did retrieval surface the facts present in the ground-truth answer? |\n| `answer_relevancy` | Does the answer actually address the question? |\n\nCalculator questions are skipped — they're tool-use, not RAG.\n\n**Run it:**\n```bash\n# ChromaDB + TEI must be running (core docker compose is enough)\ndocker compose up -d\nuv sync --group eval\nuv run python -m eval.run_ragas                  # uses Settings.prompt_version\nuv run python -m eval.run_ragas --prompt-version v1\n```\n\nPer-question scores print to stdout and persist to `data/eval_results_\u003cversion\u003e.json`.\n\n\u003e Uses Groq `llama-3.3-70b-versatile` as both answerer and LLM judge, and the project's TEI server for embeddings. `GROQ_API_KEY` must be set.\n\n### Latest results\n\n18 RAG questions, `llama-3.3-70b-versatile`, top-k = 5.\n\n| Metric | Score |\n|--------|------:|\n| `faithfulness` | **0.974** |\n| `context_recall` | **0.778** |\n| `answer_relevancy` | 0.569 \\* |\n\n\\* `answer_relevancy` is undercounted: RAGAS requests `n=3` LLM samples per question to score variance, but Groq's chat API only supports `n=1`. ~5 of 18 rows came back as `NaN` and were dropped from the mean. Real number is likely meaningfully higher — a future change should either swap the judge to an `n\u003e1`-capable model or move to the newer `ragas.metrics.collections` API.\n\n**Known weak spots**\n\n- CIS Control 1, 2, and 3 lookups (`cis_001`–`cis_003`) returned `context_recall = 0` — the canonical \"Control N: \u003cname\u003e\" strings live inside table-of-contents chunks of the source PDF and aren't surfacing as the top hit. Knowledge-base quality issue, not a metric artifact.\n\n### Prompt versioning\n\nBoth the agent's system prompt and the eval-time RAG prompt live in `prompts.py` as a versioned registry. The agent picks its active version from `Settings.prompt_version` (env: `PROMPT_VERSION`); the eval harness takes `--prompt-version v1`. Every Langfuse trace is tagged with `prompt_version` as metadata, so you can filter runs by version in the Langfuse UI to compare quality across prompt revisions.\n\nTo A/B a new prompt: add a `v2` entry to `AGENT_SYSTEM` and/or `EVAL_RAG` in `prompts.py`, then:\n\n```bash\nuv run python -m eval.run_ragas --prompt-version v1   # baseline\nuv run python -m eval.run_ragas --prompt-version v2   # candidate\ndiff data/eval_results_v1.json data/eval_results_v2.json\n```\n\nEach run writes to `data/eval_results_\u003cversion\u003e.json` so you keep both score sets side by side.\n\n---\n\n## Troubleshooting\n\n| Issue | Solution |\n|-------|----------|\n| `Could not connect to ChromaDB` | Make sure ChromaDB is running on port 8000 |\n| `GROQ_API_KEY not set` | Check your `.env` file or export the variable in your shell |\n| Slow first run | TEI downloads the embedding model on first start — subsequent starts use the cached volume |\n| MCP server fails to start | ChromaDB and TEI must both be reachable before the MCP server starts |\n| TEI stuck in healthcheck | First start downloads the model (~90 MB) — wait up to 2 minutes |\n| Agent can't reach MCP server | Check `MCP_SERVER_URL` in `.env` — default is `http://localhost:8001/mcp` |\n| Langfuse login fails | Wipe both Postgres and ClickHouse volumes and restart: `docker compose down \u0026\u0026 docker volume rm resilio_langfuse_db resilio_langfuse_clickhouse \u0026\u0026 docker compose up -d` |\n| No traces in Langfuse | Verify `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` in `.env` match **Settings → API Keys** in the Langfuse UI |\n\n---\n\n## Security Notes\n\n- Never commit API keys — keep `.env` in `.gitignore`\n- Rotate `GROQ_API_KEY` regularly\n- The MCP server listens on port 8001 — restrict access if deploying beyond localhost\n- Default Langfuse secrets (`NEXTAUTH_SECRET`, `SALT`, `ENCRYPTION_KEY`) in `.env.example` are placeholders — generate real values before exposing Langfuse beyond localhost","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Futs58%2Fresilio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Futs58%2Fresilio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Futs58%2Fresilio/lists"}