{"id":35809738,"url":"https://github.com/ca-srg/ragent","last_synced_at":"2026-04-01T18:30:26.896Z","repository":{"id":312074343,"uuid":"1040202747","full_name":"ca-srg/ragent","owner":"ca-srg","description":"CLI tool for building production RAG systems from Markdown, CSV, and PDF documents using hybrid search (BM25 + vector) with OpenSearch. Features MCP server, Slack bot, Web UI, multi-source ingestion (local/S3/GitHub), and multi-provider embeddings (Bedrock/Gemini).","archived":false,"fork":false,"pushed_at":"2026-03-25T09:15:04.000Z","size":17087,"stargazers_count":8,"open_issues_count":1,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-25T21:57:48.677Z","etag":null,"topics":["aws","bedrock","cli","csv","gemini","golang","hybrid-search","markdown","mcp-server","oidc","opensearch","opentelemetry","pdf","rag","s3-vectors","slack-bot","sqlite-vec","vector","webui"],"latest_commit_sha":null,"homepage":"https://ca-srg.dev","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ca-srg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-18T15:57:33.000Z","updated_at":"2026-03-25T09:14:54.000Z","dependencies_parsed_at":"2026-02-23T16:04:41.118Z","dependency_job_id":null,"html_url":"https://github.com/ca-srg/ragent","commit_stats":null,"previous_names":["ca-srg/kiberag","ca-srg/mdrag"],"tags_count":57,"template":false,"template_full_name":null,"purl":"pkg:github/ca-srg/ragent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ca-srg%2Fragent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ca-srg%2Fragent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ca-srg%2Fragent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ca-srg%2Fragent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ca-srg","download_url":"https://codeload.github.com/ca-srg/ragent/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ca-srg%2Fragent/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31290858,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T13:12:26.723Z","status":"ssl_error","status_checked_at":"2026-04-01T13:12:25.102Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","bedrock","cli","csv","gemini","golang","hybrid-search","markdown","mcp-server","oidc","opensearch","opentelemetry","pdf","rag","s3-vectors","slack-bot","sqlite-vec","vector","webui"],"created_at":"2026-01-07T14:15:57.583Z","updated_at":"2026-04-01T18:30:26.883Z","avatar_url":"https://github.com/ca-srg.png","language":"Go","readme":"- [Features](#features)\n# RAGent - RAG System Builder for Markdown Documents\n\n**[日本語版 (Japanese) / 日本語 README](README_ja.md)**\n\nRAGent is a CLI tool for building a RAG (Retrieval-Augmented Generation) system from markdown documents using hybrid search capabilities (BM25 + vector search) with Amazon S3 Vectors and OpenSearch.\n\n## Table of Contents\n\n- [Features](#features)\n- [Slack Search Integration](#slack-search-integration)\n- [Embedding-Agnostic RAG](#embedding-agnostic-rag)\n- [Environment Variable Setup Guide](#environment-variable-setup-guide)\n- [Architecture Overview](#architecture-overview)\n- [Prerequisites](#prerequisites)\n- [Required Environment Variables](#required-environment-variables)\n - [Installation](#installation)\n - [Releases](#releases)\n- [Commands](#commands)\n  - [vectorize - Vectorization and S3 Storage](#1-vectorize---vectorization-and-s3-storage)\n  - [query - Semantic Search](#2-query---semantic-search)\n  - [list - List Vectors](#3-list---list-vectors)\n  - [chat - Interactive RAG Chat](#4-chat---interactive-rag-chat)\n  - [slack-bot - Slack Bot for RAG Search](#5-slack-bot---slack-bot-for-rag-search)\n  - [mcp-server - MCP Server for Claude Desktop Integration](#6-mcp-server---mcp-server-for-claude-desktop-integration-new)\n  - [webui - Web UI for Vectorization Monitoring](#7-webui---web-ui-for-vectorization-monitoring-new)\n- [Development](#development)\n- [Typical Workflow](#typical-workflow)\n- [Troubleshooting](#troubleshooting)\n- [AWS Secrets Manager Integration](#aws-secrets-manager-integration)\n- [OpenSearch RAG Configuration](#opensearch-rag-configuration)\n- [Automated Setup (setup.sh)](#automated-setup-setupsh)\n- [License](#license)\n- [MCP Server Integration](#mcp-server-integration)\n- [Contributing](#contributing)\n\n## Features\n\n- **Vectorization**: Convert source files (markdown, CSV, and PDF) from local directories, S3, or GitHub repositories to embeddings using Amazon Bedrock\n- **GitHub Data Source**: Clone GitHub repositories and vectorize their markdown/CSV files with auto-generated metadata\n- **S3 Vector Integration**: Store generated vectors in Amazon S3 Vectors\n- **Hybrid Search**: Combined BM25 + vector search using OpenSearch\n- **Slack Search Integration**: Blend document results with Slack conversations via an iterative enrichment pipeline\n- **Semantic Search**: Semantic similarity search using S3 Vector Index\n- **Interactive RAG Chat**: Chat interface with context-aware responses\n- **Vector Management**: List vectors stored in S3\n- **Embedding-Agnostic RAG**: Optional search path that answers directly from Slack without requiring pre-built vectors\n- **MCP Server**: Model Context Protocol server for Claude Desktop integration\n- **OIDC Authentication**: OpenID Connect authentication with multiple providers\n- **IP-based Security**: IP address-based access control with configurable bypass ranges and audit logging\n- **Dual Transport**: HTTP and Server-Sent Events (SSE) transport support\n- **AWS Secrets Manager Integration**: Automatic secret injection as environment variable fallback\n\n## Slack Search Integration\n\nSlack search extends every retrieval workflow by streaming recent conversations, threads, and channel timelines directly from Slack’s Web API. When `SLACK_SEARCH_ENABLED=true`, the following behavior is enabled automatically:\n\n- `query` exposes `--enable-slack-search` flag to opt in per request.\n- `chat` surfaces Slack context in addition to documents without requiring extra flags.\n- `slack-bot` responds with combined document + Slack answers inside Block Kit messages.\n- `mcp-server` adds `enable_slack_search` parameter to the `hybrid_search` tool.\n\nEach Slack lookup performs iterative query refinement, merges timeline context from threads, and runs a sufficiency check before the final answer is generated. Results include permalinks so operators can pivot back to the original conversation instantly.\n\n### Slack URL Reference\n\nYou can include Slack message URLs directly in your queries. RAGent automatically detects and fetches the referenced messages, including thread replies, making them available as context for your question. This works across all commands (`query`, `chat`, `slack-bot`, `mcp-server`) regardless of whether `SLACK_SEARCH_ENABLED` is set.\n\n```bash\n# Fetch and analyze a specific Slack conversation\nRAGent query -q \"Summarize this discussion: https://your-workspace.slack.com/archives/C12345678/p1234567890123456\"\n\n# Reference multiple Slack messages\nRAGent chat\n\u003e Compare these two threads: https://slack.com/.../p111 and https://slack.com/.../p222\n```\n\nQuick example:\n\n```bash\n# Search across documents and Slack in one command\nRAGent query -q \"incident timeline\" --enable-slack-search\n```\n\n## Embedding-Agnostic RAG\n\nSlack search is intentionally embedding-agnostic: the system fetches live messages at query time instead of relying on pre-generated vectors. This provides several benefits:\n\n- **Operational cost savings**: no additional vector storage or nightly re-indexing jobs are required for Slack data.\n- **Real-time awareness**: answers incorporate messages posted seconds ago, keeping incident timelines and release notes fresh.\n- **Time-series preservation**: the pipeline keeps message order and thread structure so analysts can replay discussions accurately.\n- **Seamless fallbacks**: when Slack returns no hits the document-only hybrid search continues without interruption.\n\nSlack-only output is still fused with document references, giving operators a single consolidated view while staying within compliance rules.\n\n## Environment Variable Setup Guide\n\nRAGent integrates many services, and the required environment variables differ depending on your choices. This section provides visual flowcharts for the two main workflows: **vectorization (data ingestion)** and **output (search \u0026 usage)**.\n\n### Vectorization Flow (`vectorize` command)\n\nFollow this flowchart to determine which environment variables are needed for vectorization.\n\n```mermaid\nflowchart TD\n    Start([Run vectorize]) --\u003e AuthMethod\n\n    %% ── Authentication method ──\n    AuthMethod{{\"AWS authentication method?\"}}\n\n    AuthMethod --\u003e|\"IAM Credentials\"| IAMAuth[\"IAM Credentials\n    ━━━━━━━━━━━━━━━━━━━\n    AWS_REGION\n    AWS_ACCESS_KEY_ID\n    AWS_SECRET_ACCESS_KEY\n    BEDROCK_REGION (default: us-east-1)\"]\n\n    AuthMethod --\u003e|\"Bearer Token\"| BearerAuth[\"Bearer Token Auth\n    ━━━━━━━━━━━━━━━━━━━\n    BEDROCK_REGION\n    * AWS_BEARER_TOKEN_BEDROCK\n    ━━━━━━━━━━━━━━━━━━━\n    AWS_REGION etc. not required\"]\n\n    IAMAuth --\u003e OpenSearch\n    BearerAuth --\u003e OpenSearch\n\n    %% ── OpenSearch (always required) ──\n    OpenSearch[\"OpenSearch Config - BM25 Index\n    ━━━━━━━━━━━━━━━━━━━\n    * OPENSEARCH_ENDPOINT\n    * OPENSEARCH_INDEX\n    OPENSEARCH_REGION (default: us-east-1)\"]\n\n    OpenSearch --\u003e VectorDB\n\n    %% ── Vector DB backend selection ──\n    VectorDB{{\"Vector storage backend?\n    VECTOR_DB_BACKEND\"}}\n\n    VectorDB --\u003e|\"s3 (default)\"| S3Config[\"S3 Vectors Config\n    ━━━━━━━━━━━━━━━━━━━\n    * AWS_S3_VECTOR_BUCKET\n    * AWS_S3_VECTOR_INDEX\n    S3_VECTOR_REGION (default: us-east-1)\"]\n\n    VectorDB --\u003e|sqlite| SQLiteConfig[\"SQLite-vec Config\n    ━━━━━━━━━━━━━━━━━━━\n    SQLITE_VEC_DB_PATH\n    (default: ~/.ragent/vectors.db)\"]\n\n    S3Config --\u003e DataSource\n    SQLiteConfig --\u003e DataSource\n\n    %% ── Data source selection ──\n    DataSource{{\"Data source?\"}}\n\n    DataSource --\u003e|\"Local (default)\"| LocalSource[\"Local Directory\n    ━━━━━━━━━━━━━━━━━━━\n    --directory ./source\n    No additional env vars\"]\n\n    DataSource --\u003e|S3 Bucket| S3Source[\"S3 Source Config\n    ━━━━━━━━━━━━━━━━━━━\n    --enable-s3 flag\n    --s3-bucket bucket-name\n    S3_SOURCE_REGION (default: us-east-1)\"]\n\n    DataSource --\u003e|GitHub Repository| GitHubSource[\"GitHub Config\n    ━━━━━━━━━━━━━━━━━━━\n    --github-repos owner/repo\n    GITHUB_TOKEN\n    (required for private repos)\"]\n\n    DataSource --\u003e|Multiple| MultiSource[\"Multiple Sources\n    ━━━━━━━━━━━━━━━━━━━\n    Local + S3 + GitHub can be combined\"]\n\n    LocalSource --\u003e OCR\n    S3Source --\u003e OCR\n    GitHubSource --\u003e OCR\n    MultiSource --\u003e OCR\n\n    %% ── OCR (PDF support) ──\n    OCR{{\"Process PDFs?\n    OCR_PROVIDER\"}}\n\n    OCR --\u003e|\"No (default)\"| NoOCR[\"PDFs are skipped\"]\n    OCR --\u003e|bedrock| OCRBedrock[\"Bedrock OCR Config\n    ━━━━━━━━━━━━━━━━━━━\n    OCR_PROVIDER=bedrock\n    OCR_MODEL (optional)\n    OCR_TIMEOUT (default: 600s)\n    OCR_CONCURRENCY (default: 5)\"]\n    OCR --\u003e|gemini| OCRGemini[\"Gemini OCR Config\n    ━━━━━━━━━━━━━━━━━━━\n    OCR_PROVIDER=gemini\n    GEMINI_API_KEY (API key auth)\n    ━━━ OR ━━━\n    GOOGLE_APPLICATION_CREDENTIALS\n    GEMINI_GCP_PROJECT\n    GEMINI_GCP_LOCATION (default: us-central1)\n    ━━━━━━━━━━━━━━━━━━━\n    OCR_MODEL (optional)\"]\n\n    NoOCR --\u003e Optional\n    OCRBedrock --\u003e Optional\n    OCRGemini --\u003e Optional\n\n    %% ── Optional settings ──\n    Optional[\"Optional Settings\n    ━━━━━━━━━━━━━━━━━━━\n    VECTORIZER_CONCURRENCY (default: 10)\n    VECTORIZER_RETRY_ATTEMPTS (default: 10)\n    EXCLUDE_CATEGORIES\"]\n\n    Optional --\u003e Ready([Ready to run vectorize])\n\n    %% ── Styles ──\n    style Start fill:#4CAF50,color:#fff\n    style Ready fill:#4CAF50,color:#fff\n    style AuthMethod fill:#FF9800,color:#fff\n    style VectorDB fill:#FF9800,color:#fff\n    style DataSource fill:#FF9800,color:#fff\n    style OCR fill:#FF9800,color:#fff\n    style IAMAuth fill:#e3f2fd\n    style BearerAuth fill:#e3f2fd\n    style OpenSearch fill:#e3f2fd\n    style S3Config fill:#fff3e0\n    style SQLiteConfig fill:#fff3e0\n    style LocalSource fill:#e8f5e9\n    style S3Source fill:#e8f5e9\n    style GitHubSource fill:#e8f5e9\n    style MultiSource fill:#e8f5e9\n    style OCRBedrock fill:#fce4ec\n    style OCRGemini fill:#fce4ec\n    style NoOCR fill:#f5f5f5\n    style Optional fill:#f3e5f5\n```\n\n\u003e **Legend**: `*` = required (error if unset), `default: xxx` = has a default value. IAM credentials and Bearer Token (`AWS_BEARER_TOKEN_BEDROCK`) are mutually exclusive.\n\n### Output Flow (Search \u0026 Usage Commands)\n\nThe required environment variables depend on **which command** you use and **which search mode** you choose.\n\n```mermaid\nflowchart TD\n    Start([Use vectorized data]) --\u003e Command\n\n    %% ── Command selection ──\n    Command{{\"Which command?\"}}\n\n    Command --\u003e|query| QueryCmd[\"query\n    ━━━━━━━━━━━━━━━━\n    Semantic search\n    Terminal output\"]\n\n    Command --\u003e|chat| ChatCmd[\"chat\n    ━━━━━━━━━━━━━━━━\n    Interactive RAG chat\n    Claude-powered answers\"]\n\n    Command --\u003e|slack-bot| SlackBotCmd[\"slack-bot\n    ━━━━━━━━━━━━━━━━\n    Slack Bot\n    Auto-respond to mentions\"]\n\n    Command --\u003e|mcp-server| MCPCmd[\"mcp-server\n    ━━━━━━━━━━━━━━━━\n    MCP Protocol Server\n    Claude Desktop integration\"]\n\n    QueryCmd --\u003e SearchMode\n    ChatCmd --\u003e SearchMode\n    SlackBotCmd --\u003e SlackBotConfig\n    MCPCmd --\u003e MCPAuth\n\n    %% ── Search mode (query / chat) ──\n    SearchMode{{\"Search mode?\"}}\n\n    SearchMode --\u003e|\"Hybrid search (default)\"| HybridBase\n    SearchMode --\u003e|\"--only-slack\"| SlackOnly\n\n    %% ── Hybrid search common config ──\n    HybridBase[\"Common Config (required)\n    ━━━━━━━━━━━━━━━━━━━\n    AWS Auth (IAM or Bearer Token)\n    BEDROCK_REGION\n    * OPENSEARCH_ENDPOINT\n    * OPENSEARCH_INDEX\n    CHAT_MODEL (for chat command)\"]\n\n    HybridBase --\u003e SlackSearch\n\n    %% ── Slack search addition ──\n    SlackSearch{{\"Also use Slack search?\n    SLACK_SEARCH_ENABLED\"}}\n\n    SlackSearch --\u003e|\"false (default)\"| HybridOnly[\"Hybrid search only\n    ━━━━━━━━━━━━━━━━━━━\n    No additional env vars\"]\n\n    SlackSearch --\u003e|true| HybridPlusSlack[\"Add Slack Search Config\n    ━━━━━━━━━━━━━━━━━━━\n    SLACK_SEARCH_ENABLED=true\n    * SLACK_USER_TOKEN\n    ━━━━━ Optional ━━━━━\n    SLACK_SEARCH_MAX_RESULTS (default: 20)\n    SLACK_SEARCH_MAX_ITERATIONS (default: 3)\n    SLACK_SEARCH_TIMEOUT_SECONDS (default: 60)\"]\n\n    %% ── Slack-Only mode ──\n    SlackOnly[\"Slack-Only Config\n    ━━━━━━━━━━━━━━━━━━━\n    OpenSearch not required\n    ━━━━━━━━━━━━━━━━━━━\n    AWS Auth (IAM or Bearer Token)\n    BEDROCK_REGION\n    * SLACK_USER_TOKEN\n    SLACK_SEARCH_ENABLED=true\"]\n\n    %% ── Slack Bot specific config ──\n    SlackBotConfig[\"Slack Bot Config\n    ━━━━━━━━━━━━━━━━━━━\n    * SLACK_BOT_TOKEN\n    SLACK_APP_TOKEN (for Socket Mode)\n    SLACK_RESPONSE_TIMEOUT (default: 5s)\n    SLACK_ENABLE_THREADING (default: true)\"]\n\n    SlackBotConfig --\u003e SlackBotSearch\n\n    SlackBotSearch{{\"Search mode?\"}}\n\n    SlackBotSearch --\u003e|\"Hybrid (default)\"| SlackBotHybrid[\"Common Config (required)\n    ━━━━━━━━━━━━━━━━━━━\n    AWS Auth (IAM or Bearer Token)\n    BEDROCK_REGION\n    * OPENSEARCH_ENDPOINT\n    * OPENSEARCH_INDEX\"]\n\n    SlackBotSearch --\u003e|\"--only-slack\"| SlackBotOnly[\"Slack-Only Config\n    ━━━━━━━━━━━━━━━━━━━\n    OpenSearch not required\n    ━━━━━━━━━━━━━━━━━━━\n    AWS Auth (IAM or Bearer Token)\n    BEDROCK_REGION\n    * SLACK_USER_TOKEN\n    SLACK_SEARCH_ENABLED=true\"]\n\n    SlackBotHybrid --\u003e SlackBotSlackOpt\n    SlackBotSlackOpt{{\"Also use Slack search?\"}}\n    SlackBotSlackOpt --\u003e|false| SlackBotReady([Ready to run slack-bot])\n    SlackBotSlackOpt --\u003e|true| SlackBotSlack[\"SLACK_SEARCH_ENABLED=true\n    * SLACK_USER_TOKEN\"]\n    SlackBotSlack --\u003e SlackBotReady\n    SlackBotOnly --\u003e SlackBotReady\n\n    %% ── MCP Server auth config ──\n    MCPAuth{{\"Auth method?\n    --auth-method\"}}\n\n    MCPAuth --\u003e|\"ip (default)\"| MCPIP[\"IP Auth\n    ━━━━━━━━━━━━━━━━━━━\n    MCP_ALLOWED_IPS (default: 127.0.0.1,::1)\"]\n\n    MCPAuth --\u003e|oidc| MCPOIDC[\"OIDC Auth\n    ━━━━━━━━━━━━━━━━━━━\n    * OIDC_ISSUER\n    * OIDC_CLIENT_ID\n    * OIDC_CLIENT_SECRET\"]\n\n    MCPAuth --\u003e|both| MCPBoth[\"IP + OIDC both required\n    ━━━━━━━━━━━━━━━━━━━\n    IP auth config +\n    OIDC auth config\"]\n\n    MCPAuth --\u003e|either| MCPEither[\"IP or OIDC either one\n    ━━━━━━━━━━━━━━━━━━━\n    IP auth config or\n    OIDC auth config\"]\n\n    MCPIP --\u003e MCPServer\n    MCPOIDC --\u003e MCPServer\n    MCPBoth --\u003e MCPServer\n    MCPEither --\u003e MCPServer\n\n    MCPServer[\"MCP Server Config\n    ━━━━━━━━━━━━━━━━━━━\n    MCP_SERVER_HOST (default: localhost)\n    MCP_SERVER_PORT (default: 8080)\"]\n\n    MCPServer --\u003e MCPSearch\n\n    MCPSearch{{\"Search mode?\"}}\n\n    MCPSearch --\u003e|\"Hybrid (default)\"| MCPHybrid[\"Common Config (required)\n    ━━━━━━━━━━━━━━━━━━━\n    AWS Auth (IAM or Bearer Token)\n    BEDROCK_REGION\n    * OPENSEARCH_ENDPOINT\n    * OPENSEARCH_INDEX\"]\n\n    MCPSearch --\u003e|\"--only-slack\"| MCPSlackOnly[\"Slack-Only Config\n    ━━━━━━━━━━━━━━━━━━━\n    OpenSearch not required\n    ━━━━━━━━━━━━━━━━━━━\n    AWS Auth (IAM or Bearer Token)\n    BEDROCK_REGION\n    * SLACK_USER_TOKEN\n    SLACK_SEARCH_ENABLED=true\"]\n\n    MCPHybrid --\u003e MCPReady([Ready to run mcp-server])\n    MCPSlackOnly --\u003e MCPReady\n\n    HybridOnly --\u003e Ready([Ready])\n    HybridPlusSlack --\u003e Ready\n    SlackOnly --\u003e Ready\n\n    %% ── Styles ──\n    style Start fill:#4CAF50,color:#fff\n    style Ready fill:#4CAF50,color:#fff\n    style SlackBotReady fill:#4CAF50,color:#fff\n    style MCPReady fill:#4CAF50,color:#fff\n\n    style Command fill:#2196F3,color:#fff\n    style SearchMode fill:#FF9800,color:#fff\n    style SlackSearch fill:#FF9800,color:#fff\n    style SlackBotSearch fill:#FF9800,color:#fff\n    style SlackBotSlackOpt fill:#FF9800,color:#fff\n    style MCPAuth fill:#FF9800,color:#fff\n    style MCPSearch fill:#FF9800,color:#fff\n\n    style QueryCmd fill:#e3f2fd\n    style ChatCmd fill:#fce4ec\n    style SlackBotCmd fill:#e8f5e9\n    style MCPCmd fill:#f3e5f5\n\n    style HybridBase fill:#e3f2fd\n    style SlackBotConfig fill:#e8f5e9\n    style MCPServer fill:#f3e5f5\n\n    style HybridOnly fill:#e8f5e9\n    style HybridPlusSlack fill:#fff3e0\n    style SlackOnly fill:#fff3e0\n\n    style MCPIP fill:#f3e5f5\n    style MCPOIDC fill:#f3e5f5\n    style MCPBoth fill:#f3e5f5\n    style MCPEither fill:#f3e5f5\n```\n\n## Architecture Overview\n\nThe diagram below highlights how document and Slack pipelines converge before the answer is generated.\n\n```mermaid\ngraph LR\n    subgraph Sources[\"Data Sources\"]\n        MD[Markdown Files]\n        CSV[CSV Files]\n        S3[Amazon S3 Bucket]\n        GH[GitHub Repositories]\n    end\n\n    Sources --\u003e|Vectorize| VE[Amazon S3 Vectors]\n    VE --\u003e HY[Hybrid Search Engine]\n    OS[(Amazon OpenSearch)] --\u003e HY\n    SL[Slack Workspace] --\u003e|Conversations API| SS[SlackSearch Service]\n    SS --\u003e HY\n    HY --\u003e CT[Context Builder]\n    CT --\u003e AN[\"Answer Generation (Claude, Bedrock Chat)\"]\n```\n\n### Detailed Architecture Diagrams\n\n#### Command Overview\n\nRAGent provides five main commands, each serving a specific purpose in the RAG workflow:\n\n```mermaid\nflowchart LR\n    User([User])\n\n    User --\u003e|1| Vectorize[vectorize\u003cbr/\u003eMarkdown → Vectors]\n    User --\u003e|2| Query[query\u003cbr/\u003eSemantic Search]\n    User --\u003e|3| Chat[chat\u003cbr/\u003eInteractive RAG]\n    User --\u003e|4| SlackBot[slack-bot\u003cbr/\u003eSlack Integration]\n    User --\u003e|5| MCPServer[mcp-server\u003cbr/\u003eClaude Desktop]\n\n    Vectorize --\u003e|Store| S3V[(S3 Vectors)]\n    Vectorize --\u003e|Index| OS[(OpenSearch)]\n\n    Query --\u003e Hybrid[Hybrid Search\u003cbr/\u003eBM25 + Vector]\n    Chat --\u003e Hybrid\n    SlackBot --\u003e Hybrid\n    MCPServer --\u003e Hybrid\n\n    Hybrid --\u003e S3V\n    Hybrid --\u003e OS\n    Hybrid --\u003e|Optional| Slack[(Slack API)]\n\n    Hybrid --\u003e Results[Search Results]\n    Results --\u003e Answer[Generated Answer\u003cbr/\u003evia Bedrock Claude]\n\n    style Vectorize fill:#e1f5ff\n    style Query fill:#fff4e1\n    style Chat fill:#ffe1f5\n    style SlackBot fill:#e1ffe1\n    style MCPServer fill:#f5e1ff\n```\n\n#### Hybrid Search Flow\n\nThe hybrid search engine combines BM25 keyword matching with vector similarity search:\n\n```mermaid\nsequenceDiagram\n    participant User\n    participant CLI as CLI Command\n    participant Bedrock as Amazon Bedrock\n    participant OS as OpenSearch\n    participant Slack as Slack API\n    participant Engine as Hybrid Engine\n\n    User-\u003e\u003eCLI: query \"検索クエリ\"\n    CLI-\u003e\u003eBedrock: GenerateEmbedding(query)\n    Bedrock--\u003e\u003eCLI: Vector[1024]\n\n    par BM25 Search\n        CLI-\u003e\u003eOS: BM25 Search (kuromoji tokenizer)\n        OS--\u003e\u003eCLI: BM25 Results (200 docs)\n    and Vector Search\n        CLI-\u003e\u003eOS: k-NN Search (cosine similarity)\n        OS--\u003e\u003eCLI: Vector Results (100 docs)\n    end\n\n    CLI-\u003e\u003eEngine: Fuse Results (RRF/weighted_sum)\n    Engine--\u003e\u003eCLI: Fused Results (top-k)\n\n    opt Slack Search Enabled\n        CLI-\u003e\u003eSlack: search.messages(query)\n        Slack--\u003e\u003eCLI: Slack Messages\n        CLI-\u003e\u003eSlack: conversations.history(threads)\n        Slack--\u003e\u003eCLI: Thread Context\n        CLI-\u003e\u003eEngine: Merge Slack Context\n        Engine--\u003e\u003eCLI: Enriched Results\n    end\n\n    CLI--\u003e\u003eUser: Combined Results + References\n```\n\n#### Slack Bot Processing\n\nSlack Bot listens for mentions and responds with RAG-powered answers:\n\n```mermaid\nsequenceDiagram\n    participant Slack as Slack Workspace\n    participant Bot as Slack Bot (RTM/Socket)\n    participant Detector as Mention Detector\n    participant Extractor as Query Extractor\n    participant Hybrid as Hybrid Search\n    participant SlackSearch as Slack Search Service\n    participant Formatter as Block Kit Formatter\n\n    Slack-\u003e\u003eBot: @ragent-bot \"質問内容\"\n    Bot-\u003e\u003eDetector: Detect Mention\n    Detector--\u003e\u003eBot: Mention Event\n\n    Bot-\u003e\u003eExtractor: Extract Query\n    Extractor--\u003e\u003eBot: Query Text\n\n    Bot-\u003e\u003eHybrid: Search(query)\n\n    par Document Search\n        Hybrid-\u003e\u003eHybrid: BM25 + k-NN\n        Hybrid--\u003e\u003eBot: Document Results\n    and Slack Context Search (Optional)\n        Bot-\u003e\u003eSlackSearch: SearchSlack(query, channels)\n        SlackSearch-\u003e\u003eSlackSearch: Query Refinement Loop\n        SlackSearch-\u003e\u003eSlack: search.messages API\n        Slack--\u003e\u003eSlackSearch: Messages\n        SlackSearch-\u003e\u003eSlack: conversations.history (threads)\n        Slack--\u003e\u003eSlackSearch: Thread Timeline\n        SlackSearch-\u003e\u003eSlackSearch: Sufficiency Check\n        SlackSearch--\u003e\u003eBot: Slack Context\n    end\n\n    Bot-\u003e\u003eFormatter: Format Results (Block Kit)\n    Formatter--\u003e\u003eBot: Slack Blocks\n\n    Bot-\u003e\u003eSlack: Post Message (with Blocks)\n    Slack--\u003e\u003eSlack: Display Answer + References\n```\n\n#### MCP Server Integration\n\nMCP Server exposes RAGent's hybrid search to Claude Desktop and other MCP-compatible tools:\n\n```mermaid\nsequenceDiagram\n    participant Claude as Claude Desktop\n    participant MCP as MCP Server\n    participant Auth as Auth Middleware\n    participant Handler as Hybrid Search Handler\n    participant Bedrock as Amazon Bedrock\n    participant OS as OpenSearch\n    participant Slack as Slack API\n\n    Claude-\u003e\u003eMCP: JSON-RPC Request\u003cbr/\u003etools/call \"hybrid_search\"\n\n    MCP-\u003e\u003eAuth: Authenticate Request\n\n    alt IP Authentication\n        Auth-\u003e\u003eAuth: Check Client IP\n    else OIDC Authentication\n        Auth-\u003e\u003eAuth: Verify JWT Token\n    else Bypass Range\n        Auth-\u003e\u003eAuth: Check Bypass CIDR\n    end\n\n    Auth--\u003e\u003eMCP: Authentication OK\n\n    MCP-\u003e\u003eHandler: Handle hybrid_search(query, options)\n\n    Handler-\u003e\u003eBedrock: GenerateEmbedding(query)\n    Bedrock--\u003e\u003eHandler: Vector[1024]\n\n    par OpenSearch BM25\n        Handler-\u003e\u003eOS: BM25 Query\n        OS--\u003e\u003eHandler: BM25 Results\n    and OpenSearch k-NN\n        Handler-\u003e\u003eOS: k-NN Query\n        OS--\u003e\u003eHandler: Vector Results\n    end\n\n    Handler-\u003e\u003eHandler: Fuse Results (weighted_sum/RRF)\n\n    opt enable_slack_search=true\n        Handler-\u003e\u003eSlack: Search Messages + Threads\n        Slack--\u003e\u003eHandler: Slack Context\n        Handler-\u003e\u003eHandler: Merge Slack Results\n    end\n\n    Handler--\u003e\u003eMCP: Search Results (JSON)\n    MCP--\u003e\u003eClaude: JSON-RPC Response\u003cbr/\u003e{results, slack_results, metadata}\n```\n\n#### Vectorization Pipeline\n\nThe vectorize command processes source documents (Markdown, CSV, and S3 objects) and stores them in dual backends:\n\n```mermaid\nflowchart TD\n    Start([vectorize command]) --\u003e CheckFlags{Check Flags}\n\n    CheckFlags --\u003e|--clear| Clear[Delete All Vectors\u003cbr/\u003e+ Recreate Index]\n    CheckFlags --\u003e|Normal| ScanSources\n    Clear --\u003e ScanSources\n\n    subgraph ScanSources[Scan Data Sources]\n        LocalScan[Scan Local Directory\u003cbr/\u003e./source]\n        S3Scan[Scan S3 Bucket\u003cbr/\u003e--enable-s3]\n    end\n\n    ScanSources --\u003e Files[List Files\u003cbr/\u003e.md, .markdown, .csv]\n\n    Files --\u003e Loop{For Each File\u003cbr/\u003eParallel Processing}\n\n    Loop --\u003e FileType{File Type?}\n\n    FileType --\u003e|Markdown| ReadMD[Read File Content]\n    FileType --\u003e|CSV| ReadCSV[Read CSV \u0026 Expand Rows\u003cbr/\u003eEach row → 1 document]\n\n    ReadMD --\u003e Extract[Extract Metadata\u003cbr/\u003eFrontMatter Parser]\n    ReadCSV --\u003e ExtractCSV[Extract Metadata\u003cbr/\u003eCSV Column Mapping]\n\n    Extract --\u003e Embed[Generate Embedding\u003cbr/\u003eBedrock Titan v2]\n    ExtractCSV --\u003e Embed\n\n    Embed --\u003e Dual{Dual Backend Storage}\n\n    Dual --\u003e S3Store[Store to S3 Vector\u003cbr/\u003ewith Metadata]\n    Dual --\u003e OSIndex[Index to OpenSearch\u003cbr/\u003eBM25 + k-NN fields]\n\n    S3Store --\u003e Stats1[Update Stats\u003cbr/\u003eS3: Success/Fail]\n    OSIndex --\u003e Stats2[Update Stats\u003cbr/\u003eOS: Indexed/Skipped/Retry]\n\n    Stats1 --\u003e Check{More Files?}\n    Stats2 --\u003e Check\n\n    Check --\u003e|Yes| Loop\n    Check --\u003e|No| Report[Display Statistics\u003cbr/\u003eSuccess Rate, Errors]\n\n    Report --\u003e Follow{Follow Mode?}\n    Follow --\u003e|Yes| Wait[Wait Interval\u003cbr/\u003eDefault: 30m]\n    Wait --\u003e ScanSources\n    Follow --\u003e|No| End([Completed])\n\n    style Start fill:#e1f5ff\n    style Clear fill:#ffe1e1\n    style ScanSources fill:#e8f4f8\n    style Embed fill:#fff4e1\n    style Dual fill:#f5e1ff\n    style End fill:#e1ffe1\n```\n\n## Prerequisites\n\n### Prepare Source Documents\n\nBefore using RAGent, you need to prepare source documents in a `source/` directory. These documents should contain the content you want to make searchable through the RAG system.\n\n**Supported file types:**\n- **Markdown (.md, .markdown)**: Each file becomes one document\n- **CSV (.csv)**: Each row becomes one document (header row required)\n- **PDF (.pdf)**: Each page becomes one document (requires `OCR_PROVIDER=bedrock` or `OCR_PROVIDER=gemini`)\n\n```bash\n# Create source directory\nmkdir source\n\n# Place your files in this directory\ncp /path/to/your/documents/*.md source/\ncp /path/to/your/data/*.csv source/\n```\n\nFor CSV files, you can optionally provide a configuration file to specify column mappings:\n```bash\n# Copy example configuration\ncp csv-config.yaml.example csv-config.yaml\n\n# Run with CSV configuration\nRAGent vectorize --csv-config csv-config.yaml\n```\n\n#### CSV Configuration Options\n\nThe `csv-config.yaml` supports the following options:\n\n**header_row (Header Row Position):**\n\nUse this option when your CSV file has metadata or summary rows before the actual header row.\nWhen `header_row` is specified, that row is used as the column headers, and all preceding rows are skipped.\n\n```yaml\ncsv:\n  files:\n    - pattern: \"sample.csv\"\n      header_row: 7  # Row 7 is the header (1-indexed)\n                     # Rows 1-6 are skipped, data starts from row 8\n      content:\n        columns: [\"task\", \"category\"]\n      metadata:\n        title: \"task\"\n        category: \"category\"\n```\n\n- If `header_row` is not specified, the default is `1` (first row is the header)\n- Row numbers are 1-indexed\n\nFor exporting notes from Kibela, use the separate export tool available in the `export/` directory.\n\n## Required Environment Variables\n\nCreate a `.env` file in the project root and configure the following environment variables:\n\n```env\n# AWS Configuration\nAWS_REGION=your_aws_region\nAWS_ACCESS_KEY_ID=your_access_key\nAWS_SECRET_ACCESS_KEY=your_secret_key\n\n# S3 Vector Configuration\nS3_VECTOR_INDEX_NAME=your_vector_index_name\nS3_BUCKET_NAME=your_s3_bucket_name\nS3_VECTOR_REGION=us-east-1           # AWS region for S3 Vector bucket\nS3_SOURCE_REGION=ap-northeast-1      # AWS region for source S3 bucket (--enable-s3)\n\n# Vector DB Backend Selection\nVECTOR_DB_BACKEND=s3            # Backend type: \"s3\" (Amazon S3 Vectors) or \"sqlite\" (local sqlite-vec) (default: s3)\nSQLITE_VEC_DB_PATH=~/.ragent/vectors.db  # Path to sqlite-vec DB file (used when VECTOR_DB_BACKEND=sqlite)\n\n# OpenSearch Configuration (for Hybrid RAG)\nOPENSEARCH_ENDPOINT=your_opensearch_endpoint\nOPENSEARCH_INDEX=your_opensearch_index\nOPENSEARCH_REGION=us-east-1  # default\n\n# GitHub Configuration (optional)\nGITHUB_TOKEN=ghp_your_github_token  # Required for private repositories\n\n# Chat Configuration\nCHAT_MODEL=anthropic.claude-3-5-sonnet-20240620-v1:0  # default\nEXCLUDE_CATEGORIES=Personal,Daily  # Categories to exclude from search\n\n# MCP Server Configuration\nMCP_SERVER_HOST=localhost\nMCP_SERVER_PORT=8080\nMCP_IP_AUTH_ENABLED=true\nMCP_ALLOWED_IPS=127.0.0.1,::1  # Comma-separated list\n\n# MCP Bypass Configuration (optional)\nMCP_BYPASS_IP_RANGE=10.0.0.0/8,172.16.0.0/12  # Comma-separated CIDR ranges\nMCP_BYPASS_VERBOSE_LOG=false\nMCP_BYPASS_AUDIT_LOG=true\nMCP_TRUSTED_PROXIES=192.168.1.1,10.0.0.1  # Trusted proxy IPs for X-Forwarded-For\n\n# OIDC Authentication (optional)\nOIDC_ISSUER=https://accounts.google.com  # Your OIDC provider URL\nOIDC_CLIENT_ID=your_client_id\nOIDC_CLIENT_SECRET=your_client_secret\n\n# Slack Bot Configuration\nSLACK_BOT_TOKEN=xoxb-your-bot-token\nSLACK_USER_TOKEN=xoxp-your-user-token-with-search-read\nSLACK_RESPONSE_TIMEOUT=5s\nSLACK_MAX_RESULTS=5\nSLACK_ENABLE_THREADING=false\nSLACK_THREAD_CONTEXT_ENABLED=true\nSLACK_THREAD_CONTEXT_MAX_MESSAGES=10\n\n# Slack Search Configuration\nSLACK_SEARCH_ENABLED=false                     # Enable Slack search pipeline (set true to activate)\nSLACK_SEARCH_MAX_RESULTS=20                    # Max Slack messages retrieved per query (1-100)\nSLACK_SEARCH_MAX_RETRIES=5                     # Retry count for Slack API rate limits (0-10)\nSLACK_SEARCH_CONTEXT_WINDOW_MINUTES=30         # Time window (minutes) of surrounding message context\nSLACK_SEARCH_MAX_ITERATIONS=5                  # Iterative refinements performed when answers insufficient\nSLACK_SEARCH_MAX_CONTEXT_MESSAGES=100          # Max messages assembled into enriched context\nSLACK_SEARCH_TIMEOUT_SECONDS=5                 # Slack API request timeout in seconds (1-60)\n\n# OpenTelemetry Configuration (optional)\nOTEL_ENABLED=false\nOTEL_SERVICE_NAME=ragent\nOTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318\nOTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf\nOTEL_RESOURCE_ATTRIBUTES=service.namespace=ragent,environment=dev\nOTEL_TRACES_SAMPLER=always_on\nOTEL_TRACES_SAMPLER_ARG=1.0\n\n# OCR Configuration (for PDF vectorization)\nOCR_PROVIDER=bedrock                                    # OCR provider (\"bedrock\" or \"gemini\"; omit to skip PDFs)\nOCR_MODEL=anthropic.claude-3-5-sonnet-20241022-v2:0    # Model for OCR (Bedrock model ID or Gemini model name)\nOCR_TIMEOUT=120s                                        # OCR request timeout (default: 120s)\n\n# Embedding Configuration\nEMBEDDING_PROVIDER=bedrock                              # Embedding provider (\"bedrock\" [default] or \"gemini\")\nEMBEDDING_MODEL=                                        # Embedding model ID (optional; defaults to provider's default)\nEMBEDDING_DIMENSION=                                    # Output vector dimension (optional; overrides model default)\n\n# Gemini Configuration (OCR_PROVIDER=gemini and/or EMBEDDING_PROVIDER=gemini)\n# Option 1: API key authentication\nGEMINI_API_KEY=your_gemini_api_key                      # Google AI Studio API key\n# Option 2: Application Default Credentials (GOOGLE_APPLICATION_CREDENTIALS)\nGEMINI_GCP_PROJECT=your_gcp_project_id                  # GCP project ID for Vertex AI\nGEMINI_GCP_LOCATION=us-central1                         # GCP region for Vertex AI (default: us-central1)\n\n# AWS Secrets Manager Configuration (optional)\nSECRET_MANAGER_SECRET_ID=ragent/app              # Secret ID in AWS Secrets Manager (omit to disable)\nSECRET_MANAGER_REGION=us-east-1                  # AWS region for Secrets Manager (default: us-east-1)\n```\n\n### Gemini Embedding Configuration\n\nTo use Gemini for embeddings instead of Amazon Bedrock, set `EMBEDDING_PROVIDER=gemini`. Two authentication methods are supported:\n\n```bash\n# Use Gemini for embeddings (with API Key)\nexport EMBEDDING_PROVIDER=gemini\nexport EMBEDDING_MODEL=text-embedding-004\nexport GEMINI_API_KEY=your-api-key\n\n# Use Gemini for embeddings (with Vertex AI)\nexport EMBEDDING_PROVIDER=gemini\nexport GEMINI_GCP_PROJECT=your-project\nexport GEMINI_GCP_LOCATION=us-central1\n```\n\nWhen `EMBEDDING_MODEL` is omitted, Gemini defaults to `text-embedding-004` (768 dimensions). Bedrock defaults to `amazon.titan-embed-text-v2:0` (1024 dimensions).\n\nModels such as `gemini-embedding-2-preview` support configurable output dimensions. Set `EMBEDDING_DIMENSION` to match your vector store index — for example `EMBEDDING_DIMENSION=1024` for an OpenSearch index created with 1024 dimensions. When omitted, the model's default dimension is used.\n\nSlack search requires `SLACK_SEARCH_ENABLED=true`, a valid `SLACK_BOT_TOKEN`, **and** a user token `SLACK_USER_TOKEN` that includes scopes such as `search:read`, `channels:history`, `groups:history`, and other conversation history scopes relevant to the channels you query. The search-specific knobs let you tune throughput and cost per workspace without touching the core document pipeline.\n\n### MCP Bypass Configuration (Optional)\n\n- `MCP_BYPASS_IP_RANGE`: Comma-separated CIDR ranges that skip authentication for trusted networks.\n- `MCP_BYPASS_VERBOSE_LOG`: Enables detailed logging for bypass decisions to aid troubleshooting.\n- `MCP_BYPASS_AUDIT_LOG`: Emits JSON audit entries for bypassed requests (enabled by default for compliance).\n- `MCP_TRUSTED_PROXIES`: Comma-separated list of proxy IPs whose `X-Forwarded-For` headers are trusted during bypass checks.\n\n## AWS Secrets Manager Integration\n\nRAGent supports AWS Secrets Manager as a fallback for environment variables. When configured, secrets stored in Secrets Manager are automatically injected as environment variables at startup — but **only for keys that are not already set**. This means existing environment variables always take priority and are never overwritten.\n\n### How It Works\n\n1. On startup, RAGent checks for the `SECRET_MANAGER_SECRET_ID` environment variable.\n2. If set, it fetches the secret from AWS Secrets Manager (the secret must be stored as a JSON key-value pair).\n3. For each key in the JSON, if the corresponding environment variable is **not already set**, the value is injected.\n4. If the environment variable is already set (via `.env`, shell export, etc.), the Secrets Manager value is ignored.\n\nThis design allows you to:\n- Store sensitive credentials (API keys, tokens) centrally in Secrets Manager\n- Override any secret locally by setting the environment variable directly\n- Use the same secret across multiple deployment environments\n\n### Configuration\n\n| Environment Variable | Description | Default |\n|---|---|---|\n| `SECRET_MANAGER_SECRET_ID` | Secret ID or ARN in AWS Secrets Manager | _(not set — feature disabled)_ |\n| `SECRET_MANAGER_REGION` | AWS region for Secrets Manager | `us-east-1` |\n\n### Registering Secrets\n\nStore your secrets as a JSON object in AWS Secrets Manager. Each key should match the environment variable name used by RAGent.\n\n**Using AWS CLI:**\n\n```bash\n# Create a new secret\naws secretsmanager create-secret \\\n  --name \"ragent/app\" \\\n  --description \"RAGent application secrets\" \\\n  --secret-string '{\n    \"SLACK_BOT_TOKEN\": \"xoxb-your-bot-token\",\n    \"SLACK_USER_TOKEN\": \"xoxp-your-user-token\",\n    \"OIDC_CLIENT_SECRET\": \"your-oidc-secret\",\n    \"GITHUB_TOKEN\": \"ghp_your-github-token\",\n    \"GEMINI_API_KEY\": \"your-gemini-api-key\"\n  }' \\\n  --region us-east-1\n\n# Update an existing secret\naws secretsmanager put-secret-value \\\n  --secret-id \"ragent/app\" \\\n  --secret-string '{\n    \"SLACK_BOT_TOKEN\": \"xoxb-new-token\",\n    \"SLACK_USER_TOKEN\": \"xoxp-new-token\",\n    \"OIDC_CLIENT_SECRET\": \"new-secret\",\n    \"GITHUB_TOKEN\": \"ghp_new-token\",\n    \"GEMINI_API_KEY\": \"new-api-key\"\n  }' \\\n  --region us-east-1\n```\n\n**Using AWS Console:**\n\n1. Open the [AWS Secrets Manager Console](https://console.aws.amazon.com/secretsmanager/)\n2. Click **Store a new secret**\n3. Select **Other type of secret**\n4. Choose **Plaintext** and enter a JSON object with key-value pairs\n5. Name the secret (e.g., `ragent/app`) and complete the wizard\n\n### IAM Permissions\n\nThe IAM role or user running RAGent needs the following permissions:\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"secretsmanager:GetSecretValue\",\n        \"secretsmanager:DescribeSecret\"\n      ],\n      \"Resource\": \"arn:aws:secretsmanager:\u003cregion\u003e:\u003caccount-id\u003e:secret:ragent/app-*\"\n    }\n  ]\n}\n```\n\n### Usage Example\n\n```bash\n# Set only the Secrets Manager config — all other secrets are fetched automatically\nexport SECRET_MANAGER_SECRET_ID=ragent/app\nexport SECRET_MANAGER_REGION=us-east-1\n\n# Non-secret config still set via environment / .env\nexport OPENSEARCH_ENDPOINT=https://your-opensearch-endpoint\nexport OPENSEARCH_INDEX=your-index\n\n# Start RAGent — SLACK_BOT_TOKEN, GITHUB_TOKEN, etc. are loaded from Secrets Manager\nRAGent slack-bot\n```\n\n```bash\n# Override a specific secret locally (takes priority over Secrets Manager)\nexport SECRET_MANAGER_SECRET_ID=ragent/app\nexport SLACK_BOT_TOKEN=xoxb-local-override-token  # This value wins\n\nRAGent slack-bot  # Uses the local SLACK_BOT_TOKEN, other secrets from SM\n```\n\n\u003e **Note**: Only string values in the JSON are injected. Non-string values (objects, arrays, numbers, booleans, null) are silently skipped.\n\n## OpenTelemetry Observability\n\nRAGent exposes distributed traces and usage metrics through the OpenTelemetry (OTel) Go SDK. Traces are emitted for Slack Bot message handling, MCP tool calls, and the shared hybrid search service, while counters and histograms capture request rates, error rates, and response times.\n\n### Enable OTel\n\nSet the following environment variables (see `.env.example` for defaults):\n\n- `OTEL_ENABLED`: `true` to enable tracing and metrics (defaults to `false`).\n- `OTEL_SERVICE_NAME`: Logical service name (`ragent` by default).\n- `OTEL_RESOURCE_ATTRIBUTES`: Comma-separated `key=value` pairs such as `service.namespace=ragent,environment=dev`.\n- `OTEL_EXPORTER_OTLP_ENDPOINT`: OTLP endpoint URL (including scheme).\n- `OTEL_EXPORTER_OTLP_PROTOCOL`: `http/protobuf` (default) or `grpc`.\n- `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG`: Configure sampling strategy (`always_on`, `traceidratio`, etc.).\n\nWhen `OTEL_ENABLED=false`, RAGent registers no-op providers and carries no runtime overhead.\n\n### Example: Jaeger (local development)\n\n```bash\n# 1. Start Jaeger all-in-one\ndocker run --rm -it -p 4318:4318 -p 16686:16686 jaegertracing/all-in-one:1.58\n\n# 2. Enable OTel before running RAGent\nexport OTEL_ENABLED=true\nexport OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318\nexport OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf\n\n# 3. Run the Slack bot (or any other command)\ngo run main.go slack-bot\n# Visit http://localhost:16686 to explore spans\n```\n\n### Example: Prometheus via OpenTelemetry Collector\n\nUse the OTel Collector to convert OTLP metrics into Prometheus format:\n\n```yaml\n# collector.yaml\nreceivers:\n  otlp:\n    protocols:\n      http:\n        endpoint: 0.0.0.0:4318\nexporters:\n  prometheus:\n    endpoint: 0.0.0.0:9464\nservice:\n  pipelines:\n    metrics:\n      receivers: [otlp]\n      exporters: [prometheus]\n```\n\n```bash\notelcol --config collector.yaml\nexport OTEL_ENABLED=true\nexport OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318\ngo run main.go mcp-server\n# Scrape metrics at http://localhost:9464/metrics\n```\n\n### Example: AWS X-Ray with ADOT Collector\n\n```bash\n# 1. Run the AWS Distro for OpenTelemetry collector\ndocker run --rm -it -p 4317:4317 public.ecr.aws/aws-observability/aws-otel-collector:latest\n\n# 2. Configure RAGent to send spans over gRPC\nexport OTEL_ENABLED=true\nexport OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317\nexport OTEL_EXPORTER_OTLP_PROTOCOL=grpc\nexport OTEL_TRACES_SAMPLER=traceidratio\nexport OTEL_TRACES_SAMPLER_ARG=0.2\n```\n\n### Metrics \u0026 Span Names\n\n- **Spans**\n  - `slackbot.process_message`\n  - `mcpserver.hybrid_search`\n  - `search.hybrid`\n- **Metrics**\n  - `ragent.slack.requests.total`, `ragent.slack.errors.total`, `ragent.slack.response_time`\n  - `ragent.mcp.requests.total`, `ragent.mcp.errors.total`, `ragent.mcp.response_time`\n\nAttach additional attributes such as channel type, authentication method, tool name, and result totals for fine-grained analysis.\n\n### Grafana Dashboard\n\nRAGent includes a pre-configured Grafana dashboard template for visualizing OpenTelemetry metrics. The dashboard provides comprehensive monitoring of Slack Bot and MCP Server operations.\n\n![Grafana Dashboard](assets/grafana.png)\n\n**Dashboard Panels:**\n- **Slack Metrics**\n  - Slack Bot Requests (time series)\n  - Slack Response Latency (histogram)\n  - Slack Errors Rate (gauge)\n- **MCP Metrics**\n  - MCP Requests (time series)\n  - MCP Response Time (histogram)\n  - MCP Errors Rate (gauge)\n\n**Setup Instructions:**\n\n1. Enable OpenTelemetry in RAGent:\n   ```bash\n   export OTEL_ENABLED=true\n   export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318\n   ```\n\n2. Configure OTel Collector to export metrics to Prometheus (see example above)\n\n3. Import the dashboard template into Grafana:\n   ```bash\n   # Using Grafana API\n   curl -X POST http://localhost:3000/api/dashboards/import \\\n     -H \"Content-Type: application/json\" \\\n     -d @assets/grafana.json\n   \n   # Or manually import via Grafana UI\n   # Dashboard \u003e Import \u003e Upload JSON file \u003e Select assets/grafana.json\n   ```\n\n4. Configure Prometheus data source in Grafana pointing to your metrics endpoint\n\nThe dashboard will automatically start displaying metrics once RAGent begins processing requests.\n\n## Installation\n\n### Prerequisites\n\n- Go 1.25.0 or higher\n- direnv (recommended)\n\n### Build\n\n```bash\n# Clone the repository\ngit clone https://github.com/ca-srg/ragent.git\ncd RAGent\n\n# Install dependencies\ngo mod download\n\n# Build\ngo build -o RAGent\n\n# Add executable to PATH (optional)\nmv RAGent /usr/local/bin/\n```\n\n## Releases\n\nPrebuilt binaries are published on GitHub Releases for Linux (amd64, arm64).\n\n- Releases page: https://github.com/ca-srg/ragent/releases\n\n### Create a Release (maintainers)\n\nUse semantic versioning (`MAJOR.MINOR.PATCH`) and push a tag to trigger the pipeline:\n\n```bash\n# Example: v1.0.0\ngit tag v1.0.0\ngit push origin v1.0.0\n```\n\nThis triggers the GitHub Actions workflow which builds archives via GoReleaser and publishes them to the release.\n\n### Download Binaries\n\nFrom the Releases page, download the archive for your platform:\n\n- Linux amd64: `ragent_\u003cVERSION\u003e_linux_amd64.tar.gz`\n- Linux arm64: `ragent_\u003cVERSION\u003e_linux_arm64.tar.gz`\n\nExtract and place the binary on your `PATH`:\n\n```bash\ntar -xzf ragent_\u003cVERSION\u003e_linux_amd64.tar.gz\nsudo mv ragent /usr/local/bin/\n```\n\n### Verify Checksums\n\nEach release includes a SHA256 checksum file.\n\n```bash\n# Linux\nsha256sum -c checksums_\u003cVERSION\u003e.txt\n\n# macOS (if verifying from macOS)\nshasum -a 256 -c checksums_\u003cVERSION\u003e.txt\n```\n\nAll entries should report `OK`. If a mismatch occurs, re-download the artifact.\n\n## Vector DB Backend Selection\n\nRAGent supports two vector storage backends, selectable via the `VECTOR_DB_BACKEND` environment variable.\n\n### Amazon S3 Vectors (default, `VECTOR_DB_BACKEND=s3`)\n\n```env\nVECTOR_DB_BACKEND=s3\nAWS_S3_VECTOR_BUCKET=your-bucket\nAWS_S3_VECTOR_INDEX=your-index\nS3_VECTOR_REGION=us-east-1\n```\n\nSuitable for production deployments and large-scale document sets on AWS infrastructure.\n\n### sqlite-vec (local, `VECTOR_DB_BACKEND=sqlite`)\n\n```env\nVECTOR_DB_BACKEND=sqlite\nSQLITE_VEC_DB_PATH=./vectors.db\n```\n\nSuitable for local development and small-to-medium deployments without AWS dependencies.\n\n**Limitations:**\n- Single writer (WAL mode is enabled automatically)\n- Maximum 8,192 dimensions per embedding\n- Vector similarity search always uses OpenSearch (sqlite-vec stores vectors only)\n\n**Example:**\n```bash\n# Vectorize with sqlite-vec backend\nVECTOR_DB_BACKEND=sqlite SQLITE_VEC_DB_PATH=./vectors.db \\\n  OPENSEARCH_ENDPOINT=https://... OPENSEARCH_INDEX=ragent \\\n  RAGent vectorize\n\n# List stored vectors\nVECTOR_DB_BACKEND=sqlite SQLITE_VEC_DB_PATH=./vectors.db \\\n  OPENSEARCH_ENDPOINT=https://... OPENSEARCH_INDEX=ragent \\\n  RAGent list\n```\n\n\u003e **Note**: OpenSearch is required regardless of the vector backend selection.\n\n## Commands\n\n### 1. vectorize - Vectorization and S3 Storage\n\nRead source files (markdown and CSV), extract metadata, generate embeddings using Amazon Bedrock, and store them in Amazon S3 Vectors.\n\n```bash\nRAGent vectorize\n```\n\n**Options:**\n- `-d, --directory`: Directory containing source files to process (default: `./source`)\n- `--dry-run`: Display processing details without making actual API calls\n- `-c, --concurrency`: Number of concurrent processes (0 = use default value from config file)\n- `--csv-config`: Path to CSV configuration YAML file (for column mapping)\n- `--enable-s3`: Enable S3 source file fetching\n- `--s3-bucket`: S3 bucket name for source files (required when `--enable-s3` is set)\n- `--s3-prefix`: S3 prefix (directory) to scan (optional, defaults to bucket root)\n- `--s3-vector-region`: AWS region for S3 Vector bucket (overrides S3_VECTOR_REGION, default: us-east-1)\n- `--s3-source-region`: AWS region for source S3 bucket (overrides S3_SOURCE_REGION, default: us-east-1)\n- `--github-repos`: Comma-separated list of GitHub repositories to clone and vectorize (format: `owner/repo`)\n\n**S3 Source Examples:**\n```bash\n# S3 only (with prefix)\nRAGent vectorize --enable-s3 --s3-bucket my-docs-bucket --s3-prefix source/\n\n# S3 only (flat structure)\nRAGent vectorize --enable-s3 --s3-bucket my-docs-bucket\n\n# Local + S3 combined\nRAGent vectorize --directory ./local-docs --enable-s3 --s3-bucket my-docs-bucket --s3-prefix remote/\n\n# Dry run with S3 source\nRAGent vectorize --enable-s3 --s3-bucket my-docs-bucket --dry-run\n```\n\n**GitHub Source Examples:**\n```bash\n# Clone and vectorize a single GitHub repository\nRAGent vectorize --github-repos \"owner/repo\"\n\n# Multiple repositories\nRAGent vectorize --github-repos \"org/repo1,org/repo2\"\n\n# Combined with local source\nRAGent vectorize --directory ./local-docs --github-repos \"owner/repo\"\n\n# Dry run with GitHub source\nRAGent vectorize --github-repos \"owner/repo\" --dry-run\n\n# Follow mode with GitHub repos (re-clones on each cycle)\nRAGent vectorize --follow --github-repos \"owner/repo\"\n```\n\nFor private repositories, set the `GITHUB_TOKEN` environment variable.\nMetadata is auto-generated from the repository structure: owner name as author, repository name as source, parent directory as category, and a GitHub URL as reference.\n\nFor detailed documentation on the GitHub data source feature, see [doc/github.md](doc/github.md).\n\n**Features:**\n- Recursive scanning of markdown and CSV files\n- Automatic metadata extraction\n- CSV row expansion (each row becomes a document)\n- Automatic column detection for CSV files (or explicit configuration)\n- Embedding generation using Amazon Titan Text Embedding v2\n- Safe storage to S3 Vectors\n- High-speed processing through concurrency\n- PDF page extraction via OCR (each page becomes a document, requires `OCR_PROVIDER=bedrock` or `OCR_PROVIDER=gemini`)\n\n### 2. query - Semantic Search\n\nExecute semantic similarity search against S3 Vector Index.\n\n```bash\n# Basic search\nRAGent query -q \"machine learning algorithms\"\n\n# Search with detailed options\nRAGent query --query \"API documentation\" --top-k 5 --json\n\n# Search with metadata filter\nRAGent query -q \"error handling\" --filter '{\"category\":\"programming\"}'\n```\n\n**Options:**\n- `-q, --query`: Search query text (required)\n- `-k, --top-k`: Number of similar results to return (default: 10)\n- `-j, --json`: Output results in JSON format\n- `-f, --filter`: JSON metadata filter (e.g., `'{\"category\":\"docs\"}'`)\n- `--enable-slack-search`: Include Slack conversations alongside document results when Slack search is enabled\n\n**Usage Examples:**\n```bash\n# Search technical documentation\nRAGent query -q \"Docker container configuration\" --top-k 3\n\n# Search within specific category\nRAGent query -q \"authentication\" --filter '{\"type\":\"security\"}' --json\n\n# Get more results\nRAGent query -q \"database optimization\" --top-k 20\n\n# Merge Slack context for incident reviews\nRAGent query -q \"on-call handoff\" --enable-slack-search\n\n# Reference a specific Slack message by URL (automatically fetched)\nRAGent query -q \"What was discussed in https://your-workspace.slack.com/archives/C12345678/p1234567890123456\"\n```\n\n#### URL-Aware Search\n\nRAGent inspects each query for HTTP/HTTPS URLs. When a URL is present, it first performs an exact term query on the `reference` field before running the usual hybrid search pipeline.\n\n- Successful URL matches return immediately with `search_method` set to `\"url_exact_match\"` and include the URL-only results.\n- If the term query fails or returns no hits, the engine falls back to the hybrid search flow and records the fallback reason (`term_query_error` or `term_query_no_results`).\n- CLI `--json` output, Slack bot responses, and MCP tool results expose `search_method`, `url_detected`, and `fallback_reason` so callers can inspect how the result was produced.\n\n```bash\nRAGent query --json -q \"https://example.com/doc のタイトルを教えて\"\n```\n\nExample JSON response fragment:\n\n```json\n{\n  \"search_method\": \"url_exact_match\",\n  \"url_detected\": true,\n  \"fallback_reason\": \"\",\n  \"results\": [\n    { \"title\": \"Example Doc\", \"reference\": \"https://example.com/doc\" }\n  ]\n}\n```\n\nIf a URL is not detected or the exact match misses, `search_method` falls back to `\"hybrid_search\"` with the usual fused BM25/vector results.\n\n### 3. list - List Vectors\n\nDisplay a list of vectors stored in S3 Vector Index.\n\n```bash\n# Display all vectors\nRAGent list\n\n# Filter by prefix\nRAGent list --prefix \"docs/\"\n```\n\n**Options:**\n- `-p, --prefix`: Prefix to filter vector keys\n\n**Features:**\n- Display stored vector keys\n- Filtering by prefix\n- Check vector database contents\n\n### 4. chat - Interactive RAG Chat\n\nStart an interactive chat session using hybrid search (OpenSearch BM25 + vector search) for context retrieval and Amazon Bedrock (Claude) for generating responses.\n\n```bash\n# Start interactive chat with default settings\nRAGent chat\n\n# Chat with custom context size\nRAGent chat --context-size 10\n\n# Chat with custom weight balance for hybrid search\nRAGent chat --bm25-weight 0.7 --vector-weight 0.3\n\n# Chat with custom system prompt\nRAGent chat --system \"You are a helpful assistant specialized in documentation.\"\n\n# Chat with Slack context enabled via environment flag\nSLACK_SEARCH_ENABLED=true RAGent chat\n\n# Reference Slack messages by URL in your questions\n# (URLs are automatically detected and messages fetched)\nRAGent chat\n\u003e What does this Slack discussion mean? https://your-workspace.slack.com/archives/C12345678/p1234567890123456\n```\n\n**Options:**\n- `-c, --context-size`: Number of context documents to retrieve (default: 5)\n- `-i, --interactive`: Run in interactive mode (default: true)\n- `-s, --system`: System prompt for the chat\n- `-b, --bm25-weight`: Weight for BM25 scoring in hybrid search (0-1, default: 0.5)\n- `-v, --vector-weight`: Weight for vector scoring in hybrid search (0-1, default: 0.5)\n- `--use-japanese-nlp`: Use Japanese NLP optimization for OpenSearch (default: true)\n\nWhen `SLACK_SEARCH_ENABLED=true`, chat sessions automatically pull in recent Slack conversations, show live progress for each refinement iteration, and append permalinks under the final answer.\n\n**Features:**\n- Hybrid search combining BM25 and vector similarity\n- Context-aware responses using retrieved documents\n- Conversation history management\n- Reference citations with source links\n- Japanese language optimization\n\n**Chat Commands:**\n- `exit` or `quit`: End the chat session\n- `clear`: Clear conversation history\n- `help`: Show available commands\n\n### 5. slack-bot - Slack Bot for RAG Search\n\nStart a Slack Bot that listens for mentions and answers with RAG results.\n\n```bash\nRAGent slack-bot\n```\n\nRequirements:\n- Set `SLACK_BOT_TOKEN` in your environment (see `.env.example`).\n- Invite the bot user to the target Slack channel.\n- Optionally enable threading with `SLACK_ENABLE_THREADING=true`.\n- Thread context: enable contextual search with `SLACK_THREAD_CONTEXT_ENABLED=true` (default) and control history depth via `SLACK_THREAD_CONTEXT_MAX_MESSAGES` (default `10` messages).\n- Slack search: set `SLACK_SEARCH_ENABLED=true` to blend Slack conversations into every answer. Use `SLACK_SEARCH_MAX_RESULTS`, `SLACK_SEARCH_MAX_CONTEXT_MESSAGES`, and `SLACK_SEARCH_MAX_ITERATIONS` to tune throughput per workspace.\n- Requires OpenSearch configuration (`OPENSEARCH_ENDPOINT`, `OPENSEARCH_INDEX`, `OPENSEARCH_REGION`). Slack Bot does not use S3 Vector fallback.\n\nDetails: see `docs/slack-bot.md`.\n\nWhen enabled, the bot delivers a Block Kit section labelled **Conversations from Slack** with permalinks for each hit so responders can jump straight into the thread.\n\n### 6. mcp-server - MCP Server for Claude Desktop Integration (New)\n\nStart an MCP (Model Context Protocol) server that provides hybrid search capabilities to Claude Desktop and other MCP-compatible tools.\n\n```bash\n# Start with OIDC authentication only\nRAGent mcp-server --auth-method oidc\n\n# Allow either IP or OIDC authentication (recommended for development)\nRAGent mcp-server --auth-method either\n\n# Require both IP and OIDC authentication (highest security)\nRAGent mcp-server --auth-method both\n\n# IP authentication only (default)\nRAGent mcp-server --auth-method ip\n```\n\nThe `hybrid_search` MCP tool accepts two new parameters when the server is launched with `SLACK_SEARCH_ENABLED=true`:\n\n- `enable_slack_search` (boolean, default `false`): opt in per request from your MCP client.\n\nResponses include a `slack_results` array with message metadata and permalinks so downstream tools can render conversation context alongside document references.\n\n**Authentication Methods:**\n- `ip`: Traditional IP address-based authentication only\n- `oidc`: OpenID Connect authentication only\n- `both`: Requires both IP and OIDC authentication\n- `either`: Allows either IP or OIDC authentication\n\n**Bypass Authentication:**\nFor CI/CD environments and internal services, you can configure bypass IP ranges that skip authentication:\n```bash\n# Bypass authentication for specific IP ranges\nRAGent mcp-server --bypass-ip-range \"10.0.0.0/8\" --bypass-ip-range \"172.16.0.0/12\"\n\n# Enable audit logging for bypass access\nRAGent mcp-server --bypass-ip-range \"10.10.0.0/16\" --bypass-audit-log\n\n# Verbose logging for troubleshooting\nRAGent mcp-server --bypass-ip-range \"10.0.0.0/8\" --bypass-verbose-log\n\n# Configure trusted proxies for X-Forwarded-For\nRAGent mcp-server --bypass-ip-range \"10.0.0.0/8\" --trusted-proxies \"192.168.1.1\"\n```\n\n**Supported OIDC Providers:**\n- Google Workspace (`https://accounts.google.com`)\n- Microsoft Azure AD/Entra ID (`https://login.microsoftonline.com/{tenant}/v2.0`)\n- Okta (`https://{domain}.okta.com`)\n- Keycloak (`https://{server}/realms/{realm}`)\n- Custom OAuth2 providers\n\n**Features:**\n- JSON-RPC 2.0 compliant MCP protocol\n- Hybrid search tool: `ragent-hybrid_search`\n- Multiple authentication methods\n- Claude Desktop integration\n- SSE and HTTP transport support\n- Browser-based authentication flow\n\n**Requirements:**\n- OpenSearch configuration is required for MCP server functionality\n- For OIDC: Configure your OAuth2 application and set environment variables\n- For IP auth: Configure allowed IP addresses or ranges\n\n**Usage with Claude Desktop:**\nAfter authentication, add the server to Claude Desktop using the provided command:\n```bash\nclaude mcp add --transport sse ragent https://your-server.example.com/sse --header \"Authorization: Bearer \u003cJWT\u003e\"\n```\n\nDetails: see `doc/mcp-server.md` and `doc/oidc-authentication.md`.\n\n### 7. webui - Web UI for Vectorization Monitoring (New)\n\nStart a web-based user interface for monitoring and controlling the vectorization process. The Web UI provides real-time progress tracking, file browsing, and processing history.\n\n```bash\n# Start with default settings (localhost:8081)\nRAGent webui\n\n# Specify custom host and port\nRAGent webui --host 0.0.0.0 --port 8080\n\n# Specify source directory\nRAGent webui --directory ./docs\n```\n\n**Options:**\n- `--host`: Host address to bind (default: `localhost`)\n- `--port`: Port number to listen on (default: `8081`)\n- `-d, --directory`: Source directory for vectorization (default: `./source`)\n\n**Features:**\n- **Dashboard**: Real-time vectorization progress with start/stop controls\n- **File Browser**: Browse and search source files (Markdown, CSV)\n- **Processing History**: View past vectorization runs with success/failure stats\n- **Follow Mode**: Enable/disable scheduled vectorization with configurable intervals\n- **SSE Updates**: Live updates via Server-Sent Events without page refresh\n- **Cross-Process Status Sharing**: Monitor external `vectorize --follow` processes via Unix Socket IPC\n\n**Cross-Process Status Sharing:**\n\nWhen a `vectorize --follow` process is running separately, the Web UI can detect and display its status in real-time. This is achieved through Unix Socket-based IPC (Inter-Process Communication).\n\n```bash\n# Terminal 1: Start vectorize in follow mode (runs IPC server)\nRAGent vectorize --follow --interval 5m\n\n# Terminal 2: Start webui (connects as IPC client)\nRAGent webui\n\n# The dashboard will show \"running\" status from the external process\n```\n\nThe IPC mechanism uses JSON-RPC 2.0 over Unix Socket:\n- Socket location: `$XDG_RUNTIME_DIR/ragent/ragent.sock` or `/tmp/ragent-{uid}/ragent.sock`\n- Exclusive locking prevents multiple vectorize instances from running simultaneously\n- Stale socket cleanup handles crashed processes gracefully\n\n**Dashboard Controls:**\n- Start/Stop vectorization manually\n- Toggle follow mode (scheduler) for periodic vectorization\n- View current progress, elapsed time, and estimated remaining time\n- Monitor recent errors with file paths and error details\n\n**API Endpoints:**\n| Method | Path | Description |\n|--------|------|-------------|\n| GET | `/` | Dashboard page |\n| GET | `/files` | File browser page |\n| GET | `/history` | Processing history page |\n| POST | `/api/vectorize/start` | Start vectorization |\n| POST | `/api/vectorize/stop` | Stop vectorization |\n| GET | `/api/status` | Get current status (includes external process status) |\n| GET | `/api/files` | List files (with search) |\n| POST | `/api/scheduler/toggle` | Toggle scheduler |\n| GET | `/sse/progress` | SSE progress stream |\n| GET | `/sse/events` | SSE events stream |\n\n**Usage Example:**\n```bash\n# Start the Web UI\nRAGent webui --port 8081 --directory ./source\n\n# Open in browser\nopen http://localhost:8081\n\n# The dashboard will show:\n# - Current vectorization status\n# - Start/Stop controls\n# - Progress bar during processing\n# - Recent errors\n# - Follow mode toggle\n```\n\n**Technical Stack:**\n- Go templates with HTMX for dynamic updates\n- Server-Sent Events (SSE) for real-time progress\n- Unix Socket IPC for cross-process communication\n- No external JavaScript frameworks required\n- Responsive CSS design\n\n## Development\n\n### Build Commands\n\n```bash\n# Format code\ngo fmt ./...\n\n# Tidy dependencies\ngo mod tidy\n\n# Run tests (if configured)\ngo test ./...\n\n# Development execution\ngo run main.go [command]\n```\n\n### Project Structure\n\n```\nRAGent/\n├── main.go                 # Entry point\n├── cmd/                    # CLI command definitions\n│   ├── root.go            # Root command and common settings\n│   ├── query.go           # query command\n│   ├── list.go            # list command\n│   ├── chat.go            # chat command\n│   ├── slack.go           # slack-bot command\n│   ├── mcp-server.go      # mcp-server command\n│   ├── webui.go           # webui command\n│   └── vectorize.go       # vectorize command\n├── internal/              # Internal libraries\n│   ├── pkg/              # Shared infrastructure (feature-agnostic)\n│   │   ├── config/       # Config struct (91 fields)\n│   │   ├── embedding/    # Embedding generation\n│   │   │   └── bedrock/  # Amazon Bedrock client\n│   │   ├── opensearch/   # OpenSearch client and queries\n│   │   ├── slacksearch/  # Slack search service\n│   │   ├── search/       # HybridSearchService\n│   │   ├── s3vector/     # S3 Vector client\n│   │   ├── metrics/      # Metrics collection\n│   │   ├── observability/ # OpenTelemetry\n│   │   └── ipc/          # Inter-process communication\n│   ├── ingestion/        # vectorize/list/recreate-index slice\n│   │   ├── csv/\n│   │   ├── hashstore/\n│   │   ├── metadata/\n│   │   ├── scanner/\n│   │   ├── spreadsheet/\n│   │   └── vectorizer/\n│   ├── query/            # query/chat slice\n│   │   └── filter/\n│   ├── slackbot/         # slack-bot slice\n│   ├── mcpserver/        # mcp-server slice\n│   └── webui/            # webui slice\n├── source/               # Source documents (markdown and CSV, prepare before use)\n├── export/               # Separate export tool for Kibela\n├── doc/                  # Project documentation\n│   ├── github.md         # GitHub data source guide\n│   ├── mcp-server.md     # MCP Server setup guide\n│   ├── oidc-authentication.md # OIDC authentication guide\n│   ├── filter-configuration.md # Filter configuration guide\n│   ├── s3-vector.md      # S3 Vector integration notes\n│   └── score.md          # RAGスコアの基礎解説\n├── .envrc                # direnv configuration\n├── .env                  # Environment variables file\n└── CLAUDE.md            # Claude Code configuration\n```\n\n## Dependencies\n\n### Core Libraries\n\n- **github.com/spf13/cobra**: CLI framework\n- **github.com/joho/godotenv**: Environment variable loader\n- **github.com/aws/aws-sdk-go-v2**: AWS SDK v2\n  - S3 service\n  - S3 Vectors\n  - Bedrock Runtime (Titan Embeddings)\n- **gopkg.in/yaml.v3**: YAML processing\n\n### AWS Related Libraries\n\n- `github.com/aws/aws-sdk-go-v2/config`: AWS configuration management\n- `github.com/aws/aws-sdk-go-v2/service/s3`: S3 operations\n- `github.com/aws/aws-sdk-go-v2/service/s3vectors`: S3 Vector operations\n- `github.com/aws/aws-sdk-go-v2/service/bedrockruntime`: Bedrock Runtime operations\n\n### MCP Integration\n\n- `github.com/modelcontextprotocol/go-sdk`: Official MCP SDK v0.4.0\n- `github.com/coreos/go-oidc`: OpenID Connect implementation\n- JSON-RPC 2.0 protocol support\n- Multiple authentication providers\n\n## Typical Workflow\n\n1. **Initial Setup**\n   ```bash\n   # Set environment variables\n   cp .env.example .env\n   # Edit .env file\n   ```\n\n2. **Prepare Source Documents**\n   ```bash\n   # Create source directory if not exists\n   mkdir -p source\n   \n   # Place your files in the directory (markdown and/or CSV)\n   cp /path/to/docs/*.md source/\n   cp /path/to/data/*.csv source/\n   \n   # For CSV files, optionally configure column mapping:\n   cp csv-config.yaml.example csv-config.yaml\n   # Edit csv-config.yaml to specify columns\n   \n   # Or use the export tool for Kibela notes:\n   cd export\n   go build -o RAGent-export\n   ./RAGent-export\n   cd ..\n   ```\n\n3. **Vectorization and S3 Storage**\n   ```bash\n   # Verify with dry run\n   RAGent vectorize --dry-run\n   \n   # Execute actual vectorization\n   RAGent vectorize\n\n   # Vectorize with CSV configuration\n   RAGent vectorize --csv-config csv-config.yaml\n\n   # Continuously vectorize using follow mode (default 30m interval)\n   RAGent vectorize --follow\n\n   # Customize the follow mode interval (e.g., every 15 minutes)\n   RAGent vectorize --follow --interval 15m\n   ```\n\n   \u003e Note: `--follow` cannot be combined with `--dry-run` or `--clear`.\n\n4. **Check Vector Data**\n   ```bash\n   RAGent list\n   ```\n\n5. **Execute Semantic Search**\n   ```bash\n   RAGent query -q \"content to search\"\n\n6. **Start MCP Server (for Claude Desktop)**\n   ```bash\n   # Configure OIDC provider (optional)\n   export OIDC_ISSUER=\"https://accounts.google.com\"\n   export OIDC_CLIENT_ID=\"your-client-id\"\n\n   # Start MCP server\n   RAGent mcp-server --auth-method either\n\n   # Visit http://localhost:8080/login to authenticate\n   # Follow instructions to add to Claude Desktop\n   ```\n   ```\n\n## Troubleshooting\n\n### Common Errors\n\n1. **Environment variables not set**\n   ```\n   Error: required environment variable not set\n   ```\n   → Check if `.env` file is properly configured\n\n2. **Configuration error**\n   ```\n   Error: configuration not found or invalid\n   ```\n   → Check configuration and authentication settings\n\n3. **AWS authentication error**\n   ```\n   Error: AWS credentials not found\n   ```\n   → Check if AWS credentials are properly configured\n\n4. **S3 Vector Index not found**\n   ```\n   Error: vector index not found\n   ```\n   → Verify S3 Vector Index is created\n\n5. **MCP Server authentication error**\n   ```\n   Error: IP address not allowed: 192.168.1.100\n   ```\n   → Add IP to MCP_ALLOWED_IPS or use --auth-method oidc\n\n6. **OIDC authentication error**\n   ```\n   Error: OIDC provider discovery failed\n   ```\n   → Check OIDC_ISSUER URL and network connectivity\n\n7. **Slack token missing or invalid**\n   ```\n   Slack search unavailable: SLACK_BOT_TOKEN not configured\n   ```\n   → Ensure `SLACK_SEARCH_ENABLED=true` and `SLACK_BOT_TOKEN` belongs to the same workspace you are querying.\n\n8. **Slack rate limit (HTTP 429)**\n   ```\n   slack search failed: rate_limited\n   ```\n   → Respect the `Retry-After` header, increase `SLACK_SEARCH_MAX_RETRIES`, or reduce `SLACK_SEARCH_MAX_RESULTS`.\n\n9. **Bot not invited to channel**\n   ```\n   slack search failed: not_in_channel\n   ```\n   → Invite the bot user to the channels you want to search.\n\n10. **Secrets Manager access denied**\n    ```\n    failed to get secret value from Secrets Manager: AccessDeniedException\n    ```\n    → Ensure the IAM role has `secretsmanager:GetSecretValue` permission for the target secret ARN.\n\n11. **Secrets Manager secret not found**\n    ```\n    failed to get secret value from Secrets Manager: ResourceNotFoundException\n    ```\n    → Verify that `SECRET_MANAGER_SECRET_ID` matches an existing secret name/ARN and `SECRET_MANAGER_REGION` is correct.\n\n### Debugging Methods\n\n```bash\n# Execute with detailed logs\nRAGent vectorize --dry-run\n\n# Check environment variables\nenv | grep AWS\n\n# Test MCP server connectivity\ncurl -X POST -H \"Content-Type: application/json\" \\\n  -d '{\"jsonrpc\":\"2.0\",\"method\":\"tools/list\",\"id\":\"test\"}' \\\n  http://localhost:8080/mcp\n```\n\n## OpenSearch RAG Configuration\n\n### Role Mapping for AWS OpenSearch\n\nWhen using AWS OpenSearch with IAM authentication, you need to configure role mapping to allow your IAM role to access the OpenSearch cluster.\n\n#### Check Current Role Mapping\n```bash\ncurl -u \"master_user:master_pass\" -X GET \\\n  \"https://your-opensearch-endpoint/_plugins/_security/api/rolesmapping/all_access\"\n```\n\n#### Map IAM Role to OpenSearch Role\n```bash\ncurl -u \"master_user:master_pass\" -X PUT \\\n  \"https://your-opensearch-endpoint/_plugins/_security/api/rolesmapping/all_access\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"backend_roles\": [\"arn:aws:iam::123456789012:role/your-iam-role\"],\n    \"hosts\": [],\n    \"users\": []\n  }'\n```\n\n#### Create Custom Role for RAG Operations\n```bash\n# Create a custom role with necessary permissions\ncurl -u \"master_user:master_pass\" -X PUT \\\n  \"https://your-opensearch-endpoint/_plugins/_security/api/roles/RAGent_role\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"cluster_permissions\": [\n      \"cluster:monitor/health\",\n      \"indices:data/read/search\"\n    ],\n    \"index_permissions\": [{\n      \"index_patterns\": [\"RAGent-*\"],\n      \"allowed_actions\": [\n        \"indices:data/read/search\",\n        \"indices:data/read/get\",\n        \"indices:data/write/index\",\n        \"indices:data/write/bulk\",\n        \"indices:admin/create\",\n        \"indices:admin/mapping/put\"\n      ]\n    }]\n  }'\n\n# Map IAM role to the custom role\ncurl -u \"master_user:master_pass\" -X PUT \\\n  \"https://your-opensearch-endpoint/_plugins/_security/api/rolesmapping/RAGent_role\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"backend_roles\": [\"arn:aws:iam::123456789012:role/your-iam-role\"],\n    \"hosts\": [],\n    \"users\": []\n  }'\n```\n\n### Hybrid Search Configuration\n\nFor optimal RAG performance, configure hybrid search with appropriate weights:\n\n- **General search**: BM25 weight: 0.5, Vector weight: 0.5\n- **Keyword-focused**: BM25 weight: 0.7, Vector weight: 0.3\n- **Semantic-focused**: BM25 weight: 0.3, Vector weight: 0.7\n\n#### Recommended Settings for Japanese Documents\n- BM25 Operator: \"or\" (default)\n- BM25 Minimum Should Match: \"2\" or \"70%\" for precision\n- Use Japanese NLP: true (enables kuromoji tokenizer)\n\n## Automated Setup (setup.sh)\n\nUse the interactive `setup.sh` to configure AWS OpenSearch security, create/mapping roles, create the target index, and grant IAM permissions for Bedrock and S3 Vectors. This script drives AWS CLI and signs OpenSearch Security API calls with SigV4.\n\nPrerequisites\n- AWS CLI v2 configured (credentials/profile with permission to update the domain and IAM)\n- OpenSearch domain reachable (either VPC endpoint directly, or local port‑forward to `https://localhost:9200` with Host/SNI set to the VPC endpoint)\n\nRun\n```bash\nbash setup.sh\n```\n\nWhat it asks\n- AWS Account ID, OpenSearch domain/region, endpoint usage (direct vs. localhost:9200), IAM role ARNs (RAG runtime, optional master/admin), index name, S3 Vectors bucket/index/region, Bedrock region and model IDs.\n\nWhat it does\n- Updates the domain access policy to allow specified IAM roles\n- (Optional) Sets AdvancedSecurity MasterUserARN\n- Creates/updates OpenSearch role `kibela_rag_role` with cluster health + CRUD/bulk on \u003cindex\u003e* and maps backend_roles to your IAM roles\n- Creates the index if missing with Japanese analyzers and `knn_vector` (1024, lucene, cosinesimil)\n- (Optional) Temporarily maps `all_access` to the RAG role for troubleshooting\n- Adds IAM inline policies to the RAG role for Bedrock InvokeModel and S3 Vectors bucket/index operations\n\nNotes\n- If you use local port‑forwarding, the script sets the Host header to the VPC endpoint so SigV4 validation works against `https://localhost:9200`.\n- The `all_access` mapping is optional and intended for short‑term troubleshooting; remove it after verification.\n\n## License\n\nFor license information, please refer to the LICENSE file in the repository.\n\n## MCP Server Integration\n\n### Claude Desktop Configuration\n\nAfter setting up the MCP server and completing authentication, add the server to your Claude Desktop configuration:\n\n```json\n{\n  \"mcpServers\": {\n    \"ragent\": {\n      \"command\": \"curl\",\n      \"args\": [\n        \"-X\", \"POST\",\n        \"-H\", \"Content-Type: application/json\",\n        \"-H\", \"Authorization: Bearer YOUR_JWT_TOKEN\",\n        \"-d\", \"@-\",\n        \"http://localhost:8080/mcp\"\n      ],\n      \"env\": {}\n    }\n  }\n}\n```\n\nSSE clients (e.g., `claude mcp add --transport sse ...`) must target the dedicated `/sse` endpoint instead of `/mcp`.\n\n### Available MCP Tools\n\n- **ragent-hybrid_search**: Execute hybrid search using BM25 and vector search\n  - Parameters: `query`, `max_results`, `bm25_weight`, `vector_weight`, `use_japanese_nlp`\n  - Returns: Structured search results with fused scores (hybrid BM25/vector) and references\n  - Score reference: see [doc/score.md](doc/score.md) for how the fused score is calculated and interpreted\n\n### Authentication Flow\n\n1. Start MCP server: `RAGent mcp-server --auth-method oidc`\n2. Visit authentication URL: `http://localhost:8080/login`\n3. Complete OAuth2 flow with your identity provider\n4. Copy the provided Claude Desktop configuration\n5. Add configuration to Claude Desktop settings\n\n![OIDC Authentication](doc/oidc.png)\n\n## Contributing\n\nWe welcome contributions to the project. Feel free to submit issues and pull requests.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fca-srg%2Fragent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fca-srg%2Fragent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fca-srg%2Fragent/lists"}