{"id":28411808,"url":"https://github.com/aruntemme/go-rag","last_synced_at":"2026-04-16T19:45:47.394Z","repository":{"id":295436195,"uuid":"990105468","full_name":"aruntemme/go-rag","owner":"aruntemme","description":"Advanced RAG System with Go featuring intelligent adaptive chunking, hierarchical document processing, semantic search, and flexible LLM integration","archived":false,"fork":false,"pushed_at":"2025-05-25T14:52:11.000Z","size":4520,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-03T07:01:08.551Z","etag":null,"topics":["golang","llm","localllm","openai","rag","sqlite-vec","sqlite3"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aruntemme.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-25T14:13:40.000Z","updated_at":"2025-05-28T16:42:27.000Z","dependencies_parsed_at":"2025-05-29T12:49:10.853Z","dependency_job_id":null,"html_url":"https://github.com/aruntemme/go-rag","commit_stats":null,"previous_names":["aruntemme/go-rag"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/aruntemme/go-rag","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aruntemme%2Fgo-rag","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aruntemme%2Fgo-rag/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aruntemme%2Fgo-rag/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aruntemme%2Fgo-rag/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aruntemme","download_url":"https://codeload.github.com/aruntemme/go-rag/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aruntemme%2Fgo-rag/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261628449,"owners_count":23186820,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","llm","localllm","openai","rag","sqlite-vec","sqlite3"],"created_at":"2025-06-02T19:15:33.636Z","updated_at":"2026-04-16T19:45:42.323Z","avatar_url":"https://github.com/aruntemme.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🤖 Advanced RAG System with Go\n\n[![Go Version](https://img.shields.io/badge/Go-1.19+-blue)](https://golang.org/)\n[![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)\n[![SQLite-vec](https://img.shields.io/badge/Vector%20DB-SQLite--vec-orange)](https://github.com/asg017/sqlite-vec)\n\nA sophisticated **Retrieval Augmented Generation (RAG)** system built with Go, featuring intelligent adaptive chunking, hierarchical document processing, semantic search, flexible LLM integration, and command-line configuration management.\n\n## ✨ Key Features\n\n### 🧠 Intelligent Adaptive Chunking System\n- **Document-Size Aware**: Automatically adapts chunking strategy based on document characteristics\n- **5-Tier Classification**: VerySmall → Small → Medium → Large → VeryLarge with tailored strategies\n- **Context Preservation**: Smart thresholds prevent fragmentation while maintaining semantic coherence\n- **50% Better Performance**: Fewer chunks with 100% better context preservation\n\n### 🔍 Advanced Search \u0026 Retrieval\n- **Search-Only Endpoint**: Pure retrieval without LLM overhead (500x faster)\n- **Full RAG Pipeline**: Complete question-answering with context generation\n- **Semantic Thresholding**: Filter results by similarity scores\n- **Metadata Filtering**: Precise targeting with custom filters\n- **Query Expansion**: Automatic synonym and related term expansion\n\n### 📊 Multiple Chunking Strategies\n- **Structural Chunking**: Intelligent section and paragraph detection\n- **Fixed-Size Chunking**: Traditional character-based with overlap\n- **Semantic Chunking**: Content-aware based on meaning\n- **Sentence Window**: Overlapping sentence-based chunks\n- **Parent-Child Relationships**: Hierarchical organization for multi-level context\n\n### 🚀 Performance \u0026 Flexibility\n- **SQLite-vec Integration**: High-performance vector storage\n- **Concurrent Processing**: Efficient batch embedding generation\n- **Dimension Auto-Detection**: Automatic model compatibility\n- **RESTful API**: Clean, well-documented endpoints\n- **External LLM Support**: Use any OpenAI-compatible service\n- **Command-Line Interface**: Flexible configuration with CLI arguments\n- **Cross-Platform Builds**: Single build script for all platforms\n\n## 🏗️ Architecture\n\n```\n┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐\n│   Documents     │───▶│ Adaptive Chunking │───▶│  Vector Store   │\n│                 │    │     System       │    │  (SQLite-vec)   │\n└─────────────────┘    └──────────────────┘    └─────────────────┘\n                                                        │\n┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐\n│   Search API    │◀───│   Embedding      │◀───│   Raw Search    │\n│  (/search)      │    │    Service       │    │    Results      │\n└─────────────────┘    └──────────────────┘    └─────────────────┘\n        │                       │\n        ▼                       ▼\n┌─────────────────┐    ┌──────────────────┐\n│  External LLM   │    │   Full RAG API   │\n│   Processing    │    │    (/query)     │\n└─────────────────┘    └──────────────────┘\n```\n\n## 📋 Prerequisites\n\n- **Go 1.19+**\n- **OpenAI-compatible API Server** (LlamaCPP, OpenAI, Ollama, or any v1/embeddings endpoint)\n- **Embedding Model** (Nomic, OpenAI, or compatible)\n\n## 🚀 Quick Start\n\n### 1. Clone \u0026 Install\n```bash\ngit clone https://github.com/aruntemme/go-rag.git\ncd go-rag\ngo mod tidy\n```\n\n### 2. Build (Optional but Recommended)\n```bash\n# Quick build for current platform\ngo build -ldflags=\"-s -w\" -o rag-server .\n\n# Or build for all platforms\nchmod +x build.sh \u0026\u0026 ./build.sh\n```\n\n### 3. Configure\nCreate `config.json`:\n```json\n{\n  \"server_port\": \"8080\",\n  \"llamacpp_base_url\": \"http://localhost:8091/v1\",\n  \"embedding_model\": \"nomic-embed-text-v1.5\",\n  \"chat_model\": \"qwen3:8b\", \n  \"vector_db_path\": \"./rag_database.db\",\n  \"default_top_k\": 3\n}\n```\n\n### 4. Start Embedding Server\n```bash\n# Example with llama.cpp\n./server -m your-model.gguf --host 0.0.0.0 --port 8091\n\n# Or use OpenAI API\n# Set OPENAI_API_KEY and use https://api.openai.com/v1\n\n# Or use Ollama\nollama serve\n```\n\n### 5. Run the Application\n\n#### Development Mode\n```bash\ngo run main.go\n```\n\n#### Build \u0026 Run (Recommended)\n```bash\n# Build optimized executable\ngo build -ldflags=\"-s -w\" -o rag-server .\n\n# Run with default config\n./rag-server\n\n# Run with custom config\n./rag-server -config=production.json\n\n# Show help and options\n./rag-server -help\n\n# Show version\n./rag-server -version\n```\n\n🎉 Server starts on `http://localhost:8080` (or configured port)\n\n## 📚 Usage Examples\n\n### Basic Document Upload \u0026 Search\n```bash\n# 1. Create a collection\ncurl -X POST http://localhost:8080/api/v1/collections \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"name\": \"my_docs\", \"description\": \"My documents\"}'\n\n# 2. Add a document (adaptive chunking automatically applied)\ncurl -X POST http://localhost:8080/api/v1/documents \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"collection_name\": \"my_docs\",\n    \"content\": \"Your document content here...\",\n    \"source\": \"document.txt\"\n  }'\n\n# 3. Search without LLM (fast retrieval)\ncurl -X POST http://localhost:8080/api/v1/search \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"collection_name\": \"my_docs\",\n    \"query\": \"What is this about?\",\n    \"top_k\": 5\n  }'\n\n# 4. Full RAG query (with answer generation)\ncurl -X POST http://localhost:8080/api/v1/query \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"collection_name\": \"my_docs\",\n    \"query\": \"What is this about?\",\n    \"top_k\": 5\n  }'\n```\n\n### Advanced Search Features\n```bash\n# Search with semantic filtering and metadata\ncurl -X POST http://localhost:8080/api/v1/search \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"collection_name\": \"my_docs\",\n    \"query\": \"machine learning experience\",\n    \"top_k\": 10,\n    \"semantic_threshold\": 0.3,\n    \"metadata_filters\": {\n      \"section\": \"experience\",\n      \"chunk_type\": \"job_entry\"\n    }\n  }'\n```\n\n## 🔌 API Endpoints\n\n| Endpoint | Method | Purpose | Speed |\n|----------|--------|---------|-------|\n| `/health` | GET | Health check | ⚡ Instant |\n| `/api/v1/collections` | POST/GET/DELETE | Manage collections | ⚡ Fast |\n| `/api/v1/documents` | POST/GET/DELETE | Manage documents | 🐢 Processing |\n| `/api/v1/search` | POST | **Retrieval only** | ⚡ Fast |\n| `/api/v1/query` | POST | **Full RAG** | 🐢 LLM dependent |\n| `/api/v1/analyze` | POST | Detailed analysis | 🐢 LLM dependent |\n\n\u003e 📖 **Full API documentation**: [API_REFERENCE.md](API_REFERENCE.md)\n\n## 🧠 Adaptive Chunking System\n\nOur intelligent chunking system automatically optimizes based on document characteristics:\n\n### Document Size Categories\n- **VerySmall** (\u003c1KB): Single chunk or max 2-3 chunks\n- **Small** (1-3KB): 3-5 meaningful chunks, 400+ char minimum\n- **Medium** (3-10KB): Structural/semantic chunking\n- **Large** (10-50KB): Hierarchical parent-child chunks\n- **VeryLarge** (50KB+): Aggressive hierarchical chunking\n\n### Performance Benefits\n- **50% Fewer Chunks**: Reduces noise and improves relevance\n- **100% Better Context**: Maintains semantic coherence\n- **Universal Compatibility**: Works with any document type\n- **Automatic Optimization**: No manual tuning required\n\n\u003e 📖 **Detailed explanation**: [ADAPTIVE_CHUNKING.md](ADAPTIVE_CHUNKING.md)\n\n## 🔍 Search vs Query Endpoints\n\n### `/api/v1/search` - Pure Retrieval\n```json\n{\n  \"chunks_found\": 3,\n  \"chunks\": [/* detailed chunk data */],\n  \"context\": \"ready-to-use context string\",\n  \"similarity_scores\": [0.95, 0.87, 0.82],\n  \"processing_time\": 0.056\n}\n```\n**Perfect for**: External LLM processing, custom pipelines, debugging\n\n### `/api/v1/query` - Full RAG\n```json\n{\n  \"answer\": \"Generated answer based on retrieved context\",\n  \"retrieved_context\": [\"context chunks\"],\n  \"enhanced_chunks\": [/* chunks with metadata */],\n  \"processing_time\": 2.34\n}\n```\n**Perfect for**: Complete question-answering, integrated solutions\n\n\u003e 📖 **Search endpoint guide**: [SEARCH_ENDPOINT.md](SEARCH_ENDPOINT.md)\n\n## 🏃‍♂️ Performance\n\n| Operation | Time | Description |\n|-----------|------|-------------|\n| Document Upload | ~1-5s | Depends on size \u0026 chunking |\n| Search Query | ~0.05s | Pure retrieval |\n| Full RAG Query | ~2-30s | Includes LLM generation |\n| Embedding Batch | ~0.1s/chunk | Concurrent processing |\n\n## 🛠️ Development\n\n### Project Structure\n```\ngo-rag/\n├── main.go              # Application entry point\n├── config.json          # Configuration file\n├── go.mod \u0026 go.sum      # Go dependencies\n├── api/                 # HTTP handlers and routing\n├── core/                # Core business logic\n├── models/              # Data structures\n├── config/              # Configuration management\n└── docs/                # Documentation\n```\n\n### Key Components\n- **`core/document_processor.go`**: Adaptive chunking engine\n- **`core/vector_db.go`**: SQLite-vec integration\n- **`core/rag_service.go`**: RAG pipeline orchestration\n- **`api/handlers.go`**: HTTP API handlers\n\n## 🚀 Building \u0026 Deployment\n\n### Command-Line Options\nThe application supports flexible configuration through command-line arguments:\n\n```bash\nUsage: ./rag-server [options]\n\nOptions:\n  -config string\n        Path to configuration file (default \"config.json\")\n  -help\n        Show help information\n  -version\n        Show version information\n\nExamples:\n  ./rag-server                           # Use default config.json\n  ./rag-server -config=prod.json         # Use custom config file\n  ./rag-server -config=/path/to/config   # Use absolute path\n  ./rag-server -help                     # Show help\n  ./rag-server -version                  # Show version\n```\n\n### Build Options\n\n#### Single Platform Build\n```bash\n# Development build\ngo build -o rag-server .\n\n# Optimized production build\ngo build -ldflags=\"-s -w\" -o rag-server .\n```\n\n#### Cross-Platform Build\n```bash\n# Use provided build script for all platforms\nchmod +x build.sh\n./build.sh\n\n# Manual cross-compilation (note: CGO required for sqlite-vec)\nCGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build -ldflags=\"-s -w\" -o rag-server-linux .\nCGO_ENABLED=1 GOOS=windows GOARCH=amd64 go build -ldflags=\"-s -w\" -o rag-server.exe .\nCGO_ENABLED=1 GOOS=darwin GOARCH=arm64 go build -ldflags=\"-s -w\" -o rag-server-macos-arm64 .\n```\n\n\u003e **⚠️ Note**: Cross-platform builds require appropriate CGO toolchains for each target platform due to sqlite-vec dependency. Build script will attempt all platforms but may fail for platforms without proper CGO setup.\n\n### Deployment Configurations\n\n#### Development\n```json\n{\n  \"server_port\": \"8080\",\n  \"llamacpp_base_url\": \"http://localhost:8091/v1\",\n  \"embedding_model\": \"nomic-embed-text-v1.5\",\n  \"chat_model\": \"qwen3:8b\",\n  \"vector_db_path\": \"./rag_database.db\",\n  \"default_top_k\": 3\n}\n```\n\n#### Production\n```json\n{\n  \"server_port\": \"80\",\n  \"llamacpp_base_url\": \"https://your-llm-api.com/v1\",\n  \"embedding_model\": \"text-embedding-ada-002\",\n  \"chat_model\": \"gpt-4\",\n  \"vector_db_path\": \"/data/rag_database.db\",\n  \"default_top_k\": 5\n}\n```\n\n### Docker Deployment (Optional)\n```dockerfile\nFROM golang:1.23-alpine AS builder\nRUN apk add --no-cache gcc musl-dev sqlite-dev\nWORKDIR /app\nCOPY . .\nRUN CGO_ENABLED=1 go build -ldflags=\"-s -w\" -o rag-server .\n\nFROM alpine:latest\nRUN apk --no-cache add ca-certificates sqlite\nWORKDIR /root/\nCOPY --from=builder /app/rag-server .\nCOPY configs/ ./configs/\nEXPOSE 8080\nCMD [\"./rag-server\", \"-config=configs/production.json\"]\n```\n\n```bash\n# Build and run with custom config\ndocker build -t rag-server .\ndocker run -p 8080:8080 -v $(pwd)/data:/data rag-server ./rag-server -config=/data/custom.json\n```\n\n### Environment-Specific Deployments\n```bash\n# Development\n./rag-server -config=configs/dev.json\n\n# Staging\n./rag-server -config=configs/staging.json\n\n# Production\n./rag-server -config=configs/production.json\n```\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## 🙏 Acknowledgments\n\n- [SQLite-vec](https://github.com/asg017/sqlite-vec) for high-performance vector storage\n- [Gin](https://github.com/gin-gonic/gin) for the web framework\n- [LlamaCPP](https://github.com/ggerganov/llama.cpp) for embedding and LLM services\n\n## 📞 Support\n\n- 📖 **Documentation**: Check [API_REFERENCE.md](API_REFERENCE.md)\n- 🐛 **Issues**: [GitHub Issues](https://github.com/aruntemme/go-rag/issues)\n- 💬 **Discussions**: [GitHub Discussions](https://github.com/aruntemme/go-rag/discussions)\n\n---\n\nBuilt with ❤️ using Go and modern RAG techniques ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faruntemme%2Fgo-rag","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faruntemme%2Fgo-rag","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faruntemme%2Fgo-rag/lists"}