https://github.com/byerlikaya/smartrag
SmartRAG is a production-ready .NET 9.0 library that provides a complete Retrieval-Augmented Generation (RAG) solution. Features include multi-provider AI support (OpenAI, Anthropic, Gemini), enterprise vector storage (Qdrant, Redis, SQLite), and intelligent document processing (PDF, Word, Text).
https://github.com/byerlikaya/smartrag
ai anthropic csharp document-processing document-qa dotnet enterprise-ai gemini llm machine-learning natural-language-processing openai pdf-parser qdrant rag redis retrieval-augmented-generation vector-database word-parser
Last synced: about 2 months ago
JSON representation
SmartRAG is a production-ready .NET 9.0 library that provides a complete Retrieval-Augmented Generation (RAG) solution. Features include multi-provider AI support (OpenAI, Anthropic, Gemini), enterprise vector storage (Qdrant, Redis, SQLite), and intelligent document processing (PDF, Word, Text).
- Host: GitHub
- URL: https://github.com/byerlikaya/smartrag
- Owner: byerlikaya
- License: mit
- Created: 2025-08-15T18:38:32.000Z (2 months ago)
- Default Branch: main
- Last Pushed: 2025-08-22T14:22:40.000Z (about 2 months ago)
- Last Synced: 2025-08-23T00:40:03.683Z (about 2 months ago)
- Topics: ai, anthropic, csharp, document-processing, document-qa, dotnet, enterprise-ai, gemini, llm, machine-learning, natural-language-processing, openai, pdf-parser, qdrant, rag, redis, retrieval-augmented-generation, vector-database, word-parser
- Language: C#
- Homepage: https://www.nuget.org/packages/SmartRAG
- Size: 345 KB
- Stars: 0
- Watchers: 0
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# π SmartRAG - Enterprise-Grade RAG Solution
[](https://github.com/byerlikaya/SmartRAG/actions)
[](https://www.nuget.org/packages/SmartRAG)
[](https://www.nuget.org/packages/SmartRAG)
[](https://dotnet.microsoft.com/download/dotnet/9.0)
[](LICENSE)SmartRAG is a **production-ready** .NET 9.0 library that provides a complete **Retrieval-Augmented Generation (RAG)** solution for building **AI-powered question answering systems**. Upload your documents, ask questions in natural language, and get intelligent answers based on your content - all with enterprise-grade AI providers and storage options.
## β¨ Key Highlights
- π― **AI Question Answering**: Ask questions about your documents and get intelligent, contextual answers
- π§ **Smart Query Intent Detection**: Automatically distinguishes between general conversation and document search queries
- π **Language-Agnostic**: Works with any language without hardcoded patterns or keywords
- π€ **Universal AI Support**: 5 dedicated providers + CustomProvider for unlimited AI APIs
- π’ **Enterprise Storage**: Vector databases, Redis, SQL, FileSystem with advanced configurations
- π§ **Advanced RAG Pipeline**: Smart chunking, semantic retrieval, AI-powered answer generation
- β‘ **Lightning Fast**: Optimized vector search with context-aware answer synthesis
- π **Plug & Play**: Single-line integration with dependency injection
- π **Multi-Format**: PDF, Word, Excel, text files with intelligent parsing
- π― **Enhanced Semantic Search**: Advanced hybrid scoring with 80% semantic + 20% keyword relevance
- π **Smart Document Chunking**: Word boundary validation and optimal break points for context preservation
- β **Enterprise Grade**: Zero Warnings Policy, SOLID principles, comprehensive logging, XML documentation## π― What Makes SmartRAG Special
### π **Complete RAG Workflow**
```
π Document Upload β π Smart Chunking β π§ AI Embeddings β πΎ Vector Storage
β
πββοΈ User Question β π― Intent Detection β π Find Relevant Chunks β π€ AI Answer Generation β β¨ Smart Response
```### π **Production Features**
- **Smart Chunking**: Maintains context continuity between document segments with word boundary validation
- **Intelligent Query Routing**: Automatically routes general conversation to AI chat, document queries to RAG search
- **Language-Agnostic Design**: No hardcoded language patterns - works globally with any language
- **Multiple Storage Options**: From in-memory to enterprise vector databases
- **AI Provider Flexibility**: Switch between providers without code changes
- **Document Intelligence**: Advanced parsing for PDF, Word, Excel, and text formats
- **Configuration-First**: Environment-based configuration with sensible defaults
- **Dependency Injection**: Full DI container integration
- **Enhanced Semantic Search**: Advanced hybrid scoring combining semantic similarity and keyword relevance
- **VoyageAI Integration**: High-quality embeddings for Anthropic Claude models
- **Enterprise Architecture**: Zero Warnings Policy, SOLID/DRY principles, comprehensive XML documentation
- **Production Ready**: Thread-safe operations, centralized logging, proper error handling## π§ Smart Query Intent Detection
SmartRAG automatically detects whether your query is a general conversation or a document search request:
### **General Conversation** (Direct AI Chat)
- β **"How are you?"** β Direct AI response
- β **"What's the weather like?"** β Direct AI response
- β **"Tell me a joke"** β Direct AI response
- β **"Emin misin?"** β Direct AI response (Turkish)
- β **"δ½ ε₯½εοΌ"** β Direct AI response (Chinese)### **Document Search** (RAG with your documents)
- π **"What are the main benefits in the contract?"** β Searches your documents
- π **"ΓalΔ±Εan maaΕ bilgileri nedir?"** β Searches your documents (Turkish)
- π **"2025年第δΈε£εΊ¦ζ₯εηδΈ»θ¦εη°ζ―δ»δΉοΌ"** β Searches your documents (Chinese)
- π **"Show me the employee salary data"** β Searches your documents**How it works:** The system analyzes query structure (numbers, dates, formats, length) to determine intent without any hardcoded language patterns.
## π― Enhanced Semantic Search & Chunking
### **π§ Advanced Semantic Search**
SmartRAG uses a sophisticated **hybrid scoring system** that combines multiple relevance factors:```csharp
// Hybrid Scoring Algorithm (80% Semantic + 20% Keyword)
var hybridScore = (enhancedSemanticScore * 0.8) + (keywordScore * 0.2);// Enhanced Semantic Similarity
var enhancedSemanticScore = await _semanticSearchService
.CalculateEnhancedSemanticSimilarityAsync(query, chunk.Content);// Keyword Relevance
var keywordScore = CalculateKeywordRelevanceScore(query, chunk.Content);
```**Scoring Components:**
- **Semantic Similarity (80%)**: Advanced text analysis with context awareness
- **Keyword Relevance (20%)**: Traditional text matching and frequency analysis
- **Contextual Enhancement**: Semantic coherence and contextual keyword detection
- **Domain Independence**: Generic scoring without hardcoded domain patterns### **π Smart Document Chunking**
Advanced chunking algorithm that preserves context and maintains word integrity:```csharp
// Word Boundary Validation
private static int ValidateWordBoundary(string content, int breakPoint)
{
// Ensures chunks don't cut words in the middle
// Finds optimal break points at sentence, paragraph, or word boundaries
// Maintains semantic continuity between chunks
}// Optimal Break Point Detection
private static int FindOptimalBreakPoint(string content, int startIndex, int maxChunkSize)
{
// 1. Sentence boundaries (preferred)
// 2. Paragraph boundaries (secondary)
// 3. Word boundaries (fallback)
// 4. Character boundaries (last resort)
}
```**Chunking Features:**
- **Word Boundary Protection**: Never cuts words in the middle
- **Context Preservation**: Maintains semantic continuity between chunks
- **Optimal Break Points**: Intelligent selection of chunk boundaries
- **Overlap Management**: Configurable overlap for context continuity
- **Size Optimization**: Dynamic chunk sizing based on content structure## π¦ Installation
### NuGet Package Manager
```bash
Install-Package SmartRAG
```### .NET CLI
```bash
dotnet add package SmartRAG
```### PackageReference
```xml```
## π Supported Document Formats
SmartRAG supports a wide range of document formats with intelligent parsing and text extraction:
### **π Excel Files (.xlsx, .xls)**
- **Advanced Parsing**: Extracts text from all worksheets and cells
- **Structured Data**: Preserves table structure with tab-separated values
- **Worksheet Names**: Includes worksheet names for context
- **Cell Content**: Extracts all non-empty cell values
- **Format Preservation**: Maintains data organization for better context### **π Word Documents (.docx, .doc)**
- **Rich Text Extraction**: Preserves formatting and structure
- **Table Support**: Extracts content from tables and lists
- **Paragraph Handling**: Maintains paragraph breaks and flow
- **Metadata Preservation**: Keeps document structure intact### **π PDF Documents (.pdf)**
- **Multi-Page Support**: Processes all pages with text extraction
- **Layout Preservation**: Maintains document structure and flow
- **Text Quality**: High-quality text extraction for analysis
- **Page Separation**: Clear page boundaries for context### **π Text Files (.txt, .md, .json, .xml, .csv, .html, .htm)**
- **Universal Support**: Handles all text-based formats
- **Encoding Detection**: Automatic UTF-8 and encoding detection
- **Structure Preservation**: Maintains original formatting
- **Fast Processing**: Optimized for text-based content### **π Content Type Support**
SmartRAG automatically detects file types using both file extensions and MIME content types:
- **Excel**: `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`, `application/vnd.ms-excel`
- **Word**: `application/vnd.openxmlformats-officedocument.wordprocessingml.document`, `application/msword`
- **PDF**: `application/pdf`
- **Text**: `text/*`, `application/json`, `application/xml`, `application/csv`## π Quick Start
### 1. **Development Setup**
```bash
# Clone the repository
git clone https://github.com/byerlikaya/SmartRAG.git
cd SmartRAG# Copy development configuration template
cp examples/WebAPI/appsettings.Development.template.json examples/WebAPI/appsettings.Development.json# Edit appsettings.Development.json with your API keys
# - OpenAI API Key
# - Azure OpenAI credentials
# - Database connection strings
```### 2. **Basic Setup**
```csharp
using SmartRAG.Extensions;
using SmartRAG.Enums;var builder = WebApplication.CreateBuilder(args);
// Add SmartRAG with minimal configuration
builder.Services.UseSmartRAG(builder.Configuration,
storageProvider: StorageProvider.InMemory, // Start simple
aiProvider: AIProvider.OpenAI // Your preferred AI
);var app = builder.Build();
```### 3. **Upload Documents**
```csharp
public class DocumentController : ControllerBase
{
private readonly IDocumentService _documentService;[HttpPost("upload")]
public async Task Upload(IFormFile file)
{
var document = await _documentService.UploadDocumentAsync(
file.OpenReadStream(),
file.FileName,
file.ContentType,
"user-123"
);
return Ok(document);
}
}
```### 4. **AI-Powered Question Answering**
```csharp
public class QAController : ControllerBase
{
private readonly IDocumentService _documentService;[HttpPost("ask")]
public async Task AskQuestion([FromBody] QuestionRequest request)
{
// User asks: "What are the main benefits mentioned in the contract?"
var response = await _documentService.GenerateRagAnswerAsync(
request.Question,
maxResults: 5
);
// Returns intelligent answer based on document content
return Ok(response);
}
}
```### 5. **Configuration**
β οΈ **Security Note**: Never commit real API keys! Use `appsettings.Development.json` for local development.
```bash
# Copy template and add your real keys
cp examples/WebAPI/appsettings.json examples/WebAPI/appsettings.Development.json
```**appsettings.Development.json** (your real keys):
```json
{
"AI": {
"OpenAI": {
"ApiKey": "sk-proj-YOUR_REAL_KEY",
"Model": "gpt-4",
"EmbeddingModel": "text-embedding-ada-002"
},
"Anthropic": {
"ApiKey": "sk-ant-YOUR_REAL_KEY",
"Model": "claude-3.5-sonnet",
"EmbeddingApiKey": "voyage-YOUR_REAL_KEY",
"EmbeddingModel": "voyage-large-2"
}
},
"Storage": {
"InMemory": {
"MaxDocuments": 1000
}
}
}
```π **[Complete Configuration Guide](docs/configuration.md) | [π§ Troubleshooting Guide](docs/troubleshooting.md)**
### π **Important Note for Anthropic Users**
**Anthropic Claude models require a separate VoyageAI API key for embeddings:**
- **Why?** Anthropic doesn't provide embedding models, so we use VoyageAI's high-quality embeddings
- **Official Documentation:** [Anthropic Embeddings Guide](https://docs.anthropic.com/en/docs/build-with-claude/embeddings#how-to-get-embeddings-with-anthropic)
- **Get API Key:** [VoyageAI API Keys](https://console.voyageai.com/)
- **Models:** `voyage-large-2` (recommended), `voyage-code-2`, `voyage-01`
- **Documentation:** [VoyageAI Embeddings API](https://docs.voyageai.com/embeddings/)## π€ AI Providers - Universal Support
### π― **Dedicated Providers** (Optimized & Battle-Tested)
| Provider | Capabilities | Special Features |
|----------|-------------|------------------|
| **π€ OpenAI** | β Latest GPT models
β Advanced embeddings | Industry standard, reliable, extensive model family |
| **π§ Anthropic** | β Claude family models
β VoyageAI embeddings | Safety-focused, constitutional AI, long context, requires separate VoyageAI API key |
| **π Google Gemini** | β Gemini models
β Multimodal embeddings | Multimodal support, latest Google AI innovations |
| **βοΈ Azure OpenAI** | β Enterprise GPT models
β Enterprise embeddings | GDPR compliant, enterprise security, SLA support |### π οΈ **CustomProvider** - Universal API Support
**One provider to rule them all!** Connect to any OpenAI-compatible API:```json
{
"AI": {
"Custom": {
"ApiKey": "your-api-key",
"Endpoint": "https://api.openrouter.ai/v1/chat/completions",
"Model": "anthropic/claude-3.5-sonnet",
"EmbeddingModel": "text-embedding-ada-002"
}
}
}
```**Supported APIs via CustomProvider:**
- π **OpenRouter** - Access 100+ models
- β‘ **Groq** - Lightning-fast inference
- π **Together AI** - Open source models
- π **Perplexity** - Search + AI
- π«π· **Mistral AI** - European AI leader
- π₯ **Fireworks AI** - Ultra-fast inference
- π¦ **Ollama** - Local models
- π **LM Studio** - Local AI playground
- π οΈ **Any OpenAI-compatible API**## ποΈ Storage Solutions - Enterprise Grade
### π― **Vector Databases**
```json
{
"Storage": {
"Qdrant": {
"Host": "your-qdrant-host.com",
"ApiKey": "your-api-key",
"CollectionName": "documents",
"VectorSize": 1536
},
"Redis": {
"ConnectionString": "localhost:6379",
"KeyPrefix": "smartrag:",
"Database": 0
}
}
}
```### π’ **Traditional Databases**
```json
{
"Storage": {
"Sqlite": {
"DatabasePath": "smartrag.db",
"EnableForeignKeys": true
},
"FileSystem": {
"FileSystemPath": "Documents"
}
}
}
```### β‘ **Development**
```json
{
"Storage": {
"InMemory": {
"MaxDocuments": 1000
}
}
}
```## π Document Processing
### **Supported Formats**
- **π PDF**: Advanced text extraction with iText7
- **π Word**: .docx and .doc support with OpenXML
- **π Text**: .txt, .md, .json, .xml, .csv, .html
- **π€ Plain Text**: UTF-8 encoding with BOM detection### **Smart Document Parsing**
```csharp
// Automatic format detection and parsing
var document = await documentService.UploadDocumentAsync(
fileStream,
"contract.pdf", // Automatically detects PDF
"application/pdf",
"legal-team"
);// Smart chunking with overlap for context preservation
var chunks = document.Chunks; // Automatically chunked with smart boundaries
```### **Advanced Chunking Options**
```csharp
services.AddSmartRAG(configuration, options =>
{
options.MaxChunkSize = 1000; // Maximum chunk size
options.MinChunkSize = 100; // Minimum chunk size
options.ChunkOverlap = 200; // Overlap between chunks
options.SemanticSearchThreshold = 0.3; // Similarity threshold
});
```## π§ Advanced Configuration
### **Complete Configuration Example**
```json
{
"AI": {
"OpenAI": {
"ApiKey": "sk-...",
"Endpoint": "https://api.openai.com/v1",
"Model": "gpt-4",
"EmbeddingModel": "text-embedding-ada-002",
"MaxTokens": 4096,
"Temperature": 0.7
},
"Anthropic": {
"ApiKey": "sk-ant-...",
"Model": "claude-3.5-sonnet",
"MaxTokens": 4096,
"Temperature": 0.3,
"EmbeddingApiKey": "voyage-...",
"EmbeddingModel": "voyage-large-2"
}
},
"Storage": {
"Qdrant": {
"Host": "localhost:6334",
"UseHttps": false,
"CollectionName": "smartrag_docs",
"VectorSize": 1536,
"DistanceMetric": "Cosine"
},
"Redis": {
"ConnectionString": "localhost:6379",
"Password": "",
"Database": 0,
"KeyPrefix": "smartrag:",
"ConnectionTimeout": 30,
"EnableSsl": false
}
}
}
```### **Runtime Provider Switching**
```csharp
services.AddSmartRAG(configuration, options =>
{
options.AIProvider = AIProvider.OpenAI;
options.StorageProvider = StorageProvider.Qdrant;
options.EnableFallbackProviders = true;
options.FallbackProviders = [AIProvider.Anthropic, AIProvider.Gemini];
});
```## ποΈ Architecture
SmartRAG follows clean architecture principles with clear separation of concerns:
```
βββββββββββββββββββ ββββββββββββββββββββ βββββββββββββββββββ
β API Layer β β Service Layer β β Repository Layerβ
β β β β β β
β β’ Controllers βββββΆβ β’ DocumentServiceβββββΆβ β’ Redis Repo β
β β’ DTOs β β β’ AIService β β β’ Qdrant Repo β
β β’ Validation β β β’ ParserService β β β’ SQLite Repo β
βββββββββββββββββββ ββββββββββββββββββββ βββββββββββββββββββ
β
βΌ
ββββββββββββββββββββ
β AI Providers β
β β
β β’ OpenAI β
β β’ Anthropic β
β β’ Gemini β
β β’ CustomProvider β
ββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββ
β Semantic Search β
β β
β β’ Hybrid Scoring β
β β’ Context Aware β
β β’ Word Boundary β
ββββββββββββββββββββ
```### **Key Components**
- **π DocumentService**: Main orchestrator for document operations
- **π€ AIService**: Handles AI provider interactions and embeddings
- **π DocumentParserService**: Multi-format document parsing with smart chunking
- **π SemanticSearchService**: Advanced semantic search with hybrid scoring
- **π Factories**: Provider instantiation and configuration
- **π Repositories**: Storage abstraction layer
- **π§ Extensions**: Dependency injection configuration## π¨ API Examples
### **Document Management**
```bash
# Upload document
curl -X POST "http://localhost:5000/api/documents/upload" \
-F "file=@research-paper.pdf"# Get document
curl "http://localhost:5000/api/documents/{document-id}"# Delete document
curl -X DELETE "http://localhost:5000/api/documents/{document-id}"# List all documents
curl "http://localhost:5000/api/documents/search"
```### **AI Question Answering & Chat**
SmartRAG handles both document search and general conversation automatically:
```bash
# Ask questions about your documents (RAG mode)
curl -X POST "http://localhost:5000/api/search/search" \
-H "Content-Type: application/json" \
-d '{
"query": "What are the main risks mentioned in the financial report?",
"maxResults": 5
}'# General conversation (Direct AI chat mode)
curl -X POST "http://localhost:5000/api/search/search" \
-H "Content-Type: application/json" \
-d '{
"query": "How are you today?",
"maxResults": 1
}'
```**Document Search Response Example:**
```json
{
"query": "What are the main risks mentioned in the financial report?",
"answer": "Based on the financial documents, the main risks identified include: 1) Market volatility affecting revenue projections, 2) Regulatory changes in the European market, 3) Currency exchange fluctuations, and 4) Supply chain disruptions. The report emphasizes that market volatility poses the highest risk with potential 15-20% impact on quarterly earnings...",
"sources": [
{
"documentId": "doc-456",
"fileName": "Q3-financial-report.pdf",
"chunkContent": "Market volatility remains our primary concern, with projected impact of 15-20% on quarterly earnings...",
"relevanceScore": 0.94
}
],
"searchedAt": "2025-08-16T14:57:06.2312433Z",
"configuration": {
"aiProvider": "Anthropic",
"storageProvider": "Redis",
"model": "Claude + VoyageAI"
}
}
```**General Chat Response Example:**
```json
{
"query": "How are you today?",
"answer": "I'm doing well, thank you for asking! I'm here to help you with any questions you might have about your documents or just general conversation. How can I assist you today?",
"sources": [],
"searchedAt": "2025-08-16T14:57:06.2312433Z",
"configuration": {
"aiProvider": "Anthropic",
"storageProvider": "Redis",
"model": "Claude + VoyageAI"
}
}
```## π Performance & Scaling
### **Benchmarks**
- **Document Upload**: ~500ms for 100KB file, ~1-2s for 1MB file
- **Semantic Search**: ~200ms for simple queries, ~500ms for complex queries
- **AI Response**: ~2-5s for 5 sources, ~3-8s for 10 sources
- **Memory Usage**: ~50MB base + documents, ~100MB with Redis cache
- **Enhanced Chunking**: ~300ms for 10KB document with smart boundary detection
- **Hybrid Scoring**: ~150ms for semantic + keyword relevance calculation### **Scaling Tips**
- Use **Redis** or **Qdrant** for production workloads
- Enable **connection pooling** for database connections
- Implement **caching** for frequently accessed documents
- Use **background services** for bulk document processing
- Optimize **chunk sizes** based on your content type
- Use **semantic search threshold** to filter low-relevance results## π οΈ Development
### **Building from Source**
```bash
git clone https://github.com/byerlikaya/SmartRAG.git
cd SmartRAG
dotnet restore
dotnet build
dotnet test
```### **Running the Sample API**
```bash
cd examples/WebAPI
dotnet run
```Browse to `https://localhost:5001/scalar/v1` for interactive API documentation.
## π€ Contributing
We welcome contributions!
### **Development Setup**
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request## π What's New
### **Latest Release (v1.0.3)**
- π§ **Smart Query Intent Detection** - Automatically routes queries to chat vs document search
- π **Language-Agnostic Design** - Removed all hardcoded language patterns
- π **Enhanced Search Relevance** - Improved name detection and content scoring
- π€ **Unicode Normalization** - Fixed special character handling issues
- β‘ **Rate Limiting & Retry Logic** - Robust API handling with exponential backoff
- π **VoyageAI Integration** - Anthropic embedding support
- π **Enhanced Documentation** - Official documentation links
- π§Ή **Configuration Cleanup** - Removed unnecessary fields
- π― **Project Simplification** - Streamlined for better performance### **Architecture & Code Quality**
- π― **Enhanced Semantic Search** - Advanced hybrid scoring (80% semantic + 20% keyword)
- π **Smart Document Chunking** - Word boundary validation and optimal break points
- π§ **SemanticSearchService** - Dedicated service for semantic relevance scoring
- βοΈ **Configuration Management** - User settings take absolute priority
- π§ **Enterprise Error Handling** - Comprehensive logging and retry mechanisms
- π **Performance Optimizations** - Faster chunking and search algorithms
- β **Code Quality** - SOLID principles, zero warnings, comprehensive documentation## π Resources
- **π§ [Contact & Support](mailto:b.yerlikaya@outlook.com)**
- **πΌ [LinkedIn](https://www.linkedin.com/in/barisyerlikaya)**
- **π [GitHub Profile](https://github.com/byerlikaya)**
- **π¦ [NuGet Packages](https://www.nuget.org/profiles/barisyerlikaya)**## π License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
**Built with β€οΈ by BarΔ±Ε Yerlikaya**
Made in Turkey πΉπ· | [Contact](mailto:b.yerlikaya@outlook.com) | [LinkedIn](https://www.linkedin.com/in/barisyerlikaya)