{"id":33280909,"url":"https://github.com/psarno/pyragix-net","last_synced_at":"2026-04-14T19:32:50.760Z","repository":{"id":321680655,"uuid":"1086029710","full_name":"psarno/pyragix-net","owner":"psarno","description":"Local-first .NET 9 RAG engine mirroring PyRagix with ONNX embeddings, FAISS/Lucene search, and Ollama   generation.","archived":false,"fork":false,"pushed_at":"2025-11-14T13:19:41.000Z","size":196,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-14T15:20:07.648Z","etag":null,"topics":["csharp","dotnet","faiss","local-ai","lucene","ollama","onyx","rag","retrieval-augmented-generation"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/psarno.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-29T21:04:56.000Z","updated_at":"2025-11-14T13:19:44.000Z","dependencies_parsed_at":"2025-10-31T00:27:00.541Z","dependency_job_id":null,"html_url":"https://github.com/psarno/pyragix-net","commit_stats":null,"previous_names":["psarno/pyragix-net"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/psarno/pyragix-net","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/psarno%2Fpyragix-net","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/psarno%2Fpyragix-net/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/psarno%2Fpyragix-net/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/psarno%2Fpyragix-net/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/psarno","download_url":"https://codeload.github.com/psarno/pyragix-net/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/psarno%2Fpyragix-net/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31812968,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T18:05:02.291Z","status":"ssl_error","status_checked_at":"2026-04-14T18:05:01.765Z","response_time":153,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csharp","dotnet","faiss","local-ai","lucene","ollama","onyx","rag","retrieval-augmented-generation"],"created_at":"2025-11-17T12:04:40.454Z","updated_at":"2026-04-14T19:32:50.753Z","avatar_url":"https://github.com/psarno.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg width=\"500\" height=\"auto\" alt=\"image\" src=\"https://github.com/user-attachments/assets/4c3fe1f7-b078-4fa9-99df-779c7f0867d1\" /\u003e\n\n# PyRagix.Net\n\nPyRagix.Net is a local-first, educational Retrieval-Augmented Generation (RAG) project for .NET 9.0. It mirrors the Python reference implementation while showcasing how to compose modern retrieval techniques in C#. Query expansion, hybrid search (semantic + keyword), cross-encoder reranking, and Ollama-backed generation all run locally so you can study the full pipeline without depending on hosted services.\n\n![License](https://img.shields.io/badge/license-MIT-green.svg)\n![.NET](https://img.shields.io/badge/.NET-9.0-blue.svg)\n![C#](https://img.shields.io/badge/C%23-12.0-purple.svg)\n![Local-first](https://img.shields.io/badge/local--first-RAG-brightgreen)\n![Ollama](https://img.shields.io/badge/Ollama-ready-blue)\n\n## Overview\n\nPyRagix.Net ships as a class library plus console sample so you can explore ingestion and querying patterns from a C# environment. Everything runs on your infrastructure with zero external API dependencies for document processing and search.\n\n**Key Highlights:**\n- **Privacy-First**: All document processing, indexing, and search happen locally.\n- **Educational Focus**: Modules are organized so each pipeline stage is easy to inspect.\n- **Developer-Friendly**: Simple C# API with comprehensive configuration options.\n- **Cross-Platform**: Runs on Windows, Linux, and macOS via .NET 9.0.\n\nThis repository contains:\n- **`pyragix-net/`** - The RAG engine class library (see the engine [`README`](pyragix-net/README.md))\n- **`pyragix-net-console/`** - Console application demonstrating engine usage\n\n### Technology at a Glance\nTalking with stakeholders or curious users? This is the quick translation guide:\n- **RAG** – An “open-book” AI that looks up relevant docs before answering.\n- **Hybrid Search (FAISS + BM25)** – Semantic matchmaker plus keyword detective.\n- **Ollama** – Runs the local LLM for query rewrites and answer drafting.\n- **ONNX Runtime** – Keeps exported ML models humming inside .NET.\n- **Tesseract OCR** – Turns scanned PDFs into searchable text.\n\nNeed more than bullet points? See our [`technology explainer`](docs/technology-explainer.md) for a deeper plain-language tour.\n\n## Quick Start\n\n### Prerequisites\n\n1. **.NET 9.0 SDK** - [Download here](https://dotnet.microsoft.com/download/dotnet/9.0)\n2. **Ollama** - [Download here](https://ollama.com) for local LLM inference\n3. **Python 3.8+** - For one-time ONNX model export\n4. **8GB+ RAM** (16GB+ recommended for optimal performance)\n\n### Installation\n\n```bash\n# Clone repository\ngit clone https://github.com/psarno/pyragix-net.git\ncd pyragix-net\n\n# Restore NuGet packages\ndotnet restore\n\n# Build solution\ndotnet build\n\n# Run unit tests (xUnit)\ndotnet test\n```\n\nFor test-writing guidance, see [`PyRagix.Net.Tests/README.md`](PyRagix.Net.Tests/README.md).\n\n### Setup ONNX Models (One-Time)\n\nPyRagix.Net requires two ONNX models for embeddings and reranking. Export from Python:\n\n```bash\n# Install export tool\npip install optimum[exporters-onnx]\n\n# Export embedding model (sentence-transformers)\noptimum-cli export onnx \\\n  --model sentence-transformers/all-MiniLM-L6-v2 \\\n  --task feature-extraction \\\n  pyragix-net/Models/embeddings\n\n# Export reranker model (cross-encoder)\noptimum-cli export onnx \\\n  --model cross-encoder/ms-marco-MiniLM-L-6-v2 \\\n  --task text-classification \\\n  pyragix-net/Models/reranker\n```\n\nSee [`docs/ONNX_SETUP.md`](docs/ONNX_SETUP.md) for detailed instructions.\n\n### Configure and Run\n\n```bash\n# Copy example configuration\ncp pyragix-net-console/settings.example.toml pyragix-net-console/settings.toml\n\n# Start Ollama server (in separate terminal)\nollama pull qwen2.5:7b\nollama serve\n\n# Run console app\ncd pyragix-net-console\n\n# Ingest documents (append --fresh to rebuild indexes from scratch)\ndotnet run -- ingest ./docs\n\n# Query the RAG system\ndotnet run -- query \"What is retrieval-augmented generation?\"\n```\n\n### Vector Index Backends (Windows vs. Linux/WSL)\n\nPyRagix.Net selects the vector index implementation automatically:\n\n- **Windows (native)** – Uses [FaissNet](https://www.nuget.org/packages/FaissNet) with the FAISS C++ backend for best performance.\n- **Linux / macOS / WSL** – Uses the built-in managed inner-product index (no native dependencies). This keeps the project runnable when FAISS binaries are unavailable.\n\nWhen switching operating systems (e.g., migrating from Windows to WSL), delete previously generated vector artifacts before re-ingesting:\n\n```bash\nrm -f faiss_index.bin\nrm -f pyragix.db\nrm -rf lucene_index\n```\n\nThen rerun ingestion inside the target environment so the index is regenerated in the correct format. You can also run `dotnet run -- ingest ./docs --fresh` to let the CLI handle cleanup before rebuilding.\n\n## Usage\n\n### As a Library\n\nThe primary use case is integrating PyRagix.Net as a library in your C# applications:\n\n```csharp\nusing PyRagix.Net.Core;\nusing PyRagix.Net.Config;\n\n// Load configuration from TOML file\nvar engine = RagEngine.FromSettings(\"settings.toml\");\n\n// Or configure programmatically\nvar config = new PyRagixConfig\n{\n    OllamaEndpoint = \"http://localhost:11434\",\n    OllamaModel = \"qwen2.5:7b\",\n    EmbeddingModelPath = \"./pyragix-net/Models/embeddings/model.onnx\",\n    RerankerModelPath = \"./pyragix-net/Models/reranker/model.onnx\",\n    EnableQueryExpansion = true,\n    EnableHybridSearch = true,\n    EnableReranking = true\n};\nvar engine = new RagEngine(config);\n\n// Ingest documents (PDF, HTML, images). Set fresh: true to recreate indexes from scratch.\nawait engine.IngestDocumentsAsync(\"./my-documents\", fresh: false);\n\n// Query with natural language\nvar answer = await engine.QueryAsync(\"What are the key findings?\");\nConsole.WriteLine(answer);\n```\n\n### Console Application\n\nThe included console app (`pyragix-net-console/`) demonstrates basic usage:\n\n```bash\n# Ingest a folder of documents (append --fresh to purge existing indexes)\ndotnet run -- ingest \u003cfolder_path\u003e\n\n# Query the indexed documents\ndotnet run -- query \"\u003cyour question\u003e\"\n```\n\nPassing `--fresh` (or `-f`) to the ingest command clears FAISS, Lucene, and SQLite artifacts before rebuilding.\n\n## Architecture\n\nPyRagix.Net implements a multi-stage retrieval pipeline optimized for accuracy and privacy.\n\n### Query Pipeline\n\n```\nUser Query\n  ↓\nMulti-Query Expansion (3-5 variants via local LLM)\n  ↓\nHybrid Search (FAISS semantic 70% + Lucene BM25 keyword 30%)\n  ↓\nCross-Encoder Reranking (top-20 → top-7 by relevance)\n  ↓\nAnswer Generation (local Ollama LLM)\n```\n\n```mermaid\nflowchart LR\n    Q[User Question] --\u003e QE[Query Expander\u003cbr/\u003e(LLM variants)]\n    QE --\u003e EMB[Embedding Service\u003cbr/\u003eONNX Runtime]\n    EMB --\u003e HR[Hybrid Retriever\u003cbr/\u003ereciprocal rank fusion]\n    FAISS[FAISS Vector Index] --\u003e HR\n    LUCENE[Lucene BM25 Index] --\u003e HR\n    HR --\u003e RR[Cross-Encoder Reranker]\n    RR --\u003e GEN[Ollama Generator]\n    GEN --\u003e A[Final Answer]\n```\n\n### Ingestion Pipeline\n\n```\nDocument Input (PDF, HTML, Images)\n  ↓\nText Extraction (PdfPig, AngleSharp, Tesseract OCR)\n  ↓\nSemantic Chunking (sentence-boundary aware)\n  ↓\nEmbedding Generation (ONNX Runtime - CPU/GPU)\n  ↓\nDual Indexing (FAISS vector + Lucene BM25 keyword)\n  ↓\nSQLite Metadata Storage\n```\n\n```mermaid\nflowchart LR\n    D[Document Input\u003cbr/\u003ePDF · HTML · Images] --\u003e P[Document Processor\u003cbr/\u003ePdfPig · AngleSharp · Tesseract]\n    P --\u003e C[Semantic Chunker\u003cbr/\u003esentence-aware windows]\n    C --\u003e E[Embedding Service\u003cbr/\u003eONNX Runtime]\n    E --\u003e I[Index Service]\n    I --\u003e F[FAISS Vector Index]\n    I --\u003e L[Lucene BM25 Index]\n    I --\u003e S[SQLite Chunk Metadata]\n```\n\n**Performance Benefits:**\n- Query expansion: 20-30% improved recall\n- Cross-encoder reranking: 15-25% better precision\n- Hybrid search: 30-40% better structured query handling\n\n## Key Features\n\n### Learning-Focused Design\n\n- **Traceable Pipelines**: Ingestion, retrieval, and orchestration live in separate namespaces so you can follow each step.\n- **Strict Typing as Documentation**: Nullable reference types and explicit models make contracts obvious.\n- **Configurable Experiments**: Toggle query expansion, reranking, and hybrid search independently to observe impact.\n- **Local-Only Stack**: Reproduce workshops or internal trainings without cloud dependencies or API quotas.\n\n### Modern RAG Techniques\n\n- **Query Expansion**: Generates multiple query variants to capture diverse phrasing and improve recall on ambiguous questions\n- **Cross-Encoder Reranking**: Re-scores retrieved chunks using a specialized relevance model for precision\n- **Hybrid Search**: Combines semantic similarity (FAISS) with keyword matching (Lucene BM25) for balanced retrieval\n- **Semantic Chunking**: Respects sentence boundaries to preserve context coherence\n\n### Privacy \u0026 Control\n\n- **100% Local Operation**: All document processing, indexing, and search happen on your infrastructure\n- **No External APIs**: Zero dependencies on cloud services for core functionality (Ollama runs locally)\n- **Data Sovereignty**: Your documents never leave your network\n- **Configurable Models**: Choose and run any Ollama-compatible LLM\n\n### Document Processing\n\n- **Multi-Format Support**: PDF (PdfPig), HTML/HTM (AngleSharp), Images (Tesseract OCR)\n- **Metadata Tracking**: SQLite database for chunk provenance and filtering\n- **Incremental Updates**: Add new documents without reprocessing entire corpus (coming soon)\n\n### .NET Integration\n\n- **.NET 9.0 Class Library**: Clean, modern C# API with nullable reference types\n- **EF Core SQLite**: Robust metadata storage\n- **TOML Configuration**: Human-readable settings via `settings.toml`\n- **Dependency Injection Ready**: Easy integration with ASP.NET Core and other DI frameworks\n\n## Configuration\n\nPyRagix.Net uses `settings.toml` for configuration. Copy `settings.example.toml` and customize:\n\n```toml\n# Core Paths\nEmbeddingModelPath = \"./pyragix-net/Models/embeddings/model.onnx\"\nRerankerModelPath = \"./pyragix-net/Models/reranker/model.onnx\"\nDatabasePath = \"pyragix.db\"\nFaissIndexPath = \"faiss_index.bin\"\nLuceneIndexPath = \"lucene_index\"\n\n# Ollama LLM\nOllamaEndpoint = \"http://localhost:11434\"\nOllamaModel = \"qwen2.5:7b\"\nOllamaTimeout = 180\n\n# RAG Features (enable modern techniques)\nEnableQueryExpansion = true      # Multi-query generation\nEnableHybridSearch = true        # FAISS + BM25 fusion\nEnableReranking = true           # Cross-encoder scoring\nEnableSemanticChunking = true   # Sentence-aware splitting\n\n# Performance Tuning\nEmbeddingBatchSize = 16         # Higher = faster (more RAM)\nDefaultTopK = 7                  # Top chunks for answer generation\nHybridAlpha = 0.7               # 70% semantic, 30% keyword\nQueryExpansionCount = 3         # Number of query variants\n\n# GPU Acceleration (requires CUDA)\nExecutionProviderPreference = \"Auto\"  # Options: \"Auto\", \"Cpu\", \"Gpu\"\nGpuDeviceId = 0\n```\n\n### Hardware Tuning\n\n**For 8-12GB RAM systems:**\n```toml\nEmbeddingBatchSize = 8\nOcrBaseDpi = 100\n```\n\n**For 32GB+ RAM systems:**\n```toml\nEmbeddingBatchSize = 32\nOcrBaseDpi = 200\n```\n\n**For CUDA-capable GPUs:**\n```toml\nExecutionProviderPreference = \"Gpu\"\nGpuDeviceId = 0\n```\n\n## Project Structure\n\n```\npyragix-net/\n├── README.md                          # This file\n├── pyragix-net.sln                    # Visual Studio solution\n├── docs/\n│   └── ONNX_SETUP.md                  # Model export guide\n│\n├── pyragix-net/                       # RAG Engine (Class Library)\n│   ├── Config/\n│   │   ├── PyRagixConfig.cs           # TOML configuration loader\n│   │   ├── settings.toml              # User configuration (gitignored)\n│   │   └── settings.example.toml      # Configuration template\n│   ├── Core/\n│   │   ├── RagEngine.cs               # Public API entry point\n│   │   ├── Models/\n│   │   │   └── ChunkMetadata.cs       # EF Core metadata entity\n│   │   └── Data/\n│   │       └── PyRagixDbContext.cs    # SQLite database context\n│   ├── Ingestion/\n│   │   ├── DocumentProcessor.cs       # PDF/HTML/OCR extraction\n│   │   ├── SemanticChunker.cs         # Sentence-aware text splitting\n│   │   ├── EmbeddingService.cs        # ONNX embedding generation\n│   │   ├── IndexService.cs            # FAISS + Lucene indexing\n│   │   └── IngestionService.cs        # Pipeline orchestration\n│   ├── Retrieval/\n│   │   ├── QueryExpander.cs           # Multi-query generation\n│   │   ├── HybridRetriever.cs         # FAISS + BM25 fusion (RRF)\n│   │   ├── Reranker.cs                # Cross-encoder ONNX scoring\n│   │   ├── OllamaGenerator.cs         # LLM answer generation\n│   │   └── RetrievalService.cs        # Pipeline orchestration\n│   ├── Models/                        # .onnx files (gitignored)\n│   └── pyragix-net.csproj             # .NET 9.0 class library\n│\n└── pyragix-net-console/               # Console Demo App\n    ├── Program.cs                     # CLI implementation\n    ├── Models/                        # .onnx models location\n    ├── settings.toml                  # App-specific config (gitignored)\n    └── pyragix-net-console.csproj     # .NET 9.0 executable\n```\n\n## Dependencies\n\n### Core AI/ML\n- **Microsoft.SemanticKernel** (1.66.0+) - AI orchestration framework\n- **Microsoft.ML.OnnxRuntime** (1.23.2+) - Embedding/reranking inference (CPU)\n- **Microsoft.ML.OnnxRuntime.Gpu** (1.23.2+) - Optional GPU acceleration\n\n### Search \u0026 Indexing\n- **FaissNet** (1.1.0+) - Facebook AI Similarity Search (vector search)\n- **Lucene.Net** (4.8.0+) - BM25 keyword search\n- **Lucene.Net.Analysis.Common** - Text analysis and tokenization\n- **Lucene.Net.QueryParser** - Query parsing utilities\n\n### Document Processing\n- **UglyToad.PdfPig** (1.7.0+) - PDF text extraction\n- **AngleSharp** (1.1.2+) - HTML/XML parsing\n- **Tesseract** (5.2.0+) - OCR for images\n\n### Infrastructure\n- **Microsoft.EntityFrameworkCore.Sqlite** (9.0.10+) - Metadata storage\n- **Tomlyn** (0.19.0+) - TOML configuration parsing\n- **System.Text.Json** (9.0.10+) - JSON serialization\n\n## Performance\n\n**Optimization strategies:**\n- Increase `EmbeddingBatchSize` to 32+ for faster ingestion (requires more RAM)\n- Disable `EnableQueryExpansion` for 2x faster queries (slight recall reduction)\n- Set `ExecutionProviderPreference = \"Gpu\"` (or `\"Auto\"` to pick automatically) for 5-10x faster embedding and reranking inference when CUDA is available\n\n**When to use PyRagix.Net:**\n- Teaching or learning RAG concepts in .NET-first teams.\n- Prototyping local knowledge tools that must stay on Windows/WSL infrastructure.\n- Demonstrating how FAISS/Lucene, ONNX, and Ollama cooperate inside a C# codebase.\n- Exploring strongly typed, DI-friendly implementations before scaling further.\n\n**When to use PyRagix (Python):**\n- Need the reference web UI or richer ingestion tooling.\n- Prefer PaddleOCR or Python’s ML ecosystem for experiments.\n- Rapid prototyping with Jupyter/CLI workflows.\n- Want parity with the article/tutorial content that inspired this project.\n\n## Use Cases\n\n- **Workshops \u0026 Lunch-and-Learns**: Walk developers through a full RAG stack using only local dependencies.\n- **Internal Prototypes**: Spin up proof-of-concept search tools over private corpora before investing in heavier infrastructure.\n- **Regulated Labs**: Experiment with retrieval techniques on machines that cannot reach external APIs.\n- **Desktop \u0026 Embedded Demos**: Embed a transparent RAG pipeline inside WPF/WinForms/Avalonia showcase apps.\n- **Dev Enablement**: Use the console app + library to compare toggles (expansion, hybrid search, reranking) in real time.\n\n**Development setup:**\n```bash\ngit clone https://github.com/psarno/pyragix-net.git\ncd pyragix-net\ndotnet restore\ndotnet build\n```\n\n**Code standards:**\n- .NET 9.0 with nullable reference types enabled\n- Follow existing architectural patterns (service-based pipeline design)\n- Add XML documentation comments for public APIs\n- Ensure TOML configuration compatibility\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n## Acknowledgements\n\n- **Python Original**: [PyRagix](https://github.com/psarno/pyragix)\n- **FAISS**: Meta AI Research (Facebook AI Similarity Search)\n- **Ollama**: Ollama Team (local LLM inference)\n- **Semantic Kernel**: Microsoft (AI orchestration framework)\n- **ONNX Runtime**: Microsoft (cross-platform ML inference)\n- **Sentence Transformers**: UKP Lab (embedding models)\n\n**Built with privacy, performance, and pure .NET in mind.**\n\n---\n\n## Related Projects\n\n- **[PyRagix](https://github.com/psarno/pyragix)** - Python implementation with web UI and strict type safety\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpsarno%2Fpyragix-net","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpsarno%2Fpyragix-net","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpsarno%2Fpyragix-net/lists"}