{"id":23714899,"url":"https://github.com/teilomillet/raggo","last_synced_at":"2025-04-11T19:08:35.486Z","repository":{"id":263047870,"uuid":"833803262","full_name":"teilomillet/raggo","owner":"teilomillet","description":"A lightweight, production-ready RAG (Retrieval Augmented Generation) library in Go.","archived":false,"fork":false,"pushed_at":"2024-11-23T17:20:44.000Z","size":581,"stargazers_count":49,"open_issues_count":1,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-25T15:01:39.890Z","etag":null,"topics":["ai","chromadb","document-search","embeddings","golang","llm","milvus","openai","question-answering","rag","retrieval-augmented-generation","vector-database","vector-search"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/teilomillet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-25T19:28:03.000Z","updated_at":"2025-03-24T10:13:30.000Z","dependencies_parsed_at":"2024-11-15T20:27:41.024Z","dependency_job_id":"0d01f3fe-338f-4b1c-b6de-c41ce98e7d01","html_url":"https://github.com/teilomillet/raggo","commit_stats":null,"previous_names":["teilomillet/raggo"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teilomillet%2Fraggo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teilomillet%2Fraggo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teilomillet%2Fraggo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teilomillet%2Fraggo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/teilomillet","download_url":"https://codeload.github.com/teilomillet/raggo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248465314,"owners_count":21108244,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","chromadb","document-search","embeddings","golang","llm","milvus","openai","question-answering","rag","retrieval-augmented-generation","vector-database","vector-search"],"created_at":"2024-12-30T20:51:31.297Z","updated_at":"2025-04-11T19:08:35.469Z","avatar_url":"https://github.com/teilomillet.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Raggo - Retrieval Augmented Generation Library\n\n\u003e A flexible RAG (Retrieval Augmented Generation) library for Go, designed to make document processing and context-aware AI interactions simple and efficient.\n\n\u003cp align=\"center\"\u003e\n  \u003cstrong\u003e🔍 Smart Document Search • 💬 Context-Aware Responses • 🤖 Intelligent RAG\u003c/strong\u003e\n\u003c/p\u003e\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/teilomillet/raggo.svg)](https://pkg.go.dev/github.com/teilomillet/raggo)\n[![Go Report Card](https://goreportcard.com/badge/github.com/teilomillet/raggo)](https://goreportcard.com/report/github.com/teilomillet/raggo)\n[![License](https://img.shields.io/github/license/teilomillet/raggo)](https://github.com/teilomillet/raggo/blob/main/LICENSE)\n\n\n## Quick Start\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"github.com/teilomillet/raggo\"\n)\n\nfunc main() {\n\t// Initialize RAG with default settings\n\trag, err := raggo.NewSimpleRAG(raggo.DefaultConfig())\n\tif err != nil {\n\t\tfmt.Printf(\"Error: %v\\n\", err)\n\t\treturn\n\t}\n\tdefer rag.Close()\n\n\t// Add documents from a directory\n\terr = rag.AddDocuments(context.Background(), \"./docs\")\n\tif err != nil {\n\t\tfmt.Printf(\"Error: %v\\n\", err)\n\t\treturn\n\t}\n\n\t// Search with natural language\n\tresponse, _ := rag.Search(context.Background(), \"What are the key features?\")\n\tfmt.Printf(\"Answer: %s\\n\", response)\n}\n```\n\n## Configuration\n\nRaggo provides a flexible configuration system that can be loaded from multiple sources (environment variables, JSON files, or programmatic defaults):\n\n```go\n// Load configuration (automatically checks standard paths)\ncfg, err := config.LoadConfig()\nif err != nil {\n    log.Fatal(err)\n}\n\n// Or create a custom configuration\ncfg := \u0026config.Config{\n    Provider:   \"milvus\",           // Vector store provider\n    Model:      \"text-embedding-3-small\",\n    Collection: \"my_documents\",\n    \n    // Search settings\n    DefaultTopK:     5,      // Number of similar chunks to retrieve\n    DefaultMinScore: 0.7,    // Similarity threshold\n    \n    // Document processing\n    DefaultChunkSize:    300,  // Size of text chunks\n    DefaultChunkOverlap: 50,   // Overlap between chunks\n}\n\n// Create RAG instance with config\nrag, err := raggo.NewSimpleRAG(cfg)\n```\n\nConfiguration can be saved for reuse:\n```go\nerr := cfg.Save(\"~/.raggo/config.json\")\n```\n\nEnvironment variables (take precedence over config files):\n- `RAGGO_PROVIDER`: Service provider\n- `RAGGO_MODEL`: Model identifier\n- `RAGGO_COLLECTION`: Collection name\n- `RAGGO_API_KEY`: Default API key\n\n\n\n## Table of Contents\n\n### Part 1: Core Components\n1. [Quick Start](#quick-start)\n2. [Building Blocks](#building-blocks)\n   - [Document Loading](#document-loading)\n   - [Text Parsing](#text-parsing)\n   - [Text Chunking](#text-chunking)\n   - [Embeddings](#embeddings)\n   - [Vector Storage](#vector-storage)\n\n### Part 2: RAG Implementations\n1. [Simple RAG](#simple-rag)\n   - [Basic Usage](#basic-usage)\n   - [Document Q\u0026A](#document-qa)\n   - [Configuration](#configuration)\n2. [Contextual RAG](#contextual-rag)\n   - [Advanced Features](#advanced-features)\n   - [Context Window](#context-window)\n   - [Hybrid Search](#hybrid-search)\n3. [Memory Context](#memory-context)\n   - [Chat Applications](#chat-applications)\n   - [Memory Management](#memory-management)\n   - [Context Enhancement](#context-enhancement)\n4. [Advanced Use Cases](#advanced-use-cases)\n   - [Full Processing Pipeline](#full-processing-pipeline)\n   - [Concurrent Processing](#concurrent-processing)\n   - [Rate Limiting](#rate-limiting)\n\n## Part 1: Core Components\n\n### Quick Start\n\n#### Prerequisites\n```bash\n# Set API key\nexport OPENAI_API_KEY=your-api-key\n\n# Install Raggo\ngo get github.com/teilomillet/raggo\n```\n\n### Building Blocks\n\n#### Document Loading\n```go\nloader := raggo.NewLoader(raggo.SetTimeout(1*time.Minute))\ndoc, err := loader.LoadURL(context.Background(), \"https://example.com/doc.pdf\")\n```\n\n#### Text Parsing\n```go\nparser := raggo.NewParser()\ndoc, err := parser.Parse(\"document.pdf\")\n```\n\n#### Text Chunking\n```go\nchunker := raggo.NewChunker(raggo.ChunkSize(100))\nchunks := chunker.Chunk(doc.Content)\n```\n\n#### Embeddings\n```go\nembedder := raggo.NewEmbedder(\n    raggo.SetProvider(\"openai\"),\n    raggo.SetModel(\"text-embedding-3-small\"),\n)\n```\n\n#### Vector Storage\n```go\ndb := raggo.NewVectorDB(raggo.WithMilvus(\"collection\"))\n```\n\n## Part 2: RAG Implementations\n\n### Simple RAG\nBest for straightforward document Q\u0026A:\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"log\"\n    \"github.com/teilomillet/raggo\"\n)\n\nfunc main() {\n    // Initialize SimpleRAG\n    rag, err := raggo.NewSimpleRAG(raggo.SimpleRAGConfig{\n        Collection: \"docs\",\n        Model:      \"text-embedding-3-small\",\n        ChunkSize:  300,\n        TopK:       3,\n    })\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer rag.Close()\n\n    // Add documents\n    err = rag.AddDocuments(context.Background(), \"./documents\")\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    // Search with different strategies\n    basicResponse, _ := rag.Search(context.Background(), \"What is the main feature?\")\n    hybridResponse, _ := rag.SearchHybrid(context.Background(), \"How does it work?\", 0.7)\n    \n    log.Printf(\"Basic Search: %s\\n\", basicResponse)\n    log.Printf(\"Hybrid Search: %s\\n\", hybridResponse)\n}\n```\n\n### Contextual RAG\nFor complex document understanding and context-aware responses:\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"os\"\n\t\"path/filepath\"\n\n\t\"github.com/teilomillet/raggo\"\n)\n\nfunc main() {\n\t// Initialize RAG with default settings\n\trag, err := raggo.NewDefaultContextualRAG(\"basic_contextual_docs\")\n\tif err != nil {\n\t\tfmt.Printf(\"Failed to initialize RAG: %v\\n\", err)\n\t\tos.Exit(1)\n\t}\n\tdefer rag.Close()\n\n\t// Add documents - the system will automatically:\n\t// - Split documents into semantic chunks\n\t// - Generate rich context for each chunk\n\t// - Store embeddings with contextual information\n\tdocsPath := filepath.Join(\"examples\", \"docs\")\n\tif err := rag.AddDocuments(context.Background(), docsPath); err != nil {\n\t\tfmt.Printf(\"Failed to add documents: %v\\n\", err)\n\t\tos.Exit(1)\n\t}\n\n\t// Simple search with automatic context enhancement\n\tquery := \"What are the key features of the product?\"\n\tresponse, err := rag.Search(context.Background(), query)\n\tif err != nil {\n\t\tfmt.Printf(\"Failed to search: %v\\n\", err)\n\t\tos.Exit(1)\n\t}\n\n\tfmt.Printf(\"\\nQuery: %s\\nResponse: %s\\n\", query, response)\n}\n```\n\n### Advanced Configuration\n\n```go\n// Create a custom configuration\nconfig := \u0026raggo.ContextualRAGConfig{\n\tCollection:   \"advanced_contextual_docs\",\n\tModel:        \"text-embedding-3-small\", // Embedding model\n\tLLMModel:     \"gpt-4o-mini\",           // Model for context generation\n\tChunkSize:    300,                      // Larger chunks for more context\n\tChunkOverlap: 75,                       // 25% overlap for better continuity\n\tTopK:         5,                        // Number of similar chunks to retrieve\n\tMinScore:     0.7,                      // Higher threshold for better relevance\n}\n\n// Initialize RAG with custom configuration\nrag, err := raggo.NewContextualRAG(config)\nif err != nil {\n\tlog.Fatalf(\"Failed to initialize RAG: %v\", err)\n}\ndefer rag.Close()\n```\n\n### Memory Context\nFor chat applications and long-term context retention:\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"log\"\n    \"github.com/teilomillet/raggo\"\n    \"github.com/teilomillet/gollm\"\n)\n\nfunc main() {\n    // Initialize Memory Context\n    memoryCtx, err := raggo.NewMemoryContext(\n        os.Getenv(\"OPENAI_API_KEY\"),\n        raggo.MemoryTopK(5),\n        raggo.MemoryCollection(\"chat\"),\n        raggo.MemoryStoreLastN(100),\n        raggo.MemoryMinScore(0.7),\n    )\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer memoryCtx.Close()\n\n    // Initialize Contextual RAG\n    rag, err := raggo.NewContextualRAG(\u0026raggo.ContextualRAGConfig{\n        Collection: \"docs\",\n        Model:     \"text-embedding-3-small\",\n    })\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer rag.Close()\n\n    // Example chat interaction\n    messages := []gollm.MemoryMessage{\n        {Role: \"user\", Content: \"How does the authentication system work?\"},\n    }\n    \n    // Store conversation\n    err = memoryCtx.StoreMemory(context.Background(), messages)\n    if err != nil {\n        log.Fatal(err)\n    }\n    \n    // Get enhanced response with context\n    prompt := \u0026gollm.Prompt{Messages: messages}\n    enhanced, _ := memoryCtx.EnhancePrompt(context.Background(), prompt, messages)\n    response, _ := rag.Search(context.Background(), enhanced.Messages[0].Content)\n    \n    log.Printf(\"Response: %s\\n\", response)\n}\n```\n\n### Advanced Use Cases\n\n#### Full Processing Pipeline\nProcess large document sets with rate limiting and concurrent processing:\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"log\"\n    \"sync\"\n    \"time\"\n    \"github.com/teilomillet/raggo\"\n    \"golang.org/x/time/rate\"\n)\n\nconst (\n    GPT_RPM_LIMIT   = 5000    // Requests per minute\n    GPT_TPM_LIMIT   = 4000000 // Tokens per minute\n    MAX_CONCURRENT  = 10      // Max concurrent goroutines\n)\n\nfunc main() {\n    // Initialize components\n    parser := raggo.NewParser()\n    chunker := raggo.NewChunker(raggo.ChunkSize(500))\n    embedder := raggo.NewEmbedder(\n        raggo.SetProvider(\"openai\"),\n        raggo.SetModel(\"text-embedding-3-small\"),\n    )\n\n    // Create rate limiters\n    limiter := rate.NewLimiter(rate.Limit(GPT_RPM_LIMIT/60), GPT_RPM_LIMIT)\n    \n    // Process documents concurrently\n    var wg sync.WaitGroup\n    semaphore := make(chan struct{}, MAX_CONCURRENT)\n\n    files, _ := filepath.Glob(\"./documents/*.pdf\")\n    for _, file := range files {\n        wg.Add(1)\n        semaphore \u003c- struct{}{} // Acquire semaphore\n        \n        go func(file string) {\n            defer wg.Done()\n            defer func() { \u003c-semaphore }() // Release semaphore\n            \n            // Wait for rate limit\n            limiter.Wait(context.Background())\n            \n            // Process document\n            doc, _ := parser.Parse(file)\n            chunks := chunker.Chunk(doc.Content)\n            embeddings, _ := embedder.CreateEmbeddings(chunks)\n            \n            log.Printf(\"Processed %s: %d chunks\\n\", file, len(chunks))\n        }(file)\n    }\n    \n    wg.Wait()\n}\n```\n\n## Best Practices\n\n### Resource Management\n- Always use `defer Close()`\n- Monitor memory usage\n- Clean up old data\n\n### Performance\n- Use concurrent processing for large datasets\n- Configure appropriate chunk sizes\n- Enable hybrid search when needed\n\n### Context Management\n- Use Memory Context for chat applications\n- Configure context window size\n- Clean up old memories periodically\n\n## Examples\n\nCheck `/examples` for more:\n- Basic usage: `/examples/simple/`\n- Context-aware: `/examples/contextual/`\n- Chat applications: `/examples/chat/`\n- Memory usage: `/examples/memory_enhancer_example.go`\n- Full pipeline: `/examples/full_process.go`\n- Benchmarks: `/examples/process_embedding_benchmark.go`\n\n## License\n\nMIT License - see [LICENSE](LICENSE) file\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteilomillet%2Fraggo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fteilomillet%2Fraggo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteilomillet%2Fraggo/lists"}