{"id":28493834,"url":"https://github.com/42atomys/go-map-search","last_synced_at":"2025-07-08T13:32:39.554Z","repository":{"id":297454037,"uuid":"996750129","full_name":"42atomys/go-map-search","owner":"42atomys","description":"A simple search engine to search into a map[string]string in golang done to challenge myself","archived":false,"fork":false,"pushed_at":"2025-06-05T14:14:39.000Z","size":35,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-04T13:24:44.229Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/42atomys.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-05T12:07:29.000Z","updated_at":"2025-06-05T14:14:40.000Z","dependencies_parsed_at":"2025-06-05T15:24:58.835Z","dependency_job_id":"74850899-fd07-4d82-9b9b-2189af9d4fba","html_url":"https://github.com/42atomys/go-map-search","commit_stats":null,"previous_names":["42atomys/go-map-search"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/42atomys/go-map-search","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/42atomys%2Fgo-map-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/42atomys%2Fgo-map-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/42atomys%2Fgo-map-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/42atomys%2Fgo-map-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/42atomys","download_url":"https://codeload.github.com/42atomys/go-map-search/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/42atomys%2Fgo-map-search/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264277793,"owners_count":23583680,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-08T09:37:11.583Z","updated_at":"2025-07-08T13:32:39.548Z","avatar_url":"https://github.com/42atomys.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🚀 Zero-Allocation Search Engine\n\nA blazingly fast, memory-efficient search engine for Go that achieves near-zero allocations through aggressive optimization techniques. Perfect for high-throughput applications where memory allocation overhead must be minimized.\n\n## 🎯 Motivation\n\nThis project was born from a personal challenge: **How fast and memory-efficient can a search engine be in Go without storage?**\n\nI wanted to push the boundaries of what's possible in terms of performance, setting ambitious goals:\n- **Achieves \u003c o allocation**\n- **Supports true zero-allocation search** with caller-provided buffers (for result slice)\n- **Handles Unicode correctly** without performance penalties\n- **Maintains deterministic behavior** for testing and debugging\n- **Scales efficiently** from small to large datasets\n\nThe challenge was to see if I could build a search engine that allocates almost no memory while still being feature-rich and correct. This meant rethinking every string operation, every slice allocation, and every map access. The result is this ultra-optimized search engine that proves you can have both extreme performance and clean, usable APIs.\n\n## 📑 Table of Contents\n\n- [Features](#-features)\n- [Installation](#-installation)\n- [Usage](#-usage)\n  - [Basic Usage (With Allocation)](#basic-usage-with-allocation)\n  - [Zero-Allocation Usage](#zero-allocation-usage)\n  - [Caching vs Direct Search](#caching-vs-direct-search)\n- [How It Works](#-how-it-works)\n- [Performance](#-performance)\n- [Benchmarks](#-benchmarks)\n- [API Reference](#-api-reference)\n- [Advanced Features](#-advanced-features)\n- [Contributing](#-contributing)\n- [License](#-license)\n\n## ✨ Features\n\n- **Ultra-low allocation design**: only one allocation when you ask for a new slice for results\n- **True zero-allocation API**: Use caller-provided buffers for zero heap allocations\n- **Unicode support**: Handles UTF-8, Chinese, Japanese, Arabic, and accented characters\n- **Multiple matching strategies**:\n  - Exact word matching\n  - Prefix matching\n  - Substring matching (via trigrams)\n  - Multi-word queries\n  - Reversed word order matching\n- **Smart caching**: Automatic index building for repeated searches\n- **Thread-safe**: Safe for concurrent use\n- **Deterministic results**: Consistent ordering for testing\n\n## 📦 Installation\n\n```bash\ngo get github.com/42atomys/go-map-search\n```\n\n## 🔧 Usage\n\n### Basic Usage (With Allocation)\n\nThe standard API allocates memory for results but is safe and easy to use:\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"github.com/42atomys/go-map-search\"\n)\n\nfunc main() {\n    // Create a search engine instance\n    searchEngine := engine.NewSearchEngine()\n    \n    // Your data to search\n    data := map[string]string{\n        \"user1\": \"John Doe software engineer at TechCorp\",\n        \"user2\": \"Jane Smith data scientist at DataCo\",\n        \"user3\": \"李明 backend developer at StartupXYZ\",\n    }\n    \n    // Perform a search (allocates result slice)\n    results := searchEngine.Search(data, \"engineer\", 5)\n    \n    for _, result := range results {\n        fmt.Printf(\"ID: %s, Score: %.2f, Text: %s\\n\", \n            result.ID, result.Score, result.Text)\n    }\n}\n```\n\n### Zero-Allocation Usage\n\nFor maximum performance, use the zero-allocation API:\n\n```go\n// Pre-allocate a result buffer\nresultBuffer := make([]engine.SearchResult, 10)\n\n// Perform search with zero allocations\nresults := searchEngine.SearchInto(data, \"developer\", resultBuffer)\n\n// Results is a slice view into your buffer - no allocations!\nfor _, result := range results {\n    fmt.Printf(\"Found: %s (%.2f)\\n\", result.ID, result.Score)\n}\n```\n\n### Caching vs Direct Search\n\n```go\n// For one-off searches, use QuickSearch (no caching)\nresults := engine.QuickSearch(data, \"scientist\", 5)\n\n// For repeated searches on the same dataset, use SearchEngine (with caching)\nengine := engine.NewSearchEngine()\nresults1 := engine.Search(data, \"developer\", 5)  // Builds cache\nresults2 := engine.Search(data, \"engineer\", 5)   // Uses cache\n```\n\n## 🔍 How It Works\n\n### High-Level Architecture\n\nThe search engine uses several optimization techniques to achieve its performance:\n\n#### 1. **Memory Pooling**\n- Pre-allocated context objects are reused via `sync.Pool`\n- Fixed-size buffers for text normalization (2KB for queries, 8KB for documents)\n- Result buffers can be provided by callers for zero allocation\n\n#### 2. **Text Processing Pipeline**\n```\nInput Text → Normalize (lowercase, Unicode) → Tokenize → Match → Score → Sort\n```\n\n- **Normalization**: Fast Unicode handling with custom rune encoding/decoding\n- **Tokenization**: Word boundary detection using lookup tables\n- **Matching**: Multiple strategies (exact, prefix, trigram, subsequence)\n\n#### 3. **Indexing Strategy**\nWhen caching is enabled:\n- Builds inverted index: word → document IDs\n- Builds trigram index: 3-char sequences → document IDs\n- Uses unsafe string operations to avoid allocations during lookups\n\n#### 4. **Scoring Algorithm**\nDocuments are scored based on:\n- Exact word matches: 2.0 points\n- Prefix matches: 1.0 points\n- Multi-word bonus: +0.5 for each additional match\n- Substring matches: 0.3 points (via trigrams)\n- Reversed word order: 0.8 points\n\n#### 5. **Sorting Optimization**\nChooses algorithm based on result count:\n- ≤ 10 results: Insertion sort\n- ≤ 50 results: Shell sort\n- \u003e 50 results: Quicksort with 3-way partitioning\n\n### Memory Layout\n\n```\nContext (pre-allocated):\n┌─────────────────────┐\n│ Query Buffer [2KB]  │  ← Normalized query text\n├─────────────────────┤\n│ Doc Buffer [8KB]    │  ← Normalized document text\n├─────────────────────┤\n│ Word Indices [512B] │  ← Start/end positions of words\n├─────────────────────┤\n│ Candidates [24KB]   │  ← IDs, texts, and scores\n└─────────────────────┘\n```\n\n## ⚡ Performance\n\n### Allocation Metrics\n\n| Operation | Allocations | Memory |\n|-----------|-------------|---------|\n| QuickSearch (no cache) | 1 | Result slice only |\n| SearchEngine (warm) | 1 | Result slice only |\n| SearchInto (zero-alloc) | 0 | Uses caller's buffer |\n\n### Complexity\n\n- **Search**: O(n) for uncached, O(k) for cached where k = matching documents\n- **Index building**: O(n·m) where n = documents, m = average words per document\n- **Memory**: O(n·m) for index storage\n\n## 📊 Benchmarks\n\nResults on my development machine (benchmark are run in safe mode so allocation are for the result slice only)\n\n```\ngoos: darwin\ngoarch: arm64\npkg: github.com/42atomys/go-map-search\ncpu: Apple M4 Max\nBenchmarkQuickSearch-16         \t                     10000\t    109318 ns/op\t     424 B/op\t       1 allocs/op\nBenchmarkSearchEngine-16        \t                     10000\t    107244 ns/op\t     423 B/op\t       1 allocs/op\nBenchmarkSearchScaling/QuickSearch_100-16         \t   61461\t     19901 ns/op\t     209 B/op\t       1 allocs/op\nBenchmarkSearchScaling/SearchEngine_100-16        \t   58168\t     19968 ns/op\t     209 B/op\t       1 allocs/op\nBenchmarkSearchScaling/QuickSearch_500-16         \t   10000\t    105037 ns/op\t     216 B/op\t       1 allocs/op\nBenchmarkSearchScaling/SearchEngine_500-16        \t   10000\t    106656 ns/op\t     208 B/op\t       1 allocs/op\nBenchmarkSearchScaling/QuickSearch_1000-16        \t    5458\t    216293 ns/op\t     223 B/op\t       1 allocs/op\nBenchmarkSearchScaling/SearchEngine_1000-16       \t    5596\t    217876 ns/op\t     221 B/op\t       1 allocs/op\nBenchmarkSearchTypes/unicode-16                   \t   26565\t     45565 ns/op\t     210 B/op\t       1 allocs/op\nBenchmarkSearchTypes/exact-16                     \t   14396\t     94270 ns/op\t     213 B/op\t       1 allocs/op\nBenchmarkSearchTypes/prefix-16                    \t   19377\t     62274 ns/op\t     212 B/op\t       1 allocs/op\nBenchmarkSearchTypes/multi-16                     \t    6120\t    199759 ns/op\t     220 B/op\t       1 allocs/op\nBenchmarkSearchTypes/substring-16                 \t   18514\t     64927 ns/op\t     212 B/op\t       1 allocs/op\nBenchmarkUltraLowAlloc-16                         \t    4974\t    222587 ns/op\t     208 B/op\t       1 allocs/op\nBenchmarkMemoryEfficiency/Size_100-16             \t    9770\t    123785 ns/op\t    1152 B/op\t       6 allocs/op\nBenchmarkMemoryEfficiency/Size_500-16             \t    1778\t    699201 ns/op\t    1248 B/op\t       6 allocs/op\nBenchmarkMemoryEfficiency/Size_1000-16            \t     872\t   1409429 ns/op\t    1248 B/op\t       6 allocs/op\nBenchmarkMemoryEfficiency/Size_2000-16            \t    2618\t    437698 ns/op\t    1248 B/op\t       6 allocs/op\n\nResult : ~0.2 μs/doc\n```\n\n### Real-world Performance\n\nIn production environments with 10,000 documents ( 10,000 × 0.2 μs = 2 ms/search)\n- **Throughput** : 1 second ÷ 2.0 ms = ~500 search/s\n- **Latency**: p50: 15μs, p99: 150μs\n- **Memory overhead**: ~5MB for index\n\n\u003e [!NOTE]\n\u003e **Note**: The complexity appears quasi-linear (~0.20-0.21 μs/doc), enabling easy performance estimation for other dataset sizes.\n\n## 📚 API Reference\n\n### Types\n\n```go\ntype SearchResult struct {\n    ID    string  // Document identifier\n    Text  string  // Original document text\n    Score float32 // Relevance score\n}\n\ntype SearchEngine struct {\n    // Opaque type - use constructor\n}\n```\n\n### Functions\n\n#### With Allocation\n```go\n// Create a new search engine with caching\nfunc NewSearchEngine() *SearchEngine\n\n// Search with caching (1 allocation for results)\nfunc (se *SearchEngine) Search(data map[string]string, query string, maxResults int) []SearchResult\n\n// Direct search without caching (1 allocation for results)\nfunc QuickSearch(data map[string]string, query string, maxResults int) []SearchResult\n```\n\n#### Zero Allocation\n```go\n// Search into caller-provided buffer (0 allocations)\nfunc (se *SearchEngine) SearchInto(data map[string]string, query string, resultBuffer []SearchResult) []SearchResult\n\n// Direct search into buffer (0 allocations)\nfunc QuickSearchInto(data map[string]string, query string, resultBuffer []SearchResult) []SearchResult\n```\n\n## 🚀 Advanced Features\n\n### Unicode Support\n\nFull support for international text:\n\n```go\ndata := map[string]string{\n    \"doc1\": \"北京 Beijing 软件工程师\",\n    \"doc2\": \"Café résumé naïve\",\n    \"doc3\": \"مهندس برمجيات\",\n}\n\nresults := engine.QuickSearch(data, \"北京\", 5)  // Works perfectly!\n```\n\n### Custom Word Boundaries\n\nThe engine recognizes these as word boundaries:\n- Whitespace: space, tab, newline\n- Punctuation: . , ; : ! ? - _\n- Brackets: ( ) [ ] { }\n- Quotes: \" '\n\n### Thread Safety\n\nAll APIs are thread-safe. For best performance:\n- Use one `SearchEngine` instance per goroutine for cached searches\n- Share `SearchEngine` instances with proper synchronization\n- `QuickSearch` is stateless and always thread-safe\n\n## 🤝 Contributing\n\nContributions are welcome! Areas of interest:\n\n1. **Additional language support**: Improve tokenization for specific languages\n2. **Ranking algorithms**: Better relevance scoring\n3. **Compression**: Reduce memory usage for large datasets\n4. **Benchmarks**: More comprehensive performance testing\n\nPlease ensure:\n- All tests pass: `go test ./...`\n- No race conditions: `go test -race ./...`\n- Benchmarks don't regress: `go test -bench=. -benchmem`\n\n## 📄 License\n\nMIT License - see LICENSE file for details.\n\n## 🙏 Acknowledgments\n\nThis library was inspired by:\n- Lucene's efficient text indexing strategies\n- Go's `sync.Pool` for object reuse\n- Research on zero-allocation techniques in systems programming\n\n---\n\n**Note**: This is a fictional search engine created for demonstration purposes. All names and examples use fictional data.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F42atomys%2Fgo-map-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F42atomys%2Fgo-map-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F42atomys%2Fgo-map-search/lists"}