https://github.com/42atomys/go-map-search

A simple search engine to search into a map[string]string in golang done to challenge myself
https://github.com/42atomys/go-map-search

Last synced: about 1 year ago
JSON representation

A simple search engine to search into a map[string]string in golang done to challenge myself

Host: GitHub
URL: https://github.com/42atomys/go-map-search
Owner: 42atomys
Created: 2025-06-05T12:07:29.000Z (about 1 year ago)
Default Branch: main
Last Pushed: 2025-06-05T14:14:39.000Z (about 1 year ago)
Last Synced: 2025-07-04T13:24:44.229Z (about 1 year ago)
Language: Go
Size: 34.2 KB
Stars: 0
Watchers: 0
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

# 🚀 Zero-Allocation Search Engine

A blazingly fast, memory-efficient search engine for Go that achieves near-zero allocations through aggressive optimization techniques. Perfect for high-throughput applications where memory allocation overhead must be minimized.

## 🎯 Motivation

This project was born from a personal challenge: **How fast and memory-efficient can a search engine be in Go without storage?**

I wanted to push the boundaries of what's possible in terms of performance, setting ambitious goals:
- **Achieves < o allocation**
- **Supports true zero-allocation search** with caller-provided buffers (for result slice)
- **Handles Unicode correctly** without performance penalties
- **Maintains deterministic behavior** for testing and debugging
- **Scales efficiently** from small to large datasets

The challenge was to see if I could build a search engine that allocates almost no memory while still being feature-rich and correct. This meant rethinking every string operation, every slice allocation, and every map access. The result is this ultra-optimized search engine that proves you can have both extreme performance and clean, usable APIs.

## 📑 Table of Contents

- [Features](#-features)
- [Installation](#-installation)
- [Usage](#-usage)
- [Basic Usage (With Allocation)](#basic-usage-with-allocation)
- [Zero-Allocation Usage](#zero-allocation-usage)
- [Caching vs Direct Search](#caching-vs-direct-search)
- [How It Works](#-how-it-works)
- [Performance](#-performance)
- [Benchmarks](#-benchmarks)
- [API Reference](#-api-reference)
- [Advanced Features](#-advanced-features)
- [Contributing](#-contributing)
- [License](#-license)

## ✨ Features

- **Ultra-low allocation design**: only one allocation when you ask for a new slice for results
- **True zero-allocation API**: Use caller-provided buffers for zero heap allocations
- **Unicode support**: Handles UTF-8, Chinese, Japanese, Arabic, and accented characters
- **Multiple matching strategies**:
- Exact word matching
- Prefix matching
- Substring matching (via trigrams)
- Multi-word queries
- Reversed word order matching
- **Smart caching**: Automatic index building for repeated searches
- **Thread-safe**: Safe for concurrent use
- **Deterministic results**: Consistent ordering for testing

## 📦 Installation

```bash
go get github.com/42atomys/go-map-search
```

## 🔧 Usage

### Basic Usage (With Allocation)

The standard API allocates memory for results but is safe and easy to use:

```go
package main

import (
"fmt"
"github.com/42atomys/go-map-search"
)

func main() {
// Create a search engine instance
searchEngine := engine.NewSearchEngine()

// Your data to search
data := map[string]string{
"user1": "John Doe software engineer at TechCorp",
"user2": "Jane Smith data scientist at DataCo",
"user3": "李明 backend developer at StartupXYZ",
}

// Perform a search (allocates result slice)
results := searchEngine.Search(data, "engineer", 5)

for _, result := range results {
fmt.Printf("ID: %s, Score: %.2f, Text: %s\n",
result.ID, result.Score, result.Text)
}
}
```

### Zero-Allocation Usage

For maximum performance, use the zero-allocation API:

```go
// Pre-allocate a result buffer
resultBuffer := make([]engine.SearchResult, 10)

// Perform search with zero allocations
results := searchEngine.SearchInto(data, "developer", resultBuffer)

// Results is a slice view into your buffer - no allocations!
for _, result := range results {
fmt.Printf("Found: %s (%.2f)\n", result.ID, result.Score)
}
```

### Caching vs Direct Search

```go
// For one-off searches, use QuickSearch (no caching)
results := engine.QuickSearch(data, "scientist", 5)

// For repeated searches on the same dataset, use SearchEngine (with caching)
engine := engine.NewSearchEngine()
results1 := engine.Search(data, "developer", 5) // Builds cache
results2 := engine.Search(data, "engineer", 5) // Uses cache
```

## 🔍 How It Works

### High-Level Architecture

The search engine uses several optimization techniques to achieve its performance:

#### 1. **Memory Pooling**
- Pre-allocated context objects are reused via `sync.Pool`
- Fixed-size buffers for text normalization (2KB for queries, 8KB for documents)
- Result buffers can be provided by callers for zero allocation

#### 2. **Text Processing Pipeline**
```
Input Text → Normalize (lowercase, Unicode) → Tokenize → Match → Score → Sort
```

- **Normalization**: Fast Unicode handling with custom rune encoding/decoding
- **Tokenization**: Word boundary detection using lookup tables
- **Matching**: Multiple strategies (exact, prefix, trigram, subsequence)

#### 3. **Indexing Strategy**
When caching is enabled:
- Builds inverted index: word → document IDs
- Builds trigram index: 3-char sequences → document IDs
- Uses unsafe string operations to avoid allocations during lookups

#### 4. **Scoring Algorithm**
Documents are scored based on:
- Exact word matches: 2.0 points
- Prefix matches: 1.0 points
- Multi-word bonus: +0.5 for each additional match
- Substring matches: 0.3 points (via trigrams)
- Reversed word order: 0.8 points

#### 5. **Sorting Optimization**
Chooses algorithm based on result count:
- ≤ 10 results: Insertion sort
- ≤ 50 results: Shell sort
- > 50 results: Quicksort with 3-way partitioning

### Memory Layout

```
Context (pre-allocated):
┌─────────────────────┐
│ Query Buffer [2KB] │ ← Normalized query text
├─────────────────────┤
│ Doc Buffer [8KB] │ ← Normalized document text
├─────────────────────┤
│ Word Indices [512B] │ ← Start/end positions of words
├─────────────────────┤
│ Candidates [24KB] │ ← IDs, texts, and scores
└─────────────────────┘
```

## ⚡ Performance

### Allocation Metrics

| Operation | Allocations | Memory |
|-----------|-------------|---------|
| QuickSearch (no cache) | 1 | Result slice only |
| SearchEngine (warm) | 1 | Result slice only |
| SearchInto (zero-alloc) | 0 | Uses caller's buffer |

### Complexity

- **Search**: O(n) for uncached, O(k) for cached where k = matching documents
- **Index building**: O(n·m) where n = documents, m = average words per document
- **Memory**: O(n·m) for index storage

## 📊 Benchmarks

Results on my development machine (benchmark are run in safe mode so allocation are for the result slice only)

```
goos: darwin
goarch: arm64
pkg: github.com/42atomys/go-map-search
cpu: Apple M4 Max
BenchmarkQuickSearch-16 10000 109318 ns/op 424 B/op 1 allocs/op
BenchmarkSearchEngine-16 10000 107244 ns/op 423 B/op 1 allocs/op
BenchmarkSearchScaling/QuickSearch_100-16 61461 19901 ns/op 209 B/op 1 allocs/op
BenchmarkSearchScaling/SearchEngine_100-16 58168 19968 ns/op 209 B/op 1 allocs/op
BenchmarkSearchScaling/QuickSearch_500-16 10000 105037 ns/op 216 B/op 1 allocs/op
BenchmarkSearchScaling/SearchEngine_500-16 10000 106656 ns/op 208 B/op 1 allocs/op
BenchmarkSearchScaling/QuickSearch_1000-16 5458 216293 ns/op 223 B/op 1 allocs/op
BenchmarkSearchScaling/SearchEngine_1000-16 5596 217876 ns/op 221 B/op 1 allocs/op
BenchmarkSearchTypes/unicode-16 26565 45565 ns/op 210 B/op 1 allocs/op
BenchmarkSearchTypes/exact-16 14396 94270 ns/op 213 B/op 1 allocs/op
BenchmarkSearchTypes/prefix-16 19377 62274 ns/op 212 B/op 1 allocs/op
BenchmarkSearchTypes/multi-16 6120 199759 ns/op 220 B/op 1 allocs/op
BenchmarkSearchTypes/substring-16 18514 64927 ns/op 212 B/op 1 allocs/op
BenchmarkUltraLowAlloc-16 4974 222587 ns/op 208 B/op 1 allocs/op
BenchmarkMemoryEfficiency/Size_100-16 9770 123785 ns/op 1152 B/op 6 allocs/op
BenchmarkMemoryEfficiency/Size_500-16 1778 699201 ns/op 1248 B/op 6 allocs/op
BenchmarkMemoryEfficiency/Size_1000-16 872 1409429 ns/op 1248 B/op 6 allocs/op
BenchmarkMemoryEfficiency/Size_2000-16 2618 437698 ns/op 1248 B/op 6 allocs/op

Result : ~0.2 μs/doc
```

### Real-world Performance

In production environments with 10,000 documents ( 10,000 × 0.2 μs = 2 ms/search)
- **Throughput** : 1 second ÷ 2.0 ms = ~500 search/s
- **Latency**: p50: 15μs, p99: 150μs
- **Memory overhead**: ~5MB for index

> [!NOTE]
> **Note**: The complexity appears quasi-linear (~0.20-0.21 μs/doc), enabling easy performance estimation for other dataset sizes.

## 📚 API Reference

### Types

```go
type SearchResult struct {
ID string // Document identifier
Text string // Original document text
Score float32 // Relevance score
}

type SearchEngine struct {
// Opaque type - use constructor
}
```

### Functions

#### With Allocation
```go
// Create a new search engine with caching
func NewSearchEngine() *SearchEngine

// Search with caching (1 allocation for results)
func (se *SearchEngine) Search(data map[string]string, query string, maxResults int) []SearchResult

// Direct search without caching (1 allocation for results)
func QuickSearch(data map[string]string, query string, maxResults int) []SearchResult
```

#### Zero Allocation
```go
// Search into caller-provided buffer (0 allocations)
func (se *SearchEngine) SearchInto(data map[string]string, query string, resultBuffer []SearchResult) []SearchResult

// Direct search into buffer (0 allocations)
func QuickSearchInto(data map[string]string, query string, resultBuffer []SearchResult) []SearchResult
```

## 🚀 Advanced Features

### Unicode Support

Full support for international text:

```go
data := map[string]string{
"doc1": "北京 Beijing 软件工程师",
"doc2": "Café résumé naïve",
"doc3": "مهندس برمجيات",
}

results := engine.QuickSearch(data, "北京", 5) // Works perfectly!
```

### Custom Word Boundaries

The engine recognizes these as word boundaries:
- Whitespace: space, tab, newline
- Punctuation: . , ; : ! ? - _
- Brackets: ( ) [ ] { }
- Quotes: " '

### Thread Safety

All APIs are thread-safe. For best performance:
- Use one `SearchEngine` instance per goroutine for cached searches
- Share `SearchEngine` instances with proper synchronization
- `QuickSearch` is stateless and always thread-safe

## 🤝 Contributing

Contributions are welcome! Areas of interest:

1. **Additional language support**: Improve tokenization for specific languages
2. **Ranking algorithms**: Better relevance scoring
3. **Compression**: Reduce memory usage for large datasets
4. **Benchmarks**: More comprehensive performance testing

Please ensure:
- All tests pass: `go test ./...`
- No race conditions: `go test -race ./...`
- Benchmarks don't regress: `go test -bench=. -benchmem`

## 📄 License

MIT License - see LICENSE file for details.

## 🙏 Acknowledgments

This library was inspired by:
- Lucene's efficient text indexing strategies
- Go's `sync.Pool` for object reuse
- Research on zero-allocation techniques in systems programming

---

**Note**: This is a fictional search engine created for demonstration purposes. All names and examples use fictional data.

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/42atomys/go-map-search

Awesome Lists containing this project

README