{"id":50453425,"url":"https://github.com/mg52/search52","last_synced_at":"2026-06-01T01:04:08.399Z","repository":{"id":289557165,"uuid":"971655999","full_name":"mg52/search52","owner":"mg52","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-29T21:17:33.000Z","size":434,"stargazers_count":17,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-29T23:09:55.261Z","etag":null,"topics":["full-text-search","go","search","search-engine","searchengine"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mg52.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-04-23T21:21:53.000Z","updated_at":"2026-05-29T21:17:37.000Z","dependencies_parsed_at":"2025-05-05T01:15:07.742Z","dependency_job_id":"2677bdcb-6e87-460e-9ff8-32b82c136444","html_url":"https://github.com/mg52/search52","commit_stats":null,"previous_names":["mg52/search","mg52/search52"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mg52/search52","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mg52%2Fsearch52","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mg52%2Fsearch52/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mg52%2Fsearch52/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mg52%2Fsearch52/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mg52","download_url":"https://codeload.github.com/mg52/search52/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mg52%2Fsearch52/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33755375,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-31T02:00:06.040Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["full-text-search","go","search","search-engine","searchengine"],"created_at":"2026-06-01T01:04:07.573Z","updated_at":"2026-06-01T01:04:08.390Z","avatar_url":"https://github.com/mg52.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![build](https://github.com/mg52/search52/actions/workflows/go.yml/badge.svg)\n\n# In-Memory Search Engine\n\nA lightweight, fast, in-memory inverted-index search engine with HTTP handlers and on-disk persistence.  \n\n## Features\n\n- **Full-text indexing** on arbitrary JSON documents  \n- **Prefix \u0026 fuzzy matching** — prefix map for instant completions + SymSpell for Levenshtein fuzzy suggestions  \n- **Filters** on arbitrary document fields (OR within a field, AND across fields)  \n- **HTTP API** for index creation, search, single-doc upsert/delete, bulk add  \n- **Persistence** — saves documents + metadata and rebuilds all derived indexes on load  \n- **Docker-ready** — runs as an unprivileged user, persists under a mounted volume  \n\n---\n\n## Benchmark\n\nAll numbers are from an Apple M1 Pro. The MusicBrainz load test uses the real `mb_5m.json` dataset. The in-process benchmark below uses generated 1 000 000 and 5 000 000 document datasets with **100 000 unique vocabulary words**, `title` (3-20 words) and `tags` (1-10 words) index fields, and `year` (2000-2024) as the filter field.\n\n### MusicBrainz 5 M HTTP load test\n\nEnvironment: Apple M1 Pro, darwin/arm64, Go 1.25.4, `GOMAXPROCS=10`.\n\nDataset: `mb_5m.json`, 5 000 000 MusicBrainz documents, 585 MB JSON. Index fields: `title`, `artist`, `album`. Filter field: `year`. Result size: 100. Hard-coded prefix map cap: 5 000. Multi-term last-token prefix expansion is adaptive: 100 completions for 1-2 chars, 60 for 3-5 chars, and 50 after that. Query vocabulary: all unique tokens extracted from the same indexed text fields. Single-term queries are 65% exact, 25% prefix, 10% misspelled; multi-term queries use 2-4 tokens, 10% have one misspelled token, and the last token is prefix-truncated 25% of the time. `-mode-mix balanced` sends equal traffic to Single/NoFilter, Single/Filter, Multi/NoFilter, and Multi/Filter.\n\nIndex/load summary:\n\n| Metric | Value |\n|---|---:|\n| Documents indexed | 5 000 000 |\n| JSON file size | 585 MB |\n| Vocabulary size | All unique tokens from `title`, `artist`, `album` |\n| Total index time | 94.85 s |\n\nHTTP load-test results:\n\nThe load-test client drains each response body before recording latency.\n\n| Scope | Workers | Queries | Errors | HTTP statuses | Wall time | RPS | avg | p50 | p95 | p99 |\n|---|---:|---:|---:|---|---:|---:|---:|---:|---:|---:|\n| Overall | 16 | 100 000 | 0 | 200:100000 | 20.349 s | 4 914.4 | 3.24 ms | 1.78 ms | 11.41 ms | 21.33 ms |\n| Single / NoFilter | 16 | 25 000 | 0 | 200:25000 | 20.349 s | 1 228.6 | 4.30 ms | 1.83 ms | 15.40 ms | 26.51 ms |\n| Single / Filter | 16 | 25 000 | 0 | 200:25000 | 20.349 s | 1 228.6 | 2.57 ms | 1.13 ms | 10.23 ms | 21.41 ms |\n| Multi / NoFilter | 16 | 25 000 | 0 | 200:25000 | 20.349 s | 1 228.6 | 3.28 ms | 2.24 ms | 8.92 ms | 17.80 ms |\n| Multi / Filter | 16 | 25 000 | 0 | 200:25000 | 20.349 s | 1 228.6 | 2.80 ms | 1.89 ms | 7.90 ms | 15.15 ms |\n\n### In-process benchmark\n\nThis benchmark runs entirely inside the Go process: it loads a JSON dataset, builds the engine in memory, warms up query paths, and measures search latency without HTTP/network overhead.\n\n#### 1 M docs — heap delta ~800 MB\n\n| Mode | avg | p50 | p99 | B/op | allocs/op |\n|---|---:|---:|---:|---:|---:|\n| SingleTerm / NoFilter | 29.4 µs | 21.2 µs |  88.9 µs | 14 136 | 10 |\n| SingleTerm / Filter   |  8.8 µs |  6.6 µs |  26.1 µs |  3 457 | 13 |\n| MultiTerm  / NoFilter | 16.6 µs | 12.9 µs |  73.2 µs |  2 831 | 25 |\n| MultiTerm  / Filter   | 12.7 µs | 12.1 µs |  32.8 µs |  2 862 | 27 |\n\n#### 5 M docs — heap delta ~2 773 MB\n\n| Mode | avg | p50 | p99 | B/op | allocs/op |\n|---|---:|---:|---:|---:|---:|\n| SingleTerm / NoFilter | 67.7 µs | 40.4 µs | 318.6 µs | 14 148 | 10 |\n| SingleTerm / Filter   | 53.3 µs | 36.5 µs | 230.7 µs |  6 715 | 13 |\n| MultiTerm  / NoFilter | 82.0 µs | 41.8 µs | 594.6 µs |  3 472 | 25 |\n| MultiTerm  / Filter   | 46.9 µs | 43.0 µs | 173.9 µs |  3 492 | 27 |\n\nFilter queries are faster than no-filter equivalents because the bitset pre-prunes the candidate set before the posting-list scan.\n\nBench tooling documentation lives in [cmd/bench/README.md](cmd/bench/README.md)\n\n---\n\n## Quickstart\n\n### Build \u0026 Run\n\n```bash\ngo run ./cmd/service\n```\n\nThe API service listens on `:8080`. A small admin UI listens on `:8081`.\n\n### Docker\n\n```bash\ndocker build -t searchengine:latest .\n\ndocker run -d \\\n  -p 8080:8080 \\\n  -p 8081:8081 \\\n  -v search_data:/data \\\n  -e SEARCH52_INDEX_DATA_DIR=/data \\\n  --name searchengine \\\n  searchengine:latest\n```\n\n---\n\n## How It Works\n\n### 1. Inverted Index\n\n```go\nDataMap map[string]map[uint32]int\n// term → internalDocID → score\n```\n\nEvery document is tokenized from the configured `IndexFields`. Tokenization lowercases the text, strips every non-alphanumeric character (via a compiled regexp), and drops a fixed stop-word list (`a`, `the`, `and`). Index fields can optionally define `fieldWeights`; missing weights default to `1`, while the HTTP API rejects non-positive weights. Each token receives a normalized score based on its field weight, so a token from a field with weight `3` is worth three times a token from a field with weight `1` while keeping the document's total score budget roughly stable. If the same token appears multiple times its scores are summed, so denser matches rank higher. `DataMap` is the primary posting-list and ranking structure used by the search hot path.\n\n### 2. Internal IDs and Tombstones\n\nDocuments are identified internally by a monotonically increasing `uint32`:\n\n| Map | Purpose |\n|---|---|\n| `ExternalToInternal` | caller's string ID → current internal ID |\n| `InternalToExternal` | internal ID → caller's string ID |\n| `Documents` | internal ID → raw field map |\n| `DocDeleted` | internal ID → tombstoned? |\n\n**Update semantics**: updating a document assigns a new internal ID and sets `DocDeleted[oldID] = true`. Searches skip tombstoned IDs at scan time, while the new version is tokenized and indexed normally. To reclaim old posting-list entries, call the compact endpoint; it rebuilds the inverted index, filter bitsets, prefix map, and fuzzy dictionary from active documents only.\n\n### 3. Prefix and Fuzzy Matching\n\nAt index time, every new term seeds two auxiliary structures:\n\n- **`Prefix map[string][]string`** — maps every prefix of a term to a hard-coded list of up to 5 000 completions. Lookup is a single map read — O(1). At query time the first 3 completions are used for single-term prefix search. Multi-term search uses an adaptive prefix count for the last token: 100 completions for 1-2 chars, 60 for 3-5 chars, and 50 after that.\n- **`SymSpell`** — a Levenshtein-distance index used for fuzzy suggestions when the query token has no prefix candidates (i.e. it is not a known prefix of any indexed term). Only terms with length ≥ 4 are added to SymSpell to avoid noise from very short tokens.\n\nSingle-term search includes the exact term when it exists, then adds prefix candidates (up to 3). If there are no prefix candidates and no exact term, it falls back to SymSpell suggestions. Multi-term search applies exact + fuzzy expansion on all-but-last tokens, and exact + adaptive prefix expansion on the last token (the partially-typed word).\n\n### 4. Bitset Filters\n\nFilter fields (e.g. `year`) are stored as permanent per-value bitsets instead of per-query maps:\n\n```go\nFilterBits map[string][]uint64\n// \"year:2020\" → []uint64  (bit i set ↔ internalDocID i matches)\n```\n\nAt index time each matching internal ID flips one bit: `bits[id\u003e\u003e6] |= 1 \u003c\u003c (id\u002663)`. At query time a single bit test replaces a map lookup:\n\n```\nfilterBitTest(allowed, id)  →  bits[id\u003e\u003e6] \u0026 (1\u003c\u003c(id\u002663)) != 0\n```\n\n**Memory**: ~1.25 MB per filter value per 1 M documents, paid once at index time.  \n**Per-query allocation**: zero for the common case (single field, single value) — `applyFilterLocked` returns a direct slice reference into `se.FilterBits` with no copy.\n\nMulti-value filters within one field are ORed (bitwise union); multiple fields are ANDed (bitwise intersection). Both operations produce a new `[]uint64` and are O(N/64) where N is the highest internal ID.\n\n### 5. Top-k Extraction with a Specialized Min-Heap\n\nSearch maintains a bounded min-heap of the top-k candidates:\n\n```go\ntype internalHit struct { id uint32; score int }\n```\n\nThe heap operations (`heapPushHit`, `heapReplaceTop`, `siftDownHit`) are inlined directly over `[]internalHit` with no interface dispatch or boxing — unlike `container/heap` which boxes every element into `any`. At k = 100 this saves ~200 allocations per query.\n\n**Fill phase**: while `len(h) \u003c k`, push every passing candidate. Once full, replace the root only when `candidate.score \u003e h[0].score` (the current minimum).\n\n**Extraction phase**: repeated inline heap-pop fills the result slice from the last index down, yielding results in descending score order without an extra sort pass.\n\n### 6. Concurrency and Zero-Copy Filter Resolution\n\nThe engine uses a single `sync.RWMutex` plus a lock-free `termSet`:\n\n- All search paths hold **`RLock`** — unlimited concurrent readers, no blocking between searches.\n- Index writes (`InsertDocs`, `BuildDocumentIndex`, `AddOrUpdateDocument`) hold **`Lock`** — exclusive, blocks new readers until the write completes.\n- **`termSet sync.Map`** — a lock-free set of every indexed term. `SingleTermSearch` and `MultiTermSearch` check exact-term existence here before acquiring `RLock`, avoiding a mutex round-trip for the common case.\n\n`SingleTermSearchLoop` and `MultiTermSearchLoop` acquire `RLock` **once at the top** and hold it across both filter resolution and the posting-map scan:\n\n```\nRLock acquired\n  └─ applyFilterLocked()   →  returns direct []uint64 reference (no copy)\n  └─ posting-map scan      →  filterBitTest reads from same reference\nRLock released\n```\n\nFor the common case (single field, single value), `applyFilterLocked` returns a slice header pointing directly into `se.FilterBits` with zero allocation.\n\n### 7. Multi-Term Search: Anchor-Group Strategy\n\nMulti-term queries use a boolean AND-across-groups, OR-within-group model. A \"group\" is a set of synonyms or expansions for one query token.\n\nRather than intersecting all groups eagerly, the engine picks the **smallest group** (fewest total posting entries) as the anchor and iterates only its candidates. For each candidate it checks membership in every other group with a map lookup — O(1) per group. This avoids materialising a full intersection set and keeps multi-term search fast even when individual terms are common.\n\nScore for a matching document is the sum of its scores across all matched groups.\n\n### 8. Persistence\n\n`SaveAll` serialises only the raw document store and metadata (IDs, field config) to a single gob file. `LoadAll` restores the documents and then rebuilds all derived structures — `DataMap`, `FilterBits`, `Prefix`, `SymSpell` — by replaying the tokenisation pass. This keeps the snapshot compact and means the on-disk format never needs a schema migration when internal data structures change. If `engine.gob` is corrupt or fails validation during the HTTP load endpoint, the existing in-memory index is left unchanged and the endpoint returns a clear error.\n\n---\n\n## HTTP API\n\nAll endpoints return JSON and use HTTP status codes (`201 Created`, `200 OK`, `400 Bad Request`, `404 Not Found`, `405 Method Not Allowed`, `409 Conflict`, `504 Gateway Timeout`, `500 Internal Server Error`). Error responses have a fixed schema:\n\n```json\n{\n  \"status\": \"error\",\n  \"statusCode\": 400,\n  \"error\": \"message\"\n}\n```\n\nInput validation is intentionally strict: index/filter/field names may contain only letters, numbers, `_`, `-`, and `.`; unknown JSON fields are rejected; `resultCount` must be between 1 and 10 000; search queries are capped at 512 characters; filter strings are capped at 2 048 characters; individual filter values are capped at 256 characters; JSON request bodies are capped at 1 MiB except single-document writes, which allow 5 MiB; bulk uploads allow up to 1 GiB.\n\n### 1. Create Index\n\n```bash\ncurl -X POST http://localhost:8080/create-index \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n    \"indexName\":   \"products\",\n    \"indexFields\": [\"name\", \"tags\"],\n    \"fieldWeights\": { \"name\": 3, \"tags\": 1 },\n    \"filters\":     [\"year\"],\n    \"resultCount\": 10\n  }'\n```\n\n### 2. List Indexes\n\n```bash\ncurl http://localhost:8080/list-indexes\n```\n\nThe response includes every in-memory index and its current active document count. `storedDocs` includes old tombstoned document versions retained for update/delete performance, while `activeDocs` is the current live record count.\n\n```json\n{\n  \"status\": \"success\",\n  \"statusCode\": 200,\n  \"total\": 1,\n  \"indexes\": [\n    {\n      \"name\": \"products\",\n      \"activeDocs\": 2,\n      \"storedDocs\": 3,\n      \"deletedVersions\": 1,\n      \"indexFields\": [\"name\", \"tags\"],\n      \"filters\": [\"year\"],\n      \"resultCount\": 10\n    }\n  ]\n}\n```\n\n### 3. Bulk Add to Index\n\n```bash\ncurl -X POST 'http://localhost:8080/add-to-index?indexName=products' \\\n  -F 'file=@docs.json'\n```\n\nThe uploaded file can be either a JSON array of objects or a CSV file with a header row. Every document should include an `id` field; documents without a usable `id` are skipped by the indexer.\n\nJSON example:\n\n```json\n[\n  { \"id\": \"1\", \"name\": \"foo\", \"tags\": [\"a\", \"b\"], \"year\": \"2020\" },\n  { \"id\": \"2\", \"name\": \"bar\", \"tags\": [\"c\"],       \"year\": \"2021\" }\n]\n```\n\nCSV example:\n\n```csv\nid,name,tags,year\n1,foo,a b,2020\n2,bar,c,2021\n```\n\n### 4. Search\n\n```bash\n# Simple query\ncurl 'http://localhost:8080/search?index=products\u0026q=laptop'\n\n# With a single filter\ncurl 'http://localhost:8080/search?index=products\u0026q=laptop\u0026filter=year:2020'\n\n# With multiple filters (AND across fields, OR within a field)\ncurl 'http://localhost:8080/search?index=products\u0026q=laptop\u0026filter=year:2020,year:2021,category:electronics'\n```\n\n| Param | Description |\n|---|---|\n| `index` | Index name |\n| `q` | Search query (single or multi-term) |\n| `filter` | Comma-separated `field:value` pairs |\n\nSearch requests use a 10 second context timeout. If a query exceeds that budget, the endpoint returns `504 Gateway Timeout`.\n\n### 5. Add or Update Single Document\n\n```bash\ncurl -X POST 'http://localhost:8080/document?indexName=products' \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n    \"document\": { \"id\": \"14\", \"name\": \"New Name\", \"tags\": [\"x\"], \"year\": \"2021\" }\n  }'\n```\n\n`indexName` can also be sent in the JSON body instead of the query string.\n\n### 6. Delete Single Document\n\n```bash\ncurl -X DELETE 'http://localhost:8080/document?indexName=products\u0026id=14'\n```\n\n### 7. Save Index\n\n```bash\ncurl -X POST http://localhost:8080/save-controller \\\n  -H 'Content-Type: application/json' \\\n  -d '{ \"indexName\": \"products\" }'\n```\n\nIndexes are saved under `$SEARCH52_INDEX_DATA_DIR/\u003cindexName\u003e/engine.gob`; if `SEARCH52_INDEX_DATA_DIR` is not set, the service uses `./data`.\n\n### 8. Load Index\n\n```bash\ncurl -X POST http://localhost:8080/load-controller \\\n  -H 'Content-Type: application/json' \\\n  -d '{ \"indexName\": \"products\" }'\n```\n\nLoad is rollback-safe: a corrupt or invalid `engine.gob` does not replace an already-loaded in-memory index.\n\n### 9. Compact Index\n\n```bash\ncurl -X POST http://localhost:8080/compact-index \\\n  -H 'Content-Type: application/json' \\\n  -d '{ \"indexName\": \"products\" }'\n```\n\nCompaction removes tombstoned old document versions from all postings, filters, prefix arrays, fuzzy data, and document maps. The response includes before/after active and stored document counts.\n\n### 10. Health Check\n\n```bash\ncurl http://localhost:8080/health\n```\n\n```json\n{ \"status\": \"ok\", \"statusCode\": 200, \"duration\": \"5µs\", \"durationMs\": 0 }\n```\n\n---\n\n## Admin UI\n\nStart the service and open:\n\n```text\nhttp://localhost:8081\n```\n\nThe admin UI runs on a separate port and shares the same in-memory engine instance as the API server. It can:\n\n- list indexes and active document counts\n- create an index\n- add or update a single JSON document\n- delete a single document by ID\n- run search queries with optional filters\n- save and load indexes from disk\n- compact an index to remove tombstoned versions\n\nThe UI port can be changed with `SEARCH52_ADMIN_ADDR`; the API port can be changed with `SEARCH52_API_ADDR`.\n\n---\n\n## Testing\n\n```bash\n# Run all tests\ngo test -count=1 ./...\n\n# With race detection\ngo test -race -count=1 ./...\n\n# With filtered coverage\n./scripts/coverage.sh coverage.out\ngo tool cover -func=coverage.out\n```\n\n---\n\n## License\n\nMIT © mg52\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmg52%2Fsearch52","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmg52%2Fsearch52","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmg52%2Fsearch52/lists"}