Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/dusk-network/stroma

Neutral corpus and indexing substrate for local semantic retrieval
https://github.com/dusk-network/stroma

Last synced: 3 months ago
JSON representation

Neutral corpus and indexing substrate for local semantic retrieval

Awesome Lists containing this project

README 

          
# Stroma



Stroma is a neutral corpus and indexing substrate.



It owns the lowest-level operations needed to ingest text artifacts, chunk them, embed them, persist them in SQLite plus `sqlite-vec`, retrieve semantically close sections, and call OpenAI-compatible embedding and chat completion endpoints over a shared HTTP substrate. Callers consume Stroma through its APIs and treat the SQLite snapshot as an opaque local artifact. It does not own governance, specifications, compliance, drift analysis, prompt templates, product-specific output semantics, MCP, or CLI workflows.



## Scope



Stroma is for products that need a reusable text corpus layer with:



- canonical records with deterministic content fingerprints

- pluggable chunking strategies (`chunk.Policy` — `MarkdownPolicy` default, `KindRouterPolicy` for per-record-kind dispatch, `LateChunkPolicy` for parent/leaf hierarchy)

- pluggable embedders (`Embedder` / `ContextualEmbedder`) with a deterministic fixture and an OpenAI-compatible HTTP embedder

- OpenAI-compatible chat completion client (`chat.OpenAI`) sharing the same substrate as `embed.OpenAI`: retry with `Retry-After` (capped), classified failures (`auth` / `rate_limit` / `timeout` / `server` / `transport` / `schema_mismatch` / `dependency_unavailable`), preserved lower-level causes on provider errors, APIToken redaction, and a product-neutral structured JSON helper

- hybrid retrieval: dense vector + FTS5, fused via a pluggable `FusionStrategy` (`RRFFusion` by default) with per-arm provenance surfaced to downstream rerankers

- quantization knobs: `float32` (default), `int8` (4× smaller), `binary` (1-bit sign-packed `vec0` prefilter that is 32× smaller for the prefilter representation; full-precision vectors are retained in a companion table for cosine rescoring, so total snapshot size is not 32× smaller)

- optional Matryoshka prefilter at a truncated dimension with full-dim cosine rescore (`SearchParams.SearchDimension`)

- atomic rebuilds and incremental `Update` with embedding reuse at the section level, chaining schema migrations v2 → v3 → v4 → v5 in one transaction



Stroma is not for:



- spec governance

- source discovery or repository scanning

- code compliance or doc drift analysis

- prompt templates, system prompts, or semantic interpretation of structured chat responses

- product-specific adapters and transports



## Packages



- `corpus` — canonical record model, `NewRecord` helper, `Normalize`, deterministic `Fingerprint`

- `chunk` — `Policy` interface with `MarkdownPolicy`, `KindRouterPolicy`, `LateChunkPolicy`; `MarkdownWithOptions` returns `ErrTooManySections` when a body exceeds the DoS cap

- `embed` — `Embedder` and `ContextualEmbedder` interfaces; deterministic `Fixture`; OpenAI-compatible HTTP embedder with `MaxBatchSize` batching, deadline scaling across batches, and `APIToken` redaction in `String`/`GoString`/`LogValue`

- `chat` — OpenAI-compatible chat completion client (`chat.OpenAI`, `chat.Message`, `ChatCompletionText`, `ChatCompletionJSON`); tolerates string and multi-part array content; structured JSON responses decode into caller-owned targets and malformed JSON returns `schema_mismatch`; `APIToken` redaction parity with `embed.OpenAIConfig`

- `provider` — shared HTTP substrate used by `embed` and `chat`: retry with capped `Retry-After`, response-size bounding, negative `MaxRetries` normalization to zero, and a stable `FailureClass` taxonomy surfaced via `*provider.Error`. Callers branch on `FailureClass` to retry / degrade / propagate, and can unwrap lower-level transport/decode causes where available

- `store` — SQLite readiness probes, `sqlite-vec` readiness, quantization blob helpers (`QuantizationFloat32` / `QuantizationInt8` / `QuantizationBinary`)

- `index` — atomic `Rebuild` plus streaming `RebuildFromSource` with embedding reuse and explicit reuse diagnostics, incremental `Update` with `MaxPlannedRecords` batching guard, long-lived `Snapshot` readers, `Stats`, hybrid `Search` with provenance and explicit `MaxSearchLimit`, `ExpandContext` for parent/neighbor walks



## Retrieval Evidence And Batch Use



Use `OpenSnapshot` when issuing many searches against one built index. A `Snapshot` is safe for concurrent reads; callers own the concurrency limit, so use a bounded worker pool or semaphore sized for the host and workload, then close the snapshot after all searches and context expansions finish.



For durable evidence handles, persist at least:



- `Stats.ContentFingerprint` from the opened snapshot, identifying the indexed content generation

- `SearchHit.ChunkID`, identifying a chunk only within that snapshot generation

- `SearchHit.Ref` plus any caller-needed record metadata or `SourceRef`



`ChunkID` is not a cross-rebuild identity. Before expanding a previously saved hit, reopen the snapshot, compare `Stats.ContentFingerprint` with the saved value, and rerun search if it differs. `SearchHit.Score` and `HitProvenance` are ranking evidence for the query that produced the hit; keep them for audit/debugging, but do not use them as identity fields.



`ExpandContext(hit.ChunkID, opts)` returns the hit chunk plus requested parent/neighbor sections in document order. On flat snapshots, parent expansion is a no-op and neighbors are same-record chunks. On hierarchical snapshots, parent expansion follows `parent_chunk_id` one level and neighbors stay in the same sibling group. A missing chunk returns an empty slice and nil error, which lets wrappers treat stale handles as "not found" after they have already checked the content fingerprint.



## Rebuild Streaming, Update Memory And Batching



Use `RebuildFromSource` when the corpus lives behind a filesystem, database, or

other lazy loader and callers should not build one full `[]corpus.Record` before

indexing. A `RecordSource` returns one `corpus.Record` at a time; Stroma

normalizes each record, chunks and embeds records in bounded internal batches,

writes them to the staging snapshot, rejects duplicate refs through the

snapshot's primary key, and computes the final content fingerprint from

persisted `(ref, content_hash)` pairs. This keeps record bodies bounded to the

current source record plus the current planned batch; Stroma retains that

batch's chunks/vectors until it is flushed. `Rebuild` remains the slice

convenience API and still sorts the provided records before delegating to the

same write path. `RebuildFromSource` preserves source order, so source order

determines snapshot-local `ChunkID` assignment; callers that need repeatable

chunk IDs across streaming rebuilds should emit records in a stable order.



`RebuildFromSource` keeps the same atomic staging-file contract as `Rebuild`: a

source, chunking, embedding, or duplicate-ref error discards the staged file and

leaves the destination snapshot unchanged. It is not a resumable checkpointing

API. The staging SQLite transaction stays open while the source is consumed and

embeddings are produced, so callers should keep `RecordSource.Next` responsive to

context cancellation and avoid doing unrelated slow work inside it.



`Update` chunks, contextualizes, reuse-plans, and embeds added/replaced records before opening its SQLite write transaction. That keeps external embedder latency out of the transaction and preserves stale-plan rollback semantics, but the pre-transaction plan retains each added record's chunks, reuse decisions, and new vectors until the write phase.



For large ingests, split added records into caller-sized batches and set `UpdateOptions.MaxPlannedRecords` to that batch size. A batch above the cap fails before embedding and before the write transaction starts with an error wrapping `index.ErrUpdatePlanTooLarge`, so callers can retry smaller batches without changing the on-disk snapshot. `MaxChunkSections` still bounds per-record section expansion.



## Example



```go

ctx := context.Background()



records := []corpus.Record{

    corpus.NewRecord(

        "widget-overview",

        "Widget Overview",

        "# Overview\n\nWidgets are synchronized in batches.",

    ),

}



fixture, err := embed.NewFixture("fixture-demo", 16)

if err != nil {

    log.Fatal(err)

}



if _, err := index.Rebuild(ctx, records, index.BuildOptions{

    Path:     "stroma.db",

    Embedder: fixture,

}); err != nil {

    log.Fatal(err)

}



hits, err := index.Search(ctx, index.SearchQuery{

    Path: "stroma.db",

    SearchParams: index.SearchParams{

        Text:     "synchronized batches",

        Limit:    5,

        Embedder: fixture,

        // Fusion / Reranker / SearchDimension are optional; zero values

        // give hybrid RRF over vector+FTS with the full stored dimension.

    },

})

if err != nil {

    log.Fatal(err)

}



fmt.Println(hits[0].Ref)

```



See the [v2.0.0 release notes](https://github.com/dusk-network/stroma/releases/tag/v2.0.0) for the full API surface.



## Status



v2.0.0 (current) ships the stable substrate: hybrid retrieval, pluggable fusion, quantization, matryoshka, contextual retrieval, adaptive chunking, and incremental update. Higher-order products should consume the library rather than re-embedding their own indexing substrate.