{"id":50119502,"url":"https://github.com/yonk-labs/chunkshop","last_synced_at":"2026-05-31T00:01:42.376Z","repository":{"id":353150776,"uuid":"1215439514","full_name":"yonk-labs/chunkshop","owner":"yonk-labs","description":"Standalone ingest-to-pgvector with a built-in chunker × embedder bakeoff. Python + Rust.","archived":false,"fork":false,"pushed_at":"2026-05-30T17:09:03.000Z","size":64766,"stargazers_count":0,"open_issues_count":5,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-30T19:05:20.396Z","etag":null,"topics":["chunking","embeddings","fastembed","onnx","pgvector","rag","vector-search"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yonk-labs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-19T23:00:55.000Z","updated_at":"2026-05-30T17:09:06.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/yonk-labs/chunkshop","commit_stats":null,"previous_names":["yonk-labs/chunkshop"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/yonk-labs/chunkshop","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yonk-labs%2Fchunkshop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yonk-labs%2Fchunkshop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yonk-labs%2Fchunkshop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yonk-labs%2Fchunkshop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yonk-labs","download_url":"https://codeload.github.com/yonk-labs/chunkshop/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yonk-labs%2Fchunkshop/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33714033,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-30T02:00:06.278Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chunking","embeddings","fastembed","onnx","pgvector","rag","vector-search"],"created_at":"2026-05-23T18:04:13.905Z","updated_at":"2026-05-31T00:01:42.356Z","avatar_url":"https://github.com/yonk-labs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# chunkshop\n\n[![CI](https://github.com/yonk-labs/chunkshop/actions/workflows/ci.yml/badge.svg)](https://github.com/yonk-labs/chunkshop/actions/workflows/ci.yml)\n[![PyPI](https://img.shields.io/pypi/v/chunkshop?label=pypi\u0026color=blueviolet)](https://pypi.org/project/chunkshop/)\n[![crates.io](https://img.shields.io/crates/v/chunkshop-rs?label=crates.io\u0026color=blueviolet)](https://crates.io/crates/chunkshop-rs)\n[![status: beta](https://img.shields.io/badge/status-beta-yellow)](python/pyproject.toml)\n[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue)](python/pyproject.toml)\n[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)\n\nA small, standalone, embeddable ingestion tool. Pulls text from a source,\nchunks it, embeds it, optionally tags it, and lands the result in a vector\ntable on one of **four supported backends — Postgres + pgvector, MariaDB\n11.7+, SQLite + sqlite-vec, or ClickHouse 24.10+.** Designed to be consumed\nas a library or driven from the command line.\n\n**One YAML config = one end-to-end ingest (\"cell\").** Same YAML, two\nlanguages: Python is the reference; Rust ships to crates.io. Vectors written\nby either are interchangeable.\n\n## v0.6.0 — incremental sync primitives + RawStore (SP-1) + Rust parity (RM-B)\n\n- **SP-1 sync primitives** — `SyncMode` enum (`full_resync` / `cursor` /\n  `fingerprint`), `IncrementalSource` and `PrunableSource` protocols/traits,\n  `StaleCursorError`, `Document.fingerprint`. Cursor wire format is\n  byte-identical Python ↔ Rust.\n- **`pg_table` tuple cursor** — `(updated_at, id::text) \u003e (?, ?)` defends\n  against silent row loss at boundary timestamps. Activated by setting\n  `updated_at_column` in YAML. Python commit `ff01268`; Rust mirror in RM-B\n  Task 2.\n- **`s3` ETag IncrementalSource** — cursor is `{key: etag}` map; unchanged\n  ETags skip the GET. Cross-implementation cursor compatible.\n- **`http` depth-crawl + ETag/Last-Modified cursor + robots.txt** — BFS\n  link crawl with `crawl_depth`, conditional GETs (`If-None-Match` /\n  `If-Modified-Since`), polite delays, configurable `User-Agent`.\n- **`RawStore` primitive** — pluggable storage for the original bytes\n  (filesystem + S3). SHA-256-keyed paths defend against `doc_id`\n  traversal. LocalRawStore byte-identical to Python's; S3RawStore stores\n  fingerprint in S3 object metadata for cross-impl compatibility.\n- **Rust covers all the SP-1 surfaces** — see `rust/README.md` for the\n  at-a-glance parity table. The chunkshop-connectors plugin layer\n  (gdrive/github/blob/rss/notion/dropbox/gitlab + 20 stubs), the codeparse\n  foundation, the `code_aware` / `symbol_aware` chunkers, the\n  `code_relationships` / `code_summary` extractors, OAuth providers, and\n  the PDF/DOCX/PPTX/XLSX file parsers are explicitly Python-only by design\n  (spec D6).\n\n## v0.5.0 — modular backends + document-table groundwork\n\n- **4 sink backends** (postgres, mariadb, sqlite, clickhouse) — all 4 in\n  Python AND Rust.\n- **9 sources** (`files`, `json_corpus`, `http`, `s3`, `inline`,\n  `pg_table`, `mariadb_table`, `sqlite_table`, `clickhouse_table`).\n- **The 16-cell cross-backend matrix** — every DB-source × DB-sink combo\n  round-trips in both languages. Pinned in CI (`cargo test --test\n  cross_backend_matrix` / `pytest tests/chunkshop/test_cross_backend_matrix.py`).\n- **Cross-backend bakeoff** — one YAML, one `chunkshop bakeoff` command,\n  leaderboards across all 4 backends side-by-side (Python; Rust bakeoff is\n  PG-only for now).\n- **Opt-in Postgres document table** — Python/Postgres can write a companion\n  one-row-per-document table with `target.documents.enabled: true`. This is\n  Python/Postgres-only in v0.5.0; Rust rejects enabled document stores until\n  Rust/Postgres parity lands.\n- **Identical retrieval quality across backends.** The v0.4.0 validation\n  bakeoff (NTSB corpus, 20 docs, 12 gold queries, 2 chunkers, 1 embedder)\n  produced **identical MRR (0.903 sentence_aware, 0.896 hierarchy) on all 4\n  backends.** Differences are wall time, not accuracy.\n\n## Read side — hybrid search + Fast-mode summarization\n\nchunkshop also ships an in-process **read API** over the tables it writes:\n`semantic_search` (vector top-K), `keyword_search` (full-text), and\n`hybrid_search` (both legs, RRF or weighted fusion) — on all four backends\n(pg/sqlite/mariadb full FTS; clickhouse degraded by design). On top of it,\n`summarize_hits` is a **Fast-mode RAG** helper: collapse the K retrieved chunks\ninto one query-biased summary before sending to an LLM — **~90% fewer input\ntokens for ~2–3 ms**, costing about one query in ten of accuracy. See\n[`docs/hybrid-search.md`](docs/hybrid-search.md) (numbers:\n[`docs/fast-mode-rag-benchmarks.md`](docs/fast-mode-rag-benchmarks.md)).\n\n## Headline benchmarks\n\nThree reproducible benches in [`docs/samples/benchmarks/`](docs/samples/benchmarks/),\nfull writeup in [`docs/benchmarks.md`](docs/benchmarks.md). Numbers from a\n24-core / 122 GiB box with all 4 backends on `localhost`.\n\n| Bench | Setup | Result |\n|---|---|---|\n| **HNSW vs brute-force** (PG) | sentence_aware + bge-small-int8, 3 corpus sizes | At 75 chunks: HNSW 0.75× (slower). At 1k chunks: parity. **At 3.8k chunks: 4.20× faster query.** MRR identical at every scale. |\n| **Concurrent ingest** | `chunkshop orchestrate` on 8 PG cells, sentence_aware + bge-small | c=1: 42.3s. c=2: 1.71× speedup (85% efficient). **c=4: 2.45× speedup (61% efficient — sweet spot).** c=8: 2.81× (35%, contention). |\n| **8k-chunk throughput** | SCOTUS, 8079 chunks, all 4 backends, `hnsw: true` everywhere | Query mean: SQLite 3.21ms ≈ MariaDB 3.66ms \u003c PG+HNSW 9.22ms \u003c CH 15.06ms. All ≤16ms. |\n\nMariaDB had a 158ms cosine query in pre-v0.4.1 because chunkshop's query\nshape bypassed the MariaDB `VECTOR INDEX` (which only accelerates\n`VEC_DISTANCE_EUCLIDEAN`, not cosine). The v0.4.1 sink uses a hybrid\nquery — euclidean in `ORDER BY`, cosine in `SELECT` — closing the cliff to\n3.66ms (43× speedup). For L2-normalized embeddings (every chunkshop\nembedder) the ranking is mathematically equivalent. Write-up:\n[`docs/benchmarks.md`](docs/benchmarks.md) → Read 2.\n\nToday PG with `hnsw: true` is the production sweet spot; SQLite is the\nzero-ops dark horse at \u003c1M chunks; MariaDB is now competitive on query.\n\n## Pipeline\n\n```mermaid\nflowchart LR\n    S[Source\u003cbr/\u003efiles · json_corpus · http · s3\u003cbr/\u003epg_table · mariadb_table\u003cbr/\u003esqlite_table · clickhouse_table\u003cbr/\u003einline] --\u003e F[Framer\u003cbr/\u003eidentity · heading_boundary\u003cbr/\u003eregex_boundary · jsonpath]\n    F --\u003e C[Chunker\u003cbr/\u003esentence_aware · fixed_overlap\u003cbr/\u003ehierarchy · neighbor_expand\u003cbr/\u003esemantic · summary_embed\u003cbr/\u003ehierarchical_summary]\n    C --\u003e E[Embedder\u003cbr/\u003efastembed · ONNX · int8 or fp32]\n    E --\u003e X[Extractor\u003cbr/\u003enone · rake_keywords · keybert_phrases\u003cbr/\u003espacy_entities · lang_detect · composite]\n    X --\u003e SK[Sink\u003cbr/\u003epostgres · mariadb\u003cbr/\u003esqlite · clickhouse]\n    SK --\u003e DB[(Vector table + index)]\n```\n\nEach arrow is a contract — swap any box without touching the others. See\n[`docs/architecture.md`](docs/architecture.md) for the per-module\nbreakdown and the trait surface that makes the matrix work.\n\n## The user journey (start here)\n\nchunkshop has one canonical loop. Every adoption goes through these five\nsteps:\n\n1. **Bring real data.** Your corpus — sales notes, NTSB reports, court\n   rulings, a docs site, whatever you actually need to retrieve over.\n2. **Write a small gold set.** ~10 hand-crafted queries, each paired with\n   the doc that *should* rank #1.\n3. **Run the bakeoff.** `chunkshop bakeoff` runs every (chunker × embedder\n   × backend) combo against your corpus and gold set. Out comes a\n   leaderboard and a `recommended.yaml` — the recipe that won on *your*\n   data on *your* backend.\n4. **Ship the recommended cell.** `chunkshop ingest --config recommended.yaml`\n   runs that recipe in production.\n5. **New corpus → repeat from step 2.** Each domain gets its own bakeoff;\n   each production pipeline gets its own tuned cell.\n\nThe bakeoff is **step 1 of every adoption**, not a sample. It's the\nexperiment that picks the recipe you'll ship with.\n\n```bash\n# 1a. Install from source when you want samples, dev tooling, or the\n#     freshest branch work:\ngit clone https://github.com/yonk-labs/chunkshop \u0026\u0026 cd chunkshop\ncd python \u0026\u0026 uv sync --extra dev --extra all-backends \u0026\u0026 cd ..\n\n# 1b. Install the published Python package:\npip install 'chunkshop[all-backends]'\n\n# 2. Pick your backend. Postgres is the default; SQLite for zero-server.\nexport CHUNKSHOP_DSN=\"postgresql://postgres:postgres@localhost:5432/mydb\"\n\n# 3. Run the canonical demo: bakeoff against the NTSB aviation-accident corpus\nchunkshop bakeoff --config docs/samples/bakeoff-ntsb/bakeoff-ntsb.yaml --yes\n\n# 4. Take the recommended cell to production\nchunkshop ingest --config skill-output/bakeoff/ntsb_bakeoff/recommended.yaml\n```\n\nThe walkthrough — install → bakeoff → recommended → ingest, with the\nNTSB corpus as the worked example — lives in\n[**`docs/getting-started.md`**](docs/getting-started.md).\n\nAlready past step 3 and just want the runtime? `chunkshop ingest --config\n\u003cyour-cell\u003e.yaml`. Same for Rust: `chunkshop-rs ingest --config\n\u003cyour-cell\u003e.yaml`.\n\n## Pick your backend\n\n| Backend | When to use it | Engine doc |\n|---|---|---|\n| **Postgres** + pgvector | Default. Mature, full feature surface, HNSW. | [`docs/engines/postgres.md`](docs/engines/postgres.md) |\n| **MariaDB 11.7+** | MySQL-family stack; no extension install. | [`docs/engines/mariadb.md`](docs/engines/mariadb.md) |\n| **SQLite** + sqlite-vec | Embedded / no server. Notebooks, CI, edge. | [`docs/engines/sqlite.md`](docs/engines/sqlite.md) |\n| **ClickHouse 24.10+** | OLAP / append-only / big corpora. | [`docs/engines/clickhouse.md`](docs/engines/clickhouse.md) |\n\nMixing sources from one engine into sinks on another is fully supported and\ntest-pinned. See [`docs/mixing-sources-and-sinks.md`](docs/mixing-sources-and-sinks.md).\n\n## Status\n\n| Impl | Path | State |\n|---|---|---|\n| Python reference | `python/` | Published on [PyPI](https://pypi.org/project/chunkshop/). All features. int8 default. |\n| Rust | `rust/` | Published on [crates.io](https://crates.io/crates/chunkshop-rs). Full pipeline + 4 backends. Bakeoff is PG-only (multi-target Rust bakeoff is a v0.4.1 follow-up). Orchestrator is Python-only. |\n| Go | `go/` | Not started. |\n\n### What \"parity\" means and doesn't mean\n\n| Layer | Python | Rust |\n|---|---|---|\n| Source / framer / chunker / embedder / extractor / sink | ✅ all 4 backends | ✅ all 4 backends |\n| `Pipeline` (inline / library mode) | ✅ | ✅ |\n| `chunkshop ingest` (one YAML → one cell) | ✅ | ✅ |\n| **16-cell cross-backend matrix (4 sources × 4 sinks)** | ✅ | ✅ |\n| `chunkshop bakeoff` (matrix → leaderboard → recommended.yaml) | ✅ multi-backend | ✅ PG-only |\n| `chunkshop orchestrate` (N cells as parallel subprocesses) | ✅ | ❌ |\n| `target.documents` companion document table | ✅ Postgres only | ❌ fails loudly |\n| Embedder registry breadth | full fastembed catalogue + custom-registered HF | BGE int8 (bit-near-exact) + nomic v1.5 + stock fastembed-rs catalogue + YAML-driven HF |\n\n## Defaults\n\nThe example config ships with `chunker.type: hierarchy` and\n`embedder.model_name: Xenova/bge-base-en-v1.5-int8`.\n\n**Chunker choice is benchmark-backed.** chunkshop's factorial on a 772-doc\nlegal QA corpus (30 gold questions, `gpt-4.1-mini` answer + judge) found:\n\n- **Hierarchy chunker wins across every embedder column** — prepending the\n  section heading to each embedded chunk adds free framing context.\n- **int8 \u003e= fp32 in aggregate** (160 vs 152 fully_correct across 12 cells)\n  with 2× faster ingest.\n- Zero hallucinations across 720 answers — prompt discipline, not model\n  choice.\n\n**Embedder default is MTEB-backed.** `bge-base` beats `bge-small` by ~3–5\npoints on public retrieval benchmarks (MTEB). Swap to\n`Xenova/bge-small-en-v1.5-int8` for a smaller footprint (~35 MB, 384 dim)\nor `nomic-ai/nomic-embed-text-v1.5-Q` for long-context (8k tokens). Run\nthe factorial configs against your own corpus to confirm. See\n[`docs/embedders.md`](docs/embedders.md).\n\n## YAML shape\n\nOne YAML = one cell = one end-to-end ingest. Six sections (framer optional,\ndefaults to identity):\n\n| Section | Types |\n|---|---|\n| source | files · json_corpus · http · s3 · pg_table · mariadb_table · sqlite_table · clickhouse_table · inline |\n| framer | identity (default) · heading_boundary · regex_boundary · jsonpath |\n| chunker | sentence_aware · fixed_overlap · hierarchy · neighbor_expand · semantic · summary_embed · hierarchical_summary |\n| embedder | fastembed (ONNX via `fastembed` in Python, `ort` in Rust) |\n| extractor | none · rake_keywords · keybert_phrases · spacy_entities · lang_detect · composite (opt-in extras) |\n| target | postgres · mariadb · sqlite · clickhouse; `mode: overwrite \\| append \\| create_if_missing`; `source_tag` + `promote_metadata` for multi-source tables |\n\nFull field-by-field reference: [`python/README.md`](python/README.md).\nPer-engine specifics: [`docs/engines/`](docs/engines/).\n\n## Target table schema (PG variant, others structurally identical)\n\n```sql\nCREATE TABLE {schema}.{table} (\n    id                  text PRIMARY KEY,        -- \"{doc_id}::{seq_num}\"\n    doc_id              text NOT NULL,\n    seq_num             int  NOT NULL,\n    original_content    text NOT NULL,           -- raw chunk, for grep / fact-match / audit\n    embedded_content    text NOT NULL,           -- what was embedded (may include heading prefix, neighbors)\n    tags                text[] NOT NULL DEFAULT '{}',\n    metadata            jsonb NOT NULL DEFAULT '{}',\n    embedding           vector({dim}) NOT NULL,\n    source              text,                    -- write-once provenance\n    created_at          timestamptz NOT NULL DEFAULT now()\n);\n-- plus: UNIQUE on (doc_id, seq_num); HNSW index on embedding if hnsw=true\n```\n\nMariaDB uses `VECTOR(N)` + `JSON` columns; SQLite uses a two-table dance\n(main + vec0 virtual); ClickHouse uses `Array(Float32)` + `MergeTree`\nengine. Storage shape is identical; only column types differ. See\n[`docs/storage-model.md`](docs/storage-model.md) and per-engine docs.\n\nPython/Postgres can also write an opt-in companion document table with\n`target.documents.enabled: true`; that support is not universal backend or\nRust parity yet. See [`docs/storage-model.md`](docs/storage-model.md).\n\n## Documentation\n\n### Start here\n\n| Doc | For |\n|---|---|\n| [`docs/executive-summary.md`](docs/executive-summary.md) | Two-page overview: what, why, current state, who should use it. |\n| [`docs/getting-started.md`](docs/getting-started.md) | Zero-to-retrieval end-to-end walkthrough. |\n| [`docs/architecture.md`](docs/architecture.md) | The trait surface, the pipeline, the cross-language parity story. |\n| [`docs/upgrading.md`](docs/upgrading.md) | Version-to-version migration notes. |\n| [`docs/benchmarks.md`](docs/benchmarks.md) | Measured performance + accuracy across backends. HNSW vs brute-force, concurrent-ingest scaling, 8k-chunk throughput. |\n\n### Pick your backend\n\n| Doc | For |\n|---|---|\n| [`docs/engines/postgres.md`](docs/engines/postgres.md) | Postgres + pgvector reference. |\n| [`docs/engines/mariadb.md`](docs/engines/mariadb.md) | MariaDB 11.7+ reference. |\n| [`docs/engines/sqlite.md`](docs/engines/sqlite.md) | SQLite + sqlite-vec reference. |\n| [`docs/engines/clickhouse.md`](docs/engines/clickhouse.md) | ClickHouse 24.10+ reference. |\n| [`docs/mixing-sources-and-sinks.md`](docs/mixing-sources-and-sinks.md) | How sources and sinks compose — the 16-cell matrix, when to mix. |\n\n### Pipeline references\n\n| Doc | For |\n|---|---|\n| [`docs/chunkers.md`](docs/chunkers.md) | Each chunker: what it does, when to pick it, oversize behavior. |\n| [`docs/embedders.md`](docs/embedders.md) | Embedder mechanics: registration patterns, BYO mode, A/B testing. |\n| [`docs/embedder-catalogue.md`](docs/embedder-catalogue.md) | Verified-working models, known-broken ones, dim / size. |\n| [`docs/extractors.md`](docs/extractors.md) | Each extractor: why use it, config, promoted-column pairing. |\n| [`docs/summaries.md`](docs/summaries.md) | summary_embed + hierarchical chunker reference. |\n| [`docs/storage-model.md`](docs/storage-model.md) | What chunkshop writes per row + how to query each payload. |\n| [`docs/query-clients.md`](docs/query-clients.md) | Query the ingested table from Python, JS/TS, Rust, Go (raw SQL). |\n| [`docs/hybrid-search.md`](docs/hybrid-search.md) | Read-side Python API: semantic / keyword / hybrid search + fusion, and `summarize_hits` Fast-mode RAG. |\n| [`docs/fast-mode-rag-benchmarks.md`](docs/fast-mode-rag-benchmarks.md) | Fast-mode token-savings + accuracy benchmarks behind `summarize_hits`. |\n\n### Tutorials\n\n| Doc | For |\n|---|---|\n| [`docs/tutorial.md`](docs/tutorial.md) | Start here. End-to-end walkthrough. |\n| [`docs/tutorial-bakeoff.md`](docs/tutorial-bakeoff.md) | Bakeoff walkthrough: pick the best combo for your corpus. |\n| [`docs/tutorial-multi-source.md`](docs/tutorial-multi-source.md) | Multi-source ingest: two cells, one table, filter by source. |\n| [`docs/tutorial-framers.md`](docs/tutorial-framers.md) | DocFramer walkthrough: heading splits + JSON expansion. |\n| [`docs/tutorial-metadata.md`](docs/tutorial-metadata.md) | Metadata extraction + promoted columns + filtered queries. |\n| [`docs/tutorial-semantic.md`](docs/tutorial-semantic.md) | Semantic chunker walkthrough. |\n| [`docs/tutorial-summaries.md`](docs/tutorial-summaries.md) | summary_embed + hierarchical walkthrough. |\n\n### Quickstart recipe cards\n\n| Doc | For |\n|---|---|\n| [`docs/quickstart-bakeoff.md`](docs/quickstart-bakeoff.md) | Common bakeoff shapes (embedder-only, chunker-only, full factorial). |\n| [`docs/quickstart-multi-source.md`](docs/quickstart-multi-source.md) | Schema-flex modes + append pre-flight. |\n| [`docs/quickstart-framers.md`](docs/quickstart-framers.md) | Which framer for which source shape. |\n| [`docs/quickstart-extractors.md`](docs/quickstart-extractors.md) | Copy-paste YAML per extractor. |\n| [`docs/quickstart-semantic.md`](docs/quickstart-semantic.md) | Semantic chunker knobs. |\n| [`docs/quickstart-summaries.md`](docs/quickstart-summaries.md) | summary_embed + hierarchical_summary recipes. |\n\n## Monorepo layout\n\n```\nchunkshop/\n├── python/                 reference implementation; published to PyPI\n│   ├── src/chunkshop/\n│   │   ├── backends/       Backend dialect/connect per engine\n│   │   ├── sinks/          Sink data-model semantics per engine\n│   │   ├── sources/        Source impls (files, http, s3, *_table, ...)\n│   │   ├── chunkers/\n│   │   ├── embedders/\n│   │   ├── extractors/\n│   │   ├── framers/\n│   │   ├── summarizers/\n│   │   ├── bakeoff/\n│   │   ├── runner.py\n│   │   ├── orchestrator.py\n│   │   └── cli.py\n│   └── tests/\n├── rust/                   second implementation; published to crates.io\n│   └── chunkshop/\n│       ├── src/\n│       │   ├── backends/\n│       │   ├── sinks/\n│       │   ├── sources/\n│       │   ├── bakeoff/\n│       │   ├── chunker.rs\n│       │   ├── embedder.rs\n│       │   ├── extractor.rs\n│       │   ├── framer.rs\n│       │   ├── runner.rs\n│       │   └── main.rs\n│       └── tests/\n├── go/                     planned\n└── docs/\n    ├── architecture.md\n    ├── mixing-sources-and-sinks.md\n    ├── engines/            ← per-engine docs (postgres, mariadb, sqlite, clickhouse)\n    ├── tutorial.md\n    ├── chunkers.md\n    ├── embedders.md\n    ├── extractors.md\n    ├── storage-model.md\n    └── samples/            ← sample corpora + runnable configs + bakeoffs\n```\n\n## License\n\nMIT. See `LICENSE`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyonk-labs%2Fchunkshop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyonk-labs%2Fchunkshop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyonk-labs%2Fchunkshop/lists"}