{"id":49182251,"url":"https://github.com/apajon/mempalace-migrator","last_synced_at":"2026-04-23T02:01:07.993Z","repository":{"id":352431132,"uuid":"1215123209","full_name":"apajon/mempalace-migrator","owner":"apajon","description":"Experimental tool for reconstructing MemPalace data across ChromaDB versions (no correctness guarantees).","archived":false,"fork":false,"pushed_at":"2026-04-19T14:43:59.000Z","size":86,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-19T16:19:58.467Z","etag":null,"topics":["ai-memory","chromadb","data-migration","data-reconstruction","experimental","mempalace","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apajon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-19T14:09:47.000Z","updated_at":"2026-04-19T14:44:02.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/apajon/mempalace-migrator","commit_stats":null,"previous_names":["apajon/mempalace-migrator"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/apajon/mempalace-migrator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apajon%2Fmempalace-migrator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apajon%2Fmempalace-migrator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apajon%2Fmempalace-migrator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apajon%2Fmempalace-migrator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apajon","download_url":"https://codeload.github.com/apajon/mempalace-migrator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apajon%2Fmempalace-migrator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32162611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-22T17:06:48.269Z","status":"online","status_checked_at":"2026-04-23T02:00:06.710Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-memory","chromadb","data-migration","data-reconstruction","experimental","mempalace","sqlite"],"created_at":"2026-04-23T02:01:06.113Z","updated_at":"2026-04-23T02:01:07.977Z","avatar_url":"https://github.com/apajon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mempalace-migrator\n\nReconstruction-based migration tool. Reads a MemPalace stored under\nChromaDB `0.6.x` and rebuilds it as a new palace under ChromaDB `1.x`.\n\nScope is narrow and intentionally constrained: one supported version\npair, one collection name, tested against a small fixture. Semantic\ncorrectness of the reconstructed palace is not guaranteed. See section\n4 for the exact supported scope.\n\n---\n\n## 1. What this project is\n\n`mempalace-migrator` reads a ChromaDB `0.6.x` SQLite database directly,\nextracts the documents and metadata it can identify, and reconstructs a\nnew ChromaDB `1.x` palace from that extracted material.\n\nIt works by **reconstruction**, not by in-place migration:\n\n- The source palace is opened in SQLite read-only URI mode (`mode=ro`)\n  and is never modified.\n- A new, separate target palace is built from scratch using the public\n  ChromaDB `1.x` Python client.\n- Extracted records are inserted via the `1.x` client API, which means\n  the embedding function used by `1.x` decides how vectors are produced.\n\nThere is no shared format between the two versions. The tool does not\nupgrade a database; it transcribes what it can read into a new one.\n\n**Current status:** all five pipeline stages are implemented.\n`migrate` runs the full pipeline end-to-end and writes a target palace.\n`analyze` remains available for read-only inspection without writing.\nThe supported scope is narrow: one version pair, one collection name.\nSee section 4.\n\n---\n\n## 2. What this project is NOT\n\n- Not an in-place upgrade tool. The source palace is never modified.\n  A partial write to the target is rolled back automatically on pipeline\n  failure, but a completed run cannot be undone without deleting the\n  target directory manually.\n- Not a general migration utility. Scope is hard-coded to the single\n  version pair in section 4.\n- Not a general ChromaDB conversion utility. It is hard-coded to one\n  source structure and one target version.\n- Not guaranteed to produce a target palace equivalent to the source.\n- Not guaranteed to preserve every record from the source.\n- Not guaranteed to produce semantically correct output, even when the\n  run exits with code `0`.\n- Not guaranteed to terminate successfully on inputs it has not been\n  tested against.\n- Not a substitute for taking a backup of the source data.\n\n---\n\n## 3. Limitations and risks\n\nRead this section before running the tool.\n\n- **Embedding vectors are not transferred.** ChromaDB `1.x` recomputes\n  embeddings using its own embedding function. Search results in the\n  reconstructed palace will not be identical to those in the source,\n  even when documents are byte-identical.\n- **Output may be structurally valid but semantically incorrect.** The\n  reconstructed palace may load, accept queries, and return results that\n  differ from the source in ways the tool cannot detect. Semantic\n  correctness is not guaranteed and cannot be verified by the tool.\n- **Completeness is not guaranteed.** Rows that fail per-row integrity\n  checks are excluded from the reconstruction and listed in the report.\n  The tool continues; it does not refuse to produce a partial output.\n- **Correctness is not guaranteed.** The tool verifies what it can\n  (PRAGMA integrity, ID uniqueness, document presence, metadata\n  resolvability). It cannot verify what it does not know to look for.\n- **The tool may fail on inputs that other tooling considers valid.**\n  Detection requires a manifest with a recognised `chromadb_version`.\n  Palaces without one are rejected, even if they are otherwise readable.\n- **Tested coverage is narrow.** Only the version pair listed in\n  section 5 has been exercised. Behaviour on other versions, schemas\n  produced by other tooling, or palaces written by patched ChromaDB\n  builds is undefined.\n- **Concurrent access to the source is not detected reliably.** The\n  tool refuses to run when an uncheckpointed WAL file is present, but\n  it cannot detect a concurrent reader-writer outside that signal.\n- **Manifest authenticity is not verified.** A forged or stale manifest\n  will be trusted.\n\nA run that exits with code `0` means the tool completed without raising\na critical error. It does **not** mean the reconstructed palace is\ncorrect.\n\n---\n\n## 4. Supported scope\n\nThe tool refuses to run outside this list.\n\n| Source ChromaDB | Target ChromaDB |\n|-----------------|-----------------|\n| `0.6.3`         | `1.5.7`         |\n\nDetection requires a manifest file (`mempalace-bridge-manifest.json`)\nin the source palace directory containing both `compatibility_line` and\n`chromadb_version` fields. Without these, the tool aborts before\nextraction.\n\nThere are no plans for additional version pairs in this repository.\nEach new pair requires re-validation against real palaces.\n\n---\n\n## 5. Philosophy\n\nThese are the design constraints the codebase is held to. They are\ndescribed here so that contributors and users understand why the tool\nbehaves as it does.\n\n- **Traceability over convenience.** Every excluded row, every\n  inconsistency, and every ambiguity is recorded as a structured\n  anomaly in the report. The tool does not silently drop data.\n- **Explicit reporting over silent success.** Each report contains an\n  `explicitly_not_checked` list naming the conditions the tool does\n  not verify. Silence in the output is not a guarantee.\n- **Strict boundaries over broad support.** Anything outside the\n  supported version pair, or below the required detection confidence,\n  is rejected. There is no `--force` option.\n- **Read-only by construction.** The source database is opened in\n  SQLite `mode=ro`. The target palace is built in a separate location.\n  There is no codepath that writes to the source.\n- **Failure model is documented, not improvised.** Critical conditions\n  raise and abort. Per-row issues are collected and reported. The\n  difference is defined in code, not left to the caller.\n\n---\n\n## 6. Quickstart\n\n\u003e **Warning: back up the source palace before running this tool.**\n\u003e Even though the source is opened read-only, the surrounding workflow\n\u003e (renames, moves, scripted cleanup) is the operator's responsibility.\n\u003e\n\u003e **Warning: do not point the target path at an existing palace** unless\n\u003e you intend to overwrite it. The tool may refuse, but do not rely on this.\n\u003e\n\u003e **Warning: inspect the report after every run.** A successful exit\n\u003e code is not a correctness claim. Semantic correctness is not\n\u003e guaranteed.\n\nInstall:\n\n```bash\nuv venv .venv --python 3.12\nuv pip install --python .venv/bin/python -e .\n```\n\nAvailable commands:\n\n```bash\n# Read-only: detect format and extract records. No target written.\n.venv/bin/mempalace-migrator analyze /path/to/source-palace\n.venv/bin/mempalace-migrator analyze /path/to/source-palace --json-output\n\n# Full migration: detect → extract → transform → reconstruct → validate.\n# TARGET must not exist or be empty. Partial writes are rolled back on failure.\n.venv/bin/mempalace-migrator migrate /path/to/source-palace --target /path/to/new-palace\n.venv/bin/mempalace-migrator migrate /path/to/source-palace --target /path/to/new-palace --json-output\n\n# Inspect without writing: detect, extract, transform, validate (no target).\n# Parity checks are listed as not-performed (no reconstruction ran).\n.venv/bin/mempalace-migrator inspect /path/to/source-palace\n\n# Re-render a JSON report saved from a previous run.\n.venv/bin/mempalace-migrator report /path/to/report.json\n```\n\n`analyze` runs detection and extraction only. It does not write a\ntarget palace.\n\n`migrate` runs all five stages and writes a new ChromaDB `1.x` palace at\n`--target`. The source palace is never modified. See section 3 for\nlimitations that remain even after a successful run.\n\n---\n\n## 7. Output and reporting\n\nEach run produces a structured report printed to stdout, or as JSON\nwith `--json-output`. The report contains:\n\n- `detection`: classification, numeric confidence, source version,\n  evidence list (every fact and inconsistency observed).\n- `extraction`: collection name, PRAGMA integrity check result, full\n  list of `failed_rows` with the reason each row was excluded.\n- `extraction_stats`: `total_rows`, `parsed_rows`, `failed_rows`,\n  `parse_rate`.\n- `anomalies`: each anomaly is a structured object with\n  `type`, `severity` (`low`, `medium`, `high`, `critical`), `stage`,\n  `message`, and a `context` dictionary.\n- `anomaly_summary`: counts by severity and by type.\n- `explicitly_not_checked`: the list of conditions the tool does not\n  verify. Always present, always non-empty.\n\nOperators are expected to inspect the report. A non-empty\n`failed_rows` list means data was excluded from the reconstruction. A\nnon-empty `anomalies` list with `severity \u003e= high` means the run\ncontains conditions that warrant manual review before the output is\ntrusted for any purpose.\n\nExit codes:\n\n| Code | Meaning |\n|------|---------|\n| `0`  | Pipeline completed without raising a critical error |\n| `2`  | Detection failed (unsupported format, version, or insufficient confidence) |\n| `3`  | Extraction failed at a critical pre-flight check |\n| `6`  | Report could not be built |\n| `10` | Unexpected error (use `--debug` to surface the traceback) |\n\n---\n\n## 8. Target audience\n\nThis tool is intended for operators who:\n\n- understand the structural differences between ChromaDB `0.6.x` and\n  `1.x`,\n- can read SQLite directly to verify what the tool reports,\n- accept that semantic correctness of the output is not guaranteed and\n  that the reconstructed palace may need to be discarded after\n  inspection,\n- do not need a turnkey upgrade path.\n\nIf you need a supported migration product, this is not it.\n\n---\n\n## 9. Related projects\n\n- **[mempalace-mcp-bridge](https://github.com/apajon/mempalace-mcp-bridge)**\n  — the stable bridge between MemPalace and the Model Context Protocol.\n  It is the production-oriented project. `mempalace-migrator` exists\n  separately so that experimental reconstruction work does not affect\n  the bridge's stability or its supported scope.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapajon%2Fmempalace-migrator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapajon%2Fmempalace-migrator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapajon%2Fmempalace-migrator/lists"}