https://github.com/ericmey/musubi
Musubi (結び) — Aoi's shared memory and thought layer. The braiding of threads between presences.
https://github.com/ericmey/musubi
ai-agents ai-memory cosign fastapi grpc homelab hybrid-search livekit mcp memory-system model-context-protocol obsidian ollama pydantic python qdrant rag retrieval-augmented-generation self-hosted vector-search
Last synced: about 1 month ago
JSON representation
Musubi (結び) — Aoi's shared memory and thought layer. The braiding of threads between presences.
- Host: GitHub
- URL: https://github.com/ericmey/musubi
- Owner: ericmey
- License: apache-2.0
- Created: 2026-04-01T16:23:54.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2026-04-21T22:26:48.000Z (2 months ago)
- Last Synced: 2026-04-21T22:28:00.180Z (2 months ago)
- Topics: ai-agents, ai-memory, cosign, fastapi, grpc, homelab, hybrid-search, livekit, mcp, memory-system, model-context-protocol, obsidian, ollama, pydantic, python, qdrant, rag, retrieval-augmented-generation, self-hosted, vector-search
- Language: Python
- Homepage: https://github.com/ericmey/musubi
- Size: 4.07 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 9
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md
- Agents: AGENTS.md
Awesome Lists containing this project
README
Musubi 結び
Shared memory for a small fleet of AI agents — three planes, local inference, a lifecycle engine that matures raw captures into a human-reviewable knowledge base.
---
Musubi (結び — *"to tie, to join, to bind"*) is a memory server built for the moment when a single AI assistant is not enough: you're running several, each with its own role — one drafts notes, one answers questions, one cleans up the vault at 3am — and they need a shared substrate so that what one learns, the others can use.
It is a standalone Python service. Every downstream interface (MCP, LiveKit, a CLI, a browser extension) is an adapter that depends on Musubi's SDK. The core owns the memory model and the API; adapters own the surface.
## The three planes
```
┌──────────────────────────────────────────────────────────────────────────┐
│ MUSUBI CORE │
│ │
│ episodic ──────► concept ──────► curated │
│ (raw captures) (synthesized (human-reviewed │
│ themes) Obsidian notes) │
│ │
│ + artifact plane (binary blobs — images, audio, pdfs) │
│ + thoughts plane (pub/sub channel for agent ↔ agent messaging) │
│ │
└──────────────────────────────────────────────────────────────────────────┘
```
- **Episodic** — every sentence, quote, and observation an agent ingests. Written fast, scored by an LLM for importance, matured to `matured` after a dwell window if no contradictions surface.
- **Concept** — a daily pass clusters mature episodics by shared topic + semantic similarity, asks an LLM to summarise each cluster into a `SynthesizedConcept`, checks for contradictions between concepts, and persists the result. Concepts reinforce (not duplicate) when similar clusters recur.
- **Curated** — concepts that clear a promotion gate (reinforcement count + importance + age) are rendered as markdown and written to an [Obsidian](https://obsidian.md) vault. A human reviewer sees them, edits them, moves them around. Edits flow back into Musubi through the vault sync.
A **lifecycle engine** runs five sweeps on cron: maturation (hourly), synthesis (03:00), promotion (04:00), demotion (05:00, sweep unreinforced rows out), reflection (06:00, writes a daily digest back to the vault). Each sweep is file-locked, idempotent, and emits structured events to a SQLite journal.
## Why not a single RAG index?
One reason: **lifecycle**. A plain vector store keeps everything forever and retrieves by cosine. Musubi's episodic rows expire on a TTL if they aren't matured; mature ones flow up through synthesis; promoted ones become curated rows the human owns. Nothing sits in a "everything I've ever said" bucket — the system makes opinions about what's worth keeping and surfaces them for review.
The other reason: **agent ↔ agent memory**. Thoughts are a first-class message channel — agents can `send` and `subscribe` without a coordinator process. It's not a chat log, it's a shared board of short-lived state.
Design choices are captured as ADRs in [`docs/Musubi/13-decisions/`](docs/Musubi/13-decisions/).
## Stack
- **Python 3.12**, `pydantic v2`, strict `mypy`, `ruff` format + lint.
- **Qdrant** for named-vector hybrid search (dense + sparse + rerank).
- **TEI (text-embeddings-inference)** for BGE-M3 dense + SPLADE sparse + BGE-reranker — all GPU-hostable, CPU-fallback OK.
- **Ollama** for LLM calls (maturation scoring, synthesis, promotion rendering, reflection). Defaults to Qwen 2.5 7B; any Ollama-tagged model works.
- **FastAPI + HTTPX** HTTP surface; **gRPC** generated from `proto/` (partial). Both exposed on the same port.
- **Docker Compose** for local / single-box deploy. **Ansible** playbooks for a managed-host rollout (`deploy/ansible/`). Every published image is [cosign](https://github.com/sigstore/cosign)-signed by digest, Trivy-scanned, and ships with a CycloneDX SBOM attestation.
## Try it
```bash
# 1. Clone
git clone https://github.com/ericmey/musubi && cd musubi
# 2. Install (Python 3.12 + uv required — https://docs.astral.sh/uv/)
make install
# 3. Run the local test suite
make check
```
A single-box Docker Compose deploy is laid out in [`deploy/ansible/templates/docker-compose.yml.j2`](deploy/ansible/templates/docker-compose.yml.j2); a first-deploy runbook lives at [`deploy/runbooks/upgrade-image.md`](deploy/runbooks/upgrade-image.md). Pin by digest — tags are mutable, digests aren't, and the rest of the repo (ansible, CI, SECURITY.md) verifies by digest:
```
ghcr.io/ericmey/musubi-core@sha256:
```
Find the digest for a specific release on its GitHub Release page (the `publish-core-image` workflow attaches it) or via `docker buildx imagetools inspect ghcr.io/ericmey/musubi-core:`.
Verify the signature before pinning in production:
```bash
cosign verify \
--certificate-identity-regexp 'https://github.com/ericmey/musubi/.*' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
ghcr.io/ericmey/musubi-core@sha256:
```
## Repository layout
```
src/musubi/ importable package
types/ shared pydantic types — the schema is the contract
store/ Qdrant layout, collection names, vector specs
embedding/ TEI client + Embedder protocol + FakeEmbedder
planes/ episodic / concept / curated / artifact / thoughts
retrieve/ scoring, hybrid search, fast/deep paths
lifecycle/ maturation / synthesis / promotion / demotion / reflection / runner
llm/ Ollama client + frozen prompt files (per-name versioned)
api/ FastAPI app, OpenAPI, /v1/* routes
sdk/ Python client
adapters/ MCP, LiveKit, OpenClaw (SDK + types only)
vault/ Obsidian watcher + writer + write-log
observability/ structured logging + Prometheus metrics
tests/ mirrors src/musubi/ path-for-path
docs/Musubi/ the architecture vault (Obsidian) — source of truth for design
deploy/ ansible, prometheus, grafana, docker-compose templates
```
## Status
**v1.0 — released.** The API shape is sealed, agent-as-tenant namespace model is in ([ADR 0030](docs/Musubi/13-decisions/0030-agent-as-tenant.md)), both downstream integrations (openclaw-livekit, openclaw-musubi plugin) are on the canonical API, and the release chain from conventional commit through pinned-digest PR is fully hands-off.
In v1.0:
- ✅ Three-plane memory (episodic / curated / concept) + artifacts + thoughts, per-plane collections, KSUID-addressed rows
- ✅ All five lifecycle sweeps (maturation, synthesis, promotion, demotion, reflection) running on cron, SQLite-journaled, with a hard three-strikes rejection cap on promotion
- ✅ Hybrid retrieval — dense BGE-M3 + sparse SPLADE + BGE-reranker + 2-segment cross-plane fanout in one call
- ✅ Full HTTP surface (gRPC partial; lives behind the runtime flag `MUSUBI_GRPC`, default off)
- ✅ Plane-aligned endpoint paths: `/v1/episodic`, `/v1/curated`, `/v1/concepts`, `/v1/artifacts`
- ✅ Agent-as-tenant namespace model: `//` ([ADR 0030](docs/Musubi/13-decisions/0030-agent-as-tenant.md))
- ✅ `Last-Event-ID` replay on `/v1/thoughts/stream` — reconnect without losing thoughts
- ✅ MCP + LiveKit + OpenClaw adapters (external repos), static-bearer auth model with per-agent token support
- ✅ Supply-chain: cosign + SBOM + Trivy on every published image
- ✅ Fully automated release chain — conventional commit → release PR auto-merges → tag → signed image → digest pin PR auto-merges. Operator runs `ansible-playbook update.yml` from the control host and that is the only human step.
- ✅ Operator tooling: `musubi promote force|reject` CLI, vault large-file skip-with-warning, rate-limited vault watcher
Post-v1.0:
- ⏳ Fleet orchestration — single-node today; multi-node HA is a post-1.0 design space and will need its own ADR
- ⏳ Auto-deploy pipeline (image publish is automated; host rollout is still operator-driven via ansible)
- ⏳ gRPC transport ADR ([#98](https://github.com/ericmey/musubi/issues/98)) — priority-low, not a 1.0 blocker
- ⏳ Vault-wide sweep to update illustrative `eric/...` examples to agent-as-tenant (normative specs already flipped; docs carry a banner pointing at [ADR 0030](docs/Musubi/13-decisions/0030-agent-as-tenant.md))
Roadmap detail lives in [`docs/Musubi/12-roadmap/`](docs/Musubi/12-roadmap/).
## Contributing
This is a personal project that's been opened up for others to follow along, fork, and riff on. Contributions are welcome but the bar is: opened issue → discussion → PR. See [CONTRIBUTING.md](CONTRIBUTING.md) for the workflow and conventions.
The internal design is captured in an Obsidian-style vault at [`docs/Musubi/`](docs/Musubi/). It's readable as-is on GitHub but renders best in Obsidian. Every architectural decision has an ADR; every working module has a `slice` spec with a Test Contract.
## Security
If you find a vulnerability, please don't open a public issue. See [SECURITY.md](SECURITY.md) for the disclosure process.
## License
[Apache 2.0](LICENSE) © 2025–2026 Eric Mey.