{"id":51192356,"url":"https://github.com/tiny-systems/store-module","last_synced_at":"2026-06-27T16:30:53.351Z","repository":{"id":358713334,"uuid":"1242506467","full_name":"tiny-systems/store-module","owner":"tiny-systems","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-18T18:14:56.000Z","size":21,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-18T20:27:51.435Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tiny-systems.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-18T13:41:57.000Z","updated_at":"2026-05-18T18:14:58.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/tiny-systems/store-module","commit_stats":null,"previous_names":["tiny-systems/store-module"],"tags_count":3,"template":false,"template_full_name":"tiny-systems/example-module","purl":"pkg:github/tiny-systems/store-module","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tiny-systems%2Fstore-module","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tiny-systems%2Fstore-module/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tiny-systems%2Fstore-module/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tiny-systems%2Fstore-module/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tiny-systems","download_url":"https://codeload.github.com/tiny-systems/store-module/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tiny-systems%2Fstore-module/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34860892,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-27T02:00:06.362Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-27T16:30:50.745Z","updated_at":"2026-06-27T16:30:53.291Z","avatar_url":"https://github.com/tiny-systems.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tiny Systems Store Module\n\nEmbedded, persistent storage components for Tiny Systems flows. The\nbacking store is [bbolt](https://github.com/etcd-io/bbolt) — pure Go,\nsingle-file, transactional, used inside etcd itself. Data lives on a PVC\nmounted inside the operator pod so it survives pod restarts.\n\n## When to use this vs. alternatives\n\n| Use case | Reach for |\n|---|---|\n| Small in-component state (counters, last-seen, port runtime data) | SDK `State` via `module.Base.State()` — protected by `MaxStateBytes = 900KB` since SDK v0.10.9 |\n| Per-flow persistence above ~1MB (chat history, agent scratchpads, retrieval caches) | **document_store** (this module) |\n| Shared persistence across flows / HA requirements | `postgres_*` / `redis_*` in `database-module-v0` |\n\n`document_store` is the \"no external infra\" path. One container, one\nfile, one PVC. Configure once and any flow can use it via standard\nedges.\n\n## Components\n\n### `document_store`\n\nEmbedded KV store with per-collection buckets. Four operation ports\n(`put`, `get`, `delete`, `find`) each with a matching source result port.\n\n**Settings**\n\n| Field | Type | Notes |\n|---|---|---|\n| `path` | string | Absolute path to the bbolt file. Default `/data/store.db`. Mount a PVC at this directory. |\n| `collections` | `[{name}]` | Named buckets. Writes to undeclared collections fail. At least one required. |\n| `maxSizeMB` | int | Soft cap on file size. Puts above this route to error port (`diskFull: true`). Default 1024. |\n| `enableErrorPort` | bool | Route operational failures (disk full, missing collection, marshal errors) to the error port. |\n\n**Ports**\n\nInput (target):\n\n- `put` — `{context, collection, key, value}` → emits on `put_ok`\n- `get` — `{context, collection, key}` → emits on `get_ok` with `{value, found}`\n- `delete` — `{context, collection, key}` → emits on `delete_ok` with `{deleted}` (false if key was absent)\n- `find` — `{context, collection, prefix?, limit?}` → emits on `find_ok` with `{items: [{key, value}], count}`\n\nOutput (source): `put_ok`, `get_ok`, `delete_ok`, `find_ok`, and (when\nenabled) `error` with optional `diskFull: true`.\n\nValues are stored as JSON. Any JSON-serialisable Go value works —\nstrings, numbers, objects, arrays, nested structures.\n\n## Deployment patterns\n\nbbolt is single-writer. The .db file has an exclusive OS-level lock,\nso the deployment shape determines availability:\n\n### 1. Single replica (default) — RWO PVC, `replicas: 1`\n\nSimplest. One pod owns the file. Failure modes:\n\n- **Pod crash / restart on same node**: k8s reschedules, PVC remains\n  attached, new pod opens bbolt within ~5-10s.\n- **Node failure**: PVC detaches from dead node and attaches to a new\n  one — typically 30-90s depending on storage class. During this\n  window all requests fail; flow authors should route store errors\n  through a retry component.\n- **Storage class matters**: GKE `standard-rwo` typically detaches in\n  ~30s; some CSI drivers are slower. Test your failure mode.\n\nThis is what `Settings.LeaderOnly = false` configures. Good enough for\nsingle-tenant clusters, internal tooling, demos.\n\n### 2. Leader-only mode — RWX PVC, `replicas: N`\n\nFor deployments that want fast failover and can pay for RWX storage.\nSet `Settings.LeaderOnly = true`. Behaviour:\n\n- All N pods come up; SDK leader-election picks one.\n- Leader opens bbolt, holds the file lock, serves requests.\n- Followers refuse with `Retryable: true` on the error port.\n- Leader dies → election fires (~10s) → new leader retries\n  `bbolt.Open` for up to 30s until the prior leader's lock releases\n  (clean death is instant; dirty death waits for NFS/CSI cleanup).\n\nTradeoffs:\n\n- **RWX storage (Filestore / EFS) is ~10× the cost of block storage**\n  and slower per-op. The whole point of bbolt was cheap embedded\n  storage; using RWX moves you partway toward \"just use postgres.\"\n- **Followers don't serve reads.** Single writer, single reader. The\n  extra replicas are warm standbys, not horizontal scaling.\n- **Load balancer doesn't know about leadership.** Requests that land\n  on a follower fail with `retryable=true`. Callers (or a Kubernetes\n  Service with a smarter health check) need to handle the retry.\n\n### 3. External storage — `postgres_*` / `redis_*` from database-module-v0\n\nWhen you need real horizontal scaling, multi-region, or sub-second\nfailover. The components are stateless clients; the database does\nHA at its layer. Same flow shape — swap `document_store.put` for\n`postgres_exec`, `document_store.get` for `postgres_query` — and the\nLLM never knows the difference.\n\n## Other notes\n\n**PVC required.** Without persistent storage at `Settings.path`, all\ndata is lost on pod restart. The Helm chart for `tinysystems-operator`\nsupports volume mounts — point a PVC at `/data` (or wherever `path`\nlives) when installing. For LeaderOnly mode, the PVC must be RWX.\n\n**Lock contention messages.** When `bbolt.Open` times out (30s\nceiling), the error message specifically names \"another pod may be\nholding the file lock\" so the operator knows where to look. Common\ncauses: a Pending pod with the same PVC, a stuck NFS client on a dead\nnode, or a developer running `go run cmd/main.go` against the same\n.db while a pod is also running.\n\n**Backup story.** bbolt is a single file. Copy `path` while the pod\nis quiesced. Scheduled backup is out of scope for v1.\n\n## Pattern: persistent chat history\n\n```\nhttp_server.request → json_decode → document_store.get (conversation key)\n                                  → join messages + new user turn\n                                  → llm_chat (stateless)\n                                  → document_store.put (conversation key)\n                                  → http_server.response\n```\n\n`llm_chat` stays stateless and reusable — the flow composes\npersistence around it. Swap `document_store` for `kv` (in\n`common-module-v0`) for small histories, or `postgres_exec` when you\nneed HA.\n\n## Run locally\n\n```shell\nmkdir -p /tmp/store-module-data\ngo run cmd/main.go run \\\n  --name=tinysystems/store-module \\\n  --namespace=tinysystems-tinysystems \\\n  --version=0.1.0\n```\n\nBind `/data` (or your configured `Settings.path` directory) to a\nwritable host path when running outside Kubernetes.\n\n## Deploy\n\n```shell\ndocker build -t myregistry/store-module:0.1.0 .\ndocker push myregistry/store-module:0.1.0\n\nhelm install store-module tinysystems/tinysystems-operator \\\n  --set controllerManager.manager.image.repository=myregistry/store-module \\\n  --set persistentVolume.enabled=true \\\n  --set persistentVolume.mountPath=/data \\\n  --set persistentVolume.size=10Gi \\\n  --set controllerManager.replicas=1\n```\n\nSee `helm get values tinysystems-llm-module-v0` (or any other module)\nfor the existing Helm shape — the operator chart is shared across all\nmodules.\n\n## License\n\nMIT for this module's source. Depends on the [Tiny Systems Module SDK](https://github.com/tiny-systems/module) (BSL 1.1) and [bbolt](https://github.com/etcd-io/bbolt) (MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftiny-systems%2Fstore-module","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftiny-systems%2Fstore-module","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftiny-systems%2Fstore-module/lists"}