{"id":50372127,"url":"https://github.com/MauricioPerera/just-bash-data","last_synced_at":"2026-06-15T22:00:48.737Z","repository":{"id":354140201,"uuid":"1222323662","full_name":"MauricioPerera/just-bash-data","owner":"MauricioPerera","description":"just-bash plugin: db (MongoDB-style document store) and vec (vector similarity search) commands for in-shell agents. JWT auth, RBAC, AES-256-GCM encryption, 4 quantizations.","archived":false,"fork":false,"pushed_at":"2026-05-05T10:58:59.000Z","size":321,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-02T19:41:54.680Z","etag":null,"topics":["agent-shell","document-database","encryption","just-bash","jwt","llm-tools","plugin","rag","rbac","vector-search"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MauricioPerera.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-04-27T08:53:28.000Z","updated_at":"2026-05-26T02:00:46.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/MauricioPerera/just-bash-data","commit_stats":null,"previous_names":["mauricioperera/just-bash-data"],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/MauricioPerera/just-bash-data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MauricioPerera%2Fjust-bash-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MauricioPerera%2Fjust-bash-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MauricioPerera%2Fjust-bash-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MauricioPerera%2Fjust-bash-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MauricioPerera","download_url":"https://codeload.github.com/MauricioPerera/just-bash-data/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MauricioPerera%2Fjust-bash-data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34381762,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-15T02:00:07.085Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-shell","document-database","encryption","just-bash","jwt","llm-tools","plugin","rag","rbac","vector-search"],"created_at":"2026-05-30T08:00:19.050Z","updated_at":"2026-06-15T22:00:48.728Z","avatar_url":"https://github.com/MauricioPerera.png","language":"TypeScript","funding_links":[],"categories":["Libraries"],"sub_categories":[],"readme":"# `just-bash-data`\n\n[![npm](https://img.shields.io/npm/v/just-bash-data.svg)](https://www.npmjs.com/package/just-bash-data)\n[![CI](https://github.com/MauricioPerera/just-bash-data/actions/workflows/ci.yml/badge.svg)](https://github.com/MauricioPerera/just-bash-data/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n[![CHANGELOG](https://img.shields.io/badge/CHANGELOG-md-blue.svg)](CHANGELOG.md)\n\nA plugin for [`just-bash`](https://github.com/vercel-labs/just-bash) that gives an in-shell agent two structured-data commands:\n\n- **`db`** — MongoDB-style document store backed by [`js-doc-store`](https://github.com/MauricioPerera/js-doc-store): CRUD, indexes, aggregations, JWT auth, RBAC, optional AES-256-GCM encryption.\n- **`vec`** — vector similarity search backed by [`js-vector-store`](https://github.com/MauricioPerera/js-vector-store): float32 + 3 quantizations, IVF, matryoshka, cross-collection search.\n\nBoth share a single in-memory state hydrated from `IFileSystem` on first use and atomically flushed back after every mutating command.\n\n\u003e **Sister projects**: [`agent-skills`](https://github.com/MauricioPerera/agent-skills) is an open specification for distributing tools to LLM agents (an alternative to MCP) that uses `just-bash-data` as its reference runtime. [`just-bash-wiki`](https://github.com/MauricioPerera/just-bash-wiki) builds on this plugin to implement [Karpathy's LLM Wiki pattern](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) — a persistent, LLM-maintained knowledge base with semantic search.\n\n\u003e **Benchmark**: 8 Cloudflare Workers AI models (Granite 4.0, Llama 3.1/3.2 8B family, Llama 4 Scout MoE, GPT-OSS-20B, Gemma 4 26B) tested as agents driving the `db` command. **Granite wins on cost** ($0.000107/task, 7/7 completion). **GPT-OSS-20B wins on turns** (2 turns to DONE, 92% precision). **Gemma 4 wins on precision** (100%, zero retries, 16× the cost). Permissive-parsing aliases (v0.2.0 → v0.3.1) cut total cost by **41%** vs v0.1.0 with zero breaking changes. v0.6.0–v0.8.1 added lenient JSON parsing + operator `$`-prefix validation, taking the agent-trace replay to **103/107 commands exit 0 (96.3%)** with zero regressions across all 8 models — full v0.8.1 retest report at [examples/smoke/v8-benchmark-report.md](examples/smoke/v8-benchmark-report.md). Original cost analysis: [examples/smoke/BENCHMARK.md](examples/smoke/BENCHMARK.md).\n\n## Install\n\n```bash\nnpm i just-bash-data just-bash\n# or pin to a specific version\nnpm i just-bash-data@1.1.2 just-bash\n```\n\nThe plugin pulls its two upstream libs ([`js-doc-store`](https://www.npmjs.com/package/js-doc-store), [`js-vector-store`](https://www.npmjs.com/package/js-vector-store)) from GitHub at install time. Both are also available on npm independently.\n\nIf you need a specific commit or branch instead of the published release:\n\n```bash\nnpm i github:MauricioPerera/just-bash-data#v1.1.0 just-bash\n```\n\n## Quick start\n\n```typescript\nimport { Bash, InMemoryFs } from \"just-bash\";\nimport { createDataPlugin } from \"just-bash-data\";\n\nconst bash = new Bash({\n  fs: new InMemoryFs({}),\n  customCommands: createDataPlugin({\n    authSecret: \"jwt-signing-secret\",   // optional: enables RBAC\n    encryptionKey: \"aes-key\",           // optional: AES-256-GCM at rest\n    rootDir: \"/data\",                   // optional: default \"/data\"\n  }),\n});\n\nawait bash.exec(`db users insert '{\"name\":\"Alice\",\"age\":30}'`);\nawait bash.exec(`db users find '{\"age\":{\"$gte\":18}}' --sort age:-1`);\n\nawait bash.exec(`vec create docs --dim 1536 --quantize int8`);\nawait bash.exec(`vec store docs doc-1 '${JSON.stringify(embedding)}'`);\nawait bash.exec(`vec search docs '${JSON.stringify(query)}' --k 5`);\n```\n\n`createDataPlugin` returns a `CustomCommand[]` ready to plug into `Bash`. Each call gets its own state, scoped to the `IFileSystem` instance it sees at runtime — multiple `Bash` instances do not share data unless they share an `IFileSystem`.\n\n## Architecture\n\nThe two upstream libraries call their adapters synchronously, while just-bash `IFileSystem` is fully async. Three thin layers bridge them:\n\n```\n            IFileSystem  (async)\n                 ↑ ↓     Persister    hydrate / flush\n            MemoryAdapter  (sync, Map-backed)\n                 ↑        EncryptedAdapter (json) / EncryptedBinAdapter (bin)  ← optional\n       DocStore / VectorStore  (sync clients)\n```\n\n- The `MemoryAdapter` is the single source of truth at runtime. Reads and writes are O(1).\n- The `Persister` hydrates the adapter from `\u003croot\u003e/` lazily on first command, then flushes dirty entries atomically (`\u003cname\u003e.tmp` + `fs.mv`) after every mutating command.\n- Encryption is an optional sandwich: `EncryptedAdapter` (from upstream) wraps JSON; a small internal `EncryptedBinAdapter` mirrors the same pattern for binary blobs (vector files).\n- A `PluginRegistry` is cached per `IFileSystem` instance via `WeakMap`, so different `Bash` instances get isolated state automatically.\n\nConcurrent `flush` calls are serialized: each call appends a `doFlush` to a chain so its `takeDirty()` runs after all earlier flushes complete. Failures don't poison the chain.\n\n## `db` command\n\n```\ndb \u003ccoll|auth\u003e \u003csubcommand\u003e [args...] [flags...]\n```\n\n### Document operations\n\n| Subcommand | Effect |\n|---|---|\n| `db \u003ccoll\u003e insert \u003cjson\u003e` | Insert one doc. Generates `_id` if missing. Returns `{_id}`. |\n| `db \u003ccoll\u003e find \u003cfilter\u003e [--sort f:1\\|-1] [--limit N] [--skip N] [--project f1,f2]` | Mongo-style query. Returns `Doc[]`. |\n| `db \u003ccoll\u003e count \u003cfilter\u003e` | Returns `{count}`. |\n| `db \u003ccoll\u003e update \u003cfilter\u003e \u003cupdate\u003e [--many]` | Returns `{matched, modified}` (matched is capped at 1 in single mode). |\n| `db \u003ccoll\u003e remove \u003cfilter\u003e [--many]` | Returns `{removed}`. |\n| `db \u003ccoll\u003e aggregate \u003cpipeline\u003e` | Pipeline: `$match`/`$lookup`/`$group`/`$sort`/`$limit`/`$skip`/`$project`/`$unwind`. Accumulators: `$count`/`$sum`/`$avg`/`$min`/`$max`/`$push`/`$first`/`$last`. |\n| `db \u003ccoll\u003e drop` | Removes the collection. **Requires `admin` role** when auth is configured. |\n| `db \u003ccoll\u003e stats` | Returns `{count, indexes, sizeBytes}`. |\n| `db \u003ccoll\u003e export` | Returns `{exported, docs}` with all documents. |\n| `db \u003ccoll\u003e import \u003cjson\u003e` | Imports an array of documents. |\n\nAny positional JSON argument can be `-` to read from stdin: `echo '{\"a\":1}' \\| db users insert -` (only one `-` per invocation).\n\n### Indexes\n\n```bash\ndb \u003ccoll\u003e index create \u003cfield\u003e [--sorted] [--unique]\ndb \u003ccoll\u003e index drop \u003cfield\u003e\ndb \u003ccoll\u003e index list\n```\n\n- Default is a hash index. `--sorted` creates a sorted index for range queries.\n- `--unique` adds a uniqueness constraint. Insert/update collisions exit with `5`.\n\n### Query operators (all upstream)\n\n`$eq` `$ne` `$gt` `$gte` `$lt` `$lte` `$in` `$nin` `$exists` `$regex` `$contains` `$size` `$and` `$or` `$not` — plus dot notation for nested fields (`addr.city`).\n\n### Update operators\n\n`$set` `$unset` `$inc` `$push` `$pull` `$rename`.\n\n### Auth\n\nSet `PluginOptions.authSecret` to enable JWT + RBAC. When unset, the entire auth surface is a no-op (every command is public).\n\n```bash\ndb auth register \u003cemail\u003e \u003cpass\u003e [--roles=role1,role2]\ndb auth login    \u003cemail\u003e \u003cpass\u003e                       # → {token, expiresAt}\ndb auth verify   [--token=\u003cjwt\u003e]                      # → {user, roles, expiresAt}\ndb auth logout   [--token=\u003cjwt\u003e] [--all]\ndb auth role assign \u003cuser-id\u003e \u003crole\u003e [--token=\u003cjwt\u003e]  # admin only\ndb auth role remove \u003cuser-id\u003e \u003crole\u003e [--token=\u003cjwt\u003e]\n```\n\nThe token is read from `--token=\u003cjwt\u003e` first, falling back to `ctx.env.AUTH_TOKEN`. Use `--token` in tests; use the env var in real shell pipelines.\n\nWhen `authSecret` is set, these operations require a valid JWT:\n\n`insert`, `update`, `remove`, `drop`, `index create`, `index drop`, `auth role assign`, `auth role remove`.\n\nThese remain public:\n\n`find`, `count`, `aggregate`, `stats`, `index list`, `auth verify`, `auth login`, `auth register`, `auth logout`.\n\n`drop`, `auth role assign`, and `auth role remove` additionally require the `admin` role.\n\n## `vec` command\n\n```\nvec \u003csubcommand\u003e [args...] [flags...]\n```\n\n| Subcommand | Effect |\n|---|---|\n| `vec create \u003ccoll\u003e --dim N [--quantize float32\\|int8\\|polar\\|binary] [--metric cosine\\|euclidean\\|dot\\|manhattan]` | Creates a collection. `--dim` capped at 65536. |\n| `vec store \u003ccoll\u003e \u003cid\u003e \u003cvector-json\u003e [--meta \u003cjson\u003e]` | Single vector. JSON arg may be `-` for stdin. |\n| `vec store-batch \u003ccoll\u003e \u003cjsonl-path-or--\u003e` | One JSONL record per line `{id, vector, meta?}`. Reports `{stored, skipped, errors[]}`. ID collisions abort the batch with exit 5. |\n| `vec search \u003ccoll\u003e \u003cvector-json\u003e [--k N=10] [--metric M] [--matryoshka 64,256,1024]` | Returns `[{id, score, metadata?}, ...]` sorted by score desc (higher = more similar, regardless of metric). |\n| `vec search-across \u003ccoll-csv\u003e \u003cvector-json\u003e [--k N]` | Merges per-collection search results, returns `[{id, score, coll, metadata?}, ...]`. |\n| `vec get \u003ccoll\u003e \u003cid\u003e` | Returns `{id, vector, metadata}`. |\n| `vec remove \u003ccoll\u003e \u003cid\u003e` | Returns `{removed}`. |\n| `vec stats \u003ccoll\u003e` | Returns `{dim, count, quantize, metric}`. |\n| `vec export \u003ccoll\u003e` | Returns `{exported, records}`. |\n| `vec import \u003ccoll\u003e \u003cjson-path-or--\u003e` | Imports an array of `{id, vector, metadata?}` records. |\n| `vec drop \u003ccoll\u003e` | Removes the collection. |\n| `vec verify \u003ccoll\u003e` | Checks encryption integrity. Returns `{coll, ok, encrypted, binFile}`. |\n| `vec ivf build \u003ccoll\u003e [--sample-dims=N]` | Builds IVF (K-means) index for faster search. |\n| `vec ivf stats \u003ccoll\u003e` | Returns IVF index statistics. |\n| `vec ivf drop \u003ccoll\u003e` | Drops the IVF index. |\n\n### Quantization choices\n\n| Quantize | Bits/dim | Compression vs float32 | Recall trade-off |\n|---|---:|---:|---|\n| `float32` | 32 | 1× | exact |\n| `int8` | 8 | ~4× | ≥ 0.7 top-K overlap |\n| `polar` | 3 | ~21× | rough, fast |\n| `binary` | 1 | ~32× | very rough, very fast |\n\nQuantization is per-collection and immutable. To change it: `vec drop` + `vec create` + reinsert.\n\n## Plugin options\n\n```typescript\ntype PluginOptions = {\n  encryptionKey?: string;  // AES-256-GCM at rest (PBKDF2 100k iter)\n  authSecret?: string;     // HMAC-SHA256 JWT signing key\n  rootDir?: string;        // default: \"/data\"\n};\n```\n\n### Encryption\n\nWhen `encryptionKey` is set:\n\n- All JSON files (including `_users.docs.json`) are encrypted via upstream `EncryptedAdapter` before reaching `IFileSystem`.\n- Vector binary files are encrypted via the internal `EncryptedBinAdapter` with the same key (separate IV per write).\n- A consumer with the wrong key sees empty collections — no plaintext leaks.\n\n### Auth secret\n\nWhen `authSecret` is set, JWT issuance and verification become mandatory for the auth-required operations listed above. Tokens default to 24h expiry.\n\n### Custom root dir\n\nThe default `rootDir` is `/data` inside the virtual FS. Files written are listed below.\n\n## Persistence layout (under `\u003crootDir\u003e`)\n\n| File | Owner |\n|---|---|\n| `\u003ccoll\u003e.docs.json` | doc-store: documents |\n| `\u003ccoll\u003e.meta.json` | doc-store: index metadata |\n| `\u003ccoll\u003e.\u003cfield\u003e.idx.json` | doc-store: hash index |\n| `\u003ccoll\u003e.\u003cfield\u003e.sidx.json` | doc-store: sorted index |\n| `\u003ccoll\u003e.bin` / `\u003ccoll\u003e.q8.bin` / `\u003ccoll\u003e.b1.bin` / `\u003ccoll\u003e.p3.bin` | vector-store: vectors per quantization |\n| `\u003ccoll\u003e.json` (and quantized variants) | vector-store: per-collection manifest |\n| `_users.docs.json` | auth users |\n| `_sessions.docs.json` | active JWT sessions |\n| `_vec.registry.json` | per-coll vec metadata (dim, quantize, metric) |\n\n## Exit codes (uniform across both commands)\n\n| Code | Meaning |\n|---:|---|\n| `0` | success — `stdout` is JSON |\n| `1` | runtime / internal error |\n| `2` | usage error (bad args, malformed JSON, dim too large) |\n| `3` | not found (missing collection or id) |\n| `4` | auth error (missing / invalid / expired token, role required) |\n| `5` | validation error (unique constraint, dim mismatch, collection exists, etc.) |\n\n`stderr` is always plain text on non-zero exit.\n\n## Common recipes\n\n### RAG with external embeddings + auth\n\n```bash\nTOKEN=$(db auth login alice@x.com 's3cret123' | jq -r '.token')\nEMB=$(curl -s api.openai.com/v1/embeddings \\\n       -H \"Authorization: Bearer $OPENAI_KEY\" \\\n       -d \"{\\\"input\\\":\\\"$Q\\\",\\\"model\\\":\\\"text-embedding-3-small\\\"}\" \\\n     | jq '.data[0].embedding')\n\nvec search docs \"$EMB\" --k 5 \\\n  | jq -r '.[] | .id' \\\n  | xargs -I{} db chunks find \"{\\\"_id\\\":\\\"{}\\\"}\" --token=\"$TOKEN\" \\\n  | jq '{title, body}'\n```\n\n### Indexed read-heavy workload\n\n```bash\ndb users index create email --unique\ndb users index create age --sorted\ndb users find '{\"age\":{\"$gte\":21}}' --sort age:1 --limit 100\n```\n\n### Cross-collection semantic search\n\n```bash\nvec create faq --dim 384\nvec create tickets --dim 384\nvec create docs --dim 384\n# … populate each …\nvec search-across \"faq,tickets,docs\" \"$EMB\" --k 10\n```\n\n### Encrypted persistent state across shells\n\n```typescript\nconst fs = new InMemoryFs({});\nconst opts = { encryptionKey: \"k\", authSecret: \"s\" };\n\nimport { createDataPlugin } from \"just-bash-data\";\n\n// Shell 1\nconst bash1 = new Bash({ fs, customCommands: createDataPlugin(opts) });\nawait bash1.exec(`db notes insert '{\"secret\":\"value\"}'`);\n\n// Shell 2 (same fs, same key) — data rehydrates and decrypts transparently\nconst bash2 = new Bash({ fs, customCommands: createDataPlugin(opts) });\nawait bash2.exec(`db notes find '{}'`);   // sees the doc\n```\n\n## Limitations / known deviations from spec\n\n- **`searchAcross` is implemented in this plugin, not upstream.** Each `vec create` produces an independent store instance, so cross-collection search is performed by merging per-collection searches by score. Functionally equivalent for non-IVF cases.\n- **`db.\u003ccoll\u003e.\u003cmethod\u003e(...)` parens form is uninterceptable.** Bash fails with a parse error on `(` before any command dispatch happens. The space-separated `db \u003ccoll\u003e \u003cmethod\u003e` form is the only one this plugin can serve. The dot-syntax-without-parens form (`db.\u003ccoll\u003e \u003cmethod\u003e`) is caught by sentinels for ~30 common collection names — see v0.7.0 release notes.\n- **TypeScript consumers may need `--skipLibCheck`** if the installed `just-bash@2.14.3` reproduces the upstream `.d.ts` packaging issue (publishes `.d.ts` files referencing paths not in the tarball). This plugin's own types are clean.\n\n## Development\n\n```bash\npnpm install\npnpm typecheck     # strict, no `any`\npnpm lint\npnpm test          # 264 unit + integration tests\npnpm build         # ESM + CJS + .d.ts via tsup\npnpm pack          # produces local-just-bash-data-0.0.0.tgz\n```\n\nA full E2E smoke lives at `examples/smoke/smoke-full.mjs` (243 assertions across every subcommand, every operator, every exit code, plus encryption + salt round-trip, IVF lifecycle, sentinels, operator validators, and `vec verify`).\n\n```bash\npnpm install\npnpm smoke         # builds + runs the full E2E\n```\n\n## License\n\nMIT.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMauricioPerera%2Fjust-bash-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMauricioPerera%2Fjust-bash-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMauricioPerera%2Fjust-bash-data/lists"}