https://github.com/MauricioPerera/just-bash-data
just-bash plugin: db (MongoDB-style document store) and vec (vector similarity search) commands for in-shell agents. JWT auth, RBAC, AES-256-GCM encryption, 4 quantizations.
https://github.com/MauricioPerera/just-bash-data
agent-shell document-database encryption just-bash jwt llm-tools plugin rag rbac vector-search
Last synced: 24 days ago
JSON representation
just-bash plugin: db (MongoDB-style document store) and vec (vector similarity search) commands for in-shell agents. JWT auth, RBAC, AES-256-GCM encryption, 4 quantizations.
- Host: GitHub
- URL: https://github.com/MauricioPerera/just-bash-data
- Owner: MauricioPerera
- License: mit
- Created: 2026-04-27T08:53:28.000Z (2 months ago)
- Default Branch: main
- Last Pushed: 2026-05-05T10:58:59.000Z (2 months ago)
- Last Synced: 2026-06-02T19:41:54.680Z (about 1 month ago)
- Topics: agent-shell, document-database, encryption, just-bash, jwt, llm-tools, plugin, rag, rbac, vector-search
- Language: TypeScript
- Size: 313 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
- Agents: AGENTS.md
Awesome Lists containing this project
README
# `just-bash-data`
[](https://www.npmjs.com/package/just-bash-data)
[](https://github.com/MauricioPerera/just-bash-data/actions/workflows/ci.yml)
[](LICENSE)
[](CHANGELOG.md)
A plugin for [`just-bash`](https://github.com/vercel-labs/just-bash) that gives an in-shell agent two structured-data commands:
- **`db`** — MongoDB-style document store backed by [`js-doc-store`](https://github.com/MauricioPerera/js-doc-store): CRUD, indexes, aggregations, JWT auth, RBAC, optional AES-256-GCM encryption.
- **`vec`** — vector similarity search backed by [`js-vector-store`](https://github.com/MauricioPerera/js-vector-store): float32 + 3 quantizations, IVF, matryoshka, cross-collection search.
Both share a single in-memory state hydrated from `IFileSystem` on first use and atomically flushed back after every mutating command.
> **Sister projects**: [`agent-skills`](https://github.com/MauricioPerera/agent-skills) is an open specification for distributing tools to LLM agents (an alternative to MCP) that uses `just-bash-data` as its reference runtime. [`just-bash-wiki`](https://github.com/MauricioPerera/just-bash-wiki) builds on this plugin to implement [Karpathy's LLM Wiki pattern](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) — a persistent, LLM-maintained knowledge base with semantic search.
> **Benchmark**: 8 Cloudflare Workers AI models (Granite 4.0, Llama 3.1/3.2 8B family, Llama 4 Scout MoE, GPT-OSS-20B, Gemma 4 26B) tested as agents driving the `db` command. **Granite wins on cost** ($0.000107/task, 7/7 completion). **GPT-OSS-20B wins on turns** (2 turns to DONE, 92% precision). **Gemma 4 wins on precision** (100%, zero retries, 16× the cost). Permissive-parsing aliases (v0.2.0 → v0.3.1) cut total cost by **41%** vs v0.1.0 with zero breaking changes. v0.6.0–v0.8.1 added lenient JSON parsing + operator `$`-prefix validation, taking the agent-trace replay to **103/107 commands exit 0 (96.3%)** with zero regressions across all 8 models — full v0.8.1 retest report at [examples/smoke/v8-benchmark-report.md](examples/smoke/v8-benchmark-report.md). Original cost analysis: [examples/smoke/BENCHMARK.md](examples/smoke/BENCHMARK.md).
## Install
```bash
npm i just-bash-data just-bash
# or pin to a specific version
npm i just-bash-data@1.1.2 just-bash
```
The plugin pulls its two upstream libs ([`js-doc-store`](https://www.npmjs.com/package/js-doc-store), [`js-vector-store`](https://www.npmjs.com/package/js-vector-store)) from GitHub at install time. Both are also available on npm independently.
If you need a specific commit or branch instead of the published release:
```bash
npm i github:MauricioPerera/just-bash-data#v1.1.0 just-bash
```
## Quick start
```typescript
import { Bash, InMemoryFs } from "just-bash";
import { createDataPlugin } from "just-bash-data";
const bash = new Bash({
fs: new InMemoryFs({}),
customCommands: createDataPlugin({
authSecret: "jwt-signing-secret", // optional: enables RBAC
encryptionKey: "aes-key", // optional: AES-256-GCM at rest
rootDir: "/data", // optional: default "/data"
}),
});
await bash.exec(`db users insert '{"name":"Alice","age":30}'`);
await bash.exec(`db users find '{"age":{"$gte":18}}' --sort age:-1`);
await bash.exec(`vec create docs --dim 1536 --quantize int8`);
await bash.exec(`vec store docs doc-1 '${JSON.stringify(embedding)}'`);
await bash.exec(`vec search docs '${JSON.stringify(query)}' --k 5`);
```
`createDataPlugin` returns a `CustomCommand[]` ready to plug into `Bash`. Each call gets its own state, scoped to the `IFileSystem` instance it sees at runtime — multiple `Bash` instances do not share data unless they share an `IFileSystem`.
## Architecture
The two upstream libraries call their adapters synchronously, while just-bash `IFileSystem` is fully async. Three thin layers bridge them:
```
IFileSystem (async)
↑ ↓ Persister hydrate / flush
MemoryAdapter (sync, Map-backed)
↑ EncryptedAdapter (json) / EncryptedBinAdapter (bin) ← optional
DocStore / VectorStore (sync clients)
```
- The `MemoryAdapter` is the single source of truth at runtime. Reads and writes are O(1).
- The `Persister` hydrates the adapter from `/` lazily on first command, then flushes dirty entries atomically (`.tmp` + `fs.mv`) after every mutating command.
- Encryption is an optional sandwich: `EncryptedAdapter` (from upstream) wraps JSON; a small internal `EncryptedBinAdapter` mirrors the same pattern for binary blobs (vector files).
- A `PluginRegistry` is cached per `IFileSystem` instance via `WeakMap`, so different `Bash` instances get isolated state automatically.
Concurrent `flush` calls are serialized: each call appends a `doFlush` to a chain so its `takeDirty()` runs after all earlier flushes complete. Failures don't poison the chain.
## `db` command
```
db [args...] [flags...]
```
### Document operations
| Subcommand | Effect |
|---|---|
| `db insert ` | Insert one doc. Generates `_id` if missing. Returns `{_id}`. |
| `db find [--sort f:1\|-1] [--limit N] [--skip N] [--project f1,f2]` | Mongo-style query. Returns `Doc[]`. |
| `db count ` | Returns `{count}`. |
| `db update [--many]` | Returns `{matched, modified}` (matched is capped at 1 in single mode). |
| `db remove [--many]` | Returns `{removed}`. |
| `db aggregate ` | Pipeline: `$match`/`$lookup`/`$group`/`$sort`/`$limit`/`$skip`/`$project`/`$unwind`. Accumulators: `$count`/`$sum`/`$avg`/`$min`/`$max`/`$push`/`$first`/`$last`. |
| `db drop` | Removes the collection. **Requires `admin` role** when auth is configured. |
| `db stats` | Returns `{count, indexes, sizeBytes}`. |
| `db export` | Returns `{exported, docs}` with all documents. |
| `db import ` | Imports an array of documents. |
Any positional JSON argument can be `-` to read from stdin: `echo '{"a":1}' \| db users insert -` (only one `-` per invocation).
### Indexes
```bash
db index create [--sorted] [--unique]
db index drop
db index list
```
- Default is a hash index. `--sorted` creates a sorted index for range queries.
- `--unique` adds a uniqueness constraint. Insert/update collisions exit with `5`.
### Query operators (all upstream)
`$eq` `$ne` `$gt` `$gte` `$lt` `$lte` `$in` `$nin` `$exists` `$regex` `$contains` `$size` `$and` `$or` `$not` — plus dot notation for nested fields (`addr.city`).
### Update operators
`$set` `$unset` `$inc` `$push` `$pull` `$rename`.
### Auth
Set `PluginOptions.authSecret` to enable JWT + RBAC. When unset, the entire auth surface is a no-op (every command is public).
```bash
db auth register [--roles=role1,role2]
db auth login # → {token, expiresAt}
db auth verify [--token=] # → {user, roles, expiresAt}
db auth logout [--token=] [--all]
db auth role assign [--token=] # admin only
db auth role remove [--token=]
```
The token is read from `--token=` first, falling back to `ctx.env.AUTH_TOKEN`. Use `--token` in tests; use the env var in real shell pipelines.
When `authSecret` is set, these operations require a valid JWT:
`insert`, `update`, `remove`, `drop`, `index create`, `index drop`, `auth role assign`, `auth role remove`.
These remain public:
`find`, `count`, `aggregate`, `stats`, `index list`, `auth verify`, `auth login`, `auth register`, `auth logout`.
`drop`, `auth role assign`, and `auth role remove` additionally require the `admin` role.
## `vec` command
```
vec [args...] [flags...]
```
| Subcommand | Effect |
|---|---|
| `vec create --dim N [--quantize float32\|int8\|polar\|binary] [--metric cosine\|euclidean\|dot\|manhattan]` | Creates a collection. `--dim` capped at 65536. |
| `vec store [--meta ]` | Single vector. JSON arg may be `-` for stdin. |
| `vec store-batch ` | One JSONL record per line `{id, vector, meta?}`. Reports `{stored, skipped, errors[]}`. ID collisions abort the batch with exit 5. |
| `vec search [--k N=10] [--metric M] [--matryoshka 64,256,1024]` | Returns `[{id, score, metadata?}, ...]` sorted by score desc (higher = more similar, regardless of metric). |
| `vec search-across [--k N]` | Merges per-collection search results, returns `[{id, score, coll, metadata?}, ...]`. |
| `vec get ` | Returns `{id, vector, metadata}`. |
| `vec remove ` | Returns `{removed}`. |
| `vec stats ` | Returns `{dim, count, quantize, metric}`. |
| `vec export ` | Returns `{exported, records}`. |
| `vec import ` | Imports an array of `{id, vector, metadata?}` records. |
| `vec drop ` | Removes the collection. |
| `vec verify ` | Checks encryption integrity. Returns `{coll, ok, encrypted, binFile}`. |
| `vec ivf build [--sample-dims=N]` | Builds IVF (K-means) index for faster search. |
| `vec ivf stats ` | Returns IVF index statistics. |
| `vec ivf drop ` | Drops the IVF index. |
### Quantization choices
| Quantize | Bits/dim | Compression vs float32 | Recall trade-off |
|---|---:|---:|---|
| `float32` | 32 | 1× | exact |
| `int8` | 8 | ~4× | ≥ 0.7 top-K overlap |
| `polar` | 3 | ~21× | rough, fast |
| `binary` | 1 | ~32× | very rough, very fast |
Quantization is per-collection and immutable. To change it: `vec drop` + `vec create` + reinsert.
## Plugin options
```typescript
type PluginOptions = {
encryptionKey?: string; // AES-256-GCM at rest (PBKDF2 100k iter)
authSecret?: string; // HMAC-SHA256 JWT signing key
rootDir?: string; // default: "/data"
};
```
### Encryption
When `encryptionKey` is set:
- All JSON files (including `_users.docs.json`) are encrypted via upstream `EncryptedAdapter` before reaching `IFileSystem`.
- Vector binary files are encrypted via the internal `EncryptedBinAdapter` with the same key (separate IV per write).
- A consumer with the wrong key sees empty collections — no plaintext leaks.
### Auth secret
When `authSecret` is set, JWT issuance and verification become mandatory for the auth-required operations listed above. Tokens default to 24h expiry.
### Custom root dir
The default `rootDir` is `/data` inside the virtual FS. Files written are listed below.
## Persistence layout (under ``)
| File | Owner |
|---|---|
| `.docs.json` | doc-store: documents |
| `.meta.json` | doc-store: index metadata |
| `..idx.json` | doc-store: hash index |
| `..sidx.json` | doc-store: sorted index |
| `.bin` / `.q8.bin` / `.b1.bin` / `.p3.bin` | vector-store: vectors per quantization |
| `.json` (and quantized variants) | vector-store: per-collection manifest |
| `_users.docs.json` | auth users |
| `_sessions.docs.json` | active JWT sessions |
| `_vec.registry.json` | per-coll vec metadata (dim, quantize, metric) |
## Exit codes (uniform across both commands)
| Code | Meaning |
|---:|---|
| `0` | success — `stdout` is JSON |
| `1` | runtime / internal error |
| `2` | usage error (bad args, malformed JSON, dim too large) |
| `3` | not found (missing collection or id) |
| `4` | auth error (missing / invalid / expired token, role required) |
| `5` | validation error (unique constraint, dim mismatch, collection exists, etc.) |
`stderr` is always plain text on non-zero exit.
## Common recipes
### RAG with external embeddings + auth
```bash
TOKEN=$(db auth login alice@x.com 's3cret123' | jq -r '.token')
EMB=$(curl -s api.openai.com/v1/embeddings \
-H "Authorization: Bearer $OPENAI_KEY" \
-d "{\"input\":\"$Q\",\"model\":\"text-embedding-3-small\"}" \
| jq '.data[0].embedding')
vec search docs "$EMB" --k 5 \
| jq -r '.[] | .id' \
| xargs -I{} db chunks find "{\"_id\":\"{}\"}" --token="$TOKEN" \
| jq '{title, body}'
```
### Indexed read-heavy workload
```bash
db users index create email --unique
db users index create age --sorted
db users find '{"age":{"$gte":21}}' --sort age:1 --limit 100
```
### Cross-collection semantic search
```bash
vec create faq --dim 384
vec create tickets --dim 384
vec create docs --dim 384
# … populate each …
vec search-across "faq,tickets,docs" "$EMB" --k 10
```
### Encrypted persistent state across shells
```typescript
const fs = new InMemoryFs({});
const opts = { encryptionKey: "k", authSecret: "s" };
import { createDataPlugin } from "just-bash-data";
// Shell 1
const bash1 = new Bash({ fs, customCommands: createDataPlugin(opts) });
await bash1.exec(`db notes insert '{"secret":"value"}'`);
// Shell 2 (same fs, same key) — data rehydrates and decrypts transparently
const bash2 = new Bash({ fs, customCommands: createDataPlugin(opts) });
await bash2.exec(`db notes find '{}'`); // sees the doc
```
## Limitations / known deviations from spec
- **`searchAcross` is implemented in this plugin, not upstream.** Each `vec create` produces an independent store instance, so cross-collection search is performed by merging per-collection searches by score. Functionally equivalent for non-IVF cases.
- **`db..(...)` parens form is uninterceptable.** Bash fails with a parse error on `(` before any command dispatch happens. The space-separated `db ` form is the only one this plugin can serve. The dot-syntax-without-parens form (`db. `) is caught by sentinels for ~30 common collection names — see v0.7.0 release notes.
- **TypeScript consumers may need `--skipLibCheck`** if the installed `just-bash@2.14.3` reproduces the upstream `.d.ts` packaging issue (publishes `.d.ts` files referencing paths not in the tarball). This plugin's own types are clean.
## Development
```bash
pnpm install
pnpm typecheck # strict, no `any`
pnpm lint
pnpm test # 264 unit + integration tests
pnpm build # ESM + CJS + .d.ts via tsup
pnpm pack # produces local-just-bash-data-0.0.0.tgz
```
A full E2E smoke lives at `examples/smoke/smoke-full.mjs` (243 assertions across every subcommand, every operator, every exit code, plus encryption + salt round-trip, IVF lifecycle, sentinels, operator validators, and `vec verify`).
```bash
pnpm install
pnpm smoke # builds + runs the full E2E
```
## License
MIT.