{"id":31364432,"url":"https://github.com/esptoolkit/esp-jsondb","last_synced_at":"2026-02-12T14:01:08.921Z","repository":{"id":314406565,"uuid":"1054936871","full_name":"ESPToolKit/esp-jsondb","owner":"ESPToolKit","description":"Lightweight JSON document database for ESP32 with a MongoDB‑like API, RAM caching, and optional autosync to LittleFS","archived":false,"fork":false,"pushed_at":"2025-12-12T12:07:15.000Z","size":1786,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-13T22:15:51.696Z","etag":null,"topics":["arduino","arduino-library","database","document-database","embedded","esp","iot","json","littlefs","nosql","storage"],"latest_commit_sha":null,"homepage":"https://www.esptoolkit.hu/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ESPToolKit.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-11T14:33:10.000Z","updated_at":"2025-12-12T12:07:19.000Z","dependencies_parsed_at":"2026-02-12T13:06:38.322Z","dependency_job_id":null,"html_url":"https://github.com/ESPToolKit/esp-jsondb","commit_stats":null,"previous_names":["esptoolkit/esp-jsondb"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/ESPToolKit/esp-jsondb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ESPToolKit%2Fesp-jsondb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ESPToolKit%2Fesp-jsondb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ESPToolKit%2Fesp-jsondb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ESPToolKit%2Fesp-jsondb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ESPToolKit","download_url":"https://codeload.github.com/ESPToolKit/esp-jsondb/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ESPToolKit%2Fesp-jsondb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29367814,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-12T08:51:36.827Z","status":"ssl_error","status_checked_at":"2026-02-12T08:51:26.849Z","response_time":55,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arduino","arduino-library","database","document-database","embedded","esp","iot","json","littlefs","nosql","storage"],"created_at":"2025-09-27T07:08:58.090Z","updated_at":"2026-02-12T14:01:08.915Z","avatar_url":"https://github.com/ESPToolKit.png","language":"C++","readme":"# ESPJsonDB\n\nA lightweight document database for ESP32 devices. ESPJsonDB borrows the ergonomics of MongoDB/Mongoose while embracing embedded constraints: collections live as JSON on LittleFS, memory use is capped through an optional cache, and every API leans on ArduinoJson types so you can stay inside a single document representation.\n\n## CI / Release / License\n[![CI](https://github.com/ESPToolKit/esp-jsondb/actions/workflows/ci.yml/badge.svg)](https://github.com/ESPToolKit/esp-jsondb/actions/workflows/ci.yml)\n[![Release](https://img.shields.io/github/v/release/ESPToolKit/esp-jsondb?sort=semver)](https://github.com/ESPToolKit/esp-jsondb/releases)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE.md)\n\n## Features\n- Simple, mongoose-like API for embedded projects (create/update/remove/find with predicates or JSON filters).\n- Optional in-memory cache with dirty-document tracking and change detection to avoid needless flash I/O.\n- Automatic LittleFS synchronisation on a background FreeRTOS task (`ESPJsonDBConfig` controls interval, stack, priority, and cache usage).\n- MessagePack compression + StreamUtils for efficient read/write pipelines.\n- Schema registry with required fields, defaults, type validation, and collection-level unique constraints.\n- Event + error callbacks so firmware can observe sync cycles or take action when validation fails.\n- Snapshot/restore helpers for backups plus diagnostics that report per-collection counts and config details.\n- Generic file storage helpers under `/\u003cbaseDir\u003e/_files` with chunked stream read/write for any file type (text, binary, etc.).\n\n## Examples\nQuick start:\n\n```cpp\n#include \u003cESPJsonDB.h\u003e\n\nESPJsonDB db;\n\nvoid setup() {\n    Serial.begin(115200);\n\n    ESPJsonDBConfig cfg;\n    cfg.intervalMs = 3000;  // autosync every 3s\n    cfg.autosync = true;\n\n    if (!db.init(\"/test_db\", cfg).ok()) {\n        Serial.println(\"DB init failed\");\n        return;\n    }\n\n    db.onEvent([](DBEventType evt){\n        Serial.printf(\"Event: %s\\n\", dbEventTypeToString(evt));\n    });\n    db.onError([](const DbStatus \u0026st){\n        Serial.printf(\"Error: %s\\n\", st.message);\n    });\n}\n```\n\nWorking with documents is intentionally `JsonDocument`-centric:\n\n```cpp\nJsonDocument doc;\ndoc[\"email\"] = \"user@example.com\";\ndoc[\"role\"] = \"admin\";\nauto createRes = db.create(\"users\", doc.as\u003cJsonObjectConst\u003e());\n\nif (createRes.status.ok()) {\n    const std::string\u0026 id = createRes.value;\n    auto found = db.findById(\"users\", id);\n    if (found.status.ok()) {\n        Serial.printf(\"Role: %s\\n\", found.value[\"role\"].as\u003cconst char*\u003e());\n    }\n    db.updateById(\"users\", id, [](DocView\u0026 view){\n        view[\"role\"].set(\"owner\");\n    });\n    db.removeById(\"users\", id);\n}\n```\n\nSee the sketches under `examples/` for end-to-end flows:\n- `QuickStart` – database initialisation and simple CRUD.\n- `Collections` – create/drop collections at runtime.\n- `CacheDisabled` – run without the RAM cache.\n- `BulkOperations` – batch inserts, updates, and queries.\n- `SchemaValidation` – enforce required fields and custom validators.\n- `UniqueFields` – per-collection uniqueness guarantees.\n- `References` – store one-to-many relations and populate them lazily.\n- `FileStreaming` – store and stream `txt` / `json` / `csv` / `bin` / custom extension payloads.\n- `AsyncFileUpload` – non-blocking, callback-driven chunk upload on a background task.\n\nFile storage example:\n\n```cpp\nESPJsonDBFileOptions fileOpts;\nfileOpts.chunkSize = 256;\ndb.writeTextFile(\"notes/readme.txt\", \"hello from esp-jsondb\");\n\nFile firmwareChunk = LittleFS.open(\"/fw/chunk.bin\", FILE_READ);\ndb.writeFileStream(\"firmware/chunk.bin\", firmwareChunk, firmwareChunk.size(), fileOpts);\nfirmwareChunk.close();\n```\n\n## Gotchas\n- Each collection lives in RAM when the cache is enabled; disable the cache or add PSRAM when handling large documents.\n- All payloads are JSON; converting to structs is optional but deserialisation still costs memory—size your `JsonDocument` objects carefully.\n- Sync callbacks run on the background task; keep them short to avoid blocking periodic flushes.\n- Unique constraints and validators run inside write operations. Long-running validators will increase latency for the calling task.\n- `writeFileStream()` and `readFileStream()` hold the filesystem lock while processing the stream; use reasonable chunk sizes and avoid blocking stream sources/sinks.\n- `writeFileStreamAsync()` runs producer callbacks on a background task; callbacks must be short and thread-safe.\n- `/_files` is an internal reserved directory used for file storage and cannot be used as a collection name.\n- `getSnapshot()` and `restoreFromSnapshot()` currently cover document collections only; file storage under `/_files` is not included.\n\n## API Reference\n- `DbStatus init(const char* baseDir = \"/db\", const ESPJsonDBConfig\u0026 cfg = {})` – mount LittleFS (`cfg.initFileSystem`) and create the autosync task (optional). Diagnostics are lightweight and served from in-memory counters.\n- `void onEvent(std::function\u003cvoid(DBEventType)\u003e)` / `void onError(std::function\u003cvoid(const DbStatus\u0026)\u003e)` – receive sync, CRUD, and validation events.\n- Collection management: `collection(name)`, `dropCollection(name)`, `dropAll()`, `getAllCollectionName()`.\n- Document helpers:\n  - Create: `create`, `createMany` (JSON array) plus direct `Collection::create*` variants.\n  - Read: `findById`, `findOne`, `findMany` (predicate or JSON filter) returning `DocView` so you can read/write lazily.\n  - Update/delete: `updateOne`, `updateById`, `updateMany`, `removeById`, `removeMany` (predicate or JSON filter).\n- Schemas: `registerSchema(name, Schema)`, `unRegisterSchema(name)`; `Schema` exposes fields with type/default/unique flags plus optional custom `validate` callables.\n- References: store `{ \"collection\": \"authors\", \"_id\": \"...\" }` inside a document and call `DocView::populate(fieldName)` to expand the reference into an embedded object.\n- Sync + diagnostics: `syncNow()`, `getDiag()` (JSON summary), `getSnapshot()` / `restoreFromSnapshot()` for backups.\n  - `getDiag()` does not touch the filesystem; it reports cached counters overlaid with currently loaded collection sizes.\n- File storage:\n  - `writeFileStream(path, in, bytesToWrite, opts)` / `readFileStream(path, out, chunkSize)` for chunked stream transfer.\n  - `writeFileStreamAsync(path, pullCb, opts, doneCb)` for non-blocking producer-driven uploads.\n  - `cancelFileUpload(uploadId)`, `getFileUploadState(uploadId)` for async job control.\n  - `writeFile(path, data, size)` / `readFile(path)` for direct byte buffers.\n  - `writeTextFile(path, text)` / `readTextFile(path)` for UTF-8 or plain text payloads.\n  - `fileExists(path)`, `fileSize(path)`, `removeFile(path)` for file lifecycle utilities.\n  - File paths are relative to `/\u003cbaseDir\u003e/_files` and path traversal segments are rejected.\n  - `ESPJsonDBFileOptions`: `overwrite` and `chunkSize` controls for stream writes.\n  - `DbFileUploadPullCb`: callback receives `(requested, buffer, produced, eof)` and fills bytes into `buffer`.\n\n`ESPJsonDBConfig` knobs:\n- `intervalMs`, `stackSize`, `priority`, `coreId` – background autosync cadence \u0026 FreeRTOS tuning.\n- `autosync`, `coldSync`, `cacheEnabled` – enable/disable timers and caches.\n- `fs`, `initFileSystem`, `formatOnFail`, `partitionLabel`, `maxOpenFiles` – file system integration; pass your own `fs::FS` if you mount LittleFS elsewhere.\n\nStack sizes are expressed in bytes.\n\n## Restrictions\n- Designed for ESP32 + LittleFS. Other platforms/FSes are untested.\n- Large documents are only practical on boards with PSRAM when the cache is enabled.\n- Requires ArduinoJson 6+, StreamUtils, and a FreeRTOS-capable environment (Arduino-ESP32 or ESP-IDF with C++17).\n\n## Tests\nAn integration harness (`test/`) runs CRUD, bulk, schema, reference, and diagnostic scenarios via the `DbTester` class. Build it as a PlatformIO test or ESP-IDF component (include `test/dbTest.cpp` in your project) and run it on hardware to validate changes. Contributions that expand automated coverage are welcome.\n\n## License\nMIT — see [LICENSE.md](LICENSE.md).\n\n## ESPToolKit\n- Check out other libraries: \u003chttps://github.com/orgs/ESPToolKit/repositories\u003e\n- Hang out on Discord: \u003chttps://discord.gg/WG8sSqAy\u003e\n- Support the project: \u003chttps://ko-fi.com/esptoolkit\u003e\n- Visit the website: \u003chttps://www.esptoolkit.hu/\u003e\n","funding_links":["https://ko-fi.com/esptoolkit"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fesptoolkit%2Fesp-jsondb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fesptoolkit%2Fesp-jsondb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fesptoolkit%2Fesp-jsondb/lists"}