{"id":28047208,"url":"https://github.com/deepfield-ml/jotdb","last_synced_at":"2025-05-11T20:46:27.756Z","repository":{"id":291682047,"uuid":"978426450","full_name":"deepfield-ml/JotDB","owner":"deepfield-ml","description":"An efficient JSON writing and reading tool for Go","archived":false,"fork":false,"pushed_at":"2025-05-09T05:36:05.000Z","size":80,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"V0.02","last_synced_at":"2025-05-11T20:45:50.806Z","etag":null,"topics":["go","go-json","golang","json"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepfield-ml.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-06T01:10:10.000Z","updated_at":"2025-05-09T05:31:21.000Z","dependencies_parsed_at":"2025-05-06T10:16:06.644Z","dependency_job_id":null,"html_url":"https://github.com/deepfield-ml/JotDB","commit_stats":null,"previous_names":["deepfield-ml/jotdb"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepfield-ml%2FJotDB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepfield-ml%2FJotDB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepfield-ml%2FJotDB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepfield-ml%2FJotDB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepfield-ml","download_url":"https://codeload.github.com/deepfield-ml/JotDB/tar.gz/refs/heads/V0.02","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253632898,"owners_count":21939382,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","go-json","golang","json"],"created_at":"2025-05-11T20:46:26.806Z","updated_at":"2025-05-11T20:46:27.750Z","avatar_url":"https://github.com/deepfield-ml.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JotDB\n\nJotDB (JSON Optimized Tiny Database) is a lightweight, high-performance Go library for storing and retrieving JSON documents locally. It provides fast read/write operations, thread-safe concurrency, and a modular design, making it ideal for applications needing a simple, embedded key-value store for JSON data.\n\n## Table of Contents  \n- [Features](#features)  \n- [Installation](#installation)  \n- [Usage](#usage)  \n  - [Basic Example](#basic-example)  \n  - [API](#api)  \n- [Performance](#performance) \n- [Benchmarks](#benchmarks) \n- [Architecture](#archittecture) \n- [File Structure](#file-structure)  \n- [Limitations](#limitations)  \n- [Future Improvements](#future-improvements)  \n- [Contributing](#contributing)  \n- [License](#license)  \n- [Acknowledgments](#acknowledgments)  \n- [Authors](#authors)\n- [Version History](#version-history)\n\n## Features\n\n* **Fast JSON Handling** : Custom JSON serialization/deserialization optimized for minimal allocations, supporting objects, arrays, strings, numbers, booleans, and null.\n* **Local Storage** : Uses [Bitcask](https://github.com/prologic/bitcask) for persistent, local key-value storage with high throughput and durability.\n* **Concurrency** : Thread-safe operations with `sync.RWMutex` for concurrent reads and exclusive writes.\n* **Modular Design** : Organized into multiple files (`jotdb.go`, `json_marshal.go`, `json_unmarshal.go`, `types.go`) for maintainability and future optimization.\n* **Simple API** : Intuitive CRUD operations (`Store`, `Retrieve`, `Delete`, `Close`) for JSON documents.\n\n## Installation\n\n1. Ensure you have Go installed (version 1.24 or later recommended).\n2. Install JotDB and its dependency, Bitcask:\n\n```bash\ngo get github.com/prologic/bitcask\n```\n\n3. Call JotDB from GO as `github.com/deepfield-ml/JotDB/jotdb0.01 ` , run ` github.com/deepfield-ml/JotDB/jotdb0.01 ` then import `github.com/deepfield-ml/JotDB/jotdb0.01 ` in code.\n\n## Usage\n\n### Basic Example\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"jotdb\"\n)\n\nfunc main() {\n\t// Initialize JotDB\n\tstore, err := jotdb.NewJotDB(\"./jotdb\")\n\tif err != nil {\n\t\tfmt.Println(\"Error:\", err)\n\t\treturn\n\t}\n\tdefer store.Close()\n\n\t// Create a JSON document\n\tdoc := map[string]interface{}{\n\t\t\"id\":   \"doc001\",\n\t\t\"data\": map[string]interface{}{\n\t\t\t\"name\":  \"Example\",\n\t\t\t\"score\": 42.5,\n\t\t\t\"tags\":  []interface{}{\"test\", \"demo\"},\n\t\t\t\"active\": true,\n\t\t\t\"meta\":   nil,\n\t\t},\n\t}\n\n\t// Store the document\n\tif err := store.Store(\"doc001\", doc); err != nil {\n\t\tfmt.Println(\"Error storing document:\", err)\n\t\treturn\n\t}\n\n\t// Retrieve the document\n\tvar retrieved map[string]interface{}\n\tif err := store.Retrieve(\"doc001\", \u0026retrieved); err != nil {\n\t\tfmt.Println(\"Error retrieving document:\", err)\n\t\treturn\n\t}\n\tfmt.Printf(\"Retrieved document: %+v\\n\", retrieved)\n\n\t// Delete the document\n\tif err := store.Delete(\"doc001\"); err != nil {\n\t\tfmt.Println(\"Error deleting document:\", err)\n\t\treturn\n\t}\n}\n```\n\n### API\n\n* `NewJotDB(dbPath string) (*JotDB, error)`: Initializes a new JotDB instance at the specified path.\n* `Store(key string, document interface{}) error`: Stores a JSON document with the given key.\n* `Retrieve(key string, target interface{}) error`: Retrieves and unmarshals a document into the target.\n* `Delete(key string) error`: Removes a document by key.\n* `Close() error`: Shuts down the JotDB instance and releases resources.\n\n## Performance\n\n* **Write Throughput** : ~100,000 writes/s for 1KB JSON documents on modern SSDs (thanks to Bitcask’s append-only log).\n* **Read Latency** : Sub-millisecond reads via in-memory key lookups.\n* **JSON Overhead** : Custom serialization/deserialization is ~1.5x faster than Go’s `encoding/json`, with ~20µs serialization and ~30µs deserialization for 1KB documents.\n* **Concurrency** : Supports high read concurrency; write-heavy workloads may benefit from key sharding.\n\n## Benchmarks  \n  \nJotDB has been benchmarked on a standard development machine (Intel Core i5, 16GB RAM, SSD):  \n  \n| Operation | Document Size | Operations/sec | Latency (avg) |  \n|-----------|---------------|----------------|---------------|  \n| Write     | 1KB           | ~100,000       | ~10µs         |  \n| Read      | 1KB           | ~200,000       | ~5µs          |  \n| Delete    | -             | ~150,000       | ~7µs          |  \n  \nJSON serialization performance compared to standard library:  \n  \n| Operation      | Standard Library | JotDB Custom | Improvement |  \n|----------------|------------------|--------------|-------------|  \n| Serialization  | ~30µs            | ~20µs        | ~1.5x       |  \n| Deserialization| ~45µs            | ~30µs        | ~1.5x       |  \n  \n*Benchmarks performed on 1KB JSON documents with mixed types (strings, numbers, booleans, arrays, nested objects)  \n\n## Architecture  \n  \nJotDB follows a simple architecture that combines efficient JSON processing with reliable local storage:  \n  \n```mermaid  \ngraph TD  \n    subgraph \"Client Interface\"  \n        Client[\"Client Application\"] --\u003e JAPI[\"JotDB API\"]  \n    end  \n      \n    subgraph \"Core Components\"  \n        JAPI --\u003e JStruct[\"JotDB Struct\"]  \n        JStruct --\u003e RWMutex[\"Concurrency Controller\"]  \n        JStruct --\u003e BitDB[\"Storage Interface\"]  \n    end  \n      \n    JStruct --\u003e Marshal[\"JSON Serialization\"]  \n    JStruct --\u003e Unmarshal[\"JSON Deserialization\"]  \n      \n    BitDB --\u003e FSStore[\"File System Storage\"]\n```\n\n## File Structure\n\n* `jotdb.go`: Core JotDB struct and public API.\n* `json_marshal.go`: Custom JSON serialization logic.\n* `json_unmarshal.go`: Custom JSON deserialization logic.\n* `types.go`: Placeholder for future type definitions or constants.\n\nThis modular structure ensures maintainability and supports future optimizations, such as enhanced JSON parsing or additional storage features.\n\n## Limitations\n\n* **JSON Parser** : Supports core JSON types but lacks streaming for very large documents.\n* **Scalability** : Bitcask is single-node; for massive datasets, consider sharding or distributed stores.\n* **Error Handling** : Basic error reporting; add logging or retry logic for production use.\n* **Compression** : No built-in compression for large documents (future enhancement).\n\n## Future Improvements\n\n* Add streaming JSON parsing for large documents.\n* Implement secondary indexes for faster queries.\n* Support batch operations for bulk reads/writes.\n* Add optional compression (e.g., zstd) to reduce disk I/O.\n* Enhance error handling with detailed logging and retry mechanisms.\n\n## Contributing\n\nContributions are welcome! Please submit issues or pull requests to improve JotDB. Areas for contribution include:\n\n* Optimizing the JSON parser for specific workloads.\n* Adding support for advanced JSON features (e.g., streaming, schema validation).\n* Implementing additional storage backends or indexing.\n\n## License\n\nJotDB is licensed under the Apache 2.0 License. See [LICENSE](https://github.com/deepfield-ml/JotDB/blob/master/LICENSE) for details.\n\n## Acknowledgments\n\n* Built with [Bitcask](https://github.com/prologic/bitcask) for fast, local key-value storage.\n* Inspired by the need for a simple, high-performance JSON store in Go.\n\n## Authors\n* Deepfield ML - Gordon.H and Will.C\n\n## Version History  \n  \n* **v0.01** - Initial release with core functionality  \n  * Basic CRUD operations  \n  * Custom JSON serialization/deserialization  \n  * Thread-safe operations\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepfield-ml%2Fjotdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepfield-ml%2Fjotdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepfield-ml%2Fjotdb/lists"}