{"id":21126985,"url":"https://github.com/pyrustic/jinbase","last_synced_at":"2025-04-06T17:12:14.643Z","repository":{"id":198338701,"uuid":"697280155","full_name":"pyrustic/jinbase","owner":"pyrustic","description":"Multi-model transactional embedded database","archived":false,"fork":false,"pushed_at":"2024-12-10T20:58:01.000Z","size":71,"stargazers_count":68,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-30T16:12:12.782Z","etag":null,"topics":["blob","database","embedded-data","kv-store","multi-model","persistence","pyrustic","python","queue","relational-database","sql","sqlite","stack"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pyrustic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-27T12:10:13.000Z","updated_at":"2025-03-15T21:27:48.000Z","dependencies_parsed_at":"2025-01-29T00:13:53.074Z","dependency_job_id":"edd77ae9-9474-4c62-ba22-7e42eae4b80a","html_url":"https://github.com/pyrustic/jinbase","commit_stats":null,"previous_names":["pyrustic/jinbase"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyrustic%2Fjinbase","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyrustic%2Fjinbase/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyrustic%2Fjinbase/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyrustic%2Fjinbase/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pyrustic","download_url":"https://codeload.github.com/pyrustic/jinbase/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247517915,"owners_count":20951719,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blob","database","embedded-data","kv-store","multi-model","persistence","pyrustic","python","queue","relational-database","sql","sqlite","stack"],"created_at":"2024-11-20T04:46:10.529Z","updated_at":"2025-04-06T17:12:14.617Z","avatar_url":"https://github.com/pyrustic.png","language":"Python","readme":"[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![PyPI package version](https://img.shields.io/pypi/v/jinbase)](https://pypi.org/project/jinbase)\n[![Downloads](https://static.pepy.tech/badge/jinbase)](https://pepy.tech/project/jinbase)\n\n\n\u003c!-- Cover --\u003e\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/pyrustic/misc/master/assets/jinbase/cover.jpg\" alt=\"Cover image\" width=\"800\"\u003e\n    \u003cp align=\"center\"\u003e\n        \u003ca href=\"https://commons.wikimedia.org/wiki/File:Rudolf_Reschreiter_Blick_von_der_H%C3%B6llentalangerh%C3%BCtte_zum_H%C3%B6llentalgletscher_und_den_Riffelwandspitzen_1921.jpg\"\u003eRudolf Reschreiter\u003c/a\u003e, Public domain, via Wikimedia Commons\n    \u003c/p\u003e\n\u003c/div\u003e\n\n\u003c!-- Intro Text --\u003e\n# Jinbase\n\u003cb\u003eMulti-model transactional embedded database\u003c/b\u003e\n\n\n## Table of contents\n- [Overview](#overview)\n    - [Multiple data models coexisting in a single embedded database](#multiple-data-models-coexisting-in-a-single-embedded-database)\n    - [Support for transactions and complex data of arbitrary size](#support-for-transactions-and-complex-data-of-arbitrary-size)\n    - [Bulk and partial access to records from byte-level to field-level](#bulk-and-partial-access-to-records-from-byte-level-to-field-level)\n    - [Highly configurable database and timestamped records](#highly-configurable-database-and-timestamped-records)\n- [Why use Jinbase ?](#why-use-jinbase-)\n- [Data models and their corresponding storage interfaces](#data-models-and-their-corresponding-storage-interfaces)\n    - [Kv](#kv)\n    - [Depot](#depot)\n    - [Queue](#queue)\n    - [Stack](#stack)\n    - [Relational](#relational)\n- [The unified BLOB interface](#the-unified-blob-interface)\n- [Command line interface](#command-line-interface)\n- [Related projects](#related-projects)\n- [Testing and contributing](#testing-and-contributing)\n- [Installation](#installation)\n\n\n# Overview\n**Jinbase** (pronounced as **/ˈdʒɪnˌbeɪs/**) is a multi-model [transactional](https://en.wikipedia.org/wiki/Database_transaction) [embedded database](https://en.wikipedia.org/wiki/Embedded_database) that uses [SQLite](https://www.sqlite.org/) as storage engine. Its reference implementation is an eponymous lightweight [Python](https://www.python.org/) library available on [PyPI](#installation).\n\n\n## Multiple data models coexisting in a single embedded database\nA single Jinbase database supports **key-value**, **depot**, **queue**, **stack**, and **relational** data models. While a Jinbase file can be populated with multi-model data, depending on the needs, it is quite possible to dedicate a database file to a given model.\n\nFor each of the first four data models, there is a programmatic interface accessible via an eponymous property of a Jinbase instance.\n\n## Support for transactions and complex data of arbitrary size\nHaving SQLite as the storage engine allows Jinbase to benefit from [transactions](https://www.sqlite.org/lang_transaction.html). Jinbase ensures that at the top level, reads and writes on key-value, depot, queue and stack stores are transactional. For user convenience, context managers are exposed to create transactions of different modes.\n\nWhen for a write operation, the user submits data, whether it is a dictionary, string or integer, Jinbase serializes (except binary data), chunks and stores the data iteratively with the [Paradict](https://github.com/pyrustic/paradict) compact binary data format. This then allows for the smooth storage of complex data of [arbitrary size](https://www.sqlite.org/limits.html) with Jinbase.\n\n## Bulk and partial access to records from byte-level to field-level\nJinbase not only offers bulk access to records, but also two levels of partial access granularity.\n\nSQLite has an impressive capability which is [incremental I/O](https://sqlite.org/c3ref/blob_open.html) for [BLOBs](https://www.sqlite.org/datatype3.html). While this capability is designed to target an individual BLOB column in a row, Jinbase extends this so that for each record, incremental reads cover all chunks as if they were a single [unified BLOB](#the-unified-blob-interface).\n\nFor [dictionary](https://en.wikipedia.org/wiki/Associative_array) records only, Jinbase automatically creates and maintains a lightweight index consisting of pointers to root fields, which then allows extracting from an arbitrary record the contents of a field automatically deserialized before being returned.\n\n## Highly configurable database and timestamped records\nJinbase exposes a database connection object to the underlying SQLite storage engine, allowing for sophisticated configuration. The [Paradict](https://github.com/pyrustic/paradict) binary data format used for serializing records also allows for customizing data types via a `paradict.TypeRef` object.\n\nEach record stored in a key-value, depot, queue, or stack store is automatically timestamped. This allows the user to provide a `time_range` tuple when querying records. The precision of the timestamp, which defaults to milliseconds, can also be configured.\n\n\n\n# Why use Jinbase ?\nJinbase implements persistence for familiar data models whose stores coexist in a single file with an intuitive programmatic interface. Supported [data types](https://github.com/pyrustic/paradict) range from simple to complex and of [arbitrary size](https://www.sqlite.org/limits.html).\n\nFor convenience, all Jinbase-related tables are prefixed with `jinbase_`, allowing the user to define their own tables and interact with them as they would with a regular SQLite database. \n\nThanks to its multi-model coexistence capability, Jinbase can be used to open legacy SQLite databases to add four useful data models (key-value, depot, queue, and stack).\n\nAll this makes Jinbase relevant from prototype to production stages of software development of various sizes and scopes. \n\nFollowing are few of the most obvious use cases:\n\n- Storing user preferences\n- Persisting session data before exit\n- Order-based processing of data streams\n- Exposing data for other processes\n- Upgrading legacy SQLite files with new data models\n- Bespoke data persistence solution\n\n# Data models and their corresponding storage interfaces\nFollowing subsections discuss data models and their corresponding storage interfaces.\n\n## Kv\nThe [key-value](https://en.wikipedia.org/wiki/Key%E2%80%93value_database) data model associates to a string or an integer key, a value that is serializable with the [Paradict](https://github.com/pyrustic/paradict) binary data format. \n\nString keys can be searched with a [glob](https://en.wikipedia.org/wiki/Glob_(programming)) pattern and integer keys can be searched within a range of numbers. Since records are automatically timestamped, a `time_range` tuple can be provided by the user to search for them as well as keys.\n\nExample:\n\n```python\nimport os.path\nfrom datetime import datetime\nfrom jinbase import Jinbase, JINBASE_HOME\n\nuser_data = {\"id\": 42, \"name\": \"alex\", \"created_at\": datetime.now(),\n             \"photo\": b'\\x45\\xA6\\x42\\xDF\\x69',\n             \"books\": {\"sci-fi\": [\"book 1\", \"book 2\"],\n                       \"thriller\": [\"book 3\", [\"book4\"]]}}\n\ndb_filename = os.path.join(JINBASE_HOME, \"test.db\")\n\nwith Jinbase(db_filename) as db:\n  # set 'user'\n  kv_store = db.kv\n  kv_store.set(\"user\", user_data)  # returns a UID\n\n  # get 'user'\n  data = kv_store.get(\"user\")\n  assert data == user_data\n\n  # count total records and bytes\n  print(kv_store.count_records())\n  print(kv_store.count_bytes(\"user\"))\n\n  # list keys (the time_range is optional)\n  time_range = (\"2024-11-20 10:00:00Z\", \"2035-11-20 10:00:00Z\")\n  print(tuple(kv_store.keys(time_range=time_range)))\n\n  # find string keys with a glob pattern\n  print(tuple(kv_store.str_keys(glob=\"use*\")))\n\n  # load the 'books' field (partial access)\n  books = kv_store.load_field(\"user\", \"books\")\n  assert books == user_data[\"books\"]\n\n  # iterate (descending order)\n  for key, value in kv_store.iterate(asc=False):\n    pass\n```\n\u003e Check out the API reference for the [key-value store](https://github.com/pyrustic/jinbase/blob/master/docs/api/modules/jinbase/store/kv/class-Kv.md).\n\n\n## Depot\nThe depot data model shares similarities with the [List](https://en.wikipedia.org/wiki/List_(abstract_data_type)) data structure. An\nunique identifier (UID) is automatically assigned to a record appended to the store. This record can be retrieved later either by its unique identifier or by its [0-based](https://en.wikipedia.org/wiki/Zero-based_numbering) position in the store.\n\nExample:\n\n```python\nimport os.path\nfrom datetime import datetime\nfrom jinbase import Jinbase, JINBASE_HOME\n\nuser_data = {\"id\": 42, \"name\": \"alex\", \"created_at\": datetime.now(),\n             \"photo\": b'\\x45\\xA6\\x42\\xDF\\x69',\n             \"books\": {\"sci-fi\": [\"book 1\", \"book 2\"],\n                       \"thriller\": [\"book 3\", [\"book4\"]]}}\n\ndb_filename = os.path.join(JINBASE_HOME, \"test.db\")\n\nwith Jinbase(db_filename) as db:\n    # append 'user_data' to the depot\n    depot_store = db.depot\n    uid = depot_store.append(user_data)\n\n    # get 'user_data'\n    data = depot_store.get(uid)\n    assert data == user_data\n\n    # get the record at position 0 in the depot\n    print(depot_store.uid(0))  # prints the UID\n    # get the position of a record in the depot\n    print(depot_store.position(uid))  # prints the position\n\n    # count total records and bytes\n    print(depot_store.count_records())\n    print(depot_store.count_bytes(uid))\n\n    # list UIDs (unique identifiers)\n    time_range = (\"2024-11-20 10:00:00Z\", \"2035-11-20 10:00:00Z\")\n    print(tuple(depot_store.uids(time_range=time_range)))\n\n    # load the 'books' field (partial access)\n    books = depot_store.load_field(uid, \"books\")\n    assert books == user_data[\"books\"]\n\n    # iterate (descending order)\n    for uid, data in depot_store.iterate(asc=False):\n        pass\n```\n\n\u003e Check out the API reference for the [depot store](https://github.com/pyrustic/jinbase/blob/master/docs/api/modules/jinbase/store/depot/class-Depot.md).\n\n## Queue\nThe [queue](https://en.wikipedia.org/wiki/Queue_(abstract_data_type)) data model like other stores, is transactional. While this store provides methods to enqueue and dequeue records, there is also `peek_xxx` methods to look at the record at the front or the back of the queue, that is, read it without dequeuing.\n\nExample:\n \n```python\nimport os.path\nfrom datetime import datetime\nfrom jinbase import Jinbase, JINBASE_HOME\n\nuser_data = {\"id\": 42, \"name\": \"alex\", \"created_at\": datetime.now(),\n             \"photo\": b'\\x45\\xA6\\x42\\xDF\\x69',\n             \"books\": {\"sci-fi\": [\"book 1\", \"book 2\"],\n                       \"thriller\": [\"book 3\", [\"book4\"]]}}\n\ndb_filename = os.path.join(JINBASE_HOME, \"test.db\")\n\nwith Jinbase(db_filename) as db:\n    # enqueue 'user_data'\n    queue_store = db.queue\n    queue_store.enqueue(user_data)  # returns a UID\n\n    # peek\n    data1 = queue_store.peek_front()\n    data2 = queue_store.peek_back()\n    assert data1 == data2 == user_data\n\n    # dequeue\n    data = queue_store.dequeue()\n    assert data == user_data\n\n    # we could have dequeued the record inside a transaction\n    # to ensure that its processing completed successfully\n    # (if it fails, an automatic rollback is performed)\n    with db.write_transaction():\n        data = queue_store.dequeue()\n        # from here, process the data\n        ...\n```\n\n\u003e Check out the API reference for the [queue store](https://github.com/pyrustic/jinbase/blob/master/docs/api/modules/jinbase/store/queue/class-Queue.md).\n\n## Stack\nThe [stack](https://en.wikipedia.org/wiki/Stack_(abstract_data_type)) data model like other stores, is transactional. While this store provides methods to push and pop records, there is also a `peek` method to look at the record on top of the stack, that is, read it without popping it from the stack.\n\n```python\nimport os.path\nfrom datetime import datetime\nfrom jinbase import Jinbase, JINBASE_HOME\n\nuser_data = {\"id\": 42, \"name\": \"alex\", \"created_at\": datetime.now(),\n             \"photo\": b'\\x45\\xA6\\x42\\xDF\\x69',\n             \"books\": {\"sci-fi\": [\"book 1\", \"book 2\"],\n                       \"thriller\": [\"book 3\", [\"book4\"]]}}\n\ndb_filename = os.path.join(JINBASE_HOME, \"test.db\")\n\nwith Jinbase(db_filename) as db:\n    # push 'user_data' on top of the stack\n    stack_store = db.stack\n    stack_store.push(user_data)\n\n    # peek\n    data = stack_store.peek()\n    assert data == user_data\n\n    # pop 'user_data'\n    data = stack_store.pop()\n    assert data == user_data\n\n    # we could have popped the record inside a transaction\n    # to ensure that its processing completed successfully\n    # (if it fails, an automatic rollback is performed)\n    with db.write_transaction():\n        data = stack_store.pop()\n        # from here, process the data\n        ...\n```\n\n\u003e Check out the API reference for the [stack store](https://github.com/pyrustic/jinbase/blob/master/docs/api/modules/jinbase/store/stack/class-Stack.md).\n\n## Relational\nAs Jinbase uses [SQLite](https://en.wikipedia.org/wiki/SQLite) as its storage engine, it de facto supports the [relational](https://en.wikipedia.org/wiki/Relational_model) data model for which it exposes an interface, [LiteDBC](https://github.com/pyrustic/litedbc), for querying SQLite.\n\nLiteDBC is an SQL interface compliant with the DB-API 2.0 specification described by [PEP 249](https://peps.python.org/pep-0249/), itself wrapping Python's [sqlite3](https://docs.python.org/3/library/sqlite3.html) module for a more intuitive interface and multithreading support by default.\n\nExample:\n\n```python\nimport os.path\nfrom jinbase import Jinbase, JINBASE_HOME\n\ndb_filename = os.path.join(JINBASE_HOME, \"test.db\")\n\nwith Jinbase(db_filename) as db:\n    lite_dbc = db.dbc\n    with lite_dbc.transaction() as cursor:\n        # query the table names that exist in this database\n        query = (\"SELECT name FROM sqlite_master \"\n                 \"WHERE type='table' AND name NOT LIKE 'sqlite_%'\")\n        cursor.execute(query)\n        # Although the Cursor object already has the traditional \"fetchone\",\n        # \"fetchmany\" and \"fetchall\" methods, LiteDBC adds a new lazy \"fetch\"\n        # method for intuitive iteration.\n        # Note that 'fetch()' accepts 'limit' and 'buffer_size' as arguments.\n        for row in cursor.fetch():\n            table_name = row[0]\n            print(table_name)\n\n```\n\n\u003e Check out [LiteDBC](https://github.com/pyrustic/litedbc).\n\n# The unified BLOB interface\nWhen for a write operation, the user submits data, whether it is a dictionary, string or integer, Jinbase serializes (except binary data), chunks and stores the data iteratively with the [Paradict](https://github.com/pyrustic/paradict) compact binary data format. Under the hood, these chunks are actually stored as SQLite Binary Large Objects ([BLOBs](https://www.sqlite.org/datatype3.html)). \n\nSQLite has an impressive capability which is [incremental I/O](https://sqlite.org/c3ref/blob_open.html) for BLOBs. While this capability is designed to target an individual BLOB column in a row, Jinbase extends it to enable incremental reads of record chunks as if they form a single [unified BLOB](#the-unified-blob-interface).\n\nExample:\n\n```python\nimport os.path\nfrom jinbase import Jinbase, JINBASE_HOME\n\ndb_filename = os.path.join(JINBASE_HOME, \"test.db\")\nCHUNK_SIZE = 1  # 1 byte, thus a 5-byte input will have 5 chunks\n\n# The 'chunk_size' can be defined only once when Jinbase creates or opens\n# the database for first time. New values for 'chunk_size' will be ignored.\n# So, for this example to work, ensure that the 'db_filename' is nonexistent.\nwith Jinbase(db_filename, chunk_size=CHUNK_SIZE) as db:\n    # some binary data\n    USER_DATA = b'\\x20\\x55\\xA9\\xBC\\x69\\x42\\xD1'  # seven bytes !\n    # set the data\n    kv_store = db.kv\n    kv_store.set(\"user\", USER_DATA)\n    # count chunks\n    n_chunks = kv_store.count_chunks(\"user\")\n    assert n_chunks == len(USER_DATA)  # seven bytes !\n\n    # access the unified blob interface for incremental reads\n    with kv_store.open_blob(\"user\") as blob:\n        # read the entire unified blob\n        data = blob.read()\n        assert data == USER_DATA\n        assert blob.tell() == len(USER_DATA)  # cursor position\n        assert blob.read() == b''\n        # move the cursor back to the beginning of the blob\n        blob.seek(0)\n        # read the first byte\n        assert blob.read(1) == bytes([USER_DATA[0]]) \n        # read the last byte\n        assert blob[-1] == bytes([USER_DATA[-1]])\n        # read a slice\n        slice_obj = slice(2, 5)\n        assert blob[slice_obj] == USER_DATA[slice_obj]\n```\n\n\u003e The unified BLOB interface for incremental reads will only work on Python \u003e=3.11\n\n# Command line interface\nNot yet implemented.\n\n# Related projects\n- [LiteDBC](https://github.com/pyrustic/litedbc): Lite database connector\n- [Paradict](https://github.com/pyrustic/paradict): Streamable multi-format serialization with schema \n- [Asyncpal](https://github.com/pyrustic/asyncpal): Preemptive concurrency and parallelism for sporadic workloads \n- [KvF](https://github.com/pyrustic/kvf): The key-value file format with sections \n\n# Testing and contributing\nFeel free to **open an issue** to report a bug, suggest some changes, show some useful code snippets, or discuss anything related to this project. You can also directly email [me](https://pyrustic.github.io/#contact).\n\n## Setup your development environment\nFollowing are instructions to setup your development environment\n\n```bash\n# create and activate a virtual environment\npython -m venv venv\nsource venv/bin/activate\n\n# clone the project then change into its directory\ngit clone https://github.com/pyrustic/jinbase.git\ncd jinbase\n\n# install the package locally (editable mode)\npip install -e .\n\n# run tests\npython -m tests\n\n# deactivate the virtual environment\ndeactivate\n```\n\n\u003cp align=\"right\"\u003e\u003ca href=\"#readme\"\u003eBack to top\u003c/a\u003e\u003c/p\u003e\n\n# Installation\n**Jinbase** is **cross-platform**. It is built on [Ubuntu](https://ubuntu.com/download/desktop) and should work on **Python 3.8** or **newer**.\n\n## Create and activate a virtual environment\n```bash\npython -m venv venv\nsource venv/bin/activate\n```\n\n## Install for the first time\n\n```bash\npip install jinbase\n```\n\n## Upgrade the package\n```bash\npip install jinbase --upgrade --upgrade-strategy eager\n```\n\n## Deactivate the virtual environment\n```bash\ndeactivate\n```\n\n\u003cp align=\"right\"\u003e\u003ca href=\"#readme\"\u003eBack to top\u003c/a\u003e\u003c/p\u003e\n\n# About the author\nHello world, I'm Alex (😎️), a tech enthusiast and the architect of [Pyrustic](https://pyrustic.github.io) ! Feel free to get in touch with [me](https://pyrustic.github.io/#contact) !\n\n\u003cbr\u003e\n\u003cbr\u003e\n\u003cbr\u003e\n\n[Back to top](#readme)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyrustic%2Fjinbase","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpyrustic%2Fjinbase","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyrustic%2Fjinbase/lists"}