{"id":31635114,"url":"https://github.com/ctimmer/simple-db","last_synced_at":"2025-10-07T00:48:25.283Z","repository":{"id":315974346,"uuid":"1060242783","full_name":"ctimmer/simple-db","owner":"ctimmer","description":"micropython relational DB using btree","archived":false,"fork":false,"pushed_at":"2025-10-02T18:51:19.000Z","size":88,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-02T20:36:42.176Z","etag":null,"topics":["btree","database","micropython","relational-db"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ctimmer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-19T15:40:46.000Z","updated_at":"2025-10-02T18:51:23.000Z","dependencies_parsed_at":"2025-09-22T00:25:42.024Z","dependency_job_id":"6d94f4b9-748e-4829-b53d-fe7215eb2a0a","html_url":"https://github.com/ctimmer/simple-db","commit_stats":null,"previous_names":["ctimmer/simple-db"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ctimmer/simple-db","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctimmer%2Fsimple-db","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctimmer%2Fsimple-db/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctimmer%2Fsimple-db/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctimmer%2Fsimple-db/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ctimmer","download_url":"https://codeload.github.com/ctimmer/simple-db/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctimmer%2Fsimple-db/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278703580,"owners_count":26031205,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-06T02:00:05.630Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["btree","database","micropython","relational-db"],"created_at":"2025-10-07T00:48:17.157Z","updated_at":"2025-10-07T00:48:25.275Z","avatar_url":"https://github.com/ctimmer.png","language":"Python","readme":"# simple-db - micropython relational DB using btree\n\n## Table of Contents\n\n- __[Overview](#overview)__\n- __[Module Functions](#module-functions)__\n  - [Parameter Formats](#parameter-formats)\n  - [Functions](#functions)\n  - [Utility Functions](#utility-functions)\n- __[Remote Server](#client-server-implementation)__\n- __[Internal Database Structure](#internal-database-structure)__\n  - [Other Storage Considerations](#other-storage-considerations)\n    - [msgpack](#msgpack)\n    - [btrees DB](#btrees-db)\n    - [Decimal Numbers](#decimal-numbers)\n    - [Date/Time values](#date-and-time)\n- __[Flowcharts](#flowcharts)__\n  - [Basic](#basic-simpledb-usage)\n  - [Remote Server](#remote-simpledb-usage)\n- __[Installation](#installation)__\n  - [To device](#install-directly-on-the-device-with-mpremote)\n  - [Download file(s) from github](#download-the-source-files-from-github)\n- __[Files](#files)__\n\n## Overview\n\nBasic relational type database interface.\n\n## Quick Start\n\nInstall simple_db.py:\n```\nmpremote mip install github:ctimmer/simple-db/simple_db.py\n```\nIf you don't have mpremote, download simple_db.py from [here](https://github.com/ctimmer/simple-db). Copy the file to your micropython device.  Note: not all firmware versions have the btree module. The unix port does contain the btree module.\n\nRun the following micropython code:\n```\n# Quick Start\n\nimport sys\n\nfrom simple_db import SimpleDB, simpledb_available\n\nif not simpledb_available:\n    print (\"SimpleDB did not initialize: probably missing btree module\")\n    sys.exit ()\n\n# 1. create/open database\nqs_db = SimpleDB (\"qs.db\")\n\n# 2. Add create \"cust\" table, add a row with a primary key: \"cust_no\"\nqs_db.write_row (\"cust\", \"cust_no\", {\"cust_no\":\"0100\",\"first\":\"curt\",\"last\":\"timmerman\"})\n\n# Read row just created\nprint (\"New:\",qs_db.read_row (\"cust\", \"0100\"))\n\n# 3. Add \"location\" to row\nqs_db.rewrite_row (\"cust\", \"0100\", {\"location\":\"Alaska\"})\n\n# Read row just updated\nprint (\"Upd:\",qs_db.read_row (\"cust\", \"0100\"))\n\n# 4. Dump database (1 row) to a text file (\"qs.db.dump.txt\")\nqs_db.dump_all ()\n\n# 5. we are done\nqs_db.close ()\n```\n\nThis code:\n- (1) Creates a btree database called \"qs.db\"\n- (2) Adds a row to the \"cust\" table\n- (3) Updates the row\n- (4) Dumps qs.db to a text file called \"qs.db.dump.txt\"\n  - Shows the btree key, text up to the \"~\"\n  - Row data, text after the \"~\"\n- (5) Closes the database.\n\n## Module Functions\n\n### Parameter Formats\n\n- table_name\n  - Table containing the row data\n  - Name should not contain key_separator or dump_separator\n- pk primary key\n  - ID(s) in the row data used to build the primary key\n  - Must be correct type for the row data (dict or array)\n  - pk can be a scalar value (1 key) or an array (multiple keys)\n  - Key value should not contain key_separator or dump_separator\n- column_list\n  - ID(s) in the row data used to return column values\n  - Must be correct type for the row data (dict or array)\n  - Invalid IDs will be returned with a None value\n  - A dictionary is returned by this function\n- key, start_key, end_key\n  - key value(s) used to access row data\n  - start_key default is low value\n  - end_key default is high value\n- row_data\n  - row data associated with to primary key\n  - 2 possible formats\n    - dict containing ID/VAL pairs\n    - array containing only values\n  - Must include the primary key(s)\n- update_data\n  - Always a dictionary.\n  - Used ID/VAL to update specific columns in the row data\n  - Dictionary rows\n    - if the ID is missing from the row it will be created\n  - Array rows\n    - Only existing columns can be updated\n    - The row array cannot be extended with this function\n- limit\n  - Sets the maximum number of returned values for those functions that return lists\n- epoch_seconds Default: None\n  - If None, returns current date/time values\n  - If set, returns epoch_seconds date/time values\n    - epoch_seconds is typically returned from time.mktime()\n\n### Functions\n\n__init (db_file_path, key_separator, dump_separator, auto_commit)__\n- db_file_path Required\n  - File name of the database\n  - If the file is missing it will be created\n- key_separator Default: \".\"\n  - Character used to separate the concatenated table name and key(s)\n- dump_separator Default: \"~\"\n  - Character used to separate the primary key from the row values\n- auto_commit Default: True\n\n__write_row (table_name, pk, row_data)__\n- Creates or overwrites the row for the specified table/key\n\n__rewrite_row (table_name, key, update_data)__\n- Updates only those table/key columns specified in update_data \n\n__read_row (table_name, key)__\n- Read a row for the specified table/key\n- None is returned if the key does not exist\n\n__read_columns (table_name, key, column_list)__\n- Read a row for the specified table/key\n- None is returned if the key does not exist\n- Returns only those column values from column list\n\n__first_row (table_name, key)__\n- Returns first row with a pk \u003e= key in table\n\n__next_row (table_name, key)__\n- Returns first row with a pk \u003e key in table\n\n__row_exists (table_name, key)__\n- Returns True if the key exists in table\n\n__get_table_keys (table_name, start_key,  end_key, limit)__\n- Returns a list of keys in table from start_key up to end_key\n\n__get_table_rows (table_name, start_key,  end_key, limit)__\n- Returns a list of rows in table from start_key up to end_key\n\n__get_table_items (table_name, start_key, end_key, limit)__\n- Returns a list of key and rows in table from start_key up to end_key\n\n__dump_all (file_path)__\n- Dumps the entire database to a file\n- Format: \"primary key\" + \"~\" + row_data\n- row_data will always be in json format\n\n__load (file_path)__\n- Loads database from file created by dump_all\n\n__commit ()__\n- Flushes updated cached buffers\n\n__close ()__\n- closes btree instance and database file\n\n### Utility Functions\n\n#### Date and Time Functions\n\nThese functions allow storing date/time data in a consistent format. If epoch_seconds is not provided, current date/time is used.\n\n__get_date_time (epoch_seconds)__\n- Returns 'YYYY-MM-DD HH:MM:SS'\n\n__get_date (epoch_seconds)__\n- Returns 'YYYY-MM-DD'\n\n__get_time (epoch_seconds)__\n- Returns 'HH:MM:SS'\n\n## Remote Server Implementation\n\nThese modules will eventually be more generic\n\n- Modules needed\n  - Client application\n    - simple_db_client.py\n      - Creates the json rpc message that is processed by simple_db_server.py\n      - SimpleDBClient class does not handle the network communications.\n      - send_request function handles to network interface.\n  - Server\n    - simple_db_microdot.py\n      - HTTP server using mocrodot module.\n      - POST requests are in json rpc format.\n      - GET requests parameters are converted to json rpc,\n      - Runs on a micropython processor or with the unix port.\n    - simple_db_server.py\n      - Processes the rpc message created by simple_db_client.py\n      - SimpleDBServer class does not handle the network communications.\n    - simple_db_btrees.py (optional)\n      - simple database that uses the btrees module.\n      - Will not run under MP, there is no btrees port.\n      - Btrees is a much more robust database engine.\n      - Database files are NOT compatible with the btree files.\n\n## Internal Database Structure\n\nMicropython's btree database only stores a simple ID/Value pair. Refer to the dump_all example below.\n\n To simulate a database table with a primary key(s) the btree ID is built from the table name concatenated with 1 or more keys in the row data. In the 1st line below the table name is \"customer\" and the primary key is \"000100\" (customer_number). A \"~\" separates the primary key from the row data.\n\n The row data can be in 2 formats, json object or array.\n\nThe 1st line below was created with the following calls (json object):\n\n```\nmy_db.write_row (\"customer\", # table_name\n                \"customer_number\" , # primary key in object\n                {\"customer_number\" : \"000100\" , # row data\n                    \"name\":\"Curt\" ,\n                    \"dob\":19560606 ,\n                    \"occupation\":\"retired\"})\nmy_db.rewrite_row (\"customer\", # table name\n                    \"000100\" , # key\n                    {\"location\" : \"Alaska\"}) # row update\n```\nThe last line below was created with the following calls (json array):\n\n```\nmy_db.write_row (\"log\", # table name\n                0 ,     # primary key array index\n                [\"20250904141020\", # row data\n                \"Warning\",\n                \"Log warning\"])\n```\n\n### dump_all example:\n\n__Format:__ \\\u003ctable_name\\\u003e.\\\u003cprimary key(s)\\\u003e~\\\u003crow data\\\u003e\n```\ncustomer.000100~{\"dob\": 19560606, \"customer_number\": \"000100\", \"name\": \"Curt\", \"occupation\": \"retired\", \"location\": \"Alaska\"}\ncustomer.000500~{\"dob\": 19200101, \"customer_number\": \"000500\", \"name\": \"Moe\", \"occupation\": \"Three stooges\"}\ncustomer.001000~{\"dob\": 19250303, \"customer_number\": \"001000\", \"name\": \"Curly\", \"occupation\": \"Three stooges\"}\ncustomer.010000~{\"dob\": 19210202, \"customer_number\": \"010000\", \"name\": \"Larry\", \"occupation\": \"Three stooges\"}\ninvoice.090001~{\"invoice_number\": \"090001\", \"customer_number\": \"001000\"}\ninvoice_line.090001.0001~{\"invoice_number\": \"090001\", \"sku\": \"Snake Oil\", \"price\": \"100.00\", \"line_number\": \"0001\"}\ninvoice_line.090001.0002~{\"invoice_number\": \"090001\", \"sku\": \"Aspirin\", \"price\": \"12.00\", \"line_number\": \"0002\"}\nlog.20250903122010~[\"20250903122010\", \"Error\", \"Log error\"]\nlog.20250904141020~[\"20250904141020\", \"Warning\", \"Log warning\"]\n```\n\n### Other Storage Considerations\n\n#### msgpack\n\nmsgpack is another way to serialize python dictionaries and arrays.\nIt is more compact and faster than json.\nThe simple_db module can be configured to use msgpack by setting USE_JSON to False.\nThe source can be found [here](https://github.com/peterhinch/micropython-msgpack).\n\n- Notes\n  - msgpack can serialize python entities that are not compatible with json. This would only be a problem if you planned to use the dump_all and load functions (not tested).\n\n#### btrees DB\n\nsimple_db_btrees.py is a direct replacement for simple_db.py.\nIt uses btrees database, a much more robust DB system.\n\n- Notes\n  - This module will NOT run under Micropython. It can only be used with a remote server. See simple_db_microdot.py.\n  - Requires in a virtual environment\n    - pip install BTrees\n    - pip install ZODB\n  - msgpack (optional) requires in the same virtual environment\n    - pip install u-msgpack-python\n\n#### Decimal Numbers\n\nDecimal numbers (e.g. currency) should be stored as strings.\nThe client application can then use [mpy_decimal](https://github.com/mpy-dev/micropython-decimal-number) to manipulate without floating point errors.\n\n#### Date and Time\n\nIf at all possible date/time values should be in the same format. See [utility functions](#utility-functions) for help with this.\n\n\n## Flowcharts\n\n### Basic SimpleDB usage\n\n- All micropython\n- Application and database on same device\n\n```mermaid\nflowchart TD\n    mp_app[MicroPython Application\n    Calls SimpleDB methods]\n    sdb_mod[SimpleDB module\n    Formats keys/row data passed to btree]\n    btree_mod[btree module\n    Manipulates btree database file]\n    btree_db[(btree Database)]\n    mp_app\u003c-- requests / replies --\u003esdb_mod\n    sdb_mod\u003c-- CRUD --\u003ebtree_mod\n    btree_mod\u003c-- update / retrieve --\u003ebtree_db\n```\n\n### Remote SimpleDB usage\n\n- All micropython\n- Database server on a separate device\n- micropython unix port can be used on unix server\n- Btree database will be compatible with other micropython created btree files \n\n```mermaid\nflowchart TD\n    mp_app[MicroPython Application\n    Calls SimpleDBClient methods, same as SimpleDB]\n    sdb_client_mod[SimpleDB client module\n    Formats method calls to json RPC]\n    comm_interface[Communications\n    Delivers json message to server and returns reply]\n    sdb_server_mod[SimpleDB server\n    Translates json RPC requests to SimpleDB method calls]\n    btree_mod[btree module]\n    btree_db[(btree Database)]\n    mp_app\u003c-- requests / replies --\u003esdb_client_mod\n    sdb_client_mod\u003c-- SimpleDB methods to json RPC --\u003ecomm_interface\n    comm_interface\u003c-- json interface --\u003esdb_server_mod\n    sdb_server_mod\u003c-- json RPC to SimpleDB methods --\u003esdb_mod\n    sdb_mod[SimpleDB module\n    Formats keys/row data passed to btree]\u003c-- CRUD --\u003ebtree_mod[btree module\n    Manipulates btree database file]\n    btree_mod\u003c-- update / retrieve --\u003ebtree_db\n```\n\n## Installation\n\n### install directly on the device with mpremote:\n\n__simple_db.py__\n```\nmpremote mip install github:ctimmer/simple-db/simple_db.py\n```\n\n__simple_db_client.py__\n```\nmpremote mip install github:ctimmer/simple-db/simple_db_client.py\n```\n\n__simple_db_server.py__\n```\nmpremote mip install github:ctimmer/simple-db/simple_db_server.py\n```\n\n__simple_db_microdot.py__\n```\nmpremote mip install github:ctimmer/simple-db/simple_db_microdot.py\n```\n\n### Download the source files from github:\n\n__simple_db.py__\n```\nwget https://raw.githubusercontent.com/ctimmer/simple-db/main/simple_db.py\n```\n\n__simple_db_client.py__\n```\nwget https://raw.githubusercontent.com/ctimmer/simple-db/main/simple_db_client.py\n```\n\n__simple_db_server.py__\n```\nwget https://raw.githubusercontent.com/ctimmer/simple-db/main/simple_db_server.py\n```\n\n__simple_db_microdot.py__\n```\nwget https://raw.githubusercontent.com/ctimmer/simple-db/main/simple_db_microdot.py\n```\n\n__simple_db_btrees.py__\n```\nwget https://raw.githubusercontent.com/ctimmer/simple-db/main/simple_db_btrees.py\n```\n\n## Files\n\n- simple_db.py\n  - Provides a relational type database interface using btree\n- simple_db_btrees.py\n  - Experimental, uses btrees database but is functionally equivalent to simple_db.py\n- simple_db_tester.py\n  - Not implemented yet but I plan to move the main function code from the modules to this application.\n- simple_db_client.py\n  - simple_db interface to a remote server.\n  - Function call are the same as simple_db.py\n  - Tested with simple_db_microdot.py server.\n- simple_db_server.py\n  - Accepts json RPC database requests and returns the results.\n  - This module does not handle any communications.\n  - Imports btree.\n- simple_db_microdot.py\n  - HTTP server that handles POST (json RPC) and GET requests.\n  - All results are returned in json RPC format. \n  - Imports simple_db_server\n- README.md\n  - This documentation file","funding_links":[],"categories":["Libraries"],"sub_categories":["Storage"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fctimmer%2Fsimple-db","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fctimmer%2Fsimple-db","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fctimmer%2Fsimple-db/lists"}