{"id":23910060,"url":"https://github.com/michelp/postgres-sqlite","last_synced_at":"2025-02-23T16:46:33.209Z","repository":{"id":270923993,"uuid":"910936470","full_name":"michelp/postgres-sqlite","owner":"michelp","description":"In Memory SQLlite as a Postgres data type","archived":false,"fork":false,"pushed_at":"2025-01-04T03:48:34.000Z","size":30,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-04T04:24:50.270Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michelp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-01T20:53:59.000Z","updated_at":"2025-01-04T03:48:37.000Z","dependencies_parsed_at":"2025-01-04T04:24:54.611Z","dependency_job_id":"7c3511f8-b2f1-45ad-89cb-9bcd24607d19","html_url":"https://github.com/michelp/postgres-sqlite","commit_stats":null,"previous_names":["michelp/postgres-sqlite"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michelp%2Fpostgres-sqlite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michelp%2Fpostgres-sqlite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michelp%2Fpostgres-sqlite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michelp%2Fpostgres-sqlite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michelp","download_url":"https://codeload.github.com/michelp/postgres-sqlite/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240347782,"owners_count":19787231,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-05T06:40:03.117Z","updated_at":"2025-02-23T16:46:33.190Z","avatar_url":"https://github.com/michelp.png","language":"C","funding_links":[],"categories":["C"],"sub_categories":[],"readme":"# postgres-sqlite\n\n`postgres-sqlite` is a PostgreSQL extension that introduces SQLite\ndatabases as a first-class data type within PostgreSQL.\n\nBy leveraging the PostgreSQL \"expanded datum\" API and SQLite's\n`sqlite3_serialize`/`sqlite3_deserialize` APIs, `postgres-sqlite`\nenables the efficient creation and storage of small SQLite databases\n(up to 1GB) directly within PostgreSQL tables. This efficiency is\nachieved through PostgreSQL's compressed TOAST storage in conjunction\nwith SQLite's native serialization API. Importantly, `postgres-sqlite`\ndoes not require filesystem access—SQLite databases are fully managed\nwithin PostgreSQL tables.\n\n### Key Advantages\n\n#### Multitenancy\n`postgres-sqlite` simplifies multitenancy by embedding isolated SQLite\ndatabases directly into PostgreSQL. Each SQLite database is entirely\nindependent from its surrounding PostgreSQL environment, eliminating\nthe need for complex RLS policies, permissions, alternate database in\na cluster, or other traditional multitenancy mechanisms. This allows\nuser data to be securely stored in separate rows without additional\nconfiguration.\n\n#### Client-Server Synchronization\nAnother powerful use case is pairing `postgres-sqlite` with the\nofficial SQLite Wasm (WebAssembly) build to synchronize client-side\nand server-side data. You can seamlessly exchange data between client\nand server using either SQLite’s native SQL text format or the\nstandard binary format via `sqlite_serialize()` and\n`sqlite_deserialize()`.\n\n## Creating SQLite Databases\n\nThe extension can be installed into a Postgres in the normal way:\n\n```\nCREATE EXTENSION sqlite;\n```\n\nThere are three main objects in the extension, the `sqlite` type and\nthe `sqlite_exec()` and `sqlite_query()` functions.  New databases can\nbe created by casting an initialization string to the `sqlite` type:\n\n```\nSELECT 'CREATE TABLE user_config (key text, value text)'::sqlite;\n┌──────────────────────────────────────────────────┐\n│                      sqlite                      │\n├──────────────────────────────────────────────────┤\n│ PRAGMA foreign_keys=OFF;                        ↵│\n│ BEGIN TRANSACTION;                              ↵│\n│ CREATE TABLE user_config (key text, value text);↵│\n│ COMMIT;                                         ↵│\n│                                                  │\n└──────────────────────────────────────────────────┘\n(1 row)\n```\n\nNote that while you are *seeing* the SQL text representation of the\nsqlite object, it is stored internally in Postgres as a compact binary\nrepresentation of an in-memory sqlite database.  When the sqlite\ndatabase is accessed, it is automatically \"expanded\" from the on-disk\nbinary representation to a live sqlite database using\n`sqlite3_deserialize()`.  Conversely if and when it is written to a\ntable, it is serialized and \"TOASTed\" into a binary flat\nrepresentation with `sqlite3_serialize()`.\n\n## Modifying SQLite Objects\n\nThis new sqlite instance can now be inserted into a table and\nmanipulated with the `sqlite_exec()` function, for example:\n\n```\nCREATE TABLE customer (\n    id bigserial PRIMARY KEY,\n    name text NOT NULL,\n    data sqlite DEFAULT 'CREATE TABLE user_config (key text, value text);'\n    );\n\nINSERT INTO customer (name) VALUES ('bob') RETURNING *;\n┌────┬──────┬──────────────────────────────────────────────────┐\n│ id │ name │                       data                       │\n├────┼──────┼──────────────────────────────────────────────────┤\n│  1 │ bob  │ PRAGMA foreign_keys=OFF;                        ↵│\n│    │      │ BEGIN TRANSACTION;                              ↵│\n│    │      │ CREATE TABLE user_config (key text, value text);↵│\n│    │      │ COMMIT;                                         ↵│\n│    │      │                                                  │\n└────┴──────┴──────────────────────────────────────────────────┘\n(1 row)\n\nUPDATE customer\n    SET data = sqlite_exec(data, $$INSERT INTO user_config VALUES ('color', 'blue')$$)\n    RETURNING *;\n┌────┬──────┬──────────────────────────────────────────────────────────────────────┐\n│ id │ name │                                 data                                 │\n├────┼──────┼──────────────────────────────────────────────────────────────────────┤\n│  1 │ bob  │ PRAGMA foreign_keys=OFF;                                            ↵│\n│    │      │ BEGIN TRANSACTION;                                                  ↵│\n│    │      │ CREATE TABLE user_config (key text, value text);                    ↵│\n│    │      │ INSERT INTO user_config(rowid,\"key\",value) VALUES(1,'color','blue');↵│\n│    │      │ COMMIT;                                                             ↵│\n│    │      │                                                                      │\n└────┴──────┴──────────────────────────────────────────────────────────────────────┘\n(1 row)\n```\n\nNotice how the `DEFAULT` value for the sqlite column in the new table\nwill initialize a new sqlite database into that column.  All new rows\nin `customer` will contain contain a sqlite database named `data`\nwhich in turn contains a sqlite table `user_config`.\n\nThe `sqlite_exec(db, query)` function takes a sqlite database and a\nquery as an argument, executes that query and returns the same\ndatabase, so this can be used for chaining updates to the same\ndatabase through multiple calls.\n\n## Querying SQLite Objects\n\nThe `sqlite_query(db, query)` function is a Set Returning Function\n(SRF) that `RETURNS SETOF RECORD` with a postgres row for each sqlite\nresult row from the given sqlite query.  Using the standard 'AS'\nsyntax these values can be mapped to table like Postgres results:\n\n```\nSELECT * from sqlite_query(\n        (SELECT data FROM customer),\n        'SELECT rowid, key, value from user_config')\n    AS (id integer, key text, value text);\n┌────┬───────┬───────┐\n│ id │  key  │ value │\n├────┼───────┼───────┤\n│  1 │ color │ blue  │\n└────┴───────┴───────┘\n(1 row)\n```\n\npostgres-sqlite support integers, floats and text.\n\n## Serialize/Deserialize\n\npostgres-sqlite has support for serializing and deserializing sqlite\ndatabases as `bytea` byte array types.\n\n```\nSELECT pg_typeof(sqlite_serialize('create table foo (x)'));\n┌───────────┐\n│ pg_typeof │\n├───────────┤\n│ bytea     │\n└───────────┘\n(1 row)\n```\n\n`sqlite_deserialize` takes a `bytea` representation of the database\nand expands it into sqlite object:\n\n```\nSELECT sqlite_deserialize(sqlite_serialize('create table foo (x)'));\n┌──────────────────────────┐\n│    sqlite_deserialize    │\n├──────────────────────────┤\n│ PRAGMA foreign_keys=OFF;↵│\n│ BEGIN TRANSACTION;      ↵│\n│ CREATE TABLE foo (x);   ↵│\n│ COMMIT;                 ↵│\n│                          │\n└──────────────────────────┘\n(1 row)\n```\n\n## How it Works\n\nMost Postgres data types, like numbers and text, are \"flat\" and have\nidentical in-memory and on-disk representations.  Postgres loads those\nfrom disk into memory as is, and copies those datum around during the\ncourse of a SQL query.\n\nSome data types however, like arrays, and in this case, sqlite\ndatabases, are more efficiently represented in memory differently than\ntheir on-disk representation.  The act of copying flat datum around\nalso can cost a lot of cycles and memory for large, flat variable\nlength objects.\n\nPostgres has an API called the [Expanded Datum\nAPI](https://www.postgresql.org/docs/current/xtypes.html#XTYPES-TOAST)\nthat will \"expand\" a flat value into a more efficient in-memory\nrepresentation (in this case, a live in-memory sqlite db deserialized\nfrom TOAST storage) and \"flatten\" the expanded object back into TOAST\nwhen it's time to write it to the database.  The\nsqlite3_serialize/deserialize API is used to store the sqlite database\nin its native compact binary storage format inside Postgres.\n\nBy expanding a serialized \"flat\" sqlite database into an in-memory\nobject, this datum copying just copies a single pointer to the\ndatabase around during a sql command instead of the entire flattened\nobject.  This is particularly useful for plpgsql which can detect\nexpanded objects and handle references to them as pointers instead of\nflat objects.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichelp%2Fpostgres-sqlite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichelp%2Fpostgres-sqlite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichelp%2Fpostgres-sqlite/lists"}