{"id":13393445,"url":"https://github.com/jlongster/absurd-sql","last_synced_at":"2025-05-14T23:04:20.067Z","repository":{"id":38322028,"uuid":"384302754","full_name":"jlongster/absurd-sql","owner":"jlongster","description":"sqlite3 in ur indexeddb (hopefully a better backend soon)","archived":false,"fork":false,"pushed_at":"2023-08-06T03:06:36.000Z","size":12754,"stargazers_count":4236,"open_issues_count":41,"forks_count":104,"subscribers_count":38,"default_branch":"master","last_synced_at":"2025-04-13T19:49:50.451Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jlongster.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-07-09T02:46:06.000Z","updated_at":"2025-04-10T13:57:21.000Z","dependencies_parsed_at":"2024-02-03T20:49:35.524Z","dependency_job_id":null,"html_url":"https://github.com/jlongster/absurd-sql","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jlongster%2Fabsurd-sql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jlongster%2Fabsurd-sql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jlongster%2Fabsurd-sql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jlongster%2Fabsurd-sql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jlongster","download_url":"https://codeload.github.com/jlongster/absurd-sql/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254243358,"owners_count":22038046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T17:00:52.909Z","updated_at":"2025-05-14T23:04:15.033Z","avatar_url":"https://github.com/jlongster.png","language":"JavaScript","funding_links":["https://www.buymeacoffee.com/jlongster"],"categories":["JavaScript"],"sub_categories":[],"readme":"\nThis is an absurd project.\n\nIt implements a backend for [sql.js](https://github.com/sql-js/sql.js/) (sqlite3 compiled for the web) that treats IndexedDB like a disk and stores data in blocks there. That means your sqlite3 database is persisted. And not in the terrible way of reading and writing the whole image at once -- it reads and writes your db in small chunks.\n\nIt basically stores a whole database into another database. Which is absurd.\n\n[See the demo](https://priceless-keller-d097e5.netlify.app/). You can also view an entire app using this [here](https://app-next.actualbudget.com/?wtf_source=absurd).\n\nYou should also read [this blog post](https://jlongster.com/future-sql-web) which explains the project in great detail.\n\nIf you like my work, feel free to [buy me a coffee!](https://www.buymeacoffee.com/jlongster)\n\n## How do I use it?\n\nYou can check out the [example project](https://github.com/jlongster/absurd-example-project) to get started. Or follow the steps below:\n\nFirst you install the packages:\n\n```\nyarn add @jlongster/sql.js absurd-sql\n```\n\nRight now you need to use my fork of `sql.js`, but I'm going to open a PR and hopefully get it merged. The changes are minimal.\n\nabsurd-sql **must** run in a worker. This is fine because you really shouldn't be blocking the main thread anyway. So on the main thread, do this:\n\n```js\nimport { initBackend } from 'absurd-sql/dist/indexeddb-main-thread';\n\nfunction init() {\n  let worker = new Worker(new URL('./index.worker.js', import.meta.url));\n  // This is only required because Safari doesn't support nested\n  // workers. This installs a handler that will proxy creating web\n  // workers through the main thread\n  initBackend(worker);\n}\n\ninit();\n```\n\nThen in `index.worker.js` do this:\n\n```js\nimport initSqlJs from '@jlongster/sql.js';\nimport { SQLiteFS } from 'absurd-sql';\nimport IndexedDBBackend from 'absurd-sql/dist/indexeddb-backend';\n\nasync function run() {\n  let SQL = await initSqlJs({ locateFile: file =\u003e file });\n  let sqlFS = new SQLiteFS(SQL.FS, new IndexedDBBackend());\n  SQL.register_for_idb(sqlFS);\n\n  SQL.FS.mkdir('/sql');\n  SQL.FS.mount(sqlFS, {}, '/sql');\n\n  const path = '/sql/db.sqlite';\n  if (typeof SharedArrayBuffer === 'undefined') {\n    let stream = SQL.FS.open(path, 'a+');\n    await stream.node.contents.readIfFallback();\n    SQL.FS.close(stream);\n  }\n\n  let db = new SQL.Database(path, { filename: true });\n  // You might want to try `PRAGMA page_size=8192;` too!\n  db.exec(`\n    PRAGMA journal_mode=MEMORY;\n  `);\n\n   // Your code\n}\n```\n\n## Requirements\n\nBecause this uses `SharedArrayBuffer` and the `Atomics` API, there are some requirement for code to run.\n\n* It must be run in a worker thread (you shouldn't block the main thread with queries anyway)\n* Your server must respond with the following headers:\n\n```\nCross-Origin-Opener-Policy: same-origin\nCross-Origin-Embedder-Policy: require-corp\n```\n\nThose headers are required because browsers only enable `SharedArrayBuffer` if you tell it to isolate the process. There are potential security problems if `SharedArrayBuffer` was available everywhere.\n\n## Fallback mode\n\nWe do support browsers without `SharedArrayBuffer` (only Safari). Read more about it here: https://jlongster.com/future-sql-web#fallback-mode-without-sharedarraybuffer\n\nThere are some limitations in this mode: only one tab can be writing the database at a time. The database will never be corrupted; if multiple tabs try to write it will just throw an error (in the future it should call a handler that you provide so you can notify the user).\n\n## Performance\n\nIt consistently beats IndexedDB performance up to 10x:\n\nRead performance: doing something like `SELECT SUM(value) FROM kv`:\n\n\u003cimg width=\"610\" alt=\"perf-sum-chrome\" src=\"https://user-images.githubusercontent.com/17031/129102253-8adf163a-76b6-4af8-a1cf-8e2e39012ab0.png\"\u003e\n\nWrite performance: doing a bulk insert:\n\n\u003cimg width=\"609\" alt=\"perf-writes-chrome\" src=\"https://user-images.githubusercontent.com/17031/129102454-b4c362b3-1b0a-4625-ac96-72fc276497f3.png\"\u003e\n\nThese are all on a 2015 macbook pro. Benchmark code is in `src/examples/bench`.\n\n## How does it work?\n\nRead [this blog post](https://jlongster.com/future-sql-web) for more details.\n\n## Where you can help\n\nThere are several things that could be done:\n\n* Add a bunch more tests\n* Implement a `webkitFileSystem` backend\n  * I already started it [here](https://gist.github.com/jlongster/ec00ddbb47b4b29897ab5939b8e32fbe), but initial results showed that it was way slower?\n* Bug fixes\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjlongster%2Fabsurd-sql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjlongster%2Fabsurd-sql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjlongster%2Fabsurd-sql/lists"}