{"id":13586018,"url":"https://github.com/simonw/sqlite-diffable","last_synced_at":"2025-08-21T00:32:07.311Z","repository":{"id":44163179,"uuid":"195145678","full_name":"simonw/sqlite-diffable","owner":"simonw","description":"Tools for dumping/loading a SQLite database to diffable directory structure","archived":false,"fork":false,"pushed_at":"2025-05-05T05:44:56.000Z","size":43,"stargazers_count":125,"open_issues_count":6,"forks_count":5,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-06T07:41:46.834Z","etag":null,"topics":["datasette-io","datasette-tool","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-04T00:58:46.000Z","updated_at":"2025-07-01T17:47:59.000Z","dependencies_parsed_at":"2022-09-18T04:40:35.657Z","dependency_job_id":null,"html_url":"https://github.com/simonw/sqlite-diffable","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/simonw/sqlite-diffable","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fsqlite-diffable","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fsqlite-diffable/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fsqlite-diffable/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fsqlite-diffable/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonw","download_url":"https://codeload.github.com/simonw/sqlite-diffable/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fsqlite-diffable/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271409462,"owners_count":24754716,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-20T02:00:09.606Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datasette-io","datasette-tool","sqlite"],"created_at":"2024-08-01T15:05:16.607Z","updated_at":"2025-08-21T00:32:07.306Z","avatar_url":"https://github.com/simonw.png","language":"Python","readme":"# sqlite-diffable\n\n[![PyPI](https://img.shields.io/pypi/v/sqlite-diffable.svg)](https://pypi.org/project/sqlite-diffable/)\n[![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-diffable?include_prereleases\u0026label=changelog)](https://github.com/simonw/sqlite-diffable/releases)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-diffable/blob/main/LICENSE)\n\nTools for dumping/loading a SQLite database to diffable directory structure\n\n## Installation\n\n    pip install sqlite-diffable\n\n## Demo\n\nThe repository at [simonw/simonwillisonblog-backup](https://github.com/simonw/simonwillisonblog-backup) contains a backup of the database on my blog, https://simonwillison.net/ - created using this tool.\n\n## Dumping a database\n\nGiven a SQLite database called `fixtures.db` containing a table `facetable`, the following will dump out that table to the `dump/` directory:\n```bash\nsqlite-diffable dump fixtures.db dump/ facetable\n```\nTo dump out every table in that database, use `--all`:\n```bash\nsqlite-diffable dump fixtures.db dump/ --all\n```\nTo dump all table except some specific ones, use `--exclude` one or more times:\n```bash\nsqlite-diffable dump fixtures.db dump/ --all \\\n  --exclude unwanted_first_table \\\n  --exclude unwanted_second_table\n```\n## Loading a database\n\nTo load a previously dumped database, run the following:\n```bash\nsqlite-diffable load restored.db dump/\n```\nThis will show an error if any of the tables that are being restored already exist in the database file.\n\nYou can replace those tables (dropping them before restoring them) using the `--replace` option:\n```bash\nsqlite-diffable load restored.db dump/ --replace\n```\n## Converting to JSON objects\n\nTable rows are stored in the `.ndjson` files as newline-delimited JSON arrays, like this:\n\n```\n[\"a\", \"a\", \"a-a\", 63, null, 0.7364712141640124, \"$null\"]\n[\"a\", \"b\", \"a-b\", 51, null, 0.6020187290499803, \"$null\"]\n```\n\nSometimes it can be more convenient to work with a list of JSON objects.\n\nThe `sqlite-diffable objects` command can read a `.ndjson` file and its accompanying `.metadata.json` file and output JSON objects to standard output:\n```bash\nsqlite-diffable objects fixtures.db dump/sortable.ndjson\n```\nThe output of that command looks something like this:\n```\n{\"pk1\": \"a\", \"pk2\": \"a\", \"content\": \"a-a\", \"sortable\": 63, \"sortable_with_nulls\": null, \"sortable_with_nulls_2\": 0.7364712141640124, \"text\": \"$null\"}\n{\"pk1\": \"a\", \"pk2\": \"b\", \"content\": \"a-b\", \"sortable\": 51, \"sortable_with_nulls\": null, \"sortable_with_nulls_2\": 0.6020187290499803, \"text\": \"$null\"}\n```\n\nAdd `-o` to write that output to a file:\n```bash\nsqlite-diffable objects fixtures.db dump/sortable.ndjson -o output.txt\n```\nAdd `--array` to output a JSON array of objects, as opposed to a newline-delimited file:\n```bash\nsqlite-diffable objects fixtures.db dump/sortable.ndjson --array\n```\nOutput:\n```\n[\n{\"pk1\": \"a\", \"pk2\": \"a\", \"content\": \"a-a\", \"sortable\": 63, \"sortable_with_nulls\": null, \"sortable_with_nulls_2\": 0.7364712141640124, \"text\": \"$null\"},\n{\"pk1\": \"a\", \"pk2\": \"b\", \"content\": \"a-b\", \"sortable\": 51, \"sortable_with_nulls\": null, \"sortable_with_nulls_2\": 0.6020187290499803, \"text\": \"$null\"}\n]\n```\n\n## Storage format\n\nEach table is represented as two files. The first, `table_name.metadata.json`, contains metadata describing the structure of the table. For a table called `redirects_redirect` that file might look like this:\n\n```json\n{\n    \"name\": \"redirects_redirect\",\n    \"columns\": [\n        \"id\",\n        \"domain\",\n        \"path\",\n        \"target\",\n        \"created\"\n    ],\n    \"schema\": \"CREATE TABLE [redirects_redirect] (\\n   [id] INTEGER PRIMARY KEY,\\n   [domain] TEXT,\\n   [path] TEXT,\\n   [target] TEXT,\\n   [created] TEXT\\n)\"\n}\n```\n\nIt is an object with three keys: `name` is the name of the table, `columns` is an array of column strings and `schema` is the SQL schema text used for tha table.\n\nThe second file, `table_name.ndjson`, contains newline-delimited JSON (aka [JSON Lines](https://jsonlines.org/)) for every row in the table. Each row is represented as a JSON array with items corresponding to each of the columns defined in the metadata.\n\nThat file for the `redirects_redirect.ndjson` table might look like this:\n\n```\n[1, \"feeds.simonwillison.net\", \"swn-everything\", \"https://simonwillison.net/atom/everything/\", \"2017-10-01T21:11:36.440537+00:00\"]\n[2, \"feeds.simonwillison.net\", \"swn-entries\", \"https://simonwillison.net/atom/entries/\", \"2017-10-01T21:12:32.478849+00:00\"]\n[3, \"feeds.simonwillison.net\", \"swn-links\", \"https://simonwillison.net/atom/links/\", \"2017-10-01T21:12:54.820729+00:00\"]\n```\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fsqlite-diffable","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonw%2Fsqlite-diffable","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fsqlite-diffable/lists"}