{"id":13586374,"url":"https://github.com/simonw/db-to-sqlite","last_synced_at":"2025-04-04T07:09:49.797Z","repository":{"id":44434190,"uuid":"166159072","full_name":"simonw/db-to-sqlite","owner":"simonw","description":"CLI tool for exporting tables or queries from any SQL database to a SQLite file","archived":false,"fork":false,"pushed_at":"2023-02-28T07:16:21.000Z","size":72,"stargazers_count":370,"open_issues_count":21,"forks_count":29,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-10-30T00:33:28.226Z","etag":null,"topics":["datasette","datasette-io","datasette-tool","sqlalchemy","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-01-17T04:16:48.000Z","updated_at":"2024-10-23T09:14:49.000Z","dependencies_parsed_at":"2024-01-15T15:47:28.510Z","dependency_job_id":"c1baa736-58ab-4833-b31f-6cd84dfbf7d9","html_url":"https://github.com/simonw/db-to-sqlite","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fdb-to-sqlite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fdb-to-sqlite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fdb-to-sqlite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fdb-to-sqlite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonw","download_url":"https://codeload.github.com/simonw/db-to-sqlite/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247135147,"owners_count":20889421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datasette","datasette-io","datasette-tool","sqlalchemy","sqlite"],"created_at":"2024-08-01T15:05:31.372Z","updated_at":"2025-04-04T07:09:49.770Z","avatar_url":"https://github.com/simonw.png","language":"Python","readme":"# db-to-sqlite\n\n[![PyPI](https://img.shields.io/pypi/v/db-to-sqlite.svg)](https://pypi.python.org/pypi/db-to-sqlite)\n[![Changelog](https://img.shields.io/github/v/release/simonw/db-to-sqlite?include_prereleases\u0026label=changelog)](https://github.com/simonw/db-to-sqlite/releases)\n[![Tests](https://github.com/simonw/db-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/db-to-sqlite/actions?query=workflow%3ATest)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/db-to-sqlite/blob/main/LICENSE)\n\nCLI tool for exporting tables or queries from any SQL database to a SQLite file.\n\n## Installation\n\nInstall from PyPI like so:\n\n    pip install db-to-sqlite\n\nIf you want to use it with MySQL, you can install the extra dependency like this:\n\n    pip install 'db-to-sqlite[mysql]'\n\nInstalling the `mysqlclient` library on OS X can be tricky - I've found [this recipe](https://gist.github.com/simonw/90ac0afd204cd0d6d9c3135c3888d116) to work (run that before installing `db-to-sqlite`).\n\nFor PostgreSQL, use this:\n\n    pip install 'db-to-sqlite[postgresql]'\n\n## Usage\n```\nUsage: db-to-sqlite [OPTIONS] CONNECTION PATH\n\n  Load data from any database into SQLite.\n\n  PATH is a path to the SQLite file to create, e.c. /tmp/my_database.db\n\n  CONNECTION is a SQLAlchemy connection string, for example:\n\n      postgresql://localhost/my_database\n      postgresql://username:passwd@localhost/my_database\n\n      mysql://root@localhost/my_database\n      mysql://username:passwd@localhost/my_database\n\n  More: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls\n\nOptions:\n  --version                     Show the version and exit.\n  --all                         Detect and copy all tables\n  --table TEXT                  Specific tables to copy\n  --skip TEXT                   When using --all skip these tables\n  --redact TEXT...              (table, column) pairs to redact with ***\n  --sql TEXT                    Optional SQL query to run\n  --output TEXT                 Table in which to save --sql query results\n  --pk TEXT                     Optional column to use as a primary key\n  --index-fks / --no-index-fks  Should foreign keys have indexes? Default on\n  -p, --progress                Show progress bar\n  --postgres-schema TEXT        PostgreSQL schema to use\n  --help                        Show this message and exit.\n```\n\nFor example, to save the content of the `blog_entry` table from a PostgreSQL database to a local file called `blog.db` you could do this:\n\n    db-to-sqlite \"postgresql://localhost/myblog\" blog.db \\\n        --table=blog_entry\n\nYou can specify `--table` more than once.\n\nYou can also save the data from all of your tables, effectively creating a SQLite copy of your entire database. Any foreign key relationships will be detected and added to the SQLite database. For example:\n\n    db-to-sqlite \"postgresql://localhost/myblog\" blog.db \\\n        --all\n\nWhen running `--all` you can specify tables to skip using `--skip`:\n\n    db-to-sqlite \"postgresql://localhost/myblog\" blog.db \\\n        --all \\\n        --skip=django_migrations\n\nIf you want to save the results of a custom SQL query, do this:\n\n    db-to-sqlite \"postgresql://localhost/myblog\" output.db \\\n        --output=query_results \\\n        --sql=\"select id, title, created from blog_entry\" \\\n        --pk=id\n\nThe `--output` option specifies the table that should contain the results of the query.\n\n## Using db-to-sqlite with PostgreSQL schemas\n\nIf the tables you want to copy from your PostgreSQL database aren't in the default schema, you can specify an alternate one with the `--postgres-schema` option:\n\n    db-to-sqlite \"postgresql://localhost/myblog\" blog.db \\\n        --all \\\n        --postgres-schema my_schema\n\n## Using db-to-sqlite with MS SQL\n\nThe best way to get the connection string needed for the MS SQL connections below is to use urllib from the Standard Library as below\n\n    params = urllib.parse.quote_plus(\n        \"DRIVER={SQL Server Native Client 11.0};\"\n        \"SERVER=localhost;\"\n        \"DATABASE=my_database;\"\n        \"Trusted_Connection=yes;\"\n    )\n\nThe above will resolve to\n\n    DRIVER%3D%7BSQL+Server+Native+Client+11.0%7D%3B+SERVER%3Dlocalhost%3B+DATABASE%3Dmy_database%3B+Trusted_Connection%3Dyes\n\nYou can then use the string above in the odbc_connect below\n\n    mssql+pyodbc:///?odbc_connect=DRIVER%3D%7BSQL+Server+Native+Client+11.0%7D%3B+SERVER%3Dlocalhost%3B+DATABASE%3Dmy_database%3B+Trusted_Connection%3Dyes\n    mssql+pyodbc:///?odbc_connect=DRIVER%3D%7BSQL+Server+Native+Client+11.0%7D%3B+SERVER%3Dlocalhost%3B+DATABASE%3Dmy_database%3B+UID%3Dusername%3B+PWD%3Dpasswd\n\n## Using db-to-sqlite with Heroku Postgres\n\nIf you run an application on [Heroku](https://www.heroku.com/) using their [Postgres database product](https://www.heroku.com/postgres), you can use the `heroku config` command to access a compatible connection string:\n\n    $ heroku config --app myappname | grep HEROKU_POSTG\n    HEROKU_POSTGRESQL_OLIVE_URL: postgres://username:password@ec2-xxx-xxx-xxx-x.compute-1.amazonaws.com:5432/dbname\n\nYou can pass this to `db-to-sqlite` to create a local SQLite database with the data from your Heroku instance.\n\nYou can even do this using a bash one-liner:\n\n    $ db-to-sqlite $(heroku config --app myappname | grep HEROKU_POSTG | cut -d: -f 2-) \\\n        /tmp/heroku.db --all -p\n    1/23: django_migrations\n    ...\n    17/23: blog_blogmark\n    [####################################]  100%\n    ...\n\n## Related projects\n\n* [Datasette](https://github.com/simonw/datasette): A tool for exploring and publishing data. Works great with SQLite files generated using `db-to-sqlite`.\n* [sqlite-utils](https://github.com/simonw/sqlite-utils): Python CLI utility and library for manipulating SQLite databases.\n* [csvs-to-sqlite](https://github.com/simonw/csvs-to-sqlite): Convert CSV files into a SQLite database.\n\n## Development\n\nTo set up this tool locally, first checkout the code. Then create a new virtual environment:\n\n    cd db-to-sqlite\n    python3 -m venv venv\n    source venv/bin/activate\n\nOr if you are using `pipenv`:\n\n    pipenv shell\n\nNow install the dependencies and test dependencies:\n\n    pip install -e '.[test]'\n\nTo run the tests:\n\n    pytest\n\nThis will skip tests against MySQL or PostgreSQL if you do not have their additional dependencies installed.\n\nYou can install those extra dependencies like so:\n\n    pip install -e '.[test_mysql,test_postgresql]'\n\nYou can alternative use `pip install psycopg2-binary` if you cannot install the `psycopg2` dependency used by the `test_postgresql` extra.\n\nSee [Running a MySQL server using Homebrew](https://til.simonwillison.net/homebrew/mysql-homebrew) for tips on running the tests against MySQL on macOS, including how to install the `mysqlclient` dependency.\n\nThe PostgreSQL and MySQL tests default to expecting to run against servers on localhost. You can use environment variables to point them at different test database servers:\n\n- `MYSQL_TEST_DB_CONNECTION` - defaults to `mysql://root@localhost/test_db_to_sqlite`\n- `POSTGRESQL_TEST_DB_CONNECTION` - defaults to `postgresql://localhost/test_db_to_sqlite`\n\nThe database you indicate in the environment variable - `test_db_to_sqlite` by default - will be deleted and recreated on every test run.\n","funding_links":[],"categories":["Python","Tools"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fdb-to-sqlite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonw%2Fdb-to-sqlite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fdb-to-sqlite/lists"}