{"id":13468835,"url":"https://github.com/simonw/csvs-to-sqlite","last_synced_at":"2025-05-15T16:06:15.187Z","repository":{"id":26809563,"uuid":"110509816","full_name":"simonw/csvs-to-sqlite","owner":"simonw","description":"Convert CSV files into a SQLite database","archived":false,"fork":false,"pushed_at":"2025-04-10T17:00:54.000Z","size":139,"stargazers_count":902,"open_issues_count":42,"forks_count":71,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-05-12T05:49:42.810Z","etag":null,"topics":["click","csv","datasette","datasette-io","datasette-tool","pandas","python","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonw.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-11-13T06:38:21.000Z","updated_at":"2025-05-01T19:28:38.000Z","dependencies_parsed_at":"2024-01-13T09:36:43.851Z","dependency_job_id":"436a337f-91dc-4978-8695-ebed42683b8e","html_url":"https://github.com/simonw/csvs-to-sqlite","commit_stats":{"total_commits":85,"total_committers":7,"mean_commits":"12.142857142857142","dds":0.3529411764705882,"last_synced_commit":"1a7a9dfea69b5ea7d5c0e664ef85f68d77b53afe"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fcsvs-to-sqlite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fcsvs-to-sqlite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fcsvs-to-sqlite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fcsvs-to-sqlite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonw","download_url":"https://codeload.github.com/simonw/csvs-to-sqlite/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254374470,"owners_count":22060611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["click","csv","datasette","datasette-io","datasette-tool","pandas","python","sqlite"],"created_at":"2024-07-31T15:01:20.046Z","updated_at":"2025-05-15T16:06:15.181Z","avatar_url":"https://github.com/simonw.png","language":"Python","readme":"# csvs-to-sqlite\n\n[![PyPI](https://img.shields.io/pypi/v/csvs-to-sqlite.svg)](https://pypi.org/project/csvs-to-sqlite/)\n[![Changelog](https://img.shields.io/github/v/release/simonw/csvs-to-sqlite?include_prereleases\u0026label=changelog)](https://github.com/simonw/csvs-to-sqlite/releases)\n[![Tests](https://github.com/simonw/csvs-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/csvs-to-sqlite/actions?query=workflow%3ATest)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/csvs-to-sqlite/blob/main/LICENSE)\n\nConvert CSV files into a SQLite database. Browse and publish that SQLite database with [Datasette](https://github.com/simonw/datasette).\n\n\u003e [!NOTE]\n\u003e This tool is **infrequently maintained**. I suggest [using sqlite-utils](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-csv-or-tsv-data) for importing CSV and TSV to SQLite instead for most cases.\n\nBasic usage:\n```bash\ncsvs-to-sqlite myfile.csv mydatabase.db\n```\nThis will create a new SQLite database called `mydatabase.db` containing a\nsingle table, `myfile`, containing the CSV content.\n\nYou can provide multiple CSV files:\n```\ncsvs-to-sqlite one.csv two.csv bundle.db\n```\nThe `bundle.db` database will contain two tables, `one` and `two`.\n\nThis means you can use wildcards:\n```bash\ncsvs-to-sqlite ~/Downloads/*.csv my-downloads.db\n```\nIf you pass a path to one or more directories, the script will recursively\nsearch those directories for CSV files and create tables for each one.\n```bash\ncsvs-to-sqlite ~/path/to/directory all-my-csvs.db\n```\n## Handling TSV (tab-separated values)\n\nYou can use the `-s` option to specify a different delimiter. If you want\nto use a tab character you'll need to apply shell escaping like so:\n```bash\ncsvs-to-sqlite my-file.tsv my-file.db -s $'\\t'\n```\n## Refactoring columns into separate lookup tables\n\nLet's say you have a CSV file that looks like this:\n```csv\ncounty,precinct,office,district,party,candidate,votes\nClark,1,President,,REP,John R. Kasich,5\nClark,2,President,,REP,John R. Kasich,0\nClark,3,President,,REP,John R. Kasich,7\n```\n([Real example taken from the Open Elections project](https://github.com/openelections/openelections-data-sd/blob/master/2016/20160607__sd__primary__clark__precinct.csv))\n\nYou can now convert selected columns into separate lookup tables using the new\n`--extract-column` option (shortname: `-c`) - for example:\n```bash\ncsvs-to-sqlite openelections-data-*/*.csv \\\n    -c county:County:name \\\n    -c precinct:Precinct:name \\\n    -c office -c district -c party -c candidate \\\n    openelections.db\n```\nThe format is as follows:\n```bash\ncolumn_name:optional_table_name:optional_table_value_column_name\n```\nIf you just specify the column name e.g. `-c office`, the following table will\nbe created:\n```sql\nCREATE TABLE \"office\" (\n    \"id\" INTEGER PRIMARY KEY,\n    \"value\" TEXT\n);\n```\nIf you specify all three options, e.g. `-c precinct:Precinct:name` the table\nwill look like this:\n```sql\nCREATE TABLE \"Precinct\" (\n    \"id\" INTEGER PRIMARY KEY,\n    \"name\" TEXT\n);\n```\nThe original tables will be created like this:\n```sql\nCREATE TABLE \"ca__primary__san_francisco__precinct\" (\n    \"county\" INTEGER,\n    \"precinct\" INTEGER,\n    \"office\" INTEGER,\n    \"district\" INTEGER,\n    \"party\" INTEGER,\n    \"candidate\" INTEGER,\n    \"votes\" INTEGER,\n    FOREIGN KEY (county) REFERENCES County(id),\n    FOREIGN KEY (party) REFERENCES party(id),\n    FOREIGN KEY (precinct) REFERENCES Precinct(id),\n    FOREIGN KEY (office) REFERENCES office(id),\n    FOREIGN KEY (candidate) REFERENCES candidate(id)\n);\n```\nThey will be populated with IDs that reference the new derived tables.\n\n## Installation\n\n```bash\npip install csvs-to-sqlite\n```\n\n`csvs-to-sqlite` now requires Python 3. If you are running Python 2 you can install the last version to support Python 2:\n```bash\npip install csvs-to-sqlite==0.9.2\n```\n\n## csvs-to-sqlite --help\n\n\u003c!-- [[[cog\nimport cog\nfrom csvs_to_sqlite import cli\nfrom click.testing import CliRunner\nrunner = CliRunner()\nresult = runner.invoke(cli.cli, [\"--help\"])\nhelp = result.output.replace(\"Usage: cli\", \"Usage: csvs-to-sqlite\")\ncog.out(\n    \"```\\n{}\\n```\".format(help)\n)\n]]] --\u003e\n```\nUsage: csvs-to-sqlite [OPTIONS] PATHS... DBNAME\n\n  PATHS: paths to individual .csv files or to directories containing .csvs\n\n  DBNAME: name of the SQLite database file to create\n\nOptions:\n  -s, --separator TEXT            Field separator in input .csv\n  -q, --quoting INTEGER           Control field quoting behavior per csv.QUOTE_*\n                                  constants. Use one of QUOTE_MINIMAL (0),\n                                  QUOTE_ALL (1), QUOTE_NONNUMERIC (2) or\n                                  QUOTE_NONE (3).\n  --skip-errors                   Skip lines with too many fields instead of\n                                  stopping the import\n  --replace-tables                Replace tables if they already exist\n  -t, --table TEXT                Table to use (instead of using CSV filename)\n  -c, --extract-column TEXT       One or more columns to 'extract' into a\n                                  separate lookup table. If you pass a simple\n                                  column name that column will be replaced with\n                                  integer foreign key references to a new table\n                                  of that name. You can customize the name of\n                                  the table like so:     state:States:state_name\n                                  \n                                  This will pull unique values from the 'state'\n                                  column and use them to populate a new 'States'\n                                  table, with an id column primary key and a\n                                  state_name column containing the strings from\n                                  the original column.\n  -d, --date TEXT                 One or more columns to parse into ISO\n                                  formatted dates\n  -dt, --datetime TEXT            One or more columns to parse into ISO\n                                  formatted datetimes\n  -df, --datetime-format TEXT     One or more custom date format strings to try\n                                  when parsing dates/datetimes\n  -pk, --primary-key TEXT         One or more columns to use as the primary key\n  -f, --fts TEXT                  One or more columns to use to populate a full-\n                                  text index\n  -i, --index TEXT                Add index on this column (or a compound index\n                                  with -i col1,col2)\n  --shape TEXT                    Custom shape for the DB table - format is\n                                  csvcol:dbcol(TYPE),...\n  --filename-column TEXT          Add a column with this name and populate with\n                                  CSV file name\n  --fixed-column \u003cTEXT TEXT\u003e...   Populate column with a fixed string\n  --fixed-column-int \u003cTEXT INTEGER\u003e...\n                                  Populate column with a fixed integer\n  --fixed-column-float \u003cTEXT FLOAT\u003e...\n                                  Populate column with a fixed float\n  --no-index-fks                  Skip adding index to foreign key columns\n                                  created using --extract-column (default is to\n                                  add them)\n  --no-fulltext-fks               Skip adding full-text index on values\n                                  extracted using --extract-column (default is\n                                  to add them)\n  --just-strings                  Import all columns as text strings by default\n                                  (and, if specified, still obey --shape,\n                                  --date/datetime, and --datetime-format)\n  --version                       Show the version and exit.\n  --help                          Show this message and exit.\n\n```\n\u003c!-- [[[end]]] --\u003e\n","funding_links":[],"categories":["Python","Text-to-SQLite","Data Loading \u0026 Extraction"],"sub_categories":["Desktop"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fcsvs-to-sqlite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonw%2Fcsvs-to-sqlite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fcsvs-to-sqlite/lists"}