{"id":15050595,"url":"https://github.com/mustakimali/csvtosqlite","last_synced_at":"2026-03-11T05:02:13.974Z","repository":{"id":255878597,"uuid":"846677102","full_name":"mustakimali/csvtosqlite","owner":"mustakimali","description":"Import CSV files into SQLite database","archived":false,"fork":false,"pushed_at":"2024-09-07T15:21:50.000Z","size":20,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-31T04:41:34.972Z","etag":null,"topics":["csv","csv-import","csv-import-to-database","csv-importer","sqlite","sqlite3"],"latest_commit_sha":null,"homepage":"https://crates.io/crates/csvtosqlite","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mustakimali.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-23T18:01:49.000Z","updated_at":"2025-07-10T04:46:06.000Z","dependencies_parsed_at":"2024-09-07T16:57:12.378Z","dependency_job_id":"3d87f714-b8a4-4023-9058-6027223933ed","html_url":"https://github.com/mustakimali/csvtosqlite","commit_stats":null,"previous_names":["mustakimali/csvtosqlite"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mustakimali/csvtosqlite","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mustakimali%2Fcsvtosqlite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mustakimali%2Fcsvtosqlite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mustakimali%2Fcsvtosqlite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mustakimali%2Fcsvtosqlite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mustakimali","download_url":"https://codeload.github.com/mustakimali/csvtosqlite/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mustakimali%2Fcsvtosqlite/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30372126,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T21:41:54.280Z","status":"online","status_checked_at":"2026-03-11T02:00:07.027Z","response_time":84,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","csv-import","csv-import-to-database","csv-importer","sqlite","sqlite3"],"created_at":"2024-09-24T21:27:41.893Z","updated_at":"2026-03-11T05:02:13.950Z","avatar_url":"https://github.com/mustakimali.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CSV to SQLite Importer\n\nThis Rust project imports CSV data into an SQLite database. It reads a CSV file, creates a corresponding SQLite table, and inserts the data into the table. The table's schema is inferred from the CSV file's contents.\n\n[![csvtosqlite Demo](https://asciinema.org/a/674935.svg)](https://asciinema.org/a/674935)\n\n## Features\n\n- **CSV to SQLite:** Automatically create an SQLite table based on the CSV headers and data types.\n- **Customizable Database and Table Names:** Specify the database name and table name via command-line arguments.\n- **Dry Run Mode:** Preview the table creation and SQL insert statements without executing them.\n- **Batch Insert:** Efficiently inserts rows in batches to optimize performance.\n\n## Installation\n\n### Install from Crates.io\n\n```bash\ncargo install --locked csvtosqlite\n```\n\n### Build from sourcce\n\nMake sure you have Rust and Cargo installed on your machine. Then, build the project using:\n\n```bash\ncargo build --release\n```\n\n## Usage\n\nRun the program using the command line. The following options are available:\n\n```bash\ncsvtosqlite [OPTIONS]\n```\n\n### Options:\n\n- `-f, --file \u003cFILE\u003e`: The CSV file to import. Default is `sample.csv`.\n- `-t, --table \u003cTABLE\u003e`: The name of the SQLite table. If not specified, the table name defaults to the CSV file name (without extension).\n- `-d, --db \u003cDB\u003e`: The SQLite database file. Default is `data.db`.\n- `--dry-run`: When set, it prints the SQL commands without executing them. Default is `false`.\n- `--no-auto-detect-types`: When set, skip detecting types. All columns will be treated as strings. This is useful rows have mixed data types\n- `-b`, --batch-size \u003cBATCH_SIZE\u003e`: Batch size (Default 10000). When debugging: Reduce to 1 to identify the row that is causing the error. Query will be written in a file when insert fails.\n\n### Example Commands\n\n1. **Basic Import:**\n\n   Import a CSV file (`sample.csv`) into the default database (`data.db`) and create a table with the same name as the CSV file:\n\n   ```bash\n   csv-to-sqlite --file sample.csv\n   ```\n\n2. **Custom Table Name:**\n\n   Import a CSV file and create a custom table name:\n\n   ```bash\n   csv-to-sqlite --file data.csv --table my_table\n   ```\n\n3. **Dry Run:**\n\n   Preview the SQL commands without actually creating the table or inserting data:\n\n   ```bash\n   csvtosqlite --file data.csv --dry-run true\n   ```\n\n4. Disable auto infer data type and specify the data type for each column:\n  For CSV files with invalid data types across rows.\n\n  ```bash\n   csvtosqlite -f sample_file.csv --no-auto-detect-types\n   ```\n\n## How It Works\n\n1. **CSV Parsing:**\n   - The program reads the CSV file and extracts the headers.\n   - It determines the data type of each column based on the first row's values.\n\n2. **SQL Table Creation:**\n   - A SQL `CREATE TABLE` statement is generated based on the headers and their inferred types (e.g., `INTEGER`, `TEXT`, `REAL`, etc.).\n   - If `dry-run` is enabled, it only prints the SQL statements without executing them.\n\n3. **Data Insertion:**\n   - The program inserts the CSV rows into the SQLite table in batches of 10,000 rows for efficiency.\n   - It displays the number of rows inserted and the speed of insertion.\n\n## Performance\n\nThe program uses batching and transactions to optimize the performance of inserting data into SQLite. You can monitor the progress and speed of the insertion process, which is displayed in rows per second (wps).\n\n30s to import 6.4million rows from [Geographic units, by industry and statistical area: 2000–2023 descending order – CSV](https://www.stats.govt.nz/assets/Uploads/New-Zealand-business-demography-statistics/New-Zealand-business-demography-statistics-At-February-2023/Download-data/geographic-units-by-industry-and-statistical-area-20002023-descending-order-.zip) from [stats.govt.nz](https://www.stats.govt.nz/large-datasets/csv-files-for-download/) site.\n```bash\n❯ time ./csvtosqlite -f sample_nz_geo_units.csv\nOpening CSV file: sample_nz_geo_units.csv\nParsing headers...\nDetermining data type based on the first row...\n\u003e anzsic06 (anzsic06): String\n\u003e area (Area): String\n\u003e year (year): Integer\n\u003e geo_count (geo_count): Integer\n\u003e ec_count (ec_count): Integer\nCreating database: data.db\nCreating table:\nCREATE TABLE IF NOT EXISTS sample_nz_geo_units (\n  anzsic06 TEXT,\n  area TEXT,\n  year INTEGER,\n  geo_count INTEGER,\n  ec_count INTEGER\n);\nInserting rows...\nFinalising...Done! Inserted 6,457,053 rows (217,304,041 bytes) at (214150.80 rps)\n./csvtosqlite -f sample_nz_geo_units.csv   28.76s  user 0.81s system 98% cpu 30.166 total\navg shared (code):         0 KB\navg unshared (data/stack): 0 KB\ntotal (sum):               0 KB\nmax memory:                23\npage faults from disk:     0\nother page faults:         114444\n```\n\n## Dependencies\n\nThis project relies on the following Rust crates:\n\n- **clap:** For command-line argument parsing.\n- **csv:** For reading and parsing CSV files.\n- **sqlx:** For interacting with the SQLite database.\n- **tokio:** For asynchronous programming.\n\nYou can find the dependencies in the `Cargo.toml` file.\n\n## Contributing\n\nIf you would like to contribute to this project, please open a pull request or an issue with your suggestions or bug reports.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmustakimali%2Fcsvtosqlite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmustakimali%2Fcsvtosqlite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmustakimali%2Fcsvtosqlite/lists"}