{"id":13998141,"url":"https://github.com/asg017/sqlite-xsv","last_synced_at":"2025-04-12T17:44:00.744Z","repository":{"id":64402843,"uuid":"561892029","full_name":"asg017/sqlite-xsv","owner":"asg017","description":"the fastest CSV SQLite extension, written in Rust","archived":false,"fork":false,"pushed_at":"2025-02-06T03:04:08.000Z","size":395,"stargazers_count":132,"open_issues_count":8,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-03T21:11:18.598Z","etag":null,"topics":["sqlite","sqlite-extension"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/asg017.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-04T18:23:21.000Z","updated_at":"2025-03-18T23:35:19.000Z","dependencies_parsed_at":"2024-01-15T19:44:10.010Z","dependency_job_id":"ba84f517-0c2a-460a-8787-793e2011267c","html_url":"https://github.com/asg017/sqlite-xsv","commit_stats":{"total_commits":44,"total_committers":1,"mean_commits":44.0,"dds":0.0,"last_synced_commit":"f8c43f1c32773a0378659d6ae393dcb513c53e8a"},"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asg017%2Fsqlite-xsv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asg017%2Fsqlite-xsv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asg017%2Fsqlite-xsv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asg017%2Fsqlite-xsv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/asg017","download_url":"https://codeload.github.com/asg017/sqlite-xsv/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248609081,"owners_count":21132843,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["sqlite","sqlite-extension"],"created_at":"2024-08-09T19:01:25.467Z","updated_at":"2025-04-12T17:44:00.721Z","avatar_url":"https://github.com/asg017.png","language":"Rust","readme":"# sqlite-xsv\n\nA fast and performant SQLite extension for CSV files, written in Rust! Based on [`sqlite-loadable-rs`](https://github.com/asg017/sqlite-loadable-rs) and the wonderful [csv crate](https://github.com/BurntSushi/rust-csv).\n\n- Query CSVs, TSVs, and other-SVs as SQLite virtual tables\n- The \"reader\" interface lets you query CSVs from other data sources, such as [`sqlite-http`](https://github.com/asg017/sqlite-http)\n- Builtin support for querying CSVs with gzip or zstd compression\n\nSee [_Introducing sqlite-xsv: The Fastest CSV Parser for SQLite_](https://observablehq.com/@asg017/introducing-sqlite-xsv) (Jan 2023) for more details!\n\n\u003e **Note**\n\u003e Nothing to do with [xsv](https://github.com/BurntSushi/xsv), but is based on the same csv crate. This is named `sqlite-xsv` to distinguish between the official [SQLite CSV Virtual table](https://www.sqlite.org/csv.html) and the [`sqlean` vsv extension](https://github.com/nalgeon/sqlean/blob/main/docs/vsv.md).\n\n![](./benchmarks/count.png)\n\n## Usage\n\n```sql\n.load ./xsv0\n\ncreate virtual table temp.students using csv(\n  filename=\"students.csv\"\n);\n\nselect * from temp.students;\n/*\n┌────┬───────┬─────┬─────────┐\n│ id │ name  │ age │ process │\n├────┼───────┼─────┼─────────┤\n│ 1  │ alex  │ 10  │ .9      │\n│ 2  │ brian │ 20  │ .7      │\n│ 3  │ craig │ 30  │ .3      │\n└────┴───────┴─────┴─────────┘\n*/\n```\n\nProvide a schema for CSVs that lack headers, or to provide types on columns.\n\n```sql\ncreate virtual table temp.students_no_header using csv(\n  filename=\"students_no_header.csv\",\n  header=false,\n  id text,\n  name text,\n  age int,\n);\n\nselect * from temp.students_no_header;\n\n```\n\nQuery files that are gzip'ed or compressed with `zstd` directly.\n\n```sql\ncreate virtual table temp.students_gz using csv(\n  filename=\"students.csv.gz\"\n);\n\nselect * from temp.students_gz;\n\n\ncreate virtual table temp.students_zst using csv(\n  filename=\"students.csv.zst\"\n);\n\nselect * from temp.students_zst;\n\n\n```\n\nUse the [`csv_reader`](/docs.md#xsv_reader) API and the `fsdir()` function in the SQLite CLI to read from several CSV files in one query.\n\n```sql\ncreate virtual table temp.students_reader using csv_reader(\n  id integer,\n  name text,\n  age integer,\n  progess real\n);\n\nwith files as (\n   select name as path\n   from fsdir('tests/data/student_files')\n\n)\nselect\n  files.path,\n  students.*\nfrom files\njoin students_reader(files.path) as students\nwhere files.path like '%.csv';\n/*\n┌────────────────────────────────┬────┬───────────┬─────┬─────────┐\n│              path              │ id │   name    │ age │ progess │\n├────────────────────────────────┼────┼───────────┼─────┼─────────┤\n│ tests/data/student_files/a.csv │ 1  │ alex      │ 10  │ 0.9     │\n│ tests/data/student_files/a.csv │ 2  │ adrian    │ 20  │ 0.8     │\n│ tests/data/student_files/a.csv │ 3  │ andres    │ 30  │ 0.7     │\n│ tests/data/student_files/c.csv │ 1  │ craig     │ 70  │ 0.4     │\n│ tests/data/student_files/c.csv │ 2  │ catherine │ 90  │ 0.5     │\n│ tests/data/student_files/c.csv │ 3  │ coin      │ 80  │ 0.6     │\n│ tests/data/student_files/b.csv │ 1  │ brian     │ 60  │ 0.1     │\n│ tests/data/student_files/b.csv │ 2  │ beto      │ 50  │ 0.2     │\n│ tests/data/student_files/b.csv │ 3  │ brandy    │ 40  │ 0.3     │\n└────────────────────────────────┴────┴───────────┴─────┴─────────┘\n*/\n```\n\nQuery CSVs from HTTP endpoints, with the reader API and [`sqlite-http`](https://github.com/asg017/sqlite-http). Note: Only works for CSVs that work in memory, for now.\n\n```sql\n.load ./http0\n-- Reading a CSV from the wonderful LA Times COVID project\n-- https://github.com/datadesk/california-coronavirus-data\n\ncreate virtual table temp.cdph_age_reader using csv_reader(\n  date,\n  age text,\n  confirmed_cases_total int,\n  confirmed_cases_percent float,\n  deaths_total int,\n  deaths_percent float\n);\n\ncreate table cdph_age as\n  select *\n  from temp.cdph_age_reader(\n    http_get_body(\n      'https://raw.githubusercontent.com/datadesk/california-coronavirus-data/master/cdph-age.csv'\n    )\n  );\n\nselect *\nfrom cdph_age\nlimit 5;\n\n/*\n┌────────────┬───────┬───────────────────────┬─────────────────────────┬──────────────┬────────────────┐\n│    date    │  age  │ confirmed_cases_total │ confirmed_cases_percent │ deaths_total │ deaths_percent │\n├────────────┼───────┼───────────────────────┼─────────────────────────┼──────────────┼────────────────┤\n│ 2023-01-03 │ 0-4   │ 371691                │ 0.034                   │ 32           │ 0.0            │\n│ 2023-01-03 │ 80+   │ 292252                │ 0.027                   │ 37038        │ 0.378          │\n│ 2023-01-03 │ 18–34 │ 3416056               │ 0.312                   │ 1655         │ 0.017          │\n│ 2023-01-03 │ 35–49 │ 2530259               │ 0.231                   │ 6135         │ 0.063          │\n│ 2023-01-03 │ 50–59 │ 1379087               │ 0.126                   │ 10892        │ 0.111          │\n└────────────┴───────┴───────────────────────┴─────────────────────────┴──────────────┴────────────────┘\n*/\n```\n\n## Documentation\n\nSee [`docs.md`](./docs.md) for a full API reference.\n\n## Installing\n\n| Language       | Install                                                    |                                                                                                                                                                                    |\n| -------------- | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Python         | `pip install sqlite-xsv`                                   | [![PyPI](https://img.shields.io/pypi/v/sqlite-xsv.svg?color=blue\u0026logo=python\u0026logoColor=white)](https://pypi.org/project/sqlite-xsv/)                                               |\n| Node.js        | `npm install sqlite-xsv`                                   | [![npm](https://img.shields.io/npm/v/sqlite-xsv.svg?color=green\u0026logo=nodedotjs\u0026logoColor=white)](https://www.npmjs.com/package/sqlite-xsv)                                         |\n| Deno           | [`deno.land/x/sqlite_xsv`](https://deno.land/x/sqlite_xsv) | [![deno.land/x release](https://img.shields.io/github/v/release/asg017/sqlite-xsv?color=fef8d2\u0026include_prereleases\u0026label=deno.land%2Fx\u0026logo=deno)](https://deno.land/x/sqlite_xsv) |\n| Ruby           | `gem install sqlite-xsv`                                   | ![Gem](https://img.shields.io/gem/v/sqlite-xsv?color=red\u0026logo=rubygems\u0026logoColor=white)                                                                                            |\n| Rust           | `cargo add sqlite-xsv`                                     | [![Crates.io](https://img.shields.io/crates/v/sqlite-xsv?logo=rust)](https://crates.io/crates/sqlite-xsv)                                                                          |\n| Github Release |                                                            | ![GitHub tag (latest SemVer pre-release)](https://img.shields.io/github/v/tag/asg017/sqlite-xsv?color=lightgrey\u0026include_prereleases\u0026label=Github+release\u0026logo=github)              |\n\n\u003c!--\n| Datasette      | `datasette install datasette-sqlite-xsv`                   | [![Datasette](https://img.shields.io/pypi/v/datasette-sqlite-xsv.svg?color=B6B6D9\u0026label=Datasette+plugin\u0026logoColor=white\u0026logo=python)](https://datasette.io/plugins/datasette-sqlite-xsv) |\n| Elixir         | [`hex.pm/packages/sqlite_xsv`](https://hex.pm/packages/sqlite_xsv) | [![Hex.pm](https://img.shields.io/hexpm/v/sqlite_xsv?color=purple\u0026logo=elixir)](https://hex.pm/packages/sqlite_xsv)                                                                       |\n| Go             | `go get -u github.com/asg017/sqlite-xsv/bindings/go`               | [![Go Reference](https://pkg.go.dev/badge/github.com/asg017/sqlite-xsv/bindings/go.svg)](https://pkg.go.dev/github.com/asg017/sqlite-xsv/bindings/go)                                     |\n--\u003e\n\nThe [Releases page](https://github.com/asg017/sqlite-xsv/releases) contains pre-built binaries for Linux amd64, MacOS amd64 (no arm yet), and Windows.\n\n### As a loadable extension\n\nIf you want to use `sqlite-xsv` as a [Runtime-loadable extension](https://www.sqlite.org/loadext.html), Download the `xsv0.dylib` (for MacOS), `xsv0.so` (Linux), or `xsv0.dll` (Windows) file from a release and load it into your SQLite environment.\n\n\u003e **Note:**\n\u003e The `0` in the filename (`xsv0.dylib`/ `xsv0.so`/`xsv0.dll`) denotes the major version of `sqlite-xsv`. Currently `sqlite-xsv` is pre v1, so expect breaking changes in future versions.\n\nFor example, if you are using the [SQLite CLI](https://www.sqlite.org/cli.html), you can load the library like so:\n\n```sql\n.load ./xsv0\nselect xsv_version();\n-- v0.0.1\n```\n\nOr in Python, using the builtin [sqlite3 module](https://docs.python.org/3/library/sqlite3.html):\n\n```python\nimport sqlite3\n\ncon = sqlite3.connect(\":memory:\")\n\ncon.enable_load_extension(True)\ncon.load_extension(\"./xsv0\")\n\nprint(con.execute(\"select xsv_version()\").fetchone())\n# ('v0.0.1',)\n```\n\nOr in Node.js using [better-sqlite3](https://github.com/WiseLibs/better-sqlite3):\n\n```javascript\nconst Database = require(\"better-sqlite3\");\nconst db = new Database(\":memory:\");\n\ndb.loadExtension(\"./xsv0\");\n\nconsole.log(db.prepare(\"select xsv_version()\").get());\n// { 'xsv_version()': 'v0.0.1' }\n```\n\nFor [Datasette](https://datasette.io/), it is currently NOT recommended to load `sqlite-xsv` in public Datasette instances. This is because the SQL API reads files from the filesystem, which is dangerous on Datasette instances. This may be changed in future version of `sqlite-xsv.\n","funding_links":[],"categories":["Rust","People","sqlite"],"sub_categories":["As Main Database"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasg017%2Fsqlite-xsv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fasg017%2Fsqlite-xsv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasg017%2Fsqlite-xsv/lists"}