{"id":13466997,"url":"https://github.com/neilotoole/sq","last_synced_at":"2026-05-16T06:20:42.324Z","repository":{"id":40628443,"uuid":"51800292","full_name":"neilotoole/sq","owner":"neilotoole","description":"sq data wrangler","archived":false,"fork":false,"pushed_at":"2025-04-16T22:55:33.000Z","size":51525,"stargazers_count":2283,"open_issues_count":30,"forks_count":33,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-05-12T05:19:30.394Z","etag":null,"topics":["azure-sql-edge","csv","data-wrangler","database","excel","go","golang","json","jsona","jsonl","markdown","mysql","postgres","sql","sqlserver","tsv","xlsx","xml"],"latest_commit_sha":null,"homepage":"https://sq.io","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/neilotoole.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-02-16T02:01:18.000Z","updated_at":"2025-05-09T02:28:35.000Z","dependencies_parsed_at":"2023-09-24T22:22:11.082Z","dependency_job_id":"44feb769-6c12-4dd0-a2ba-a6c14e51e15f","html_url":"https://github.com/neilotoole/sq","commit_stats":{"total_commits":645,"total_committers":3,"mean_commits":215.0,"dds":0.0728682170542636,"last_synced_commit":"9dc178a9933ebaa592d2190121b40f14f59405d0"},"previous_names":[],"tags_count":99,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neilotoole%2Fsq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neilotoole%2Fsq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neilotoole%2Fsq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neilotoole%2Fsq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/neilotoole","download_url":"https://codeload.github.com/neilotoole/sq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254101558,"owners_count":22014908,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure-sql-edge","csv","data-wrangler","database","excel","go","golang","json","jsona","jsonl","markdown","mysql","postgres","sql","sqlserver","tsv","xlsx","xml"],"created_at":"2024-07-31T15:00:52.094Z","updated_at":"2026-05-16T06:20:42.316Z","avatar_url":"https://github.com/neilotoole.png","language":"Go","funding_links":[],"categories":["Template Engines","Go","Text Processing","文本处理","Other","SQL","\u003ca name=\"data-management-tabular\"\u003e\u003c/a\u003eData management - Tabular data","database"],"sub_categories":["Formatters","格式器","Über SQL"],"readme":"[![Go Reference](https://pkg.go.dev/badge/github.com/neilotoole/sq.svg)](https://pkg.go.dev/github.com/neilotoole/sq)\n[![Go Report Card](https://goreportcard.com/badge/neilotoole/sq)](https://goreportcard.com/report/neilotoole/sq)\n[![License](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/neilotoole/sq/blob/master/LICENSE)\n![Main pipeline](https://github.com/neilotoole/sq/actions/workflows/main.yml/badge.svg)\n\n# sq data wrangler\n\n`sq` is a command line tool that provides jq-style access to\nstructured data sources: SQL databases, or document formats like CSV or Excel.\nIt is the lovechild of sql+jq.\n\n![sq](.images/splash.png)\n\n`sq` executes jq-like [queries](https://sq.io/docs/query), or database-native [SQL](https://sq.io/docs/cmd/sql/).\nIt can [join](https://sq.io/docs/query#cross-source-joins) across sources: join a CSV file to a Postgres table, or\nMySQL with Excel.\n\n`sq` outputs to a multitude of [formats](https://sq.io/docs/output#formats)\nincluding [JSON](https://sq.io/docs/output#json),\n[Excel](https://sq.io/docs/output#xlsx), [CSV](https://sq.io/docs/output#csv),\n[HTML](https://sq.io/docs/output#html), [Markdown](https://sq.io/docs/output#markdown)\nand [XML](https://sq.io/docs/output#xml), and can [insert](https://sq.io/docs/output#insert) query\nresults directly to a SQL database.\n\n`sq` can also [inspect](https://sq.io/docs/inspect) sources to view metadata about the source structure (tables,\ncolumns, size). You can use [`sq diff`](https://sq.io/docs/diff) to compare tables, or\nentire databases. `sq` has commands for common database operations to\n[copy](https://sq.io/docs/cmd/tbl-copy), [truncate](https://sq.io/docs/cmd/tbl-truncate),\nand [drop](https://sq.io/docs/cmd/tbl-drop) tables.\n\nFind out more at [sq.io](https://sq.io).\n\n\u003e [!TIP]\n\u003e The rest of this doc is mainly for `sq` end users and first-timers. Contributors (bug reports, feature requests, pull requests),\n\u003e see [CONTRIBUTING.md](./CONTRIBUTING.md).\n\n## Drivers\n\n`sq` knows how to deal with a data source type via a [driver](https://sq.io/docs/drivers)\nimplementation. To view the installed/supported drivers:\n\n```shell\n$ sq driver ls\nDRIVER      DESCRIPTION\nsqlite3     SQLite\npostgres    PostgreSQL\nsqlserver   Microsoft SQL Server / Azure SQL Edge\nmysql       MySQL\nclickhouse  ClickHouse\noracle      Oracle Database (experimental)\nduckdb      DuckDB\ncsv         Comma-Separated Values\ntsv         Tab-Separated Values\njson        JSON\njsona       JSON Array: LF-delimited JSON arrays\njsonl       JSON Lines: LF-delimited JSON objects\nxlsx        Microsoft Excel XLSX\n```\n\n\u003e [!NOTE]\n\u003e ClickHouse Driver support is currently in beta. Full details of support can be\n\u003e found in the [ClickHouse README](drivers/clickhouse/README.md).\n\n\u003e [!NOTE]\n\u003e **Oracle Driver (Experimental):** Oracle Database support is experimental.\n\u003e The driver uses pure Go ([go-ora](https://github.com/sijms/go-ora)) and does\n\u003e not require Oracle Instant Client. See the [Oracle driver docs](drivers/oracle/README.md) for setup.\n\n## Install\n\n### macOS\n\n```shell\nbrew install sq\n```\n\n\u003e [!IMPORTANT]\n\u003e `sq` is now a [core brew formula](https://formulae.brew.sh/formula/sq#default). Previously, `sq` was available via `brew install neilotoole/sq/sq`. If you have installed `sq` this way, you should uninstall it (`brew uninstall neilotoole/sq/sq`) before installing the new formula via `brew install sq`.\n\n### Linux\n\n```shell\n/bin/sh -c \"$(curl -fsSL https://sq.io/install.sh)\"\n```\n\n### Windows\n\n```shell\nscoop bucket add sq https://github.com/neilotoole/sq\nscoop install sq\n```\n\n### Go\n\n```shell\ngo install github.com/neilotoole/sq\n```\n\n### Docker\n\nThe [`ghcr.io/neilotoole/sq`](https://github.com/neilotoole/sq/pkgs/container/sq)\nimage is preloaded with `sq` and a handful of related tools like `jq`.\n\n#### Local\n\n```shell\n# Shell into a one-time container.\n$ docker run -it ghcr.io/neilotoole/sq zsh\n\n# Start detached (background) container named \"sq-shell\".\n$ docker run -d --name sq-shell ghcr.io/neilotoole/sq\n# Shell into that container.\n$ docker exec -it sq-shell zsh\n```\n\n#### Kubernetes\n\nRunning `sq` in a Kubernetes environment is useful for DB migrations,\nas well as general data wrangling.\n\n```shell\n# Start pod named \"sq-shell\".\n$ kubectl run sq-shell --image ghcr.io/neilotoole/sq\n# Shell into the pod.\n$ kubectl exec -it sq-shell -- zsh\n```\n\nSee other [install options](https://sq.io/docs/install/).\n\n## Agent skills\n\nAn [Agent Skills](https://agentskills.io/) skill for this repo lives under\n[`skills/sq/`](./skills/sq/) and helps coding assistants use an installed `sq`\n(CLI patterns, `references/` for drivers). **User-facing install and usage** are\ndocumented at **[sq.io/docs/agent-skills](https://sq.io/docs/agent-skills)**.\n\nQuick install from GitHub:\n\n```shell\nnpx skills add neilotoole/sq\n```\n\n## Documentation site (sq.io)\n\nUser-facing docs at [sq.io](https://sq.io) are built from the [`site/`](./site/) directory in this repository (Hugo + Bun + Netlify). To work on the website locally, see [`site/README.md`](./site/README.md). The standalone **`sq-web`** repository is **archived**; open issues and pull requests here against **`neilotoole/sq`** for doc and site changes.\n\n## Overview\n\nUse `sq help` to see command help. Docs are over at [sq.io](https://sq.io).\nRead the [overview](https://sq.io/docs/overview/), and\n[tutorial](https://sq.io/docs/tutorial/). The [cookbook](https://sq.io/docs/cookbook/) has\nrecipes for common tasks, and the [query guide](https://sq.io/docs/query) covers `sq`'s query language.\n\nThe major concept is: `sq` operates on data sources, which are treated as SQL databases (even if the\nsource is really a CSV or XLSX file etc.).\n\nIn a nutshell, you [`sq add`](https://sq.io/docs/cmd/add) a source (giving it a [`handle`](https://sq.io/docs/concepts#handle)), and then execute commands against the\nsource.\n\n### Sources\n\nInitially there are no [sources](https://sq.io/docs/source).\n\n```shell\n$ sq ls\n\n```\n\nLet's [add](https://sq.io/docs/cmd/add) a source. First we'll add a [SQLite](https://sq.io/docs/drivers/sqlite)\ndatabase, but this could also be [Postgres](https://sq.io/docs/drivers/postgres),\n[SQL Server](https://sq.io/docs/drivers/sqlserver), [MySQL](https://sq.io/docs/drivers/mysql), [Oracle](https://sq.io/docs/drivers/oracle) etc., or a document source such as [Excel](https://sq.io/docs/drivers/xlsx) or\n[CSV](https://sq.io/docs/drivers/csv).\n\nDownload the sample DB, and `sq add` the source.\n\n```shell\n$ wget https://sq.io/testdata/sakila.db\n\n$ sq add ./sakila.db\n@sakila  sqlite3  sakila.db\n\n$ sq ls -v\nHANDLE   ACTIVE  DRIVER   LOCATION                         OPTIONS\n@sakila  active  sqlite3  sqlite3:///Users/demo/sakila.db\n\n$ sq ping @sakila\n@sakila       1ms  pong\n\n$ sq src\n@sakila  sqlite3  sakila.db\n```\n\nThe [`sq ping`](https://sq.io/docs/cmd/ping) command simply pings the source\nto verify that it's available.\n\n[`sq src`](https://sq.io/docs/cmd/src) lists the [_active source_](https://sq.io/docs/source#active-source), which in our\ncase is `@sakila`.\nYou can change the active source using `sq src @other_src`.\nWhen there's an active source specified, you can usually omit the handle from `sq` commands.\nThus you could instead do:\n\n```shell\n$ sq ping\n@sakila  1ms  pong\n```\n\n\u003e [!TIP]\n\u003e Document sources such as CSV or Excel can be added from the local filesystem, or\n\u003e from an HTTP URL.\n\u003e\n\u003e ```shell\n\u003e $ sq add https://acme.s3.amazonaws.com/sales.csv\n\u003e ```\n\u003e\n\u003e ![sq inspect remote](./.images/sq_inspect_remote_s3.png)\n\u003e See the [sources](https://sq.io/docs/source#download) docs for more.\n\n### Query\n\nFundamentally, `sq` is for querying data. The jq-style syntax is covered in\ndetail in the [query guide](https://sq.io/docs/query).\n\n![sq query where slq](./.images/sq_query_where_slq.png)\n\nThe above query selected some rows from the `actor` table. You could also\nuse [native SQL](https://sq.io/docs/cmd/sql), e.g.:\n\n![sq query where sql](./.images/sq_query_where_sql.png)\n\nBut we're flying a bit blind here: how did we know about the `actor` table?\n\n### Inspect\n\n[`sq inspect`](https://sq.io/docs/inspect) is your friend.\n\n![sq inspect](./.images/sq_inspect_source_text.png)\n\nUse [`sq inspect -v`](https://sq.io/docs/cmd/inspect) to see more detail.\nOr use [`-j`](https://sq.io/docs/output#json) to get JSON output:\n\n![sq inspect -j](./.images/sq_inspect_sakila_sqlite_json.png)\n\nCombine `sq inspect` with [jq](https://jqlang.github.io/jq/) for some useful capabilities.\nHere's how to [list](https://sq.io/docs/cookbook#list-table-names)\nall the table names in the active source:\n\n```shell\n$ sq inspect -j | jq -r '.tables[] | .name'\nactor\naddress\ncategory\ncity\ncountry\ncustomer\n[...]\n```\n\nAnd here's how you\ncould [export](https://sq.io/docs/cookbook#export-all-table-data-to-csv) each table\nto a CSV file:\n\n```shell\n$ sq inspect -j | jq -r '.tables[] | .name' | xargs -I % sq .% --csv --output %.csv\n$ ls\nactor.csv     city.csv      customer_list.csv  film_category.csv  inventory.csv  rental.csv                  staff.csv\naddress.csv   country.csv   film.csv           film_list.csv      language.csv   sales_by_film_category.csv  staff_list.csv\ncategory.csv  customer.csv  film_actor.csv     film_text.csv      payment.csv    sales_by_store.csv          store.csv\n```\n\nNote that you can also inspect an individual table:\n\n![sq inspect actor verbose](./.images/sq_inspect_actor_verbose.png)\n\nRead more about [`sq inspect`](https://sq.io/docs/inspect).\n\n### Diff\n\nUse [`sq diff`](https://sq.io/docs/diff) to compare metadata, or row data, for sources, or individual tables.\n\nThe default behavior is to diff table schema and row counts. Table row data is not compared in this mode.\n\n![sq diff](.images/sq_diff_src_default.png)\n\nUse [`--data`](https://sq.io/docs/diff#--data) to compare row data.\n\n![sq diff data](.images/sq_diff_table_data.png)\n\nThere are many more options available. See the [diff docs](https://sq.io/docs/diff).\n\n### Insert query results\n\n`sq` query results can be [output](https://sq.io/docs/output) in various formats\n([`text`](https://sq.io/docs/output#text),\n[`json`](https://sq.io/docs/output#json),\n[`csv`](https://sq.io/docs/output#csv), etc.). Those results can also be \"outputted\"\nas an [_insert_](https://sq.io/docs/output#insert) into a database table.\n\nThat is, you can use `sq` to insert results from a Postgres query into a MySQL table,\nor copy an Excel worksheet into a SQLite table, or a push a CSV file into\na SQL Server table etc.\n\n\u003e [!TIP]\n\u003e If you want to copy a table inside the same (database) source,\n\u003e use [`sq tbl copy`](https://sq.io/docs/cmd/tbl-copy) instead, which uses the database's native table copy functionality.\n\nHere we query a CSV file, and insert the results into a Postgres table.\n\n![sq query insert inspect](./.images/sq_query_insert_inspect.png)\n\n### Cross-source joins\n\n`sq` can perform the usual [joins](https://sq.io/docs/query#joins). Here's how you would\njoin tables `actor`, `film_actor`, and `film`:\n\n```shell\n$ sq '.actor | join(.film_actor, .actor_id) | join(.film, .film_id) | .first_name, .last_name, .title'\n```\n\nBut `sq` can also join across data sources. That is, you can join an Excel worksheet with a\nPostgres table, or join a CSV file with MySQL, and so on.\n\nThis example joins a Postgres database, an Excel worksheet, and a CSV file.\n\n![sq join multi source](./.images/sq_join_multi_source.png)\n\nRead more about cross-source joins in the [query guide](https://sq.io/docs/query#joins).\n\n### Table commands\n\n`sq` provides several handy commands for working with tables:\n[`tbl copy`](/docs/cmd/tbl-copy), [`tbl truncate`](/docs/cmd/tbl-truncate)\nand [`tbl drop`](/docs/cmd/tbl-drop).\nNote that these commands work directly\nagainst SQL database sources, using their native SQL commands.\n\n```shell\n$ sq tbl copy .actor .actor_copy\nCopied table: @sakila.actor --\u003e @sakila.actor_copy (200 rows copied)\n\n$ sq tbl truncate .actor_copy\nTruncated 200 rows from @sakila.actor_copy\n\n$ sq tbl drop .actor_copy\nDropped table @sakila.actor_copy\n```\n\n### UNIX pipes\n\nFor file-based sources (such as CSV or XLSX), you can `sq add` the source file,\nbut you can also pipe it:\n\n```shell\n$ cat ./example.xlsx | sq .Sheet1\n```\n\nSimilarly, you can inspect:\n\n```shell\n$ cat ./example.xlsx | sq inspect\n```\n\n## Output formats\n\n`sq` has many [output formats](https://sq.io/docs/output):\n\n- `--text`: [Text](https://sq.io/docs/output#text)\n- `--json`: [JSON](https://sq.io/docs/output#json)\n- `--jsona`: [JSON Array](https://sq.io/docs/output#jsona)\n- `--jsonl`: [JSON Lines](https://sq.io/docs/output#jsonl)\n- `--csv` / `--tsv` : [CSV](https://sq.io/docs/output#csv) / [TSV](https://sq.io/docs/output#tsv)\n- `--xlsx`: [XLSX](https://sq.io/docs/output#xlsx) (Microsoft Excel)\n- `--html`: [HTML](https://sq.io/docs/output#html)\n- `--xml`: [XML](https://sq.io/docs/output#xml)\n- `--yaml`: [YAML](https://sq.io/docs/output#yaml)\n- `--markdown`: [Markdown](https://sq.io/docs/output#markdown)\n- `--raw`: [Raw](https://sq.io/docs/output#raw) (bytes)\n\n## Contributing\n\nSee [CONTRIBUTING.md](./CONTRIBUTING.md).\n\n## CHANGELOG\n\nSee [CHANGELOG.md](./CHANGELOG.md).\n\n## Acknowledgements\n\n- Thanks to [Diego Souza](https://github.com/diegosouza) for creating\n  the [Arch Linux package](https://aur.archlinux.org/packages/sq-bin), and [`@icp`](https://github.com/icp1994)\n  for creating the [Void Linux package](https://github.com/void-linux/void-packages/blob/master/srcpkgs/sq/template).\n- Much inspiration is owed to [jq](https://jqlang.github.io/jq/).\n- See [`go.mod`](https://github.com/neilotoole/sq/blob/master/go.mod) for a list of third-party\n  packages.\n- `sq` imports a bunch of [`usql`](https://github.com/xo/usql) functionality.\n- Additionally, `sq` incorporates modified versions of:\n - [`olekukonko/tablewriter`](https://github.com/olekukonko/tablewriter)\n - [`segmentio/encoding`](https://github.com/segmentio/encoding) for JSON encoding.\n- The [_Sakila_](https://dev.mysql.com/doc/sakila/en/) example databases were lifted\n  from [jOOQ](https://github.com/jooq/jooq), which in turn owe their heritage to earlier work on\n  Sakila.\n- Date rendering via [`ncruces/go-strftime`](https://github.com/ncruces/go-strftime).\n- A modified version [`dolmen-go/contextio`](https://github.com/dolmen-go/contextio) is\n  incorporated into the codebase.\n- [`djherbis/buffer`](https://github.com/djherbis/buffer) is used for caching.\n- A forked version of [`nightlyone/lockfile`](https://github.com/nightlyone/lockfile) is incorporated.\n- The human-friendly `text` log format handler is a fork of [`lmittmann/tint`](https://github.com/lmittmann/tint).\n\n## Similar, related, or noteworthy projects\n\n- [`usql`](https://github.com/xo/usql)\n- [`textql`](https://github.com/dinedal/textql)\n- [`golang-migrate`](https://github.com/golang-migrate/migrate)\n- [`octosql`](https://github.com/cube2222/octosql)\n- [`rq`](https://github.com/dflemstr/rq)\n- [`miller`](https://github.com/johnkerl/miller)\n- [`jsoncolor`](https://github.com/neilotoole/jsoncolor) is a JSON colorizer created for `sq`.\n- [`streamcache`](https://github.com/neilotoole/streamcache) is a Go in-memory byte cache mechanism created for `sq`.\n- [`fifomu`](https://github.com/neilotoole/fifomu) is a FIFO mutex, used by `streamcache`, and thus upstream in `sq`.\n- [`tailbuf`](https://github.com/neilotoole/tailbuf) is a fixed-size object tail buffer created for `sq`.\n- [`oncecache`](https://github.com/neilotoole/oncecache) is an in-memory object cache created for `sq`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneilotoole%2Fsq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneilotoole%2Fsq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneilotoole%2Fsq/lists"}