{"id":13395398,"url":"https://github.com/harelba/q","last_synced_at":"2025-05-13T20:09:41.482Z","repository":{"id":2346309,"uuid":"3308920","full_name":"harelba/q","owner":"harelba","description":"q - Run SQL directly on delimited files and multi-file sqlite databases","archived":false,"fork":false,"pushed_at":"2024-07-23T01:17:37.000Z","size":3202,"stargazers_count":10268,"open_issues_count":124,"forks_count":425,"subscribers_count":164,"default_branch":"master","last_synced_at":"2025-04-28T10:52:51.425Z","etag":null,"topics":["cli","command-line","command-line-tool","csv","database","python","q","qtextasdata","sql","sqlite","sqlite3","textasdata","tsv"],"latest_commit_sha":null,"homepage":"http://harelba.github.io/q/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/harelba.png","metadata":{"files":{"readme":"README.markdown","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"harelba"}},"created_at":"2012-01-30T21:12:09.000Z","updated_at":"2025-04-27T10:18:25.000Z","dependencies_parsed_at":"2023-07-06T09:48:51.574Z","dependency_job_id":"eb712421-0e07-4014-a3b9-2a4c22a375c7","html_url":"https://github.com/harelba/q","commit_stats":{"total_commits":367,"total_committers":24,"mean_commits":"15.291666666666666","dds":0.08446866485013627,"last_synced_commit":"a1da26ce85936b700ccefd1e35c279f9d2b21ecb"},"previous_names":[],"tags_count":60,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harelba%2Fq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harelba%2Fq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harelba%2Fq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harelba%2Fq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/harelba","download_url":"https://codeload.github.com/harelba/q/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254020606,"owners_count":22000753,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","command-line","command-line-tool","csv","database","python","q","qtextasdata","sql","sqlite","sqlite3","textasdata","tsv"],"created_at":"2024-07-30T17:01:56.656Z","updated_at":"2025-05-13T20:09:41.465Z","avatar_url":"https://github.com/harelba.png","language":"Python","funding_links":["https://github.com/sponsors/harelba","https://www.patreon.com/harelba"],"categories":["Python","Uncategorized","其他__大数据","Command line","cli","⚡ Productivity","Bash","Text-to-SQLite"],"sub_categories":["Uncategorized","网络服务_其他","Useful Linux Tools","Desktop"],"readme":"[![Build and Package](https://github.com/harelba/q/workflows/BuildAndPackage/badge.svg?branch=master)](https://github.com/harelba/q/actions?query=branch%3Amaster)\n\n# q - Text as Data\nq's purpose is to bring SQL expressive power to the Linux command line and to provide easy access to text as actual data.\n\nq allows the following:\n\n* Performing SQL-like statements directly on tabular text data, auto-caching the data in order to accelerate additional querying on the same file. \n* Performing SQL statements directly on multi-file sqlite3 databases, without having to merge them or load them into memory\n\nThe following table shows the impact of using caching:\n\n|    Rows   | Columns | File Size | Query time without caching | Query time with caching | Speed Improvement |\n|:---------:|:-------:|:---------:|:--------------------------:|:-----------------------:|:-----------------:|\n| 5,000,000 |   100   |   4.8GB   |    4 minutes, 47 seconds   |       1.92 seconds      |        x149       |\n| 1,000,000 |   100   |   983MB   |        50.9 seconds        |      0.461 seconds      |        x110       |\n| 1,000,000 |    50   |   477MB   |        27.1 seconds        |      0.272 seconds      |        x99        |\n|  100,000  |   100   |    99MB   |         5.2 seconds        |      0.141 seconds      |        x36        |\n|  100,000  |    50   |    48MB   |         2.7 seconds        |      0.105 seconds      |        x25        |\n\nNotice that for the current version, caching is **not enabled** by default, since the caches take disk space. Use `-C readwrite` or `-C read` to enable it for a query, or add `caching_mode` to `.qrc` to set a new default.\n \nq's web site is [https://harelba.github.io/q/](https://harelba.github.io/q/) or [https://q.textasdata.wiki](https://q.textasdata.wiki) It contains everything you need to download and use q immediately.\n\n\n## Usage Examples\nq treats ordinary files as database tables, and supports all SQL constructs, such as `WHERE`, `GROUP BY`, `JOIN`s, etc. It supports automatic column name and type detection, and provides full support for multiple character encodings.\n\nHere are some example commands to get the idea:\n\n```bash\n$ q \"SELECT COUNT(*) FROM ./clicks_file.csv WHERE c3 \u003e 32.3\"\n\n$ ps -ef | q -H \"SELECT UID, COUNT(*) cnt FROM - GROUP BY UID ORDER BY cnt DESC LIMIT 3\"\n\n$ q \"select count(*) from some_db.sqlite3:::albums a left join another_db.sqlite3:::tracks t on (a.album_id = t.album_id)\"\n```\n\nDetailed examples are in [here](https://harelba.github.io/q/#examples)\n\n## Installation.\n**New Major Version `3.1.6` is out with a lot of significant additions.**\n\nInstructions for all OSs are [here](https://harelba.github.io/q/#installation).\n\nThe previous version `2.0.19` Can still be downloaded from [here](https://github.com/harelba/q/releases/tag/2.0.19)  \n\n## Contact\nAny feedback/suggestions/complaints regarding this tool would be much appreciated. Contributions are most welcome as well, of course.\n\nLinkedin: [Harel Ben Attia](https://www.linkedin.com/in/harelba/)\n\nTwitter [@harelba](https://twitter.com/harelba)\n\nEmail [harelba@gmail.com](mailto:harelba@gmail.com)\n\nq on twitter: [#qtextasdata](https://twitter.com/hashtag/qtextasdata?src=hashtag_click)\n\nPatreon: [harelba](https://www.patreon.com/harelba) - All the money received is donated to the [Center for the Prevention and Treatment of Domestic Violence](https://www.gov.il/he/departments/bureaus/molsa-almab-ramla) in my hometown - Ramla, Israel.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharelba%2Fq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fharelba%2Fq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharelba%2Fq/lists"}