{"id":23989603,"url":"https://github.com/candidtim/qurcol","last_synced_at":"2026-07-15T17:30:17.414Z","repository":{"id":57459544,"uuid":"269208772","full_name":"candidtim/qurcol","owner":"candidtim","description":"Command line tool to quickly explore the content of a file in a column-oriented storage format","archived":false,"fork":false,"pushed_at":"2020-06-03T22:39:34.000Z","size":27,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-08T06:17:28.992Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/candidtim.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-03T22:36:02.000Z","updated_at":"2021-07-15T16:13:37.000Z","dependencies_parsed_at":"2022-09-13T16:02:43.085Z","dependency_job_id":null,"html_url":"https://github.com/candidtim/qurcol","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/candidtim%2Fqurcol","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/candidtim%2Fqurcol/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/candidtim%2Fqurcol/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/candidtim%2Fqurcol/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/candidtim","download_url":"https://codeload.github.com/candidtim/qurcol/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240599181,"owners_count":19826959,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-07T17:32:16.992Z","updated_at":"2026-07-15T17:30:17.356Z","avatar_url":"https://github.com/candidtim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# qurcol - view, query and convert columnar storage files from command line\n\n`qurcol` (as in \"query columnar ...\") is a tool that enables its users to\nquickly explore the content of a file in a column-oriented storage format (like\n[Apache Parquet](https://parquet.apache.org/), for example), using command line\nonly and without the need for more complex software like Spark or Pandas.\n\nIt allows viewing the file content, schema, querying the content with SQL and\nconverting the data to CSV.\n\nThis tool only targets the use case of a basic exploration of a file content.\nThe author believes that aforementioned Spark, Pandas, etc. should be used\ninstead in any scenario which goes beyond that.\n\n\n## Features\n\nCommand line tool to:\n\n - view a columnar file content\n - print a columnar file schema\n - convert a columnar file to a CSV file\n - run SQL queries on the data from a columnar file\n\nList of supported columnar file formats:\n\n - [Apache Parquet](https://parquet.apache.org/)\n\nUsers should be aware of the size of the source files and keep in mind that\nthe file is read in memory when being processed by this tool.\n\n\n## Status\n\nThis software is generally available. This software is intended to be used in\ncommand line by individual users. It is not intended for use in a production\nenvironment.\n\nThe software is provided \"as is\", without warranty of any kind, express or\nimplied, including but not limited to the warranties of merchantability,\nfitness for a particular purpose and noninfringement. In no event shall the\nauthors or copyright holders be liable for any claim, damages or other\nliability, whether in an action of contract, tort or otherwise, arising from,\nout of or in connection with the software or the use or other dealings in the\nsoftware.\n\n\n\n## Installation\n\nInstall from PyPI:\n\n    pip install [--user] qurcol\n\nAlternatively, you can download a release from the Release page in GitHub.\n\n\n## Usage\n\nBuilt-in help provides detailed usage information with examples:\n\n    qurcol --help\n\n\nFew examples are given below to demonstrate the usage. Please, refer to the\nbuilt-in help for all details though (it is not practical to keep a duplicate\nof the \"help\" in this README).\n\nPrint an extract (few first rows, few last rows) of the file content:\n\n    qurcol view [[--head=N] [--tail=N] | [--all]] [--output=table|csv] FILE\n\nFor example, view to the snippet of a file content:\n\n    qurcol view FILE\n\nor, export entire file in CSV format:\n\n    qurcol --all --output=csv FILE \u003e OUTPUT_FILE\n\nPrint the file schema:\n\n    qurcol schema [--output=table|csv] FILE\n\nRun an inline SQL query on the file content:\n\n    qurcol sql -c FILE \"QUERY\"\n\n\n### SQL syntax\n\n`qurcol sql` command loads the file into an in-memory SQLite database. Therefore,\ndata types and SQL syntax are those of SQLite v.3. The command will attempt\nits best to map the data types from the source file to the data types available\nin SQLite, but users should be aware of the fact that SQLite data types are\noften less expressive (for example, there is no data type to represent\ndate/time information).\n\nData is loaded into a table with the name `data`.\n\nFinally, for a complete example, the following command:\n\n    qurcol sql -c --output=csv FILE \"select * from data\"\n\nwill have same effect as:\n\n    qurcol view --all --output=csv FILE\n\n\n## Why?\n\nHere are few sample use cases:\n\n - As a Data Engineer I want to review the schema of a file in a data\n   lake, in order to consume them accordingly in my software.\n\n - As a Data Ops Engineer I want to review the content of the sample file from\n   a data lake, in order to ensure that the data is being produced into it.\n\n - As a Data Engineer I want to query some data from a file in a data lake, in\n   order to review/confirm the properties of the data I am going to use.\n\n - As a Product Manager I want to load into a spreadsheet the `.parquet` file\n   shared with me by Data Science team, in order to review its content.\n\n\n## Not features\n\nFor any tool it is equally important to know what can and and what cannot be\ndone with it. Following potential features were considered for inclusion but\ndecided to be not in the current scope of this tool, unless a clear use case is\ndefined for them.\n\n - Conversion to other \"complex\" data formats (e.g. Excel), because it will add\n   more dependencies to the tool, while CSV can be imported to a spreadsheet\n   easily.\n\n - Reading data from files in other \"file systems\" (HDFS, S3, etc.), because\n   there exist command line tools to \"dowload\" data from these.\n\n - Any advanced data exploration and plotting, because there is no single way\n   to do that. You may use a combination of [Jupyter](https://jupyter.org/),\n   [Pandas](https://pandas.pydata.org/) and\n   [Seaborn](https://seaborn.pydata.org/) instead, for example.\n\n - Any advanced form of querying the data that goes beyond SQL. You may use\n   [Pandas](https://pandas.pydata.org/) or [Apache\n   Spark](https://spark.apache.org/) instead, for example.\n\n\n## Contributions\n\nContributions are very welcome. Please, feel free to submit an issue or create\na Pull Request.\n\n### Development environment\n\nThis software is written in Python 3, and has a modern development environment\nthat depends on [Poetry](https://python-poetry.org/docs/) and\n[Nox](https://nox.thea.codes/en/stable/).\n\nRun all tests:\n\n    nox -s tests\n\nOr, to quickly run tests on a single Python version only:\n\n    poetry run pytest --cov\n\nRun linters:\n\n    nox lint\n\nReformat code:\n\n    nox -s black\n\nOr, check code format without reformatting:\n\n    nox -s black -- --check .\n\nFull pre-commit check:\n\n    nox\n\nNote: Nox is configured to reuse virtualenvs by default; if you want to run Nox\nin a clean environment, add `--no-reuse-existing-virtualenvs` argument.\n\n\n### Criteria for Pull Requests\n\n - The PR should pass on CI. CI is configured to run all essential controls\n   (tests, flake8, mypy, black). You can easily run same controls in a local\n   development environment before the PR submission.\n\n - Beyond code formatting, please, try to stick to the overall code style, such\n   as the choice of variable names, code structure, etc.\n\n\n## License\n\nLicensed under the [Apache License v2.0](http://www.apache.org/licenses/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcandidtim%2Fqurcol","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcandidtim%2Fqurcol","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcandidtim%2Fqurcol/lists"}