https://github.com/ktrueda/parquet-tools
easy install parquet-tools
https://github.com/ktrueda/parquet-tools
cli parquet parquet-tools
Last synced: about 1 month ago
JSON representation
easy install parquet-tools
- Host: GitHub
- URL: https://github.com/ktrueda/parquet-tools
- Owner: ktrueda
- License: mit
- Created: 2020-05-02T06:46:50.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2024-07-09T19:15:45.000Z (over 1 year ago)
- Last Synced: 2025-04-21T17:45:20.848Z (7 months ago)
- Topics: cli, parquet, parquet-tools
- Language: Python
- Homepage:
- Size: 522 KB
- Stars: 176
- Watchers: 4
- Forks: 24
- Open Issues: 20
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# parquet-tools
This is a pip installable [parquet-tools](https://github.com/apache/parquet-mr).
In other words, parquet-tools is a CLI tools of [Apache Arrow](https://github.com/apache/arrow).
You can show parquet file content/schema on local disk or on Amazon S3.
It is incompatible with original parquet-tools.
## Features
- Read Parquet data (local file or file on S3)
- Read Parquet metadata/schema (local file or file on S3)
## Installation
```bash
$ pip install parquet-tools
```
## Usage
```bash
$ parquet-tools --help
usage: parquet-tools [-h] {show,csv,inspect} ...
parquet CLI tools
positional arguments:
{show,csv,inspect}
show Show human readble format. see `show -h`
csv Cat csv style. see `csv -h`
inspect Inspect parquet file. see `inspect -h`
optional arguments:
-h, --help show this help message and exit
```
## Usage Examples
#### Show local parquet file
```bash
$ parquet-tools show test.parquet
+-------+-------+---------+
| one | two | three |
|-------+-------+---------|
| -1 | foo | True |
| nan | bar | False |
| 2.5 | baz | True |
+-------+-------+---------+
```
#### Show parquet file on S3
```bash
$ parquet-tools show s3://bucket-name/prefix/*
+-------+-------+---------+
| one | two | three |
|-------+-------+---------|
| -1 | foo | True |
| nan | bar | False |
| 2.5 | baz | True |
+-------+-------+---------+
```
#### Inspect parquet file schema
```bash
$ parquet-tools inspect /path/to/parquet
```
Inspect output
```
############ file meta data ############
created_by: parquet-cpp version 1.5.1-SNAPSHOT
num_columns: 3
num_rows: 3
num_row_groups: 1
format_version: 1.0
serialized_size: 2226
############ Columns ############
one
two
three
############ Column(one) ############
name: one
path: one
max_definition_level: 1
max_repetition_level: 0
physical_type: DOUBLE
logical_type: None
converted_type (legacy): NONE
############ Column(two) ############
name: two
path: two
max_definition_level: 1
max_repetition_level: 0
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
############ Column(three) ############
name: three
path: three
max_definition_level: 1
max_repetition_level: 0
physical_type: BOOLEAN
logical_type: None
converted_type (legacy): NONE
```
#### Cat CSV parquet and transform [csvq](https://github.com/mithrandie/csvq)
```bash
$ parquet-tools csv s3://bucket-name/test.parquet |csvq "select one, three where three"
+-------+-------+
| one | three |
+-------+-------+
| -1.0 | True |
| 2.5 | True |
+-------+-------+
```