https://github.com/grihabor/pyiceberg-hdfs-native
Provides a pyiceberg.io.FileIO implementation that uses hdfs-native client.
https://github.com/grihabor/pyiceberg-hdfs-native
fsspec hdfs hdfs-native iceberg pyiceberg
Last synced: 7 months ago
JSON representation
Provides a pyiceberg.io.FileIO implementation that uses hdfs-native client.
- Host: GitHub
- URL: https://github.com/grihabor/pyiceberg-hdfs-native
- Owner: grihabor
- License: mit
- Created: 2025-06-09T22:32:05.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2025-08-27T11:14:22.000Z (9 months ago)
- Last Synced: 2025-08-27T20:32:32.648Z (9 months ago)
- Topics: fsspec, hdfs, hdfs-native, iceberg, pyiceberg
- Language: Python
- Homepage:
- Size: 7.81 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# pyiceberg-hdfs-native
Provides a `pyiceberg.io.FileIO` implementation that uses
[`hdfs-native`](https://github.com/Kimahriman/hdfs-native) client.
## How to use
Install with uv:
```bash
uv tool install --with pyiceberg-hdfs-native pyiceberg
```
Configure pyiceberg via `~/.pyiceberg.yaml`:
```bash
default:
uri: https://iceberg.example.com/
py-io-impl: pyiceberg_hdfs_native.HdfsFileIO
```
Configure hdfs-native:
```bash
export HADOOP_CONF_DIR=/opt/hadoop/conf
```
If using kerberos, run `kinit`.
Now `files` command should work:
```
pyiceberg files db.table
```
## Read iceberg table with polars
```bash
uv run --with polars --with pyarrow --with pyiceberg-hdfs-native python
```
```python
from pyiceberg.catalog import load_catalog
import polars as pl
def read_table(table_name):
catalog = load_catalog(name='default') # will read config from ~/.pyiceberg.yaml
table = catalog.load_table(table_name)
metadata_location = table.metadata_location
storage_options = {'py-io-impl': 'pyiceberg_hdfs_native.HdfsFileIO'}
return pl.scan_iceberg(metadata_location, storage_options=storage_options, reader_override='pyiceberg')
read_table('db.tbl').head().collect()
```