{"id":51201221,"url":"https://github.com/davanstrien/polars-hf","last_synced_at":"2026-06-28T00:31:00.613Z","repository":{"id":361887654,"uuid":"1256196088","full_name":"davanstrien/polars-hf","owner":"davanstrien","description":"Read and write Hugging Face Hub buckets with Polars (pure-Python IO plugin)","archived":false,"fork":false,"pushed_at":"2026-06-11T20:36:58.000Z","size":64,"stargazers_count":1,"open_issues_count":3,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-12T02:22:22.430Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davanstrien.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-01T14:50:39.000Z","updated_at":"2026-06-03T12:16:45.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/davanstrien/polars-hf","commit_stats":null,"previous_names":["davanstrien/polars-hf"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/davanstrien/polars-hf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davanstrien%2Fpolars-hf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davanstrien%2Fpolars-hf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davanstrien%2Fpolars-hf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davanstrien%2Fpolars-hf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davanstrien","download_url":"https://codeload.github.com/davanstrien/polars-hf/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davanstrien%2Fpolars-hf/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34873663,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-27T02:00:06.362Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-28T00:31:00.554Z","updated_at":"2026-06-28T00:31:00.605Z","avatar_url":"https://github.com/davanstrien.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# polars-hf\n\nRead and write [Hugging Face Hub buckets](https://huggingface.co/docs/hub/storage-buckets) with\n[Polars](https://pola.rs), as a pure-Python **IO plugin**. No fork of Polars, no compiled\nextensions — just install and scan.\n\n\u003e **Status:** alpha, pre-release (not on PyPI yet — install from git, see below).\n\u003e Reads (`scan_bucket`) and writes (`sink_bucket`, including partitioned) are implemented.\n\n## Why\n\nStock Polars already reads `hf://datasets/...` and `hf://spaces/...` natively. It does **not** yet\nread `hf://buckets/...`. `polars-hf` fills that gap from the outside.\n\nIt returns a **native** `pl.scan_parquet` LazyFrame: bucket files are XET-backed, so `scan_bucket`\nfollows the authenticated Hub `resolve` redirect to a presigned `cas-bridge.xethub.hf.co` URL and\nhands that to Polars. Polars' own Rust object store then does async, concurrent, **range-read**\nscans — so **projection, predicate, and slice pushdown**, streaming, and multi-file concurrency all\nwork natively and only the column chunks actually needed are transferred. (This is the same read\nmechanism upstream's `hf://` reader uses; we just resolve the signed URL in Python because stock\nPolars can't attach a bearer token to a generic `https://` URL.)\n\n\u003e [!NOTE]\n\u003e **This may be a stopgap.** Native `hf://buckets/...` support is proposed upstream in Polars —\n\u003e [pola-rs/polars#27611](https://github.com/pola-rs/polars/issues/27611) (reads) and\n\u003e [pola-rs/polars#26909](https://github.com/pola-rs/polars/issues/26909) (streaming sink). If those\n\u003e land, `polars-hf` becomes redundant; until then, it fills the gap from the outside.\n\n## Install\n\nNot on PyPI yet — install from git:\n\n```bash\nuv add \"polars-hf @ git+https://github.com/davanstrien/polars-hf\"\n# or: pip install \"git+https://github.com/davanstrien/polars-hf\"\n```\n\nRequires `polars\u003e=1.40,\u003c1.50` and `huggingface_hub\u003e=1.12`.\n\n### On Hugging Face Jobs\n\nUse a [PEP 723](https://peps.python.org/pep-0723/) inline-dependency script so the Job pulls the\nplugin straight from git — no build, no PyPI:\n\n```python\n# /// script\n# requires-python = \"\u003e=3.10\"\n# dependencies = [\"polars-hf @ git+https://github.com/davanstrien/polars-hf@main\"]\n# ///\nimport polars as pl\nimport polars_hf as plhf\n\nplhf.scan_bucket(\"hf://buckets/me/data/*.parquet\").filter(pl.col(\"score\") \u003e 0.5).collect()\n```\n\n```bash\nhf jobs uv run --secrets HF_TOKEN --flavor cpu-upgrade my_script.py\n```\n\nSee [`examples/run_on_hf_jobs.py`](examples/run_on_hf_jobs.py) for a runnable example.\n\n## Usage\n\n```python\nimport polars as pl\nimport polars_hf as plhf\n\n# A single file, a glob, or a whole bucket/directory (expanded to **/*.parquet):\nlf = plhf.scan_bucket(\"hf://buckets/my-namespace/my-bucket/data/*.parquet\")\n\ndf = (\n    lf.filter(pl.col(\"label\") == 1)   # predicate pushdown\n      .select(\"text\", \"label\")        # projection pushdown\n      .head(100)                       # row-limit pushdown\n      .collect()\n)\n```\n\n`scan_bucket` returns a lazy `LazyFrame` and works with the streaming engine.\n\n### Writing\n\n```python\n# Single file (parquet/csv/ipc/ndjson; format inferred from the extension):\nplhf.sink_bucket(lf, \"hf://buckets/ns/name/out.parquet\")\n\n# Partitioned: pass a base prefix + partition options (native pl.PartitionBy):\nplhf.sink_bucket(lf, \"hf://buckets/ns/name/by_year\", partition_by=\"year\")\nplhf.sink_bucket(lf, \"hf://buckets/ns/name/shards\", max_rows_per_file=1_000_000)\n```\n\n`sink_bucket` accepts a `LazyFrame` (streaming) or a `DataFrame`. Partitioned writes split by key\n(hive `key=value/` layout), by size, or both. Two modes:\n\n- `atomic=True` (default) — stage partitions locally, upload in one commit; bounded by local disk.\n- `atomic=False` — stream each partition straight to the bucket; handles bigger-than-disk, one commit\n  per file (cheap on buckets, which are not git-backed).\n\n### Authentication\n\nBy default the token is resolved by `huggingface_hub` (the `HF_TOKEN` environment variable or your\ncached `hf auth login`). You can also pass one explicitly:\n\n```python\nplhf.scan_bucket(\"hf://buckets/ns/name/data.parquet\", token=\"hf_...\")\n```\n\n## Supported URIs\n\n```\nhf://buckets/{namespace}/{name}/{path}\n```\n\n- `{path}` may be a single `.parquet` file, a glob (`data/*.parquet`), or a directory / the whole\n  bucket (expanded to `**/*.parquet`).\n- Buckets have **no** revision concept, so `@revision` is rejected (matching the Hub).\n- `hf://datasets/...` and `hf://spaces/...` are read natively by Polars — use\n  `pl.scan_parquet(...)` for those.\n\nSigned URLs are resolved when `scan_bucket` is called and are valid for ~1 hour. Collect within that\nwindow; for long-lived query plans, call `scan_bucket` again to refresh.\n\n## Performance\n\nBucket reads fetch many small range requests from the XET CDN. Two things dominate:\n\n- **Concurrency.** polars' default cloud-IO concurrency (`max(cpu_threads, 10)`) is low for\n  high-latency object stores. `polars-hf` raises `POLARS_CONCURRENCY_BUDGET` to `64` by default\n  (override by setting it yourself). This is a large win on warm/repeated scans and ~15% on cold.\n- **Cold vs warm CDN.** The *first* read of freshly written/copied data pays a cold-CDN penalty\n  (the bytes aren't at the edge yet); subsequent reads are much faster. For repeated large-scale\n  reads, consider [pre-warming](https://huggingface.co/docs/hub/storage-buckets#pre-warming-and-cdn)\n  the bucket.\n\n## Limitations\n\n- **Heterogeneous schemas across globbed files are not yet supported** (the equivalent of native\n  Polars `missing_columns=\"insert\"`). Like Polars' own default, mismatched schemas raise. Planned.\n- Reads cover parquet; writes cover parquet/csv/ipc/ndjson. Delta/Iceberg are out of scope.\n\n## Development\n\n```bash\nuv sync\nuv run ruff check .\nuv run pytest                 # network tests skip without an HF token\n```\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavanstrien%2Fpolars-hf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavanstrien%2Fpolars-hf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavanstrien%2Fpolars-hf/lists"}