{"id":51201209,"url":"https://github.com/davanstrien/bucketbag","last_synced_at":"2026-06-28T00:30:59.696Z","repository":{"id":367741587,"uuid":"1282119768","full_name":"davanstrien/bucketbag","owner":"davanstrien","description":"A minimal, toolz-style helper for batched, bounded-memory access to Hugging Face bucket files.","archived":false,"fork":false,"pushed_at":"2026-06-27T12:29:14.000Z","size":21,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-27T13:04:29.894Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davanstrien.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-27T10:52:56.000Z","updated_at":"2026-06-27T12:29:18.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/davanstrien/bucketbag","commit_stats":null,"previous_names":["davanstrien/bucketbag"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/davanstrien/bucketbag","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davanstrien%2Fbucketbag","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davanstrien%2Fbucketbag/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davanstrien%2Fbucketbag/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davanstrien%2Fbucketbag/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davanstrien","download_url":"https://codeload.github.com/davanstrien/bucketbag/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davanstrien%2Fbucketbag/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34873663,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-27T02:00:06.362Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-28T00:30:59.637Z","updated_at":"2026-06-28T00:30:59.688Z","avatar_url":"https://github.com/davanstrien.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bucketbag\n\nA minimal, [toolz](https://toolz.readthedocs.io/)-style helper for **batched, bounded-memory** reads of\n**Hugging Face bucket** files. Not a framework — a few composable functions for the\n*download-a-batch → process → delete → repeat* loop you'd otherwise rewrite in every script.\n\n## Install\n\n```toml\n# uv script PEP 723 header\n# /// script\n# dependencies = [\"bucketbag @ git+https://github.com/davanstrien/bucketbag\"]\n# ///\n```\n(or `uv add \"bucketbag @ git+https://github.com/davanstrien/bucketbag\"` / `pip install -e .`)\n\n## `batched_files` — the one verb\n\n`partition_all` for bucket files: it lists/downloads a batch to a temp dir (`/dev/shm`), yields it, and\n**deletes it before the next** — so disk stays bounded. **File-type agnostic.** Cleanup is automatic\n(even on exception) — you never touch temp files.\n\n```python\nfrom bucketbag import batched_files\n\n# Bound each batch by SIZE (recommended) — predictable footprint whatever the file sizes:\nfor batch in batched_files(\"davanstrien/my-bucket\", include=\"**/*.jp2\", max_bytes=4 * 2**30):\n    for it in batch:                 # LoadedItem, already on local disk\n        work(it.path)                # or it.bytes / it.image / it.text() / it.json()\n    # ↑ this batch's files are deleted as the loop advances — nothing to clean up\n\n# …or bound by file count instead:\nfor batch in batched_files(\"davanstrien/my-bucket\", include=\"**/*.jp2\", n=32):\n    ...\n```\n\nDisk high-water ≈ `(prefetch + 1) × max_bytes`. `prefetch=2` (default) overlaps downloads with your work.\nThe only rule: **don't keep a `LoadedItem`/`.path` past its batch** — the file is already gone.\n\n## Resume (your loop, your rules)\n\n```python\nfrom bucketbag import iter_keys, batched_files, completed_keys, write_parquet\n\ndone = completed_keys(OUT)                                    # __source_key values already written\nkeys = [k for k in iter_keys(SRC, include=\"**/*.jp2\") if k not in done]\nfor batch in batched_files(SRC, keys=keys, max_bytes=4 * 2**30):\n    rows = [{\"__source_key\": it.key, **work(it)} for it in batch]\n    write_parquet(rows, OUT, f\"part-{batch[0].key.replace('/', '_')}.parquet\")\n```\n\n## API\n\n| | |\n| --- | --- |\n| `batched_files(bucket, *, keys, include, exclude, n=20, max_bytes, dir, prefetch=2, max_workers, start_after, limit, token)` | download batches → yield `list[LoadedItem]` → auto-delete |\n| `iter_keys(bucket, *, prefix, include, exclude, start_after, limit, token)` | list + glob-filter + sort keys (no download) |\n| `completed_keys(out_bucket, *, prefix, column=\"__source_key\", token)` | set of done keys, for resume |\n| `write_parquet(rows, out_bucket, key, *, token)` | write a list of dicts as one parquet object |\n| `boost(*, file_concurrency=32, high_performance=True)` | raise xet download concurrency (~2.5× on small files) |\n| `LoadedItem` | `.key` `.path` + lazy `.bytes` `.image` `.text()` `.json()` |\n| `partition_all` | re-exported from `toolz` |\n\n`bucket` = `\"ns/bucket\"`, `\"ns/bucket/prefix\"`, or `\"hf://buckets/ns/bucket/prefix\"`. Globs: `*` within a\npath segment, `**` across `/`. `n=None` + `max_bytes` gives pure size-based batches.\n\n## Performance\n\nCold + disjoint, replicated (`examples/bench.py`, l4x1, ~0.8 MB jp2; ranges, ±10–20%):\n\n| | default xet | + `boost()` |\n| --- | --- | --- |\n| `bucketbag` (prefetch 2–4) | ~85–110 img/s | **~200–270 img/s** |\n| raw `download_bucket_files` | ~85–105 | ~210–260 |\n| `HfFileSystem` (32 threads) | ~90–110 | ~110 (bypasses xet) |\n| FUSE mount | ~22 (avoid) | — |\n\nOn the default transport bucketbag is **competitive, not faster** — the win is bounded disk + cleanup +\nresume + overlap-with-compute at ~no cost. The one real throughput lever is xet's concurrent-file cap\n(default 8): `boost()` raises it for **~2.5× on small files** — a single env var, **no Rust**. Skip\n`boost()` for *large* files (it would over-subscribe). `HF_XET_HIGH_PERFORMANCE=1` is on by default (opt\nout `BUCKETBAG_NO_XET_TUNE=1`); for cross-stage re-reads, enable `HF_XET_CHUNK_CACHE_SIZE_BYTES`.\n\n## Scope\n\nIntentionally small and **pure Python**. Out of scope (for now): Jobs fan-out — would be a separate thin\n`bucketbag.jobs` module on top of these helpers. Possible future: adaptive \"auto batch size\".\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavanstrien%2Fbucketbag","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavanstrien%2Fbucketbag","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavanstrien%2Fbucketbag/lists"}