{"id":37066446,"url":"https://github.com/terrafloww/rasteret","last_synced_at":"2026-04-11T07:10:50.351Z","repository":{"id":271154503,"uuid":"902385710","full_name":"terrafloww/rasteret","owner":"terrafloww","description":"Rasteret is a library for 20x+ faster reads of GeoTIFF than Rasterio/GDAL. Interops with TorchGeo, Xarray, DuckDB, Polars","archived":false,"fork":false,"pushed_at":"2026-04-09T19:57:55.000Z","size":4414,"stargazers_count":118,"open_issues_count":2,"forks_count":5,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-09T21:33:40.339Z","etag":null,"topics":["cache","cloud-optimized","geoparquet","geospatial","geotiff","raster","remote-sensing","satellite-imagery","satellite-imagery-pipeline"],"latest_commit_sha":null,"homepage":"https://terrafloww.github.io/rasteret/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/terrafloww.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":"codemeta.json","zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-12T13:15:48.000Z","updated_at":"2026-04-09T19:57:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"33fa8630-29bb-4fa2-85cb-8fc9b90db894","html_url":"https://github.com/terrafloww/rasteret","commit_stats":null,"previous_names":["terrafloww/rasteret"],"tags_count":25,"template":false,"template_full_name":null,"purl":"pkg:github/terrafloww/rasteret","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terrafloww%2Frasteret","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terrafloww%2Frasteret/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terrafloww%2Frasteret/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terrafloww%2Frasteret/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/terrafloww","download_url":"https://codeload.github.com/terrafloww/rasteret/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terrafloww%2Frasteret/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31671879,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-10T17:19:37.612Z","status":"online","status_checked_at":"2026-04-11T02:00:05.776Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","cloud-optimized","geoparquet","geospatial","geotiff","raster","remote-sensing","satellite-imagery","satellite-imagery-pipeline"],"created_at":"2026-01-14T07:48:03.692Z","updated_at":"2026-04-11T07:10:50.345Z","avatar_url":"https://github.com/terrafloww.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e🛰️ Rasteret\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cstrong\u003eThe AI practitioner's multiplier for cloud-native satellite data.\u003c/strong\u003e\u003cbr\u003e\n  \u003cem\u003eA high-performance rasterio/GDAL alternative for scaleable ML workflows.\u003c/em\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\nRasteret helps you manage and read massive satellite imagery collections with zero friction. \u003cbr\u003e\nIt provides a high-performance \"drop-in\" backend for **TorchGeo**, **xarray**, and **NumPy** that is up to 20x faster than traditional GDAL-based workflows.\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://terrafloww.github.io/rasteret\"\u003e\u003cimg src=\"https://img.shields.io/badge/docs-terrafloww.github.io%2Frasteret-009DD1\" alt=\"Documentation\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://discord.gg/86NgTB3Xa\"\u003e\u003cimg src=\"https://img.shields.io/badge/Discord-chat-5865F2?logo=discord\u0026logoColor=white\" alt=\"Discord\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/rasteret/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/rasteret?color=blue\" alt=\"PyPI\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/rasteret/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/pyversions/rasteret\" alt=\"Python\"\u003e\u003c/a\u003e\n  \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-Apache--2.0-blue\" alt=\"License\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n## Why Rasteret?\n\nGeospatial data science is often 80% \"plumbing.\" You spend hours writing `pystac-client` loops, manual `ThreadPoolExecutor` code, and fragile CRS-alignment logic just to get a batch of pixels for your model.\n\n**Rasteret turns those 80% into a single line of code.**\n\nIt separates the **Control Plane** (managing your scenes, labels, and splits in a local Parquet index) from the **Data Plane** (streaming pixels directly from cloud COGs).\n\n### The \"Friction\" vs. \"Flow\" Comparison\n\n**The Old Way (25+ lines of fragile plumbing)**:\n1. Search STAC catalog ✅\n2. Loop over items ✅\n3. Handle pagination ✅\n4. Filter by cloud cover ✅\n5. **Wait 500ms per file** to parse remote TIFF headers (GDAL cold start) ❌\n6. Manage `ThreadPoolExecutor` manually ❌\n7. Manually stack results and align CRS ❌\n\n**The Rasteret Way (3 lines of robust code)**:\n```python\nimport rasteret\n\n# 1. Load or Build your collection (Index is local, metadata is relational)\ncollection = rasteret.load(\"my_s2_experiment\")\n\n# 2. Query like a Table: \"Give me the training scenes with \u003c10% clouds\"\nfiltered = collection.subset(split=\"train\", cloud_cover_lt=10)\n\n# 3. Batch Read: \"Fetch aligned pixels for these 1000 polygons\"\ndata = filtered.get_numpy(geometries=my_polygons, bands=[\"B04\", \"B08\"])\n```\n\n---\n\n## Key Features\n\n- **🚀 20x Faster Cold Starts**: By caching tile-layout metadata locally, Rasteret jumps straight to the pixels, skipping expensive remote header parsing, which happens in every new environment.\n- **📦 Seamless \"Drop-in\" Backends**: Boost **TorchGeo** or **xarray** performance by simply swapping the reader. No need to rewrite your training code.\n- **🧬 Relational Imagery**: Store your labels, `train/val/test` splits, and custom metadata directly in the imagery index. No more separate CSVs.\n- **🛠️ Zero-Config Throughput**: Automatic cloud storage presigning with `Obstore`, and custom async I/O handles the networking so you don't have to.\n\n## Performance\n\nRasteret's claims are backed by rigorous, reproducible benchmarks. We measure across three dimensions: cold-start latency, cloud-native scale, and comparison against legacy \"data-inside-parquet\" patterns.\n\n### 1. Cold-start comparison with TorchGeo\nSame AOIs, same scenes, same sampler, same DataLoader. Rasteret eliminates the \"cold start tax\" by caching IFD headers in the local Parquet index.\n\n| Scenario | rasterio/GDAL (Standard) | Rasteret (Index-First) | Speedup |\n|---|---|---|---|\n| Single AOI, 15 scenes | 9.08 s | 1.14 s | **8x** |\n| Multi-AOI, 30 scenes | 42.05 s | 2.25 s | **19x** |\n| Cross-CRS boundary | 12.47 s | 0.59 s | **21x** |\n\n![Processing time comparison](./assets/benchmark_results.png)\n![Speedup breakdown](./assets/benchmark_breakdown.png)\n\n### 2. The Cloud vs. Edge Comparison\nHow does Rasteret stack up against **Google Earth Engine (GEE)** or a highly parallelized Rasterio setup for time-series extraction?\n\n| Library | First Run (Cold) | Subsequent Runs (Hot) |\n|---------|-----------------|-----------------------|\n| **Rasterio** + ThreadPool | 32 s | 24 s |\n| **Google Earth Engine** | 10–30 s | 3–5 s |\n| **Rasteret** | **3 s** | **3 s** |\n\n![Single request performance](./assets/single_timeseries_request.png)\n\n### 3. HuggingFace `MajorTOM` vs. Rasteret\nRecent \"images-inside-Parquet\" approaches (like MajorTOM) try to store image bytes in Parquet files. Rasteret keeps imagery in cloud COGs while using Parquet as a high-performance index—delivering better throughput without the data movement overhead.\n\n| Patches | HF `datasets` (streaming) | Rasteret index+COGs | Speedup |\n|---:|---:|---:|---:|\n| 120 | 46.83 s | 12.09 s | **3.88x** |\n| 1000 | 771.59 s | 118.69 s | **6.50x** |\n\n![HF vs Rasteret speedup](./assets/benchmark_hf_speedup.png)\n\n*All numbers measured on AWS us-west-2 4CPU machine (same region as data) vs. cold-start GDAL.*\n\n---\n\n## Technical Deep Dives\n\nFor the full architectural rationale, methodology, and reproducibility scripts, see:\n\n- [**Full Benchmarks Guide**](https://terrafloww.github.io/rasteret/explanation/benchmark/): Methodology and results.\n- [**Design Decisions**](https://terrafloww.github.io/rasteret/explanation/design-decisions.md): Why we chose Parquet + COGs\n- [**Schema Contract**](https://terrafloww.github.io/rasteret/explanation/schema-contract/): The internal anatomy of a Collection.\n\n```text\nSTAC API / GeoParquet  --\u003e  Parquet Collection  --\u003e  Tile-level byte reads\n       (once)                  (queryable)             (no GDAL hot path)\n```\n\n## Quick Start\n\n### 1. Build a Collection\n```python\nimport rasteret\n\n# Build from any STAC API or Parquet Metadata table\ncollection = rasteret.build(\n    \"earthsearch/sentinel-2-l2a\",\n    name=\"s2_training\",\n    bbox=(77.5, 12.9, 77.7, 13.1),\n    date_range=(\"2024-01-01\", \"2024-06-30\")\n)\n```\n\n### 2. Turbocharge your ML (TorchGeo)\nRasteret provides a high-performance backend that honors the `GeoDataset` contract.\n\n```python\nfrom torch.utils.data import DataLoader\nfrom torchgeo.samplers import RandomGeoSampler\n\n# Same API as TorchGeo, much faster pixel pipe\ndataset = collection.to_torchgeo_dataset(bands=[\"B04\", \"B08\"], chip_size=256)\n\nsampler = RandomGeoSampler(dataset, size=256, length=100)\nloader  = DataLoader(dataset, sampler=sampler, batch_size=4)\n```\n\n### 3. Fast Xarray creation\n```python\nds = collection.get_xarray(geometries=my_aoi, bands=[\"B04\", \"B08\"])\nndvi = (ds.B08 - ds.B04) / (ds.B08 + ds.B04)\n```\n\n## Key Entry Points\n\nRasteret is built for flexibility. Choose the output format that fits your existing workflow:\n\n| Method | Output | Purpose |\n|---|---|---|\n| [**`to_torchgeo_dataset()`**](https://terrafloww.github.io/rasteret/reference/integrations/torchgeo/) | `RasteretGeoDataset` | Drop-in high-performance backend for **TorchGeo** training. |\n| [**`get_xarray()`**](https://terrafloww.github.io/rasteret/reference/core/collection/#rasteret.core.collection.Collection.get_xarray) | `xarray.Dataset` | Quick create Xarray for analysis. |\n| [**`get_numpy()`**](https://terrafloww.github.io/rasteret/reference/core/collection/#rasteret.core.collection.Collection.get_numpy) | `numpy.ndarray` | Raw pixel arrays (`[N, C, H, W]`) directly. |\n| [**`get_gdf()`**](https://terrafloww.github.io/rasteret/reference/core/collection/#rasteret.core.collection.Collection.get_gdf) | `GeoDataFrame` | Metadata and pixel arrays as a standard geopandas dataframe. |\n| [**`sample_points()`**](https://terrafloww.github.io/rasteret/reference/core/collection/#rasteret.core.collection.Collection.sample_points) | `DataFrame` | Exact pixel values at points geometries with intuitive configurable fallback for nodata pixels |\n\n---\n\nFull documentation at **[terrafloww.github.io/rasteret](https://terrafloww.github.io/rasteret)**:\n\n- [**Conceptual Roadmap**](https://terrafloww.github.io/rasteret/explanation/conceptual-roadmap/): Why Rasteret?\n- [**Transitioning from Rasterio**](https://terrafloww.github.io/rasteret/how-to/transitioning-from-rasterio/): Side-by-side patterns.\n- [**Turbocharging TorchGeo**](https://terrafloww.github.io/rasteret/how-to/turbocharging-torchgeo/): Scaling your DL loaders.\n- [**Tutorials**](https://terrafloww.github.io/rasteret/tutorials/): Hands-on examples.\n\n## License\n\nCode: [Apache-2.0](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fterrafloww%2Frasteret","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fterrafloww%2Frasteret","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fterrafloww%2Frasteret/lists"}