{"id":18084177,"url":"https://github.com/developmentseed/obstore","last_synced_at":"2025-05-15T02:08:01.273Z","repository":{"id":258690547,"uuid":"874419862","full_name":"developmentseed/obstore","owner":"developmentseed","description":"The simplest, highest-throughput Python interface to S3, GCS \u0026 Azure Storage, powered by Rust.","archived":false,"fork":false,"pushed_at":"2025-05-09T17:19:15.000Z","size":15107,"stargazers_count":344,"open_issues_count":46,"forks_count":14,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-05-09T18:24:52.855Z","etag":null,"topics":["amazon-s3","azure","azure-storage","cloudflare-r2","google-cloud-storage","object-store","python","rust","s3"],"latest_commit_sha":null,"homepage":"https://developmentseed.org/obstore","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/developmentseed.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-17T19:46:34.000Z","updated_at":"2025-05-09T17:24:08.000Z","dependencies_parsed_at":"2025-02-10T07:03:28.334Z","dependency_job_id":"d29f6854-61eb-41b4-b526-3982950f88c9","html_url":"https://github.com/developmentseed/obstore","commit_stats":null,"previous_names":["developmentseed/object-store-rs","developmentseed/object-store-py","developmentseed/obstore"],"tags_count":32,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developmentseed%2Fobstore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developmentseed%2Fobstore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developmentseed%2Fobstore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developmentseed%2Fobstore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/developmentseed","download_url":"https://codeload.github.com/developmentseed/obstore/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254259383,"owners_count":22040820,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-s3","azure","azure-storage","cloudflare-r2","google-cloud-storage","object-store","python","rust","s3"],"created_at":"2024-10-31T15:05:55.734Z","updated_at":"2025-05-15T02:07:56.263Z","avatar_url":"https://github.com/developmentseed.png","language":"Python","readme":"# obstore\n\n[![PyPI][pypi_badge]][pypi_link]\n[![Conda Version][conda_version_badge]][conda_version]\n[![PyPI - Downloads][pypi-img]][pypi-link]\n\n[pypi_badge]: https://badge.fury.io/py/obstore.svg\n[pypi_link]: https://pypi.org/project/obstore/\n[conda_version_badge]: https://img.shields.io/conda/vn/conda-forge/obstore.svg\n[conda_version]: https://prefix.dev/channels/conda-forge/packages/obstore\n[pypi-img]: https://img.shields.io/pypi/dm/obstore\n[pypi-link]: https://pypi.org/project/obstore/\n\nThe simplest, highest-throughput [^1] Python interface to [S3][s3], [GCS][gcs], [Azure Storage][azure_storage], \u0026 other S3-compliant APIs, powered by Rust.\n\n[s3]: https://aws.amazon.com/s3/\n[gcs]: https://cloud.google.com/storage\n[azure_storage]: https://learn.microsoft.com/en-us/azure/storage/common/storage-introduction\n\n- Sync and async API with **full type hinting**.\n- **Streaming downloads** with configurable chunking.\n- **Streaming uploads** from files or async or sync iterators.\n- **Streaming list**, with no need to paginate.\n- Automatic [**multipart uploads**](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) for large file objects.\n- Automatic **credential refresh** before expiration.\n- File-like object API and [fsspec](https://github.com/fsspec/filesystem_spec) integration.\n- Easy to install with **no required Python dependencies**.\n- Optionally return list results in [Apache Arrow](https://arrow.apache.org/) format, which is faster and more memory-efficient than materializing Python `dict`s.\n- Zero-copy data exchange between Rust and Python via the [buffer protocol](https://jakevdp.github.io/blog/2014/05/05/introduction-to-the-python-buffer-protocol/).\n\nFor Rust developers looking to add `object_store` support to their own Python packages, refer to [`pyo3-object_store`](https://docs.rs/pyo3-object_store/latest/pyo3_object_store/).\n\n[^1]: Benchmarking is ongoing, but preliminary results indicate roughly [9x higher throughput than fsspec](https://github.com/geospatial-jeff/pyasyncio-benchmark/blob/fe8f290cb3282dcc3bc96cae06ed5f90ad326eff/test_results/cog_header_results.csv) and [2.8x higher throughput than aioboto3](https://github.com/geospatial-jeff/pyasyncio-benchmark/blob/40e67509a248c5102a6b1608bcb9773295691213/test_results/20250218_results/ec2_m5/aggregated_results.csv) for many concurrent, small, get requests from an async context.\n\n## Installation\n\nTo install obstore using pip:\n\n```sh\npip install obstore\n```\n\nObstore is on [conda-forge](https://prefix.dev/channels/conda-forge/packages/obstore) and can be installed using [conda](https://docs.conda.io), [mamba](https://mamba.readthedocs.io/), or [pixi](https://pixi.sh/). To install obstore using conda:\n\n```\nconda install -c conda-forge obstore\n```\n\n## Documentation\n\n[Full documentation is available on the website](https://developmentseed.org/obstore).\n\nHead to [Getting Started](https://developmentseed.org/obstore/latest/getting-started/) to dig in.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevelopmentseed%2Fobstore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevelopmentseed%2Fobstore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevelopmentseed%2Fobstore/lists"}