{"id":32692896,"url":"https://github.com/trissim/polystore","last_synced_at":"2026-04-12T18:03:21.058Z","repository":{"id":321866220,"uuid":"1087449883","full_name":"trissim/polystore","owner":"trissim","description":"Framework-agnostic multi-backend storage abstraction for ML and scientific computing","archived":false,"fork":false,"pushed_at":"2025-11-01T01:30:29.000Z","size":2676,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-01T02:29:40.646Z","etag":null,"topics":["backend","data","io","jax","ml","multi-framework","numpy","pytorch","scientific-computing","storage","tensorflow","zarr"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trissim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-31T23:52:04.000Z","updated_at":"2025-10-31T23:52:27.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/trissim/polystore","commit_stats":null,"previous_names":["trissim/polystore"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/trissim/polystore","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trissim%2Fpolystore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trissim%2Fpolystore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trissim%2Fpolystore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trissim%2Fpolystore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trissim","download_url":"https://codeload.github.com/trissim/polystore/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trissim%2Fpolystore/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":282166178,"owners_count":26625192,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-01T02:00:06.759Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend","data","io","jax","ml","multi-framework","numpy","pytorch","scientific-computing","storage","tensorflow","zarr"],"created_at":"2025-11-01T16:02:07.004Z","updated_at":"2026-04-12T18:03:21.045Z","avatar_url":"https://github.com/trissim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# polystore\n\n**Framework-agnostic multi-backend storage abstraction for ML and scientific computing**\n\n[![PyPI version](https://badge.fury.io/py/polystore.svg)](https://badge.fury.io/py/polystore)\n[![Documentation Status](https://readthedocs.org/projects/polystore/badge/?version=latest)](https://polystore.readthedocs.io/en/latest/?badge=latest)\n[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Coverage](https://raw.githubusercontent.com/trissim/polystore/main/.github/badges/coverage.svg)](https://trissim.github.io/polystore/coverage/)\n\n## Features\n\n- **Pluggable Backends**: Disk, memory, Zarr, and streaming backends with auto-registration\n- **Multi-Framework I/O**: Seamless support for NumPy, PyTorch, JAX, TensorFlow, CuPy\n- **Atomic Operations**: Cross-platform atomic file writes with automatic locking\n- **Batch Operations**: Efficient batch loading and saving\n- **Format Detection**: Automatic format detection and routing\n- **Type-Safe**: Full type hints and mypy support\n- **Zero Dependencies**: Core requires only NumPy (framework support is optional)\n\n## Quick Start\n\n```python\nfrom polystore import FileManager, BackendRegistry\n\n# Create registry and file manager\nregistry = BackendRegistry()\nfm = FileManager(registry)\n\n# Save data to disk\nimport numpy as np\ndata = np.array([[1, 2], [3, 4]])\nfm.save(data, \"output.npy\", backend=\"disk\")\n\n# Load data back\nloaded = fm.load(\"output.npy\", backend=\"disk\")\n\n# Use memory backend for testing\nfm.save(data, \"test.npy\", backend=\"memory\")\ncached = fm.load(\"test.npy\", backend=\"memory\")\n```\n\n## Installation\n\n```bash\n# Base installation (NumPy only)\npip install polystore\n\n# With specific frameworks\npip install polystore[zarr]\npip install polystore[torch]\npip install polystore[jax]\npip install polystore[tensorflow]\npip install polystore[cupy]\n\n# With streaming support\npip install polystore[streaming]\n\n# With all optional dependencies\npip install polystore[all]\n```\n\n## Supported Backends\n\n| Backend | Description | Storage | Dependencies |\n|---------|-------------|---------|--------------|\n| **disk** | Local filesystem | Persistent | None |\n| **memory** | In-memory cache | Volatile | None |\n| **zarr** | Zarr/OME-Zarr arrays | Persistent | zarr, ome-zarr |\n| **streaming** | ZeroMQ streaming | None | pyzmq |\n\n## Supported Formats\n\n| Format | Extensions | Frameworks |\n|--------|-----------|------------|\n| **NumPy** | `.npy`, `.npz` | NumPy, PyTorch, JAX, TensorFlow, CuPy |\n| **TIFF** | `.tif`, `.tiff` | NumPy, PyTorch, JAX, TensorFlow, CuPy |\n| **Zarr** | `.zarr` | NumPy, PyTorch, JAX, TensorFlow, CuPy |\n| **PyTorch** | `.pt`, `.pth` | PyTorch |\n| **CSV** | `.csv` | NumPy, pandas |\n| **JSON** | `.json` | Python dicts |\n\n## Architecture\n\n```\npolystore/\n├── base.py              # Abstract interfaces (DataSink, DataSource, StorageBackend)\n├── backend_registry.py  # Auto-registration system\n├── disk.py              # Disk storage backend\n├── memory.py            # In-memory backend\n├── zarr.py              # Zarr backend\n├── streaming.py         # ZeroMQ streaming backend\n├── filemanager.py       # High-level API\n├── atomic.py            # Atomic file operations\n└── exceptions.py        # Custom exceptions\n```\n\n## Advanced Usage\n\n### Custom Backends\n\n```python\nfrom polystore import StorageBackend\n\nclass MyBackend(StorageBackend):\n    _backend_type = 'my_backend'  # Auto-registers\n    \n    def save(self, data, file_path, **kwargs):\n        # Your save logic\n        pass\n    \n    def load(self, file_path, **kwargs):\n        # Your load logic\n        pass\n```\n\n### Batch Operations\n\n```python\n# Save multiple files\ndata_list = [np.random.rand(100, 100) for _ in range(10)]\npaths = [f\"image_{i}.npy\" for i in range(10)]\nfm.save_batch(data_list, paths, backend=\"disk\")\n\n# Load multiple files\nloaded_list = fm.load_batch(paths, backend=\"disk\")\n```\n\n### Atomic Writes\n\n```python\nfrom polystore import atomic_write, atomic_write_json\n\n# Atomic file write with automatic locking\nwith atomic_write(\"output.txt\") as f:\n    f.write(\"data\")\n\n# Atomic JSON write\natomic_write_json({\"key\": \"value\"}, \"config.json\")\n```\n\n## Why polystore?\n\n**Before** (Manual backend management):\n```python\nif backend == 'disk':\n    np.save(path, data)\nelif backend == 'memory':\n    cache[path] = data\nelif backend == 'zarr':\n    zarr.save(path, data)\n# ... 50 more lines of if/elif ...\n```\n\n**After** (polystore):\n```python\nfm.save(data, path, backend=backend)\n```\n\n## Documentation\n\nFull documentation available at [polystore.readthedocs.io](https://polystore.readthedocs.io)\n\n## Addons\n\nExtend polystore with additional backends:\n\n- **polystore-napari**: Napari viewer streaming backend\n- **polystore-fiji**: Fiji/ImageJ streaming backend\n- **polystore-omero**: OMERO server backend\n\n## Performance\n\n- **Zero-copy** conversions between frameworks via DLPack (when possible)\n- **Lazy loading** for optional dependencies\n- **Batch operations** for efficient I/O\n- **Atomic writes** with minimal overhead\n\n## License\n\nMIT License - see LICENSE file for details\n\n## Contributing\n\nContributions welcome! Please see CONTRIBUTING.md for guidelines.\n\n## Credits\n\nDeveloped by Tristan Simas. Extracted from the [OpenHCS](https://github.com/trissim/openhcs) project.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrissim%2Fpolystore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrissim%2Fpolystore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrissim%2Fpolystore/lists"}