{"id":21843993,"url":"https://github.com/altescy/minato","last_synced_at":"2025-04-14T12:06:53.413Z","repository":{"id":39884948,"uuid":"365624268","full_name":"altescy/minato","owner":"altescy","description":"⚓ Minato: A Unified File I/O Library for Python","archived":false,"fork":false,"pushed_at":"2024-12-06T12:02:43.000Z","size":679,"stargazers_count":13,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-14T12:06:45.447Z","etag":null,"topics":["cache","cli","filesystem","google-cloud-storage","http","huggingface-hub","python","s3"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/minato/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/altescy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-08T22:41:51.000Z","updated_at":"2024-12-06T12:01:17.000Z","dependencies_parsed_at":"2024-12-06T09:22:19.417Z","dependency_job_id":"7937a28d-d879-43e6-a797-3cb42c2938c7","html_url":"https://github.com/altescy/minato","commit_stats":{"total_commits":225,"total_committers":2,"mean_commits":112.5,"dds":0.008888888888888835,"last_synced_commit":"48f07a6da5d676198b93e2cdb3ebf2990aa27e33"},"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/altescy%2Fminato","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/altescy%2Fminato/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/altescy%2Fminato/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/altescy%2Fminato/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/altescy","download_url":"https://codeload.github.com/altescy/minato/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248877983,"owners_count":21176243,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","cli","filesystem","google-cloud-storage","http","huggingface-hub","python","s3"],"created_at":"2024-11-27T22:18:05.151Z","updated_at":"2025-04-14T12:06:53.388Z","avatar_url":"https://github.com/altescy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ⚓ Minato\n\n[![Actions Status](https://github.com/altescy/minato/workflows/CI/badge.svg)](https://github.com/altescy/minato/actions/workflows/ci.yml)\n[![Python version](https://img.shields.io/pypi/pyversions/minato)](https://github.com/altescy/minato)\n[![License](https://img.shields.io/github/license/altescy/minato)](https://github.com/altescy/minato/blob/master/LICENSE)\n[![pypi version](https://img.shields.io/pypi/v/minato)](https://pypi.org/project/minato/)\n\nA Unified File I/O Library for Python\n\n\nMinato is a Python library that provides a unified and simple interface to work with local and remote files, as well as compressed and archived files.\nWith Minato, you can seamlessly read and write files from various sources like local filesystem, HTTP(S), Amazon S3, Google Cloud Storage, and Hugging Fase Hub.\nIt also supports reading and writing compressed files such as gzip, bz2, and lzma, as well as directly accessing files inside archives like zip and tar.\n\nOne of Minato's key features is its built-in caching mechanism, which allows you to cache remote files locally, and manage the cache with a provided CLI.\nThe cache is automatically updated based on ETag headers, ensuring that you always work with the latest version of the files.\n\n## Features\n\n- Unified file I/O for local and remote files (HTTP(S), S3, GCP, Hugging Face Hub)\n- Support for reading and writing compressed files (gzip, bz2, lzma)\n- Direct access to files inside archives (zip, tar)\n- Local caching of remote files with cache management CLI\n- Automatic cache updates based on ETag headers\n\n## Installation\n\nInstall Minato using pip:\n\n```bash\npip install minato                   # minimal installation for only local/http(s) file I/O\npip install minato[s3]               # for Amazon S3\npip install minato[gcs]              # for Google Cloud Storage\npip install minato[huggingface-hub]  # for Hugging Face Hub\npip install minato[all]              # for all supported file I/O\n```\n\n## Usage\n\n### Quick Start\n\nHere's a simple example demonstrating how to read and write files on online storage:\n\n```python\nimport minato\n\n# Write a file to an S3 bucket\ns3_path = \"s3://your_bucket/path/to/file\"\nwith minato.open(s3_path, \"w\") as f:\n    f.write(\"Create a new file on AWS S3!\")\n```\n\nAccess cached online resources in local storage:\n\n```python\n# Cache a remote file and get its local path\nremote_path = \"http://example.com/path/to/archive.zip!inner/path/to/file\"\nlocal_filename = minato.cached_path(remote_path)\n```\n\nAccess files inside archives like zip by connecting the archive path and inner file path with an exclamation mark (`!`) like above.\n\nAutomatically decompress files with gzip / lzma / bz2 compression:\n\n```python\nwith minato.open(\"data.txt.gz\", \"rt\", decompress=True) as f:\n    content = f.read()\n```\n\nIn the example above, Minato will automatically detect the file format based on the file's content or filename and decompress the file accordingly.\n\n### Cache Management\n\n```bash\n❯ poetry run minato --help\nusage: minato\n\npositional arguments:\n  {cache,list,remove,update}\n    cache               cache remote file and return cached local file path\n    list                show list of cached files\n    remove              remove cached files\n    update              update cached files\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --version             show program's version number and exit\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faltescy%2Fminato","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faltescy%2Fminato","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faltescy%2Fminato/lists"}