{"id":37070965,"url":"https://github.com/foundation-model-stack/fastsafetensors","last_synced_at":"2026-02-10T04:01:23.331Z","repository":{"id":241135969,"uuid":"804120523","full_name":"foundation-model-stack/fastsafetensors","owner":"foundation-model-stack","description":"High-performance safetensors model loader","archived":false,"fork":false,"pushed_at":"2026-02-09T07:33:50.000Z","size":1668,"stargazers_count":101,"open_issues_count":2,"forks_count":18,"subscribers_count":9,"default_branch":"main","last_synced_at":"2026-02-09T11:37:36.685Z","etag":null,"topics":["python","pytorch","safetensors"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/foundation-model-stack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"code-of-conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-05-22T01:57:20.000Z","updated_at":"2026-02-09T08:54:03.000Z","dependencies_parsed_at":"2024-06-11T08:42:51.774Z","dependency_job_id":"e2e23e69-e7d4-4aa8-8e60-b0fc7d748bd0","html_url":"https://github.com/foundation-model-stack/fastsafetensors","commit_stats":null,"previous_names":["foundation-model-stack/fastsafetensors"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/foundation-model-stack/fastsafetensors","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foundation-model-stack%2Ffastsafetensors","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foundation-model-stack%2Ffastsafetensors/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foundation-model-stack%2Ffastsafetensors/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foundation-model-stack%2Ffastsafetensors/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/foundation-model-stack","download_url":"https://codeload.github.com/foundation-model-stack/fastsafetensors/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foundation-model-stack%2Ffastsafetensors/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29290460,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-10T03:42:42.660Z","status":"ssl_error","status_checked_at":"2026-02-10T03:42:41.897Z","response_time":65,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","pytorch","safetensors"],"created_at":"2026-01-14T08:17:18.512Z","updated_at":"2026-02-10T04:01:23.325Z","avatar_url":"https://github.com/foundation-model-stack.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"fastsafetensors is an efficient safetensors model loader.\nThis library is tested with python 3.9-13 and pytorch 2.1-2.7.\n\nDisclaimer: This repository contains a research prototype. It should be used with caution.\n\n# Features\n\nWe introduced three major features to optimize model loading performance:\n1. Batched, lazy tensor instantiations\n2. GPU offloading for sharding, type conversions, and device pointer alignment.\n3. GPU Direct Storage enablement for file loading from storage to GPU memory\n\nA major design difference from the original safetensors file loader is *NOT* to use `mmap`.\nIt loads tensors on-demand with mmap'ed files,\nbut unfortunately, it cannot fully utilize high-throughput I/O such as NVMe SSDs.\nSo, we asynchronously transfer files in parallel to saturate storage throughput.\nThen, fastsafetensors lazily instantiates tensors at GPU device memory with DLPack.\n\nAnother design change is to offload sharding and other manipulations on tensors to GPUs.\nThe original loader provides slicing for sharding at user programs before copying to device memory. However, it incurrs high CPU usages for host memory accesses.\nSo, we introduce a special APIs to run sharding with `torch.distributed` collective operations such as `broadcast` and `scatter`.\nThe offloading is also applied to other tensor manipulations such as type conversions.\n\nThe above two design can be naturally extended to utilize device-to-device data transfers with GPU Direct Storage.\nThe technology helps to minimize copy overheads from NVMe SSDs to GPU memory with host CPU and memory bypassed.\n\n## Basic API usages\n\n`SafeTensorsFileLoader` is a low-level entrypoint. To use it, pass either `SingleGroup()` for simple inference or `ProcessGroup()` (from `torch.distributed`) for tensor-parallel inference. The loader supports both CPU and CUDA devices, with optional GPU Direct Storage (GDS) support. You can specify the device and GDS settings using the `device` and `nogds` arguments, respectively. Note that if GDS is not available, the loader will fail to open files when `nogds=False`. For more information on enabling GDS, please refer to the NVIDIA documentation.\n\nAfter creating a `SafeTensorsFileLoader` instance, first map target files and a rank using the `.add_filenames()` method. Then, call `.copy_file_to_device()` to trigger the actual file copies on aggregated GPU memory fragments and directly instantiate a group of Tensors. Once the files are loaded, you can retrieve a tensor using the `.get_tensor()` method. Additionally, you can obtain sharded tensors by `.get_sharded()`, which internally run collective operations in `torch.distributed`.\n\nImportant: To release the GPU memory allocated for tensors, you must explicitly call the `.close()` method. This is because Fastsafetensors allows multiple tensors to share a limited number of GPU memory fragments. As a result, it is the user's responsibility to ensure that all tensors are properly released before calling `.close()`, which will then safely release the underlying GPU memory.\n\n`fastsafe_open` is an easier entrypoint. You can force turning off GDS and run in the fallback mode if `nogds==True`. However, users must be aware of the above tricky memory management model, which should be fixed in future releases.\n\n```python\nwith fastsafe_open(filenames=[filename], nogds=True, device=\"cpu\", debug_log=True) as f:\n    for key in f.get_keys():\n        t = f.get_tensor(key).clone().detach() # clone if t is used outside\n```\n\n## Development\n\n### Pre-commit Hooks\n\nThis repository uses pre-commit hooks for automatic code formatting and linting. To set up:\n\n1. Install development dependencies:\n```bash\npip install -e \".[dev]\"\n```\n\n2. Install pre-commit hooks:\n```bash\npre-commit install\n```\n\nNow, every time you commit, the following checks will run automatically:\n- **black**: Code formatting\n- **isort**: Import sorting\n- **flake8**: Basic linting (syntax errors, undefined names)\n- **mypy**: Type checking\n- **trailing-whitespace**: Remove trailing whitespace\n- **end-of-file-fixer**: Ensure files end with a newline\n- **check-yaml**: Validate YAML files\n- **check-toml**: Validate TOML files\n- **check-merge-conflict**: Detect merge conflict markers\n- **debug-statements**: Detect debug statements\n\nTo manually run pre-commit on all files:\n```bash\npre-commit run --all-files\n```\n\nTo skip pre-commit hooks (not recommended):\n```bash\ngit commit --no-verify\n```\n\n## Code of Conduct\n\nPlease refer to [Foundation Model Stack Community Code of Conduct](https://github.com/foundation-model-stack/foundation-model-stack/blob/main/code-of-conduct.md).\n\n## Publication\n\nTakeshi Yoshimura, Tatsuhiro Chiba, Manish Sethi, Daniel Waddington, Swaminathan Sundararaman. (2025) Speeding up Model Loading with fastsafetensors [arXiv:2505.23072](https://arxiv.org/abs/2505.23072) and IEEE CLOUD 2025.\n\n## For NVIDIA\n\n### Install from PyPI\n\nSee https://pypi.org/project/fastsafetensors/\n\n```bash\npip install fastsafetensors\n```\n\n### Install from source\n\n```bash\npip install .\n```\n\n## For ROCm\n\nOn ROCm, there are not GDS equivalent support. So fastsafetensors support only supports `nogds=True` mode.\nThe performance gain example can be found at [amd-perf.md](./docs/amd-perf.md)\n\n### Install from Github Source\n\n```bash\nROCM_PATH=/opt/rocm pip install git+https://github.com/foundation-model-stack/fastsafetensors.git\n```\n\n### Install from source\n\n```bash\nROCM_PATH=/opt/rocm pip install .\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffoundation-model-stack%2Ffastsafetensors","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffoundation-model-stack%2Ffastsafetensors","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffoundation-model-stack%2Ffastsafetensors/lists"}