https://github.com/pfnet/pfio
IO library to access various filesystems with unified API
https://github.com/pfnet/pfio
Last synced: about 1 year ago
JSON representation
IO library to access various filesystems with unified API
- Host: GitHub
- URL: https://github.com/pfnet/pfio
- Owner: pfnet
- License: mit
- Created: 2019-04-23T04:50:28.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2024-11-25T06:41:41.000Z (over 1 year ago)
- Last Synced: 2025-04-04T03:47:29.084Z (over 1 year ago)
- Language: Python
- Homepage: https://pfio.readthedocs.io/
- Size: 1.93 MB
- Stars: 52
- Watchers: 72
- Forks: 22
- Open Issues: 15
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## PFIO
PFIO is an IO abstraction library developed by PFN, optimized for deep
learning training with batteries included. It supports
- Filesystem API abstraction with unified error semantics,
- Explicit user-land caching system,
- IO performance tracing and metrics stats, and
- Fileset container utilities to save metadata.
## Dependency
- HDFS client and libhdfs for HDFS access
- CPython >= 3.9
## Installation and Document build
Installation
```shell
$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .
```
Documentation
```sh
$ cd pfio/docs
$ make html
$ open build/html/index.html
```
## How to use
Please refer to the [official document](https://pfio.readthedocs.io) for more information about the usage.
## Release
Check [the official document](https://packaging.python.org/tutorials/packaging-projects/) for latest release procedure.
Run tests locally:
```sh
$ pip install tox
$ tox
```
Bump version numbers in `pfio/version.py` .
Push and open a pull request to invoke CI. Once CI passed and the pull request merged,
tag a release:
```sh
$ git tag -s X.Y.Z
$ git push --tags
```
Build:
```sh
$ rm -rf dist
$ pip3 install --user build
$ python3 -m build
```
Release to PyPI:
```sh
$ python3 -m pip install --user --upgrade twine
$ python3 -m twine upload --repository testpypi dist/*
```