https://github.com/donkomura/fsspec-chfs
fsspec implementations for CHFS
https://github.com/donkomura/fsspec-chfs
dask fsspec high-performance-computing python
Last synced: 4 months ago
JSON representation
fsspec implementations for CHFS
- Host: GitHub
- URL: https://github.com/donkomura/fsspec-chfs
- Owner: donkomura
- License: mit
- Created: 2022-08-31T04:10:10.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2022-09-07T05:16:41.000Z (over 3 years ago)
- Last Synced: 2025-09-23T01:55:52.059Z (8 months ago)
- Topics: dask, fsspec, high-performance-computing, python
- Language: Python
- Homepage:
- Size: 19.5 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# fsspec-chfs
fsspec-chfs is a file system interface to CHFS.
CHFS is a parallel consistent hashing file system created instantly using node-local storages such as persistent memory and NVMe SSD for high performance computing.
This repository includes the integration for Dask.
# Requirements
* [CHFS](https://github.com/otatebe/chfs)
* [PyCHFS](https://github.com/donkomura/PyCHFS.git)
# Getting Started
## Installation
```
$ pip install fsspec-chfs
```
## Create file system
You can create CHFS by `chfsctl` and set `CHFS_SERVER` environmental variable.
```
$ eval `chfsctl start`
$ chlist # show started servers
```
## How to use fsspec-chfs
```python
import fsspec
fs = fsspec.filesystem('chfs')
with fs.open('/hello') as f:
f.write(b'world')
```
### Use in Dask
fsspec-chfs provides `CHFSClientDaemon` plugin for Dask worker, and it optimizes CHFS initialization/termination in Dask.
```python
client = Client(LocalCluster())
plugin = CHFSClientDaemon()
client.register_worker_plugin(plugin)
def func(path, data):
fs = fsspec.filesystem("chfs_stub")
fs.pipe(path, data)
return 0
future = client.submit(func, "/tmp/foo", b'abcde')
counts = future.result()
```
# Developing
## VSCode devcontainer
You can use VSCode devcontainer to develop fsspec-chfs.
The setup steps are follows:
1. Install Docker and Remote-Container extension.
2. Press Ctrl+Shift+P in VSCode.
3. select `Remote-Container: Open the folder in the Container`
## Testing
```
$ eval `chfsctl start` # start the server and set CHFS_SERVER
$ tox
```