{"id":41918957,"url":"https://github.com/donkomura/fsspec-chfs","last_synced_at":"2026-01-25T16:36:15.203Z","repository":{"id":237399918,"uuid":"530930878","full_name":"donkomura/fsspec-chfs","owner":"donkomura","description":"fsspec implementations for CHFS","archived":false,"fork":false,"pushed_at":"2022-09-07T05:16:41.000Z","size":20,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-23T01:55:52.059Z","etag":null,"topics":["dask","fsspec","high-performance-computing","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/donkomura.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-31T04:10:10.000Z","updated_at":"2024-05-01T06:46:16.000Z","dependencies_parsed_at":"2024-05-01T08:50:44.285Z","dependency_job_id":"7ce0e447-e807-4d00-ad04-f269a945a19d","html_url":"https://github.com/donkomura/fsspec-chfs","commit_stats":null,"previous_names":["donkomura/fsspec-chfs"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/donkomura/fsspec-chfs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donkomura%2Ffsspec-chfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donkomura%2Ffsspec-chfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donkomura%2Ffsspec-chfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donkomura%2Ffsspec-chfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/donkomura","download_url":"https://codeload.github.com/donkomura/fsspec-chfs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donkomura%2Ffsspec-chfs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28755558,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-25T16:32:25.380Z","status":"ssl_error","status_checked_at":"2026-01-25T16:32:09.189Z","response_time":113,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dask","fsspec","high-performance-computing","python"],"created_at":"2026-01-25T16:36:14.616Z","updated_at":"2026-01-25T16:36:15.197Z","avatar_url":"https://github.com/donkomura.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fsspec-chfs\n\nfsspec-chfs is a file system interface to CHFS.\nCHFS is a parallel consistent hashing file system created instantly using node-local storages such as persistent memory and NVMe SSD for high performance computing.\nThis repository includes the integration for Dask.\n\n# Requirements\n\n* [CHFS](https://github.com/otatebe/chfs)\n* [PyCHFS](https://github.com/donkomura/PyCHFS.git)\n\n# Getting Started\n\n## Installation\n\n```\n$ pip install fsspec-chfs\n```\n\n## Create file system\n\nYou can create CHFS by `chfsctl` and set `CHFS_SERVER` environmental variable.\n\n```\n$ eval `chfsctl start`\n$ chlist # show started servers\n```\n\n## How to use fsspec-chfs\n\n```python\nimport fsspec\n\nfs = fsspec.filesystem('chfs')\n\nwith fs.open('/hello') as f:\n\tf.write(b'world')\n```\n\n### Use in Dask\n\nfsspec-chfs provides `CHFSClientDaemon` plugin for Dask worker, and it optimizes CHFS initialization/termination in Dask.\n\n```python\nclient = Client(LocalCluster())\nplugin = CHFSClientDaemon()\nclient.register_worker_plugin(plugin)\ndef func(path, data):\n\tfs = fsspec.filesystem(\"chfs_stub\")\n\tfs.pipe(path, data)\n\treturn 0\nfuture = client.submit(func, \"/tmp/foo\", b'abcde')\ncounts = future.result()\n```\n\n# Developing\n\n## VSCode devcontainer\n\nYou can use VSCode devcontainer to develop fsspec-chfs.\nThe setup steps are follows:\n\n1. Install Docker and Remote-Container extension.\n2. Press Ctrl+Shift+P in VSCode.\n3. select `Remote-Container: Open the folder in the Container`\n\n## Testing\n\n```\n$ eval `chfsctl start` # start the server and set CHFS_SERVER\n$ tox\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdonkomura%2Ffsspec-chfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdonkomura%2Ffsspec-chfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdonkomura%2Ffsspec-chfs/lists"}