{"id":13468515,"url":"https://github.com/fsspec/filesystem_spec","last_synced_at":"2025-12-11T22:49:36.797Z","repository":{"id":37792801,"uuid":"130727736","full_name":"fsspec/filesystem_spec","owner":"fsspec","description":"A specification that python filesystems should adhere to.","archived":false,"fork":false,"pushed_at":"2025-12-04T23:58:59.000Z","size":3330,"stargazers_count":1259,"open_issues_count":330,"forks_count":422,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-12-06T05:44:00.101Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fsspec.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-04-23T16:45:08.000Z","updated_at":"2025-12-05T15:04:17.000Z","dependencies_parsed_at":"2024-01-13T21:20:32.318Z","dependency_job_id":"6e138e34-dfce-4fb8-89e3-01b01e1ef16d","html_url":"https://github.com/fsspec/filesystem_spec","commit_stats":{"total_commits":1909,"total_committers":274,"mean_commits":6.967153284671533,"dds":0.6013619696176009,"last_synced_commit":"9a161714f0bbfe44ee769f259420f2f7db975471"},"previous_names":["intake/filesystem_spec"],"tags_count":96,"template":false,"template_full_name":null,"purl":"pkg:github/fsspec/filesystem_spec","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fsspec%2Ffilesystem_spec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fsspec%2Ffilesystem_spec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fsspec%2Ffilesystem_spec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fsspec%2Ffilesystem_spec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fsspec","download_url":"https://codeload.github.com/fsspec/filesystem_spec/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fsspec%2Ffilesystem_spec/sbom","scorecard":{"id":412846,"data":{"date":"2025-08-11","repo":{"name":"github.com/fsspec/filesystem_spec","commit":"c46db870f828f7f1318ed7a3dc26ecc8b48b3880"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":5.6,"checks":[{"name":"Code-Review","score":6,"reason":"Found 18/30 approved changesets -- score normalized to 6","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":10,"reason":"28 commit(s) and 24 issue activity found in the last 90 days -- score normalized to 10","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/main.yaml:1","Warn: no topLevel permission defined: .github/workflows/pypipublish.yaml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/main.yaml:28: update your workflow using https://app.stepsecurity.io/secureworkflow/fsspec/filesystem_spec/main.yaml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/main.yaml:33: update your workflow using https://app.stepsecurity.io/secureworkflow/fsspec/filesystem_spec/main.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/main.yaml:53: update your workflow using https://app.stepsecurity.io/secureworkflow/fsspec/filesystem_spec/main.yaml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/main.yaml:58: update your workflow using https://app.stepsecurity.io/secureworkflow/fsspec/filesystem_spec/main.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/main.yaml:72: update your workflow using https://app.stepsecurity.io/secureworkflow/fsspec/filesystem_spec/main.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/main.yaml:73: update your workflow using https://app.stepsecurity.io/secureworkflow/fsspec/filesystem_spec/main.yaml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/main.yaml:76: update your workflow using https://app.stepsecurity.io/secureworkflow/fsspec/filesystem_spec/main.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/main.yaml:84: update your workflow using https://app.stepsecurity.io/secureworkflow/fsspec/filesystem_spec/main.yaml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/main.yaml:89: update your workflow using https://app.stepsecurity.io/secureworkflow/fsspec/filesystem_spec/main.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/main.yaml:127: update your workflow using https://app.stepsecurity.io/secureworkflow/fsspec/filesystem_spec/main.yaml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/main.yaml:130: update your workflow using https://app.stepsecurity.io/secureworkflow/fsspec/filesystem_spec/main.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pypipublish.yaml:11: update your workflow using https://app.stepsecurity.io/secureworkflow/fsspec/filesystem_spec/pypipublish.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pypipublish.yaml:13: update your workflow using https://app.stepsecurity.io/secureworkflow/fsspec/filesystem_spec/pypipublish.yaml/master?enable=pin","Warn: pipCommand not pinned by hash: install_s3fs.sh:13","Warn: pipCommand not pinned by hash: .github/workflows/pypipublish.yaml:18","Warn: pipCommand not pinned by hash: .github/workflows/pypipublish.yaml:19","Info:   0 out of   8 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   5 third-party GitHubAction dependencies pinned","Info:   0 out of   3 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: BSD 3-Clause \"New\" or \"Revised\" License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":10,"reason":"project is fuzzed","details":["Info: OSSFuzz integration found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-18T23:08:53.934Z","repository_id":37792801,"created_at":"2025-08-18T23:08:53.934Z","updated_at":"2025-08-18T23:08:53.934Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27616970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-09T02:00:09.185Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T15:01:12.672Z","updated_at":"2025-12-11T22:49:36.789Z","avatar_url":"https://github.com/fsspec.png","language":"Python","funding_links":[],"categories":["Python","others","Libraries","File \u0026 Path Utilities"],"sub_categories":[],"readme":"# filesystem_spec\n\n[![PyPI version](https://badge.fury.io/py/fsspec.svg)](https://pypi.python.org/pypi/fsspec/)\n[![Anaconda-Server Badge](https://anaconda.org/conda-forge/fsspec/badges/version.svg)](https://anaconda.org/conda-forge/fsspec)\n![Build](https://github.com/fsspec/filesystem_spec/workflows/CI/badge.svg)\n[![Docs](https://readthedocs.org/projects/filesystem-spec/badge/?version=latest)](https://filesystem-spec.readthedocs.io/en/latest/?badge=latest)\n\nA specification for pythonic filesystems.\n\n## Install\n\n```bash\npip install fsspec\n```\n\nwould install the base fsspec. Various optionally supported features might require specification of custom\nextra require, e.g. `pip install fsspec[ssh]` will install dependencies for `ssh` backends support.\nUse `pip install fsspec[full]` for installation of all known extra dependencies.\n\nUp-to-date package also provided through conda-forge distribution:\n\n```bash\nconda install -c conda-forge fsspec\n```\n\n\n## Purpose\n\nTo produce a template or specification for a file-system interface, that specific implementations should follow,\nso that applications making use of them can rely on a common behaviour and not have to worry about the specific\ninternal implementation decisions with any given backend. Many such implementations are included in this package,\nor in sister projects such as `s3fs` and `gcsfs`.\n\nIn addition, if this is well-designed, then additional functionality, such as a key-value store or FUSE\nmounting of the file-system implementation may be available for all implementations \"for free\".\n\n## Documentation\n\nPlease refer to [RTD](https://filesystem-spec.readthedocs.io/en/latest/?badge=latest)\n\n## Develop\n\nfsspec uses GitHub Actions for CI. Environment files can be found\nin the \"ci/\" directory. Note that the main environment is called \"py38\",\nbut it is expected that the version of python installed be adjustable at\nCI runtime. For local use, pick a version suitable for you.\n\n```bash\n# For a new environment (mamba / conda).\nmamba create -n fsspec -c conda-forge  python=3.10 -y\nconda activate fsspec\n\n# Standard dev install with docs and tests.\npip install -e \".[dev,doc,test]\"\n\n# Full tests except for downstream\npip install s3fs\npip uninstall s3fs\npip install -e .[dev,doc,test_full]\npip install s3fs --no-deps\npytest -v\n\n# Downstream tests.\nsh install_s3fs.sh\n# Windows powershell.\ninstall_s3fs.sh\n```\n\n### Testing\n\nTests can be run in the dev environment, if activated, via ``pytest fsspec``.\n\nThe full fsspec suite requires a system-level docker, docker-compose, and fuse\ninstallation. If only making changes to one backend implementation, it is\nnot generally necessary to run all tests locally.\n\nIt is expected that contributors ensure that any change to fsspec does not\ncause issues or regressions for either other fsspec-related packages such\nas gcsfs and s3fs, nor for downstream users of fsspec. The \"downstream\" CI\nrun and corresponding environment file run a set of tests from the dask\ntest suite, and very minimal tests against pandas and zarr from the\ntest_downstream.py module in this repo.\n\n### Code Formatting\n\nfsspec uses [Black](https://black.readthedocs.io/en/stable) to ensure\na consistent code format throughout the project.\nRun ``black fsspec`` from the root of the filesystem_spec repository to\nauto-format your code. Additionally, many editors have plugins that will apply\n``black`` as you edit files. ``black`` is included in the ``tox`` environments.\n\nOptionally, you may wish to setup [pre-commit hooks](https://pre-commit.com) to\nautomatically run ``black`` when you make a git commit.\nRun ``pre-commit install --install-hooks`` from the root of the\nfilesystem_spec repository to setup pre-commit hooks. ``black`` will now be run\nbefore you commit, reformatting any changed files. You can format without\ncommitting via ``pre-commit run`` or skip these checks with ``git commit\n--no-verify``.\n\n## Support\n\nWork on this repository is supported in part by:\n\n\"Anaconda, Inc. - Advancing AI through open source.\"\n\n\u003ca href=\"https://anaconda.com/\"\u003e\u003cimg src=\"https://camo.githubusercontent.com/b8555ef2222598ed37ce38ac86955febbd25de7619931bb7dd3c58432181d3b6/68747470733a2f2f626565776172652e6f72672f636f6d6d756e6974792f6d656d626572732f616e61636f6e64612f616e61636f6e64612d6c617267652e706e67\" alt=\"anaconda logo\" width=\"40%\"/\u003e\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffsspec%2Ffilesystem_spec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffsspec%2Ffilesystem_spec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffsspec%2Ffilesystem_spec/lists"}