{"id":41624020,"url":"https://github.com/climateimpactlab/datafs","last_synced_at":"2026-01-24T14:09:08.136Z","repository":{"id":49863891,"uuid":"74184663","full_name":"ClimateImpactLab/DataFS","owner":"ClimateImpactLab","description":"An abstraction layer for data storage systems","archived":false,"fork":false,"pushed_at":"2022-12-26T20:36:15.000Z","size":942,"stargazers_count":6,"open_issues_count":50,"forks_count":2,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-09-10T03:53:20.493Z","etag":null,"topics":["command-line-tool","data-version-control","file-management","python","storage"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ClimateImpactLab.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-11-19T03:32:09.000Z","updated_at":"2023-09-08T17:17:18.000Z","dependencies_parsed_at":"2023-01-31T01:31:02.373Z","dependency_job_id":null,"html_url":"https://github.com/ClimateImpactLab/DataFS","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/ClimateImpactLab/DataFS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClimateImpactLab%2FDataFS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClimateImpactLab%2FDataFS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClimateImpactLab%2FDataFS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClimateImpactLab%2FDataFS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ClimateImpactLab","download_url":"https://codeload.github.com/ClimateImpactLab/DataFS/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClimateImpactLab%2FDataFS/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28729411,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-24T10:24:43.181Z","status":"ssl_error","status_checked_at":"2026-01-24T10:24:36.112Z","response_time":89,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["command-line-tool","data-version-control","file-management","python","storage"],"created_at":"2026-01-24T14:09:07.015Z","updated_at":"2026-01-24T14:09:08.127Z","avatar_url":"https://github.com/ClimateImpactLab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"=========================================\nDataFS Data Management System\n=========================================\n\n.. image:: https://img.shields.io/pypi/v/datafs.svg\n        :target: https://pypi.python.org/pypi/datafs\n\n.. image:: https://travis-ci.org/ClimateImpactLab/DataFS.svg?branch=master\n        :target: https://travis-ci.org/ClimateImpactLab/DataFS?branch=master\n\n.. image:: https://coveralls.io/repos/github/ClimateImpactLab/DataFS/badge.svg?branch=master\n        :target: https://coveralls.io/github/ClimateImpactLab/DataFS?branch=master\n\n.. image:: https://readthedocs.org/projects/datafs/badge/?version=latest\n        :target: https://datafs.readthedocs.io/en/latest/?badge=latest\n        :alt: Documentation Status\n\n.. image:: https://pyup.io/repos/github/ClimateImpactLab/DataFS/shield.svg\n        :target: https://pyup.io/repos/github/ClimateImpactLab/DataFS/\n        :alt: Updates\n\n.. image:: https://api.codacy.com/project/badge/Grade/5e095453424840e092e71c42b8ad8b52\n        :alt: Codacy Badge\n        :target: https://www.codacy.com/app/delgadom/DataFS?utm_source=github.com\u0026utm_medium=referral\u0026utm_content=ClimateImpactLab/DataFS\u0026utm_campaign=badger\n\n.. image:: https://badges.gitter.im/DataFS/Lobby.svg\n   :alt: Join the chat at https://gitter.im/DataFS/Lobby\n   :target: https://gitter.im/DataFS/Lobby?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge\n\n\nDataFS is a package manager for data. It manages file versions, dependencies, and metadata for individual use or large organizations.\n\nConfigure and connect to a metadata `Manager \u003chttp://datafs.readthedocs.io/en/latest/configure.manager.html\u003e`_ and multiple data `Services \u003chttp://datafs.readthedocs.io/en/latest/configure.authorities.html\u003e`_ using a specification file and you'll be sharing, tracking, and using your data in seconds.\n\n\n* Free software: MIT license\n* Documentation: https://datafs.readthedocs.io.\n\n\nFeatures\n--------\n\n* Explicit version and metadata management for teams\n* Unified read/write interface across file systems\n* Easily create out-of-the-box configuration files for users\n* Track data dependencies and usage logs\n* Use datafs from python or from the command line\n* Permissions handled by managers \u0026 services, giving you control over user access\n\n\nUsage\n-----\n\nFirst, `configure an API \u003chttp://datafs.readthedocs.io/en/latest/configure.html\u003e`_. Don't worry. It's not too bad. Check out the `quickstart \u003chttp://datafs.readthedocs.io/en/latest/quickstart.yml\u003e`_ to follow along.\n\nWe'll assume we already have an API object created and attached to a service called \"local\". Once you have this, you can start using DataFS to create and use archives.\n\n.. code-block:: bash\n\n    $ datafs create my_new_data_archive --description \"a test archive\"\n    created versioned archive \u003cDataArchive local://my_new_data_archive\u003e\n    \n    $ echo \"initial file contents\" \u003e my_file.txt\n    \n    $ datafs update my_new_data_archive my_file.txt\n    \n    $ datafs cat my_new_data_archive\n    initial file contents\n\nVersions are tracked explicitly. Bump versions on write, and read old versions \nif desired.\n\n.. code-block:: bash\n\n    $ echo \"updated contents\" \u003e my_file.txt\n    \n    $ datafs update my_new_data_archive my_file.txt --bumpversion minor\n    uploaded data to \u003cDataArchive local://my_new_data_archive\u003e. version bumped 0.0.1 --\u003e 0.1.\n    \n    $ datafs cat my_new_data_archive\n    updated contents\n    \n    $ datafs cat my_new_data_archive --version 0.0.1\n    initial file contents\n\nPin versions using a requirements file to set the default version\n\n.. code-block:: bash\n\n    $ echo \"my_new_data_archive==0.0.1\" \u003e requirements_data.txt\n    \n    $ datafs cat my_new_data_archive\n    initial file contents\n\nAll of these features are available from (and faster in) python:\n\n.. code-block:: python\n\n    \u003e\u003e\u003e import datafs\n    \u003e\u003e\u003e api = datafs.get_api()\n    \u003e\u003e\u003e archive = api.get_archive('my_new_data_archive')\n    \u003e\u003e\u003e with archive.open('r', version='latest') as f:\n    ...     print(f.read())\n    ...\n    updated contents\n\n\nIf you have permission to delete archives, it's easy to do. See `administrative tools \u003chttp://datafs.readthedocs.io/en/latest/admin.html\u003e`_ for tips on setting permissions.\n\n.. code-block:: bash\n\n    $ datafs delete my_new_data_archive\n    deleted archive \u003cDataArchive local://my_new_data_archive\u003e\n\nSee `examples \u003chttp://datafs.readthedocs.io/en/latest/examples.html\u003e`_ for more extensive use cases.\n\n\n\nInstallation\n------------\n\n``pip install datafs``\n\n\nAdditionally, you'll need a manager and services:\n\nManagers:\n\n* MongoDB: ``pip install pymongo``\n* DynamoDB: ``pip install boto3``\n\nServices:\n\n* Ready out-of-the-box:\n\n  - local\n  - shared\n  - mounted\n  - zip\n  - ftp\n  - http/https\n  - in-memory\n\n* Requiring additional packages:\n\n  - AWS/S3: ``pip install boto``\n  - SFTP: ``pip install paramiko``\n  - XMLRPC: ``pip install xmlrpclib``\n\n\nRequirements\n------------\n\nFor now, DataFS requires python 2.7. We're working on 3x support.\n\n\nTodo\n----\n\nSee `issues \u003chttps://github.com/ClimateImpactLab/DataFS/issues\u003e`_ to see and add to our todos.\n\n\nCredits\n---------\n\nThis package was created by `Justin Simcock \u003chttps://github.com/jgerardsimcock\u003e`_ and `Michael Delgado \u003chttps://github.com/delgadom\u003e`_ of the `Climate Impact Lab \u003chttp://impactlab.org\u003e`_. Check us out on `github \u003chttps://github.com/ClimateImpactLab\u003e`_.\n\nMajor kudos to the folks at `PyFilesystem \u003chttps://github.com/PyFilesystem\u003e`_. Thanks also to `audreyr \u003chttps://github.com/audreyr\u003e`_ for the wonderful `cookiecutter \u003chttps://github.com/audreyr/cookiecutter-pypackage\u003e`_ package, and to `Pyup \u003chttps://pyup.io\u003e`_, a constant source of inspiration and our silent third contributor.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclimateimpactlab%2Fdatafs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclimateimpactlab%2Fdatafs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclimateimpactlab%2Fdatafs/lists"}