{"id":19056948,"url":"https://github.com/pyfilesystem/s3fs","last_synced_at":"2025-04-12T21:31:10.571Z","repository":{"id":23327765,"uuid":"98723914","full_name":"PyFilesystem/s3fs","owner":"PyFilesystem","description":"Amazon S3 filesystem for PyFilesystem2","archived":false,"fork":false,"pushed_at":"2024-07-16T00:00:33.000Z","size":67,"stargazers_count":154,"open_issues_count":44,"forks_count":55,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-04T01:07:33.737Z","etag":null,"topics":["amazon","filesystem","pyfilesystem2","s3"],"latest_commit_sha":null,"homepage":"http://fs-s3fs.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PyFilesystem.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-29T09:53:35.000Z","updated_at":"2024-11-07T06:04:36.000Z","dependencies_parsed_at":"2024-06-18T13:49:03.953Z","dependency_job_id":"01d89d15-5dc8-41d7-903c-a0964fc91807","html_url":"https://github.com/PyFilesystem/s3fs","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PyFilesystem%2Fs3fs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PyFilesystem%2Fs3fs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PyFilesystem%2Fs3fs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PyFilesystem%2Fs3fs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PyFilesystem","download_url":"https://codeload.github.com/PyFilesystem/s3fs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248635244,"owners_count":21137194,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon","filesystem","pyfilesystem2","s3"],"created_at":"2024-11-08T23:52:49.665Z","updated_at":"2025-04-12T21:31:10.548Z","avatar_url":"https://github.com/PyFilesystem.png","language":"Python","readme":"# S3FS\n\nS3FS is a [PyFilesystem](https://www.pyfilesystem.org/) interface to\nAmazon S3 cloud storage.\n\nAs a PyFilesystem concrete class, [S3FS](http://fs-s3fs.readthedocs.io/en/latest/) allows you to work with S3 in the\nsame way as any other supported filesystem.\n\n## Installing\n\nYou can install S3FS from pip as follows:\n\n```\npip install fs-s3fs\n```\n\n## Opening a S3FS\n\nOpen an S3FS by explicitly using the constructor:\n\n```python\nfrom fs_s3fs import S3FS\ns3fs = S3FS('mybucket')\n```\n\nOr with a FS URL:\n\n```python\n  from fs import open_fs\n  s3fs = open_fs('s3://mybucket')\n```\n\n## Downloading Files\n\nTo *download* files from an S3 bucket, open a file on the S3\nfilesystem for reading, then write the data to a file on the local\nfilesystem. Here's an example that copies a file `example.mov` from\nS3 to your HD:\n\n```python\nfrom fs.tools import copy_file_data\nwith s3fs.open('example.mov', 'rb') as remote_file:\n    with open('example.mov', 'wb') as local_file:\n        copy_file_data(remote_file, local_file)\n```\n\nAlthough it is preferable to use the higher-level functionality in the\n`fs.copy` module. Here's an example:\n\n```python\nfrom fs.copy import copy_file\ncopy_file(s3fs, 'example.mov', './', 'example.mov')\n```\n\n## Uploading Files\n\nYou can *upload* files in the same way. Simply copy a file from a\nsource filesystem to the S3 filesystem.\nSee [Moving and Copying](https://docs.pyfilesystem.org/en/latest/guide.html#moving-and-copying)\nfor more information.\n\n## ExtraArgs\n\nS3 objects have additional properties, beyond a traditional\nfilesystem. These options can be set using the ``upload_args``\nand ``download_args`` properties. which are handed to upload\nand download methods, as appropriate, for the lifetime of the\nfilesystem instance.\n\nFor example, to set the ``cache-control`` header of all objects\nuploaded to a bucket:\n\n```python\nimport fs, fs.mirror\ns3fs = S3FS('example', upload_args={\"CacheControl\": \"max-age=2592000\", \"ACL\": \"public-read\"})\nfs.mirror.mirror('/path/to/mirror', s3fs)\n```\n\nsee [the Boto3 docs](https://boto3.readthedocs.io/en/latest/reference/customizations/s3.html#boto3.s3.transfer.S3Transfer.ALLOWED_UPLOAD_ARGS)\nfor more information.\n\n`acl` and `cache_control` are exposed explicitly for convenience, and can be used in URLs.\nIt is important to URL-Escape the `cache_control` value in a URL, as it may contain special characters.\n\n```python\nimport fs, fs.mirror\nwith open fs.open_fs('s3://example?acl=public-read\u0026cache_control=max-age%3D2592000%2Cpublic') as s3fs\n    fs.mirror.mirror('/path/to/mirror', s3fs)\n```\n\n\n## S3 URLs\n\nYou can get a public URL to a file on a S3 bucket as follows:\n\n```python\nmovie_url = s3fs.geturl('example.mov')\n```\n\n## Documentation\n\n- [PyFilesystem Wiki](https://www.pyfilesystem.org)\n- [S3FS Reference](http://fs-s3fs.readthedocs.io/en/latest/)\n- [PyFilesystem Reference](https://docs.pyfilesystem.org/en/latest/reference/base.html)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyfilesystem%2Fs3fs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpyfilesystem%2Fs3fs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyfilesystem%2Fs3fs/lists"}