{"id":20853720,"url":"https://github.com/wolfeidau/s3iofs","last_synced_at":"2026-03-07T02:14:19.122Z","repository":{"id":54813097,"uuid":"333395103","full_name":"wolfeidau/s3iofs","owner":"wolfeidau","description":"This package provides an S3 implementation for Go1.16 filesystem interface.","archived":false,"fork":false,"pushed_at":"2025-04-21T03:54:47.000Z","size":172,"stargazers_count":10,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-21T04:46:34.478Z","etag":null,"topics":["aws","golang","s3"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wolfeidau.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-27T11:15:11.000Z","updated_at":"2025-04-21T03:53:58.000Z","dependencies_parsed_at":"2024-01-17T06:55:12.197Z","dependency_job_id":"2fb4142a-262a-44e2-b273-e14f6b7facc6","html_url":"https://github.com/wolfeidau/s3iofs","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wolfeidau%2Fs3iofs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wolfeidau%2Fs3iofs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wolfeidau%2Fs3iofs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wolfeidau%2Fs3iofs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wolfeidau","download_url":"https://codeload.github.com/wolfeidau/s3iofs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253682166,"owners_count":21946893,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","golang","s3"],"created_at":"2024-11-18T03:22:55.089Z","updated_at":"2026-03-07T02:14:19.058Z","avatar_url":"https://github.com/wolfeidau.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# s3iofs [![Go Report Card](https://goreportcard.com/badge/github.com/wolfeidau/s3iofs)](https://goreportcard.com/report/github.com/wolfeidau/s3iofs) [![Documentation](https://godoc.org/github.com/wolfeidau/s3iofs?status.svg)](https://godoc.org/github.com/wolfeidau/s3iofs)\n\nThis package provides an S3 implementation for [Go1.16 filesystem interface](https://tip.golang.org/doc/go1.16#fs).\n\n# Overview\n\nThis package provides an S3 implementation for the Go1.16 filesystem interface using the [AWS SDK for Go v2](https://github.com/aws/aws-sdk-go-v2).\n\nThe `S3FS` implements the following interfaces:\n\n- `fs.FS`\n- `fs.StatFS`\n- `fs.ReadDirFS`\n\nThe `s3File` implements the following interfaces:\n\n- `fs.FileInfo`\n- `fs.DirEntry`\n- `fs.ReadDirFile`\n- `io.ReaderAt`\n- `io.Seeker`\n\nIn addition to this the `S3FS` also implements the following interfaces:\n\n- `RemoveFS`, which provides a `Remove(name string) error` method.\n- `WriteFileFS` which provides a `WriteFile(name string, data []byte, perm fs.FileMode) error` method.\n\nThe `Seek` and `ReadAt` operations enable libraries such as [apache arrow](https://arrow.apache.org/) to read parts of a parquet file from S3, without downloading the entire file.\n\n# Usage \n\n```go\n\t// Load the Shared AWS Configuration (~/.aws/config) and enable request logging\n\tawscfg, err := config.LoadDefaultConfig(context.TODO(),\n\t\tconfig.WithClientLogMode(aws.LogRetries|aws.LogRequest),\n\t\tconfig.WithLogger(logging.NewStandardLogger(os.Stdout)),\n\t)\n\tif err != nil {\n\t\t// ...\n\t}\n\n\ts3fs := s3iofs.New(os.Getenv(\"TEST_BUCKET_NAME\"), awscfg)\n\n\terr = fs.WalkDir(s3fs, \".\", func(path string, d fs.DirEntry, err error) error {\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\n\t\tif d.IsDir() {\n\t\t\tfmt.Println(\"dir:\", path)\n\t\t\treturn nil\n\t\t}\n\t\tfmt.Println(\"file:\", path)\n\t\treturn nil\n\t})\n\tif err != nil {\n\t\t// ...\n\t}\n```\n\n# Integration Tests\n\nThe integration tests for this package are in a separate module under the `integration`\tdirectory, this to avoid polluting the main module with docker based testing dependencies used to run the tests locally against [minio](https://min.io/).\n\n# Links\n\nS3 implements ranges based on [HTTP Request ranges](https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests), it well worth reading up on this if your using the `io.ReadSeek` interface.\n\n# License\n\nThis application is released under Apache 2.0 license and is copyright [Mark Wolfe](https://www.wolfe.id.au).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwolfeidau%2Fs3iofs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwolfeidau%2Fs3iofs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwolfeidau%2Fs3iofs/lists"}