{"id":13441210,"url":"https://github.com/gaul/s3proxy","last_synced_at":"2025-05-16T17:00:25.526Z","repository":{"id":19093400,"uuid":"22321368","full_name":"gaul/s3proxy","owner":"gaul","description":"Access other storage backends via the S3 API","archived":false,"fork":false,"pushed_at":"2025-05-02T03:30:14.000Z","size":1680,"stargazers_count":1912,"open_issues_count":150,"forks_count":238,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-05-08T20:55:45.125Z","etag":null,"topics":["atmos","aws-s3","azure","backblaze-b2","google-cloud-storage","openstack-swift","proxy","s3"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gaul.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2014-07-27T20:32:24.000Z","updated_at":"2025-05-08T14:50:18.000Z","dependencies_parsed_at":"2023-10-17T06:27:05.122Z","dependency_job_id":"1caebc1e-9d6f-4117-8051-687e847b68f4","html_url":"https://github.com/gaul/s3proxy","commit_stats":{"total_commits":1155,"total_committers":74,"mean_commits":"15.608108108108109","dds":"0.22510822510822515","last_synced_commit":"83bc9f7f7f7bf8b901521ad4df414540c5b367d9"},"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gaul%2Fs3proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gaul%2Fs3proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gaul%2Fs3proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gaul%2Fs3proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gaul","download_url":"https://codeload.github.com/gaul/s3proxy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253156684,"owners_count":21862980,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atmos","aws-s3","azure","backblaze-b2","google-cloud-storage","openstack-swift","proxy","s3"],"created_at":"2024-07-31T03:01:31.171Z","updated_at":"2025-05-08T22:26:14.976Z","avatar_url":"https://github.com/gaul.png","language":"Java","readme":"# S3Proxy\n\n[![Github All Releases](https://img.shields.io/github/downloads/gaul/s3proxy/total.svg)](https://github.com/gaul/s3proxy/releases/)\n[![Docker Pulls](https://img.shields.io/docker/pulls/andrewgaul/s3proxy.svg)](https://hub.docker.com/r/andrewgaul/s3proxy/)\n[![Maven Central](https://img.shields.io/maven-central/v/org.gaul/s3proxy.svg)](https://search.maven.org/#search%7Cga%7C1%7Ca%3A%22s3proxy%22)\n[![Twitter Follow](https://img.shields.io/twitter/follow/S3Proxy.svg?style=social\u0026label=Follow)](https://twitter.com/S3Proxy)\n\nS3Proxy implements the\n[S3 API](https://en.wikipedia.org/wiki/Amazon_S3#S3_API_and_competing_services)\nand *proxies* requests, enabling several use cases:\n\n* translation from S3 to Backblaze B2, EMC Atmos, Google Cloud, Microsoft Azure, and OpenStack Swift\n* testing without Amazon by using the local filesystem\n* extension via middlewares\n* embedding into Java applications\n\n## Usage with Docker\n\n[Docker Hub](https://hub.docker.com/r/andrewgaul/s3proxy/) hosts a Docker image\nand has instructions on how to run it.\n\n## Usage without Docker\n\nUsers can [download releases](https://github.com/gaul/s3proxy/releases)\nfrom GitHub.  Developers can build the project by running `mvn package` which\nproduces a binary at `target/s3proxy`.  S3Proxy requires Java 11 or newer to\nrun.\n\nConfigure S3Proxy via a properties file.  An example using the local\nfile system as the storage backend with anonymous access:\n\n```\ns3proxy.authorization=none\ns3proxy.endpoint=http://127.0.0.1:8080\njclouds.provider=filesystem\njclouds.filesystem.basedir=/tmp/s3proxy\n```\n\nFirst create the filesystem basedir:\n\n```\nmkdir /tmp/s3proxy\n```\n\nNext run S3Proxy.  Linux and Mac OS X users can run the executable jar:\n\n```\nchmod +x s3proxy\ns3proxy --properties s3proxy.conf\n```\n\nWindows users must explicitly invoke java:\n\n```\njava -jar s3proxy --properties s3proxy.conf\n```\n\nFinally test by creating a bucket then listing all the buckets:\n\n```\n$ curl --request PUT http://localhost:8080/testbucket\n\n$ curl http://localhost:8080/\n\u003c?xml version=\"1.0\" ?\u003e\u003cListAllMyBucketsResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"\u003e\u003cOwner\u003e\u003cID\u003e75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a\u003c/ID\u003e\u003cDisplayName\u003eCustomersName@amazon.com\u003c/DisplayName\u003e\u003c/Owner\u003e\u003cBuckets\u003e\u003cBucket\u003e\u003cName\u003etestbucket\u003c/Name\u003e\u003cCreationDate\u003e2015-08-05T22:16:24.000Z\u003c/CreationDate\u003e\u003c/Bucket\u003e\u003c/Buckets\u003e\u003c/ListAllMyBucketsResult\u003e\n```\n\n## Usage with Java\n\nMaven Central hosts S3Proxy artifacts and the wiki has\n[instructions on Java use](https://github.com/gaul/s3proxy/wiki/Using-S3Proxy-in-Java-projects).\n\n## Supported storage backends\n\n* atmos\n* aws-s3 (Amazon-only)\n* azureblob\n* azureblob-sdk (newer but lacks multi-part upload, see [Azure/azure-sdk-for-java#42603](https://github.com/Azure/azure-sdk-for-java/issues/42603))\n* b2\n* filesystem (on-disk storage, deprecated)\n* filesystem-nio2 (on-disk storage, recommended)\n* google-cloud-storage\n* openstack-swift\n* rackspace-cloudfiles-uk and rackspace-cloudfiles-us\n* s3 (all implementations)\n* transient (in-memory storage, deprecated)\n* transient-nio2 (in-memory storage, recommended)\n\nSee the wiki for [examples of configurations](https://github.com/gaul/s3proxy/wiki/Storage-backend-examples).\n\n## Assigning buckets to backends\n\nS3Proxy can be configured to assign buckets to different backends with the same\ncredentials. The configuration in the properties file is as follows:\n```\ns3proxy.bucket-locator.1=bucket\ns3proxy.bucket-locator.2=another-bucket\n```\n\nIn addition to the explicit names, [glob syntax](https://docs.oracle.com/javase/tutorial/essential/io/fileOps.html#glob) can be used to configure many\nbuckets for a given backend.\n\nA bucket (or a glob) cannot be assigned to multiple backends.\n\n## Middlewares\n\nS3Proxy can modify its behavior based on middlewares:\n\n* [bucket aliasing](https://github.com/gaul/s3proxy/wiki/Middleware-alias-blobstore)\n* [bucket locator](https://github.com/gaul/s3proxy/wiki/Middleware-bucket-locator)\n* [eventual consistency modeling](https://github.com/gaul/s3proxy/wiki/Middleware---eventual-consistency)\n* [large object mocking](https://github.com/gaul/s3proxy/wiki/Middleware-large-object-mocking)\n* [read-only](https://github.com/gaul/s3proxy/wiki/Middleware-read-only)\n* [regex rename blobs](https://github.com/gaul/s3proxy/wiki/Middleware-regex)\n* [sharded backend containers](https://github.com/gaul/s3proxy/wiki/Middleware-sharded-backend)\n* [storage class override](https://github.com/gaul/s3proxy/wiki/Middleware-storage-class-override)\n* [user metadata replacer](https://github.com/gaul/s3proxy/wiki/Middleware-user-metadata-replacer)\n\n## SSL Support\n\nS3Proxy can listen on HTTPS by setting the `secure-endpoint` and [configuring a keystore](http://wiki.eclipse.org/Jetty/Howto/Configure_SSL#Generating_Keys_and_Certificates_with_JDK_keytool). You can read more about how configure S3Proxy for SSL Support in [the dedicated wiki page](https://github.com/gaul/s3proxy/wiki/SSL-support) with Docker, Kubernetes or simply Java.\n\n## Limitations\n\nS3Proxy has broad compatibility with the S3 API, however, it does not support:\n\n* ACLs other than private and public-read\n* BitTorrent hosting\n* bucket logging\n* bucket policies\n* [CORS bucket operations](https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html#how-do-i-enable-cors) like getting or setting the CORS configuration for a bucket. S3Proxy only supports a static configuration (see below).\n* hosting static websites\n* object server-side encryption\n* object tagging\n* object versioning, see [#74](https://github.com/gaul/s3proxy/issues/74)\n* POST upload policies, see [#73](https://github.com/gaul/s3proxy/issues/73)\n* requester pays buckets\n* [select object content](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html)\n\nS3Proxy emulates the following operations:\n\n* copy multi-part objects, see [#76](https://github.com/gaul/s3proxy/issues/76)\n\nS3Proxy has basic CORS preflight and actual request/response handling. It can be configured within the properties\nfile (and corresponding ENV variables for Docker):\n\n```\ns3proxy.cors-allow-origins=https://example\\.com https://.+\\.example\\.com https://example\\.cloud\ns3proxy.cors-allow-methods=GET PUT\ns3proxy.cors-allow-headers=Accept Content-Type\ns3proxy.cors-allow-credential=true\n```\n\nCORS cannot be configured per bucket. `s3proxy.cors-allow-all=true` will accept any origin and header.\nActual CORS requests are supported for GET, PUT, POST, HEAD and DELETE methods.\n\nThe wiki collects\n[compatibility notes](https://github.com/gaul/s3proxy/wiki/Storage-backend-compatibility)\nfor specific storage backends.\n\n## Support\n\n* [GitHub issues](https://github.com/gaul/s3proxy/issues)\n* [Stack Overflow](https://stackoverflow.com/questions/tagged/s3proxy)\n* [commercial support](mailto:andrew@gaul.org)\n\n## References\n\n* [Apache jclouds](https://jclouds.apache.org/) provides storage backend support for S3Proxy\n* [Ceph s3-tests](https://github.com/ceph/s3-tests) help maintain and improve compatibility with the S3 API\n* [fake-s3](https://github.com/jubos/fake-s3), [gofakes3](https://github.com/johannesboyne/gofakes3), [minio](https://github.com/minio/minio), [S3 ninja](https://github.com/scireum/s3ninja), and [s3rver](https://github.com/jamhall/s3rver) provide functionality similar to S3Proxy when using the filesystem backend\n* [GlacierProxy](https://github.com/bouncestorage/glacier-proxy) and [SwiftProxy](https://github.com/bouncestorage/swiftproxy) provide similar functionality for the Amazon Glacier and OpenStack Swift APIs\n* [s3mock](https://github.com/adobe/S3Mock) - Adobe's s3 mock implementation\n* [sbt-s3](https://github.com/localytics/sbt-s3) runs S3Proxy via the Scala Build Tool\n* [swift3](https://github.com/openstack/swift3) provides an S3 middleware for OpenStack Swift\n* [Zenko](https://www.zenko.io/) provide similar multi-cloud functionality\n\n## License\n\nCopyright (C) 2014-2025 Andrew Gaul\n\nLicensed under the Apache License, Version 2.0\n","funding_links":[],"categories":["HarmonyOS","Java"],"sub_categories":["Windows Manager"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgaul%2Fs3proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgaul%2Fs3proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgaul%2Fs3proxy/lists"}