{"id":41373176,"url":"https://github.com/mirakl/s3proxy","last_synced_at":"2026-01-23T09:52:23.193Z","repository":{"id":35192708,"uuid":"118642190","full_name":"mirakl/s3proxy","owner":"mirakl","description":"Manipulate objects, centralize credentials and access rights for S3","archived":false,"fork":false,"pushed_at":"2025-06-16T14:22:50.000Z","size":16139,"stargazers_count":26,"open_issues_count":3,"forks_count":3,"subscribers_count":36,"default_branch":"master","last_synced_at":"2025-11-16T23:27:01.774Z","etag":null,"topics":["aws","aws-s3","go","golang","s3","terraform-managed"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mirakl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-01-23T17:07:45.000Z","updated_at":"2025-08-05T15:06:20.000Z","dependencies_parsed_at":"2024-06-21T15:40:30.724Z","dependency_job_id":"6b6233a4-bd6c-48c4-b9c0-78923335c604","html_url":"https://github.com/mirakl/s3proxy","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"purl":"pkg:github/mirakl/s3proxy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirakl%2Fs3proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirakl%2Fs3proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirakl%2Fs3proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirakl%2Fs3proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mirakl","download_url":"https://codeload.github.com/mirakl/s3proxy/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mirakl%2Fs3proxy/sbom","scorecard":{"id":649091,"data":{"date":"2025-08-11","repo":{"name":"github.com/mirakl/s3proxy","commit":"458b27399ff9059e543bd4e73fd96693618ccb99"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":5.6,"checks":[{"name":"Maintained","score":10,"reason":"12 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":10,"reason":"all changesets reviewed","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Info: jobLevel 'contents' permission set to 'read': .github/workflows/codeql.yml:18","Info: jobLevel 'actions' permission set to 'read': .github/workflows/codeql.yml:17","Warn: no topLevel permission defined: .github/workflows/ci.yml:1","Warn: no topLevel permission defined: .github/workflows/codeql.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:36: update your workflow using https://app.stepsecurity.io/secureworkflow/mirakl/s3proxy/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:42: update your workflow using https://app.stepsecurity.io/secureworkflow/mirakl/s3proxy/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:17: update your workflow using https://app.stepsecurity.io/secureworkflow/mirakl/s3proxy/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:23: update your workflow using https://app.stepsecurity.io/secureworkflow/mirakl/s3proxy/ci.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:29: update your workflow using https://app.stepsecurity.io/secureworkflow/mirakl/s3proxy/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/codeql.yml:30: update your workflow using https://app.stepsecurity.io/secureworkflow/mirakl/s3proxy/codeql.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/codeql.yml:34: update your workflow using https://app.stepsecurity.io/secureworkflow/mirakl/s3proxy/codeql.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/codeql.yml:39: update your workflow using https://app.stepsecurity.io/secureworkflow/mirakl/s3proxy/codeql.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/codeql.yml:48: update your workflow using https://app.stepsecurity.io/secureworkflow/mirakl/s3proxy/codeql.yml/master?enable=pin","Warn: containerImage not pinned by hash: Dockerfile:1","Warn: containerImage not pinned by hash: Dockerfile:17","Warn: containerImage not pinned by hash: Dockerfile:25: pin your Docker image by updating alpine:3.20 to alpine:3.20@sha256:b3119ef930faabb6b7b976780c0c7a9c1aa24d0c75e9179ac10e6bc9ac080d0d","Info:   0 out of   8 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 third-party GitHubAction dependencies pinned","Info:   0 out of   3 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":3,"reason":"7 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2022-0635","Warn: Project is vulnerable to: GO-2022-0646","Warn: Project is vulnerable to: GO-2024-3321 / GHSA-v778-237x-gjrc","Warn: Project is vulnerable to: GO-2025-3487 / GHSA-hcg3-q754-cr77","Warn: Project is vulnerable to: GO-2024-3333","Warn: Project is vulnerable to: GO-2025-3503 / GHSA-qxp5-gwg8-xv66","Warn: Project is vulnerable to: GO-2025-3595 / GHSA-vvgc-356p-c3xw"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":7,"reason":"SAST tool detected but not run on all commits","details":["Info: SAST configuration detected: CodeQL","Warn: 5 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-21T12:57:45.327Z","repository_id":35192708,"created_at":"2025-08-21T12:57:45.327Z","updated_at":"2025-08-21T12:57:45.327Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28687170,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-23T05:48:07.525Z","status":"ssl_error","status_checked_at":"2026-01-23T05:48:07.129Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-s3","go","golang","s3","terraform-managed"],"created_at":"2026-01-23T09:52:22.464Z","updated_at":"2026-01-23T09:52:23.186Z","avatar_url":"https://github.com/mirakl.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"s3proxy\n=======\n\nA S3 proxy server between your application and S3 for upload and download of objects. \nWhy use s3proxy ? To centralize credentials and access rights in your application infrastructure.\n\nS3Proxy can create presigned urls to allow another application (e.g. mmp) to upload or download objects from S3, and delete or duplicate objects.\n\nPlease note that uploads and downloads are using presigned urls, so S3proxy generates the url and returns it to your app, which then uses it to perform the upload or download.\nGenerating a presigned url is purely local and does not entails communicating with S3 (or Minio). This means that the presigned url may be rejected if the S3proxy credentials do not allow accessing the bucket.\n\nDeletes and copies are proxyfied, and S3Proxy itself sends the HTTP request to S3.\n\n## Architecture\n\ns3proxy has been developed as a REST service. It is generating pre-signed authentified upload and download URL on a object in a bucket.\nThis presigned URL has a duration period and can be used by any basic HTTP client in any language.\n\n## Requirements \n\n* Docker version 17.12.0+\n* Go 1.17+\n\n\n## Build\n\n* Install [Go](https://golang.org/doc/install)\n* Set $GOPATH variable, by default it should be something like $HOME/go\n* Set $GOBIN variable, by default it should be something like $GOPATH/bin\n* Make sure $GOBIN is in your $PATH : `export $PATH=$PATH:$GOBIN`\n\nThe previous steps are needed for some of the tools this projects uses, mainly goimports.\n\nThis project itself uses go modules, and so does NOT need to be in your GOPATH.\nIt would be better to clone it outside of the $GOPATH (or else you'll need to export `GO111MODULE=on`). \n\nClone the project : `git clone git@github.com:mirakl/s3proxy.git` and run `make`.\n\n\n## Build the docker image\n\n* run `make docker-image`\n\n\n## Push the docker image\n\n* run `make docker-image-push`\n\n\n## Add a new release\n\nYou only need to add a tag version in this repo without \"v\". [Dockerhub](https://hub.docker.com/repository/docker/mirakl/s3proxy) will see this tag and create a new artefact (docker image).\n\n\n## s3proxy Configuration\n\nYou can use environment variables or command line options for configuration.\n\n\n### Command Line Options\n\n```\nUsage of s3proxy:\n    --http-port : The port that the proxy binds to\n    --api-key : Define server side API key for API call authorization\n    --use-rsyslog : Add rsyslog as second logging destination by specifying the rsyslog host and port (ex. localhost:514)\n    --use-minio : Use minio as backend by specifying the minio server host and port (ex. localhost:9000)\n    --minio-access-key : Minion AccessKey equivalent to a AWS_ACCESS_KEY_ID\n    --minio-secret-key : Minion AccessKey equivalent to a AWS_SECRET_ACCESS_KEY   \n```\n\n\n### Environment variables\n\nThe following environment variables can be used in place of the corresponding command-line arguments:\n\n- `S3PROXY_HTTP_PORT`\n- `S3PROXY_API_KEY`\n- `S3PROXY_USE_RSYSLOG`\n- `S3PROXY_USE_MINIO`\n- `S3PROXY_MINIO_ACCESS_KEY`\n- `S3PROXY_MINIO_SECRET_KEY`\n\n\n### Minimum configuration for S3 backend\n\nThe minimum configuration for s3proxy is defined by the AWS credentials, to do so define the following env. variables :\n\n* `AWS_REGION` : endpoint used to interact with S3 (ex: eu-west-1)\n* `AWS_ACCESS_KEY` : iam user used access key for S3 (from aws console)\n* `AWS_SECRET_ACCESS_KEY` : iam user used secret for S3 (from aws console)\n\n\n### Minimum configuration for Minio backend\n\nThe minimum configuration for minio backend, you can set the following env. variables or command line options :\n\n* `AWS_REGION` : you have to define this variable even if it is not used for minio (ex: eu-west-1)\n* `S3PROXY_USE_MINIO (or --use-minio)` : minion backend url (ex: localhost:9000)\n* `S3PROXY_MINIO_ACCESS_KEY (or --minio-access-key)` : minio access key (check minio server stdout)\n* `S3PROXY_MINIO_SECRET_KEY (or --minio-secret-key)` : minio secret key (check minio server stdout)\n\n\n### Advanced configuration\n\nYou can customize the http port, define a remote syslog server for centralized logs or define an s3 compatible backend like minio.\nMinio (https://minio.io) is useful when you need to run integration tests or develop on local  without access to S3.\n\nexample :\n\n```\n./s3proxy \\\n    --http-port 80 \\\n    --api-key 3f300bdc-0028-11e8-ba89-0ed5f89f718b \\\n    --use-rsyslog rsyslog:514 \\\n    --use-minio minio:9000 \\\n    --minio-access-key AKIAIOSFODNN7EXAMPLE \\\n    --minio-secret-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY     \n```\n\n\n## Logging Format\n\nBy default, s3proxy logs requests to stdout the following format :\n\n```\n%{time:15:04:05.000} %{shortfunc} ▶ %{level:.4s} %{id:03x} %{message}\n```\n\n\n## Endpoints Documentation\n\ns3proxy responds directly to the following endpoints.\n\n### Health check :\n\n* `/` \n    - returns a 200 OK response with the version number (used for health checks)\n\n### Presigned URL API :\n\n* Create URL for upload : `POST /api/v1/presigned/url/:bucket/:key`    \n   - returns an 200 OK : create a URL for upload\n    \n* Create URL for download : `GET /api/v1/presigned/url/:bucket/:key`  \n    - return an 200 OK : create a URL for download\n\n### Object API\n\n* Delete object : `DELETE /api/v1/object/:bucket/:key`  \n    - return an 200 OK response : delete the object defined by the bucket and the key\n    \n* Bulk Delete object : `POST /api/v1/object/delete/:bucket` with a body containing list of keys \"key=...\u0026key=...\"  \n    - return an 200 OK response : delete the object defined by the bucket and the key\n    - return 400 Bad request if key parameter is missing\n\n* Copy object : `POST /api/v1/object/copy/:bucket/:key?destBucket=...\u0026destKey=...`\n    - return an 200 OK : copy the object defined by the bucket and the key to the destBucket and destKey\n    - return 400 Bad request if destBucket or destKey are missing\n    - return 404 Not Found if the bucket or the key are not found (also destBucket)\n\n### Parameters\n\n* bucket : name of the bucket for example : mybucket\n* key : relative path to the object for example : folder1/folder2/file.txt\n* destBucket : destination bucket for example : mybucket\n* destKey : destination key for example : /folder/file2.txt \n\n## curl examples\n\n* Create a URL for upload :\n\n```\ncurl -H \"Authorization: ${API_KEY}\" -X POST \\ \n    http://localhost:8080/api/v1/presigned/url/my-bucket/folder1/file.txt`\n```\n\nResponse : HTTP CODE 200\n\n`{\"url\" : \"http://...\"}`\n\nYou can use the url in the response to upload file to the backend\n\n`curl -v -H 'Expect:' --upload-file /tmp/file1.txt \"${URL}\"`\n\n* Create a URL for download :\n\n```\ncurl -H \"Authorization: ${API_KEY}\" -X GET \\ \n    http://localhost:8080/api/v1/presigned/url/my-bucket/folder1/file.txt`\n```\n\nResponse : HTTP CODE 200\n\n`{\"url\" : \"http://...\"}`\n\nYou can use the url in the response to download file from the backend\n\n`curl -v -o /tmp/file1.txt \"${URL}\"`\n\n* Delete an object :\n\n```\ncurl -H \"Authorization: ${API_KEY}\" -X DELETE \\ \n    http://localhost:8080/api/v1/object/my-bucket/folder1/file.txt`\n```\n\nResponse : HTTP CODE 200\n\n`{\"response\" : \"ok\"}`\n\n* Bulk delete an object :\n\n```\ncurl -H \"Authorization: ${API_KEY}\" -d \"key=/folder1/file1.txt\u0026key=/folder1/file2.txt\" \\ \n    http://localhost:8080/api/v1/object/my-bucket`\n```\n\nResponse : HTTP CODE 200\n\n`{\"response\" : \"ok\"}`\n\n\n* Copy an object :\n\n```\ncurl -H \"Authorization: ${API_KEY}\" -X POST \\ \n    http://localhost:8080/api/v1/object/copy/my-bucket/folder1/file.txt?destBucket=my-bucket\u0026destKey=/folder1/file2.txt`\n```\n\nResponse : HTTP CODE 200\n\n`{\"response\" : \"ok\"}`\n\n\n* Errors : If an error has occurred then a response code != 200 is sent with a response body\n\n`{\"error\" : \"\u003cmessage of the error\u003e\"}`\n\nIf the object does not exists, it returns always 200.\n\n## Development\n\nTo setup your development environment you can run the docker-compose.\nThis will start minio and a syslog server\n\nTo run :  \n\n`docker-compose -f ./test/docker-compose.yml up -d minio rsyslog createbuckets`\n\nTo stop : \n\n`docker-compose down`\n\n\n## Tests\n\nSeveral kind of tests are available :\n\n* Unit tests\n\n* Integration tests\n\n* End-to-end tests\n\n\n### Unit tests\n\nUnit tests are used to verify the wanted behaviour of the s3proxy.\nThey don't need any external components, they use a fake S3 backend for running tests. \n\nTo run the unit tests : `make test`\n\n\n### Integration tests\n\nIntegration tests are used to verify the integration with a real s3 backend and a rsyslog server. \nIn our tests we are using minio server which provides a S3 compatible API.\n\nTo run the tests : `make integration-test`\n\n\n### End-to-end tests\n\nEnd-to-end tests are used to verify the whole integration with a s3proxy running standalone in its container, a S3 backend (minio) and a rsyslog.\nThe integration tests simulate external call from a program using the s3proxy. \n\nTo run the tests : `make end2end-test VERSION=1.0.0`\n\nwhere `VERSION` is the container version of the s3proxy (make sure to build the image first)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmirakl%2Fs3proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmirakl%2Fs3proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmirakl%2Fs3proxy/lists"}