{"id":18963852,"url":"https://github.com/janeliascicomp/x2s3","last_synced_at":"2025-12-11T18:35:58.842Z","repository":{"id":245340731,"uuid":"815673691","full_name":"JaneliaSciComp/x2s3","owner":"JaneliaSciComp","description":"S3 Proxy Service","archived":false,"fork":false,"pushed_at":"2025-03-10T14:21:01.000Z","size":219,"stargazers_count":7,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-29T07:51:28.732Z","etag":null,"topics":["s3","s3-bucket","s3-storage"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JaneliaSciComp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-15T19:46:40.000Z","updated_at":"2025-03-10T14:21:05.000Z","dependencies_parsed_at":"2024-06-21T14:57:47.002Z","dependency_job_id":"8f9165cd-5c0e-4520-bc80-6fa4d1fcbe69","html_url":"https://github.com/JaneliaSciComp/x2s3","commit_stats":null,"previous_names":["janeliascicomp/jproxy"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JaneliaSciComp%2Fx2s3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JaneliaSciComp%2Fx2s3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JaneliaSciComp%2Fx2s3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JaneliaSciComp%2Fx2s3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JaneliaSciComp","download_url":"https://codeload.github.com/JaneliaSciComp/x2s3/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249201117,"owners_count":21229004,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["s3","s3-bucket","s3-storage"],"created_at":"2024-11-08T14:22:03.511Z","updated_at":"2025-12-11T18:35:58.837Z","avatar_url":"https://github.com/JaneliaSciComp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# x2s3\n\n[![Python CI](https://github.com/JaneliaSciComp/x2s3/actions/workflows/python-ci.yml/badge.svg)](https://github.com/JaneliaSciComp/x2s3/actions/workflows/python-ci.yml)\n\nRESTful web service which makes any storage system *X* available as an S3-compatible REST API, hence the name \"X to S3\". It was initially built to support cloud-compatible microscopy image viewers such as [N5 Viewer](https://github.com/saalfeldlab/n5-viewer) (BigDataViewer) and [Neuroglancer](https://github.com/google/neuroglancer).\n\nAt Janelia, we use **x2s3** to make private buckets on Seagate Lyve appear public, and to proxy internal resources (e.g. VAST S3). It can also be used as a pop-up file service for quickly viewing local images in BigDataViewer or Neuroglancer.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/JaneliaSciComp/x2s3/main/docs/use_cases.png\"\u003e\n\u003c/p\u003e\n\n# Features\n\n* Extensible support for backend storage systems\n* Optional web-based bucket explorer\n* Hidden buckets\n* Partial buckets (chroot-like prefixes)\n* Non-blocking object streaming\n\nInspired by S3 proxies such as [oxyno-zeta/s3-proxy](https://github.com/oxyno-zeta/s3-proxy) and [pottava/aws-s3-proxy](https://github.com/pottava/aws-s3-proxy), this service goes a step further to implement enough of the [AWS S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/Type_API_Reference.html) to be useable by AWS clients, such as  BigDataViewer. The S3 proxy implements which do this well (e.g. [gaul/s3proxy](https://github.com/gaul/s3proxy)) only proxy a single bucket at a time.\n\nS3 endpoints implemented:\n* [GetBucketAcl](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketAcl.html)\n* [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html)\n* [HeadObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html)\n* [ListBuckets](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html)\n* [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html)\n\nS3 features omitted:\n* Permissions\n* Encryption\n* Versioning\n* RequestPayer\n* etc.\n\n# Running\n\nCreate a `config.yaml` file that contains all of the buckets you want to serve. You can get started quickly by using the provided example template:\n\n```bash\ncp config.template.yaml config.yaml\n```\n\nSee the [documentation](docs/Config.md) for more information about the configuration file.\n\nThe simplest way to run the service is to use Docker:\n\n```bash\ndocker run -it -p 8000:8000 -v ./config.yaml:/app/x2s3/config.yaml ghcr.io/janeliascicomp/x2s3:latest\n```\n\nYou can also run the service with Python/Uvicorn directly. See the [development documentation](docs/Development.md) for more information on setting that up.\n\n\n## Production Deployment\n\nFor production deployments, we recommend using an orchestrator (like Docker Compose) to run the prebuilt Docker container along with an Nginx reverse proxy which provides caching and TLS termination.\n\nCreate a `./docker/.env` file that looks like this:\n\n```bash\nCONFIG_FILE=/path/to/config.yaml\nVAR_DIR=/path/to/var/dir\nCERT_DIR=/path/to/certs\nNGINX_CACHE_DIR=/path/to/cache\n```\n\nThese properties configure the service as follows:\n* `CONFIG_FILE`: path to the `config.yaml` settings file\n* `VAR_DIR`: optional path to the var directory containing access keys referenced by `config.yaml`\n* `CERT_FILE`: optional path to the SSL cert file\n* `KEY_FILE`: optional path to the SSL key file\n* `NGINX_CACHE_DIR`: path for Nginx response caching (you can disable caching by editing `nginx.conf`)\n\nNow you can bring up the container:\n\n```bash\ncd docker/\ndocker compose up -d\n```\n\n# Additional Documentation\n\n* [Configuation](docs/Config.md) - how to use `config.yaml` to configure the service\n* [Development](docs/Development.md) - notes on developing the service codebase\n\n\n# Attributions\n\n* Proxy icons created by [Uniconlabs - Flaticon](https://www.flaticon.com/free-icons/proxy)\n* [AWS S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/Type_API_Reference.html) Reference\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaneliascicomp%2Fx2s3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaneliascicomp%2Fx2s3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaneliascicomp%2Fx2s3/lists"}