{"id":45411773,"url":"https://github.com/bluedynamics/zodb-s3blobs","last_synced_at":"2026-04-02T19:17:02.890Z","repository":{"id":338343856,"uuid":"1150968929","full_name":"bluedynamics/zodb-s3blobs","owner":"bluedynamics","description":"Store ZODB blobs in S3-compatible object storage.","archived":false,"fork":false,"pushed_at":"2026-03-30T13:23:35.000Z","size":70,"stargazers_count":6,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-03-30T15:17:49.170Z","etag":null,"topics":["plone","s3","zodb","zope"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/zodb-s3blobs/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bluedynamics.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-05T22:38:58.000Z","updated_at":"2026-03-30T13:23:08.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/bluedynamics/zodb-s3blobs","commit_stats":null,"previous_names":["bluedynamics/zodb-s3blobs"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/bluedynamics/zodb-s3blobs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluedynamics%2Fzodb-s3blobs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluedynamics%2Fzodb-s3blobs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluedynamics%2Fzodb-s3blobs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluedynamics%2Fzodb-s3blobs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bluedynamics","download_url":"https://codeload.github.com/bluedynamics/zodb-s3blobs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluedynamics%2Fzodb-s3blobs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31314272,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T12:59:32.332Z","status":"ssl_error","status_checked_at":"2026-04-02T12:54:48.875Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["plone","s3","zodb","zope"],"created_at":"2026-02-21T23:15:49.985Z","updated_at":"2026-04-02T19:17:02.869Z","avatar_url":"https://github.com/bluedynamics.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# zodb-s3blobs\n\nStore ZODB blobs in S3-compatible object storage.\n\n## Features\n\n- Wraps any ZODB base storage (FileStorage, RelStorage, MappingStorage, ...)\n- Works with any S3-compatible service (AWS S3, MinIO, Ceph, DigitalOcean Spaces)\n- Local LRU filesystem cache for fast reads\n- Full ZODB two-phase commit integration (transactional safety)\n- ZConfig integration for `zope.conf` configuration\n- Supports MVCC storages (`new_instance()`)\n- Garbage collection of orphaned S3 objects during `pack()`\n\n## Installation\n\n```bash\npip install zodb-s3blobs\n```\n\n## Configuration via zope.conf\n\nAdd `%import zodb_s3blobs` and use the `\u003cs3blobstorage\u003e` section wrapping any base storage.\n\n### With FileStorage\n\n```xml\n%import zodb_s3blobs\n\n\u003czodb_db main\u003e\n    \u003cs3blobstorage\u003e\n        bucket-name my-zodb-blobs\n        s3-endpoint-url http://minio:9000\n        s3-access-key $S3_ACCESS_KEY\n        s3-secret-key $S3_SECRET_KEY\n        cache-dir /var/cache/zodb-s3-blobs\n        cache-size 2GB\n        \u003cfilestorage\u003e\n            path /var/lib/zodb/Data.fs\n        \u003c/filestorage\u003e\n    \u003c/s3blobstorage\u003e\n\u003c/zodb_db\u003e\n```\n\nZConfig expands `$VARIABLE` and `${VARIABLE}` from the process environment.\nFor production, consider omitting `s3-access-key` and `s3-secret-key` entirely\nand relying on the boto3 credential chain (IAM roles, instance profiles,\n`~/.aws/credentials`, or the `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`\nenvironment variables).\n\n### With RelStorage\n\nWhen wrapping RelStorage, `zodb-s3blobs` overrides RelStorage's blob handling.\nBlobs go to S3 instead of the `blob_chunk` table. RelStorage still handles object data (pickles) in the RDBMS.\n\n```xml\n%import zodb_s3blobs\n\n\u003czodb_db main\u003e\n    \u003cs3blobstorage\u003e\n        bucket-name my-zodb-blobs\n        cache-dir /var/cache/zodb-s3-blobs\n        cache-size 2GB\n        \u003crelstorage\u003e\n            \u003cpostgresql\u003e\n                dsn dbname='zodb' user='zodb' host='localhost'\n            \u003c/postgresql\u003e\n        \u003c/relstorage\u003e\n    \u003c/s3blobstorage\u003e\n\u003c/zodb_db\u003e\n```\n\n**Note:** When wrapping RelStorage, `zodb-s3blobs` automatically enables\nRelStorage's `LOCK_EARLY` mode (equivalent to `RELSTORAGE_LOCK_EARLY=1`). This\ncauses the database commit lock to be held slightly longer — through the S3 blob\nupload — to ensure transactional consistency. This is the same behavior as\nRelStorage 2.x and is required so that blob uploads complete before the\ntransaction commits.\n\n## Configuration Reference\n\n| Parameter | Default | Description |\n|-----------|---------|-------------|\n| `bucket-name` | *(required)* | S3 bucket name |\n| `s3-prefix` | `\"\"` | Key prefix in bucket |\n| `s3-endpoint-url` | `None` | For MinIO, Ceph, etc. |\n| `s3-region` | `None` | AWS region |\n| `s3-access-key` | `None` | Uses boto3 credential chain if omitted. Use `$ENV_VAR` substitution — never hardcode credentials. |\n| `s3-secret-key` | `None` | Uses boto3 credential chain if omitted. Use `$ENV_VAR` substitution — never hardcode credentials. |\n| `s3-use-ssl` | `true` | Whether to use SSL for S3 connections |\n| `s3-addressing-style` | `auto` | S3 addressing style: `path`, `virtual`, or `auto` |\n| `s3-sse-customer-key` | `None` | Base64-encoded 256-bit key for SSE-C encryption. Requires SSL. |\n| `cache-dir` | *(required)* | Local cache directory path |\n| `cache-size` | `1GB` | Maximum local cache size |\n\n## How It Works\n\n`zodb-s3blobs` uses the same proxy/wrapper pattern as ZODB's built-in `BlobStorage`. It wraps any base storage via `__getattr__` and explicitly overrides all blob methods so they always take precedence.\n\n### Two-Phase Commit Flow\n\n1. **`storeBlob`**: Object data (pickle) is stored in the base storage. The blob file is staged locally.\n2. **`tpc_vote`**: Staged blobs are uploaded to S3. If any upload fails, the transaction aborts cleanly.\n3. **`tpc_finish`**: No S3 operations (this method must not fail per ZODB contract). Staged files are moved into the local cache.\n4. **`tpc_abort`**: Uploaded S3 objects are deleted (best-effort). Local staged files are cleaned up.\n\n### S3 Key Layout\n\n```\nblobs/{oid_hex}/{tid_hex}.blob\n```\n\nWith a configured prefix: `{prefix}/blobs/{oid_hex}/{tid_hex}.blob`\n\n### Local Cache\n\nThe local filesystem cache provides fast reads after the first access. It uses LRU eviction with a background daemon thread that removes the oldest files (by access time) when the total size exceeds the configured maximum. The cache is required -- S3 latency makes direct access impractical for ZODB's synchronous access patterns.\n\n### Garbage Collection\n\nDuring `pack()`, the base storage is packed first, then S3 is scanned for keys referencing OIDs that are no longer reachable. Orphaned keys are deleted. This also cleans up any objects left behind by failed abort operations.\n\n### S3 Bucket Security\n\nEnsure your S3 bucket has appropriate access controls (Block Public Access enabled, restrictive bucket policy). The minimum IAM policy required by `zodb-s3blobs`:\n\n```json\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [{\n        \"Effect\": \"Allow\",\n        \"Action\": [\n            \"s3:GetObject\",\n            \"s3:PutObject\",\n            \"s3:DeleteObject\",\n            \"s3:ListBucket\"\n        ],\n        \"Resource\": [\n            \"arn:aws:s3:::BUCKET_NAME\",\n            \"arn:aws:s3:::BUCKET_NAME/*\"\n        ]\n    }]\n}\n```\n\n### Encryption at Rest (SSE-C)\n\n`zodb-s3blobs` supports SSE-C (Server-Side Encryption with Customer-Provided Keys).\nThe S3 service encrypts/decrypts data using your key but never stores it.\nWorks with AWS S3, Hetzner Object Storage, MinIO (with KES), and other S3-compatible services.\n\n\u003e **Warning — AWS SSE-C deprecation (April 2026):** AWS will disable SSE-C by default\n\u003e on new S3 buckets starting April 2026. Existing buckets are unaffected. For new buckets,\n\u003e you must explicitly enable SSE-C in the bucket policy, or consider migrating to SSE-KMS.\n\u003e If you receive 403 errors with SSE-C configured, this is the likely cause.\n\u003e See the [AWS announcement](https://aws.amazon.com/blogs/storage/advanced-notice-amazon-s3-to-disable-the-use-of-sse-c-encryption-by-default-for-all-new-buckets-and-select-existing-buckets-in-april-2026/) for details.\n\nGenerate a 256-bit key:\n\n```bash\npython -c \"import base64, os; print(base64.b64encode(os.urandom(32)).decode())\"\n```\n\nConfigure via environment variable:\n\n```xml\ns3-sse-customer-key $S3_SSE_KEY\n```\n\n**Important:** If you lose the key, encrypted data is irrecoverable. SSL is required (enforced at startup).\n\n**Security note:** The SSE-C key is held in process memory for the lifetime of the storage instance. In long-running servers, consider using IAM-based encryption (SSE-KMS) instead if memory exposure is a concern. Python's string handling makes secure memory clearing impractical.\n\n## Using with MinIO (dev setup)\n\n**Warning:** The credentials below are MinIO defaults for local development only. Never use default credentials in production.\n\n```yaml\n# docker-compose.yml\nservices:\n  minio:\n    image: minio/minio\n    command: server /data --console-address \":9001\"\n    ports:\n      - \"9000:9000\"\n      - \"9001:9001\"\n    environment:\n      MINIO_ROOT_USER: minioadmin\n      MINIO_ROOT_PASSWORD: minioadmin\n```\n\nCreate the bucket:\n\n```bash\nmc alias set local http://localhost:9000 minioadmin minioadmin\nmc mb local/zodb-blobs\n```\n\n## Development\n\n```bash\ngit clone https://github.com/bluedynamics/zodb-s3blobs.git\ncd zodb-s3blobs\nuv venv\nuv pip install -e \".[test]\"\npytest\n```\n\nFor **reproducible deployments** (production), pin dependencies with a lockfile:\n\n```bash\nuv pip compile pyproject.toml -o requirements.txt\nuv pip install -r requirements.txt\n```\n\n## Source Code and Contributions\n\nThe source code is managed in a Git repository, with its main branches hosted on GitHub.\nIssues can be reported there too.\n\nWe'd be happy to see many forks and pull requests to make this package even better.\nWe welcome AI-assisted contributions, but expect every contributor to fully understand and be able to explain the code they submit.\nPlease don't send bulk auto-generated pull requests.\n\nMaintainers are Jens Klein and the BlueDynamics Alliance developer team.\nWe appreciate any contribution and if a release on PyPI is needed, please just contact one of us.\nWe also offer commercial support if any training, coaching, integration or adaptations are needed.\n\n## License\n\nZPL-2.1\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbluedynamics%2Fzodb-s3blobs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbluedynamics%2Fzodb-s3blobs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbluedynamics%2Fzodb-s3blobs/lists"}