{"id":35381619,"url":"https://github.com/tigrisdata/tigris-boto3-ext","last_synced_at":"2026-05-02T07:10:32.606Z","repository":{"id":319019723,"uuid":"1076572043","full_name":"tigrisdata/tigris-boto3-ext","owner":"tigrisdata","description":"Extend boto3 with Tigris-specific features","archived":false,"fork":false,"pushed_at":"2026-04-03T23:17:01.000Z","size":65,"stargazers_count":1,"open_issues_count":6,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-03T23:25:09.924Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://www.tigrisdata.com/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tigrisdata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-15T03:51:38.000Z","updated_at":"2026-02-12T21:33:10.000Z","dependencies_parsed_at":"2025-10-17T19:46:42.255Z","dependency_job_id":"b5005d39-9f54-4648-b711-2d1b8410299f","html_url":"https://github.com/tigrisdata/tigris-boto3-ext","commit_stats":null,"previous_names":["tigrisdata/tigris-boto3-ext"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/tigrisdata/tigris-boto3-ext","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tigrisdata%2Ftigris-boto3-ext","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tigrisdata%2Ftigris-boto3-ext/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tigrisdata%2Ftigris-boto3-ext/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tigrisdata%2Ftigris-boto3-ext/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tigrisdata","download_url":"https://codeload.github.com/tigrisdata/tigris-boto3-ext/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tigrisdata%2Ftigris-boto3-ext/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31504897,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T03:10:19.677Z","status":"ssl_error","status_checked_at":"2026-04-07T03:10:13.982Z","response_time":105,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-02T05:10:33.010Z","updated_at":"2026-05-02T07:10:32.598Z","avatar_url":"https://github.com/tigrisdata.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tigris-boto3-ext\n\n[![CI](https://github.com/tigrisdata/tigris-boto3-ext/actions/workflows/ci.yml/badge.svg)](https://github.com/tigrisdata/tigris-boto3-ext/actions/workflows/ci.yml)\n[![codecov](https://codecov.io/gh/tigrisdata/tigris-boto3-ext/branch/main/graph/badge.svg)](https://codecov.io/gh/tigrisdata/tigris-boto3-ext)\n[![Python Version](https://img.shields.io/pypi/pyversions/tigris-boto3-ext.svg)](https://pypi.org/project/tigris-boto3-ext/)\n[![PyPI version](https://badge.fury.io/py/tigris-boto3-ext.svg)](https://badge.fury.io/py/tigris-boto3-ext)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\nExtend boto3 with Tigris-specific features like snapshots and bucket forking, while maintaining full boto3 compatibility.\n\n## Features\n\n- **Bundle API**: Fetch thousands of objects in a single request as a streaming tar archive — designed for ML training workloads\n- **Snapshot Support**: Create, list, and read from bucket snapshots\n- **Bucket Forking**: Create forked buckets from existing buckets or snapshots\n- **Object Rename**: Rename objects in place without rewriting their data\n- **Multiple Usage Patterns**: Context managers, decorators, helper functions, or wrapper client\n- **Zero Configuration**: Works with existing boto3 code\n- **Type Safe**: Full type hints for IDE support\n- **Pythonic API**: Uses familiar Python patterns\n\n## Installation\n\n```bash\npip install tigris-boto3-ext\n```\n\n## Usage Patterns\n\n### 1. Context Managers (Recommended)\n\n#### Enable Snapshots for Bucket Creation\n\n```python\nfrom tigris_boto3_ext import TigrisSnapshotEnabled\n\nwith TigrisSnapshotEnabled(s3_client):\n    s3_client.create_bucket(Bucket='my-snapshot-bucket')\n```\n\n#### Work with Snapshots\n\n```python\nfrom tigris_boto3_ext import TigrisSnapshot\n\n# List snapshots for a bucket\nwith TigrisSnapshot(s3_client, 'my-bucket'):\n    snapshots = s3_client.list_buckets()\n\n# Read objects from a specific snapshot\nwith TigrisSnapshot(s3_client, 'my-bucket', snapshot_version='12345'):\n    obj = s3_client.get_object(Bucket='my-bucket', Key='file.txt')\n    objects = s3_client.list_objects_v2(Bucket='my-bucket')\n```\n\n#### Create Forked Buckets\n\n```python\nfrom tigris_boto3_ext import TigrisFork\n\n# Fork from current state\nwith TigrisFork(s3_client, 'source-bucket'):\n    s3_client.create_bucket(Bucket='forked-bucket')\n\n# Fork from specific snapshot\nwith TigrisFork(s3_client, 'source-bucket', snapshot_version='12345'):\n    s3_client.create_bucket(Bucket='forked-from-snapshot')\n```\n\n#### Rename Objects\n\nTigris implements rename as a `copy_object` request plus the `X-Tigris-Rename: true`\nheader — no data is rewritten, only the key changes. Keep the context tight so\nunrelated `copy_object` calls are not turned into renames.\n\n```python\nfrom tigris_boto3_ext import TigrisRename\n\nwith TigrisRename(s3_client):\n    s3_client.copy_object(\n        Bucket='my-bucket',\n        CopySource='my-bucket/old-name.txt',\n        Key='new-name.txt',\n    )\n```\n\n### 2. Decorators\n\n```python\nfrom tigris_boto3_ext import snapshot_enabled, with_snapshot, forked_from, with_rename\n\n@snapshot_enabled\ndef create_snapshot_enabled_bucket(s3_client, bucket_name):\n    return s3_client.create_bucket(Bucket=bucket_name)\n\n# List available snapshots\n@with_snapshot('my-bucket')\ndef list_snapshots(s3_client):\n    return s3_client.list_buckets()\n\n# Read from specific snapshot\n@with_snapshot('my-bucket', snapshot_version='12345')\ndef read_from_snapshot(s3_client, key):\n    return s3_client.get_object(Bucket='my-bucket', Key=key)\n\n@forked_from('source-bucket', snapshot_version='12345')\ndef create_my_fork(s3_client, new_bucket):\n    return s3_client.create_bucket(Bucket=new_bucket)\n\n@with_rename\ndef rename_file(s3_client, bucket, old_key, new_key):\n    return s3_client.copy_object(\n        Bucket=bucket,\n        CopySource=f'{bucket}/{old_key}',\n        Key=new_key,\n    )\n\n# Use the decorated functions\ncreate_snapshot_enabled_bucket(s3_client, 'my-bucket')\nsnapshots = list_snapshots(s3_client)\nobj = read_from_snapshot(s3_client, 'file.txt')\ncreate_my_fork(s3_client, 'my-fork')\nrename_file(s3_client, 'my-bucket', 'old.txt', 'new.txt')\n```\n\n### 3. Helper Functions\n\n```python\nfrom tigris_boto3_ext import (\n    create_snapshot_bucket,\n    create_snapshot,\n    list_snapshots,\n    create_fork,\n    get_object_from_snapshot,\n    get_snapshot_version,\n    list_objects_from_snapshot,\n    head_object_from_snapshot,\n    has_snapshot_enabled,\n    get_bucket_info,\n    rename_object,\n)\n\n# Create snapshot-enabled bucket\ncreate_snapshot_bucket(s3_client, 'my-bucket')\n\n# Check if bucket has snapshots enabled\nif has_snapshot_enabled(s3_client, 'my-bucket'):\n    print(\"Snapshots are enabled!\")\n\n# Get comprehensive bucket information\ninfo = get_bucket_info(s3_client, 'my-bucket')\nprint(f\"Snapshot enabled: {info['snapshot_enabled']}\")\n\n# Create snapshots\nresult = create_snapshot(s3_client, 'my-bucket', snapshot_name='backup-1')\nversion = get_snapshot_version(result)\n\n# List snapshots\nsnapshots = list_snapshots(s3_client, 'my-bucket')\n\n# Create forks\ncreate_fork(s3_client, 'new-bucket', 'source-bucket', snapshot_version=version)\n\n# Access snapshot data\nobj = get_object_from_snapshot(s3_client, 'my-bucket', 'file.txt', version)\nobjects = list_objects_from_snapshot(s3_client, 'my-bucket', '12345', Prefix='data/')\nmetadata = head_object_from_snapshot(s3_client, 'my-bucket', 'file.txt', '12345')\n\n# Rename an object in place (no data rewrite)\nrename_object(s3_client, 'my-bucket', 'old-name.txt', 'new-name.txt')\n```\n\n## Complete Examples\n\n### Example 1: Backup and Restore Workflow\n\n```python\nimport boto3\nfrom tigris_boto3_ext import (\n    create_snapshot_bucket,\n    create_snapshot,\n    list_snapshots,\n    create_fork,\n    get_snapshot_version,\n)\n\ns3 = boto3.client('s3')\n\n# Create a snapshot-enabled bucket\ncreate_snapshot_bucket(s3, 'production-data')\n\n# Add some data\ns3.put_object(Bucket='production-data', Key='important.txt', Body=b'critical data')\n\n# Create a snapshot\nsnapshot_result = create_snapshot(s3, 'production-data', snapshot_name='daily-backup')\nsnapshot_version = get_snapshot_version(snapshot_result)\n\n# List all snapshots\nsnapshots = list_snapshots(s3, 'production-data')\nfor bucket in snapshots.get('Buckets', []):\n    print(f\"Snapshot: {bucket['Name']}\")\n\n# Restore from snapshot by creating a fork\ncreate_fork(s3, 'restored-data', 'production-data', snapshot_version=snapshot_version)\n```\n\n### Example 2: Testing with Snapshot Isolation\n\n```python\nimport boto3\nfrom tigris_boto3_ext import create_fork, create_snapshot, get_snapshot_version\n\ns3 = boto3.client('s3')\n\n# Create a snapshot of production data\nsnapshot_result = create_snapshot(s3, 'production-data', snapshot_name='test-snapshot')\nsnapshot_version = get_snapshot_version(snapshot_result)\n\n# Fork for testing (isolated copy)\ncreate_fork(s3, 'test-data', 'production-data', snapshot_version=snapshot_version)\n\n# Run tests against test-db without affecting production\ns3.put_object(Bucket='test-data', Key='test-data.txt', Body=b'test data')\n\n# Clean up test bucket when done\ns3.delete_bucket(Bucket='test-data')\n```\n\n### Example 3: Time-Travel Queries\n\n```python\nimport boto3\nfrom tigris_boto3_ext import get_object_from_snapshot, list_objects_from_snapshot\n\ns3 = boto3.client('s3')\n\n# Get object as it was at a specific snapshot\nhistorical_obj = get_object_from_snapshot(\n    s3,\n    'my-bucket',\n    'config.json',\n    snapshot_version='12345'\n)\nold_config = historical_obj['Body'].read()\n\n# List all objects in historical snapshot\nhistorical_objects = list_objects_from_snapshot(\n    s3,\n    'my-bucket',\n    snapshot_version='12345',\n    Prefix='logs/2024/'\n)\n\nfor obj in historical_objects.get('Contents', []):\n    print(f\"Historical object: {obj['Key']}\")\n```\n\n### Example 4: Retrieving Bucket Snapshot and Fork Information\n\n```python\nimport boto3\nfrom tigris_boto3_ext import (\n    create_snapshot_bucket,\n    create_snapshot,\n    create_fork,\n    get_snapshot_version,\n    has_snapshot_enabled,\n    get_bucket_info,\n)\n\ns3 = boto3.client('s3')\n\n# Check if a bucket has snapshots enabled\nbucket_name = 'my-bucket'\n\ncreate_snapshot_bucket(s3, bucket_name)\n\nif has_snapshot_enabled(s3, bucket_name):\n    print(f\"✓ Snapshots are enabled for {bucket_name}\")\nelse:\n    print(f\"✗ Snapshots are not enabled for {bucket_name}\")\n\n# Get comprehensive bucket information\ninfo = get_bucket_info(s3, bucket_name)\nprint(f\"Snapshot enabled: {info['snapshot_enabled']}\")\n\n# Example: Check fork lineage\nsource_bucket = 'production-data'\ncreate_snapshot_bucket(s3, source_bucket)\n\n# Create a snapshot\nsnapshot_result = create_snapshot(s3, source_bucket, snapshot_name='v1')\nsnapshot_version = get_snapshot_version(snapshot_result)\n\n# Create a fork\nforked_bucket = 'test-data'\ncreate_fork(s3, forked_bucket, source_bucket, snapshot_version=snapshot_version)\n\n# Inspect the fork\nfork_info = get_bucket_info(s3, forked_bucket)\nprint(f\"Forked from: {fork_info['fork_source_bucket']}\")\nprint(f\"Snapshot version: {fork_info['fork_source_snapshot']}\")\n```\n\n### Example 5: Bundle API — Fetch Multiple Objects in One Request\n\n```python\nimport tarfile\nimport boto3\nfrom tigris_boto3_ext import bundle_objects, BundleError, BUNDLE_ON_ERROR_FAIL\n\ns3 = boto3.client('s3')\n\n# Fetch a batch of training images as a streaming tar archive\nkeys = [f\"dataset/train/img_{i:05d}.jpg\" for i in range(1000)]\nresponse = bundle_objects(s3, 'my-dataset-bucket', keys)\n\nwith tarfile.open(fileobj=response, mode=\"r|\") as tar:\n    for member in tar:\n        if member.name == \"__bundle_errors.json\":\n            continue  # skip the error manifest\n        f = tar.extractfile(member)\n        if f is not None:\n            image_bytes = f.read()\n            # feed to training pipeline\n\n# Use fail mode for inference where every object must be present\ntry:\n    response = bundle_objects(\n        s3, 'my-bucket', keys, on_error=BUNDLE_ON_ERROR_FAIL\n    )\nexcept BundleError as e:\n    print(f\"Bundle failed (HTTP {e.status_code}): {e.body}\")\n```\n\nSee [`examples/bundle_usage.py`](examples/bundle_usage.py) for more patterns including error handling, response metadata, and ML training batches.\n\n## How It Works\n\nThis library uses boto3's event system to inject Tigris-specific headers into S3 API requests:\n\n### Request Headers (Sent to Tigris)\n\n- **`X-Tigris-Enable-Snapshot: true`** - Enables snapshot support for bucket creation\n- **`X-Tigris-Snapshot: true; name=\u003cname\u003e`** - Creates a snapshot\n- **`X-Tigris-Snapshot: \u003cbucket_name\u003e`** - Lists snapshots for a bucket\n- **`X-Tigris-Snapshot-Version: \u003cversion\u003e`** - Reads from specific snapshot version\n- **`X-Tigris-Fork-Source-Bucket: \u003cbucket\u003e`** - Specifies fork source\n- **`X-Tigris-Fork-Source-Bucket-Snapshot: \u003cversion\u003e`** - Forks from specific snapshot\n- **`X-Tigris-Rename: true`** - Turns a `CopyObject` request into an in-place rename\n\n### Response Headers (Returned by Tigris)\n\nThe following custom headers are returned in HeadBucket responses and can be accessed via `get_bucket_info()` and `has_snapshot_enabled()`:\n\n- **`X-Tigris-Enable-Snapshot: true`** - Present when snapshots are enabled for the bucket\n- **`X-Tigris-Fork-Source-Bucket: \u003cbucket_name\u003e`** - Present on forked buckets, indicates the parent bucket\n- **`X-Tigris-Fork-Source-Bucket-Snapshot: \u003cversion\u003e`** - Present on forked buckets, indicates the snapshot version\n\nThe library registers event handlers on `before-sign.s3.*` events to add request headers transparently.\n\n## Requirements\n\n- Python 3.9+\n- boto3 \u003e= 1.26.0\n\n## Development\n\n### Setup\n\n```bash\n# Clone the repository\ngit clone https://github.com/tigrisdata/tigris-boto3-ext.git\ncd tigris-boto3-ext\n\n# Install with dev dependencies using uv\nuv sync --all-extras\n\n# Or with pip\npip install -e \".[dev]\"\n```\n\n### Running Tests\n\n#### Integration Tests\n\nIntegration tests run against a real Tigris S3 service. See [`tests/integration/README.md`](tests/integration/README.md) for detailed setup instructions.\n\n```bash\n# Set up environment variables\nexport AWS_ENDPOINT_URL_S3=\"https://t3.storage.dev\"\nexport AWS_ACCESS_KEY_ID=\"your-access-key\"\nexport AWS_SECRET_ACCESS_KEY=\"your-secret-key\"\n\n# Run integration tests\nuv run pytest tests/integration/ -v\n```\n\n### Code Quality\n\n```bash\n# Type checking\nuv run mypy tigris_boto3_ext\n\n# Linting\nuv run ruff check tigris_boto3_ext\n\n# Auto-fix linting issues\nuv run ruff check --fix tigris_boto3_ext\n\n# Code formatting\nuv run ruff format tigris_boto3_ext\n\n# Check formatting without making changes\nuv run ruff format --check tigris_boto3_ext\n```\n\n## License\n\nApache-2.0\n\n## Contributing\n\nContributions welcome! Please open an issue or PR on GitHub.\n\n## Support\n\nFor issues and questions:\n\n- GitHub Issues: \u003chttps://github.com/tigrisdata/tigris-boto3-ext/issues\u003e\n- Documentation: \u003chttps://www.tigrisdata.com/docs\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftigrisdata%2Ftigris-boto3-ext","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftigrisdata%2Ftigris-boto3-ext","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftigrisdata%2Ftigris-boto3-ext/lists"}