{"id":30219116,"url":"https://github.com/engineervix/s3tob2","last_synced_at":"2025-08-28T02:43:24.183Z","repository":{"id":307086623,"uuid":"1028276080","full_name":"engineervix/s3tob2","owner":"engineervix","description":"transfer files from Amazon S3 to Backblaze B2","archived":false,"fork":false,"pushed_at":"2025-07-29T09:50:09.000Z","size":14,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-14T07:47:49.573Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/engineervix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-29T09:36:07.000Z","updated_at":"2025-07-29T09:50:14.000Z","dependencies_parsed_at":"2025-07-29T11:51:57.899Z","dependency_job_id":"a52d77ac-a51d-4f41-baa9-d91dbbb1b4ca","html_url":"https://github.com/engineervix/s3tob2","commit_stats":null,"previous_names":["engineervix/s3tob2"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/engineervix/s3tob2","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/engineervix%2Fs3tob2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/engineervix%2Fs3tob2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/engineervix%2Fs3tob2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/engineervix%2Fs3tob2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/engineervix","download_url":"https://codeload.github.com/engineervix/s3tob2/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/engineervix%2Fs3tob2/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272424372,"owners_count":24932893,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-28T02:00:10.768Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-14T07:47:45.459Z","updated_at":"2025-08-28T02:43:24.161Z","avatar_url":"https://github.com/engineervix.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# S3 to B2 Transfer Script\n\n[![CI](https://github.com/engineervix/s3tob2/actions/workflows/main.yml/badge.svg)](https://github.com/engineervix/s3tob2/actions/workflows/main.yml)\n\nA Python script to efficiently transfer files from Amazon S3 to Backblaze B2 storage with support for concurrent transfers, checksum verification, and resume capability.\n\n## Features\n\n- **Concurrent transfers** - Transfer multiple files in parallel for faster migration\n- **Resume support** - Skip files that already exist in B2 (configurable)\n- **Checksum verification** - Verify file integrity after transfer\n- **Move or copy** - Option to delete files from S3 after successful transfer\n- **Prefix filtering** - Transfer only files matching a specific prefix\n- **Progress logging** - Detailed logs with transfer progress and summary\n- **Error handling** - Graceful handling of transfer failures with detailed error reporting\n\n## Requirements\n\n- Python 3.10+\n- Required packages:\n   - boto3\n   - b2sdk\n   - python-dotenv\n\n  ```bash\n  pip install -r requirements.txt\n  ```\n\n## Configuration\n\nCreate a `.env` file in the same directory as the script with the following variables:\n\n```env\n# Required S3 settings\nS3_BUCKET=your-s3-bucket-name\n\n# Required B2 settings\nB2_BUCKET=your-b2-bucket-name\nB2_APPLICATION_KEY_ID=your-b2-key-id\nB2_APPLICATION_KEY=your-b2-application-key\n\n# Optional S3 settings\nS3_PREFIX=path/to/files/              # Filter objects by prefix (default: \"\")\nAWS_ACCESS_KEY_ID=your-aws-key        # Uses default AWS credentials if not set\nAWS_SECRET_ACCESS_KEY=your-aws-secret  # Uses default AWS credentials if not set\nAWS_REGION=us-east-1                   # AWS region (default: us-east-1)\n\n# Optional transfer settings\nDELETE_FROM_S3=false                   # Delete from S3 after transfer (default: false)\nMAX_WORKERS=5                          # Number of concurrent transfers (default: 5)\nVERIFY_CHECKSUMS=true                  # Verify file integrity (default: true)\nSKIP_EXISTING=true                     # Skip files that exist in B2 (default: true)\n```\n\n## Usage\n\n1. Set up your `.env` file with the required credentials\n2. Run the script:\n   ```bash\n   python s3tob2.py\n   ```\n\nThe script will:\n- List all objects in the S3 bucket (with optional prefix filter)\n- Transfer each file to B2, skipping existing files if configured\n- Log progress to both console and `s3_to_b2_transfer.log`\n- Display a summary of successful and failed transfers\n\n## AWS Credentials\n\nThe script supports two methods for AWS authentication:\n\n1. **Explicit credentials** - Set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in `.env`\n2. **Default credentials** - Leave AWS credentials empty to use:\n   - IAM instance role (when running on EC2)\n   - AWS credentials file (`~/.aws/credentials`)\n   - Environment variables\n\n## B2 Setup\n\n1. Create a B2 application key with the following capabilities:\n   - `listBuckets`\n   - `listFiles`\n   - `readFiles`\n   - `writeFiles`\n   - `deleteFiles` (only needed if `DELETE_FROM_S3=true`)\n\n2. Create or use an existing B2 bucket for the destination\n\n## Transfer Options\n\n### Copy vs Move\n- **Copy mode** (`DELETE_FROM_S3=false`) - Files remain in S3 after transfer\n- **Move mode** (`DELETE_FROM_S3=true`) - Files are deleted from S3 after successful transfer\n\n### Resume Capability\nWhen `SKIP_EXISTING=true`, the script checks if each file already exists in B2 before transferring. This allows you to resume interrupted transfers without re-uploading files.\n\n### Concurrent Transfers\nAdjust `MAX_WORKERS` based on your network capacity and file sizes. Higher values may improve throughput for many small files, while lower values may be better for large files.\n\n## Logging\n\nThe script creates detailed logs in `s3_to_b2_transfer.log` including:\n- File transfer progress\n- Error messages for failed transfers\n- Transfer summary with success/failure counts\n\n## Error Handling\n\nThe script continues transferring remaining files even if individual transfers fail. Failed transfers are logged and counted in the final summary.\n\n## Example Output\n\n```\nConfiguration loaded:\n  S3 Bucket: my-s3-bucket\n  S3 Prefix: 'uploads/2024/'\n  AWS Region: us-east-1\n  B2 Bucket: my-b2-bucket\n  Delete from S3: False\n  Max Workers: 5\n  Verify Checksums: True\n  Skip Existing: True\n\n2024-01-15 10:30:45 - INFO - Found 150 objects in S3 bucket\n2024-01-15 10:30:45 - INFO - Starting transfer of 150 objects...\n2024-01-15 10:30:46 - INFO - Downloading uploads/2024/file1.jpg (1048576 bytes) from S3...\n2024-01-15 10:30:47 - INFO - Uploading uploads/2024/file1.jpg to B2...\n2024-01-15 10:30:48 - INFO - Successfully transferred uploads/2024/file1.jpg\n...\n==================================================\nTRANSFER SUMMARY\n==================================================\nTotal objects: 150\nSuccessful transfers: 148\nFailed transfers: 2\nAction: Copy\n```\n\n## Troubleshooting\n\n### Authentication Errors\n- Verify your AWS and B2 credentials are correct\n- Ensure your B2 application key has the required capabilities\n- Check that the bucket names exist and are accessible\n\n### Transfer Failures\n- Check the log file for detailed error messages\n- Verify network connectivity to both S3 and B2\n- Ensure sufficient permissions on both source and destination\n\n### Performance Issues\n- Adjust `MAX_WORKERS` based on your network and file characteristics\n- For very large files, consider reducing concurrent transfers\n- Monitor memory usage for large file transfers\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fengineervix%2Fs3tob2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fengineervix%2Fs3tob2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fengineervix%2Fs3tob2/lists"}