{"id":24870944,"url":"https://github.com/schachte/s3-integrity-checks","last_synced_at":"2026-05-13T07:08:10.581Z","repository":{"id":275110266,"uuid":"925093438","full_name":"Schachte/s3-integrity-checks","owner":"Schachte","description":"Integrity check utilities for s3-compatible storage providers across multiple SDKs","archived":false,"fork":false,"pushed_at":"2025-01-31T08:48:06.000Z","size":24,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-31T09:26:02.237Z","etag":null,"topics":["aws","cloudflare","crc32","r2","s3"],"latest_commit_sha":null,"homepage":"https://ryan-schachte.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Schachte.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-31T08:02:26.000Z","updated_at":"2025-01-31T08:48:09.000Z","dependencies_parsed_at":"2025-01-31T09:26:14.798Z","dependency_job_id":"01054b36-2b54-498f-a1e5-13f6e2f1b011","html_url":"https://github.com/Schachte/s3-integrity-checks","commit_stats":null,"previous_names":["schachte/s3-integrity-checks"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Schachte%2Fs3-integrity-checks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Schachte%2Fs3-integrity-checks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Schachte%2Fs3-integrity-checks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Schachte%2Fs3-integrity-checks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Schachte","download_url":"https://codeload.github.com/Schachte/s3-integrity-checks/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245744843,"owners_count":20665338,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cloudflare","crc32","r2","s3"],"created_at":"2025-02-01T04:17:44.072Z","updated_at":"2026-05-13T07:08:10.524Z","avatar_url":"https://github.com/Schachte.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# S3 Integrity Checker\n\nMinimal tool that facilitates integrity checking of various operations and checksum algorithms for S3-compatible object stores.\n\n# Setup\n```bash\ngit clone git@github.com:Schachte/s3-integrity-checks.git\ncd s3-integrity-checks\nmake install\n```\n\n## Usage\n\n### Options\n- `--bucket`: S3 bucket name (required)\n- `--file`: Path to the file to upload (mutually exclusive with --text)\n- `--text`: Text content to upload (mutually exclusive with --file)\n- `--key`: S3 object key (required)\n- `--endpoint-url`: Custom S3 endpoint URL\n- `--access-key`: AWS access key ID (Python only)\n- `--secret-key`: AWS secret access key (Python only)\n- `--region`: AWS region (default: us-east-1)\n- `--profile`: AWS profile name\n- `--verbose`: Enable verbose output\n- `--part-size`: Size of each part in bytes (Go only, default: 5MB)\n- `--parts`: Comma-separated list of part indices to include in final object (Go only, e.g., '1,2,4')\n- `--upload-empty-part`: Upload an empty part as the final part (Go only)\n\n### Run Tests\n```bash\n# Run all tests\nmake test\nmake test-python\nmake test-go\nmake test-aws-cli\n```\n\n### Manual Usage\n\n#### Text Upload\n```bash\n# Python\npython src/python/integrity.py \\\n  --bucket my-bucket \\\n  --text \"Hello, World\" \\\n  --key test.txt \\\n  --endpoint-url https://my-endpoint.example.com \\\n  --profile default \\\n  --region auto\n\n# Go\n./bin/integrity \\\n  -bucket my-bucket \\\n  -text \"Hello, World\" \\\n  -key test.txt \\\n  -endpoint-url https://my-endpoint.example.com \\\n  -profile default \\\n  -region auto\n\n# AWS CLI\n./src/aws-cli/integrity.sh \\\n  --bucket my-bucket \\\n  --text \"Hello, World\" \\\n  --key test.txt \\\n  --endpoint-url https://my-endpoint.example.com \\\n  --profile default \\\n  --region auto\n```\n\n#### File Upload\n```bash\n# Python\npython src/python/integrity.py \\\n  --bucket my-bucket \\\n  --file path/to/myfile.txt \\\n  --key uploads/myfile.txt \\\n  --endpoint-url https://my-endpoint.example.com \\\n  --profile default \\\n  --region auto\n\n# Go (with new options)\n./bin/integrity \\\n  -bucket my-bucket \\\n  -file path/to/myfile.txt \\\n  -key uploads/myfile.txt \\\n  -endpoint-url https://my-endpoint.example.com \\\n  -profile default \\\n  -region auto \\\n  -part-size 10485760 \\  # Use 10MB parts\n  -parts \"1,2,4\" \\       # Only include parts 1, 2, and 4 in final object\n  -upload-empty-part     # Add empty final part\n\n# AWS CLI\n./src/aws-cli/integrity.sh \\\n  --bucket my-bucket \\\n  --file path/to/myfile.txt \\\n  --key uploads/myfile.txt \\\n  --endpoint-url https://my-endpoint.example.com \\\n  --profile default \\\n  --region auto\n```\n\nAll implementations produce similar output:\n```\nInitiating multipart upload...\nUploading part 1...\n✓ Part 1 uploaded and verified (12/12 bytes)\n\nCompleting multipart upload...\n✓ Upload completed: myfile.txt → my-bucket/uploads/myfile.txt\n\n=== Upload Phase Summary ===\n✓ upload initialization: Upload initiated successfully\n✓ part upload (Part 1): Uploaded and verified (12/12 bytes)\n✓ upload completion: Upload completed successfully\n```\n\n### Verbose Mode\nAdd the `-v` or `--verbose` flag for detailed output:\n\n```bash\n# Python with file upload\npython src/python/integrity.py \\\n  --bucket my-bucket \\\n  --file path/to/myfile.txt \\\n  --key uploads/myfile.txt \\\n  --endpoint-url https://my-endpoint.example.com \\\n  --profile default \\\n  --region auto \\\n  --verbose\n```\n\nVerbose output includes:\n- Configuration details\n- API request/response information\n- CRC32 checksum details\n- Part upload verification\n- Complete upload verification\n- Local vs Server checksum comparison (AWS CLI only)\n\n\n_Example Output_\n```\n✓ Part 9 uploaded and verified (73958028/73958028 bytes)\n\nCompleting multipart upload...\nVerbose mode enabled\n{\n  \"Response\": {\n    \"Body\": {\n      \"Bucket\": \"crc32-3\",\n      \"ETag\": \"03f72a5ae72e7b35cf4b43ee97eab189-9\",\n      \"Key\": \"test.txt\",\n      \"Location\": \"https://endpoint.example.com/crc32-3/test.txt\",\n      \"VersionId\": \"7e6b447187b6544b5a1415b0e19814f8\"\n    },\n    \"Metadata\": {\n      \"RequestId\": \"tx00000000000000000009b-0065c3c8c8-1234567\"\n    }\n  }\n}\n\nChecksum Verification:\nLocal CRC32 (Base64): 8uQxUA==\nServer CRC32 (Base64): 8uQxUA==\n✓ Checksums match\n```\n\n### Environment Variables\n- `S3_ENDPOINT`: Override the default S3 endpoint URL (takes precedence over --endpoint-url)\n\nExample:\n```bash\n# Environment variable takes precedence over command line argument\nS3_ENDPOINT=\"https://my-custom-endpoint.com\" \\\npython src/python/integrity.py \\\n  --endpoint-url \"https://ignored-endpoint.com\" \\\n  --bucket my-bucket \\\n  --file path/to/myfile.txt \\\n  --key uploads/myfile.txt \\\n  --profile default \\\n  --region auto\n```\n\n### Implementation Comparison\n```bash\nmake compare-implementations\n```\n\nThis will:\n1. Run all three implementations with identical parameters\n2. Save outputs to:\n   - tmp/python_output.txt\n   - tmp/go_output.txt\n   - tmp/aws_cli_output.txt\n3. Allow easy comparison of behavior and responses\n\n## Building\n```bash\n# Build all\nmake build\nmake build-python\nmake build-go\nmake build-aws-cli\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fschachte%2Fs3-integrity-checks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fschachte%2Fs3-integrity-checks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fschachte%2Fs3-integrity-checks/lists"}