{"id":37650193,"url":"https://github.com/umccr/copyrite","last_synced_at":"2026-06-04T00:01:10.865Z","repository":{"id":284288171,"uuid":"869794885","full_name":"umccr/copyrite","owner":"umccr","description":"A CLI tool for efficient checksum/copy operations across object stores","archived":false,"fork":false,"pushed_at":"2026-06-02T02:13:45.000Z","size":793,"stargazers_count":2,"open_issues_count":14,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-06-02T03:10:16.630Z","etag":null,"topics":["aws","cloud","gcs","s3"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/umccr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-10-08T22:46:04.000Z","updated_at":"2026-05-12T04:25:42.000Z","dependencies_parsed_at":"2025-03-25T06:21:34.252Z","dependency_job_id":"f7aac37f-afa0-462b-8565-be2951c2be50","html_url":"https://github.com/umccr/copyrite","commit_stats":null,"previous_names":["umccr/cloud-checksum","umccr/copyrite"],"tags_count":31,"template":false,"template_full_name":null,"purl":"pkg:github/umccr/copyrite","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umccr%2Fcopyrite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umccr%2Fcopyrite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umccr%2Fcopyrite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umccr%2Fcopyrite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/umccr","download_url":"https://codeload.github.com/umccr/copyrite/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umccr%2Fcopyrite/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33884734,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-03T02:00:06.370Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cloud","gcs","s3"],"created_at":"2026-01-16T11:32:12.839Z","updated_at":"2026-06-04T00:01:10.836Z","avatar_url":"https://github.com/umccr.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# copyrite\n\nA CLI tool for efficient checksum and copy operations across object stores.\n\ncopyrite can calculate checksums on files in object stores and copy them to other locations.\n\nThe aims of copyrite are to:\n* Calculate multiple checksums in parallel as efficiently as possible.\n  * Multiple checksums are supported which can be computed by reading through the data once.\n* Copy files across object stores with verifiable integrity.\n  * Files are copied concurrently, and verified with part sizes that are as optimal as possible.\n* Avoid doing unnecessary work.\n  * If checksums exists natively in object stores, they are not computed again.\n  * Checksums are saved and propagated through a [`.sums` file][sums] for future operations on the same file.\n* Have a wide range of configurable options for copying tags, metadata, or part sizes.\n* Output detailed statistics on operations that can be consumed by other processes.\n\n## Usage\n\nRun the help command:\n\n```\ncopyrite -h\n```\n\nGenerate checksums for an input file:\n\n```\ncopyrite generate --checksum md5,sha1,sha256 \u003cINPUT_FILE\u003e\n```\n\nAWS style etags are supported, with either a `-\u003cpart_size\u003e` suffix or `-\u003cpart_number\u003e` suffix.\nFor example, `-8` represents splitting the checksum into 8 parts, where as `-8mib` represents\nsplitting the checksum into 8mib chunks.\n\n```\ncopyrite generate --checksum md5-aws-8,md5-aws-8mib \u003cINPUT_FILE\u003e\n```\n\nTo see if files are identical, use the check command:\n\n```\ncopyrite check \u003cINPUT_FILE_1\u003e \u003cINPUT_FILE_2\u003e\n```\n\nObjects on S3 are also supported by using the `s3://bucket/key` syntax:\n\n```\ncopyrite generate --checksum md5-aws-8,md5-aws-8mib s3://bucket/key\ncopyrite check s3://bucket/key1 s3://bucket/key2\n```\n\nCopy files, this supports S3 and local files for source and destination:\n\n```sh\n# Server-side copy in S3.\ncopyrite copy s3://bucket/key1 s3://bucket/key2\n# Local to local\ncopyrite copy local_file1 local_file2\n\n# S3 to local\ncopyrite copy s3://bucket/key1 local_file\n# Local to S3\ncopyrite copy local_file s3://bucket/key1\n```\n\nThe credentials used for the source and destination side are customizable through different providers and environment\nvariables. See the \"Credentials\" section in the long help:\n\n```sh\ncopyrite --help\n```\n\n## Design\n\nThis tool aims to be as efficient and performant as possible when calculating checksums. This means that it only\nreads the data once, and simultaneously calculates desired sets of checksums as it reads through the data. On S3, it always\nuses metadata fields like ETags and additional checksums to obtain data without reading the file if it is able to.\n\nThis tool requires generating `.sums` files to allow checking it. This means that a `generate` command should always be\nperformed before a `check`. To avoid specifying checksums, use `--missing` on the `generate` command to generate only\nthe needed checksums to perform a `check`.\n\n## Tests\n\nRun unit tests using:\n\n```sh\ncargo test --all-features\n```\n\nRun bench marks using:\n\n```sh\ncargo bench --all-features\n```\n\nIntegration tests are ignored by default. They perform operations on an S3 bucket directly, and need to have a\n`COPYRITE_TEST_BUCKET_URI` environment set, to a bucket and prefix that files can be written to. Run the tests\nusing:\n\n```sh\nCOPYRITE_TEST_BUCKET_URI=\"s3://bucket/prefix\" cargo test --all-features -- --ignored\n```\n\nThe endpoint URL can also be set for S3-compatible endpoint tests:\n\n```sh\nCOPYRITE_TEST_BUCKET_URI=\"s3://bucket/prefix\" COPYRITE_TEST_ENDPOINT_URL=\"https://storage.googleapis.com\" cargo test --all-features -- --ignored\n```\n\nThis can also source credentials from a secret, and set compatibility options. The full test environment variables are:\n\n| Variable                      | Description                                                                                                  |\n|-------------------------------|--------------------------------------------------------------------------------------------------------------|\n| `COPYRITE_TEST_BUCKET_URI`    | The S3 bucket and prefix to use.                                                                             |\n| `COPYRITE_TEST_ENDPOINT_URL`  | The S3 endpoint URL.                                                                                         |\n| `COPYRITE_TEST_SECRET`        | The AWS Secrets Manager secret name or ARN, this will also set the credential provider type to `aws-secret`. |\n| `COPYRITE_TEST_REGION`        | The AWS region.                                                                                              |\n| `COPYRITE_TEST_S3_COMPATIBLE` | Set to `true` to enable S3-compatibility.                                                                    |\n\n[sums]: docs/ARCHITECTURE.md#the-sums-file\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fumccr%2Fcopyrite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fumccr%2Fcopyrite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fumccr%2Fcopyrite/lists"}