{"id":43045429,"url":"https://github.com/as/ccp","last_synced_at":"2026-01-31T09:46:38.894Z","repository":{"id":57627951,"uuid":"400933548","full_name":"as/ccp","owner":"as","description":"ccp (cloud copy) copies files across local storage, s3, gs, and http/https protocols. it accelerates web downloads and supports transparent authentication","archived":false,"fork":false,"pushed_at":"2025-04-06T20:43:54.000Z","size":4279,"stargazers_count":7,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-06T21:31:46.834Z","etag":null,"topics":["accelerator","amazon","awscli","copy","download","file","filesystem","google","gs","http","https","list","read","s3","s3-bucket","upload"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/as.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-29T02:32:09.000Z","updated_at":"2025-04-06T20:43:58.000Z","dependencies_parsed_at":"2024-04-18T18:45:58.851Z","dependency_job_id":"5e85c53d-893a-45db-8515-4cb0c78f8ad5","html_url":"https://github.com/as/ccp","commit_stats":null,"previous_names":[],"tags_count":51,"template":false,"template_full_name":null,"purl":"pkg:github/as/ccp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/as%2Fccp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/as%2Fccp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/as%2Fccp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/as%2Fccp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/as","download_url":"https://codeload.github.com/as/ccp/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/as%2Fccp/sbom","scorecard":{"id":210586,"data":{"date":"2025-08-11","repo":{"name":"github.com/as/ccp","commit":"93489346653dea426d3b44e3a0e6731cb86aa21d"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.9,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact v0.4.7 not signed: https://api.github.com/repos/as/ccp/releases/177679680","Warn: release artifact v0.4.7 does not have provenance: https://api.github.com/repos/as/ccp/releases/177679680"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Pinned-Dependencies","score":6,"reason":"dependency not pinned by hash detected -- score normalized to 6","details":["Warn: goCommand not pinned by hash: vendor/google.golang.org/grpc/regenerate.sh:35","Warn: goCommand not pinned by hash: vendor/google.golang.org/grpc/vet.sh:37","Info:   3 out of   5 goCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Vulnerabilities","score":0,"reason":"23 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2022-0635","Warn: Project is vulnerable to: GO-2022-0646","Warn: Project is vulnerable to: GO-2025-3488 / GHSA-6v2p-p543-phr9","Warn: Project is vulnerable to: GO-2023-2153 / GHSA-m425-mq94-257g / GHSA-qppj-fm5r-hxr3","Warn: Project is vulnerable to: GO-2024-2611 / GHSA-8r3f-844c-mc37","Warn: Project is vulnerable to: GO-2022-0236 / GHSA-h86h-8ppg-mxmh","Warn: Project is vulnerable to: GO-2021-0238 / GHSA-83g2-8m93-v3w7","Warn: Project is vulnerable to: GO-2022-0288","Warn: Project is vulnerable to: GO-2022-0969 / GHSA-69cg-p879-7622","Warn: Project is vulnerable to: GO-2022-1144 / GHSA-xrjj-mj9h-534m","Warn: Project is vulnerable to: GO-2023-1571 / GHSA-vvpx-j8f3-3w6h","Warn: Project is vulnerable to: GO-2023-1988 / GHSA-2wrh-6pvc-2jm9","Warn: Project is vulnerable to: GO-2023-2102 / GHSA-4374-p667-p6c8","Warn: Project is vulnerable to: GO-2024-2687 / GHSA-4v7x-pqxf-cx7m","Warn: Project is vulnerable to: GO-2024-3333","Warn: Project is vulnerable to: GO-2025-3503 / GHSA-qxp5-gwg8-xv66","Warn: Project is vulnerable to: GO-2025-3595 / GHSA-vvgc-356p-c3xw","Warn: Project is vulnerable to: GO-2022-0536 / GHSA-39qc-96h7-956f / GHSA-hgr8-6h9x-f7q9","Warn: Project is vulnerable to: GO-2020-0015 / GHSA-5rcv-m4m3-hfh7","Warn: Project is vulnerable to: GO-2021-0113 / GHSA-ppp9-7jff-5vj2","Warn: Project is vulnerable to: GO-2022-1059 / GHSA-69ch-w2m2-3vjp","Warn: Project is vulnerable to: GO-2025-3372 / GHSA-6wxm-mpqj-6jpf","Warn: Project is vulnerable to: GO-2022-0493 / GHSA-p782-xgp4-8hr8"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-17T00:42:38.728Z","repository_id":57627951,"created_at":"2025-08-17T00:42:38.728Z","updated_at":"2025-08-17T00:42:38.728Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28937598,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-31T08:53:31.997Z","status":"ssl_error","status_checked_at":"2026-01-31T08:51:38.521Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["accelerator","amazon","awscli","copy","download","file","filesystem","google","gs","http","https","list","read","s3","s3-bucket","upload"],"created_at":"2026-01-31T09:46:37.844Z","updated_at":"2026-01-31T09:46:38.890Z","avatar_url":"https://github.com/as.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ccp\nccp transparently copies files across different file servers, such as AWS (s3), google cloud storage (gs), http, and local filesystems. It supports download acceleration and streaming output, making it ideal for workflows that require pipelining with accelerated downloads. The latter feature is unavailable with most download accelerators, as they download the entire file as partial data and stitch it together after its downloaded. This is disk-inefficient and makes the file impossible to stream to standard output.\n\nOn an AWS s3 instance, ccp downloaded a 1.5 TiB file in less than 10 minutes on a 2 TiB volume.\n\n## Install\n\n```\ngo install github.com/as/ccp@latest\n```\n\n## Usage\n\n### Copy\n\n```\nccp http://example.com /tmp/file\nccp /tmp/file s3://bucket/file\nccp s3://bucket/file gs://bucket/file\nccp s3://bucket/file s3://bucket/file2\n```\n\n### List\n\n```\nccp -ls s3://bucket/\nccp -ls s3://bucket/dir\n```\n\n\n### Delete\n\n# Note: This is currently non-recursive and only forks for os and s3\n```\nccp -d test.txt\nccp -d s3://bucket/file s3://bucket/file2 ... s3://bucket/fileN\n```\n\n### Test\n\nThe `test` flag will cause `ccp` to verify that it can read and write to the locations without copying data. The first example will check read access only, whereas the second also creates an empty `file2`. This is useful when you need to verify bucket permissions in advance before copying large files, however, it will create empty files.\n\n```\nccp -test s3://bucket/file -\nccp -test s3://bucket/file s3://bucket/file2\n```\n\nYou can also use `ccp -ls` to check access controls, however some schemes will allow accounts to `list` a file and then deny read access to that file.\n\n## Ranges (seek+skip)\n\nUsing the `-seek` and `-skip` flag allows copying byte ranges from source files. This is only supported for some protocols.\n\n```\nccp -q -seek 41 -count 7 http://example.com -\nExample\n```\n\nThis works transparently with any download acceleration used by `ccp` at runtime (byte ranges are repartitioned into blocks and accelerated).\n\n## Feature Matrix\n\nSCHEME | SRC | DST | DELETE | SEEK+SKIP | COMMENT\n-- | -- | -- | --| --| --\ns3 | x | x |x|x| amazon s3\ngs | x | x ||| google cloud storage\nhttp/https || x || x  |  \nftp/sftp |  |||   |  \nssh |  |||   |  \nfile | x | x |x|| local file\n  | x | x |x|| alias for file\n\n## Configuration\n\ngs/s3 are configured through the common environment variables used by the respective sdks\n\n\n## UNIX-like Usage Example\n\nA more in-depth example is necessary to see how ccp can integrate with your existing UNIX tools.\n```\n# Step 1:\n# Create four test files in /tmp/ccp\n\nccp http://example.com /tmp/ccp/file1\nccp http://example.com /tmp/ccp/file2\nccp http://example.com /tmp/ccp/file3\necho hello | ccp - /tmp/ccp/file4\n\n# Step 2:\n# Use ccp -ls to list these files\n# This will generate a tab seperated list of $size, $file\nccp -ls /tmp/ccp\n\n\t1256\t/tmp/ccp/file1\n\t1256\t/tmp/ccp/file2\n\t1256\t/tmp/ccp/file3\n\t6\t/tmp/ccp/file4\n\n# Step 3:\n# To see what ccp is going to do, run it -dry\n# The output can be piped to the shell to execute the copy\n# Or you can just remove the -dry flag.\n#\nccp -l -dry /tmp/ccp /tmp/ccp.bak/\n\n\tccp \"/tmp/ccp/file1\" \"/tmp/ccp.bak/file1\" # 1256\n\tccp \"/tmp/ccp/file2\" \"/tmp/ccp.bak/file2\" # 1256\n\tccp \"/tmp/ccp/file3\" \"/tmp/ccp.bak/file3\" # 1256\n\tccp \"/tmp/ccp/file4\" \"/tmp/ccp.bak/file4\" # 6\n\n# Step 4:\n# Earlier you ran a command to consume stdin and write\n# that to file4. CCP can, in addition to reading file contents,\n# read a newline-seperated list of files from stdin with the \"ccp -l\" flag\n\nccp -ls /tmp/ccp | ccp -l -dry - /tmp/ccp.bak/\n\n\tccp \"/tmp/ccp/file1\" \"/tmp/ccp.bak/file1\" # 1256\n\tccp \"/tmp/ccp/file2\" \"/tmp/ccp.bak/file2\" # 1256\n\tccp \"/tmp/ccp/file3\" \"/tmp/ccp.bak/file3\" # 1256\n\tccp \"/tmp/ccp/file4\" \"/tmp/ccp.bak/file4\" # 6\n\n# Step 5:\n# The above output is identical to \"ccp -l -dry /tmp/ccp /tmp/ccp.bak/\"\n# However, it is more powerful.\n\n\n# Filter out the undesirable \"file4\" from the copy\nccp -ls /tmp/ccp | egrep -v file4 | ccp -l -dry - /tmp/ccp.bak/\n\n\tccp \"/tmp/ccp/file1\" \"/tmp/ccp.bak/file1\" # 1256\n\tccp \"/tmp/ccp/file2\" \"/tmp/ccp.bak/file2\" # 1256\n\tccp \"/tmp/ccp/file3\" \"/tmp/ccp.bak/file3\" # 1256\n\n# Filter files larger than 10 bytes\nccp -ls /tmp/ccp | awk '{ if ($1 \u003c= 10) print }'  | ccp -l -dry - /tmp/ccp.bak/\n\n\tccp \"/tmp/ccp/file4\" \"/tmp/ccp.bak/file4\" # 6\n\n# All of these tricks can be used with gs and s3 buckets as well\n# to copy files across different cloud providers.\n```\n\n## Notes\n\n### Transfer Acceleration\n\n- Since v0.2.4, ccp will download http/https inputs larger than (default 64MiB) by spawning multiple connection (similar to aria2c) and using temporary files. The streaming nature of ccp is preserved and the process is transparent to the user. Disable it with `ccp -slow` to revert to pre-v0.2.4 behavior.\n\n- Additionally, `ccp` will attempt to presign urls and download them over http when possible. If `ccp -secure` is used, it prevents presigned urls from being stripped to http from https.\n\n- Use `ccp -secure -slow` to disable these two optimizations\n\n- The temporary folder used for disk-backed files is $TEMP, or can be overridden on the command line. \n\n### GS to S3 compatibility mode\n\n- The `gs` protocol supports an `s3` compatibility mode wherein an s3 client can speak to a `gs` bucket using the `s3` protocol. This usage mode is not well-documented, and involves generating aws-compatible hmac keys (aka $AWS_ACCESS_KEY_ID, $AWS_SECRET_ACCESS_KEY). This usage mode is not supported and in my experience does not work reliably. To fix this, use `GOOGLE_APPLICATION_CREDENTIALS` or some other credentials auto-detected by the google SDK.\n\n### Cross-Account Copying\n\n- Currently, it is not possible to copy files across accounts using the same scheme. For example, two `s3` buckets managed by different accounts. This is not possible to resolve without introducing a config file that properly initializes credentials based on the source or destination path.\n\n- One workaround is to use the subshell along with two `ccp` processes, each containing the initialized credentials\n\n```\n# assuming ~/env/dev and ~/env/prod contain AWS_SECRET variables\n(. ~/env/dev; ccp s3://dev/file.webm - ) | ( . ~/env/prod; ccp - s3://prod/file.webm)\n```\n\n### Cross-Scheme Directory Structures\n\n- Schemes like `s3` (and possibly `gs`) do not have a concept of directories. Hence, it is possible to create `s3://bucket/file.mp4` and `s3://bucket/file.mp4/file.txt`. The `ccp` program always operates on a directory structure, so `ccp -ls s3://bucket/file.mp4` on such a layout will produce a list of outputs including both files. The best solution is to use a sane directory-based layout to migitage potentially undefined behavior when copying these resources to a local filesystem.\n\n ### China\n\n- This software is not affiliated with or sponsored by the People's Republic of China\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fas%2Fccp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fas%2Fccp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fas%2Fccp/lists"}