{"id":36470694,"url":"https://github.com/evalphobia/cloud-label-uploader","last_synced_at":"2026-01-12T00:36:31.721Z","repository":{"id":49728211,"uuid":"135277020","full_name":"evalphobia/cloud-label-uploader","owner":"evalphobia","description":"Download file from CSV file via http; Create training CSV file for AutoML and Sagemaker Ground Truth; Upload file to GCS and S3","archived":false,"fork":false,"pushed_at":"2021-06-10T08:49:46.000Z","size":56,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-06-20T12:48:30.843Z","etag":null,"topics":["automl","csv","gcs","image"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/evalphobia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-29T10:02:14.000Z","updated_at":"2021-06-10T08:49:32.000Z","dependencies_parsed_at":"2022-09-06T01:30:58.552Z","dependency_job_id":null,"html_url":"https://github.com/evalphobia/cloud-label-uploader","commit_stats":null,"previous_names":["evalphobia/csv-file-downloader"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/evalphobia/cloud-label-uploader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evalphobia%2Fcloud-label-uploader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evalphobia%2Fcloud-label-uploader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evalphobia%2Fcloud-label-uploader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evalphobia%2Fcloud-label-uploader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/evalphobia","download_url":"https://codeload.github.com/evalphobia/cloud-label-uploader/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evalphobia%2Fcloud-label-uploader/sbom","scorecard":{"id":385360,"data":{"date":"2025-08-11","repo":{"name":"github.com/evalphobia/cloud-label-uploader","commit":"ffdf50e65210fa34a6f4ab4949468215892daa83"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.3,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/12 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:18: update your workflow using https://app.stepsecurity.io/secureworkflow/evalphobia/cloud-label-uploader/release.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:28: update your workflow using https://app.stepsecurity.io/secureworkflow/evalphobia/cloud-label-uploader/release.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release.yml:35: update your workflow using https://app.stepsecurity.io/secureworkflow/evalphobia/cloud-label-uploader/release.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:18: update your workflow using https://app.stepsecurity.io/secureworkflow/evalphobia/cloud-label-uploader/test.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:28: update your workflow using https://app.stepsecurity.io/secureworkflow/evalphobia/cloud-label-uploader/test.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:40: update your workflow using https://app.stepsecurity.io/secureworkflow/evalphobia/cloud-label-uploader/test.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:50: update your workflow using https://app.stepsecurity.io/secureworkflow/evalphobia/cloud-label-uploader/test.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/test.yml:55: update your workflow using https://app.stepsecurity.io/secureworkflow/evalphobia/cloud-label-uploader/test.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:65: update your workflow using https://app.stepsecurity.io/secureworkflow/evalphobia/cloud-label-uploader/test.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:70: update your workflow using https://app.stepsecurity.io/secureworkflow/evalphobia/cloud-label-uploader/test.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/test.yml:74: update your workflow using https://app.stepsecurity.io/secureworkflow/evalphobia/cloud-label-uploader/test.yml/master?enable=pin","Info:   0 out of   8 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   3 third-party GitHubAction dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/release.yml:1","Warn: no topLevel permission defined: .github/workflows/test.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 7 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":9,"reason":"1 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2025-3488 / GHSA-6v2p-p543-phr9"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-18T16:28:10.600Z","repository_id":49728211,"created_at":"2025-08-18T16:28:10.601Z","updated_at":"2025-08-18T16:28:10.601Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28329805,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T00:36:25.062Z","status":"ssl_error","status_checked_at":"2026-01-12T00:36:15.229Z","response_time":60,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automl","csv","gcs","image"],"created_at":"2026-01-12T00:36:31.316Z","updated_at":"2026-01-12T00:36:31.716Z","avatar_url":"https://github.com/evalphobia.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"cloud-label-uploader\n----\n\n[![GoDoc][1]][2] [![License: MIT][3]][4] [![Release][5]][6] [![Build Status][7]][8] [![Go Report Card][13]][14] [![Code Climate][19]][20] [![BCH compliance][21]][22]\n\n[1]: https://godoc.org/github.com/evalphobia/cloud-label-uploader?status.svg\n[2]: https://godoc.org/github.com/evalphobia/cloud-label-uploader\n[3]: https://img.shields.io/badge/License-MIT-blue.svg\n[4]: LICENSE.md\n[5]: https://img.shields.io/github/release/evalphobia/cloud-label-uploader.svg\n[6]: https://github.com/evalphobia/cloud-label-uploader/releases/latest\n[7]: https://travis-ci.org/evalphobia/cloud-label-uploader.svg?branch=master\n[8]: https://travis-ci.org/evalphobia/cloud-label-uploader\n[9]: https://coveralls.io/repos/evalphobia/cloud-label-uploader/badge.svg?branch=master\u0026service=github\n[10]: https://coveralls.io/github/evalphobia/cloud-label-uploader?branch=master\n[11]: https://codecov.io/github/evalphobia/cloud-label-uploader/coverage.svg?branch=master\n[12]: https://codecov.io/github/evalphobia/cloud-label-uploader?branch=master\n[13]: https://goreportcard.com/badge/github.com/evalphobia/cloud-label-uploader\n[14]: https://goreportcard.com/report/github.com/evalphobia/cloud-label-uploader\n[15]: https://img.shields.io/github/downloads/evalphobia/cloud-label-uploader/total.svg?maxAge=1800\n[16]: https://github.com/evalphobia/cloud-label-uploader/releases\n[17]: https://img.shields.io/github/stars/evalphobia/cloud-label-uploader.svg\n[18]: https://github.com/evalphobia/cloud-label-uploader/stargazers\n[19]: https://codeclimate.com/github/evalphobia/cloud-label-uploader/badges/gpa.svg\n[20]: https://codeclimate.com/github/evalphobia/cloud-label-uploader\n[21]: https://bettercodehub.com/edge/badge/evalphobia/cloud-label-uploader?branch=master\n[22]: https://bettercodehub.com/\n\n`cloud-label-uploader` download and upload files from url in CSV.\nAnd create labeled CSV file for machine learning of Google Cloud AutoML / AWS Sagemaker.\n\n# Installation\n\nInstall cloud-label-uploader by command below,\n\n```bash\n$ go get github.com/evalphobia/cloud-label-uploader\n```\n\n# Usage\n\n## root command\n\n```bash\n$ cloud-label-uploader\nCommands:\n\n  help       show help\n  download   Download files from --file csv\n  list       Create list file from --input dir images\n  upload     Upload files to Cloud Bucket(S3, GCS) from --input dir\n  vott       Create object-detection list file from VoTT results\n```\n\n## download command\n\n`download` downloads file from url in CSV file.\n\n```bash\n$ cloud-label-uploader help download\nDownload files from --file csv\n\nOptions:\n\n  -h, --help           display help information\n  -i, --input         *input CSV file --input='/path/to/dir/input.csv'\n  -n, --name          *column name for filename --name='name'\n  -l, --label         *column name for label --label='group'\n  -u, --url           *column name for URL --url='path'\n  -m, --parallel[=2]   parallel number (multiple download) --parallel=2\n  -o, --output         outout dir --output='/path/to/dir/'\n```\n\n```bash\n# Save CSV file with name, label and URL.\n$ cat my_file_list.csv\n\nid,label,image_url\n1,cat,http://example.com/foo.jpg\n2,dog,http://example.com/bar.jpg\n3,cat,https://example.com/foo2.JPG\n4,human,https://example.com/baz.png?q=1\n5,human,https://example.com/baz2.png\n\n\n# Download files from URL in CSV.\n$ cloud-label-uploader download -i ./my_file_list.csv -o ./save -n \"id\" -l \"label\" -u \"image_url\"\n\n\n# Chech downloaded files.\n$ tree ./save\n\n./save\n├── cat\n│   ├── 1.jpg\n│   ├── 3.JPG\n├── dog\n│   ├── 2.jpg\n└── human\n    ├── 4.png\n    └── 5.png\n\n3 directories, 5 files\n```\n\n\n## list command\n\n`list` creates a CSV file from images files, containing label and expected path on GCS/S3.\n from url in CSV file. (for multi label classification)\n\n```bash\n$ cloud-label-uploader help list\nCreate list file from --input dir images\n\nOptions:\n\n  -h, --help                      display help information\n  -i, --input                    *image dir path --input='/path/to/image_dir'\n  -o, --output[=./output.csv]    *output CSV file path --output='./output.csv'\n  -a, --all                       use all files\n  -t, --type[=jpg,jpeg,png,gif]   comma separate file extensions --type='jpg,jpeg,png,gif'\n  -f, --format[=csv]              set output format --format='[csv,sagemaker]'\n  -p, --prefix                   *prefix for file path --prefix='gs://\u003cyour-bucket-name\u003e'\n```\n\n```bash\n# Create file list from given dir and save it to output CSV file.\n$ cloud-label-uploader list -i ./save -o result.csv -p \"gs://my-bucket/test-project\"\n\n\n# Check saved CSV file.\n$ cat result.csv\n\ngs://my-bucket/test-project/cat/1.jpg,cat\ngs://my-bucket/test-project/cat/3.JPG,cat\ngs://my-bucket/test-project/dog/2.jpg,dog\ngs://my-bucket/test-project/human/4.png,human\ngs://my-bucket/test-project/human/5.png,human\n```\n\n\n## upload command\n\n`upload` uploads image files in a directory to GCS/S3 bucket.\n\n```bash\n$ cloud-label-uploader help upload\nUpload files to Cloud Bucket(S3, GCS) from --input dir\n\nOptions:\n\n  -h, --help                      display help information\n  -i, --input                    *image dir path --input='/path/to/image_dir'\n  -t, --type[=jpg,jpeg,png,gif]   comma separate file extensions --type='jpg,jpeg,png,gif'\n  -a, --all                       use all files\n  -l, --label                     label file for training (outputted CSV file) --label='/path/to/output.csv'\n  -c, --provider                 *cloud provider name for the bucket --provider='[s3,gcs]'\n  -b, --bucket                   *bucket name of S3/GCS --bucket='\u003cyour-bucket-name\u003e'\n  -p, --prefix                   *prefix for S3/GCS --prefix='foo/bar'\n  -m, --parallel[=2]              parallel number (multiple upload) --parallel=2\n```\n\n```bash\n# Before uploading, create GCS bucket\n# $ gsutil mb gs://example-bucket\n\n# Create file list from given dir and save it to output CSV file.\n# $ export GOOGLE_API_GO_PRIVATEKEY=`cat /path/to/gcs.pem`\n# $ export GOOGLE_API_GO_EMAIL=gcs@example.iam.gserviceaccount.com\n$ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcs.json\n$ cloud-label-uploader upload -i ./save -b 'example-bucket' -t 'jpg,png' -p 'automl_model/20180401' -c 'gcs' -l './result.csv' -m 10\n\n# upload files to gs://example-bucket/automl_model/20180401/ ...\n```\n\n\n## vott command\n\n`vott` creates a CSV file for AutoML Vision object-detection from VoTT's tagging result json files.\n\n```bash\n$ cloud-label-uploader help vott\nCreate object-detection list file from VoTT results\n\nOptions:\n\n  -h, --help                    display help information\n  -i, --input                  *VoTT json results dir path --input='/path/to/vott_json_dir'\n  -o, --output[=./output.csv]  *output CSV file path --output='./output.csv'\n  -p, --prefix[=gs://]         *prefix for file path --prefix='gs://\u003cyour-bucket-name\u003e'\n  -r, --recursive[=false]       read files in sub directories\n```\n\n```bash\n# Create file list from given dir and save it to output CSV file.\n$ cloud-label-uploader list -j ./vott/result -o result.csv -p \"gs://my-bucket/test-project/\"\n\n\n# Check saved CSV file.\n$ cat result.csv\n\nUNASSIGNED,gs://my-bucket/test-project/cat/1.jpg,cat,0.17785499052004333,0.3945237235067437,0.28786057692307687,0.3945237235067437,0.28786057692307687,0.5815947133911368,0.17785499052004333,0.5815947133911368\nUNASSIGNED,gs://my-bucket/test-project/cat/3.JPG,cat,0.7158391915641477,0.44460227272727265,0.8379421133567663,0.44460227272727265,0.8379421133567663,0.5285943675889327,0.7158391915641477,0.5285943675889327\nUNASSIGNED,gs://my-bucket/test-project/dog/2.jpg,dog,0.6664113285482123,0.4608950407608696,0.7451203615926327,0.4608950407608696,0.7451203615926327,0.6013332201086957,0.6664113285482123,0.6013332201086957\nUNASSIGNED,gs://my-bucket/test-project/human/4.png,human,0.730020144907909,0.4057476065751445,0.8625490926327194,0.4057476065751445,0.8625490926327194,0.5787572254335259,0.730020144907909,0.5787572254335259\nUNASSIGNED,gs://my-bucket/test-project/human/5.png,human,,0.6799583559046587,0.4373871026011561,0.7823545842361863,0.4373871026011561,0.7823545842361863,0.5737953847543352,0.6799583559046587,0.5737953847543352\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevalphobia%2Fcloud-label-uploader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevalphobia%2Fcloud-label-uploader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevalphobia%2Fcloud-label-uploader/lists"}