{"id":15983171,"url":"https://github.com/suyashkumar/deeplesion-gcp-loader","last_synced_at":"2025-04-04T20:32:34.978Z","repository":{"id":82739673,"uuid":"148703471","full_name":"suyashkumar/deeplesion-gcp-loader","owner":"suyashkumar","description":"Get the DeepLesion CT Image data set into a GCP Storage Bucket","archived":false,"fork":false,"pushed_at":"2018-09-13T23:04:38.000Z","size":9,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-10T05:13:20.635Z","etag":null,"topics":["bucket","data","data-loader","data-loading","data-science","deep-learning","deep-lesion","deeplesion","gcp","gcp-bucket","loader","storage"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/suyashkumar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-13T22:06:12.000Z","updated_at":"2025-01-10T19:47:17.000Z","dependencies_parsed_at":"2023-03-07T03:15:13.836Z","dependency_job_id":null,"html_url":"https://github.com/suyashkumar/deeplesion-gcp-loader","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suyashkumar%2Fdeeplesion-gcp-loader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suyashkumar%2Fdeeplesion-gcp-loader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suyashkumar%2Fdeeplesion-gcp-loader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suyashkumar%2Fdeeplesion-gcp-loader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/suyashkumar","download_url":"https://codeload.github.com/suyashkumar/deeplesion-gcp-loader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247247218,"owners_count":20907933,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bucket","data","data-loader","data-loading","data-science","deep-learning","deep-lesion","deeplesion","gcp","gcp-bucket","loader","storage"],"created_at":"2024-10-08T01:40:54.969Z","updated_at":"2025-04-04T20:32:34.953Z","avatar_url":"https://github.com/suyashkumar.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DeepLesion GCP Loader\nThis program is a simple way to fetch, uncompress, and upload the [DeepLesion](https://www.nih.gov/news-events/news-releases/nih-clinical-center-releases-dataset-32000-ct-images) dataset of 32,000 CT images into a google cloud bucket. Usage is simple:\n\n```sh\n./deeplesion-loader --removeFiles=true --bucketName=my-bucket\n```\nWill download each 4GB zip from the dataset, unzip it, and upload the images to `my-bucket`. This configuration with `removeFiles=true` will delete each zip file after it has successfully uploaded the contents to GCP. \n\n```sh\n./deeplesion-loader --bucketName=my-bucket --parallel=true\n```\nWill run all file downloads and uploads in parallel--this is *much faster*, but requires more disk space and resources. \n\n**Note:** You must ensure the machine running this program has write access to your GCP bucket (or that GCP application deafult credentials are set). See the section below for more details\n\n## General installation and setup\nYou can simply download the right binary from the [releases tab](https://github.com/suyashkumar/deeplesion-gcp-loader/releases) and run it like detailed above. You can also fetch the binary from the commandline using the following command:\n\n```sh\nwget -qO- $BINARY_RELEASE_LINK | tar xvz\n```\n\nwhere `$BINARY_RELEASE_LINK` is the link of the download from the [releases tab](https://github.com/suyashkumar/deeplesion-gcp-loader/releases).\n\n### Ensuring GCP Write access\nThe machine this program runs on needs to have write access to your bucket. This can be done in two ways:\n* Ensure [application default credentials](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login) are set. Usually: `gcloud auth application-default login` will do it \n* Or you can spin up a GCP virtual machine that has the \"Storage\" API permission set to \"Read Write\" which can be done when creating the VM by clicking \"Set access for each API\"\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuyashkumar%2Fdeeplesion-gcp-loader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsuyashkumar%2Fdeeplesion-gcp-loader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuyashkumar%2Fdeeplesion-gcp-loader/lists"}