{"id":22683336,"url":"https://github.com/ylogx/gcloud-backup","last_synced_at":"2025-03-29T14:40:27.074Z","repository":{"id":253627348,"uuid":"844062693","full_name":"ylogx/gcloud-backup","owner":"ylogx","description":"Download all files in a directory from Google Cloud Storage buckets for local archival","archived":false,"fork":false,"pushed_at":"2024-08-18T09:32:27.000Z","size":108,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-04T15:42:58.723Z","etag":null,"topics":["backup","downloader","google-cloud","google-cloud-storage"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ylogx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-18T09:08:37.000Z","updated_at":"2024-08-18T09:32:30.000Z","dependencies_parsed_at":"2024-08-18T10:26:41.516Z","dependency_job_id":"562ef4b0-0134-4ba8-b908-e730c612b2eb","html_url":"https://github.com/ylogx/gcloud-backup","commit_stats":null,"previous_names":["ylogx/gcloud-backup"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ylogx%2Fgcloud-backup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ylogx%2Fgcloud-backup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ylogx%2Fgcloud-backup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ylogx%2Fgcloud-backup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ylogx","download_url":"https://codeload.github.com/ylogx/gcloud-backup/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246200300,"owners_count":20739563,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","downloader","google-cloud","google-cloud-storage"],"created_at":"2024-12-09T21:11:45.403Z","updated_at":"2025-03-29T14:40:27.041Z","avatar_url":"https://github.com/ylogx.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Google Cloud Storage Backup\n===========================\n\nHave you gone through the pain of downloading files one-by-one from a Google Cloud Storage bucket?\n\nEasily backup your Google Cloud Storage (GCS) bucket directories to your local machine or NAS with this CLI tool.\nLeveraging Async IO for concurrent downloads, it ensures a fast and efficient download process.\nThe tool is fault-tolerant, allowing you to resume downloads after network interruptions.\nIt also compresses your files into a single tar file and uses zstd for optimal storage efficiency \u0026 speed.\nTry it now to speed up your GCS backup process!\n\nInstallation\n------------\n\nTo install the dependencies, run the following command:\n\n```bash\nmake init # Install dependencies and setup poetry virtual environment\n```\n\nGetting Started\n---------------\n\n### Credentials\n\nTo authenticate with GCP, you need to set up your credentials.\nYou can either set up your credentials using the `gcloud` CLI or\nby setting the `GOOGLE_APPLICATION_CREDENTIALS` environment variable.\nFor more information, refer to the [GCP documentation](https://cloud.google.com/docs/authentication/getting-started).\n\n```bash\nexport GOOGLE_APPLICATION_CREDENTIALS=\"${HOME}/.config/gcloud/\u003cyour_credentails/adc.json\u003e\"\n```\n\n### Usage\n\nTo see the available options, run the following command:\n\n```bash\npoetry run python run.py --help\n\nUsage: run.py cp [OPTIONS]\n\nOptions:\n  -b, --bucket-name TEXT          Name of the GCS bucket  [required]\n  -f, --folder-path TEXT          Path to the folder in the bucket  [required]\n  -l, --local-download-path TEXT  Local path to download the files\n  -o, --output-filename TEXT      Name of the output file\n  -p, --gcloud-project TEXT       Name of the GCP project\n  --help\n```\n\n### Downloading files from GCS\nTo download files from a GCS bucket, run the following command:\n\n```bash\npoetry run python run.py cp \\\n    --bucket us.artifacts.foobar.appspot.com \\\n    --folder-path containers/images \\\n    --gcloud-project foobar\n# poetry run cp -b us.artifacts.foobar.appspot.com -f containers/images -p foobar # Shorthand format\n\n2024-08-18 01:55:46.300 | DEBUG    | downloder.download:_validate_local_download_path:89 - Creating directory /Users/example/gcloud-downloader/images\n2024-08-18 01:55:46.300 | INFO     | downloder.download:download_files:22 - Downloading files from us.artifacts.lineage-sc.appspot.com/containers/images to /Users/example/gcloud-downloader/images\n2024-08-18 01:55:46.729 | DEBUG    | downloder.download:download_blob:43 - Downloading containers/images/file1\naa9e90e8b to /Users/example/gcloud-downloader/images/file1.tmp\n2024-08-18 01:55:46.730 | DEBUG    | downloder.download:download_blob:43 - Downloading containers/images/file2\naeb790a03 to /Users/example/gcloud-downloader/images/file2.tmp\n2024-08-18 01:55:46.731 | DEBUG    | downloder.download:download_blob:43 - Downloading containers/images/file3\n67d570afa to /Users/example/gcloud-downloader/images/file3.tmp\n...\n2024-08-18 01:59:01.111 | DEBUG    | downloder.download:download_blob:47 - Downloaded containers/images/file1 to /Users/example/gcloud-downloader/images/file1\n2024-08-18 01:59:01.462 | DEBUG    | downloder.download:download_blob:47 - Downloaded containers/images/file2 to /Users/example/gcloud-downloader/images/file2\n2024-08-18 01:59:07.763 | DEBUG    | downloder.download:download_blob:47 - Downloaded containers/images/file3 to /Users/example/gcloud-downloader/images/file3\n2024-08-18 01:59:07.768 | INFO     | downloder.download:compress_files:70 - Compressing files in /Users/example/gcloud-downloader/images to images.tar\n2024-08-18 01:59:28.653 | DEBUG    | downloder.download:compress_files:75 - Compressed the tar file using zstandard to images.tar.zst\n2024-08-18 01:59:49.002 | DEBUG    | downloder.download:compress_files:82 - Compressed to images.tar.zst\n2024-08-18 01:59:49.003 | INFO     | __main__:cp:26 - Compressed files to images.tar.zst\n```\n\nYou can find your downloaded files in the `images.tar.zst` file in the current directory as shown by logs above.\nYou can also specify the local download path using the `--local-download-path` option and the output filename using the `--output-filename` option.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fylogx%2Fgcloud-backup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fylogx%2Fgcloud-backup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fylogx%2Fgcloud-backup/lists"}