{"id":28566990,"url":"https://github.com/altinity/s3gc","last_synced_at":"2025-06-10T15:40:50.493Z","repository":{"id":235914846,"uuid":"791526825","full_name":"Altinity/s3gc","owner":"Altinity","description":"Garbage collector for S3","archived":false,"fork":false,"pushed_at":"2025-01-31T11:42:05.000Z","size":76,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-31T12:20:02.410Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Altinity.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-24T21:55:37.000Z","updated_at":"2025-01-31T11:42:10.000Z","dependencies_parsed_at":"2025-01-31T12:20:03.859Z","dependency_job_id":null,"html_url":"https://github.com/Altinity/s3gc","commit_stats":null,"previous_names":["ilejn/s3gc","altinity/s3gc"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fs3gc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fs3gc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fs3gc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fs3gc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Altinity","download_url":"https://codeload.github.com/Altinity/s3gc/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fs3gc/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259104311,"owners_count":22805853,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-10T15:40:20.105Z","updated_at":"2025-06-10T15:40:50.477Z","avatar_url":"https://github.com/Altinity.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# s3gc\nGarbage collector for ClickHouse S3 disks\n\n## description\nThe script removes orphaned objects from s3 object storage\n  Ones that are not mentioned in system.remote_data_paths table\n\nThere are two stages:\n1. Collecting.\n     Paths of all objects found in object storage are put in auxiliary ClickHouse table.\n       It's name is a concatenation of 's3objects_for_' and disk name by default.\n       Created in the same ClickHouse instance where data from system.remote_data_paths selected\n2. Removing.\n     All objects that exist in s3 and not used according to system.remote_data_paths\n       are removed from object storage.\n\nIt is possible to split these stages or do everything at one go.\n\nBesides this, it is possible to calculate objects to remove without actual removing AKA dry run.\nIf dryrun is set together with usecollected, it uses collected data.\nIf dryrun is set together with collectonly, error is raised.\n\nIt is important to use `--s3diskname` if your disk name is not `s3` which is by default.\n\nWARNING!: Please use `--dry-run` to check and compare results of what is going to be deleted, just to be on the safe side. \n\n## script invocation\n### help\n```\npython3 s3gc.py --help\n```\n### typical usage\n#### all together with dry-run\nfor https://altinity-clickhouse-data-demo20565656565620663600000001.s3.amazonaws.com/github\n```\nS3GC_S3ACCESSKEY=sdfasfaerasasf \\\nS3GC_S3SECRETKEY=werqwsdfqwersdfasf \\\nS3GC_S3IP=s3.amazonaws.com \\\nS3GC_S3PORT=443 \\\nS3GC_S3REGION=us-east-1 \\\nS3GC_S3BUCKET=altinity-clickhouse-data-demo20565656565620663600000001 \\\nS3GC_S3PATH=github/ \\\nS3GC_S3SECURE_FLAG=true \\\npython3 ./s3gc.py --verbose --dry-run\n```\n#### GCS and object storage that do not support batch delete operations\n```\nS3GC_S3ACCESSKEY=GOOG1xxxxxxxxx \\\nS3GC_S3SECRETKEY=xxxxxxxxxxx \\\nS3GC_S3IP=storage.googleapis.com \\\nS3GC_S3PORT=443 \\\nS3GC_S3BUCKET=clickhouse-altinity-main-disk \\\nS3GC_S3PATH=chi-main-main-0-0/ \\\nS3GC_S3SECURE_FLAG=true \\\nS3GC_S3DISKNAME=gcs \\\npython3 ./s3gc.py --verbose --use-remove-objects=false\n```\n\nGCS_HMAC_KEY = S3GC_S3ACCESSKEY\nGCS_HMAC_SECRET = S3GC_S3SECRETKEY\n\n\n#### collect only\n```\nS3GC_S3PORT=19000  S3GC_S3ACCESSKEY=minio99  S3GC_S3SECRETKEY=minio123  python3 ./s3gc.py --verbose --collectonly\n```\n#### use collected\n```\nS3GC_S3PORT=19000  S3GC_S3ACCESSKEY=minio99  S3GC_S3SECRETKEY=minio123 S3GC_USECOLLECTED=true  python3 ./s3gc.py --debug\n```\n\n## docker\nThere is a docker image for the script.\n\n### rebuild\n```\nmake\nsudo docker buildx build --platform linux/arm/v7,linux/arm64/v8,linux/amd64 -t ilejn/s3gc .\n```\n\n### usage\n```\nsudo docker run ilejn/s3gc --help\nsudo docker run --network=\"host\" -e S3GC_S3PORT=19000 -e S3GC_S3ACCESSKEY=minio99 -e S3GC_S3SECRETKEY=minio123 ilejn/s3gc\n```\n\n## changelog\n\n### v_0.1 Wed Jun 12 2024\n\n- object last modified in auxiliary table\n- useage command line parameter\n  \n### v_0.2 Fri Jan 31 2025\n- added option to avoid batch deletion for services like GCS\n\n## to do list\n~~1. option to avoid `remove_objects` which is reportedly not supported by GCE~~\n\n- concurrency / async\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faltinity%2Fs3gc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faltinity%2Fs3gc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faltinity%2Fs3gc/lists"}