{"id":18658551,"url":"https://github.com/cboudereau/gcs-rsync","last_synced_at":"2025-04-11T19:32:32.246Z","repository":{"id":39630903,"uuid":"416759782","full_name":"cboudereau/gcs-rsync","owner":"cboudereau","description":"Lightweight Google Cloud Storage sync Rust Client with better performance than gsutil rsync","archived":false,"fork":false,"pushed_at":"2024-03-18T13:30:17.000Z","size":159,"stargazers_count":11,"open_issues_count":1,"forks_count":5,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-03-19T09:57:11.860Z","etag":null,"topics":["gcs","gcs-rsync","gsutil","rsync","rust","rust-lang"],"latest_commit_sha":null,"homepage":"https://docs.rs/gcs-rsync/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cboudereau.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-10-13T13:41:36.000Z","updated_at":"2024-04-15T16:52:31.022Z","dependencies_parsed_at":"2024-03-18T09:41:03.217Z","dependency_job_id":"21dce30e-b6b6-4988-99f9-bb568174503d","html_url":"https://github.com/cboudereau/gcs-rsync","commit_stats":{"total_commits":117,"total_committers":1,"mean_commits":117.0,"dds":0.0,"last_synced_commit":"0eeb842f71a12fc0dc189b92a09ea27d576e96cd"},"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cboudereau%2Fgcs-rsync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cboudereau%2Fgcs-rsync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cboudereau%2Fgcs-rsync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cboudereau%2Fgcs-rsync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cboudereau","download_url":"https://codeload.github.com/cboudereau/gcs-rsync/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247666005,"owners_count":20975787,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gcs","gcs-rsync","gsutil","rsync","rust","rust-lang"],"created_at":"2024-11-07T07:33:27.444Z","updated_at":"2025-04-11T19:32:32.235Z","avatar_url":"https://github.com/cboudereau.png","language":"Rust","funding_links":[],"categories":["rust"],"sub_categories":[],"readme":"# gcs-rsync\n\n[![build](https://github.com/cboudereau/gcs-rsync/workflows/build/badge.svg)](https://github.com/cboudereau/gcs-rsync/actions/workflows/build.yml?query=event%3Apush)\n[![codecov](https://codecov.io/gh/cboudereau/gcs-rsync/branch/main/graph/badge.svg?token=AA168CFNFQ)](https://codecov.io/gh/cboudereau/gcs-rsync)\n[![License:MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![docs.rs](https://docs.rs/gcs-rsync/badge.svg)](https://docs.rs/gcs-rsync)\n[![crates.io](https://img.shields.io/crates/v/gcs-rsync.svg)](https://crates.io/crates/gcs-rsync)\n[![crates.io (recent)](https://img.shields.io/crates/dr/gcs-rsync)](https://crates.io/crates/gcs-rsync)\n[![docker](https://img.shields.io/docker/pulls/superbeeeeeee/gcs-rsync)](https://hub.docker.com/r/superbeeeeeee/gcs-rsync)\n\nLightweight and efficient Rust gcs rsync for Google Cloud Storage.\n\ngcs-rsync is faster than [gsutil rsync](https://cloud.google.com/storage/docs/gsutil/commands/rsync) according to the following benchmarks.\n\nno hard limit to 32K objects or specific conf to compute state.\n\nThis crate can be used as a library or CLI. The API managing objects (download, upload, delete, ...) can be used independently.\n\n## How to install as crate\n\nCargo.toml\n\n```bash\n[dependencies]\ngcs-rsync = \"0.4\"\n```\n\n## How to install as cli tool\n\n```bash\ncargo install --example gcs-rsync gcs-rsync\n\n~/.cargo/bin/gcs-rsync\n\n```\n\n## How to run with docker\n\n### Mirror local folder to gcs\n\n```bash\ndocker run --rm -it -v ${GOOGLE_APPLICATION_CREDENTIALS}:/creds.json:ro -v \u003cYourFolderToUpload\u003e:/source:ro superbeeeeeee/gcs-rsync -r -m /source gs://\u003cYourBucket\u003e/\u003cYourFolderToUpload\u003e/\n```\n\n### Mirror gcs to folder\n\n```bash\ndocker run --rm -it -v ${GOOGLE_APPLICATION_CREDENTIALS}:/creds.json:ro -v \u003cYourFolderToDownloadTo\u003e:/dest superbeeeeeee/gcs-rsync -r -m gs://\u003cYourBucket\u003e/\u003cYourFolderToUpload\u003e/ /dest\n```\n\n### Mirror partial gcs with prefix to folder\n\n```bash\ndocker run --rm -it -v ${GOOGLE_APPLICATION_CREDENTIALS}:/creds.json:ro -v \u003cYourFolderToDownloadTo\u003e:/dest superbeeeeeee/gcs-rsync -r -m gs://\u003cYourBucket\u003e/\u003cYourFolderToUpload\u003e/\u003cYourPrefix\u003e /dest\n```\n\n### Include or Exclude files using glob pattern\n\n#### CLI gcs-rsync\n\n```-i``` (include glob pattern) and ```-x``` (exclude glob pattern) multiple times.\n\nAn example where any json or toml are included recursively except any test.json or test.toml recursively\n```bash\ndocker run --rm -it -v ${GOOGLE_APPLICATION_CREDENTIALS}:/creds.json:ro -v \u003cYourFolderToDownloadTo\u003e:/dest superbeeeeeee/gcs-rsync -r -m -i **/*.json -i **/*.toml -x **/test.json -x **/test.toml\n gs://\u003cYourBucket\u003e/YourFolderToUpload\u003e/ /dest\n```\n\n#### Library\n```with_includes``` and ```with_excludes``` client builders are used to fill includes and excludes glob patterns.\n\n## Benchmark\n\nImportant note about gsutil: The `gsutil ls` command does not list all object items by default but instead list all prefixes while adding the `-r` flag slowdown `gsutil` performance. The `ls` performance command is very different to the `rsync` implementation.\n\n### new files only (first time sync)\n\n- gcs-rsync: 2.2s/7MB\n- gsutil: 9.93s/47MB\n\n**winner**: gcs-rsync\n\n#### gcs-rsync sync bench\n\n```bash\nrm -rf ~/Documents/test4 \u0026\u0026 cargo build --release --examples \u0026\u0026 /usr/bin/time -lp -- ./target/release/examples/bucket_to_folder_sync\n```\n\n```\nreal         2.20\nuser         0.13\nsys          0.21\n             7606272  maximum resident set size\n                   0  average shared memory size\n                   0  average unshared data size\n                   0  average unshared stack size\n                1915  page reclaims\n                   0  page faults\n                   0  swaps\n                   0  block input operations\n                   0  block output operations\n                 394  messages sent\n                1255  messages received\n                   0  signals received\n                  54  voluntary context switches\n                5814  involuntary context switches\n           636241324  instructions retired\n           989595729  cycles elapsed\n             3895296  peak memory footprint\n```\n\n#### gsutil sync bench\n\n```bash\nrm -rf ~/Documents/gsutil_test4 \u0026\u0026 mkdir ~/Documents/gsutil_test4 \u0026\u0026 /usr/bin/time -lp --  gsutil -m -q rsync -r gs://dev-bucket/sync_test4/ ~/Documents/gsutil_test4/\n```\n\n```\nOperation completed over 215 objects/50.3 KiB.\nreal         9.93\nuser         8.12\nsys          2.35\n            47108096  maximum resident set size\n                   0  average shared memory size\n                   0  average unshared data size\n                   0  average unshared stack size\n              196391  page reclaims\n                   1  page faults\n                   0  swaps\n                   0  block input operations\n                   0  block output operations\n               36089  messages sent\n               87309  messages received\n                   5  signals received\n               38401  voluntary context switches\n               51924  involuntary context switches\n            12986389  instructions retired\n            12032672  cycles elapsed\n              593920  peak memory footprint\n```\n\n### no change (second time sync)\n\n- gcs-rsync: 0.78s/8MB\n- gsutil: 2.18s/47MB\n\n**winner**: gcs-rsync (due to size and mtime check before crc32c like gsutil does)\n\n#### gcs-rsync sync bench\n\n```bash\ncargo build --release --examples \u0026\u0026 /usr/bin/time -lp -- ./target/release/examples/bucket_to_folder_sync\n```\n\n```\nreal         1.79\nuser         0.13\nsys          0.12\n             7864320  maximum resident set size\n                   0  average shared memory size\n                   0  average unshared data size\n                   0  average unshared stack size\n                1980  page reclaims\n                   0  page faults\n                   0  swaps\n                   0  block input operations\n                   0  block output operations\n                 397  messages sent\n                1247  messages received\n                   0  signals received\n                  42  voluntary context switches\n                4948  involuntary context switches\n           435013936  instructions retired\n           704782682  cycles elapsed\n             4141056  peak memory footprint\n```\n\n#### gsutil sync bench\n\n```bash\n/usr/bin/time -lp --  gsutil -m -q rsync -r gs://test-bucket/sync_test4/ ~/Documents/gsutil_test4/\n```\n\n```\nreal         2.18\nuser         1.37\nsys          0.66\n            46899200  maximum resident set size\n                   0  average shared memory size\n                   0  average unshared data size\n                   0  average unshared stack size\n              100108  page reclaims\n                1732  page faults\n                   0  swaps\n                   0  block input operations\n                   0  block output operations\n                6311  messages sent\n               12752  messages received\n                   4  signals received\n                6145  voluntary context switches\n               14219  involuntary context switches\n            13133297  instructions retired\n            13313536  cycles elapsed\n              602112  peak memory footprint\n```\n\n### gsutil rsync config\n\n```bash\ngsutil -m -q rsync -r -d ./your-dir gs://your-bucket\n```\n\n```bash\n/usr/bin/time -lp --  gsutil -m -q rsync -r gs://dev-bucket/sync_test4/ ~/Documents/gsutil_test4/\n```\n\n- [commands](https://cloud.google.com/storage/docs/gsutil/commands/rsync)\n- [crc32c for gsutil](https://cloud.google.com/storage/docs/gsutil/addlhelp/CRC32CandInstallingcrcmod)\n\n## About authentication\n\nAll default functions related to authentication use GOOGLE_APPLICATION_CREDENTIALS env var as default conf like official Google libraries do on other languages (golang, dotnet)\n\nOther functions (from and from_file) provide the custom integration mode.\n\nFor more info about OAuth2, see the related README in the oauth2 mod.\n\n## How to run tests\n\n## Unit tests\n\n```bash\ncargo test --lib\n```\n\n## Integration tests + Unit tests\n\n```bash\nTEST_SERVICE_ACCOUNT=\u003cPathToAServiceAccount\u003e TEST_BUCKET=\u003cBUCKET\u003e TEST_PREFIX=\u003cPREFIX\u003e cargo test --no-fail-fast\n```\n\n## Examples\n\n### Upload object\n\n#### Library\n```rust\nuse std::path::Path;\n\nuse gcs_rsync::storage::{credentials, Object, ObjectClient, StorageResult};\nuse tokio_util::codec::{BytesCodec, FramedRead};\n\n#[tokio::main]\nasync fn main() -\u003e StorageResult\u003c()\u003e {\n    let args = std::env::args().collect::\u003cVec\u003c_\u003e\u003e();\n    let bucket = args[1].as_str();\n    let prefix = args[2].to_owned();\n    let file_path = args[3].to_owned();\n\n    let auc = Box::new(credentials::authorizeduser::default().await?);\n    let object_client = ObjectClient::new(auc).await?;\n\n    let file_path = Path::new(\u0026file_path);\n    let name = file_path.file_name().unwrap().to_string_lossy();\n\n    let file = tokio::fs::File::open(file_path).await.unwrap();\n    let stream = FramedRead::new(file, BytesCodec::new());\n\n    let name = format!(\"{}/{}\", prefix, name);\n    let object = Object::new(bucket, name.as_str())?;\n    object_client.upload(\u0026object, stream).await.unwrap();\n    println!(\"object {} uploaded\", \u0026object);\n    Ok(())\n}\n```\n\n#### CLI\n\n```bash\ncargo run --release --example upload_object \"\u003cYourBucket\u003e\" \"\u003cYourPrefix\u003e\" \"\u003cYourFilePath\u003e\"\n```\n\n### Download object\n\n#### Library\n```rust\nuse std::path::Path;\n\nuse futures::TryStreamExt;\nuse gcs_rsync::storage::{credentials, Object, ObjectClient, StorageResult};\nuse tokio::{\n    fs::File,\n    io::{AsyncWriteExt, BufWriter},\n};\n\n#[tokio::main]\nasync fn main() -\u003e StorageResult\u003c()\u003e {\n    let args = std::env::args().collect::\u003cVec\u003c_\u003e\u003e();\n    let bucket = args[1].as_str();\n    let name = args[2].as_str();\n    let output_path = args[3].to_owned();\n\n    let auc = Box::new(credentials::authorizeduser::default().await?);\n    let object_client = ObjectClient::new(auc).await?;\n\n    let file_name = Path::new(\u0026name).file_name().unwrap().to_string_lossy();\n    let file_path = format!(\"{}/{}\", output_path, file_name);\n\n    let object = Object::new(bucket, name)?;\n    let mut stream = object_client.download(\u0026object).await.unwrap();\n\n    let file = File::create(\u0026file_path).await.unwrap();\n    let mut buf_writer = BufWriter::new(file);\n\n    while let Some(data) = stream.try_next().await.unwrap() {\n        buf_writer.write_all(\u0026data).await.unwrap();\n    }\n\n    buf_writer.flush().await.unwrap();\n    println!(\"object {} downloaded to {:?}\", \u0026object, file_path);\n    Ok(())\n}\n```\n\n#### CLI\n```bash\ncargo run --release --example download_object \"\u003cYourBucket\u003e\" \"\u003cYourObjectName\u003e\" \"\u003cYourAbsoluteExistingDirectory\u003e\"\n```\n\n### Download public object\n\n#### Library\n```rust\nuse std::path::Path;\n\nuse futures::TryStreamExt;\nuse gcs_rsync::storage::{Object, ObjectClient, StorageResult};\nuse tokio::{\n    fs::File,\n    io::{AsyncWriteExt, BufWriter},\n};\n\n#[tokio::main]\nasync fn main() -\u003e StorageResult\u003c()\u003e {\n    let bucket = \"gcs-rsync-dev-public\";\n    let name = \"hello.txt\";\n\n    let object_client = ObjectClient::no_auth();\n\n    let file_name = Path::new(\u0026name).file_name().unwrap().to_string_lossy();\n    let file_path = file_name.to_string();\n\n    let object = Object::new(bucket, \"hello.txt\")?;\n    let mut stream = object_client.download(\u0026object).await.unwrap();\n\n    let file = File::create(\u0026file_path).await.unwrap();\n    let mut buf_writer = BufWriter::new(file);\n\n    while let Some(data) = stream.try_next().await.unwrap() {\n        buf_writer.write_all(\u0026data).await.unwrap();\n    }\n\n    buf_writer.flush().await.unwrap();\n    println!(\"object {} downloaded to {:?}\", \u0026object, file_path);\n    Ok(())\n}\n```\n\n#### CLI\n```bash\ncargo run --release --example download_public_object \"\u003cYourBucket\u003e\" \"\u003cYourObjectName\u003e\" \"\u003cYourAbsoluteExistingDirectory\u003e\"\n```\n\n### Delete object\n\n#### Library\n```rust\nuse gcs_rsync::storage::{credentials, Object, ObjectClient, StorageResult};\n\n#[tokio::main]\nasync fn main() -\u003e StorageResult\u003c()\u003e {\n    let args = std::env::args().collect::\u003cVec\u003c_\u003e\u003e();\n    let bucket = args[1].as_str();\n    let name = args[2].as_str();\n    let object = Object::new(bucket, name)?;\n\n    let auc = Box::new(credentials::authorizeduser::default().await?);\n    let object_client = ObjectClient::new(auc).await?;\n\n    object_client.delete(\u0026object).await?;\n    println!(\"object {} uploaded\", \u0026object);\n    Ok(())\n}\n```\n\n#### CLI\n\n```bash\ncargo run --release --example delete_object \"\u003cYourBucket\u003e\" \"\u003cYourPrefix\u003e/\u003cYourFileName\u003e\"\n```\n\n### List objects\n\n#### Library\n```rust\nuse futures::TryStreamExt;\nuse gcs_rsync::storage::{credentials, ObjectClient, ObjectsListRequest, StorageResult};\n\n#[tokio::main]\nasync fn main() -\u003e StorageResult\u003c()\u003e {\n    let args = std::env::args().collect::\u003cVec\u003c_\u003e\u003e();\n    let bucket = args[1].as_str();\n    let prefix = args[2].to_owned();\n\n    let auc = Box::new(credentials::authorizeduser::default().await?);\n    let object_client = ObjectClient::new(auc).await?;\n\n    let objects_list_request = ObjectsListRequest {\n        prefix: Some(prefix),\n        fields: Some(\"items(name),nextPageToken\".to_owned()),\n        ..Default::default()\n    };\n\n    object_client\n        .list(bucket, \u0026objects_list_request)\n        .await\n        .try_for_each(|x| {\n            println!(\"{}\", x.name.unwrap());\n            futures::future::ok(())\n        })\n        .await?;\n\n    Ok(())\n}\n```\n\n#### CLI\n\n```bash\ncargo run --release --example list_objects \"\u003cYourBucket\u003e\" \"\u003cYourPrefix\u003e\"\n```\n\n### List objects with default service account\n\n#### Library\n```rust\nuse futures::TryStreamExt;\nuse gcs_rsync::storage::{credentials, ObjectClient, ObjectsListRequest, StorageResult};\n\n#[tokio::main]\nasync fn main() -\u003e StorageResult\u003c()\u003e {\n    let args = std::env::args().collect::\u003cVec\u003c_\u003e\u003e();\n    let bucket = args[1].as_str();\n    let prefix = args[2].to_owned();\n\n    let auc = Box::new(\n        credentials::serviceaccount::default(\n            \"https://www.googleapis.com/auth/devstorage.full_control\",\n        )\n        .await?,\n    );\n    let object_client = ObjectClient::new(auc).await?;\n\n    let objects_list_request = ObjectsListRequest {\n        prefix: Some(prefix),\n        fields: Some(\"items(name),nextPageToken\".to_owned()),\n        ..Default::default()\n    };\n\n    object_client\n        .list(bucket, \u0026objects_list_request)\n        .await\n        .try_for_each(|x| {\n            println!(\"{}\", x.name.unwrap());\n            futures::future::ok(())\n        })\n        .await?;\n\n    Ok(())\n}\n```\n\n#### CLI\n```bash\nGOOGLE_APPLICATION_CREDENTIALS=\u003cPathToJson\u003e cargo r --release --example list_objects_service_account \"\u003cYourBucket\u003e\" \"\u003cYourPrefix\u003e\"\n```\n\n#### List lots of (\u003e32K) objects\n\nlist a bucket having more than 60K objects\n\n```bash\ntime cargo run --release --example list_objects \"\u003cYourBucket\u003e\" \"\u003cYourPrefixHavingMoreThan60K\u003e\" | wc -l\n```\n\n### Profiling\n\n[Humans are terrible at guessing-about-performance](https://github.com/flamegraph-rs/flamegraph#humans-are-terrible-at-guessing-about-performance)\n\n```bash\nexport CARGO_PROFILE_RELEASE_DEBUG=true\nsudo -- cargo flamegraph --example list_objects \"\u003cYourBucket\u003e\" \"\u003cYourPrefixHavingMoreThan60K\u003e\"\n```\n\n```bash\ncargo build --release --examples \u0026\u0026 /usr/bin/time -lp -- ./target/release/examples/list_objects \"\u003cYourBucket\u003e\" \"\u003cYourPrefixHavingMoreThan60K\u003e\"\n```\n\n### Native bin build (static shared lib)\n\n```bash\ndocker rust rust:alpine3.14\napk add --no-cache musl-dev pkgconfig openssl-dev\n\nLDFLAGS=\"-static -L/usr/local/musl/lib\" LD_LIBRARY_PATH=/usr/local/musl/lib:$LD_LIBRARY_PATH CFLAGS=\"-I/usr/local/musl/include\" PKG_CONFIG_PATH=/usr/local/musl/lib/pkgconfig cargo build --release --target=x86_64-unknown-linux-musl --example bucket_to_folder_sync\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcboudereau%2Fgcs-rsync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcboudereau%2Fgcs-rsync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcboudereau%2Fgcs-rsync/lists"}