{"id":13571177,"url":"https://github.com/skyplane-project/skyplane","last_synced_at":"2025-05-15T00:06:49.845Z","repository":{"id":46922318,"uuid":"421928709","full_name":"skyplane-project/skyplane","owner":"skyplane-project","description":"🔥 Blazing fast bulk data transfers between any cloud 🔥","archived":false,"fork":false,"pushed_at":"2024-05-11T02:35:23.000Z","size":10189,"stargazers_count":1135,"open_issues_count":106,"forks_count":67,"subscribers_count":25,"default_branch":"main","last_synced_at":"2025-04-14T10:27:08.862Z","etag":null,"topics":["aws","azure","cloud","cloud-computing","data-antigravity","data-transfer","datasync","gcp","multi-cloud","multicloud","replication","rsync","sky","skyplane"],"latest_commit_sha":null,"homepage":"https://skyplane.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/skyplane-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/roadmap.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-10-27T18:20:37.000Z","updated_at":"2025-04-04T05:44:04.000Z","dependencies_parsed_at":"2024-01-14T16:14:01.602Z","dependency_job_id":"5b302f9e-4d75-44f6-8ef3-247e2008f206","html_url":"https://github.com/skyplane-project/skyplane","commit_stats":{"total_commits":656,"total_committers":27,"mean_commits":"24.296296296296298","dds":"0.33231707317073167","last_synced_commit":"4602d9cec2083142831f7f208b413ca2bf091bd8"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skyplane-project%2Fskyplane","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skyplane-project%2Fskyplane/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skyplane-project%2Fskyplane/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skyplane-project%2Fskyplane/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/skyplane-project","download_url":"https://codeload.github.com/skyplane-project/skyplane/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254249198,"owners_count":22039029,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","azure","cloud","cloud-computing","data-antigravity","data-transfer","datasync","gcp","multi-cloud","multicloud","replication","rsync","sky","skyplane"],"created_at":"2024-08-01T14:00:59.590Z","updated_at":"2025-05-15T00:06:49.692Z","avatar_url":"https://github.com/skyplane-project.png","language":"Python","readme":"\u003cp align=\"center\"\u003e\n    \u003cpicture\u003e\n        \u003csource srcset=\"docs/_static/logo-dark-mode.png\" media=\"(prefers-color-scheme: dark)\"\u003e\n        \u003cimg src=\"docs/_static/logo-light-mode.png\" width=\"300\" /\u003e\n    \u003c/picture\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://join.slack.com/t/skyplaneworkspace/shared_invite/zt-1cxmedcuc-GwIXLGyHTyOYELq7KoOl6Q\"\u003e \n    \u003cimg alt=\"Join Slack\" src=\"https://img.shields.io/badge/-Join%20Skyplane%20Slack-blue?logo=slack\"\u003e\n  \u003c/a\u003e\n  \n  \u003ca href=\"https://github.com/skyplane-project/skyplane/actions/workflows/integration-test-multiple-sizes.yml\"\u003e \n    \u003cimg alt=\"integration-test\" src=\"https://github.com/skyplane-project/skyplane/actions/workflows/integration-test-multiple-sizes.yml/badge.svg\"\u003e\n  \u003c/a\u003e\n  \n  \u003ca href=\"https://github.com/skyplane-project/skyplane/actions/workflows/docker-publish.yml\"\u003e \n    \u003cimg alt=\"docker\" src=\"https://github.com/skyplane-project/skyplane/actions/workflows/docker-publish.yml/badge.svg\"\u003e\n  \u003c/a\u003e\n  \n  \u003ca href=\"https://skyplane.readthedocs.io/en/latest/?badge=latest\"\u003e \n    \u003cimg alt=\"docs\" src=\"https://readthedocs.org/projects/skyplane/badge/?version=latest\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n**🔥 Blazing fast bulk data transfers between any cloud 🔥**\n\nSkyplane is a tool for blazingly fast bulk data transfers between object stores in the cloud. It provisions a fleet of VMs in the cloud to transfer data in parallel while using compression and bandwidth tiering to reduce cost.\n\nSkyplane is:\n1. 🔥 Blazing fast ([110x faster than AWS DataSync](https://skyplane.org/en/latest/benchmark.html))\n2. 🤑 Cheap (4x cheaper than rsync)\n3. 🌐 Universal (AWS, Azure, IBM and GCP)\n\nYou can use Skyplane to transfer data: \n* between object stores within a cloud provider (e.g. AWS us-east-1 to AWS us-west-2)\n* between object stores across multiple cloud providers (e.g. AWS us-east-1 to GCP us-central1)\n* between local storage and cloud object stores (experimental)\n\nSkyplane currently supports the following source and destination endpoints (any source and destination can be combined): \n\n| Endpoint           | Source             | Destination        |\n|--------------------|--------------------|--------------------|\n| AWS S3             | :white_check_mark: | :white_check_mark: |\n| Google Storage     | :white_check_mark: | :white_check_mark: |\n| Azure Blob Storage | :white_check_mark: | :white_check_mark: |\n| IBM Cloud Object Storage | :white_check_mark: | :white_check_mark: |\n| Local Disk         | :white_check_mark: | (in progress)      |\n\nSkyplane is an actively developed project. It will have 🔪 SHARP EDGES 🔪. Please file an issue or ask the contributors via [the #help channel on our Slack](https://join.slack.com/t/skyplaneworkspace/shared_invite/zt-1cxmedcuc-GwIXLGyHTyOYELq7KoOl6Q) if you encounter bugs.\n\n# Resources \n- [Quickstart](#quickstart)\n- [Contributing](https://skyplane.org/en/latest/contributing.html)\n- [Roadmap](https://skyplane.org/en/latest/roadmap.html)\n- [Slack Community](https://join.slack.com/t/skyplaneworkspace/shared_invite/zt-1cxmedcuc-GwIXLGyHTyOYELq7KoOl6Q)\n\n# Quickstart\n\n## 1. Installation\nWe recommend installation from PyPi:\n```\n$ pip install \"skyplane[aws]\"\n\n# install support for other clouds as needed:\n\n#   $ pip install \"skyplane[azure]\"\n#   $ pip install \"skyplane[gcp]\"\n#   $ pip install \"skyplane[ibmcloud]\"\n#   $ pip install \"skyplane[all]\"\n```\n\nSkyplane supports AWS, Azure, IBM and GCP. You can install Skyplane with support for one or more of these clouds by specifying the corresponding extras. To install two out of three clouds, you can run `pip install \"skyplane[aws,azure]\"`.\n\n*GCP support on the M1 Mac*: If you are using an M1 Mac with the arm64 architecture and want to install GCP support for Skyplane, you will need to install as follows\n`GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip install \"skyplane[aws,gcp]\"`\n\n## 2. Setup Cloud Credentials \n\nSkyplane needs access to cloud credentials to perform transfers. To get started with setting up credentials, make sure you have cloud provider CLI tools installed:\n\n```\n---\u003e For AWS:\n$ pip install awscli\n\n---\u003e For Google Cloud:\n$ pip install gcloud\n\n---\u003e For Azure:\n$ pip install azure\n\n```\nOnce you have the CLI tools setup, log into each cloud provider's CLI:\n```\n---\u003e For AWS:\n$ aws configure\n\n---\u003e For Google Cloud:\n$ gcloud auth application-default login\n\n---\u003e For Azure:\n$ az login\n\n---\u003e For IBM Cloud:\n$ Follow IBM Cloud and create an account with the resource group.\nCopy https://github.com/skyplane-project/skyplane/blob/main/skyplane/compute/ibmcloud/ibm_credentials.yaml.template\ninto `~/.bluemix/ibm_credentials` and fill your \nIBM IAM key and credentials to your IBM Cloud object storage \n\n---\u003e For SCP:\n$ # Create directory if required\n$ mkdir -p ~/.scp\n$ # Add the lines for \"access_key\", \"secret_key\", and \"project_id\" to scp_credential file\n$ echo \"access_key = \u003cyour_access_key\u003e\" \u003e\u003e ~/.scp/scp_credential\n$ echo \"secret_key = \u003cyour_secret_key\u003e\" \u003e\u003e ~/.scp/scp_credential\n$ echo \"project_id = \u003cyour_project_id\u003e\" \u003e\u003e ~/.scp/scp_credential\n\n```\nAfter authenticating with each cloud provider, you can run `skyplane init` to create a configuration file for Skyplane.\n\n```bash\n$ skyplane init\n```\n\u003cdetails\u003e\n\u003csummary\u003eskyplane init output\u003c/summary\u003e\n\u003cbr\u003e\n\n```\n$ skyplane init\n\n====================================================\n _____ _   ____   _______ _       ___   _   _  _____\n/  ___| | / /\\ \\ / / ___ \\ |     / _ \\ | \\ | ||  ___|\n\\ `--.| |/ /  \\ V /| |_/ / |    / /_\\ \\|  \\| || |__\n `--. \\    \\   \\ / |  __/| |    |  _  || . ` ||  __|\n/\\__/ / |\\  \\  | | | |   | |____| | | || |\\  || |___\n\\____/\\_| \\_/  \\_/ \\_|   \\_____/\\_| |_/\\_| \\_/\\____/\n====================================================\n\n\n(1) Configuring AWS:\n    Loaded AWS credentials from the AWS CLI [IAM access key ID: ...XXXXXX]\n    AWS region config file saved to /home/ubuntu/.skyplane/aws_config\n\n(2) Configuring Azure:\n    Azure credentials found in Azure CLI\n    Azure credentials found, do you want to enable Azure support in Skyplane? [Y/n]: Y\n    Enter the Azure subscription ID: [XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX]:\n    Azure region config file saved to /home/ubuntu/.skyplane/azure_config\n    Querying for SKU availbility in regions\n    Azure SKU availability cached in /home/ubuntu/.skyplane/azure_sku_mapping\n\n(3) Configuring GCP:\n    GCP credentials found in GCP CLI\n    GCP credentials found, do you want to enable GCP support in Skyplane? [Y/n]: Y\n    Enter the GCP project ID [XXXXXXX]:\n    GCP region config file saved to /home/ubuntu/.skyplane/gcp_config\n\n(4) Configuring SCP:\n    Loaded SCP credentials from the scp_credntial file [access key: ...XXXXXX]\n    SCP region config file saved to /home/ubuntu/.skyplane/scp_config\n\n\nConfig file saved to /home/ubuntu/.skyplane/config\n```\n\n\u003c/details\u003e\n\n## 3. Run Transfers \n\nWe’re ready to use Skyplane! Let’s use `skyplane cp` to copy files from AWS to GCP:\n```\nskyplane cp s3://... gs://...\n```\nTo transfer only new objects, you can instead use `skyplane sync`:\n```\n$ skyplane sync s3://... gs://...\n```\n\nYou can configure Skyplane to use more VMs per region with the `-n` flag. For example, to double the transfer speed with two VMs, run: \n```\n$ skyplane cp -r s3://... s3://... -n 2\n```\n\n## 4. Clean Up \nSkyplane will automatically attempt to terminate VMs that it starts, but to double check and forcefuly terminate all VMs, run `skyplane deprovision`.\n\n# Technical Details\nSkyplane is based on research at UC Berkeley into accelerated networks between cloud providers. Under the hood, Skyplane starts a fleet of VMs in the source and destination regions. It then uses a custom TCP protocol to accelerate the transfer between the VMs. Skyplane may use a L7 overlay network to route traffic around congested network hot spots. \n\n\u003cimg src=\"docs/_static/skyplane-data-plane.png\" width=\"384\" /\u003e\n\nFor more details on Skyplane, see: \n- [Technical Talk](https://skyplane.org/en/latest/architecture.html)\n- [NSDI '23 Paper](https://arxiv.org/abs/2210.07259)\n\n\n","funding_links":[],"categories":["Python","HarmonyOS"],"sub_categories":["Windows Manager"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskyplane-project%2Fskyplane","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fskyplane-project%2Fskyplane","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskyplane-project%2Fskyplane/lists"}