https://github.com/nuvo/skbn
Copy files and directories between Kubernetes and cloud storage
https://github.com/nuvo/skbn
abs aws azure azure-blob-storage go golang kubernetes minio s3
Last synced: 2 months ago
JSON representation
Copy files and directories between Kubernetes and cloud storage
- Host: GitHub
- URL: https://github.com/nuvo/skbn
- Owner: nuvo
- License: apache-2.0
- Created: 2018-08-26T20:14:41.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2024-06-12T13:35:52.000Z (about 1 year ago)
- Last Synced: 2024-12-30T12:45:38.981Z (6 months ago)
- Topics: abs, aws, azure, azure-blob-storage, go, golang, kubernetes, minio, s3
- Language: Go
- Homepage:
- Size: 110 KB
- Stars: 75
- Watchers: 10
- Forks: 75
- Open Issues: 20
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://github.com/nuvo/skbn/releases)
[](https://travis-ci.org/nuvo/skbn)
[](https://hub.docker.com/r/nuvo/skbn/)
[](https://goreportcard.com/report/github.com/nuvo/skbn)
[](https://github.com/nuvo/skbn/blob/master/LICENSE)
# Skbn
Skbn is a tool for copying files and directories between Kubernetes and cloud storage providers. It is named after the 1981 video game [Sokoban](https://en.wikipedia.org/wiki/Sokoban).
Skbn uses an in-memory buffer for the copy process, to avoid excessive memory consumption.
Skbn currently supports the following providers:
* AWS S3
* Minio S3
* Azure Blob Storage
## Install
### Prerequisites
1. git
2. [dep](https://github.com/golang/dep)
### From a release
Download the latest release from the [Releases page](https://github.com/nuvo/skbn/releases) or use it with a [Docker image](https://hub.docker.com/r/nuvo/skbn)
### From source
```
mkdir -p $GOPATH/src/github.com/nuvo && cd $_
git clone https://github.com/nuvo/skbn.git && cd skbn
make
```
## Usage
### Copy files from Kubernetes to S3
```
skbn cp \
--src k8s:///// \
--dst s3:///
```
### Copy files from S3 to Kubernetes
```
skbn cp \
--src s3:/// \
--dst k8s://///
```
### Copy files from Kubernetes to Azure Blob Storage
```
skbn cp \
--src k8s:///// \
--dst abs:////
```
### Copy files from Azure Blob Storage to Kubernetes
```
skbn cp \
--src abs://// \
--dst k8s://///
```
## Advanced usage
### Copy files from source to destination in parallel
```
skbn cp \
--src ... \
--dst ... \
--parallel
```
* `n` is the number of files to be copied in parallel (for full parallelism use 0)
### Set in-memory buffer size
Skbn copies files using an in-memory buffer. To control the buffer size:
```
skbn cp \
--src ... \
--dst ... \
--buffer-size
```
* `f` is the in-memory buffer size (in MB) to use for files copy. This flag should be used with caution when used in conjunction with `--parallel`
* The default value for `buffer-size` is 6.75 MB, and was decided based on benchmark
### Minio S3 support
Skbn supports file copy from and to a Minio S3 endpoint. To let skbn know how your minio is configured, you can set the following environment variables:
```
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_S3_ENDPOINT=http(s)://:
AWS_S3_NO_SSL=true # disables SSL
AWS_S3_FORCE_PATH_STYLE=true # enforce path style bucket access
```
## Added bonus section
### Copy files from S3 to Azure Blob Storage
```
skbn cp \
--src s3:/// \
--dst abs:////
```
### Copy files from Azure Blob Storage to S3
```
skbn cp \
--src abs://// \
--dst s3:///
```
### Copy files from Kubernetes to Kubernetes
```
skbn cp \
--src k8s:///// \
--dst k8s://///
```
### Copy files from S3 to S3
```
skbn cp \
--src s3:/// \
--dst s3:///
```
### Copy files from Azure Blob Storage to Azure Blob Storage
```
skbn cp \
--src abs://// \
--dst abs:////
```
## Credentials
### Kubernetes
Skbn tries to get credentials in the following order:
1. if `KUBECONFIG` environment variable is set - skbn will use the current context from that config file
2. if `~/.kube/config` exists - skbn will use the current context from that config file with an [out-of-cluster client configuration](https://github.com/kubernetes/client-go/tree/master/examples/out-of-cluster-client-configuration)
3. if `~/.kube/config` does not exist - skbn will assume it is working from inside a pod and will use an [in-cluster client configuration](https://github.com/kubernetes/client-go/tree/master/examples/in-cluster-client-configuration)
### AWS
Skbn uses the default AWS [credentials chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html).
In addition, the `AWS_REGION` environment variable should be set (default is `eu-central-1`).
### Azure Blob Storage
Skbn uses `AZURE_STORAGE_ACCOUNT` and `AZURE_STORAGE_ACCESS_KEY` environment variables for authentication.
## Examples
1. [In-cluster example](/examples/in-cluster)
2. [Code example](/examples/code)