https://github.com/codait/cos-utils
Cloud Object Storage utility
https://github.com/codait/cos-utils
cloud-object-storage
Last synced: 3 months ago
JSON representation
Cloud Object Storage utility
- Host: GitHub
- URL: https://github.com/codait/cos-utils
- Owner: CODAIT
- License: apache-2.0
- Created: 2019-10-24T00:36:06.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2019-11-28T00:48:56.000Z (over 5 years ago)
- Last Synced: 2025-02-04T03:02:45.707Z (4 months ago)
- Topics: cloud-object-storage
- Language: Python
- Homepage:
- Size: 55.7 KB
- Stars: 0
- Watchers: 7
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Cloud Object Storage CLI
[](https://travis-ci.com/CODAIT/cos-utils) [](https://pypi.org/project/cos-utils/) 
Table of content:
- [Getting Started](#getting-started)
- [Listing Cloud Object Storage buckets](#listing-cloud-object-storage-buckets)
- [Listing the content of a Cloud Object Storage bucket](#listing-the-content-of-a-cloud-object-storage-bucket)
- [Uploading files to a Cloud Object Storage bucket](#uploading-files-to-a-cloud-object-storage-bucket)
- [Downloading files from a Cloud Object Storage bucket](#downloading-files-from-a-cloud-object-storage-bucket)
- [Removing files from a Cloud Object Storage bucket](#removing-files-from-a-cloud-object-storage-bucket)
---
## Getting started
The utility requires Python 3.6 or above.
### Installation
You can install the utility from [PyPI](https://pypi.org/project/cos-utils) or from the [source](#install-from-source).
#### Install from pypi.org
```
$ pip install cos-utils --upgrade
```
#### Install from source code
```
$ git clone https://github.com/CODAIT/cos-utils.git
$ cd cos-utils
$ pip install .
```
#### Configuration
Set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`environment variables based on your [Cloud Object Storage HMAC credentials](https://cloud.ibm.com/docs/services/cloud-object-storage/iam?topic=cloud-object-storage-service-credentials).
```
$ export AWS_ACCESS_KEY_ID=...
$ export AWS_SECRET_ACCESS_KEY=...
```
# Listing Cloud Object Storage buckets
You can run the list utility in a terminal window using the generated console script
```
$ list_buckets --help
```
or explicitly
```
$ python -m cos_utils.list_buckets --help
```
The help lists required and optional parameters.
```
usage: list_buckets [-h] pattern
List buckets in Cloud Object Storage instance.
positional arguments:
pattern Bucket name spec (supported wildcards: * and ?)
optional arguments:
-h, --help show this help message and exit
Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be
defined to run the utility.
```
## List all buckets
```
$ list_buckets *
```
> On Linux, Unix and MacOS wildcards need to be escaped to prevent shell expansion: `list_files \*`.
## Apply a filter
Use the `*` (any character) and `?` (one character) wildcards to define a filter condition.
For example, to limit output to buckets starting with `data-`:
```
$ list_buckets data-*
```
# Listing the content of a Cloud Object Storage bucket
You can run the list utility in a terminal window using the generated console script
```
$ list_files --help
```
or explicitly
```
$ python -m cos_utils.list_files --help
```
The help lists required and optional parameters.
```
usage: list_files [-h] bucket pattern
List the content of a Cloud Object Storage bucket.
positional arguments:
bucket Bucket name
pattern Object key spec (supported wildcards: * and ?)
optional arguments:
-h, --help show this help message and exit
Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be
defined to run the utility.
```
## List the content of ``
```
$ list_files *
```
> On Linux, Unix and MacOS wildcards need to be escaped to prevent shell expansion: `list_files \*`.
## Apply a filter
Use the `*` (any character) and `?` (one character) wildcards to define a filter condition.
For example, to limit output to files ending in `.png`:
```
$ list_files *.png
```
# Uploading files to a Cloud Object Storage bucket
You can run the upload utility in a terminal window using the generated console script
```
$ upload_files --help
```
or explicitly
```
$ python -m cos_utils.upload_files --help
```
The help lists required and optional parameters. The examples listed below explain them in detail.
```
usage: upload_files [-h] [-p PREFIX] [-r] [-s] [-w] bucket pattern
Upload files to a Cloud Object Storage bucket.
positional arguments:
bucket Bucket name
pattern File or directory spec (supported wildcards: * and ?)
optional arguments:
-h, --help show this help message and exit
-p PREFIX, --prefix PREFIX
Key name prefix
-r, --recursive Include files in subdirectories
-s, --squash Exclude subdirectory name from key name
-w, --wipe Clear bucket prior to upload
Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be
defined to run the utility.
```
## Example scenario
The `` contains the following directories and files:
```
file1.png
file2.png
file3.jpg
file4.txt
dir1/file5.gif
dir1/file6.png
dir1/dir2/file7.png
dir1/dir3/file8.jpg
dir1/dir3/file1.png
```
In the examples given below `` refers to an existing bucket in Cloud Object Storage.
## Upload directories
You can upload the content of any directory.
### Upload the content of `` to bucket ``
```
$ upload_files
```
Bucket `` contains the following objects:
```
file1.png
file2.png
file3.jpg
file4.txt
```
### Same as before but clear the bucket first before uploading
Specify the optional `--wipe` parameter to clear the bucket before upload.
```
$ upload_files --wipe
```
Bucket `` contains the following objects:
```
file1.png
file2.png
file3.jpg
file4.txt
```
### Same as before but include subdirectories
Specify the optional `--recursive` parameter include files in subdirectories.
```
$ upload_files --wipe --recursive
```
Bucket `` contains the following objects:
```
file1.png
file2.png
file3.jpg
file4.txt
dir1/file5.gif
dir1/file6.png
dir1/dir2/file7.png
dir1/dir3/file8.jpg
dir1/dir3/file1.png
```
### Same as before but don't use subdirectory names during object key generation
Specify the optional `--squash` parameter to ignore subdirectory names during object key generation.
```
$ upload_files --wipe --recursive --squash
```
Bucket `` contains the following objects. Note that `` contains two files named `file1.png`. First `file1.png` is uploaded and later overwritten with the content of `dir1/dir3/file1.png`.
```
file2.png
file3.jpg
file4.txt
file5.gif
file6.png
file7.png
file8.jpg
file1.png
```
### Same as before but include a static key name prefix
Specify the optional `--prefix ` parameter to add `` to the object key for every file.
```
$ upload_files --wipe --recursive --squash --prefix data
```
Bucket `` contains the following objects:
```
data/file2.png
data/file3.jpg
data/file4.txt
data/file5.gif
data/file6.png
data/file7.png
data/file8.jpg
data/file1.png
```
## Upload files
You can upload a single file by specifying ``.
```
$ upload_files /path/to/local/directory/file1.png --wipe
```
Bucket `` contains the following object:
```
file1.png
```
You can upload multiple files by specifying a pattern ``
```
$ upload_files /path/to/local/directory/*.png --wipe
```
> On Linux, Unix and MacOS wildcards need to be escaped to prevent shell expansion: `/path/to/local/directory/\*.png`.
Bucket `` contains the following objects:
```
file1.png
file2.png
```
Use the `--recursive` parameter to extend the search to subdirectories of `/path/to/local/directory/`.
```
$ upload_files /path/to/local/directory/*.png --wipe --recursive
```
```
file1.png
file2.png
dir1/file6.png
dir1/dir2/file7.png
dir1/dir3/file1.png
```
# Downloading files from a Cloud Object Storage bucket
You can run the download utility in a terminal window using the generated console script
```
$ download_files --help
```
or explicitly
```
$ python -m cos_utils.dowload_files --help
```
The help lists required and optional parameters. The examples listed below explain them in detail.
```
usage: download_files [-h] [-d TARGET_DIR] bucket pattern
Download objects from a Cloud Object Storage bucket.
positional arguments:
bucket Bucket name
pattern Object key spec (supported wildcards: * and ?)
optional arguments:
-h, --help show this help message and exit
-d TARGET_DIR, --target_dir TARGET_DIR
Local target directory. Defaults to the current
directory.
Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be
defined to run the utility.
```
### Download complete bucket content
You can download the complete content of a bucket to the current directory:
```
$ download_files *
```
> On Linux, Unix and MacOS wildcards need to be escaped to prevent shell expansion: `download_files \*`.
### Same as before but specify a target directory
Use the `--target_dir ` parameter to specify an existing directory where the downloaded files will be stored:
```
$ download_files * --target_dir /tmp/downloads
```
### Use wildcards to selectively download files
Use the `*` (any character) and `?` (one character) wildcards to define a filter condition.
#### Download only png files
```
$ download_files *.png
```
#### Download files that contain a certain string in their name
```
$ download_files *fil*
```
# Removing files from a Cloud Object Storage bucket
You can run the remove utility in a terminal window using the generated console script
```
$ remove_files --help
```
or explicitly
```
$ python -m cos_utils.remove_files --help
```
### Remove all files from a bucket
```
$ remove_files
```
# License
[Apache-2.0](LICENSE)