https://github.com/everythingme/pystream
Stream backups directly to/from S3/HDFS without wasting disk space during the process
https://github.com/everythingme/pystream
Last synced: about 1 year ago
JSON representation
Stream backups directly to/from S3/HDFS without wasting disk space during the process
- Host: GitHub
- URL: https://github.com/everythingme/pystream
- Owner: EverythingMe
- License: bsd-2-clause
- Created: 2015-06-18T13:53:03.000Z (almost 11 years ago)
- Default Branch: master
- Last Pushed: 2015-06-19T09:23:19.000Z (almost 11 years ago)
- Last Synced: 2025-04-14T15:57:36.886Z (about 1 year ago)
- Language: Python
- Homepage:
- Size: 120 KB
- Stars: 14
- Watchers: 6
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# pystream
Stream backups directly to/from S3/HDFS without wasting disk space during the process.
This tool is a command line interface for the [smart_open](https://pypi.python.org/pypi/smart_open/) library
## Installation
`pip install pystream`
## Usage
Stream `mysqldump` directly to S3 without wasting any additional disk space during the dump process
```
mysqldump | gzip | pystream - s3://backups/mysqldump.gz
```
Restore MySQL backup directly from S3
```
pystream s3://backups/mysqldump.gz - | gunzip | mysql
```
Stream a tarball to S3
```
tar cz . | pystream - s3://backups/backup.tar.gz
```
Stream a tarball from S3
```
pystream s3://backups/backup.tar.gz - | tar xz
```
S3 `cat`
```
pystream s3://bucket/path/to/key -
```
And the usual `s3cmd cp` like usage:
```
pystream s3://bucket/path/to/key /path/on/filesystem
pystream /path/on/filesystem s3://bucket/path/to/key
```