https://github.com/eborisch/rpipe

Tool for reading from stdin and writing to rclone storage.
https://github.com/eborisch/rpipe

Last synced: about 2 months ago
JSON representation

Tool for reading from stdin and writing to rclone storage.

• Host: GitHub
• URL: https://github.com/eborisch/rpipe
• Owner: eborisch
• License: mit
• Created: 2017-03-26T04:25:13.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2021-04-27T16:12:15.000Z (over 3 years ago)
• Last Synced: 2024-07-09T00:04:38.416Z (3 months ago)
• Language: Python
• Size: 9.77 KB
• Stars: 10
• Watchers: 6
• Forks: 3
• Open Issues: 0
• Metadata Files:
  • Readme: README.txt
  • License: LICENSE

Awesome Lists containing this project

README

        
# rpipe

Tool for reading from stdin and writing to rclone storage.

usage: rpipe.py [-h] [-c CHUNKSIZE] [-b BLOCKSIZE] [-t TEMPDIR] [-r] [-j JOBS]

                [-n]

                destination

Provides pipe in to / out of rclone destination

positional arguments:

  destination

optional arguments:

  -h, --help            show this help message and exit

  -c CHUNKSIZE, --chunksize CHUNKSIZE

                        Chunk size for splitting transfer [8MB]

  -b BLOCKSIZE, --blocksize BLOCKSIZE

                        Block size for read/write [64KB]

  -t TEMPDIR, --tempdir TEMPDIR

                        Directory for storing temporary files

  -r, --replay          Write previous saved stream to stdout

  -j JOBS, --jobs JOBS  Number of simultaneous rclone jobs

  -n, --nocheck         Don't check md5 at end (eg. crypto store

Works by creating temporary files of size --chunksize in --tempdir, and

uploading those. By default runs two 'jobs', such that an upload can be

occuring while the next chunk is being built. As such, tempdir needs to

be able to hold two chunks. They are deleted and checksum-ed along the

way, and verified during retrieval.

Make sure that your destination doesn't exist (purge it first.) This will

likely be added as a default step on a future version.

Examples:

     | rpipe.py remote:some/empty/loc

     | rpipe.py --nocheck crypt:an/encrypted/loc

                    ^ As we can't check the md5s of the deposited files on an

                      encrypted store...

    rpipe.py --replay remote:some/empty/loc | 

    rpipe.py --replay --nocheck crypt:an/ecrypted/loc |