Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/ggeorgovassilis/ssh-file-uploader


https://github.com/ggeorgovassilis/ssh-file-uploader

datatransfer networking scp ssh

Last synced: 8 days ago
JSON representation

Awesome Lists containing this project

README

        

ssh-file-uploader
=================

Ssh file uploader v1.0
----------------------------

This Java programme uploads large files with scp and ssh over unreliable internet connections.
It does this by splitting the large file into small parts and uploading them in parallel over multiple scp connections.
When an upload stalls, it is repeated for that part.

Usage:
----------------------------

SSHFileUploader path_to_configuration file_to_upload path_on_remote_server

Example:
----------------------------

java -jar sshfileuploader.jar /home/jack/tools/sshfileuploader.properties ./photos/backup.zip /mnt/raid/backups/2013-06-13

Example configuration file:

# Start every that many milliseconds the next upload or restart a failed upload
INTERVAL=1000

# Wait at least that many milliseconds after starting a new upload before starting another one
PAUSE_AFTER_NEW_PROCSS=500

# Maximum number of parallel uploads
MAXPROCESSES=2

# Maximum size of file part to upload in kilobytes. Rule of thumb: the more reliable the connection the larger the value you can pick.
# Chosing large values on bad connections will stall the upload and result in very low speed.
PARTSIZE_KB=64

# Fail at most that many times when merging parts at the target server
MAX_FAILURES=20

# Location of scp binary
scp=/usr/bin/scp

# Location of ssh binary
ssh=/usr/bin/ssh

# SSH login to use at target server for scp and merging
userName=sshuser

# Name or IP of server to upload files to
server=ssh.example.com

# Location for temporary file space
tmpDir=/tmp

# Logging: normal, quiet
logging=normal

# SSH / SCP arguments
sshArgs=-o ConnectTimeout=30

System requirements
--------------------------------

Java 1.6
scp
ssh

FAQ
-------------------------------

Q: Can I specify the password to use for ssh ?
A: No. SSH must be set up to allow non-interactive login (i.e. keyfile)

Q: What is the best setting for MAXPROCESSES and PARTSIZE_KB ?
A: That depends on your connection. Unreliable connections will require smaller PARTSIZE_KBs, a slow ssh handshake will benefit from
larger MAXPROCESSES. The best thing is to try it out.

Q: It hangs
A: Assuming that your network connection to the remote server is not completely dead, make sure you have disabled ControlMaster in
ssh_config. If enabled, it will simulate multiple ssh connections over the same TCP connection which defies the purpose of this programme.

Q: Merging doesn't seem to work
A: You need a bash on the remote server

About
-------------------------------

Author: George Georgovassilis
https://github.com/ggeorgovassilis/ssh-file-uploader