https://github.com/mooerslab/multiplefiletransfer
Transfer multiple files by scp without repeated manual password entry
https://github.com/mooerslab/multiplefiletransfer
Last synced: 5 months ago
JSON representation
Transfer multiple files by scp without repeated manual password entry
- Host: GitHub
- URL: https://github.com/mooerslab/multiplefiletransfer
- Owner: MooersLab
- License: mit
- Created: 2023-01-05T16:39:58.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2025-08-13T21:46:17.000Z (11 months ago)
- Last Synced: 2025-08-13T23:31:27.118Z (11 months ago)
- Language: Shell
- Size: 40 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://opensource.org/licenses/mit)
# Transfer multiple files by scp without repeated manual password entry
*Problem:* Transferring multiple tar files from a remote computer can be laborious because you are asked to enter your password after each file transfer.
This solution requires making an authentication key using the program *sshpass*. You have to have ssh installed to use this solution.
1. Generate an authentication key. You should use a short passphrase when prompted for one.
```bash
ssh-keygen -t rsa -b 4096
```
2. Copy the authentication key to the remote server(s).
```bash
ssh-copy-id bmooers@schooner2.oscer.ou.edu:/home/bmooers
```
3. You will be prompted for the passphrase the next time you use `ssh` to log in or `scp` to transfer files. The prompt with scp will occur just once instead of the number of times corresponding to the list's files.
The above steps are done only once.
The steps below must be done at the start of a session of multiple file transfers.
You will have to enter your passphrase only once.
Due to ethernet instability, you may have to limit the amount of tar files transferred to a quarter TB.
4. Enter: `eval "$(ssh-agent -s)"`
5. Enter: `ssh-add ~/.ssh/id_rsa`
Store the following two functions in your `.zshrc` or `.bashrc` or store in a .bashFunctions file and source from your `.zshrc` or `.bashrc` file.
This first function is for putting file(s) onto the OURdisk.
```bash
function dtn2mput {
print "Enter the next two lines once for a day of transfers, then you can run this script multiple times with different batches of files."
printf 'eval "$(ssh-agent -s)"'
printf 'ssh-add ~/.ssh/id_rsa'
if [ $# -lt 1 ]; then
echo 1>&2 "$0: not enough arguments. Need one filename or a list of two or more separated by whitespaces."
echo "Usage1: smbmget fileName1 fileName2"
echo "Note the dropped leading forward slash"
return 2
fi
destination="/ourdisk/hpc/bmooers/bmooers/dont_archive"
for file in "$@"; do
echo "Put '$file' to dtn2."
scp -i ~/.ssh/id_rsa -pr "$file" bmooers@dtn2.oscer.ou.edu:"$destination"/.
done
echo "All done."
echo "Function stored in ~/.bashFunctions3."
# Call the function with the list of files "$@" to be transferred to OURdisk computer.
}
```
6. Enter `dtn2mput file1.tar file2.tar file3.tar file4.tar` and so on. Note the use of whitespaces to separate the files.
This second function is for getting file9(s) from OURdisk.
```bash
function dtn2mget {
printf "Enter the next two lines once for a day of transfers, then you can run this script mutliple times with different batches of files."
printf 'eval "$(ssh-agent -s)"'
printf 'ssh-add ~/.ssh/id_rsa'
if [ $# -lt 1 ]; then
echo 1>&2 "$0: not enough arguments. Need one filename or a list of two or more separated by whitespaces."
echo "Usage1: dtn2mget fileName1 fileName2"
return 2
fi
source="/data/bmooers"
for file in "$@"; do
echo "Fetch '$file' from dtn2."
scp -i ~/.ssh/id_rsa -pr mooers@smbcopy.slac.stanford.edu:"$source"/"$file" .
done
echo "All done."
echo "Function stored in ~/.bashFunctions3."
# Call the function with the list of files "$@" to be transferred from OURdisk computer.
}
```
## Long Duration Transfers of Large Files
Start a tmux session and launch the dtn2mput function with a list of tar files.
Fortunately, the tmux session will continue despite your premature disconnections from the server.
You can launch up to 10 or more tmux sessions in parallel from an account on a HPC computer.
## Update History
|Version | Changes | Date |
|:-----------:|:-----------------------------------------------:|:---------------:|
| Version 0.3 | Fixed typos in README.md | 2024 April 10 |
| Version 0.4 | Added Solution 1. | 2024 August 13 |
| Version 0.5 | Revised dramatically. | 2025 January 12 |
| Version 0.6 | Note about using tmux. | 2025 Augues 13 |
## Sources of funding
- NIH: R01 CA242845
- NIH: R01 AI088011
- NIH: P30 CA225520 (PI: R. Mannel)
- NIH P20GM103640 and P30GM145423 (PI: A. West)