{"id":20074114,"url":"https://github.com/juniper/splitcopy","last_synced_at":"2025-10-03T15:10:25.673Z","repository":{"id":55105870,"uuid":"173256164","full_name":"Juniper/splitcopy","owner":"Juniper","description":"Improves file transfer rates when copying files to/from JUNOS/EVO/*nix hosts.","archived":false,"fork":false,"pushed_at":"2022-11-19T10:59:38.000Z","size":619,"stargazers_count":22,"open_issues_count":0,"forks_count":5,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-17T11:50:47.250Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Juniper.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-01T07:32:48.000Z","updated_at":"2024-04-06T21:32:37.000Z","dependencies_parsed_at":"2022-08-14T12:10:25.174Z","dependency_job_id":null,"html_url":"https://github.com/Juniper/splitcopy","commit_stats":null,"previous_names":[],"tags_count":39,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Juniper%2Fsplitcopy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Juniper%2Fsplitcopy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Juniper%2Fsplitcopy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Juniper%2Fsplitcopy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Juniper","download_url":"https://codeload.github.com/Juniper/splitcopy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252580004,"owners_count":21771249,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T14:49:28.431Z","updated_at":"2025-10-03T15:10:20.630Z","avatar_url":"https://github.com/Juniper.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Splitcopy\n\nImproves file transfer rates when copying files to/from JUNOS/EVO/\\*nix hosts.\n  \nAt a minimum, sshd must be running on the remote host.  \nOn JUNOS/EVO this requires 'system services ssh' configuration.  \n\nIf using ftp to copy files then an ftp daemon must be running on the remote host.   \nOn JUNOS this requires 'system services ftp' configuration.  \nFTP is the default transfer method due to its lower resource usage and its ability to restart transfers.   \n\nScript overheads include authentication, sha hash generation/comparison, disk space check, file split and join.  \nIt can be slower than normal ftp/scp for small files as a result.\n\nBecause it opens a number of simultaneous connections,\nif the JUNOS/EVO host has connection/rate limits configured like this:\n\n```\nsystem {\n    services {\n        ssh { # or ftp\n            connection-limit 10;\n            rate-limit 10;\n        }\n    }\n}\n```\nor system login retry-options:\n\n```\nsystem {\n    login {\n        retry-options {\n            \u003c..\u003e\n        }\n    }\n}\n```\n\nThe script will deactivate these limits so it can proceed, then rollback these changes upon completion.  \n\n## Arguments \n\n`source`     Mandatory  \n`target`     Mandatory  \n`--pwd`      Optional, password  \n`--scp`      Optional, use scp instead of ftp to transfer files  \n`--ssh_key`  Optional, path to private ssh key (only required if not located in ~/.ssh/)  \n`--log`      Optional, enables additional logging, specify a logging level as argument  \n`--noverify` Optional, skips sha1 hash comparison of src and dst file  \n`--split_timeout` Optional, time to wait for remote file split operation to complete, default 120s  \n`--ssh_port` Optional, ssh port number to connect to  \n`--nocurses` Optional, disables the use of a curses window to show per-file progress statistics  \n\nThe format of source and target arguments match those of the 'scp' cmd.  \nBoth accept either a local path, or a remote path in the format - user@host:path or host@path  \n\n### To copy from local host to remote host:\n    splitcopy  \u003clocal path\u003e \u003cuser\u003e@\u003chost\u003e:\u003cpath\u003e\n### To copy from remote host to local host:\n    splitcopy  \u003cuser\u003e@\u003chost\u003e:\u003cpath\u003e \u003clocal path\u003e\n\nSupports connecting through jumphosts via 'ProxyCommand' entries in ~/.ssh/config. Example:  \n```\nHost myserver  \n  ProxyCommand ssh myjumphost.mydomain.com -W %h:%p\n```\n\n# INSTALLATION\n\nInstallation requires Python \u003e= 3.6 and associated `pip` tool  \n\n    python3 -m pip install splitcopy\n\nInstalling from Git is also supported (OS must have git installed).\n\n    To install the latest MASTER code\n    python3 -m pip install git+https://github.com/Juniper/splitcopy.git\n    -or-\n    To install a specific version, branch, tag, etc.\n    python3 -m pip install git+https://github.com/Juniper/splitcopy.git@\u003cbranch,tag,commit\u003e\n\nUpgrading has the same requirements as installation and has the same format with the addition of --upgrade\n\n    python3 -m pip install splitcopy --upgrade\n\n\n# Usage Examples \n## FTP transfer (default method)\n\n```\n$ splitcopy /var/tmp/jselective-update-ppc-J1.1-14.2R5-S3-J1.1.tgz lab@192.168.1.1:/var/tmp/\nPassword:\nchecking remote port(s) are open...\nusing FTP for file transfer\nchecking remote storage...\nsha1 not found, generating sha1...\nsplitting file...\nstarting transfer...\n100% done\ntransfer complete\njoining files...\ndeleting remote tmp directory...\ngenerating remote sha hash...\nlocal and remote sha hash match\nfile has been successfully copied to 192.168.1.1:/var/tmp/jselective-update-ppc-J1.1-14.2R5-S3-J1.1.tgz\ndata transfer = 0:00:16.831192\ntotal runtime = 0:00:31.520914\n```\n\n## SCP transfer\n\n```\n$ splitcopy lab@192.168.1.1/var/log/messages /var/tmp/ --scp  \nssh auth succeeded\nchecking remote storage...\nchecking local storage...\nsha1 not found, generating sha1...\nsplitting file...\nstarting transfer...\n100% done\ntransfer complete\njoining files...\ndeleting remote tmp directory...\ngenerating remote sha hash...\nlocal and remote sha hash match\nfile has been successfully copied to /var/tmp/messages\ndata transfer = 0:00:18.768987\ntotal runtime = 0:00:44.891370\n```\n\n## Notes on using FTP\n\nFTP is the default transfer method.  \nThe processing of each file chunk is performed by a dedicated thread  \nEach cpu core is allowed up to 5 threads, with a system max of 32 threads used  \n\nUsing FTP method will generate the following processes on the remote host:\n- for mgmt session: 1x sshd, 1x cli, 1x mgd, 1x csh\n- for transfers: up to 40x ftpd processes (depends on Python version and number of cpus as described above)\n\nIn theory, this could result in the per-user maxproc limit of 64 being exceeded:\n```\nMay  2 04:46:59   /kernel: maxproc limit exceeded by uid 2001, please see tuning(7) and login.conf(5).\n```\nThe script modulates the number of chunks to match the number of threads available   \nThe maximum number of user owned processes that could be created is \u003c= 44\n\n## Notes on using SCP\n\nThe processing of each file chunk is performed by a dedicated thread  \nEach cpu core is allowed up to 5 threads, with a system max of 32 threads used  \n\nUsing SCP method will generate the following processes on the remote host:\n- for mgmt session: 1x sshd, 1x cli, 1x mgd, 1x csh\n- for transfers:  depends on Python version, number of cpus (see above), OpenSSH and Junos FreeBSD version (see below)\n\nIn FreeBSD 11 based Junos each scp transfer creates 2 user owned processes: \n```\nlab 30366   0.0  0.0  475056   7688  -  Ss   10:39      0:00.03 cli -c scp -t /var/tmp/\nlab 30367   0.0  0.0   61324   4860  -  S    10:39      0:00.01 scp -t /var/tmp/\n```\nIn FreeBSD 10 based Junos each scp transfer creates 2 user owned processes\n```\nlab  28639   0.0  0.0  734108   4004  -  Is   12:00PM     0:00.01 cli -c scp -t /var/tmp/splitcopy_jinstall-11.4R5.5-domestic-signed.tgz/\nlab  28640   0.0  0.0   24768   3516  -  S    12:00PM     0:00.01 scp -t /var/tmp/splitcopy_jinstall-11.4R5.5-domestic-signed.tgz/\n```\nIn FreeBSD 6 based Junos each scp transfer creates 3 user owned processes: \n```\nlab  78625  0.0  0.1  2984  2144  ??  Ss    5:29AM   0:00.01 cli -c scp -t /var/tmp/splitcopy_jinstall-11.4R5.5-domestic-signed.tgz/  \nlab  78626  0.0  0.0  2252  1556  ??  S     5:29AM   0:00.00 sh -c scp -t /var/tmp/splitcopy_jinstall-11.4R5.5-domestic-signed.tgz/  \nlab  78627  0.0  0.1  3500  1908  ??  S     5:29AM   0:00.01 scp -t /var/tmp/splitcopy_jinstall-11.4R5.5-domestic-signed.tgz/  \n```\nIn addition, if OpenSSH version is \u003e= 7.4, an additional user owned process is created:\n```\nlab  2287  2.4  0.1 11912  2348  ??  S     3:49AM   0:00.15 sshd: lab@notty (sshd)\n```\nThis could result in the per-user maxproc limit of 64 being exceeded:\n```\nMay  2 04:46:59   /kernel: maxproc limit exceeded by uid 2001, please see tuning(7) and login.conf(5).\n```\nTo mitigate this, the script modulates the number of chunks to match the maximum number of simultaneous transfers possible (based on OpenSSH, Junos FreeBSD versions and the number of cpu's).  \nThe maximum number of user owned processes that could be created is \u003c= 45\n\n\n\n## LICENSE\n\nApache 2.0\n\n## CONTRIBUTORS\n\nJuniper Networks is actively contributing to and maintaining this repo. Please contact jnpr-community-netdev@juniper.net for any queries.\n\n*Contributors:*\n\n[Chris Jenn](https://github.com/ipmonk)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuniper%2Fsplitcopy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjuniper%2Fsplitcopy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuniper%2Fsplitcopy/lists"}