{"id":19756597,"url":"https://github.com/esnet/escp","last_synced_at":"2025-04-30T11:33:28.716Z","repository":{"id":47573522,"uuid":"393116708","full_name":"esnet/EScp","owner":"esnet","description":"EScp is a high performance transfer tool that uses a command line interface similar to scp.","archived":false,"fork":false,"pushed_at":"2025-04-24T21:56:46.000Z","size":626,"stargazers_count":30,"open_issues_count":0,"forks_count":6,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-04-24T22:42:56.640Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/esnet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-08-05T16:59:33.000Z","updated_at":"2025-04-24T21:56:49.000Z","dependencies_parsed_at":"2023-12-10T02:30:48.200Z","dependency_job_id":"c45ee9ed-fd4b-45c1-ad3b-86c37d17f39c","html_url":"https://github.com/esnet/EScp","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esnet%2FEScp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esnet%2FEScp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esnet%2FEScp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esnet%2FEScp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/esnet","download_url":"https://codeload.github.com/esnet/EScp/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251691633,"owners_count":21628359,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T03:16:22.720Z","updated_at":"2025-04-30T11:33:28.566Z","avatar_url":"https://github.com/esnet.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"EScp 0.9\n========\n\nEScp is an application for high speed data transfer. This version has been\ntested across the WAN at ~200 gbit/s at SC23, disk-to-disk, limited by NIC.\nEScp's command line interface is derived from [SCP](https://openssh.com) and\nthen extended to provide additional functionality. Some of the features of\nEScp include:\n\n  * Multiple TCP/IP streams per transfer ( 1/thread )\n  * Lockless multi threaded application\n  * ZeroCopy (When blocksize \u003c L3), uses shared memory, and page aligned\n  * AES-GCM 128 over the wire encryption, optimized for performance\n  * Authentication to remote systems through SSH\n  * Swappable I/O Engines ( Currently: DUMMY \u0026 POSIX )\n  * Block based interface to transferring data\n  * Compression (using zstd)\n  * Checksums, Direct I/O, API's\n  * Memory/Block Based Transfers (as well as File based transfers)\n\nIn general, the approach taken by EScp is to have algorithms optimized to give\ngood performance while providing full-set functionality. This means baseline\nsecurity (AES-GCM 128 Encryption, SSH Authorization) and File Reliability\n(Checksumming, File Verification, and Transfer Reliability) that scale with\ncore-count. Additional functionality is available through API's, CLI options,\nand so on.\n\nEScp is currently only available for x86-64 on Linux. If you would like to see\n*Platform* supported, please open a ticket at https://github.com/ESnet/EScp.\n\nRELEASE NOTES\n=============\n\n**EScp is in active development**. The latest tagged releases likely offers the\nbest stability. Currently that is 0.8.1.\n\nAt this point, most scp flags are supported although not necessarily 1:1. EScp\nadds additional flags to support high performance transfer flows.\n\nInstallation\n============\n\n```\n# Ubuntu\napt install cmake libtool g++ nasm autoconf automake rustup\ngit checkout 0.8.1\ncargo install cargo-deb\ncargo deb\nsudo dpkg -i target/debian/escp_0.8.1-1_amd64.deb\n\n# Uninstall\n# dpkg -r escp\n```\n\nMore Information\n================\n\n * [Install/Developer Build Environment](docs/install.md)\n * [Performance/Tuning](docs/perf.md)\n * [Roadmap/Notes/FAQs](docs/notes.md)\n\n\nUSAGE\n=====\n\n```\nEnergy Sciences Network transfer tool (EScp)\n\nUsage: escp [OPTIONS] [SOURCE]...\n\nArguments:\n  [SOURCE]...  [ Destination ]\n\nOptions:\n      --filelist \u003cFILE_LIST\u003e             FILE_LIST instead of SOURCE (YAML or '\\n' list) [default: ]\n  -P, --port \u003cSSH_PORT\u003e                  SSH Port\n      --escp_port \u003cESCP_PORT\u003e            ESCP Port [default: 1232]\n      --escp_portrange \u003cESCP_PORTRANGE\u003e  ESCP Port range (if unset, use 1232-1242) [default: 10]\n  -v, --verbose                          Verbose/Debug output (use with logfile)\n      --logfile \u003cLOG_FILE\u003e               Log to FILE (or syslog LOCALN where N between 0-7)\n  -q, --quiet\n  -l, --limit \u003cLIMIT\u003e                    LIMIT/thread (bytes/sec) using SO_MAX_PACING_RATE\n  -p, --preserve                         Preserve source attributes\n  -C, --compress                         Compression\n      --sparse                           Sparse file support, use with compression\n  -r, --recursive                        Copy recursively\n  -o \u003cSSH_OPTION\u003e                        SSH_OPTION to SSH\n  -S, --ssh \u003cSSH\u003e                        SSH binary [default: ssh]\n  -D, --escp \u003cESCP\u003e                      EScp binary [default: escp]\n      --blocksize \u003cBLOCK_SZ\u003e             [default: 1M]\n      --ioengine \u003cIO_ENGINE\u003e             posix,dummy [default: posix]\n  -t, --parallel \u003cTHREADS\u003e               # of IO worker threads [default: 4]\n      --bits                             Display speed in bits/s\n      --nodirect                         Disable direct (O_DIRECT) mode\n      --nochecksum                       disable checksum\n  -A, --agent                            Enable SSH Agent Forwarding\n  -c, --cipher \u003cCIPHER\u003e                  CIPHER used by SSH\n  -i, --identity \u003cIDENTITY\u003e              IDENTITY pubkey for SSH auth\n  -F, --ssh_config \u003cSSH_CONFIG\u003e          SSH_CONFIG passed to SSH [default: ]\n  -J, --jump_host \u003cJUMP_HOST\u003e            JUMP_HOST used by SSH [default: ]\n  -L, --license                          Display License\n  -h, --help                             Print help\n  -V, --version                          Print version\n\nExample:\n\n# Transfer file1 and file2 to server host using SSH\nescp file1 file2 host:/remoteDirectory\n\n\n```\n\nNotes:\n\n**filelist** option expects either a file containing newline seperated\nstrings or YAML. The YAML option allows specifying source/dest pairs,\nas an example:\n\n```\n  - [ \"SRC_A\", \"DST_A\" ]\n  - [ \"SRC_B\", \"DST_B\" ]\n```\n\n** compress ** is using zstd at compression level 3. If a block of data\nfails to compress, compression is skipped for that block.\n\n** preserve ** only applies to files at present. This should be fixed in\na later release.\n\n\n\n\nAUTHOR\n======\n\nEScp is written by Charles Shiflett with the support of [ESnet](es.net). EScp\nis a side project and is not in any way an official or supported project of\nESnet, The University of California, Lawrence Berkeley National Lab, or the\nUS Department of Energy.\n\nI'd like to thank my team at ESnet; Anne White, Goran Pejović, Dhivakaran\nMuruganantham, George Robb, Luke Baker and Shawn Kwang, as well as Brian\nTierney, Eli Dart, Ezra Kissel, Ken Miller, Eric Pouyoul, Jason Zurawski,\nSeyoung You, and Andrew Wiedlea for their support, encouragement, and\nassistance in developing EScp.\n\nLastly, thanks to you, for using this application. EScp was written for you!\nIf you are interested in contributing, you are in the unique position of\nbeing the first external collaborator to be added here.\n\nLICENSE (BSD3)\n==============\n\n```\nESnet Secure Copy (EScp) Copyright (c) 2021-24, The Regents of the\nUniversity of California, through Lawrence Berkeley National Laboratory\n(subject to receipt of any required approvals from the U.S. Dept. of\nEnergy). All rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n(1) Redistributions of source code must retain the above copyright notice,\nthis list of conditions and the following disclaimer.\n\n(2) Redistributions in binary form must reproduce the above copyright\nnotice, this list of conditions and the following disclaimer in the\ndocumentation and/or other materials provided with the distribution.\n\n(3) Neither the name of the University of California, Lawrence Berkeley\nNational Laboratory, U.S. Dept. of Energy nor the names of its contributors\nmay be used to endorse or promote products derived from this software\nwithout specific prior written permission.\n\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\nAND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\nARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\nLIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\nCONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\nSUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\nINTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\nCONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\nARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\nPOSSIBILITY OF SUCH DAMAGE.\n\nYou are under no obligation whatsoever to provide any bug fixes, patches,\nor upgrades to the features, functionality or performance of the source\ncode (\"Enhancements\") to anyone; however, if you choose to make your\nEnhancements available either publicly, or directly to Lawrence Berkeley\nNational Laboratory, without imposing a separate written license agreement\nfor such Enhancements, then you hereby grant the following license: a\nnon-exclusive, royalty-free perpetual license to install, use, modify,\nprepare derivative works, incorporate into other computer software,\ndistribute, and sublicense such enhancements or derivative works thereof,\nin binary and source code form.\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fesnet%2Fescp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fesnet%2Fescp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fesnet%2Fescp/lists"}