{"id":21541738,"url":"https://github.com/chezhang/clusrun","last_synced_at":"2025-12-31T00:16:48.038Z","repository":{"id":64303964,"uuid":"230954016","full_name":"chezhang/clusrun","owner":"chezhang","description":"Run command in cluster","archived":false,"fork":false,"pushed_at":"2022-11-08T02:35:09.000Z","size":2888,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-24T08:31:17.504Z","etag":null,"topics":["azure-vmss","docker-container","golang","grpc","hpcpack","lightweight","linux","on-premise","windows"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chezhang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-30T17:25:42.000Z","updated_at":"2022-11-08T02:35:13.000Z","dependencies_parsed_at":"2023-01-15T10:00:32.475Z","dependency_job_id":null,"html_url":"https://github.com/chezhang/clusrun","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chezhang%2Fclusrun","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chezhang%2Fclusrun/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chezhang%2Fclusrun/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chezhang%2Fclusrun/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chezhang","download_url":"https://codeload.github.com/chezhang/clusrun/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244124179,"owners_count":20401683,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure-vmss","docker-container","golang","grpc","hpcpack","lightweight","linux","on-premise","windows"],"created_at":"2024-11-24T05:07:27.924Z","updated_at":"2025-12-31T00:16:48.009Z","avatar_url":"https://github.com/chezhang.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# clusrun\n\nA simple tool to run commands in a cluster, inspired by the [clusrun](https://docs.microsoft.com/en-us/powershell/high-performance-computing/clusrun) feature from Microsoft HPC Pack.\n\n## Components\n\n\u003cimg src=\"images/components.png\" width=\"750\"\u003e\n\n- The worker executable `clusnode` or `clusnode.exe`\n\n - headnode role\n\n - receive heartbeats from clusnode role of workers and send node validatoin requests\n - receive node or job request from client\n - receive config request from client role of worker\n - dispatch job request to clusnode role of workers and redirect responses\n - store jobs information, nodes information and configs locally\n\n - clusnode role\n\n - send heartbeats to headnode role of workers and receive node validation requests \n - receive job requests from headnode role of workers\n - run command locally\n\n - client role\n\n - send config request to headnode role and clusnode role of worker\n\n- The client executable `clus` or `clus.exe`\n\n - send node or job request to headnode role of workers and receive response\n - display node or job information locally\n\n## Deployment\n\n#### Install or uninstall clusrun as service\n\nThe substance of clusrun installation is to copy the setup package to a node and to register the extracted executable as a service, the uninstallation is to unregister the service and to delete files.\n\n- Install and uninstall clusrun on physical or virtual machine\n\n 1. Download setup package from [Releases](https://github.com/chezhang/clusrun/releases) (`setup.zip` for Windows node, `setup.tar.gz` for Linux node)\n 2. Extract it to the location you want to install clusrun\n 3. Execute the install script with Administrator or root privilege to register clusrun as a service\n 4. Execute the executable `clusnode` or `clusnode.exe` to set the headnode(s) for this node to report to\n 5. Execute uninstall script to uninstall clusrun\n\n - The setup script on [Releases](https://github.com/chezhang/clusrun/releases) (`setup.ps1` for Windows node, `setup.sh` for Linux node) can help to achieve the above steps\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n \n - Install clusrun with default parameters\n \n ```PowerShell\n PS C:\\\u003e [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12\n PS C:\\\u003e (New-Object System.Net.WebClient).DownloadFile(\"https://github.com/chezhang/clusrun/releases/download/v0.2.0/setup.ps1\", \"$(pwd)\\setup.ps1\") \n PS C:\\\u003e .\\setup.ps1\n 04/14/2020 04:11:19 Setup clusrun: headnodes=localhost, location=C:\\Program Files\\clusrun, setup_url=https://github.com/chezhang/clusrun/releases/download/v0.2.0/setup.zip, reinstall=False, uninstall=False\n 04/14/2020 04:11:19 Download clusrun from https://github.com/chezhang/clusrun/releases/download/v0.2.0/setup.zip\n 04/14/2020 04:11:20 Extract clusrun from C:\\\\clusrun.setup.zip to C:\\Program Files\\clusrun\n 04/14/2020 04:11:21 Install clusrun\n [SC] CreateService SUCCESS\n\n SERVICE_NAME: clusnode\n TYPE : 10 WIN32_OWN_PROCESS\n STATE : 2 START_PENDING\n (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)\n WIN32_EXIT_CODE : 0 (0x0)\n SERVICE_EXIT_CODE : 0 (0x0)\n CHECKPOINT : 0x0\n WAIT_HINT : 0x7d0\n PID : 440\n FLAGS :\n symbolic link created for C:\\Windows\\clus.exe \u003c\u003c===\u003e\u003e C:\\Program Files\\clusrun\\clus.exe\n symbolic link created for C:\\Windows\\clusnode.exe \u003c\u003c===\u003e\u003e C:\\Program Files\\clusrun\\clusnode.exe\n 04/14/2020 04:11:24 Set headnodes to localhost\n Set headnodes result:\n \"VMSS-WIN2000001:50505\": Already connected\n 04/14/2020 04:11:24 Clusrun is installed in C:\\Program Files\\clusrun\n PS C:\\\u003e\n ```\n\n - Executable `clus.exe` and `clusnode.exe`\n \n ```PowerShell\n PS C:\\\u003e clus.exe\n\n Usage:\n clus \u003ccommand\u003e [arguments]\n\n The commands are:\n node - list nodes, add nodes to groups or remove nodes from groups in the cluster\n run - run a command or script on nodes in the cluster\n job - list, cancel or rerun jobs in the cluster\n\n Usage of node:\n clus node [options]\n clus node -h\n\n Usage of run:\n clus run [options] \u003ccommand\u003e\n clus run -h\n\n Usage of job:\n clus job [options] [jobs]\n clus job -h\n\n PS C:\\\u003e clusnode.exe\n\n Usage:\n clusnode \u003ccommand\u003e [options]\n\n The commands are:\n start - start the node\n config - configure the started node\n\n Usage of start:\n clusnode start [options]\n clusnode start -h\n\n Usage of config:\n clusnode config \u003ccommand\u003e [configs]\n clusnode config -h\n\n PS C:\\\u003e \n ```\n\n - Uninstall clusrun and keep data\n\n ```PowerShell\n PS C:\\\u003e \u0026 'C:\\Program Files\\clusrun\\uninstall.bat'\n\n SERVICE_NAME: clusnode\n TYPE : 10 WIN32_OWN_PROCESS\n STATE : 4 RUNNING\n (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)\n WIN32_EXIT_CODE : 0 (0x0)\n SERVICE_EXIT_CODE : 0 (0x0)\n CHECKPOINT : 0x0\n WAIT_HINT : 0x0\n [SC] DeleteService SUCCESS\n PS C:\\\u003e dir 'C:\\Program Files\\clusrun\\'\n\n\n Directory: C:\\Program Files\\clusrun\n\n\n Mode LastWriteTime Length Name\n ---- ------------- ------ ----\n d----- 4/14/2020 4:11 AM clusnode.exe.db\n d----- 4/14/2020 4:11 AM clusnode.exe.logs\n -a---- 4/14/2020 4:11 AM 411 clusnode.exe.config\n -a---- 4/14/2020 11:33 AM 406 uninstall.bat\n\n\n PS C:\\\u003e\n ```\n\n - Uninstall clusrun and cleanup data\n\n ```PowerShell\n PS C:\\\u003e \u0026 'C:\\Program Files\\clusrun\\uninstall.bat' -cleanup\n [SC] OpenService FAILED 1060:\n\n The specified service does not exist as an installed service.\n\n [SC] OpenService FAILED 1060:\n\n The specified service does not exist as an installed service.\n\n Could Not Find C:\\Program Files\\clusrun\\clusnode.exe\n Could Not Find C:\\Program Files\\clusrun\\clus.exe\n Could Not Find C:\\Windows\\clusnode.exe\n Could Not Find C:\\Windows\\clus.exe\n The batch file cannot be found.\n The batch file cannot be found.\n PS C:\\\u003e dir 'C:\\Program Files\\clusrun\\'\n PS C:\\\u003e\n ```\n\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n \n - Install clusrun with default parameters\n\n ```Bash\n root@ubuntu1804:~# wget https://github.com/chezhang/clusrun/releases/download/v0.2.0/setup.sh\n --2020-04-14 04:21:44-- https://github.com/chezhang/clusrun/releases/download/v0.2.0/setup.sh\n Resolving github.com (github.com)... 13.237.44.5\n Connecting to github.com (github.com)|13.237.44.5|:443... connected.\n HTTP request sent, awaiting response... 302 Found\n Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/230954016/d9cba080-7e47-11ea-9278-49c4fd988216?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200414%2Fus-east-1%2Fs3%2Faws4_request\u0026X-Amz-Date=20200414T042144Z\u0026X-Amz-Expires=300\u0026X-Amz-Signature=8a57e2c8d7c43c9d82389bae109cea50b3f1e49eb8a510119ff99a5ded01416f\u0026X-Amz-SignedHeaders=host\u0026actor_id=0\u0026repo_id=230954016\u0026response-content-disposition=attachment%3B%20filename%3Dsetup.sh\u0026response-content-type=application%2Foctet-stream [following]\n --2020-04-14 04:21:45-- https://github-production-release-asset-2e65be.s3.amazonaws.com/230954016/d9cba080-7e47-11ea-9278-49c4fd988216?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200414%2Fus-east-1%2Fs3%2Faws4_request\u0026X-Amz-Date=20200414T042144Z\u0026X-Amz-Expires=300\u0026X-Amz-Signature=8a57e2c8d7c43c9d82389bae109cea50b3f1e49eb8a510119ff99a5ded01416f\u0026X-Amz-SignedHeaders=host\u0026actor_id=0\u0026repo_id=230954016\u0026response-content-disposition=attachment%3B%20filename%3Dsetup.sh\u0026response-content-type=application%2Foctet-stream\n Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.248.196\n Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.248.196|:443... connected.\n HTTP request sent, awaiting response... 200 OK\n Length: 1213 (1.2K) [application/octet-stream]\n Saving to: ‘setup.sh’\n\n setup.sh 100%[===========================\u003e] 1.18K --.-KB/s in 0s\n\n 2020-04-14 04:21:45 (40.6 MB/s) - ‘setup.sh’ saved [1213/1213]\n\n root@ubuntu1804:~# chmod +x setup.sh\n root@ubuntu1804:~# ./setup.sh\n --2020-04-14 04:21:56-- https://github.com/chezhang/clusrun/releases/download/v0.2.0/setup.tar.gz\n Resolving github.com (github.com)... 13.237.44.5\n Connecting to github.com (github.com)|13.237.44.5|:443... connected.\n HTTP request sent, awaiting response... 302 Found\n Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/230954016/da643700-7e47-11ea-96f4-2f2796fb36f4?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200414%2Fus-east-1%2Fs3%2Faws4_request\u0026X-Amz-Date=20200414T042157Z\u0026X-Amz-Expires=300\u0026X-Amz-Signature=0bc7676267667b8d5d7d9b5b1773bdafefa254bbd9f98640bed5cffd2c2f4750\u0026X-Amz-SignedHeaders=host\u0026actor_id=0\u0026repo_id=230954016\u0026response-content-disposition=attachment%3B%20filename%3Dsetup.tar.gz\u0026response-content-type=application%2Foctet-stream [following]\n --2020-04-14 04:21:57-- https://github-production-release-asset-2e65be.s3.amazonaws.com/230954016/da643700-7e47-11ea-96f4-2f2796fb36f4?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200414%2Fus-east-1%2Fs3%2Faws4_request\u0026X-Amz-Date=20200414T042157Z\u0026X-Amz-Expires=300\u0026X-Amz-Signature=0bc7676267667b8d5d7d9b5b1773bdafefa254bbd9f98640bed5cffd2c2f4750\u0026X-Amz-SignedHeaders=host\u0026actor_id=0\u0026repo_id=230954016\u0026response-content-disposition=attachment%3B%20filename%3Dsetup.tar.gz\u0026response-content-type=application%2Foctet-stream\n Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.27.180\n Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.27.180|:443... connected.\n HTTP request sent, awaiting response... 200 OK\n Length: 13395728 (13M) [application/octet-stream]\n Saving to: ‘clusrun.setup.tar.gz’\n\n clusrun.setup.tar.gz 100%[===========================\u003e] 12.77M 4.29MB/s in 3.0s\n\n 2020-04-14 04:22:01 (4.29 MB/s) - ‘clusrun.setup.tar.gz’ saved [13395728/13395728]\n\n clus\n clusnode\n install.sh\n uninstall.sh\n /usr/local/clusrun /usr/local/clusrun\n /usr/local/clusrun\n Set headnodes result:\n \"UBUNTU1804:50505\": Already connected\n\n Clusrun is installed in /usr/local/clusrun\n\n root@ubuntu1804:~#\n ```\n\n - Executable `clus` and `clusnode`\n\n ```Bash\n root@ubuntu1804:~# clus\n\n Usage:\n clus \u003ccommand\u003e [arguments]\n\n The commands are:\n node - list nodes, add nodes to groups or remove nodes from groups in the cluster\n run - run a command or script on nodes in the cluster\n job - list, cancel or rerun jobs in the cluster\n\n Usage of node:\n clus node [options]\n clus node -h\n\n Usage of run:\n clus run [options] \u003ccommand\u003e\n clus run -h\n\n Usage of job:\n clus job [options] [jobs]\n clus job -h\n\n root@ubuntu1804:~# clusnode\n\n Usage:\n clusnode \u003ccommand\u003e [options]\n\n The commands are:\n start - start the node\n config - configure the started node\n\n Usage of start:\n clusnode start [options]\n clusnode start -h\n\n Usage of config:\n clusnode config \u003ccommand\u003e [configs]\n clusnode config -h\n\n root@ubuntu1804:~#\n ```\n\n - Uninstall clusrun and keep data\n\n ```Bash\n root@ubuntu1804:~# /usr/local/clusrun/uninstall.sh\n root@ubuntu1804:~# ll /usr/local/clusrun/\n total 24\n drwxr-xr-x 4 root root 4096 Apr 14 04:23 ./\n drwxr-xr-x 11 root root 4096 Apr 12 12:56 ../\n -rw-r--r-- 1 root root 401 Apr 14 04:22 clusnode.config\n drw-r--r-- 4 root root 4096 Apr 14 04:22 clusnode.db/\n drw-r--r-- 2 root root 4096 Apr 14 04:22 clusnode.logs/\n -rwxrwxrwx 1 root root 386 Apr 14 03:42 uninstall.sh*\n root@ubuntu1804:~#\n ```\n\n - Uninstall clusrun and cleanup data\n\n ```Bash\n root@ubuntu1804:~# /usr/local/clusrun/uninstall.sh -cleanup\n Failed to stop clusrun.service: Unit clusrun.service not loaded.\n root@ubuntu1804:~# ll /usr/local/clusrun/\n total 8\n drwxr-xr-x 2 root root 4096 Apr 14 04:23 ./\n drwxr-xr-x 11 root root 4096 Apr 12 12:56 ../\n root@ubuntu1804:~#\n ```\n\n \u003c/details\u003e\n\n- Install, update and uninstall clusrun on Azure VMSS\n\n - via PowerShell\n\n 1. Create a [VMSS](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/overview) on Azure\n 2. Optionally, create a VM in the same subnet as the headnode\n 3. Install [Azure PowerShell](https://docs.microsoft.com/en-us/powershell/azure/install-az-ps) module if it is not installed yet\n 4. [Sign in](https://docs.microsoft.com/en-us/powershell/module/az.accounts/connect-azaccount) Azure account and [select](https://docs.microsoft.com/en-us/powershell/module/az.accounts/set-azcontext) corresponding subscription in PowerShell\n 5. Download script `vmss.ps1` from [Releases](https://github.com/chezhang/clusrun/releases) and run it to install clusrun as a VMSS extension. Specify the node names of headnodes, to which the clusrun client should connect, when running the script, or the first instance in the VMSS will be used as a headnode\n\n \u003cdetails\u003e\u003csummary\u003eExample\u003c/summary\u003e\n\n - Install clusrun with default headnode\n\n ```PowerShell\n PS D:\\\u003e [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12\n PS D:\\\u003e (New-Object System.Net.WebClient).DownloadFile(\"https://github.com/chezhang/clusrun/releases/download/v0.2.0/vmss.ps1\", \"$(pwd)\\vmss.ps1\")\n PS D:\\\u003e \\vmss.ps1 -resourceGroup vmss-win -vmssName vmss-win2019\n Use the first instance vmss-win2019_1 with hostname vmss-win2000001 as headnode\n [04/14/2020 10:43:43] Current extensions:\n\n Name StatusesSummary\n ---- ---------------\n Microsoft.Azure.Geneva.GenevaMonitoring {Microsoft.Azure.Management.Compute.Models.VirtualMachineStatusCodeCount, Micr…\n\n\n [04/14/2020 10:43:43] Adding extension Install_clusrun ...\n\n\n [04/14/2020 10:47:01] Current extensions:\n\n\n Name StatusesSummary\n ---- ---------------\n Microsoft.Azure.Geneva.GenevaMonitoring {Microsoft.Azure.Management.Compute.Models.VirtualMachineStatusCodeCount}\n Install_clusrun {Microsoft.Azure.Management.Compute.Models.VirtualMachineStatusCodeCount}\n\n\n Code Count\n ---- -----\n ProvisioningState/succeeded 40\n\n PS D:\\\u003e\n ```\n\n - Update/Reinstall clusrun with default headnode\n\n ```Powershell\n PS D:\\\u003e \\vmss.ps1 -resourceGroup vmss-win -vmssName vmss-win2019\n Use the first instance vmss-win2019_1 with hostname vmss-win2000001 as headnode\n [04/14/2020 10:59:04] Current extensions:\n\n Name StatusesSummary\n ---- ---------------\n Microsoft.Azure.Geneva.GenevaMonitoring {Microsoft.Azure.Management.Compute.Models.VirtualMachineStatusCodeCount}\n Install_clusrun {Microsoft.Azure.Management.Compute.Models.VirtualMachineStatusCodeCount}\n\n\n Code Count\n ---- -----\n ProvisioningState/succeeded 40\n\n\n\n [04/14/2020 10:59:04] Removing extension Install_clusrun ...\n\n\n [04/14/2020 11:01:58] Current extensions:\n\n Name StatusesSummary\n ---- ---------------\n Microsoft.Azure.Geneva.GenevaMonitoring {Microsoft.Azure.Management.Compute.Models.VirtualMachineStatusCodeCount}\n\n\n\n [04/14/2020 11:02:00] Adding extension Install_clusrun ...\n\n\n [04/14/2020 11:04:21] Current extensions:\n\n Name StatusesSummary\n ---- ---------------\n Microsoft.Azure.Geneva.GenevaMonitoring {Microsoft.Azure.Management.Compute.Models.VirtualMachineStatusCodeCount}\n Install_clusrun {Microsoft.Azure.Management.Compute.Models.VirtualMachineStatusCodeCount}\n\n\n Code Count\n ---- -----\n ProvisioningState/succeeded 40\n\n PS D:\\\u003e\n ```\n\n - Uninstall clusrun\n\n ```Powershell\n PS D:\\\u003e \\vmss.ps1 -resourceGroup vmss-win -vmssName vmss-win2019 -uninstall\n Use the first instance vmss-win2019_1 with hostname vmss-win2000001 as headnode\n [04/14/2020 11:34:48] Current extensions:\n\n Name StatusesSummary\n ---- ---------------\n Microsoft.Azure.Geneva.GenevaMonitoring {Microsoft.Azure.Management.Compute.Models.VirtualMachineStatusCodeCount}\n Install_clusrun {Microsoft.Azure.Management.Compute.Models.VirtualMachineStatusCodeCount}\n\n\n Code Count\n ---- -----\n ProvisioningState/succeeded 40\n\n\n\n [04/14/2020 11:34:48] Removing extension Install_clusrun ...\n\n\n [04/14/2020 11:37:13] Current extensions:\n\n Name StatusesSummary\n ---- ---------------\n Microsoft.Azure.Geneva.GenevaMonitoring {Microsoft.Azure.Management.Compute.Models.VirtualMachineStatusCodeCount}\n\n\n\n [04/14/2020 11:37:17] Adding extension Uninstall_clusrun ...\n\n\n [04/14/2020 11:39:41] Current extensions:\n\n Name StatusesSummary\n ---- ---------------\n Microsoft.Azure.Geneva.GenevaMonitoring {Microsoft.Azure.Management.Compute.Models.VirtualMachineStatusCodeCount}\n Uninstall_clusrun {Microsoft.Azure.Management.Compute.Models.VirtualMachineStatusCodeCount}\n\n\n Code Count\n ---- -----\n ProvisioningState/succeeded 40\n\n\n\n [04/14/2020 11:39:43] Removing extension Uninstall_clusrun ...\n\n\n [04/14/2020 11:42:05] Current extensions:\n\n Name StatusesSummary\n ---- ---------------\n Microsoft.Azure.Geneva.GenevaMonitoring {Microsoft.Azure.Management.Compute.Models.VirtualMachineStatusCodeCount}\n\n PS D:\\\u003e\n ```\n\n \u003c/details\u003e\n\n - via Azure portal\n\n 1. Create a VMSS on Azure\n 2. Optionally, create a VM in the same subnet as the headnode\n 3. Open `Extensions` page in `Settings` of the VMSS and click `Add`\n 4. Select `Custom Script Extension` and click `Create`\n 5. Download setup script from [Releases](https://github.com/chezhang/clusrun/releases) (`setup.ps1` for Windows, `setup.sh` for Linux)\n 6. Click `Browse` in `Install extension` page, select or create a storage account and upload the setup script to a container in it, then select the setup script as the `Script file`\n 7. Set `Arguments` to specify the headnode(s) (`-headnode(s) \"\u003cheadnode(s)\u003e\"` for Windows, `-h \"\u003cheadnodes\u003e\"` for Linux)\n 8. Click `OK` to install clusrun to the VMSS as extension\n 9. Upgrade VMSS instances to the latest model to apply the extension change\n 10. Add `Custom Script Extension` again and set `Arguments` to update or uninstall clusrun (`-reinstall` or `-uninstall` for Windows, `-r` or `-u` for Linux)\n\n#### Start clusrun manually\n\n- Start clusrun on physical or virtual machine\n\n 1. Download setup package from [Releases](https://github.com/chezhang/clusrun/releases) (`setup.zip` for Windows node, `setup.tar.gz` for Linux node)\n 2. Extract it to the location you want to install clusrun\n 3. Start clusrun with extracted executable `clusnode` or `clusnode.exe`\n\n- Start clusrun inside docker container\n\n Docker container with clusrun can be started as a node on physical or virtual machine to join a cluster.\n\n - Docker container node on Linux\n\n 1. Install docker if it is not installed\n 2. Download clusrun setup package `setup.tar.gz` from [Releases](https://github.com/chezhang/clusrun/releases) and extract the executable `clusnode`\n 3. Start docker container(s) with clusrun being hosted on available port(s) of host\n\n \u003cdetails\u003e\u003csummary\u003eExample\u003c/summary\u003e\n\n ```Bash\n root@ubuntu:~# docker -v\n Docker version 18.09.7, build 2d0083d\n root@ubuntu:~# wget -q https://github.com/chezhang/clusrun/releases/download/0.2.0/setup.tar.gz\n root@ubuntu:~# tar zxvf setup.tar.gz\n clus\n clusnode\n install.sh\n uninstall.sh\n root@ubuntu:~# headnode=$(hostname):60000\n root@ubuntu:~# for port in {60000..60010}; do docker run -v $(pwd):/tmp:ro -p $port:$port -d ubuntu bash -c \"cp /tmp/clusnode . \u0026\u0026 ./clusnode start -headnodes $headnode -host $(hostname):$port\"; done\n 638f1f5703d9d20d0d1662907891e59970fbe138e7b8fdcc996ad5d89c4a4ccb\n 02658001db5ec0e504ff57e45135d69898e747c702f0aac589f6f21fceb3545d\n d925c729b2eb77a338571aabd5140f6248bd34b662c4926b9a2cd4dad14de9f0\n 64fa2ce25f18cd64184ca904293497444367c44c3bcdf256a4fc3d477a662b15\n 0e626052110baba81b0fe7b0a856fc9616640c0f8396789669744282ac4c33b9\n 77eab7045773bef6c7917f1a38cc14b6a2d1735794a02bfff00bfc1b3450f096\n b95d4600578f1b319f2506623743e8ce316819a0c938d825f285fb95f8c620ef\n 80f2fc1fd219adc8bb2f5ab5d5f19e64f0a2a01822b02fd1f06c099d7a57b82a\n 44a286831dc88e687c08f13967c007ae77cfa4567772ccf84bac46d1fc15dcc8\n 6ce13f870ff51d3aa69f34c175c8be67b80d695cf8d93657840f0cb6b33cf834\n eadf99a431416a3ccb4996c211bd5eb553ead4d37b37636acd5ceab83797e771\n root@ubuntu:~# ./clus node -headnode $headnode\n Node State\n -------------------------- -----\n 02658001DB5E(UBUNTU:60001) Ready\n 0E626052110B(UBUNTU:60004) Ready\n 44A286831DC8(UBUNTU:60008) Ready\n 638F1F5703D9(UBUNTU:60000) Ready\n 64FA2CE25F18(UBUNTU:60003) Ready\n 6CE13F870FF5(UBUNTU:60009) Ready\n 77EAB7045773(UBUNTU:60005) Ready\n 80F2FC1FD219(UBUNTU:60007) Ready\n B95D4600578F(UBUNTU:60006) Ready\n D925C729B2EB(UBUNTU:60002) Ready\n EADF99A43141(UBUNTU:60010) Ready\n ----------------------------------\n Node count: 11\n root@ubuntu:~#\n ```\n \u003c/details\u003e\n\n - Docker container node on Windows is not supported currently\n\n## Usage examples\n\n### Client executable `clus` or `clus.exe`\n\n- Show usage\n\n ```CMD or Bash\n clus\n ```\n \n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus\n\n Usage:\n clus \u003ccommand\u003e [arguments]\n\n The commands are:\n node - list nodes, add nodes to groups or remove nodes from groups in the cluster\n run - run a command or script on nodes in the cluster\n job - list, cancel or rerun jobs in the cluster\n\n Usage of node:\n clus node [options]\n clus node -h\n\n Usage of run:\n clus run [options] \u003ccommand\u003e\n clus run -h\n\n Usage of job:\n clus job [options] [jobs]\n clus job -h\n\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus\n\n Usage:\n clus \u003ccommand\u003e [arguments]\n\n The commands are:\n node - list nodes, add nodes to groups or remove nodes from groups in the cluster\n run - run a command or script on nodes in the cluster\n job - list, cancel or rerun jobs in the cluster\n\n Usage of node:\n clus node [options]\n clus node -h\n\n Usage of run:\n clus run [options] \u003ccommand\u003e\n clus run -h\n\n Usage of job:\n clus job [options] [jobs]\n clus job -h\n\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n ```CMD or Bash\n clus -h\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus -h\n\n Usage:\n clus \u003ccommand\u003e [arguments]\n\n The commands are:\n node - list nodes, add nodes to groups or remove nodes from groups in the cluster\n run - run a command or script on nodes in the cluster\n job - list, cancel or rerun jobs in the cluster\n\n Usage of node:\n clus node [options]\n clus node -h\n\n Usage of run:\n clus run [options] \u003ccommand\u003e\n clus run -h\n\n Usage of job:\n clus job [options] [jobs]\n clus job -h\n\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus -h\n\n Usage:\n clus \u003ccommand\u003e [arguments]\n\n The commands are:\n node - list nodes, add nodes to groups or remove nodes from groups in the cluster\n run - run a command or script on nodes in the cluster\n job - list, cancel or rerun jobs in the cluster\n\n Usage of node:\n clus node [options]\n clus node -h\n\n Usage of run:\n clus run [options] \u003ccommand\u003e\n clus run -h\n\n Usage of job:\n clus job [options] [jobs]\n clus job -h\n\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003eclus node\u003c/summary\u003e\n\n#### Manage nodes in cluster by command `clus node`\n\n- Show usage\n\n ```CMD or Bash\n clus node -h\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus node -h\n Usage of clus node options:\n -addgroups string\n add nodes to the specified node groups\n -format string\n format the nodes in table, list or group (default \"table\")\n -groupby string\n group the nodes by state or node group\n -groups string\n filter nodes in the specified node groups\n -headnode string\n specify the headnode to connect (default \"localhost:50505\")\n -intersect\n specify to filter nodes in intersection (union if not specified) of node groups\n -orderby string\n sort the nodes by node name or node groups (default \"name\")\n -pattern string\n filter nodes matching the specified regular expression pattern\n -removegroups string\n remove nodes from the specified node groups\n -state string\n filter nodes in the specified state (ready, error or lost)\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus node -h\n Usage of clus node options:\n -addgroups string\n add nodes to the specified node groups\n -format string\n format the nodes in table, list or group (default \"table\")\n -groupby string\n group the nodes by state or node group\n -groups string\n filter nodes in the specified node groups\n -headnode string\n specify the headnode to connect (default \"localhost:50505\")\n -intersect\n specify to filter nodes in intersection (union if not specified) of node groups\n -orderby string\n sort the nodes by node name or node groups (default \"name\")\n -pattern string\n filter nodes matching the specified regular expression pattern\n -removegroups string\n remove nodes from the specified node groups\n -state string\n filter nodes in the specified state (ready, error or lost)\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- List all nodes in the cluster\n\n ```CMD or Bash\n clus node\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus node\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n WIN2016(WIN2016:60000) Lost\n --------------------------------------------------------------\n Node count: 4\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus node\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n WIN2016(WIN2016:60000) Lost\n --------------------------------------------------------------\n Node count: 4\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- List nodes with filters\n\n ```CMD or Bash\n clus node -groups \u003cnode groups\u003e [-intersec]\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus node -groups windows,vm\n Node State Groups\n --------------- ----- -----------\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n -------------------------------------\n Node count: 2\n\n c:\\\u003eclus node -groups windows,vm -intersect\n Node State Groups\n ------- ----- -----------\n WIN2016 Ready vm, windows\n -----------------------------\n Node count: 1\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus node -groups linux,container\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n --------------------------------------------------------------\n Node count: 2\n root@vmss-ubun000000:~# clus node -groups linux,container -intersect\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n --------------------------------------------------------------\n Node count: 1\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n ```CMD or Bash\n clus node -pattern \u003cregular expression\u003e\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus node -pattern VMSS-UBUN000000\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n --------------------------------------------------------------\n Node count: 2\n\n c:\\\u003eclus node -pattern ^VMSS-UBUN000000$\n Node State Groups\n --------------- ----- ---------\n VMSS-UBUN000000 Ready linux, vm\n -----------------------------------\n Node count: 1\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus node -pattern VMSS-UBUN000000\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n --------------------------------------------------------------\n Node count: 2\n root@vmss-ubun000000:~# clus node -pattern ^VMSS-UBUN000000$\n Node State Groups\n --------------- ----- ---------\n VMSS-UBUN000000 Ready linux, vm\n -----------------------------------\n Node count: 1\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n ```CMD or Bash\n clus node -state \u003cready/error/lost\u003e\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus node -state ready\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n --------------------------------------------------------------\n Node count: 3\n\n c:\\\u003eclus node -state error\n Node count: 0\n\n c:\\\u003eclus node -state lost\n Node State\n ---------------------- -----\n WIN2016(WIN2016:60000) Lost\n ------------------------------\n Node count: 1\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus node -state ready\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n --------------------------------------------------------------\n Node count: 3\n root@vmss-ubun000000:~# clus node -state error\n Node count: 0\n root@vmss-ubun000000:~# clus node -state lost\n Node State\n ---------------------- -----\n WIN2016(WIN2016:60000) Lost\n ------------------------------\n Node count: 1\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- List nodes with different displaying format\n\n ```CMD or Bash\n clus node -format \u003ctable/list/group\u003e\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus node -format table\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n WIN2016(WIN2016:60000) Lost\n --------------------------------------------------------------\n Node count: 4\n\n c:\\\u003eclus node -format list\n Node : 67A226CD8E76(VMSS-UBUN000000:60000)\n State : Ready\n Groups : container, linux\n ----------------------------------------------------------------------------\n Node : VMSS-UBUN000000\n State : Ready\n Groups : linux, vm\n ----------------------------------------------------------------------------\n Node : WIN2016\n State : Ready\n Groups : vm, windows\n ----------------------------------------------------------------------------\n Node : WIN2016(WIN2016:60000)\n State : Lost\n ----------------------------------------------------------------------------\n Node count: 4\n\n c:\\\u003eclus node -format group -groupby nodegroup\n ----------------------------------------------------------------------------\n WIN2016(WIN2016:60000)\n --------------------------------[container]---------------------------------\n 67A226CD8E76(VMSS-UBUN000000:60000)\n ----------------------------------[linux]-----------------------------------\n 67A226CD8E76(VMSS-UBUN000000:60000) VMSS-UBUN000000\n\n ------------------------------------[vm]------------------------------------\n VMSS-UBUN000000 WIN2016\n ---------------------------------[windows]----------------------------------\n WIN2016\n ----------------------------------------------------------------------------\n Count of nodes in no group: 1\n Count of nodes in group 'container': 1\n Count of nodes in group 'linux': 2\n Count of nodes in group 'vm': 2\n Count of nodes in group 'windows': 1\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus node -format table\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n WIN2016(WIN2016:60000) Lost\n --------------------------------------------------------------\n Node count: 4\n root@vmss-ubun000000:~# clus node -format list\n Node : 67A226CD8E76(VMSS-UBUN000000:60000)\n State : Ready\n Groups : container, linux\n -------------------------------------------------------------------\n Node : VMSS-UBUN000000\n State : Ready\n Groups : linux, vm\n -------------------------------------------------------------------\n Node : WIN2016\n State : Ready\n Groups : vm, windows\n -------------------------------------------------------------------\n Node : WIN2016(WIN2016:60000)\n State : Lost\n -------------------------------------------------------------------\n Node count: 4\n root@vmss-ubun000000:~# clus node -format group -groupby nodegroup\n -------------------------------------------------------------------\n WIN2016(WIN2016:60000)\n ----------------------------[container]----------------------------\n 67A226CD8E76(VMSS-UBUN000000:60000)\n\n ------------------------------[linux]------------------------------\n 67A226CD8E76(VMSS-UBUN000000:60000)\n VMSS-UBUN000000\n\n -------------------------------[vm]--------------------------------\n VMSS-UBUN000000 WIN2016\n -----------------------------[windows]-----------------------------\n WIN2016\n -------------------------------------------------------------------\n Count of nodes in no group: 1\n Count of nodes in group 'container': 1\n Count of nodes in group 'linux': 2\n Count of nodes in group 'vm': 2\n Count of nodes in group 'windows': 1\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n ```CMD or Bash\n clus node -groupby \u003cstate/nodegroup\u003e\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus node -groupby state\n Node State Groups\n ----------------------------------- ----- ----------------\n WIN2016(WIN2016:60000) Lost\n\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n --------------------------------------------------------------\n Node count: 4\n\n c:\\\u003eclus node -groupby nodegroup\n Node State Groups\n ----------------------------------- ----- ----------------\n WIN2016(WIN2016:60000) Lost\n\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n\n VMSS-UBUN000000 Ready linux, vm\n\n WIN2016 Ready vm, windows\n --------------------------------------------------------------\n Node count: 4\n\n c:\\\u003eclus node -groupby state -format group\n -----------------------------------[Lost]-----------------------------------\n WIN2016(WIN2016:60000)\n ----------------------------------[Ready]-----------------------------------\n 67A226CD8E76(VMSS-UBUN000000:60000) VMSS-UBUN000000\n WIN2016\n ----------------------------------------------------------------------------\n Count of nodes in state 'Lost': 1\n Count of nodes in state 'Ready': 3\n\n c:\\\u003eclus node -groupby nodegroup -format group\n ----------------------------------------------------------------------------\n WIN2016(WIN2016:60000)\n --------------------------------[container]---------------------------------\n 67A226CD8E76(VMSS-UBUN000000:60000)\n ----------------------------------[linux]-----------------------------------\n 67A226CD8E76(VMSS-UBUN000000:60000) VMSS-UBUN000000\n\n ------------------------------------[vm]------------------------------------\n VMSS-UBUN000000 WIN2016\n ---------------------------------[windows]----------------------------------\n WIN2016\n ----------------------------------------------------------------------------\n Count of nodes in no group: 1\n Count of nodes in group 'container': 1\n Count of nodes in group 'linux': 2\n Count of nodes in group 'vm': 2\n Count of nodes in group 'windows': 1\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus node -groupby state\n Node State Groups\n ----------------------------------- ----- ----------------\n WIN2016(WIN2016:60000) Lost\n\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n --------------------------------------------------------------\n Node count: 4\n root@vmss-ubun000000:~# clus node -groupby nodegroup\n Node State Groups\n ----------------------------------- ----- ----------------\n WIN2016(WIN2016:60000) Lost\n\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n\n VMSS-UBUN000000 Ready linux, vm\n\n WIN2016 Ready vm, windows\n --------------------------------------------------------------\n Node count: 4\n root@vmss-ubun000000:~# clus node -groupby state -format group\n ------------------------------[Lost]-------------------------------\n WIN2016(WIN2016:60000)\n ------------------------------[Ready]------------------------------\n 67A226CD8E76(VMSS-UBUN000000:60000)\n VMSS-UBUN000000\n WIN2016\n\n -------------------------------------------------------------------\n Count of nodes in state 'Lost': 1\n Count of nodes in state 'Ready': 3\n root@vmss-ubun000000:~# clus node -groupby nodegroup -format group\n -------------------------------------------------------------------\n WIN2016(WIN2016:60000)\n ----------------------------[container]----------------------------\n 67A226CD8E76(VMSS-UBUN000000:60000)\n\n ------------------------------[linux]------------------------------\n 67A226CD8E76(VMSS-UBUN000000:60000)\n VMSS-UBUN000000\n\n -------------------------------[vm]--------------------------------\n VMSS-UBUN000000 WIN2016\n -----------------------------[windows]-----------------------------\n WIN2016\n -------------------------------------------------------------------\n Count of nodes in no group: 1\n Count of nodes in group 'container': 1\n Count of nodes in group 'linux': 2\n Count of nodes in group 'vm': 2\n Count of nodes in group 'windows': 1\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n ```CMD or Bash\n clus node -orderby \u003cname/nodegroups\u003e\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus node -orderby name\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n WIN2016(WIN2016:60000) Lost\n --------------------------------------------------------------\n Node count: 4\n\n c:\\\u003eclus node -orderby nodegroup\n Node State Groups\n ----------------------------------- ----- ----------------\n WIN2016(WIN2016:60000) Lost\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n --------------------------------------------------------------\n Node count: 4\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus node -orderby name\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n WIN2016(WIN2016:60000) Lost\n --------------------------------------------------------------\n Node count: 4\n root@vmss-ubun000000:~# clus node -orderby nodegroup\n Node State Groups\n ----------------------------------- ----- ----------------\n WIN2016(WIN2016:60000) Lost\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n --------------------------------------------------------------\n Node count: 4\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- Add or remove node groups\n\n ```CMD or Bash\n clus node -addgroups \u003cnode groups\u003e [-groups] [-pattern] [-state]\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus node -addgroups windows -pattern WIN2016\n Node State Groups\n ---------------------- ----- -----------\n WIN2016 Ready vm, windows\n WIN2016(WIN2016:60000) Lost windows\n --------------------------------------------\n Node count: 2\n Nodes are added to node groups: windows\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus node -addgroups test\n Node State Groups\n ----------------------------------- ----- ----------------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux, test\n VMSS-UBUN000000 Ready linux, test, vm\n WIN2016 Ready test, vm, windows\n WIN2016(WIN2016:60000) Lost test\n --------------------------------------------------------------------\n Node count: 4\n Nodes are added to node groups: test\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n ```CMD or Bash\n clus node -removegroups \u003cnode groups\u003e [-groups] [-pattern] [-state]\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus node -removegroups windows -pattern WIN2016:60000\n Node State\n ---------------------- -----\n WIN2016(WIN2016:60000) Lost\n ------------------------------\n Node count: 1\n Nodes are removed from node groups: windows\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus node -removegroups test\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n WIN2016(WIN2016:60000) Lost\n --------------------------------------------------------------\n Node count: 4\n Nodes are removed from node groups: test\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- List nodes in specified cluster\n\n ```CMD or Bash\n clus node -headnode \u003cnode host\u003e\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus node -headnode VMSS-UBUN000000\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n WIN2016(WIN2016:60000) Lost\n --------------------------------------------------------------\n Node count: 4\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus node -headnode win2016\n Node State Groups\n ----------------------------------- ----- ----------------\n 67A226CD8E76(VMSS-UBUN000000:60000) Ready container, linux\n VMSS-UBUN000000 Ready linux, vm\n WIN2016 Ready vm, windows\n WIN2016(WIN2016:60000) Lost\n --------------------------------------------------------------\n Node count: 4\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003eclus run\u003c/summary\u003e\n\n#### Run command in cluster by command `clus run`\n\n- Show usage\n\n ```CMD or Bash\n clus run\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus run\n\n Usage:\n clus run [options] \u003ccommand\u003e\n\n Options:\n -background\n run command without printing output\n -cache int\n specify the number of characters to cache and display for output of command on each node (default 1000)\n -dump\n save the output to file\n -groups string\n specify certain node groups to run the command\n -headnode string\n specify the headnode to connect (default \"localhost:50505\")\n -intersect\n specify to run the command in intersection (union if not specified) of node groups\n -nodes string\n specify certain nodes to run the command\n -pattern string\n specify nodes matching a certain regular expression pattern to run the command\n -prompt int\n specify the number of nodes, the output of which will be displayed promptly (default 1)\n -script string\n specify the script file containing commands to run\n -sweep string\n perform parametric sweep by replacing specified placeholder string in the command on each node to sequence number (in specified range and step optionally) with format \"placeholder[{begin[-end][:step]}]\"\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus run\n\n Usage:\n clus run [options] \u003ccommand\u003e\n\n Options:\n -background\n run command without printing output\n -cache int\n specify the number of characters to cache and display for output of command on each node (default 1000)\n -dump\n save the output to file\n -groups string\n specify certain node groups to run the command\n -headnode string\n specify the headnode to connect (default \"localhost:50505\")\n -intersect\n specify to run the command in intersection (union if not specified) of node groups\n -nodes string\n specify certain nodes to run the command\n -pattern string\n specify nodes matching a certain regular expression pattern to run the command\n -prompt int\n specify the number of nodes, the output of which will be displayed promptly (default 1)\n -script string\n specify the script file containing commands to run\n -sweep string\n perform parametric sweep by replacing specified placeholder string in the command on each node to sequence number (in specified range and step optionally) with format \"placeholder[{begin[-end][:step]}]\"\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n ```CMD or Bash\n clus run -h\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus run -h\n Usage of clus run options:\n -background\n run command without printing output\n -cache int\n specify the number of characters to cache and display for output of command on each node (default 1000)\n -dump\n save the output to file\n -groups string\n specify certain node groups to run the command\n -headnode string\n specify the headnode to connect (default \"localhost:50505\")\n -intersect\n specify to run the command in intersection (union if not specified) of node groups\n -nodes string\n specify certain nodes to run the command\n -pattern string\n specify nodes matching a certain regular expression pattern to run the command\n -prompt int\n specify the number of nodes, the output of which will be displayed promptly (default 1)\n -script string\n specify the script file containing commands to run\n -sweep string\n perform parametric sweep by replacing specified placeholder string in the command on each node to sequence number (in specified range and step optionally) with format \"placeholder[{begin[-end][:step]}]\"\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus run -h\n Usage of clus run options:\n -background\n run command without printing output\n -cache int\n specify the number of characters to cache and display for output of command on each node (default 1000)\n -dump\n save the output to file\n -groups string\n specify certain node groups to run the command\n -headnode string\n specify the headnode to connect (default \"localhost:50505\")\n -intersect\n specify to run the command in intersection (union if not specified) of node groups\n -nodes string\n specify certain nodes to run the command\n -pattern string\n specify nodes matching a certain regular expression pattern to run the command\n -prompt int\n specify the number of nodes, the output of which will be displayed promptly (default 1)\n -script string\n specify the script file containing commands to run\n -sweep string\n perform parametric sweep by replacing specified placeholder string in the command on each node to sequence number (in specified range and step optionally) with format \"placeholder[{begin[-end][:step]}]\"\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- Run command in the cluster\n\n ```CMD or Bash\n clus run \u003ccommand\u003e\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus run hostname\n Job 1 started on 3 nodes in cluster localhost:50505.\n\n ----------------------------------Command-----------------------------------\n hostname\n ----------------------------------------------------------------------------\n\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 67a226cd8e76\n [1/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 108.8428ms.\n [2/3] Command finished on node VMSS-UBUN000000 in 109.8266ms.\n [3/3] Command finished on node WIN2016 in 112.8264ms.\n\n -------------------[67A226CD8E76(VMSS-UBUN000000:60000)]--------------------\n 67a226cd8e76\n\n -----------------------------[VMSS-UBUN000000]------------------------------\n vmss-ubun000000\n\n ---------------------------------[WIN2016]----------------------------------\n win2016\n\n ----------------------------------------------------------------------------\n Runtime:\n Min=108.8428ms\n Max=112.8264ms\n Mean=110.4986ms\n Mid=109.8266ms\n SD=1.694295ms\n\n 3 of 3 nodes succeeded.\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus run hostname\n Job 72 started on 3 nodes in cluster localhost:50505.\n\n ---------------------------------Command----------------------------------\n hostname\n --------------------------------------------------------------------------\n\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 67a226cd8e76\n [1/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 59.975ms.\n [2/3] Command finished on node VMSS-UBUN000000 in 62.5264ms.\n [3/3] Command finished on node WIN2016 in 109.8528ms.\n\n ------------------[67A226CD8E76(VMSS-UBUN000000:60000)]-------------------\n 67a226cd8e76\n\n ----------------------------[VMSS-UBUN000000]-----------------------------\n vmss-ubun000000\n\n --------------------------------[WIN2016]---------------------------------\n win2016\n\n --------------------------------------------------------------------------\n Runtime:\n Min=59.975ms\n Max=109.8528ms\n Mean=77.4514ms\n Mid=62.5264ms\n SD=22.934914ms\n\n 3 of 3 nodes succeeded.\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- Run command on certain nodes\n\n ```CMD or Bash\n clus run -groups \u003cnode groups\u003e [-intersect]\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus run -groups vm hostname\n Job 2 started on 2 nodes in cluster localhost:50505.\n\n ----------------------------------Command-----------------------------------\n hostname\n ----------------------------------------------------------------------------\n\n [VMSS-UBUN000000]: vmss-ubun000000\n [1/2] Command finished on node VMSS-UBUN000000 in 112.6209ms.\n [2/2] Command finished on node WIN2016 in 114.6214ms.\n\n -----------------------------[VMSS-UBUN000000]------------------------------\n vmss-ubun000000\n\n ---------------------------------[WIN2016]----------------------------------\n win2016\n\n ----------------------------------------------------------------------------\n Runtime:\n Min=112.6209ms\n Max=114.6214ms\n Mean=113.62115ms\n Mid=113.62115ms\n SD=1.00025ms\n\n 2 of 2 nodes succeeded.\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus run -groups vm hostname\n Job 73 started on 2 nodes in cluster localhost:50505.\n\n ---------------------------------Command----------------------------------\n hostname\n --------------------------------------------------------------------------\n\n [VMSS-UBUN000000]: vmss-ubun000000\n [1/2] Command finished on node VMSS-UBUN000000 in 51.0999ms.\n [2/2] Command finished on node WIN2016 in 94.0386ms.\n\n ----------------------------[VMSS-UBUN000000]-----------------------------\n vmss-ubun000000\n\n --------------------------------[WIN2016]---------------------------------\n win2016\n\n --------------------------------------------------------------------------\n Runtime:\n Min=51.0999ms\n Max=94.0386ms\n Mean=72.56925ms\n Mid=72.56925ms\n SD=21.46935ms\n\n 2 of 2 nodes succeeded.\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n ```CMD or Bash\n clus run -pattern \u003cregular expression\u003e\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus run -pattern WIN2016 hostname\n Job 3 started on 1 nodes in cluster localhost:50505.\n\n ----------------------------------Command-----------------------------------\n hostname\n ----------------------------------------------------------------------------\n\n [WIN2016]: win2016\n [1/1] Command finished on node WIN2016 in 70.9979ms.\n\n ---------------------------------[WIN2016]----------------------------------\n win2016\n\n ----------------------------------------------------------------------------\n Runtime: Min=70.9979ms, Max=70.9979ms, Mean=70.9979ms, Mid=70.9979ms, SD=0s\n 1 of 1 nodes succeeded.\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus run -pattern VMSS hostname\n Job 74 started on 2 nodes in cluster localhost:50505.\n\n ---------------------------------Command----------------------------------\n hostname\n --------------------------------------------------------------------------\n\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 67a226cd8e76\n [1/2] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 54.392ms.\n [2/2] Command finished on node VMSS-UBUN000000 in 63.4899ms.\n\n ------------------[67A226CD8E76(VMSS-UBUN000000:60000)]-------------------\n 67a226cd8e76\n\n ----------------------------[VMSS-UBUN000000]-----------------------------\n vmss-ubun000000\n\n --------------------------------------------------------------------------\n Runtime:\n Min=54.392ms\n Max=63.4899ms\n Mean=58.94095ms\n Mid=58.94095ms\n SD=4.54895ms\n\n 2 of 2 nodes succeeded.\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n ```CMD or Bash\n clus run -nodes \u003cnode names\u003e\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus run -nodes WIN2016,VMSS-UBUN000000 hostname\n Job 5 started on 2 nodes in cluster localhost:50505.\n\n ----------------------------------Command-----------------------------------\n hostname\n ----------------------------------------------------------------------------\n\n [VMSS-UBUN000000]: vmss-ubun000000\n [1/2] Command finished on node VMSS-UBUN000000 in 56.158ms.\n [2/2] Command finished on node WIN2016 in 69.1561ms.\n\n -----------------------------[VMSS-UBUN000000]------------------------------\n vmss-ubun000000\n\n ---------------------------------[WIN2016]----------------------------------\n win2016\n\n ----------------------------------------------------------------------------\n Runtime:\n Min=56.158ms\n Max=69.1561ms\n Mean=62.65705ms\n Mid=62.65705ms\n SD=6.49905ms\n\n 2 of 2 nodes succeeded.\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus run -nodes VMSS-UBUN000000,WIN2016 hostname Job 75 started on 2 nodes in cluster localhost:50505.\n\n ---------------------------------Command----------------------------------\n hostname\n --------------------------------------------------------------------------\n\n [VMSS-UBUN000000]: vmss-ubun000000\n [1/2] Command finished on node VMSS-UBUN000000 in 51.4004ms.\n [2/2] Command finished on node WIN2016 in 94.261ms.\n\n ----------------------------[VMSS-UBUN000000]-----------------------------\n vmss-ubun000000\n\n --------------------------------[WIN2016]---------------------------------\n win2016\n\n --------------------------------------------------------------------------\n Runtime:\n Min=51.4004ms\n Max=94.261ms\n Mean=72.8307ms\n Mid=72.8307ms\n SD=21.4303ms\n\n 2 of 2 nodes succeeded.\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- Run command in background\n\n ```CMD or Bash\n clus run -backgroud\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus run -background hostname\n Job 6 started on 3 nodes in cluster localhost:50505.\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus run -background hostname\n Job 76 started on 3 nodes in cluster localhost:50505.\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- Run command with specified cache size\n\n ```CMD or Bash\n clus run -cache \u003ccharacter number\u003e\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus run -cache 5 hostname\n Job 7 started on 3 nodes in cluster localhost:50505.\n\n ----------------------------------Command-----------------------------------\n hostname\n ----------------------------------------------------------------------------\n\n [1/3] Command finished on node VMSS-UBUN000000 in 57.4392ms.\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 67a226cd8e76\n [2/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 60.4381ms.\n [3/3] Command finished on node WIN2016 in 73.4352ms.\n\n -------------------[67A226CD8E76(VMSS-UBUN000000:60000)]--------------------\n (Truncated)\n ...8e76\n\n -----------------------------[VMSS-UBUN000000]------------------------------\n (Truncated)\n ...0000\n\n ---------------------------------[WIN2016]----------------------------------\n (Truncated)\n ...016\n\n ----------------------------------------------------------------------------\n Runtime:\n Min=57.4392ms\n Max=73.4352ms\n Mean=63.770833ms\n Mid=60.4381ms\n SD=6.942542ms\n\n 3 of 3 nodes succeeded.\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus run -cache 5 hostname\n Job 77 started on 3 nodes in cluster localhost:50505.\n\n ---------------------------------Command----------------------------------\n hostname\n --------------------------------------------------------------------------\n\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 67a226cd8e76\n [1/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 53.1031ms.\n [2/3] Command finished on node VMSS-UBUN000000 in 58.7311ms.\n [3/3] Command finished on node WIN2016 in 88.8155ms.\n\n ------------------[67A226CD8E76(VMSS-UBUN000000:60000)]-------------------\n (Truncated)\n ...8e76\n\n ----------------------------[VMSS-UBUN000000]-----------------------------\n (Truncated)\n ...0000\n\n --------------------------------[WIN2016]---------------------------------\n (Truncated)\n ...016\n\n --------------------------------------------------------------------------\n Runtime:\n Min=53.1031ms\n Max=88.8155ms\n Mean=66.883233ms\n Mid=58.7311ms\n SD=15.67773ms\n\n 3 of 3 nodes succeeded.\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- Run command displaying output of different number of nodes promptly\n\n ```CMD or Bash\n clus run -prompt \u003cnode number\u003e\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus run -prompt 10 hostname\n Job 8 started on 3 nodes in cluster localhost:50505.\n\n ----------------------------------Command-----------------------------------\n hostname\n ----------------------------------------------------------------------------\n\n [VMSS-UBUN000000]: vmss-ubun000000\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 67a226cd8e76\n [1/3] Command finished on node VMSS-UBUN000000 in 28.0018ms.\n [2/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 30.0007ms.\n [WIN2016]: win2016\n [3/3] Command finished on node WIN2016 in 64.2562ms.\n\n -------------------[67A226CD8E76(VMSS-UBUN000000:60000)]--------------------\n 67a226cd8e76\n\n -----------------------------[VMSS-UBUN000000]------------------------------\n vmss-ubun000000\n\n ---------------------------------[WIN2016]----------------------------------\n win2016\n\n ----------------------------------------------------------------------------\n Runtime:\n Min=28.0018ms\n Max=64.2562ms\n Mean=40.7529ms\n Mid=30.0007ms\n SD=16.639365ms\n\n 3 of 3 nodes succeeded.\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus run -prompt 10 hostname\n Job 78 started on 3 nodes in cluster localhost:50505.\n\n ---------------------------------Command----------------------------------\n hostname\n --------------------------------------------------------------------------\n\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 67a226cd8e76\n [1/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 54.4323ms.\n [VMSS-UBUN000000]: vmss-ubun000000\n [2/3] Command finished on node VMSS-UBUN000000 in 61.0101ms.\n [WIN2016]: win2016\n [3/3] Command finished on node WIN2016 in 98.7498ms.\n\n ------------------[67A226CD8E76(VMSS-UBUN000000:60000)]-------------------\n 67a226cd8e76\n\n ----------------------------[VMSS-UBUN000000]-----------------------------\n vmss-ubun000000\n\n --------------------------------[WIN2016]---------------------------------\n win2016\n\n --------------------------------------------------------------------------\n Runtime:\n Min=54.4323ms\n Max=98.7498ms\n Mean=71.3974ms\n Mid=61.0101ms\n SD=19.5266ms\n\n 3 of 3 nodes succeeded.\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- Run command and dump output\n\n ```CMD or Bash\n clus run -dump\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus run -dump hostname\n Job 9 started on 3 nodes in cluster localhost:50505.\n Dumping output to c:\\clus.run.20200415174918.800091500\n\n ----------------------------------Command-----------------------------------\n hostname\n ----------------------------------------------------------------------------\n\n [1/3] Command finished on node VMSS-UBUN000000 in 31.0957ms.\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 67a226cd8e76\n [2/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 64.5129ms.\n [3/3] Command finished on node WIN2016 in 83.4806ms.\n\n -------------------[67A226CD8E76(VMSS-UBUN000000:60000)]--------------------\n 67a226cd8e76\n\n -----------------------------[VMSS-UBUN000000]------------------------------\n vmss-ubun000000\n\n ---------------------------------[WIN2016]----------------------------------\n win2016\n\n ----------------------------------------------------------------------------\n Runtime:\n Min=31.0957ms\n Max=83.4806ms\n Mean=59.6964ms\n Mid=64.5129ms\n SD=21.655537ms\n\n 3 of 3 nodes succeeded.\n Output is dumped to c:\\clus.run.20200415174918.800091500\n\n c:\\\u003edir clus.run.20200415174918.800091500\n Volume in drive C is Windows\n Volume Serial Number is 56D1-9C35\n\n Directory of c:\\clus.run.20200415174918.800091500\n\n 04/15/2020 05:49 PM \u003cDIR\u003e .\n 04/15/2020 05:49 PM \u003cDIR\u003e ..\n 04/15/2020 05:49 PM 0 67A226CD8E76(VMSS-UBUN000000.60000).err\n 04/15/2020 05:49 PM 13 67A226CD8E76(VMSS-UBUN000000.60000).out\n 04/15/2020 05:49 PM 0 VMSS-UBUN000000.err\n 04/15/2020 05:49 PM 16 VMSS-UBUN000000.out\n 04/15/2020 05:49 PM 0 WIN2016.err\n 04/15/2020 05:49 PM 9 WIN2016.out\n 6 File(s) 38 bytes\n 2 Dir(s) 124,614,762,496 bytes free\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus run -dump hostname\n Job 79 started on 3 nodes in cluster localhost:50505.\n Dumping output to /root/clus.run.20200415174944.716807900\n\n ---------------------------------Command----------------------------------\n hostname\n --------------------------------------------------------------------------\n\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 67a226cd8e76\n [1/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 47.7575ms.\n [2/3] Command finished on node VMSS-UBUN000000 in 53.568ms.\n [3/3] Command finished on node WIN2016 in 90.1797ms.\n\n ------------------[67A226CD8E76(VMSS-UBUN000000:60000)]-------------------\n 67a226cd8e76\n\n ----------------------------[VMSS-UBUN000000]-----------------------------\n vmss-ubun000000\n\n --------------------------------[WIN2016]---------------------------------\n win2016\n\n --------------------------------------------------------------------------\n Runtime:\n Min=47.7575ms\n Max=90.1797ms\n Mean=63.835066ms\n Mid=53.568ms\n SD=18.778893ms\n\n 3 of 3 nodes succeeded.\n Output is dumped to /root/clus.run.20200415174944.716807900\n root@vmss-ubun000000:~# ll clus.run.20200415174944.716807900/\n total 20\n drw-r--r-- 2 root root 4096 Apr 15 17:49 ./\n drwx------ 8 root root 4096 Apr 15 17:49 ../\n -rw-r--r-- 1 root root 0 Apr 15 17:49 '67A226CD8E76(VMSS-UBUN000000.60000).err'\n -rw-r--r-- 1 root root 13 Apr 15 17:49 '67A226CD8E76(VMSS-UBUN000000.60000).out'\n -rw-r--r-- 1 root root 0 Apr 15 17:49 VMSS-UBUN000000.err\n -rw-r--r-- 1 root root 16 Apr 15 17:49 VMSS-UBUN000000.out\n -rw-r--r-- 1 root root 0 Apr 15 17:49 WIN2016.err\n -rw-r--r-- 1 root root 9 Apr 15 17:49 WIN2016.out\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- Run commands in a script\n\n ```CMD or Bash\n clus run -script \u003cfile\u003e\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003enotepad script.txt\n\n c:\\\u003emore script.txt\n hostname\n echo hostname\n hostname\n\n c:\\\u003eclus run -script script.txt\n Job 10 started on 3 nodes in cluster localhost:50505.\n\n ----------------------------------Command-----------------------------------\n hostname\n echo hostname\n hostname\n ----------------------------------------------------------------------------\n\n [67A226CD8E76(VMSS-UBUN000000:60000)]: /clusnode.db/VMSS-UBUN000000.60000.command/WIN2016.50505.10.sh: line 1: $'hostname\\r': command not found\n [67A226CD8E76(VMSS-UBUN000000:60000)]: hostname\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 67a226cd8e76\n [1/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 28.9948ms.\n [2/3] Command finished on node VMSS-UBUN000000 in 31.0032ms.\n [3/3] Command finished on node WIN2016 in 81.9474ms.\n\n -------------------[67A226CD8E76(VMSS-UBUN000000:60000)]--------------------\n /clusnode.db/VMSS-UBUN000000.60000.command/WIN2016.50505.10.sh: line 1: $'hostname\\r': command not found\n hostname\n 67a226cd8e76\n\n -----------------------------[VMSS-UBUN000000]------------------------------\n /usr/local/clusrun/clusnode.db/VMSS-UBUN000000.50505.command/WIN2016.50505.10.sh: line 1: $'hostname\\r': command not found\n hostname\n vmss-ubun000000\n\n ---------------------------------[WIN2016]----------------------------------\n win2016\n hostname\n win2016\n\n ----------------------------------------------------------------------------\n Runtime:\n Min=28.9948ms\n Max=81.9474ms\n Mean=47.315133ms\n Mid=31.0032ms\n SD=24.502433ms\n\n 3 of 3 nodes succeeded.\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# vim script\n root@vmss-ubun000000:~# cat script\n hostname\n echo 1\n hostname\n root@vmss-ubun000000:~# clus run -script script\n Job 80 started on 3 nodes in cluster localhost:50505.\n\n ---------------------------------Command----------------------------------\n hostname\n echo 1\n hostname\n\n --------------------------------------------------------------------------\n\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 67a226cd8e76\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 1\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 67a226cd8e76\n [1/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 41.9381ms.\n [2/3] Command finished on node VMSS-UBUN000000 in 48.7202ms.\n [3/3] Command finished on node WIN2016 in 99.1658ms.\n\n ------------------[67A226CD8E76(VMSS-UBUN000000:60000)]-------------------\n 67a226cd8e76\n 1\n 67a226cd8e76\n\n ----------------------------[VMSS-UBUN000000]-----------------------------\n vmss-ubun000000\n 1\n vmss-ubun000000\n\n --------------------------------[WIN2016]---------------------------------\n win2016\n 1\n win2016\n\n --------------------------------------------------------------------------\n Runtime:\n Min=41.9381ms\n Max=99.1658ms\n Mean=63.2747ms\n Mid=48.7202ms\n SD=25.529427ms\n\n 3 of 3 nodes succeeded.\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- Run command with different content on each node\n\n ```CMD or Bash\n clus run -sweep \u003cplaceholder[{begin[-end][:step]}]\u003e\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus run -sweep ?? echo ??\n Job 12 started on 3 nodes in cluster localhost:50505.\n\n Sweep parameter: ??\n ----------------------------------Command-----------------------------------\n echo ??\n ----------------------------------------------------------------------------\n\n [1/3] Command finished on node VMSS-UBUN000000 in 53.9991ms.\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 0\n [2/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 80.0054ms.\n [3/3] Command finished on node WIN2016 in 81.0067ms.\n\n -------------------[67A226CD8E76(VMSS-UBUN000000:60000)]--------------------\n 0\n\n -----------------------------[VMSS-UBUN000000]------------------------------\n 1\n\n ---------------------------------[WIN2016]----------------------------------\n 2\n\n ----------------------------------------------------------------------------\n Runtime:\n Min=53.9991ms\n Max=81.0067ms\n Mean=71.6704ms\n Mid=80.0054ms\n SD=12.50218ms\n\n 3 of 3 nodes succeeded.\n\n c:\\\u003eclus run -sweep ??{10-9:-1} echo ??\n Job 13 started on 3 nodes in cluster localhost:50505.\n\n Sweep parameter: ??{10-9:-1}\n ----------------------------------Command-----------------------------------\n echo ??\n ----------------------------------------------------------------------------\n\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 10\n [1/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 24.0771ms.\n [2/3] Command finished on node VMSS-UBUN000000 in 25.9966ms.\n [3/3] Command finished on node WIN2016 in 51.1295ms.\n\n -------------------[67A226CD8E76(VMSS-UBUN000000:60000)]--------------------\n 10\n\n -----------------------------[VMSS-UBUN000000]------------------------------\n 9\n\n ---------------------------------[WIN2016]----------------------------------\n 10\n\n ----------------------------------------------------------------------------\n Runtime:\n Min=24.0771ms\n Max=51.1295ms\n Mean=33.7344ms\n Mid=25.9966ms\n SD=12.32513ms\n\n 3 of 3 nodes succeeded.\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus run -sweep ?? echo ??\n Job 82 started on 3 nodes in cluster localhost:50505.\n\n Sweep parameter: ??\n ---------------------------------Command----------------------------------\n echo ??\n --------------------------------------------------------------------------\n\n [1/3] Command finished on node VMSS-UBUN000000 in 47.2975ms.\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 0\n [2/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 47.6821ms.\n [3/3] Command finished on node WIN2016 in 91.2192ms.\n\n ------------------[67A226CD8E76(VMSS-UBUN000000:60000)]-------------------\n 0\n\n ----------------------------[VMSS-UBUN000000]-----------------------------\n 1\n\n --------------------------------[WIN2016]---------------------------------\n 2\n\n --------------------------------------------------------------------------\n Runtime:\n Min=47.2975ms\n Max=91.2192ms\n Mean=62.066266ms\n Mid=47.6821ms\n SD=20.614834ms\n\n 3 of 3 nodes succeeded.\n root@vmss-ubun000000:~# clus run -sweep ??{10-9:-1} echo ??\n Job 83 started on 3 nodes in cluster localhost:50505.\n\n Sweep parameter: ??{10-9:-1}\n ---------------------------------Command----------------------------------\n echo ??\n --------------------------------------------------------------------------\n\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 10\n [1/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 60.5268ms.\n [2/3] Command finished on node VMSS-UBUN000000 in 73.7394ms.\n [3/3] Command finished on node WIN2016 in 108.5368ms.\n\n ------------------[67A226CD8E76(VMSS-UBUN000000:60000)]-------------------\n 10\n\n ----------------------------[VMSS-UBUN000000]-----------------------------\n 9\n\n --------------------------------[WIN2016]---------------------------------\n 10\n\n --------------------------------------------------------------------------\n Runtime:\n Min=60.5268ms\n Max=108.5368ms\n Mean=80.934333ms\n Mid=73.7394ms\n SD=20.249532ms\n\n 3 of 3 nodes succeeded.\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n \n\n- Run command in specified cluster\n\n ```CMD or Bash\n clus run -headnode \u003cnode host\u003e\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus run -headnode VMSS-UBUN000000 hostname\n Job 81 started on 3 nodes in cluster VMSS-UBUN000000:50505.\n\n ----------------------------------Command-----------------------------------\n hostname\n ----------------------------------------------------------------------------\n\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 67a226cd8e76\n [1/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 51.5474ms.\n [2/3] Command finished on node VMSS-UBUN000000 in 54.548ms.\n [3/3] Command finished on node WIN2016 in 103.8009ms.\n\n -------------------[67A226CD8E76(VMSS-UBUN000000:60000)]--------------------\n 67a226cd8e76\n\n -----------------------------[VMSS-UBUN000000]------------------------------\n vmss-ubun000000\n\n ---------------------------------[WIN2016]----------------------------------\n win2016\n\n ----------------------------------------------------------------------------\n Runtime:\n Min=51.5474ms\n Max=103.8009ms\n Mean=69.965433ms\n Mid=54.548ms\n SD=23.956627ms\n\n 3 of 3 nodes succeeded.\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus run -headnode win2016 hostname\n Job 11 started on 3 nodes in cluster win2016:50505.\n\n ---------------------------------Command----------------------------------\n hostname\n --------------------------------------------------------------------------\n\n [67A226CD8E76(VMSS-UBUN000000:60000)]: 67a226cd8e76\n [1/3] Command finished on node VMSS-UBUN000000 in 52.6636ms.\n [2/3] Command finished on node 67A226CD8E76(VMSS-UBUN000000:60000) in 52.8099ms.\n [3/3] Command finished on node WIN2016 in 61.2312ms.\n\n ------------------[67A226CD8E76(VMSS-UBUN000000:60000)]-------------------\n 67a226cd8e76\n\n ----------------------------[VMSS-UBUN000000]-----------------------------\n vmss-ubun000000\n\n --------------------------------[WIN2016]---------------------------------\n win2016\n\n --------------------------------------------------------------------------\n Runtime:\n Min=52.6636ms\n Max=61.2312ms\n Mean=55.568233ms\n Mid=52.8099ms\n SD=4.004767ms\n\n 3 of 3 nodes succeeded.\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003eclus job\u003c/summary\u003e\n\n#### Manage jobs in cluster by command `clus job`\n\n- Show usage\n\n ```CMD or Bash\n clus job -h\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus job -h\n Usage of clus job options:\n -cancel\n cancel jobs\n -format string\n format the jobs in table or list\n -headnode string\n specify the headnode to connect (default \"localhost:50505\")\n -rerun\n rerun jobs\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus job -h\n Usage of clus job options:\n -cancel\n cancel jobs\n -format string\n format the jobs in table or list\n -headnode string\n specify the headnode to connect (default \"localhost:50505\")\n -rerun\n rerun jobs\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- List all jobs in the cluster\n\n ```CMD or Bash\n clus job\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus job\n Id State Nodes Command\n -- -------- ----- -------------------------------------\n 1 Finished 3 hostname\n 2 Finished 2 hostname\n 3 Finished 1 hostname\n 4 Finished 1 hostname\n 5 Finished 2 hostname\n 6 Finished 3 hostname\n 7 Finished 3 hostname\n 8 Finished 3 hostname\n 9 Finished 3 hostname\n 10 Finished 3 hostname\\r\\necho hostname\\r\\nhostname\n 11 Finished 3 hostname\n 12 Finished 3 echo ??\n 13 Finished 3 echo ??\n -------------------------------------------------------------\n Job count: 13\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus job\n Id State Nodes Command\n -- -------- ----- --------------------------------------------------\n 1 Finished 40 hostname\n 2 Finished 40 hostname\n 3 Finished 40 hostname\n 4 Canceled 40 ping localhost\n 5 Finished 40 snap install docker\n 6 Finished 40 docker version\n 7 Finished 30 hostname\n 8 Finished 30 hostname\n 9 Finished 1 echo 1\n 10 Finished 40 ping -c 4 localhost\n 11 Finished 40 hostname\n 12 Finished 40 ping -c 10 localhost\n 13 Finished 40 hostname\n 14 Finished 40 hostname\n 15 Finished 40 headnode=vmss-ubun000000\\ncd ~\\ncp /usr/local/c...\n 16 Failed 160 clus.run.20200414173133 s echo clus.run.2020041...\n 17 Finished 160 echo ??\n 18 Finished 160 echo ??\n 19 Finished 160 echo ??\n 20 Finished 160 echo ??\n 21 Finished 160 echo ??\n 22 Finished 160 hostname\n 23 Finished 160 echo x\n 24 Finished 160 echo x\n 25 Finished 160 hostname\n 26 Finished 160 hostname\n 27 Canceled 40 ping localhost\n 28 Failed 160 clusnode\n 29 Finished 1 pwd\n 30 Finished 1 ls\n 31 Finished 1 /clusnode\n 32 Failed 40 clusnode config add -headnodes win2016\n 33 Finished 160 hostname\n 34 Finished 160 hostname\n 35 Finished 160 hostname\n 36 Finished 160 hostname\n 37 Finished 160 echo ??\n 38 Failed 160 docker ps\n 39 Finished 40 docker ps\n 40 Finished 40 docker kill $(docker ps -q)\\nheadnode=vmss-ubun...\n 41 Finished 480 echo ??\n 42 Finished 480 echo ??\n 43 Failed 480 clusnode config get\n 44 Finished 480 echo ??\n 45 Finished 40 clusnode config get\n 46 Finished 40 clusnode config add -headnodes win2016\n 47 Finished 40 clusnode config get\n 48 Finished 40 clusnode config get\n 49 Finished 480 echo ??\n 50 Finished 480 echo ??\n 51 Finished 40 clusnode config get\n 52 Finished 40 clusnode config del -headnodes win2016\n 53 Finished 40 clusnode config get\n 54 Finished 40 clusnode config get\n 55 Finished 40 clusnode config add -headnodes win2016\n 56 Finished 40 clusnode config get\n 57 Finished 440 /clusnode config get\n 58 Failed 40 headnode=win2016\\nfor port in {60000..60010}\\nd...\n 59 Finished 40 headnode=win2016\\nfor port in {60000..60010}\\nd...\n 60 Finished 40 for port in {60000..60010}\\ndo\\n clusnode confi...\n 61 Finished 469 hostname\n 62 Finished 40 headnode=vmss-ubun000000\\ncd ~\\ncp /usr/local/c...\n 63 Finished 908 hostname\n 64 Finished 908 hostname\n 65 Finished 908 hostname\n 66 Finished 908 hostname\n 67 Finished 40 docker\n 68 Finished 40 echo\n 69 Finished 40 echo $(docker ps -q)\n 70 Finished 40 docker kill $(docker ps -q)\n 71 Finished 40 clusnode config set -headnodes VMSS-UBUN000001\n 72 Finished 3 hostname\n 73 Finished 2 hostname\n 74 Finished 2 hostname\n 75 Finished 2 hostname\n 76 Finished 3 hostname\n 77 Finished 3 hostname\n 78 Finished 3 hostname\n 79 Finished 3 hostname\n 80 Finished 3 hostname\\necho 1\\nhostname\\n\n 81 Finished 3 hostname\n 82 Finished 3 echo ??\n 83 Finished 3 echo ??\n --------------------------------------------------------------------------\n Job count: 83\n root@vmss-ubun000000:~#\n ```\n \u003c/details\u003e\n\n- List specified jobs\n\n ```CMD or Bash\n clus job \u003c[~]id[-id][,[~]id[-id]]\u003e [[~]id[-id][,[~]id[-id]]]\n ```\n\n \u003cdetails\u003e\u003csummary\u003eExample on Windows\u003c/summary\u003e\n\n ```CMD\n c:\\\u003eclus job 1,3-5 7 ~1-2\n Id : 1\n State : Finished\n Nodes : 67A226CD8E76(VMSS-UBUN000000:60000), VMSS-UBUN000000, WIN2016\n Create Time : 2020-04-15 17:41:41 +0000 GMT\n End Time : 2020-04-15 17:41:41 +0000 GMT\n Command : hostname\n ----------------------------------------------------------------------------\n Id : 3\n State : Finished\n Nodes : WIN2016\n Create Time : 2020-04-15 17:44:13 +0000 GMT\n End Time : 2020-04-15 17:44:13 +0000 GMT\n Command : hostname\n ----------------------------------------------------------------------------\n Id : 4\n State : Finished\n Nodes : WIN2016\n Create Time : 2020-04-15 17:44:45 +0000 GMT\n End Time : 2020-04-15 17:44:45 +0000 GMT\n Command : hostname\n ----------------------------------------------------------------------------\n Id : 5\n State : Finished\n Nodes : VMSS-UBUN000000, WIN2016\n Create Time : 2020-04-15 17:46:26 +0000 GMT\n End Time : 2020-04-15 17:46:26 +0000 GMT\n Command : hostname\n ----------------------------------------------------------------------------\n Id : 7\n State : Finished\n Nodes : 67A226CD8E76(VMSS-UBUN000000:60000), VMSS-UBUN000000, WIN2016\n Create Time : 2020-04-15 17:47:50 +0000 GMT\n End Time : 2020-04-15 17:47:51 +0000 GMT\n Command : hostname\n ----------------------------------------------------------------------------\n Id : 12\n State : Finished\n Nodes : 67A226CD8E76(VMSS-UBUN000000:60000), VMSS-UBUN000000, WIN2016\n Create Time : 2020-04-15 17:57:39 +0000 GMT\n End Time : 2020-04-15 17:57:39 +0000 GMT\n Sweep Parameter : ??\n Command : echo ??\n ----------------------------------------------------------------------------\n Id : 13\n State : Finished\n Nodes : 67A226CD8E76(VMSS-UBUN000000:60000), VMSS-UBUN000000, WIN2016\n Create Time : 2020-04-15 17:57:44 +0000 GMT\n End Time : 2020-04-15 17:57:44 +0000 GMT\n Sweep Parameter : ??{10-9:-1}\n Command : echo ??\n ----------------------------------------------------------------------------\n Job count: 7\n\n c:\\\u003e\n ```\n \u003c/details\u003e\n\n \u003cdetails\u003e\u003csummary\u003eExample on Linux\u003c/summary\u003e\n\n ```Bash\n root@vmss-ubun000000:~# clus job 1,3-5 7 ~1-2\n Id : 1\n State : Finished\n Nodes : VMSS-UBUN000000, VMSS-UBUN000001, VMSS-UBUN000002, VMSS-UBUN000003, VMSS-UBUN000004, VMSS-UBUN000005, VMSS-UBUN000006, VMSS-UBUN000007, VMSS-UBUN000008, VMSS-UBUN000009, VMSS-UBUN00000C, VMSS-UBUN00000D, VMSS-UBUN00000E, VMSS-UBUN00000F, VMSS-UBUN00000G, VMSS-UBUN00000H, VMSS-UBUN00000J, VMSS-UBUN00000M, VMSS-UBUN00000N, VMSS-UBUN00000O, VMSS-UBUN00000P, VMSS-UBUN00000Q, VMSS-UBUN00000S, VMSS-UBUN00000T, VMSS-UBUN00000U, VMSS-UBUN00000V, VMSS-UBUN00000W, VMSS-UBUN00000X, VMSS-UBUN00000Y, VMSS-UBUN00000Z, VMSS-UBUN000010, VMSS-UBUN000011, VMSS-UBUN000012, VMSS-UBUN000013, VMSS-UBUN000014, VMSS-UBUN000015, VMSS-UBUN000016, VMSS-UBUN000019, VMSS-UBUN00001A, VMSS-UBUN00001B\n Create Time : 2020-04-14 17:03:33 +0000 UTC\n End Time : 2020-04-14 17:03:33 +0000 UTC\n Command : hostname\n --------------------------------------------------------------------------\n Id : 3\n State : Finished\n Nodes : VMSS-UBUN000000, VMSS-UBUN000001, VMSS-UBUN000002, VMSS-UBUN000003, VMSS-UBUN000004, VMSS-UBUN000005, VMSS-UBUN000006, VMSS-UBUN000007, VMSS-UBUN000008, VMSS-UBUN000009, VMSS-UBUN00000C, VMSS-UBUN00000D, VMSS-UBUN00000E, VMSS-UBUN00000F, VMSS-UBUN00000G, VMSS-UBUN00000H, VMSS-UBUN00000J, VMSS-UBUN00000M, VMSS-UBUN00000N, VMSS-UBUN00000O, VMSS-UBUN00000P, VMSS-UBUN00000Q, VMSS-UBUN00000S, VMSS-UBUN00000T, VMSS-UBUN00000U, VMSS-UBUN00000V, VMSS-UBUN00000W, VMSS-UBUN00000X, VMSS-UBUN00000Y, VMSS-UBUN00000Z, VMSS-UBUN000010, VMSS-UBUN000011, VMSS-UBUN000012, VMSS-UBUN000013, VMSS-UBUN000014, VMSS-UBUN000015, VMSS-UBUN000016, VMSS-UBUN000019, VMSS-UBUN00001A, VMSS-UBUN00001B\n Create Time : 2020-04-14 17:19:24 +0000 UTC\n End Time : 2020-04-14 17:19:24 +0000 UTC\n Command : hostname\n --------------------------------------------------------------------------\n Id : 4\n State : Canceled\n Nodes : VMSS-UBUN000000, VMSS-UBUN000001, VMSS-UBUN000002, VMSS-UBUN000003, VMSS-UBUN000004, VMSS-UBUN000005, VMSS-UBUN000006, VMSS-UBUN000007, VMSS-UBUN000008, VMSS-UBUN000009, VMSS-UBUN00000C, VMSS-UBUN00000D, VMSS-UBUN00000E, VMSS-UBUN00000F, VMSS-UBUN00000G, VMSS-UBUN00000H, VMSS-UBUN00000J, VMSS-UBUN00000M, VMSS-UBUN00000N, VMSS-UBUN00000O, VMSS-UBUN00000P, VMSS-UBUN00000Q, VMSS-UBUN00000S, VMSS-UBUN00000T, VMSS-UBUN00000U, VMSS-UBUN00000V, VMSS-UBUN00000W, VMSS-UBUN00000X, VMSS-UBUN00000Y, VMSS-UBUN00000Z, VMSS-UBUN000010, VMSS-UBUN000011, VMSS-UBUN000012, VMSS-UBUN000013, VMSS-UBUN000014, VMSS-UBUN000015, VMSS-UBUN000016, VMSS-UBUN000019, VMSS-UBUN00001A, VMSS-UBUN00001B\n Create Time : 2020-04-14 17:21:25 +0000 UTC\n End Time : 2020-04-14 17:21:55 +0000 UTC\n Failed Nodes : VMSS-UBUN000000 -\u003e -1, VMSS-UBUN000001 -\u003e -1, VMSS-UBUN000002 -\u003e -1, VMSS-UBUN000003 -\u003e -1, VMSS-UBUN000004 -\u003e -1, VMSS-UBUN000005 -\u003e -1, VMSS-UBUN000006 -\u003e -1, VMSS-UBUN000007 -\u003e -1, VMSS-UBUN000008 -\u003e -1, VMSS-UBUN000009 -\u003e -1, VMSS-UBUN00000C -\u003e -1, VMSS-UBUN00000D -\u003e -1, VMSS-UBUN00000E -\u003e -1, VMSS-UBUN00000F -\u003e -1, VMSS-UBUN00000G -\u003e -1, VMSS-UBUN00000H -\u003e -1, VMSS-UBUN00000J -\u003e -1, VMSS-UBUN00000M -\u003e -1, VMSS-UBUN00000N -\u003e -1, VMSS-UBUN00000O -\u003e -1, VMSS-UBUN00000P -\u003e -1, VMSS-UBUN00000Q -\u003e -1, VMSS-UBUN00000S -\u003e -1, VMSS-UBUN00000T -\u003e -1, VMSS-UBUN00000U -\u003e -1, VMSS-UBUN00000V -\u003e -1, VMSS-UBUN00000W -\u003e -1, VMSS-UBUN00000X -\u003e -1, VMSS-UBUN00000Y -\u003e -1, VMSS-UBUN00000Z -\u003e -1, VMSS-UBUN000010 -\u003e -1, VMSS-UBUN000011 -\u003e -1, VMSS-UBUN000012 -\u003e -1, VMSS-UBUN000013 -\u003e -1, VMSS-UBUN000014 -\u003e -1, VMSS-UBUN000015 -\u003e -1, VMSS-UBUN000016 -\u003e -1, VMSS-UBUN000019 -\u003e -1, VMSS-UBUN00001A -\u003e -1, VMSS-UBUN00001B -\u003e -1\n Command : ping localhost\n --------------------------------------------------------------------------\n Id : 5\n State : Finished\n Nodes : VMSS-UBUN000000, VMSS-UBUN000001, VMSS-UBUN000002, VMSS-UBUN000003, VMSS-UBUN000004, VMSS-UBUN000005, VMSS-UBUN000006, VMSS-UBUN000007, VMSS-UBUN000008, VMSS-UBUN000009, VMSS-UBUN00000C, VMSS-UBUN00000D, VMSS-UBUN00000E, VMSS-UBUN00000F, VMSS-UBUN00000G, VMSS-UBUN00000H, VMSS-UBUN00000J, VMSS-UBUN00000M, VMSS-UBUN00000N, VMSS-UBUN00000O, VMSS-UBUN00000P, VMSS-UBUN00000Q, VMSS-UBUN00000S, VMSS-UBUN00000T, VMSS-UBUN00000U, VMSS-UBUN00000V, VMSS-UBUN00000W, VMSS-UBUN00000X, VMSS-UBUN00000Y, VMSS-UBUN00000Z, VMSS-UBUN000010, VMSS-UBUN000011, VMSS-UBUN000012, VMSS-UBUN000013, VMSS-UBUN000014, VMSS-UBUN000015, VMSS-UBUN000016, VMSS-UBUN000019, VMSS-UBUN00001A, VMSS-UBUN00001B\n Create Time : 2020-04-14 17:23:11 +0000 UTC\n End Time : 2020-04-14 17:23:49 +0000 UTC\n Command : snap install docker\n --------------------------------------------------------------------------\n Id : 7\n State : Finished\n Nodes : VMSS-UBUN000000, VMSS-UBUN000001, VMSS-UBUN000002, VMSS-UBUN000003, VMSS-UBUN000004, VMSS-UBUN000005, VMSS-UBUN000006, ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchezhang%2Fclusrun","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchezhang%2Fclusrun","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchezhang%2Fclusrun/lists"}