{"id":13530288,"url":"https://github.com/Fmstrat/cloudstomp","last_synced_at":"2025-04-01T18:31:31.393Z","repository":{"id":76612532,"uuid":"166443160","full_name":"Fmstrat/cloudstomp","owner":"Fmstrat","description":"Automated deployment of instances on EC2 via plugin for high CPU/GPU applications at the lowest price.","archived":false,"fork":false,"pushed_at":"2019-01-18T17:10:51.000Z","size":13,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-27T19:59:47.959Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Fmstrat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-01-18T16:56:55.000Z","updated_at":"2024-11-21T07:08:40.000Z","dependencies_parsed_at":null,"dependency_job_id":"ff4a5fe3-74e7-4d7b-ae8e-2ab54029e747","html_url":"https://github.com/Fmstrat/cloudstomp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fmstrat%2Fcloudstomp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fmstrat%2Fcloudstomp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fmstrat%2Fcloudstomp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fmstrat%2Fcloudstomp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Fmstrat","download_url":"https://codeload.github.com/Fmstrat/cloudstomp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246691560,"owners_count":20818533,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T07:00:47.419Z","updated_at":"2025-04-01T18:31:31.163Z","avatar_url":"https://github.com/Fmstrat.png","language":"Python","funding_links":[],"categories":["Cloud"],"sub_categories":[],"readme":"# Cloudstomp - Automated deployment of instances on EC2 via plugin for high CPU/GPU applications at the lowest price\n\nCloudstomp is a python application *that is in active development* and built to make deployment of high CPU/GPU applications in Amazon EC2 as spot instances for affordable, quick, and easy data analysis.\n\nPlugins can be created in the `json` format that enable users to quickly deploy applications. Two plugins are included for demonstration, one for a `hello world` application, and one for `hashcat`.\n\n## Install\n\nInstallation can be completed with Docker or natively. For a native local installation, start by downloading cloudstomp:\n```\n# cd /usr/bin\n# wget https://raw.githubusercontent.com/Fmstrat/cloudstomp/master/cloudstomp.py\n# chmod +x cloudstomp.py\n```\nor, if you would like to use Docker:\n```\n# alias cloudstomp.py='docker run -it --rm --name=cloudstomp -v ${HOME}/.cloudstomp:/data cloudstomp /cloudstomp.py'\n```\n## Plugins\nThe current documentation for plugins and avaialable plugins can be found at https://github.com/Fmstrat/cloudstomp-plugins and are:\n* Hello World - The example shown below\n* Hashcat - Password cracking in the cloud\n\n## AWS Credentials\nTo use `Cloudstomp` you must provide AWS credentials. To set up your credentials at the AWS Management Console:\n* Sign in at https://console.aws.amazon.com/iam/\n* Go to `Users`\n* Then `Add User`\n* Add the user `cloudstomp`\n* Check the box for `Programmatic access`\n* Click `Next: Permissions`\n* If the `Administrators` group doesn't exist, click `Add group`\n* Enter `Administrators` for `Group name`\n* Check `Administrator Access`\n* Click `Create group` which will return you to the add user screen                                                                                                                            \n* Back on the add user screen, choose `Add user to group` and check `Administrators`\n* Click `Next: Tags`\n* Click `Next: Review`\n* Click `Create user`\n* `Access key ID` and `Secret access key` are displayed which are required for configuration of `Cloudstomp`\n\n## Usage\n\nFirst, we need to download the latest plugins:\n```\n# cloudstomp.py --update\n```\nThis will download all of the preconfigured plugins into the ~/cloudstomp folder (or the folder specified in the Docker command).\n\nThen, we fire up cloudstomp:\n```\n# cloudstomp.py\n```\nThe first time `Cloudstomp` runs, it requires the AWS credentials. This sets up a credentials file completely seperate from those used in your normal AWS administration to reduce conflicts.\n\u003cpre\u003e\nAWS CLI is not configured, would you like to do this now? [y/N]: \u003ci\u003e\u003cb\u003ey\u003c/b\u003e\u003c/i\u003e\nAccess key, Secret key, and default region must all be configured.\nAWS Access Key ID [None]: \u003ci\u003e\u003cb\u003e*********\u003c/b\u003e\u003c/i\u003e\nAWS Secret Access Key [None]: \u003ci\u003e\u003cb\u003e*********\u003c/b\u003e\u003c/i\u003e\nDefault region name [None]: \u003ci\u003e\u003cb\u003eus-east-1\u003c/b\u003e\u003c/i\u003e\nDefault output format [None]: \n\u003c/pre\u003e\n\nThen, follow the prompts:\n\u003cpre\u003e\n\nSessions:\n-------------------------------------------\n c) Create\n q) Quit\n-------------------------------------------\nSelect session: \u003ci\u003e\u003cb\u003ec\u003c/b\u003e\u003c/i\u003e\n\nCreating a new session.\nPlugins:\n-------------------------------------------\n 1) Hashcat\n 2) Hello World\n-------------------------------------------\nPlease select a plugin: \u003ci\u003e\u003cb\u003e2\u003c/b\u003e\u003c/i\u003e\n\n\u003c/pre\u003e\nThe first time you choose a plugin, the AMI that instances will be created from needs to be built. This is a one-time item and the AMI is stored on S3 as they always are for faster future deployment.\n\u003cpre\u003e\n\nThe AMI for hello world needs to be built.\nWould you like to build it now? [Y/n]: \u003ci\u003e\u003cb\u003ey\u003c/b\u003e\u003c/i\u003e\n\n[2019-01-18 10:57:47] Build will take place on a t2.nano instance.\n[2019-01-18 10:57:47] Checking availability zones for t2.nano...\n[2019-01-18 10:57:48] Getting subnet...\n[2019-01-18 10:57:49]   Using subnet subnet-0bea6a005f7f8dbed.\n[2019-01-18 10:57:49] Checking VPC...\n[2019-01-18 10:57:50]   Using VPC vpc-02d8172ca01a496de.\n[2019-01-18 10:57:50] Checking security group...\n[2019-01-18 10:57:50]   Creating new security group.\n[2019-01-18 10:57:51]   Adding rules...\n[2019-01-18 10:57:52]     Added port 22.\n[2019-01-18 10:57:52]   Using security group sg-019fedf5084f3e068.\n[2019-01-18 10:57:54] Instance i-06f82a99547d8d66e started.\n[2019-01-18 10:57:54] Waiting for boot to complete...\n[2019-01-18 10:57:54]   Instance in pending state, sleeping 15s...\n[2019-01-18 10:58:10] Instance has public IP of 18.212.124.241.\n[2019-01-18 10:58:10] Checking for SSH connectivity...\n[2019-01-18 10:58:13]   Waiting for SSH to start, sleeping 15s...\n[2019-01-18 10:58:28]   Waiting for SSH to start, sleeping 15s...\n[2019-01-18 10:58:46] Starting build scripts...\n[2019-01-18 10:58:46]   Running build.sh...\n[2019-01-18 10:58:48]     Uploaded.\nGet:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]\nHit:2 http://archive.ubuntu.com/ubuntu bionic InRelease\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]\n\n... [ Output of setting up the instance (apt-get/etc) ] ...\n\nSuccessfully built PyYAML\nInstalling collected packages: docutils, jmespath, six, python-dateutil, urllib3, botocore, PyYAML, pyasn1, rsa, colorama, futures, s3transfer, awscli\nSuccessfully installed PyYAML-3.13 awscli-1.16.91 botocore-1.12.81 colorama-0.3.9 docutils-0.14 futures-3.2.0 jmespath-0.9.3 pyasn1-0.4.5 python-dateutil-2.7.5 rsa-3.4.2 s3transfer-0.1.13 six-1.12.0 urllib3-1.24.1\n[2019-01-18 10:59:59] Stopping instance...\n[2019-01-18 11:00:00]   Instance in stopping state, sleeping 15s...\n[2019-01-18 11:00:16]   Instance in stopping state, sleeping 15s...\n[2019-01-18 11:00:37] Creating AMI image...\n[2019-01-18 11:00:38]   Build is in pending state, sleeping 15s...\n[2019-01-18 11:00:54]   Build is in pending state, sleeping 15s...\n[2019-01-18 11:01:09]   Build is in pending state, sleeping 15s...\n[2019-01-18 11:01:25]   Build is in pending state, sleeping 15s...\n[2019-01-18 11:01:41]   Build is in pending state, sleeping 15s...\n[2019-01-18 11:01:57]   Build is in pending state, sleeping 15s...\n[2019-01-18 11:02:13]   Build is in pending state, sleeping 15s...\n[2019-01-18 11:02:33] Tagging with build version...\n[2019-01-18 11:02:34] Terminating source image...\n[2019-01-18 11:02:35] Build complete.\n\nReturning to session creation.\n\n\u003c/pre\u003e\nOnce the build completes, we return to creating the new session. You will notice one of the questions below is about encryption. Output data is stored on S3 for later download when processes finish. This data can be encrypted with gocryptfs, that way if it is stored for a long period of time it is more secure. For the `Hello World` plugin, data is syncronized to S3 every 15 seconds. For other plugins, the reocmmendation is every `300` seconds.\n\u003cpre\u003e\n\nWould you like to run the instance On-Demand or as a Spot Instance? [o/S]: \u003ci\u003e\u003cb\u003es\u003c/b\u003e\u003c/i\u003e\nInstances:\n-------------------------------------------\n 1) 1 vCPU, 1GB RAM     [t2.micro]\n 2) 1 vCPU, 2GB RAM     [t2.small]\n-------------------------------------------\nPlease select an instance: \u003ci\u003e\u003cb\u003e1\u003c/b\u003e\u003c/i\u003e\n\nPricing:\n--------------  ----------      ----------      ----------      ------------\nZone            Average         Latest          On Demand       Reccomended\n--------------  ----------      ----------      ----------      ------------\n1) us-east-1c   $0.003500       $0.003500       $0.011600       $0.004025 (best)\n2) us-east-1f   $0.003500       $0.003500       $0.011600       $0.004025\n3) us-east-1b   $0.003500       $0.003500       $0.011600       $0.004025\n4) us-east-1a   $0.003500       $0.003500       $0.011600       $0.004025\n5) us-east-1e   $0.003500       $0.003500       $0.011600       $0.004025\n6) us-east-1d   $0.003500       $0.003500       $0.011600       $0.004025\n--------------  ----------      ----------      ----------      ------------\n(The default is 15% over the average past two week spot price to keep your instance active.)\n\nWhich zone would you like to run in? \u003ci\u003e\u003cb\u003e1\u003c/b\u003e\u003c/i\u003e\nWhat is the max price you would pay for a spot instance? [0.004025]: \nWhat would you like to call this instance? \u003ci\u003e\u003cb\u003eTesting\u003c/b\u003e\u003c/i\u003e\nHow many seconds would you like to wait between printing 'hello world'? [10]: \u003ci\u003e\u003cb\u003e10\u003c/b\u003e\u003c/i\u003e\nEnter a password for gocryptfs encrypted output (leave blank for no encryption): \nSession Testing created.\n\nSessions:\n-------------------------------------------\n 1) Testing\n c) Create\n q) Quit\n-------------------------------------------\nSelect session: \n\u003c/pre\u003e\nNow you have the session `Testing` available. Next, we will start up an instance running the `Hello World` plugin.\n\u003cpre\u003e\nSessions:\n-------------------------------------------\n 1) Testing\n c) Create\n q) Quit\n-------------------------------------------\nSelect session: \u003ci\u003e\u003cb\u003e1\u003c/b\u003e\u003c/i\u003e\n\nTesting:\n-------------------------------------------\n 1) Start\n 2) Output\n s) Show variables\n d) Delete\n b) Back\n-------------------------------------------\nChoose an option: \u003ci\u003e\u003cb\u003e1\u003c/b\u003e\u003c/i\u003e\n\n[2019-01-18 11:06:27] Getting subnet...\n[2019-01-18 11:06:28]   Using subnet subnet-0bea6a005f7f8dbed.\n[2019-01-18 11:06:28] Checking VPC...\n[2019-01-18 11:06:29]   Using VPC vpc-02d8172ca01a496de.\n[2019-01-18 11:06:29] Checking security group...\n[2019-01-18 11:06:30]   Found existing security group.\n[2019-01-18 11:06:30]   Using security group sg-019fedf5084f3e068.\n[2019-01-18 11:06:31] Waiting on spot id sir-9jcg74mg...\n[2019-01-18 11:06:31]   Spot request in open state, sleeping 15s...\n[2019-01-18 11:07:03]   Instance in running state.\n[2019-01-18 11:07:04] Instance has public IP of 54.88.86.34.\n[2019-01-18 11:07:04] Checking for SSH connectivity...\n[2019-01-18 11:07:07]   Waiting for SSH to start, sleeping 15s...\n[2019-01-18 11:07:25] Uploading cloudstomp AWS credentials...\n[2019-01-18 11:07:26] Uploading any supporting files...\n[2019-01-18 11:07:26]   Complete.\n[2019-01-18 11:07:26] Starting init script...\n[2019-01-18 11:07:26]   Running remote.sh...\n[2019-01-18 11:07:27]     Uploaded.\n[2019-01-18 11:07:33] Launch complete.\n\nTesting:\n-------------------------------------------\n 1) Status\n 2) Connect\n 3) Stop\n 4) Output\n s) Show variables\n d) Delete\n b) Back\n-------------------------------------------\nChoose an option: \n\u003c/pre\u003e\nNow we can check on the status of our instance, connect to it and attach to the `screen` session, stop the instance, or show the outputs from S3. Outputs are available even after the instance is stopped.\n\u003cpre\u003e\nTesting:\n-------------------------------------------\n 1) Status\n 2) Connect\n 3) Stop\n 4) Output\n s) Show variables\n d) Delete\n b) Back\n-------------------------------------------\nChoose an option: \u003ci\u003e\u003cb\u003e1\u003c/b\u003e\u003c/i\u003e\n[2019-01-18 11:09:59] Instance has public IP of 54.88.86.34.\n[2019-01-18 11:09:59] Checking for SSH connectivity...\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nTesting:\n-------------------------------------------\n 1) Status\n 2) Connect\n 3) Stop\n 4) Output\n s) Show variables\n d) Delete\n b) Back\n-------------------------------------------\nChoose an option: \u003ci\u003e\u003cb\u003e3\u003c/b\u003e\u003c/i\u003e\n[2019-01-18 11:10:28] Instance has public IP of 54.88.86.34.\n[2019-01-18 11:10:28] Checking for SSH connectivity...\nThere is no screen to be resumed matching helloworld.\nThere is no screen to be resumed matching sync.\nupload: cloudstomp/plain/output.txt to s3://cloudstomp-testing/output.txt\nupload: cloudstomp/plain/sync.txt to s3://cloudstomp-testing/sync.txt\nConnection to 54.88.86.34 closed.\n\nSessions:\n-------------------------------------------\n 1) Testing     [shutting-down]\n c) Create\n q) Quit\n-------------------------------------------\nSelect session: \u003ci\u003e\u003cb\u003e1\u003c/b\u003e\u003c/i\u003e\n\nTesting:\n-------------------------------------------\n 1) Output\n s) Show variables\n d) Delete\n b) Back\n-------------------------------------------\nChoose an option: \u003ci\u003e\u003cb\u003e1\u003c/b\u003e\u003c/i\u003e\n\ndownload: s3://cloudstomp-testing/output.txt to ../../../tmp/cloudstomp-testing/plain/output.txt\ndownload: s3://cloudstomp-testing/sync.txt to ../../../tmp/cloudstomp-testing/plain/sync.txt\nStarting hello world...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\nHello world!\nSleeping 10 seconds...\n^C\n-------------------------------------------\n\nTesting:\n-------------------------------------------\n 1) Output\n s) Show variables\n d) Delete\n b) Back\n-------------------------------------------\nChoose an option: \n\u003c/pre\u003e\n\nMany of these options can also be executed from the command line (check with `cloudstomp.py --help`) for automation.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFmstrat%2Fcloudstomp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FFmstrat%2Fcloudstomp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFmstrat%2Fcloudstomp/lists"}