Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://andrewchilds.github.io/overcast/

Orchestrate VMs from your terminal
https://andrewchilds.github.io/overcast/

digitalocean orchestration virtualbox vm

Last synced: about 2 months ago
JSON representation

Orchestrate VMs from your terminal

Lists

README

        

# ![Overcast Logo](http://i.imgur.com/eCBl2NI.png)

Overcast is a tiny command line program designed to make it easy to spin up, configure and manage clusters of machines across multiple cloud providers.

## Devops made simple.

Create, reboot and destroy instances on DigitalOcean or locally using Virtualbox.

```sh
# Spin up a new Ubuntu 20.04 instance on DigitalOcean:
$ overcast digitalocean create db-01

# Perform a full system upgrade and then install Redis:
$ overcast run db-01 install/core install/redis
```

Run multiple commands or multiple scripts on any of your instances over SSH. Commands can be run sequentially or in parallel. Run and test your deployment scripts across different providers.

```sh
# Create a LAMP stack using bundled install scripts:
$ overcast run lamp-01 install/core install/apache install/mysql install/php

# Run your own scripts using relative or absolute paths:
$ overcast run app-cluster ./my-app/my-install-script /path/to/another-script

# Run sequences of commands and scripts across multiple machines in parallel:
$ overcast run app-* ./script.sh uptime "free -m" "df -h" --parallel
```

Quickly SSH in to any instance by name.

```sh
$ overcast ssh app-01
```

Push and pull files between your local machine and any of your instances.

```sh
$ overcast push app-01 nginx/myapp.conf /etc/nginx/sites-enabled/myapp.conf
$ overcast pull app-01 /var/log/syslog ./my-local-syslog-copy
```

Overcast is a thin layer on top of your SSH client. It doesn't install or leave anything on the servers you communicate with, so Overcast itself has no real attack surface.

A library of [scripts](https://github.com/andrewchilds/overcast/tree/master/scripts) and [recipes](https://github.com/andrewchilds/overcast/tree/master/recipes) are bundled to make it easy to deploy a number of common software stacks and applications.

## Installation (OS X/Linux)

1. Install [Node.js](https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager) if not already installed.

2. Install Overcast using npm.

```sh
$ npm -g install overcast
```

3. You can now use Overcast from any directory. Running any overcast command from anywhere will create the `~/.overcast` config directory if it doesn't already exist. Add your API keys to `~/.overcast/variables.json` to use their respective commands, either manually or using the `var` command:

```sh
$ overcast var set DIGITALOCEAN_API_TOKEN abc123
```

4. To make working with Overcast easier, you can add bash tab completion by adding the following to your `.bash_profile`:

```sh
# Overcast Tab completion
_overcast_completions() {
local cur=${COMP_WORDS[COMP_CWORD]}
COMPREPLY=($(compgen -W "`overcast completions`" -- "$cur"))
return 0
}
complete -F _overcast_completions overcast
```

## Installation (Windows)

Using Overcast on Windows is possible, but **unsupported**.

## Uninstallation

Since Overcast is just a wrapper around SSH, there is nothing on your remote machines to uninstall. To uninstall Overcast from your local machine:

```sh
# To remove the Overcast package:
$ npm -g remove overcast
# Optionally delete your Overcast SSH keys and configuration files:
$ rm -rf ~/.overcast
```

## Configuration

Overcast looks for an `.overcast` directory in the current directory, a parent directory, or `~/.overcast`, in that order. This means you can have multiple configurations and treat your server infrastructure like source code.

The command `overcast init` will create a new configuration in the current directory. The config directory looks like this:

```sh
/.overcast
/keys # SSH keys, can be your own or auto-generated by overcast
overcast.key
overcast.key.pub
clusters.json # Cluster/instance definitions (see example.clusters.json)
variables.json # API keys, etc (see example.variables.json)
```

## Design Goals

There are many server management frameworks out there already, but they generally involve a complex server-client implementation, a steep learning curve, or a giant, monolithic conceptual framework that requires taking a course to understand.

I wanted something that had little to no learning curve, that did only what you asked it to do on the remote machines and nothing more, that just focused on multi-server provisioning and communication and leaves problems like process/state management and system monitoring to tools designed specifically for those problems.

## Example App Recipes

- [Discourse](https://github.com/andrewchilds/overcast/blob/master/recipes/discourse)
- [Ubuntu 14.04 LAMP server](https://github.com/andrewchilds/overcast/blob/master/recipes/lamp-server)
- [Overcast Charts](https://github.com/andrewchilds/overcast-charts)

## Command Reference

### overcast aliases

```
Usage:
overcast aliases

Description:
Return a list of bash aliases for SSHing to your instances.

To use, add this to your .bash_profile:
test -f $HOME/.overcast_aliases && source $HOME/.overcast_aliases

And then create the .overcast_aliases file:
overcast aliases > $HOME/.overcast_aliases

Or to automatically refresh aliases in every new terminal window
(which will add a couple hundred milliseconds to your startup time),
add this to your .bash_profile:
overcast aliases > $HOME/.overcast_aliases
source $HOME/.overcast_aliases
```

### overcast cluster count

```
Usage:
overcast cluster count [name]

Description:
Return the number of instances in a cluster.

Examples:
$ overcast cluster count db
> 0
$ overcast instance create db.01 --cluster db
> ...
$ overcast cluster count db
> 1
```

### overcast cluster add

```
Usage:
overcast cluster add [name]

Description:
Adds a new cluster.

Examples:
$ overcast cluster add db
```

### overcast cluster rename

```
Usage:
overcast cluster rename [name] [new-name]

Description:
Renames a cluster.

Examples:
$ overcast cluster rename app-cluster app-cluster-renamed
```

### overcast cluster remove

```
Usage:
overcast cluster remove [name]

Description:
Removes a cluster from the index. If the cluster has any instances
attached to it, they will be moved to an "orphaned" cluster.

Examples:
$ overcast cluster remove db
```

### overcast completions

```
Usage:
overcast completions

Description:
Return an array of commands, cluster names, and instance names for use
in bash tab completion.

To enable tab completion in bash, add this to your .bash_profile:

_overcast_completions() {
local cur=${COMP_WORDS[COMP_CWORD]}
COMPREPLY=($(compgen -W "`overcast completions`" -- "$cur"))
return 0
}
complete -F _overcast_completions overcast
```

### overcast digitalocean boot

```
Usage:
overcast digitalocean boot [name]

Description:
Boot up an instance if powered off, otherwise do nothing.
```

### overcast digitalocean create

```
Usage:
overcast digitalocean create [name] [options...]

Description:
Creates a new instance on DigitalOcean.

Options: Defaults:
--cluster CLUSTER default
--ssh-port PORT 22
--ssh-key PATH overcast.key
--ssh-pub-key PATH overcast.key.pub
--region REGION
--image IMAGE
--size SIZE
--backups false
--monitoring false
--private-networking false
--vpc-uuid
--with-droplet-agent false

Examples:
# Match using slugs:
$ overcast digitalocean create vm-01 --size 2gb --region sfo1

# Match using IDs or names:
$ overcast digitalocean create vm-02 --region "London 1" --image 6374128
```

### overcast digitalocean destroy

```
Usage:
overcast digitalocean destroy [name] [options...]

Description:
Destroys a DigitalOcean droplet and removes it from your account.
Using --force overrides the confirm dialog.

Options: Defaults:
--force false

Examples:
$ overcast digitalocean destroy vm-01
```

### overcast digitalocean images

```
Usage:
overcast digitalocean images

Description:
List all images, including snapshots.
```

### overcast digitalocean instances

```
Usage:
overcast digitalocean instances

Description:
List all instances in your account.
```

### overcast digitalocean reboot

```
Usage:
overcast digitalocean reboot [name]

Description:
Reboot an instance using the provider API.
```

### overcast digitalocean regions

```
Usage:
overcast digitalocean regions

Description:
List all available regions.
```

### overcast digitalocean rebuild

```
Usage:
overcast digitalocean rebuild [name] [image]

Description:
Rebuilds an existing instance on DigitalOcean, preserving the IP address.
[image] can be image ID, name or slug.

Examples:
# Rebuild an instance using a readymade image:
$ overcast digitalocean rebuild vm-01 ubuntu-14-04-x64

# Rebuild an instance using a snapshot:
$ overcast digitalocean rebuild vm-01 "vm-01 backup"
```

### overcast digitalocean resize

```
Usage:
overcast digitalocean resize [name] [size] [options...]

Description:
Shutdown, resize, and reboot a DigitalOcean instance.
[size] must be a valid size slug.
If the --skip-boot flag is used, the instance will stay powered off.

Options: Defaults:
--skip-boot false

Examples:
# Resize an instance to 2gb:
$ overcast digitalocean resize vm-01 2gb
```

### overcast digitalocean snapshot

```
Usage:
overcast digitalocean snapshot [name] [snapshot-name]

Description:
Creates a named snapshot of a droplet. This will reboot the instance.

Examples:
$ overcast digitalocean snapshot vm-01 vm-01-snapshot
```

### overcast digitalocean snapshots

```
Usage:
overcast digitalocean snapshots

Description:
List all available snapshots in your account.
```

### overcast digitalocean shutdown

```
Usage:
overcast digitalocean shutdown [name]

Description:
Shut down an instance using the provider API.
```

### overcast digitalocean sizes

```
Usage:
overcast digitalocean sizes

Description:
List all available instance sizes.
```

### overcast digitalocean sync

```
Usage:
overcast digitalocean sync [name]

Description:
Fetch and update instance metadata.
```

### overcast expose

```
Usage:
overcast expose [instance|cluster|all] [port...] [options]

Description:
Reset the exposed ports on the instance or cluster using iptables.
This will fail if you don't include the current SSH port.
Specifying --allowlist will restrict all ports to the specified address(es).
These can be individual IPs or CIDR ranges, such as "192.168.0.0/24".

Expects an Ubuntu server, untested on other distributions.

Options:
--user USERNAME
--password PASSWORD
--allowlist "IP|RANGE"
--allowlist-PORT "IP|RANGE"

Examples:
Allow SSH, HTTP and HTTPS connections from anywhere:
$ overcast expose app 22 80 443

Allow SSH from anywhere, only allow Redis connections from 1.2.3.4:
$ overcast expose redis 22 6379 --allowlist-6379 "1.2.3.4"

Only allow SSH and MySQL connections from 1.2.3.4 or from 5.6.7.xxx:
$ overcast expose mysql 22 3306 --allowlist "1.2.3.4 5.6.7.0/24"
```

### overcast exposed

```
Usage:
overcast exposed [instance|cluster|all]

Description:
List the exposed ports on the instance or cluster.
Expects an Ubuntu server, untested on other distributions.

Options:
--user USERNAME
--password PASSWORD
--machine-readable, --mr
```

### overcast help

```
Usage:
overcast help

Description:
Provides help about Overcast and specific commands.
```

### overcast info

```
Usage:
overcast info
overcast info [name]

Description:
Prints the complete .overcast/clusters.json file.
Optionally display only instances matching [name].
```

### overcast init

```
Usage:
overcast init

Description:
Create an .overcast config directory in the current working directory.
No action taken if one already exists.
```

### overcast instance get

```
Usage:
overcast instance get [instance|cluster|all] [attr...] [options...]

Description:
Returns the attribute(s) for the instance or cluster, one per line,
or space-delimited using the --single-line option.
Deeply nested arrays and objects are supported.
"origin" is a compound attribute that returns user@ip:ssh-port.

Options: Defaults:
--single-line, -s false

Examples:
$ overcast instance get app-01 origin
[email protected]:22

$ overcast instance get app-cluster ip
127.0.0.1
127.0.0.2
127.0.0.3

$ overcast instance get app-01 digitalocean.image.id
103510828
```

### overcast instance add

```
Usage:
overcast instance add [name] [ip] [options...]

Description:
Adds an existing instance to a cluster.

Options: Defaults:
--cluster CLUSTER default
--ssh-port PORT 22
--ssh-key PATH overcast.key
--user USERNAME root
--password PASSWORD

Examples:
$ overcast instance add app.01 127.0.0.1 --cluster app \
--ssh-port 22222 --ssh-key $HOME/.ssh/id_rsa
```

### overcast instance list

```
Usage:
overcast instance list [cluster...]

Description:
Returns all instance names, one per line.
Optionally limit to one or more clusters.

Examples:
$ overcast instance list
$ overcast instance list app-cluster db-cluster
```

### overcast instance remove

```
Usage:
overcast instance remove [name]

Description:
Removes an instance from the index.
The server itself is not affected by this action.

Examples:
$ overcast instance remove app-01
```

### overcast instance update

```
Usage:
overcast instance update [instance|cluster|all] [options...]

Description:
Update any instance property. Specifying --cluster will move the instance
to that cluster. Specifying --name will rename the instance.

Options:
--name NAME
--cluster CLUSTER
--ip IP
--ssh-port PORT
--ssh-key PATH
--user USERNAME
--password PASSWORD

Examples:
# Update the user and ssh-key of an instance:
$ overcast instance update app.01 --user myuser --ssh-key /path/to/key

# Update ssh-port of a cluster:
$ overcast instance update app-cluster --ssh-port 22222
```

### overcast list

```
Usage:
overcast list

Description:
List your cluster and instance definitions.
```

### overcast ping

```
Usage:
overcast ping [instance|cluster|all] [options]

Description:
Display the average ping time for an instance or cluster.

Options: Defaults:
--count N, -c N 3

Examples:
$ overcast ping app-01
$ overcast ping db --count 5
```

### overcast port

```
Usage:
overcast port [instance|cluster|all] [port]

Description:
Change the SSH port for an instance or a cluster. This command fails
if the new port has not been previously opened by iptables.
See also the "expose" and "exposed" commands.

Examples:
# Expose only necessary ports:
$ overcast expose vm-01 22 55522 80 443

# Update SSH port from 22 to 55522:
$ overcast port vm-01 55522

# Close port 22:
$ overcast expose vm-01 55522 80 443
```

### overcast pull

```
Usage:
overcast pull [instance|cluster|all] [source] [dest] [options...]

Description:
Pull a file or directory from an instance or cluster using scp by default,
or using rsync if the --rsync flag is used. Source is absolute or relative
to the home directory. Destination can be absolute or relative to the
.overcast/files directory. Any reference to {instance} in the destination
will be replaced with the instance name.

Options: Defaults:
--rsync false
--user USERNAME
--password PASSWORD

Examples:
Assuming instances "app.01" and "app.02", this will expand to:
- .overcast/files/app.01.bashrc
- .overcast/files/app.02.bashrc
$ overcast pull app .bashrc {instance}.bashrc
```

### overcast push

```
Usage:
overcast push [instance|cluster|all] [source] [dest] [options...]

Description:
Push a file or directory to an instance or cluster using scp by default,
or rsync if the --rsync flag is used. Source can be absolute or relative
to the .overcast/files directory. Destination can be absolute or relative
to the home directory. Any reference to {instance} in the source will be
replaced with the instance name. The --exclude flag only works with rsync.

Options: Defaults:
--rsync false
--user USERNAME
--password PASSWORD
--exclude FILE_OR_DIRECTORY

Examples:
Assuming instances "app.01" and "app.02", this will expand to:
- .overcast/files/app.01.bashrc
- .overcast/files/app.02.bashrc
$ overcast push app {instance}.bashrc .bashrc
```

### overcast run

```
Usage:
overcast run [instance|cluster|all] [command|file...]

Description:
Execute commands or script files on an instance or cluster over SSH.
Commands will execute sequentially unless the --parallel flag is used.
An error will stop execution unless the --continueOnError flag is used.
Script files can be either absolute or relative path.

Options: Defaults:
--env "KEY=VAL KEY='1 2 3'"
--user USERNAME
--password PASSWORD
--ssh-key PATH
--ssh-args ARGS
--continueOnError false
--machine-readable, --mr false
--parallel, -p false
--shell-command "COMMAND" bash -s
--only-once false

Examples:
# Run arbirary commands and files in sequence across all instances:
$ overcast run all uptime "free -m" "df -h" /path/to/my/script

# Setting environment variables:
$ overcast run app --env "foo='bar bar' testing=123" env

# Use machine-readable output (no server prefix):
$ overcast run app-01 uptime --mr

# Run bundled and custom scripts in sequence:
$ overcast run db-* install/core install/redis ./my/install/script

# Pass along arbitrary SSH arguments, for example to force a pseudo-tty:
$ overcast run all /my/install/script --ssh-args "-tt"
```

### overcast scriptvar

```
Usage:
overcast scriptvar [instance|cluster|all] [filename] [key] [value]

Description:
Set a named variable in a remote file on an instance or cluster.
Expects a shell variable format, for example MY_VAR_NAME="my_value"

Options: Defaults:
--user USERNAME
--password PASSWORD
--continueOnError false
--machine-readable, --mr false
--parallel, -p false

Examples:
$ overcast scriptvar app-01 /path/to/file.sh MY_API_TOKEN abc123
```

### overcast slack

```
Usage:
overcast slack [message] [options...]

Description:
Sends a message to a Slack channel.
Requires a SLACK_WEBHOOK_URL property to be set in variables.json.
You can set that with the following command:
overcast var set SLACK_WEBHOOK_URL https://foo.slack.com/blah

Options: Defaults:
--channel NAME #alerts
--icon-emoji EMOJI :cloud:
--icon-url URL
--user NAME Overcast
--KEY VALUE

Examples:
$ overcast slack "Deploy completed." --icon-emoji ":satelite:"
$ overcast slack "Server stats" --channel "#general" --cpu "0.54 0.14 0.09"
```

### overcast ssh

```
Usage:
overcast ssh [instance] [options...]

Description:
Opens an interactive SSH connection to an instance.

Options:
--user USERNAME
--password PASSWORD
--ssh-key PATH

Examples:
$ overcast ssh instance-01
# To use a personal username and key in variables.json:
$ overcast vars set OVERCAST_SSH_USER my-username
$ overcast vars set OVERCAST_SSH_KEY /path/to/my.key
$ overcast ssh instance-01 # will use the above variables to attempt a connection
```

### overcast sshkey create

```
Usage:
overcast sshkey create [name]

Description:
Creates a new SSH key in the current .overcast config.

Examples:
$ overcast sshkey create myKeyName
New SSH key "myKeyName" created.
- /path/to/.overcast/keys/myKeyName.key
- /path/to/.overcast/keys/myKeyName.key.pub
```

### overcast sshkey delete

```
Usage:
overcast sshkey delete [name]

Description:
Deletes SSH public/private key files from the current .overcast config.

Examples:
$ overcast sshkey delete myKeyName
SSH key "myKeyName" deleted.
```

### overcast sshkey get

```
Usage:
overcast sshkey get [name] [option]

Description:
Display the requested SSH key data or path from the current .overcast config.
Defaults to displaying the public key data if no option found.

Options:
--public-data
--private-data
--public-path
--private-path

Examples:
$ overcast sshkey get myKeyName
[public key data]
$ overcast sshkey get myKeyName --private-data
[private key data]
```

### overcast sshkey list

```
Usage:
overcast sshkey list

Description:
List the found SSH key names in the current .overcast config.

Examples:
$ overcast sshkey list
myKeyName
overcast
```

### overcast sshkey push

```
Usage:
overcast sshkey push [instance|cluster|all] [name|path] [options...]

Description:
Push a public SSH key to an instance or cluster. Accepts a key name,
filename, or full path. This will overwrite the existing authorized_keys
file, unless you use --append.

Options: Defaults:
--user USERNAME
--append, -a false

Examples:
# Generate new SSH key pair:
$ overcast sshkey create newKey

# Push public key to instance, update instance config to use private key:
$ overcast sshkey push vm-01 newKey
$ overcast instance update vm-01 --ssh-key newKey.key

# Same as above but using key path instead of key name:
$ overcast sshkey push vm-02 "~/.ssh/id_rsa.pub"
$ overcast instance update vm-02 --ssh-key "~/.ssh/id_rsa"

# Push public key to instance using arbitrary user:
$ overcast sshkey push vm-03 newKey --user myOtherUser

# Append public key to authorized_keys instead of overwriting:
$ overcast sshkey push vm-04 newKey --append
```

### overcast tunnel

```
Usage:
overcast tunnel [instance] [local-port((:hostname):remote-port)...]

Description:
Opens an SSH tunnel to the port(s) specified.
If only one port is specified, assume the same port for local/remote.
If no remote host is specified, assume the remote host itself (127.0.0.1).
Multiple tunnels can be opened over a single connection.

Options:
--user USERNAME
--password PASSWORD
--ssh-key PATH

Examples:
# Tunnel local 5984 to remote 5984
$ overcast tunnel app-01 5984

# Tunnel local 8000 to remote 5984, local 8001 to remote 3000
$ overcast tunnel app-01 8000:5984 8001:3000

# Tunnel local 3000 to otherhost.com:4000
$ overcast tunnel app-01 3000:otherhost.com:4000
```

### overcast vars list

```
Usage:
overcast vars list

Description:
List all variables in your current config.
```

### overcast vars set

```
Usage:
overcast vars set [name] [value]

Description:
Set a variable in your current config.

Examples:
$ overcast vars set AWS_KEY myawskey12345
$ overcast vars set MY_CUSTOM_VARIABLE_NAME foo
```

### overcast vars get

```
Usage:
overcast vars get [name]

Description:
Get a variable from your current config.

Examples:
$ overcast vars get AWS_KEY
> myawskey12345

$ overcast vars get MY_CUSTOM_VARIABLE_NAME
> foo
```

### overcast vars delete

```
Usage:
overcast vars delete [name]

Description:
Delete a variable from your current config.

Examples:
$ overcast vars delete MY_CUSTOM_VARIABLE_NAME
```

### overcast virtualbox boot

```
Usage:
overcast virtualbox boot [name]

Description:
Boot up a Virtualbox instance.
```

### overcast virtualbox create

```
Usage:
overcast virtualbox create [name] [options...]

Description:
Creates a new Virtualbox instance.

Options: Defaults:
--cluster CLUSTER default
--cpus COUNT 1
--image NAME trusty64
--ram MB 512
--ip ADDRESS 192.168.22.10
--ssh-key PATH overcast.key
--ssh-pub-key PATH overcast.key.pub

Examples:
$ overcast virtualbox create vm-01
$ overcast virtualbox create vm-02 --ram 1024 --image precise64
```

### overcast virtualbox destroy

```
Usage:
overcast virtualbox destroy [name] [options...]

Description:
Destroys a Virtualbox instance.
Using --force overrides the confirm dialog.

Options: Defaults:
--force false

Examples:
$ overcast virtualbox destroy vm-01
```

### overcast virtualbox reboot

```
Usage:
overcast virtualbox reboot [name]

Description:
Reboots a Virtualbox instance.
```

### overcast virtualbox shutdown

```
Usage:
overcast virtualbox shutdown [name]

Description:
Shut down a Virtualbox instance.
```

### overcast wait

```
Usage:
overcast wait [seconds]

Description:
Show a progress bar for a specified number of seconds.

Examples:
$ overcast wait 30
```

## Running the Tests

[![Build Status](https://travis-ci.org/andrewchilds/overcast.svg?branch=master)](https://travis-ci.org/andrewchilds/overcast)

```sh
npm test
```

## Upgrading Overcast

```sh
npm -g update overcast
```

Configuration files are left alone during an upgrade.

## Contributing

Contributions are welcome. If you've got an idea for a feature or found a bug, please [open an issue](https://github.com/andrewchilds/overcast/issues). If you're a developer and want to help improve Overcast, [open a pull request](https://github.com/andrewchilds/overcast/pulls) with your changes.

### Hacking Overcast

```sh
git clone [email protected]:andrewchilds/overcast.git
cd overcast

# To use local development version:
node overcast [args...]

# To use published version:
overcast [args...]
```

## Roadmap

- Standardize Recipes
- Tagging
- Google Compute Engine support
- More comprehensive script/recipe library
- More test coverage

## License

MIT. Copyright © 2014-2022 Andrew Childs