Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/spotty-cloud/spotty
Training deep learning models on AWS and GCP instances
https://github.com/spotty-cloud/spotty
aws deep-learning docker gcp google-cloud gpu sagemaker spot-instances
Last synced: 3 months ago
JSON representation
Training deep learning models on AWS and GCP instances
- Host: GitHub
- URL: https://github.com/spotty-cloud/spotty
- Owner: spotty-cloud
- License: mit
- Created: 2017-10-30T22:26:37.000Z (about 7 years ago)
- Default Branch: dev
- Last Pushed: 2023-10-18T09:11:13.000Z (about 1 year ago)
- Last Synced: 2024-08-04T20:03:44.364Z (3 months ago)
- Topics: aws, deep-learning, docker, gcp, google-cloud, gpu, sagemaker, spot-instances
- Language: Python
- Homepage: https://spotty.cloud
- Size: 1.26 MB
- Stars: 491
- Watchers: 9
- Forks: 43
- Open Issues: 21
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
[](https://spotty.cloud)
[](https://pypi.org/project/spotty/)
Spotty drastically simplifies training of deep learning models on [AWS](https://aws.amazon.com/)
and [GCP](https://cloud.google.com/):
- it makes training on GPU instances as simple as training on your local machine
- it automatically manages all necessary cloud resources including images, volumes, snapshots and SSH keys
- it makes your model trainable in the cloud by everyone with a couple of commands
- it uses [tmux](https://en.wikipedia.org/wiki/Tmux) to easily detach remote processes from their terminals
- it saves you up to 70% of the costs by using [AWS Spot Instances](https://aws.amazon.com/ec2/spot/)
and [GCP Preemtible VMs](https://cloud.google.com/preemptible-vms/)
## Documentation
- See the [documentation page](https://spotty.cloud).
- Read [this](https://medium.com/@apls/how-to-train-deep-learning-models-on-aws-spot-instances-using-spotty-8d9e0543d365)
article on Medium for a real-world example.
## Installation
Requirements:
* Python >=3.6
* AWS CLI (see [Installing the AWS Command Line Interface](http://docs.aws.amazon.com/cli/latest/userguide/installing.html))
if you're using AWS
* Google Cloud SDK (see [Installing Google Cloud SDK](https://cloud.google.com/sdk/install))
if you're using GCP
Use [pip](http://www.pip-installer.org/en/latest/) to install or upgrade Spotty:
$ pip install -U spotty
## Get Started
1. Prepare a `spotty.yaml` file and put it to the root directory of your project:
- See the file specification [here](https://spotty.cloud/docs/user-guide/configuration-file.html).
- Read [this](https://medium.com/@apls/how-to-train-deep-learning-models-on-aws-spot-instances-using-spotty-8d9e0543d365)
article for a real-world example.
2. Start an instance:
```bash
$ spotty start
```
It will run a Spot Instance, restore snapshots if any, synchronize the project with the running instance
and start the Docker container with the environment.
3. Train a model or run notebooks.
To connect to the running container via SSH, use the following command:
```bash
$ spotty sh
```
It runs a [tmux](https://github.com/tmux/tmux/wiki) session, so you can always detach this session using
__`Ctrl + b`__, then __`d`__ combination of keys. To be attached to that session later, just use the
`spotty sh` command again.
Also, you can run your custom scripts inside the Docker container using the `spotty run ` command. Read more
about custom scripts in the documentation:
[Configuration: "scripts" section](https://spotty.cloud/docs/configuration-file/#scripts-section-optional).
## Contributions
Any feedback or contributions are welcome! Please check out the [guidelines](CONTRIBUTING.md).
## License
[MIT License](LICENSE)