Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/manjavacas/cloudy
CLOUDY automates the execution of experiments on Google Cloud.
https://github.com/manjavacas/cloudy
bash bash-script cloud gcp google google-cloud google-cloud-platform google-cloud-storage google-compute-engine python virtual-machine
Last synced: 2 days ago
JSON representation
CLOUDY automates the execution of experiments on Google Cloud.
- Host: GitHub
- URL: https://github.com/manjavacas/cloudy
- Owner: manjavacas
- License: gpl-3.0
- Created: 2024-09-13T12:03:57.000Z (20 days ago)
- Default Branch: main
- Last Pushed: 2024-09-18T08:41:18.000Z (15 days ago)
- Last Synced: 2024-09-26T01:41:15.129Z (7 days ago)
- Topics: bash, bash-script, cloud, gcp, google, google-cloud, google-cloud-platform, google-cloud-storage, google-compute-engine, python, virtual-machine
- Language: Shell
- Homepage:
- Size: 14.8 MB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[]()
[]()
**CLOUDY** automates the execution of experiments on [Google Cloud](https://console.cloud.google.com). It creates VM instances and buckets, installs dependencies, runs Python scripts, and handles resource cleanup.
## ⚙️ **How it works?**
The workflow of **CLOUDY** comprises the following steps:
1. The script `launch.sh` prepares a VM instance, according to the options specified in the `config.json` file.
2. The script `setup.sh` is executed in the VM to install dependencies and run the Python script indicated.
3. The output is saved to an existing bucket, or a new one is created as required.
4. The instance is automatically deleted once its execution has finished.
## 📄 **Scripts**
This project consists of the following scripts:
- `launch.sh`: creates a VM instance on Google Cloud according to the configuration defined in `config.json`. It also downloads and copies your repository to the VM instance.
- `setup.sh`: runs on the VM instance. Installs dependencies, runs your Python script, and saves the results to a Google Cloud bucket, creating it if necessary.
- `clean.sh`: cleans up all VM instances and buckets on Google Cloud.
- `Makefile`: enables the execution of the scripts through simple commands.
## 💻 **How to use CLOUDY?**
1. **Prerequisites**
First, create a [service account on GCP](https://cloud.google.com/iam/docs/service-accounts-create?hl=en) with the required permissions for Compute Engine and Cloud Storage (e.g., *storage administrator*, *compute instances administrator*).
Then, install the following dependencies:
- `Google Cloud SDK`: required to interact with Google Cloud from the command line.
- `jq`: used to read the JSON configuration file.
2. **Edit `config.json`**
Define your custom configuration in the `config.json` file, located in the root directory of the project. For example:
```json
{
"INSTANCE_NAME": "vm",
"BUCKET_NAME": "bucket",
"REPO_URL": "https://github.com/manjavacas/cloudy.git",
"SCRIPT_PATH": "foo/foo.py",
"SCRIPT_ARGS": "cloudy",
"DEPENDENCIES": "numpy pandas",
"SERVICE_ACCOUNT": "[email protected]",
"SETUP_SCRIPT": "setup.sh",
"MACHINE_TYPE": "n2-standard-2",
"ZONE": "europe-southwest1-b",
"IMAGE_FAMILY": "ubuntu-2004-lts",
"IMAGE_PROJECT": "ubuntu-os-cloud",
"BUCKET_ZONE": "eu"
}
```
The main options to edit are:
- `INSTANCE_NAME` and `BUCKET_NAME`: identifiers for the created instance and bucket.
- `REPO_URL`: the repository to clone. This is where the code you want to execute is located.
- `SCRIPT_PATH` and `SCRIPT_ARGS`: path to the Python script you want to execute in the repository, along with its input arguments.
- `DEPENDENCIES`: dependencies required to run the Python script.
- `SERVICE_ACCOUNT`: GCP service account to be used. It must have the necessary permissions.
3. **Run CLOUDY**
a. **Using `Makefile`**
- To launch a VM instance, run:
```bash
$ make launch
```
- To clean up all VM instances and buckets, run:
```bash
$ make clean
```
- To delete VM instances and buckets and then relaunch, run:
```bash
$ make reset
```
b. **Using `cloudy.py`**
Alternatively, you can use the Python script `cloud.py` for the same operations:
```bash
$ python cloudy.py launch
$ python cloudy.py clean
$ python cloudy.py reset
```