{"id":15042359,"url":"https://github.com/manjavacas/cloudy","last_synced_at":"2026-01-28T07:02:35.417Z","repository":{"id":256947368,"uuid":"856891057","full_name":"manjavacas/cloudy","owner":"manjavacas","description":"CLOUDY automates the execution of experiments on Google Cloud.","archived":false,"fork":false,"pushed_at":"2024-10-03T13:42:37.000Z","size":19010,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-28T08:51:15.134Z","etag":null,"topics":["bash","bash-script","cloud","gcp","google","google-cloud","google-cloud-platform","google-cloud-storage","google-compute-engine","python","virtual-machine"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/manjavacas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-13T12:03:57.000Z","updated_at":"2024-11-07T11:49:21.000Z","dependencies_parsed_at":"2024-09-25T01:41:04.707Z","dependency_job_id":"c16aadf3-3646-451c-b566-8584416cda5c","html_url":"https://github.com/manjavacas/cloudy","commit_stats":null,"previous_names":["manjavacas/cloudexec","manjavacas/cloudy"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manjavacas%2Fcloudy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manjavacas%2Fcloudy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manjavacas%2Fcloudy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manjavacas%2Fcloudy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/manjavacas","download_url":"https://codeload.github.com/manjavacas/cloudy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248956039,"owners_count":21189277,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bash","bash-script","cloud","gcp","google","google-cloud","google-cloud-platform","google-cloud-storage","google-compute-engine","python","virtual-machine"],"created_at":"2024-09-24T20:47:10.034Z","updated_at":"2026-01-28T07:02:30.381Z","avatar_url":"https://github.com/manjavacas.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"images/logo.png\" alt=\"CLOUDY\" width=50% /\u003e\n\u003c/p\u003e\n\n![License](https://img.shields.io/badge/license-GPLv3-blue)\n[![Release](https://badgen.net/github/release/manjavacas/cloudy)]()\n[![Contributors](https://badgen.net/github/contributors/manjavacas/cloudy)]()\n[![DOI](https://zenodo.org/badge/856891057.svg)](https://doi.org/10.5281/zenodo.13885959)\n\n**CLOUDY** automates the execution of experiments on [Google Cloud](https://console.cloud.google.com). It creates VM instances and buckets, installs dependencies, runs Python scripts, and handles resource cleanup.\n\n## ⚙️ **How it works?**\n\nThe workflow of **CLOUDY** comprises the following steps:\n\n1. The script `launch.sh` prepares a VM instance, according to the options specified in the `config.json` file.\n\n2. The script `setup.sh` is executed in the VM to install dependencies and run the Python script indicated.\n\n3. The output is saved to an existing bucket, or a new one is created as required.\n\n4. The instance is automatically deleted once its execution has finished.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"images/diagram.png\" alt=\"DIAGRAM\" width=80% /\u003e\n\u003c/p\u003e\n\n## 📄 **Scripts**\n\nThis project consists of the following scripts:\n\n- `launch.sh`: creates a VM instance on Google Cloud according to the configuration defined in `config.json`. It also downloads and copies your repository to the VM instance.\n- `setup.sh`: runs on the VM instance. Installs dependencies, runs your Python script, and saves the results to a Google Cloud bucket, creating it if necessary.\n- `clean.sh`: cleans up all VM instances and buckets on Google Cloud.\n- `Makefile`: enables the execution of the scripts through simple commands.\n\n## 💻 **How to use CLOUDY?**\n\n1. **Prerequisites**\n\n   First, create a [service account on GCP](https://cloud.google.com/iam/docs/service-accounts-create?hl=en) with the required permissions for Compute Engine and Cloud Storage (e.g., *storage administrator*, *compute instances administrator*).\n\n   Then, install the following dependencies:\n\n    - `Google Cloud SDK`: required to interact with Google Cloud from the command line.\n\n    - `jq`: used to read the JSON configuration file.\n\n2. **Edit `config.json`**\n\n   Define your custom configuration in the `config.json` file, located in the root directory of the project. For example:\n\n   ```json\n    {\n        \"INSTANCE_NAME\": \"vm\",\n        \"BUCKET_NAME\": \"bucket\",\n        \"REPO_URL\": \"https://github.com/manjavacas/cloudy.git\",\n        \"SCRIPT_PATH\": \"foo/foo.py\",\n        \"SCRIPT_ARGS\": \"cloudy\",\n        \"DEPENDENCIES\": \"numpy pandas\",\n        \"SERVICE_ACCOUNT\": \"service@account.iam.gserviceaccount.com\",\n        \"SETUP_SCRIPT\": \"setup.sh\",\n        \"MACHINE_TYPE\": \"n2-standard-2\",\n        \"ZONE\": \"europe-southwest1-b\",\n        \"IMAGE_FAMILY\": \"ubuntu-2004-lts\",\n        \"IMAGE_PROJECT\": \"ubuntu-os-cloud\",\n        \"BUCKET_ZONE\": \"eu\"\n    }\n    ```\n\n    The main options to edit are:\n\n    - `INSTANCE_NAME` and `BUCKET_NAME`: identifiers for the created instance and bucket.\n    - `REPO_URL`: the repository to clone. This is where the code you want to execute is located.\n    - `SCRIPT_PATH` and `SCRIPT_ARGS`: path to the Python script you want to execute in the repository, along with its input arguments.\n    - `DEPENDENCIES`: dependencies required to run the Python script.\n    - `SERVICE_ACCOUNT`: GCP service account to be used. It must have the necessary permissions.\n\n3. **Run CLOUDY**\n\n    a. **Using `Makefile`**\n\n    - To launch a VM instance, run:\n\n    ```bash\n    $ make launch\n    ```\n\n    - To clean up all VM instances and buckets, run:\n\n    ```bash\n    $ make clean\n    ```\n\n    - To delete VM instances and buckets and then relaunch, run:\n\n    ```bash\n    $ make reset\n    ```\n\n    b. **Using `cloudy.py`**\n\n    Alternatively, you can use the Python script `cloud.py` for the same operations:\n\n    ```bash\n    $ python cloudy.py launch\n    $ python cloudy.py clean\n    $ python cloudy.py reset\n    ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanjavacas%2Fcloudy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmanjavacas%2Fcloudy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanjavacas%2Fcloudy/lists"}