{"id":18825556,"url":"https://github.com/grycap/ec3","last_synced_at":"2025-04-14T01:31:34.707Z","repository":{"id":21878002,"uuid":"25201639","full_name":"grycap/ec3","owner":"grycap","description":"Elastic Cloud Computing Cluster tool","archived":false,"fork":false,"pushed_at":"2024-07-19T11:05:46.000Z","size":7545,"stargazers_count":33,"open_issues_count":1,"forks_count":15,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-27T15:52:18.978Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://www.grycap.upv.es/ec3","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grycap.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2014-10-14T10:35:27.000Z","updated_at":"2024-07-19T11:05:50.000Z","dependencies_parsed_at":"2024-02-15T09:36:07.699Z","dependency_job_id":null,"html_url":"https://github.com/grycap/ec3","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grycap%2Fec3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grycap%2Fec3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grycap%2Fec3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grycap%2Fec3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grycap","download_url":"https://codeload.github.com/grycap/ec3/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248807573,"owners_count":21164711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T00:59:57.983Z","updated_at":"2025-04-14T01:31:29.695Z","avatar_url":"https://github.com/grycap.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":".. image:: doc/EC3-logo-3d.png\n   :height: 50px\n   :width: 41 px\n   :scale: 50 %\n   :alt: alternate text\n   :align: right\n   \n.. Elastic Cloud Computing Cluster (EC3)\n=====================================\n\nElastic Cloud Computing Cluster (EC3) is a tool to create elastic virtual clusters on top\nof Infrastructure as a Service (IaaS) providers, either public (such as `Amazon Web Services`_,\n`Google Cloud`_ or `Microsoft Azure`_)\nor on-premises (such as `OpenNebula`_ and `OpenStack`_). We offer recipes to deploy `TORQUE`_\n(optionally with `MAUI`_), `SLURM`_, `SGE`_, `HTCondor`_, `Mesos`_, `Nomad`_ and `Kubernetes`_ clusters that can be self-managed with `CLUES`_:\nit starts with a single-node cluster and working nodes will be dynamically deployed and provisioned\nto fit increasing load (number of jobs at the LRMS). Working nodes will be undeployed when they are idle.\nThis introduces a cost-efficient approach for Cluster-based computing.\n   \nInstallation\n------------\n\nRequisites\n~~~~~~~~~~\n\nThe program `ec3` requires Python 2.6+, `PLY`_, `PyYAML`_, `Requests`_, `jsonschema`_ and an `IM`_ server,\nwhich is used to launch the virtual machines.\n\n`PyYAML`_ is usually available in distribution repositories (``python-yaml`` in Debian;\n``PyYAML`` in Red Hat; and ``PyYAML`` in pip).\n\n`PLY`_ is usually available in distribution repositories (``python-ply`` and ``ply`` in pip).\n\n`Requests`_ is usually available in distribution repositories (``python-requests`` and ``requests`` in pip).\n\n`jsonschema`_ is usually available in distribution repositories (``python-jsonschema`` and ``jsonschema`` in pip).\n\nBy default `ec3` uses our public `IM`_ server in `appsgrycap.i3m.upv.es`. *Optionally* you can deploy a \nlocal `IM`_ server following the instructions of the `IM manual`_.\n \nAlso ``sshpass`` command is required to provide the user with ssh access to the cluster.\n\nInstalling\n~~~~~~~~~~\n\nAs Python 2 is no longer supported, we recommend to install ec3 with Python 3.\n\nFirst you need to install pip tool. To install them in Debian and Ubuntu based distributions, do::\n\n\tsudo apt update\n\tsudo apt install -y python3-pip\n\nIn Red Hat based distributions (RHEL, CentOS, Amazon Linux, Oracle Linux, Fedora, etc.), do::\n\n\tsudo yum install -y epel-release\n\tsudo yum install -y which python3-pip\n\t\nThen you only have to call the install command of the pip tool with the `ec3-cli` package::\n\t\n    sudo pip3 install ec3-cli\n\nYou can also download the last `ec3` version from `this \u003chttps://github.com/grycap/ec3\u003e`_ git repository::\n\n   git clone https://github.com/grycap/ec3\n\nThen you can install it calling the pip tool with the current ec3 directory::\n\t\n    sudo pip3 install ./ec3\n\n\nBasic example with Amazon EC2\n-----------------------------\n\nFirst create a file ``auth.txt`` with a single line like this::\n\n   id = provider ; type = EC2 ; username = \u003c\u003cAccess Key ID\u003e\u003e ; password = \u003c\u003cSecret Access Key\u003e\u003e\n\nReplace ``\u003c\u003cAccess Key ID\u003e\u003e`` and ``\u003c\u003cSecret Access Key\u003e\u003e`` with the corresponding values\nfor the AWS account where the cluster will be deployed. It is safer to use the credentials\nof an IAM user created within your AWS account.\n\nThis file is the authorization file (see `Authorization file`_), and can have more than one set of credentials.\n\nThe next command deploys a `TORQUE`_ cluster based on an `Ubuntu`_ image::\n\n   $ ec3 launch mycluster torque ubuntu-ec2 -a auth.txt -y\n   WARNING: you are not using a secure connection and this can compromise the secrecy of the passwords and private keys available in the authorization file.\n   Creating infrastructure\n   Infrastructure successfully created with ID: 60\n      ▄▟▙▄¨        Front-end state: running, IP: 132.43.105.28\n\nIf you deployed a local `IM`_ server, use the next command instead::\n\n   $ ec3 launch mycluster torque ubuntu-ec2 -a auth.txt -u http://localhost:8899\n\nThis can take several minutes.\n\nBear in mind that you have to specify a resource manager (like ``torque`` in our example) in addition to the images that you want to deploy (e.g. ``ubuntu-ec2``). For more information about this check the `templates documentation`_.\n\nYou can show basic information about the deployed clusters by executing::\n\n    $ ec3 list\n        name       state          IP        nodes\n     ---------------------------------------------\n      mycluster  configured  132.43.105.28    0\n\nOnce the cluster has been deployed, open a ssh session to the front-end (you may need to install the ``sshpass`` library)::\n\n   $ ec3 ssh mycluster\n   Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-24-generic x86_64)\n   Documentation:  https://help.ubuntu.com/\n   ubuntu@torqueserver:~$\n\nYou may use the cluster as usual, depending on the LRMS.\nFor Torque, you can decide to submit a couple of jobs using qsub, to test elasticity in the cluster::\n\n   $ for i in 1 2; do echo \"/bin/sleep 50\" | qsub; done\n\nNotice that CLUES will intercept the jobs submited to the LRMS to deploy additional working nodes if needed.\nThis might result in a customizable (180 seconds by default) blocking delay when submitting jobs when no additional working nodes are available.\nThis guarantees that jobs will enter execution as soon as the working nodes are deployed and integrated in the cluster.\n\nWorking nodes will be provisioned and relinquished automatically to increase and decrease the cluster size according to the elasticity policies provided by CLUES.\n\nEnjoy your virtual elastic cluster!\n\n\nEC3 in Docker Hub\n-----------------\n\nEC3 has an official Docker container image available in `Docker Hub`_ and `GitHub Container Regitry`_ \nthat can be used instead of installing the CLI. You can download it by typing:: \n\n   $ sudo docker pull grycap/ec3\n   or\n   $ sudo docker pull ghcr.io/grycap/ec3\n\nYou can exploit all the potential of EC3 as if you download the CLI and run it on your computer:: \n\n   $ sudo docker run grycap/ec3 list\n   $ sudo docker run grycap/ec3 templates\n \nTo launch a cluster, you can use the recipes that you have locally by mounting the folder as a volume. Also it is recommendable to mantain the data of active clusters locally, by mounting a volume as follows::\n\n   $ sudo docker run -v /home/user/:/tmp/ -v /home/user/ec3/templates/:/etc/ec3/templates -v /home/user/.ec3/clusters:/root/.ec3/clusters grycap/ec3 launch mycluster torque ubuntu16 -a /tmp/auth.dat \n\nNotice that you need to change the local paths to the paths where you store the auth file, the templates folder and the .ec3/clusters folder. So, once the front-end is deployed and configured you can connect to it by using::\n\n   $ sudo docker run -ti -v /home/user/.ec3/clusters:/root/.ec3/clusters grycap/ec3 ssh mycluster\n\nLater on, when you need to destroy the cluster, you can type::\n\n   $ sudo docker run -ti -v /home/user/.ec3/clusters:/root/.ec3/clusters grycap/ec3 destroy mycluster\n\n\nAdditional information\n----------------------\n\n* `EC3 Command-line Interface`_.\n* `Templates`_.\n* Information about available templates: ``ec3 templates [--search \u003ctopic\u003e] [--full-description]``.\n\n.. _`CLUES`: http://www.grycap.upv.es/clues/\n.. _`RADL`: http://www.grycap.upv.es/im/doc/radl.html\n.. _`TORQUE`: http://www.adaptivecomputing.com/products/open-source/torque\n.. _`MAUI`: http://www.adaptivecomputing.com/products/open-source/maui/\n.. _`SLURM`: http://slurm.schedmd.com/\n.. _`SGE`: http://gridscheduler.sourceforge.net/\n.. _`Mesos`: http://mesos.apache.org/\n.. _`HTCondor`: https://research.cs.wisc.edu/htcondor/\n.. _`Nomad`: https://www.nomadproject.io/\n.. _`Kubernetes`: https://kubernetes.io/\n.. _`Scientific Linux`: https://www.scientificlinux.org/\n.. _`Ubuntu`: http://www.ubuntu.com/\n.. _`OpenNebula`: http://www.opennebula.org/\n.. _`OpenStack`: http://www.openstack.org/\n.. _`Amazon Web Services`: https://aws.amazon.com/\n.. _`Google Cloud`: http://cloud.google.com/\n.. _`Microsoft Azure`: http://azure.microsoft.com/\n.. _`IM`: https://github.com/grycap/im\n.. _`PyYAML`: http://pyyaml.org/wiki/PyYAML\n.. _`PLY`: http://www.dabeaz.com/ply/\n.. _`Requests`: http://docs.python-requests.org/\n.. _`EC3 Command-line Interface`: http://ec3.readthedocs.org/en/devel/ec3.html\n.. _`Command templates`: http://ec3.readthedocs.org/en/devel/ec3.html#command-templates\n.. _`Authorization file`: http://ec3.readthedocs.org/en/devel/ec3.html#authorization-file\n.. _`Templates`: http://ec3.readthedocs.org/en/devel/templates.html\n.. _`templates documentation`: http://ec3.readthedocs.org/en/devel/templates.html#ec3-types-of-templates\n.. _`Docker Hub`: https://hub.docker.com/r/grycap/ec3/\n.. _`EC3aaS`: http://servproject.i3m.upv.es/ec3/\n.. _`sshpass`: https://gist.github.com/arunoda/7790979\n.. _`ubuntu-ec2`: https://github.com/grycap/ec3/blob/devel/templates/ubuntu-ec2.radl\n.. _`IM manual`: https://imdocs.readthedocs.io/en/latest/manual.html\n.. _`jsonschema`: https://github.com/Julian/jsonschema\n.. _`GitHub Container Regitry`: https://github.com/grycap/ec3/pkgs/container/ec3","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrycap%2Fec3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrycap%2Fec3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrycap%2Fec3/lists"}