{"id":19546244,"url":"https://github.com/hewlettpackard/ezdemo","last_synced_at":"2025-04-26T19:32:25.907Z","repository":{"id":39659612,"uuid":"457491426","full_name":"HewlettPackard/ezdemo","owner":"HewlettPackard","description":"HPE Ezmeral Deployment tool for demos","archived":false,"fork":false,"pushed_at":"2023-10-03T20:50:35.000Z","size":1145,"stargazers_count":17,"open_issues_count":11,"forks_count":18,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-04-04T17:45:20.712Z","etag":null,"topics":["ai","datafabric","demo","enterprise","ezmeral","hpe","kubernetes","mapr"],"latest_commit_sha":null,"homepage":"https://www.hpe.com/ezmeral","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HewlettPackard.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-02-09T19:01:41.000Z","updated_at":"2024-07-25T09:35:27.000Z","dependencies_parsed_at":"2022-07-13T12:10:52.511Z","dependency_job_id":null,"html_url":"https://github.com/HewlettPackard/ezdemo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HewlettPackard%2Fezdemo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HewlettPackard%2Fezdemo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HewlettPackard%2Fezdemo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HewlettPackard%2Fezdemo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HewlettPackard","download_url":"https://codeload.github.com/HewlettPackard/ezdemo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251041566,"owners_count":21527216,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","datafabric","demo","enterprise","ezmeral","hpe","kubernetes","mapr"],"created_at":"2024-11-11T03:43:59.597Z","updated_at":"2025-04-26T19:32:20.896Z","avatar_url":"https://github.com/HewlettPackard.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ezmeral Container Platform Demo\n\nAutomated installation for Ezmeral Container Platform and MLOps on various platforms for demo purposes.\n\nYou need a container runtime to run the tool. It should work on any container runtime and tested on Docker. Podman doesn't work if you try to map volumes (should work fine without the mounts).\n\n## Recognition\n\nThis tool is possible because of the amazing work by;\n\n- Chris Snow (\u003chttps://github.com/snowch\u003e)\n\n  and\n\n- Dirk Derichsweiler (\u003chttps://github.com/dderichswei\u003e)\n\n## Usage\n\nIt is recommended to use a workspace (folder) to keep your settings, instead of editing settings files in the container. You need to create a user.settings and credentials file (as explained below).\n\nCreate a file named \"user.settings\" as following:\n\n```json\n{\n  \"project_id\": \"\",\n  \"user\": \"\",\n  \"admin_password\": \"ChangeMe.\",\n  \"is_mlops\": false,\n  \"is_mapr\": false,\n  \"is_mapr_ha\": true,\n  \"is_gpu\": false,\n  \"is_ha\": false,\n  \"is_runtime\": true,\n  \"is_verbose\": true,\n  \"install_ad\": true,\n  \"extra_tags\": {}\n}\n```\n\nCreate \"aws_config.json\", \"azure_config.json\", or \"dc.ini\" file in the same folder. Use one of the following templates:\n\nAWS Template:\n\n```json\n{\n  \"aws_access_key\": \"\",\n  \"aws_secret_key\": \"\",\n  \"region\": \"\"\n}\n```\n\nAzure Template:\n\n```json\n{\n  \"az_subscription\": \"\",\n  \"az_appId\": \"\",\n  \"az_password\": \"\",\n  \"az_tenant\": \"\",\n  \"region\": \"\"\n}\n```\n\nData Center Template:\n\n```ini\n[DEFAULT]\n### Data Center Settings\ndc=name_of_dc # used for specific settings\nvm_network=10.1.1.192/24 # IP allocation will start with the given IP in that subnet, controller the first (or three if HA etc)\n### VMWARE SETTINGS\nvcenter_server=\nvcenter_user=\nvcenter_password=\nvcenter_datacenter_name=\nvcenter_cluster_name=\nvcenter_template_name=            # CentOS/RHEL 7.x template with min 400GB OS disk\nvcenter_template_user=            # user with paswordless sudo configured\nvcenter_template_keyfile=         # SSH private key for the centos tempalte user\nvcenter_mapr_template_name=       # Changed to Rocky 8.5 (requires perl and cloud-init installed) / Ubuntu 20.04 template with min 100GB OS disk\nvcenter_mapr_template_user=       # user with paswordless sudo configured\nvcenter_mapr_template_keyfile=      # SSH private key for the ubuntu template user\nvcenter_folder=\nvcenter_datastore=\nvcenter_vswitch=\nvm_gateway=\nvm_dns=\nvm_domain=\n\n```\n\nMore details and customization for on premise deployment is documented in [DC Readme file](./server/dc/README.md).\n\nYou will need to copy the private key file(s) required by the templates into the container, set its permissions to 0600 (`chmod 0600 prv.key`) and provide the full path inside the container.\n\nDownload the [start script](https://raw.githubusercontent.com/HewlettPackard/ezdemo/main/start.sh), or copy/paste below to start the container.\n\n```bash\n#!/usr/bin/env bash\nVOLUMES=()\nCONFIG_FILES=(\"aws_config.json\" \"azure_config.json\")\nfor file in \"${CONFIG_FILES[@]}\"\ndo\n  target=\"${file%_*}\"\n  [[ -f \"./${file}\" ]] \u0026\u0026 VOLUMES+=(\"$(pwd)/${file}:/app/server/${target}/config.json:rw\")\ndone\n\n[[ -f \"./dc.ini\" ]] \u0026\u0026 VOLUMES+=(\"$(pwd)/dc.ini:/app/server/${target}/dc.ini:rw\")\n[[ -f \"./user.settings\" ]] \u0026\u0026 VOLUMES+=(\"$(pwd)/user.settings:/app/server/user.settings:rw\")\n\nprintf -v joined ' -v %s' \"${VOLUMES[@]}\"\n\ndocker run --name ezdemo --pull always -d -p 4000:4000 -p 8443:8443 -p 9443:9443 ${joined} erdincka/ezdemo:latest\n```\n\nOnce the container starts, you can either use the WebUI on \u003chttp://localhost:4000/\u003e or run scripts manually within the container.\n\n## Web Interface\n\nSelect your target\n![Select target](./public/Providers.jpg)\n\nSettings\n![Select target](./public/Settings.jpg)\n\nStart and Monitor Deployment\n![Select target](./public/Deployment.jpg)\n\n## Advanced Usage\n\nExec into the container either by using [exec scripts](https://raw.githubusercontent.com/HewlettPackard/ezdemo/main/docker-exec.sh) or the following command:\n\n```bash\ndocker exec -it \"$(docker ps -f \"status=running\" -f \"ancestor=erdincka/ezdemo\" -q)\" /bin/bash\n```\n\n### Run all at once\n\n`./00-run_all.sh aws|azure|dc`\n\n### Or run in stages\n\nProcess is divided into steps to provide granularity.\n\nStep 1 (Init): Prepare the environment (ensure that the required settings are in place, utilities are configured etc).\n\nStep 2 (Apply): To deploy the infrastructure nodes based on the selection (runtime, mlops, mapr etc)\n\nStep 3 (Install): Platform installation based on the selection (AD server, runtime platform, Data Fabric nodes etc).\n\nStep 4 (Configure): Further configuration of the platform(s) for selected options, such as;\n\n- Runtime (is_runtime=true in user.settings file): creates a Kubernetes cluster with an example tenant enabling Spark operator\n\n- MLOps (is_mlops=true in user.settings file): creates a Kubernetes cluster with Data Fabric on Kubernetes as Tenant Storage and configures end to end model serving scenario, such as deployment and configuration of Kubeflow, MLflow, Livy, Spark Operator and a Notebook (includes project repository for CI/CD, training cluster etc)\n\n- Data Fabric (is_mapr=true in user.settings file): creates a standalone Data Fabric cluster (single node if is_mapr=false, 5 nodes if is_mapr=true) and configures it with AD authentication, Spark, Airflow and Hive.\n\nAt any stage if script fails or if you wish to update the environment, you can restart the process for a specific step;\n\n- `./01-init.sh aws|azure|dc`\n- `./02-apply.sh aws|azure|dc`\n- `./03-install.sh aws|azure|dc`\n- `./04-configure.sh aws|azure|dc`\n\nDeployed resources will be available in /app/server/ansible/inventory.ini file.\n\n- All access to the public cloud environment is possible only through the gateway host.\n\n- Use `ssh centos@10.1.0.xx` to access hosts within the container, using their internal IP address (~/.ssh/config file is already set up for jump host via gateway)\n\n- You can copy \"./generated/controller.prv_key\" and \"~/.ssh/config\" to your workstation to access the deployed nodes directly\n\n- You can access created Kubernetes cluster with kubectl as K8s Admin context is already configured for ~/.kube/config (and you can copy that file from container to your workstation to connect to the K8s cluster directly)\n\n- Copy and install \"./generated/\\*/minica.pem\" into your browser to prevent SSL certificate errors\n\n## Reference\n\n### Utilities used in the container\n\n- AWS CLI - Download from [AWS](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)\n- Azure-CLI - Download from [Azure](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli)\n- Terraform - Download from [Terraform](https://www.terraform.io/downloads.html)\n- Ansible - Install from [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) or simply via pip (sudo pip3 install ansible)\n- python3 (apt/yum/brew install python3)\n- jq (apt/yum/brew install jq)\n- hpecp (pip3 install hpecp)\n- kubectl from [K8s](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)\n- minica (apt/yum/brew install minica)\n\n### Scripts\n\nFollowing scripts are mainly used to automate the whole process of creating the resources via Terraform (Thanks to Mr Snow) and then trigger the Ansible Playbooks (Thanks to Mr Derichsweiler).\n\n- 00-run_all.sh: Runs all scripts at once (unattended install)\n\n- 01-init.sh: Initialize Terraform, create SSH keys \u0026 certificates\n\n- 02-apply.sh: Runs `terraform apply` to deploy resources\n\n- 03-install.sh: Run Ansible scripts to install ECP\n\n- 04-configure.sh: Run Ansible scripts to configure ECP for demo\n\n- 99-destroy.sh: Destroy all created resources (**DANGER**: All resources will be destroyed, except the generated keys and certificates)\n\n### Ansible Scripts\n\nAnsible is used extensively to set up and configure the resources that are deployed via Terraform.\n\n- prepare_centos: Updates packages and requirements for ECP installation\n\n- install_falco: Updates kernel and install falco service\n\n- install_ecp: Initial installation and setup for ECP\n\n- import_hosts: Collects node information and update them as ECP worker nodes\n\n- create_k8s: Installs Kubernetes Cluster (if MLOps is not selected)\n\n- create_picasso: Installs Kubernetes Cluster and Picasso (Data Fabric on Kubernetes)\n\n- configure_picasso: Enables Picasso (Data Fabric on Kubernetes) for all tenants\n\n- configure_mlops: Configures MLOps tenant and life-cycle tools (Kubeflow, Minio, Jupyter NB etc)\n\n- and many more...\n\n## Notes\n\nDeployment defaults to EU-WEST-2 (EU - London) region on AWS, UK South (EU - London) region on Azure.\n\nPlease use following format to choose your region on AWS (config.json);\n\n```shell\n\"us-east-1\"      // N.Virginia\n\"us-east-2\"      // Ohio\n\"us-west-1\"      // N.California\n\"us-west-2\"      // Oregon\n\"ap-southeast-1\" // Singapore\n\"eu-central-1\"   // Frankfurt\n\"eu-west-1\"      // Ireland\n\"eu-west-2\"      // London\n\"eu-west-3\"      // Paris\n\"eu-north-1\"     // Stockholm\n\"ca-central-1\"   // Montréal, Québec\n```\n\nThis format should be used to select a region on Azure;\n\n```shell\n\"eastus\"\n\"eastus2\"\n\"centralus\"\n\"westus\"\n\"westus2\"\n\"canadacentral\"\n\"canadaeast\"\n\"northeurope\"\n\"westeurope\"\n\"ukwest\"\n\"uksouth\"\n\"francecentral\"\n\"germanynorth\"\n\"centralindia\"\n\"japaneast\"\n\"australiacentral\"\n\"uaenorth\"\n\"southafricawest\"\n```\n\n\\*\\* Not all regions are tested, please provide feedback if you have an issue with a region.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhewlettpackard%2Fezdemo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhewlettpackard%2Fezdemo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhewlettpackard%2Fezdemo/lists"}