{"id":16899570,"url":"https://github.com/0xfelix/gitops-demo","last_synced_at":"2025-04-12T08:34:15.874Z","repository":{"id":118937293,"uuid":"557895174","full_name":"0xFelix/gitops-demo","owner":"0xFelix","description":"Managing VirtualMachines with GitOps","archived":false,"fork":false,"pushed_at":"2023-04-27T14:08:56.000Z","size":937,"stargazers_count":20,"open_issues_count":0,"forks_count":26,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-26T03:41:41.638Z","etag":null,"topics":["gitops","kubernetes","openshift"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/0xFelix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-26T14:02:23.000Z","updated_at":"2025-02-12T06:29:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"c2444792-8378-49b3-9da0-628f07aa54fb","html_url":"https://github.com/0xFelix/gitops-demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xFelix%2Fgitops-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xFelix%2Fgitops-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xFelix%2Fgitops-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xFelix%2Fgitops-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/0xFelix","download_url":"https://codeload.github.com/0xFelix/gitops-demo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248540766,"owners_count":21121413,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gitops","kubernetes","openshift"],"created_at":"2024-10-13T17:49:31.205Z","updated_at":"2025-04-12T08:34:15.861Z","avatar_url":"https://github.com/0xFelix.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Managing VirtualMachines with GitOps\n\nThis repository contains instructions and files for a demonstration\ndeployment that shows how a CI/CD workflow of [`VirtualMachines`](https://kubevirt.io/user-guide/)\nacross multiple clusters could look like. It works with Open Cluster Management,\nArgoCD and KubeVirt or Red Hat Advanced Cluster Management, OpenShift GitOps \nand OpenShift Virtualization.\n\n## What is Open Cluster Management or Red Hat Advanced Cluster Management?\n\nOpen Cluster Management (OCM) or Red Hat Advanced Cluster Management (ACM)\nsimplifies the management of multiple clusters by offering end-to-end\nmanagement, visibility and control of the whole cluster and application life\ncycle. It acts as a central point for keeping an inventory of all your clusters\nand applications and enables multi-cluster and multi-cloud scenarios, such as\ndeploying the same application across clusters in different regions, possibly on\nseveral cloud providers. It uses a hub and spoke architecture and allows the\ntargeted distribution of Kubernetes manifests across clusters.\n\n### What are hub and managed clusters?\n\nThe hub cluster is the cluster on which OCM/ACM is running on. It\nacts as an inventory and carries out all management actions. It is usually not\nrunning any actual workloads (though still possible), these run on managed\nclusters. Managed clusters are kept in the inventory of the hub cluster.\nExisting clusters can be added to the inventory and on ACM they can also be\ncreated directly. The terms Open Cluster Management and Advanced Cluster\nManagement might be used interchangeably in the following sections. For more\ninformation have a look at the [OCM documentation](https://open-cluster-management.io/concepts/architecture/#overview).\n\n## What is the GitOps way and what is ArgoCD or OpenShift GitOps?\n\nThe GitOps way uses Git repositories as a single source of truth to deliver\ninfrastructure as code. Automation is employed to keep the desired and the live\nstate of clusters in sync at all times. This means any change to a repository\nis automatically applied to one or more clusters while changes to a cluster will\nbe automatically reverted to the state described in the single source of truth.\n\nArgoCD or Red Hat OpenShift GitOps enables declarative GitOps workflows and\nallows to deploy applications on-demand. It monitors the live state of clusters\nagainst the desired state in a Git repository and keeps them in sync. The terms\nArgoCD and OpenShift GitOps might be used interchangeably in the following\nsections. For more information have a look at\nthe [ArgoCD documentation](https://argo-cd.readthedocs.io/en/stable/#what-is-argo-cd).\n\n### A quick primer about Applications and ApplicationSets\n\nThe ArgoCD `Application` is a [`CustomResourceDefinition` (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/),\nwhich essentially describes a source of manifests and a target cluster to apply\nthe manifests to. Besides that, options like automatic creation of namespaces\nor the automatic revert of changes can be configured.\n\nThe ArgoCD `ApplicationSet` is a CRD building on ArgoCD `Applications`,\ntargeted to deploy and manage `Applications` across multiple clusters while\nusing the same manifest or declaration. It is possible to deploy multiple\n`ApplicationSets` which are contained in one monorepo. By using generators\nit is possible to dynamically select a subset of clusters available to ArgoCD to\ndeploy resources to.\n\nIn this demo we are going to use `ApplicationSets` to deploy KubeVirt or\nOpenShift Virtualization and `VirtualMachines` to multiple clusters while using\nthe same declaration of resources for all clusters.\n\nFor more information on `ApplicationSets` see the [documentation](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset).\n\n## Requirements for the setup\n\nThe following requirements need to be satisfied to build the setup described\nin this demo yourself:\n\n- A Git repository accessible by the hub cluster\n- One Kubernetes or OpenShift cluster acting as hub cluster\n    - Needs to be publicly accessible or at least accessible by the managed\n      clusters\n- One or more Kubernetes or OpenShift clusters acting as managed clusters\n    - Can be in private networks\n    - Virtualization has to be available\n    - Nested virtualization is fine for demonstration purposes\n\n### Repository preparation\n\nYou need to clone this repository to somewhere where you are able to make\nchanges to it (i.e. forking it on GitHub). Then open a terminal on your machine,\ncheck out the repository locally and change your working directory into the\ncloned repository.\n\nThe `ApplicationSets` in this repository use the URL of this repository as\n`repoURL`. To be able to make changes to your `ApplicationSets`, you need to\nadjust the `repoURL` to the URL of your own repository. If you do this later\ndo not forget to update any existing `ApplicationSets` on your hub cluster.\n\n## Kubernetes specifics\n\nThe following sections will cover the Kubernetes specific part of this demo.\n\n### Installing Open Cluster Management on the hub cluster\n\nSee the [OCM documentation](https://open-cluster-management.io/getting-started/installation/start-the-control-plane/)\non how to install Open Cluster Management.\n\nMake sure to install [multicloud-integrations](https://github.com/open-cluster-management-io/multicloud-integrations)\nfor integration with ArgoCD too.\n\n### Adding managed clusters to OCM on the hub cluster\n\nSee the [OCM documentation](https://open-cluster-management.io/getting-started/installation/register-a-cluster/)\non how to add managed clusters to OCM.\n\n### Organizing managed clusters in a set on OCM\n\nManaged clusters can be grouped into `ManagedClusterSets`. These sets can be\nbound to namespaces with a `ManagedClusterSetBinding` to make managed clusters\navailable in the bound namespaces.\n\nTo add managed clusters to a new set see the [OCM documentation](https://open-cluster-management.io/concepts/managedclusterset/).\nMake sure to name your set `managed` for compatibility with the manifests in\nthis repository.\n\nNow we have a `ManagedClusterSet` that can be used to make the managed clusters\navailable to ArgoCD.\n\n### Installing ArgoCD on the hub cluster\n\nSee the [ArgoCD documentation](https://argo-cd.readthedocs.io/en/stable/getting_started/)\non how to install ArgoCD.\n\n## OpenShift specifics\n\nThe following sections will cover the OpenShift specific part of this demo.\n\n### Installing Advanced Cluster Management on the hub cluster\n\n![ACM Install in OperatorHub](pictures/01.png)\n\n1. Login as cluster administrator on the UI of the hub cluster\n2. Open the `Administrator` view if it is not already selected\n2. In the menu click on `Operators` and open `OperatorHub`\n3. In the search type `Advanced Cluster Management for Kubernetes` and click\n   on it in the results\n4. Click on `Install`, keep defaults and click on `Install` again\n5. Wait until `MultiClusterHub` can be created and create it\n6. Wait until the created `MultiClusterHub` is ready (`Operators` --\u003e\n   `Installed Operators` --\u003e see status of ACM)\n\n### Adding managed clusters to ACM on the hub cluster\n\nManaged clusters can be added to ACM in two ways:\n\n1. Create a new cluster with ACM\n2. Add an existing cluster to ACM\n\n\u003e Note: For the sake of simplicity we will let ACM create the managed clusters\n\u003e in this blog post on a public cloud provider. Please note that nested\n\u003e virtualization is not supported in production deployments.\n\nTo create one or more managed clusters follow these steps:\n\n1. Login as cluster administrator on the UI of the hub cluster\n2. At the top of the menu select `All Clusters` (`local-cluster` should be\n   selected initially)\n3. Add credentials for you cloud provider by clicking on `Credentials` in\n   the menu and then clicking on `Add credentials`\n4. Click on `Infrastructure` and then on `Clusters` in the menu\n5. Click `Create cluster`, select your cloud provider and complete the\n   wizard (use the default cluster set for now)\n\n\u003e Note: When using Azure as cloud provider select instance type\n\u003e `Standard_D8s_v3` for the control plane and `Standard_D4s_v3` for the worker\n\u003e nodes, resources might become to tight to run virtual machines on the\n\u003e cluster otherwise.\n\n### Organizing managed clusters in a set on ACM\n\nManaged clusters can be grouped into `ManagedClusterSets`. These sets can be\nbound to namespaces with a `ManagedClusterSetBinding` to make managed clusters\navailable in the bound namespaces.\n\nTo add managed clusters to a new set follow these steps:\n\n1. Login as cluster administrator on the UI of the hub cluster\n2. At the top of the menu select `All Clusters` (`local-cluster` should be\n   selected initially)\n4. Click on `Infrastructure` and then on `Clusters` in the menu\n3. Click on `Cluster sets` and then on `Create cluster set`\n4. Enter `managed` as name for the new set and click on `Create`\n5. Click on `Managed resource assignments`\n6. Select all clusters you want to add, click on `Review` and then on `Save`\n\nNow we have a `ManagedClusterSet` that can be used to make the managed clusters\navailable to ArgoCD.\n\n![Managed cluster list](pictures/05.png)\n\nIf done correctly, the cluster list of the created `ManagedClusterSet` in ACM\nshould look like in the screenshot above.\n\n### Installing OpenShift GitOps on the hub cluster\n\n![GitOps Install in OperatorHub](pictures/03.png)\n\n1. Login as cluster administrator on the UI of the hub cluster\n2. Open the `Administrator` view if it is not already selected\n2. In the menu click on `Operators` and open `OperatorHub`\n3. In the search type `Red Hat OpenShift GitOps` and click\n   on it in the results\n4. Click on `Install`, keep defaults and click on `Install` again\n6. Wait until OpenShift GitOps is ready (`Operators` --\u003e\n   `Installed Operators` --\u003e see status of OpenShift GitOps)\n\nIf installed correctly, the list of installed operators on your cluster should\nlook like in the following screenshot:\n\n![Installed Operators](pictures/04.png)\n\n#### Accessing the OpenShift GitOps web UI\n\nThe OpenShift GitOps web UI is exposed with a `Route`. To get the\nexact URL of the `Route` follow these steps:\n\n1. Login as cluster administrator on the UI of the hub cluster\n2. Open the `Administrator` view if it is not already selected\n3. In the menu click on `Networking` and open `Routes`\n4. In the `Projects` drop down select `openshift-gitops`\n   (enable `Show default projects` if not visible)\n5. There will be a `Route` called `openshift-gitops-server`, the location of\n   this `Route` is the URL to the GitOps UI\n6. You can log in to the GitOps UI with your OpenShift credentials\n\nAlternatively you can use the command line to get the URL to the GitOps UI with\nthe following command:\n\n```shell\noc get route -n openshift-gitops openshift-gitops-server -o jsonpath='{.spec.host}'\n```\n\n## Making a set of managed clusters available to ArgoCD or OpenShift GitOps\n\nTo make a set of managed clusters available to ArgoCD or OpenShift GitOps, a\ntight integration between OCM and ArgoCD or ACM and GitOps exists. The\nintegration is controlled with the `GitOpsCluster` CRD.\n\nFollow these steps to make the managed clusters available to ArgoCD or GitOps:\n\n1. Make sure you are logged in to your cluster on the CLI\n2. Run the copied command in your terminal\n3. Create a `ManagedClusterSetBinding` in the `argocd` or `openshift-gitops`\n   namespace to make the `ManagedClusterSet` available in this namespace\n    - See file [managedclustersetbinding.yaml](acm-gitops-integration/managedclustersetbinding.yaml)\n    - On Kubernetes run `kubectl create -n argocd -f acm-gitops-integration/managedclustersetbinding.yaml`\n    - On OpenShift run `oc create -n openshift-gitops -f acm-gitops-integration/managedclustersetbinding.yaml`\n4. Create a `Placement` to let OCM/ACM decide which clusters should be made\n   available to GitOps\n    - See file [placement.yaml](acm-gitops-integration/placement.yaml)\n    - On Kubernetes run `kubectl create -n argocd -f acm-gitops-integration/placement.yaml`\n    - On OpenShift run `oc create -n openshift-gitops -f acm-gitops-integration/placement.yaml`\n    - For the sake of simplicity this will select the whole`ManagedClusterSet`,\n      but advanced use cases are possible\n5. Create a `GitOpsCluster` to finally make the selected clusters available to\n   GitOps on the hub cluster\n    - For Kubernetes see file [k8s-gitopscluster.yaml](acm-gitops-integration/k8s-gitopscluster.yaml)\n    - On Kubernetes run `kubectl create -f acm-gitops-integration/k8s-gitopscluster.yaml`\n    - For OpenShift see file [gitopscluster.yaml](acm-gitops-integration/gitopscluster.yaml)\n    - On OpenShift run `oc create -f acm-gitops-integration/gitopscluster.yaml`\n\n![ArgoCD cluster list](pictures/06.png)\n\nIn this screenshot you can see that the managed clusters were made available to\nArgoCD successfully. This view can be opened by going to ArgoCD's settings and\nopening the `Clusters` menu. Until an `Application` is deployed to the cluster\nits connection status may still be `Unknown`.\n\n## Assigning clusters to environments\n\nIn our setup we assign managed clusters to specific environments by setting a\nlabel on them. Ideally it would be possible to assign them from OCM/ACM, but\nfor the time being this still has to be done in ArgoCD. In an upcoming OCM/ACM\nrelease it will be possible to carry over labels set in OCM/ACM to ArgoCD.\n\nSee [this PR](https://github.com/open-cluster-management-io/multicloud-integrations/pull/51)\nfor details.\n\nIn this post we will work with the `dev` and the `prod` environments. Add your\nmanaged clusters to the environments by following these steps:\n\nOpen ArgoCD's settings and open the `Clusters` menu. Then click on the three\ndots on the right side of a cluster to edit it. After editing the cluster do not\nforget to save your changes.\n\n### Assigning clusters to the dev environment\n\n![ArgoCD cluster list](pictures/07.png)\n\nOne or more of the clusters should belong to the `dev` environment. This is\nachieved by setting the `env` label to the value `dev` on the managed cluster.\n\n### Assigning clusters to the prod environment\n\n![ArgoCD cluster list](pictures/08.png)\n\nOne or more of the clusters should belong to the `prod` environment. This is\nachieved by setting the `env` label to the value `prod` on the managed cluster.\n\n## Deploying KubeVirt or OpenShift Virtualization to the managed clusters\n\n![OpenShift Virtualization Application in ArgoCD](pictures/09.png)\n\nTo deploy KubeVirt or OpenShift Virtualization to the managed clusters with\nthe help of an `ApplicationSet` run the following command from your cloned\nrepository (See [Repository preparation](#Repository-preparation)):\n\nOn Kubernetes run the following commands:\n\n```shell\nkubectl create -f applicationsets/cdi/applicationset-cdi.yaml\nkubectl create -f applicationsets/kubevirt/applicationset-kubevirt.yaml\n```\n\nOn OpenShift run the following command:\n\n```shell\noc create -f applicationsets/virtualization/applicationset-virtualization.yaml\n```\n\nThis will create an `Application` for each managed cluster that deploys\nKubeVirt or OpenShift Virtualization with its default settings. The\n`Application` will ensure that the appropriate namespaces exists, and it\nwill automatically apply any changes to this repository or undo changes\nwhich are not in this repository. Sync waves are used to ensure that\nresources are created in the right order.\n\n### Order of resource creation on Kubernetes\n\n1. `Namespace`, `CustomResourceDefinition`, `ClusterRole`\n2. `ServiceAccount`, `Role`, `ConfigMap`\n3. `ClusterRoleBinding`, `RoleBinding`\n4. `Deployment`\n5. `CDI` CR, `KubeVirt` CR\n\n### Order of resource creation on OpenShift\n\n1. `OperatorGroup`\n2. `Subscription`\n3. `HyperConverged`\n\nBecause the `HyperConverged` CRD is unknown to ArgoCD, the sync option\n`SkipDryRunOnMissingResource=true` is set to allow ArgoCD to create a CR\nwithout knowing its CRD.\n\n### Forcing a specific version of OpenShift Virtualization\n\nThere is only one update channel for OpenShift Virtualization (called\n`stable`) so the appropriate version for the managed cluster is selected\nautomatically.\n\nTo force a specific version from the channel do the following:\n\n1. Make sure that [`grpcurl`](https://github.com/fullstorydev/grpcurl) and\n   [`jq`](https://stedolan.github.io/jq/) are available on your machine\n2. Extract the available [`CSV`](https://olm.operatorframework.io/docs/concepts/crds/clusterserviceversion/)\n   versions from the Operator registry\n    1. Login to the command line of the managed cluster\n    2. Run\n       `oc port-forward service/redhat-operators -n openshift-marketplace 50051:50051`\n    3. In a separate terminal run\n       `grpcurl -plaintext localhost:50051 api.Registry/ListBundles | jq 'select(.csvName | match (\"kubevirt-hyperconverged-operator\")) | .version'`\n3. Set the following fields in the `Subscription` spec\n    - `installPlanApproval`: `Manual`\n    - `startingCSV`: Your desired and available CSV version\n\nThis technique can for example be used to control the upgrade process of\nOpenShift Virtualization in a declarative way.\n\n### Health state of an Application\n\nIn ArgoCD's UI you can follow the synchronization status of the newly created\n`Application` for each cluster. Eventually every `Application` will reach the\nhealthy and synced status like in the following screenshot.\n\n![OpenShift Virtualization Application becoming ready](pictures/10.png)\n\nTo see what is actually deployed have a look into the following directories:\n\nOn Kubernetes look into `applicationsets/cdi/manifests` and\n`applicatioinsets/kubevirt/manifests`.\n\nOn OpenShift look into `applicationsets/virtualization/manifests`.\n\n## Deploying a VirtualMachine to the managed clusters\n\nTo deploy a Fedora `VirtualMachine` on all managed clusters with the help of\nan `ApplicationSet` run the following command from your cloned repository\n(See [Repository preparation](#Repository-preparation)):\n\nOn Kubernetes run the following command:\n\n```shell\nkubectl create -n argocd -f applicationsets/demo-vm/applicationset-demo-vm.yaml\n```\n\nOn OpenShift run the following command:\n\n```shell\noc create -n openshift-gitops -f applicationsets/demo-vm/applicationset-demo-vm.yaml\n```\n\nThis will create an `Application` for each managed cluster that deploys a\nsimple `VirtualMachine` on each cluster. It uses the Fedora `DataSource`\navailable on the OpenShift cluster by default to boot a Fedora cloud image.\n\nOn Kubernetes please create the DataSource with the following command:\n\n```shell\nkubectl create -f acm-gitops-integration/k8s-datasource.yaml\n```\n\n### Health state of the `Application`\n\nNotice how the health state of the created `Application` is `Suspended`. This is\nbecause the created `VirtualMachine` is still in stopped state.\n\n![New demo VM Applications in suspended state](pictures/11.png)\n\n## Applying customizations to environments\n\nInstead of using plain manifests this `ApplicationSet` is using `Kustomize`.\nThis allows to apply customizations to an `Application` depending on the\nenvironment a managed cluster belongs to. In this post it is achieved by using\nthe `metadata.labels.env` value to choose the right `Kustomize` overlay.\n\nThe `dev` overlay prefixes names of created resources with `dev-`, while the\n`prod` overlay prefixes names with `prod-`. Furthermore, the created\n`VirtualMachines` get more or less memory assigned depending on the environment.\nThese are only simple customizations, but the possibilities are endless!\n\nTo see what is actually deployed have a look into the following directory:\n`applicationsets/demo-vm/kustomize`.\n\n### Quick summary\n\nHere is a quick summary of the required steps:\n\n1. Choose to modify all environments (`base`) or a single environment (eg.\n   `dev` or `prod`)\n2. To start the `VirtualMachine` in all environments edit\n   `applicationsets/demo-vm/kustomize/base/virtualmachine.yaml`\n3. Set `spec.running` to `true`\n4. Commit and push the change to your repository\n5. Refresh ArgoCD to pick up the change\n\nThe following sections will explain the steps in more detail.\n\n### How to start or stop a VirtualMachine\n\nFirst let us have a closer look at the `Application` of the stopped\n`VirtualMachine`. Notice the `Suspended` health state. Also notice the `dev-`\nprefix of the created `VirtualMachine`. It was created on a cluster belonging\nto the `dev` environment.\n\n![Detail view of suspended demo VM Application](pictures/12.png)\n\nTo start or stop a `VirtualMachine` you need to edit the `spec.running` field of\na `VirtualMachine` and set it to the corresponding value (`false` or `true`).\nYou can do this in the `applicationsets/demo-vm/kustomize` directory.\n\n### Graceful shutdown of `VirtualMachines`\n\nIf the `VirtualMachine` has an appropriate termination grace period\n(`spec.template.spec.terminationGracePeriodSeconds`), by setting this value to\n`false` the `VirtualMachine` will be gracefully shut down. When setting the\ntimeout grace period to 0 seconds, the `VirtualMachine` is stopped immediately\nhowever.\n\n### Applying changes to specific environments\n\nWhen modifying the `VirtualMachine` you can choose to either modify the base or\na specific overlay of `Kustomize`. This allows to start or stop the\n`VirtualMachine` in every environment or just in a specific one. In this example\nthe `VirtualMachine` was started in every environment by modifying the\n`Kustomize` base.\n\n### Applying the change with ArgoCD\n\nTo apply new changes with ArgoCD you need to commit and push changes to the Git\nrepository containing your `Application`. To start or stop a `VirtualMachine`\nyou have to update the manifest and commit and push to your repository. In the\nArgoCD UI select the `Application` of the `VirtualMachine` and click `Refresh`\nto apply the change immediately. Otherwise, it will take some time until ArgoCD\nscans the repository and picks up the change.\n\n### Observing the change\n\nAfter ArgoCD picked up the change it will sync it to the `VirtualMachine` as\nvisible by the `Progressing` health state in the following screenshot:\n\n![Detail view of progressing demo VM Application](pictures/13.png)\n\nEventually the `VirtualMachine` will be running and healthy:\n\n![Detail view of healthy demo VM Application](pictures/14.png)\n\n## Advanced usage of OCM/ACM Placements with ArgoCD or OpenShift GitOps\n\nFor the sake of simplicity the `Placement` created in this demo selects the\nwhole `ManagedClusterSet`, but more advanced use cases are possible.\n\nOCM/ACM can dynamically select a subset of clusters from the `ManagedClusterSet`\nwhile following a defined set of criteria. This for example allows to schedule\n`VirtualMachines` on clusters with the most resources available at the time of\nthe placement decision.\n\nFor more on this topic see [Using the Open Cluster Management Placement for Multicluster Scheduling](https://cloud.redhat.com/blog/using-the-open-cluster-management-placement-for-multicluster-scheduling).\n\n## Alternative way of deploying OpenShift Virtualization to managed clusters\n\nAn [OCM/ACM add-on](https://open-cluster-management.io/developer-guides/addon/)\nthat deploys OpenShift Virtualization to managed clusters was implemented\nfor evaluation purposes. The add-on is fully functional and can deploy\nOpenShift Virtualization to all managed clusters that have a specific label set.\n\nAlthough the add-on the serves the purpose of deploying OpenShift\nVirtualization, it was found to be unnecessary complex when OpenShift GitOps\nis available too. The add-on is only deploying a small set of static manifests\nwhich can be deployed by GitOps too.\n\nIn contrast to its use stand the additional maintenance burden and resource\nusage of another container running on the cluster. Therefore, it was decided\nto not follow this path any further.\n\nThe add-on can be found [here](https://github.com/0xFelix/kubevirt-addon-manager).\n\n## Integration with Ansible AWX or Ansible Automation Controller\n\nOCM/ACM can be integrated with Ansible AWX or Ansible Automation Controller to\ntrigger Playbook runs after certain events. To make use of `VirtualMachines`\nin Ansible a dynamic inventory is needed, which makes the `VirtualMachines`\navailable and accessible to Ansible.\n\nThere is already a collection of [KubeVirt modules](https://github.com/kubevirt/kubevirt-ansible/)\nfor Ansible, this collection however is deprecated and no longer working.\n\nFor evaluation purposes [a fork](https://github.com/0xFelix/kubernetes.kubevirt)\nwas created. This fork provides limited functionality but shows\nthat this type of integration is still possible. A demo of this Ansible\ncollection can be found [here](https://github.com/0xFelix/kubevirt-inventory-demo).\n\n## Future use cases\n\nA future use case is to pre-configure a `VirtualMachine` on a cluster and then\nexport it into a blob format which can be stored somewhere where it can be\naccessed from other clusters. The blob could then be imported into other\nclusters to allow deployment of replicas of a pre-configured `VirtualMachine`\nacross multiple clusters.\n\nA possible blob format for this kind of export/import feature could be\n[ContainerDisks](https://kubevirt.io/user-guide/virtual_machines/disks_and_volumes/#containerdisk),\nwhich are already supported by KubeVirt or OpenShift Virtualization.\n\nTo show this is already possible with the current `ContainerDisk` implementation\na Proof-Of-Concept was created. The PoC can be found [here](https://github.com/0xFelix/kubevirt/tree/virtctl-exportcd).\n\n## Summary and outlook\n\n![View of all healthy Applications](pictures/15.png)\n\nIn this demo we set up a hub cluster and two clusters managed by OCM/ACM to\ndeploy applications to from a centralized management point. As example\napplications we deployed KubeVirt or OpenShift Virtualization with simple\nmanifests and a virtual machine with manifests customized by `Kustomize`. We\nlearned how to apply customizations to specific environments and how we can\nstart and stop virtual machines in a declarative way. All of this was\naccomplished the GitOps way by using a Git repository as a single source of\ntruth.\n\nThis is of course only the tip of the iceberg, building on this setup allows\nyou to customize your `ApplicationSets` for different environments like\ndevelopment, staging and production or to schedule your applications based on\ncustom criteria (e.g. available resources) with advanced placements rules.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0xfelix%2Fgitops-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F0xfelix%2Fgitops-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0xfelix%2Fgitops-demo/lists"}