{"id":16179218,"url":"https://github.com/jonashackt/crossplane-argocd","last_synced_at":"2026-01-20T18:56:30.891Z","repository":{"id":213373599,"uuid":"733925268","full_name":"jonashackt/crossplane-argocd","owner":"jonashackt","description":"Example project showing how to use Crossplane together with ArgoCD","archived":false,"fork":false,"pushed_at":"2024-04-11T20:39:19.000Z","size":14749,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-04-12T06:21:42.655Z","etag":null,"topics":["app-of-apps","argocd","argocd-apps","aws","crossplane","crossplane-provider-aws","eks-cluster","kubernetes"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonashackt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-12-20T12:54:53.000Z","updated_at":"2024-04-15T14:39:40.473Z","dependencies_parsed_at":"2023-12-20T17:25:20.609Z","dependency_job_id":"1a072c8a-4d8e-4289-9557-bcea4a954207","html_url":"https://github.com/jonashackt/crossplane-argocd","commit_stats":null,"previous_names":["jonashackt/crossplane-argocd"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fcrossplane-argocd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fcrossplane-argocd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fcrossplane-argocd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fcrossplane-argocd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonashackt","download_url":"https://codeload.github.com/jonashackt/crossplane-argocd/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247712112,"owners_count":20983640,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["app-of-apps","argocd","argocd-apps","aws","crossplane","crossplane-provider-aws","eks-cluster","kubernetes"],"created_at":"2024-10-10T05:26:02.543Z","updated_at":"2026-01-20T18:56:30.848Z","avatar_url":"https://github.com/jonashackt.png","language":"Shell","funding_links":[],"categories":["kubernetes"],"sub_categories":[],"readme":"# crossplane-argocd\n[![Crossplane plain ArgoCD](https://github.com/jonashackt/crossplane-argocd/workflows/crossplane-argocd/badge.svg)](https://github.com/jonashackt/crossplane-argocd/actions/workflows/crossplane-argocd.yml)\n[![Crossplane, ArgoCD \u0026 External Secrets Operator (+Doppler)](https://github.com/jonashackt/crossplane-argocd/workflows/crossplane-argocd-external-secrets/badge.svg)](https://github.com/jonashackt/crossplane-argocd/actions/workflows/crossplane-argocd-external-secrets.yml)\n![crossplane-version](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2Fjonashackt%2Fcrossplane-argocd%2Fmain%2Fcrossplane%2FChart.yaml\u0026query=%24.dependencies%5B%3A1%5D.version\u0026label=crossplane\u0026color=blue)\n![argocd-version](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2Fjonashackt%2Fcrossplane-argocd%2Fmain%2Fargocd%2Finstall%2Fkustomization.yaml\u0026query=%24.resources%5B%3A1%5D\u0026label=argocd\u0026color=rgb(236%2C%20110%2C%2076))\n![provider-aws-ec2](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2Fjonashackt%2Fcrossplane-argocd%2Fmain%2Fupbound%2Fprovider-aws%2Fprovider%2Fupbound-provider-aws-ec2.yaml\u0026query=%24.spec.package\u0026label=provider-aws-ec2\u0026color=rgb(109%2C%20100%2C%20245))\n![provider-aws-eks](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2Fjonashackt%2Fcrossplane-argocd%2Fmain%2Fupbound%2Fprovider-aws%2Fprovider%2Fupbound-provider-aws-eks.yaml\u0026query=%24.spec.package\u0026label=provider-aws-eks\u0026color=rgb(109%2C%20100%2C%20245))\n![provider-aws-iam](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2Fjonashackt%2Fcrossplane-argocd%2Fmain%2Fupbound%2Fprovider-aws%2Fprovider%2Fupbound-provider-aws-iam.yaml\u0026query=%24.spec.package\u0026label=provider-aws-iam\u0026color=rgb(109%2C%20100%2C%20245))\n![provider-aws-s3](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2Fjonashackt%2Fcrossplane-argocd%2Fmain%2Fupbound%2Fprovider-aws%2Fprovider%2Fupbound-provider-aws-s3.yaml\u0026query=%24.spec.package\u0026label=provider-aws-s3\u0026color=rgb(109%2C%20100%2C%20245))\n[![License](http://img.shields.io/:license-mit-blue.svg)](https://github.com/jonashackt/crossplane-argocd/blob/master/LICENSE)\n[![renovateenabled](https://img.shields.io/badge/renovate-enabled-yellow)](https://renovatebot.com)\n\nExample project showing how to use the crossplane together with ArgoCD\n\n\u003e This project is based on the crossplane only repository https://github.com/jonashackt/crossplane-aws-azure, where the basics about crossplane.io are explained in detail - incl. how to provision to AWS and Azure.\n\n__The idea is \"simple\": Why not treat infrastructure deployments/provisioning the same way as application deployments?!__ An ideal combination would be crossplane as control plane framework, which manages infrastructure through the Kubernetes api together with ArgoCD as [GitOps](https://www.gitops.tech/) framework to have everything in sync with our version control system.\n\n\n### TLDR: Steps from 0 to 100\n\nIf you don't want to read much text, do the following steps:\n\n```shell\n# fire up kind\nkind create cluster --image kindest/node:v1.32.0 --wait 5m --name crossplane-argocd\n\n# Install ArgoCD\nkubectl apply -k argocd/install\nkubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-server --namespace argocd --timeout=300s\n\n# Access ArgoUI\nkubectl port-forward -n argocd --address='0.0.0.0' service/argocd-server 8080:80\nkubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=\"{.data.password}\" | base64 -d; echo\n\n# Create Secret with Doppler Service Token\n# be sure to have exported the env var locally, e.g. via\n# export DOPPLER_SERVICE_TOKEN=\"dp.st.dev.dopplerservicetoken\"\nkubectl create secret generic doppler-token-auth-api --from-literal dopplerToken=\"$DOPPLER_SERVICE_TOKEN\"\n\n# Prepare Secret with ArgoCD API Token for Crossplane ArgoCD Provider (port forward can be run in subshell appending ' \u0026' + Ctrl-C and beeing deleted after running create-argocd-api-token-secret.sh via 'fg 1%' + Ctrl-C)\nkubectl port-forward -n argocd --address='0.0.0.0' service/argocd-server 8443:443\nbash create-argocd-api-token-secret.sh\n\n\n\n\n# Bootstrap Crossplane via ArgoCD\nkubectl apply -n argocd -f argocd/crossplane-eso-bootstrap.yaml \n\nkubectl get crd\n\n# Install Crossplane EKS APIs/Composition\nkubectl apply -f argocd/crossplane-apis/crossplane-apis.yaml\n\n# Create actual EKS cluster via Crossplane \u0026 register it in ArgoCD via argocd-provider\nkubectl apply -f argocd/infrastructure/aws-eks.yaml\ncrossplane beta trace kubernetesclusters.k8s.crossplane.jonashackt.io/deploy-target-eks -o wide\n\n# Optional: If you want, have a look onto the new cluster\nkubectl get secret eks-cluster-kubeconfig -o jsonpath='{.data.kubeconfig}' | base64 --decode \u003e ekskubeconfig\n# integrate the contents of `ekskubeconfig` into your `~/.kube/config` (better w/ VSCode!) \u0026 switch over to the new kube context\n\n# Run Application on EKS cluster using Argo\nkubectl apply -f argocd/applications/microservice-api-spring-boot.yaml\n```\n\nNow you should see both clusters (kind \u0026 EKS) running and the app beeing deployed:\n\n![](docs/kind-argo-crossplane-and-eks-fully-working.png)\n\n\n\n# Prerequisites: a management cluster for ArgoCD and crossplane\n\nFirst we need a simple management cluster for our ArgoCD and crossplane deployments. [As in the base project](https://github.com/jonashackt/crossplane-aws-azure) we simply use kind here:\n\nBe sure to have some packages installed. On a Mac:\n\n```shell\nbrew install kind helm kubectl kustomize argocd\n```\n\nOr on Arch/Manjaro:\n\n```shell\npamac install kind-bin helm kubectl-bin kustomize argocd\n```\n\n\nhttps://docs.crossplane.io/latest/cli/\n\nAlso we should install the crossplane CLI\n\n```shell\ncurl -sL \"https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh\" | sh\nsudo mv crossplane /usr/local/bin\n```\n\nNow the `kubectl crossplane --help` command should be ready to use.\n\n\nNow spin up a local kind cluster\n\n```shell\nkind create cluster --image kindest/node:v1.31.1 --wait 5m\n```\n\n\n\n# Pre-install preparations: Configure ArgoCD for Crossplane\n\nBefore even starting to install ArgoCD, we should be aware of [some needed configuration details](https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/) in order to let Argo run smootly with Crossplane.\n\nWe can ignore [the mentioned health status configuration](https://docs.crossplane.io/latest/guides/crossplane-with-argo-cd/#set-health-status) in the docs, since \n\n\u003e \"Some checks are supported by the community directly in Argo’s repository. For example the Provider from pkg.crossplane.io has already been declared which means there no further configuration needed.\"\n\nSo for now we should focus on the configuration of the annotation based resource tracking in ArgoCD and the exclusion of Crossplane generated `ProviderConfigUsage` CRDs.\n\n\n#### Configure annotation based resource tracking in ArgoCD\n\nAs [the docs state](https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/):\n\n\u003e \"There are different ways to configure how Argo CD tracks resources. With Crossplane, you need to configure Argo CD to use Annotation based resource tracking.\"\n\nYou may already used ArgoCD with resource tracking via the well-known label `app.kubernetes.io/instance`, which is the default resource tracking mode in Argo. But from ArgoCD 2.2 on [there are additional ways of tracking resources](https://argo-cd.readthedocs.io/en/stable/user-guide/resource_tracking/#additional-tracking-methods-via-an-annotation). One of them is the annotation based resource tracking. This has some advantages:\n\n\u003e \"The advantages of using the tracking id annotation is that there are no clashes any more with other Kubernetes tools and Argo CD is never confused about the owner of a resource. The annotation+label can also be used if you want other tools to understand resources managed by Argo CD.\"\n\nThe resource tracking method has to be configured inside the `argocd-cm` ConfigMap using the `application.resourceTrackingMethod` field:\n\n```yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: argocd-cm\ndata:\n  # Set Resource Tracking Method (see https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/#set-resource-tracking-method)\n  application.resourceTrackingMethod: annotation\n```\n\n\n### Exclude Crossplane generated ProviderConfigUsage CRDs\n\nThe second necessary configuration refers to [the exclusion of Crossplane generated `ProviderConfigUsage` CRDs](https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/#set-resource-exclusion):\n\n\u003e Crossplane providers generates a `ProviderConfigUsage` for each of the managed resource (MR) it handles. This resource enable representing the relationship between MR and a ProviderConfig so that the controller can use it as finalizer when a ProviderConfig is deleted. End-users of Crossplane are not expected to interact with this resource.\n\nWhat this means is that if we have a lot of Crossplane Resources that we work with like it is shown in the following image, the ArgoCD UI reactivity can be impacted:\n\n![](docs/crossplane-providerconfigusage-in-argo.png)\n\nAnd because these resources don't give us anymore insights, we can savely remove them as ArgoCD resources. Therefore we also configure this in the `argocd-cm` ConfigMap:\n\n```yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: argocd-cm\ndata:\n  ...\n  # Set Resource Exclusion (see https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/#set-resource-exclusion)\n  resource.exclusions: |\n    - apiGroups:\n      - \"*\"\n      kinds:\n      - ProviderConfigUsage      \n```\n\nWe will actually configure this while installing ArgoCD in a second. Because the question is: where exactly can we change parameters of the `argocd-cm` ConfigMap in ArgoCD?\n\n\n\n\n\n# Install ArgoCD into the management cluster\n\nThis question boils down to another question on a higher level: How do we install ArgoCD and change the ConfigMap in a flexible and GitOps-style way? Ideally also in a [renovatebot-enabled](https://github.com/renovatebot/renovate) fashion. And I already had that kind of question solved for me: Just [use Kustomize as described here](https://stackoverflow.com/a/71692892/4964553) and [also in the Argo docs](https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#manage-argo-cd-using-argo-cd).\n\nIn fact the ArgoCD team itself uses this approach to deploy their own ArgoCD instances. A live deployment [is available here](https://cd.apps.argoproj.io/) and the configuration used [can be found on GitHub](https://github.com/argoproj/argoproj-deployments/tree/master/argocd).\n\nUsing [Kustomize](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/) enables a great way of declaritively changing configuration in ConfigMaps, while using the default installation method (which [is this install.yaml](https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml)). And at the same time staying upgradable via Renovate. \n\nSo let's first create a directory `argocd/install` in the root of our repository. Therein we create a file called [`kustomization.yaml`](argocd/install/kustomization.yaml) with the following contents:\n\n```yaml\napiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n- github.com/argoproj/argo-cd//manifests/cluster-install?ref=v2.12.2\n- argocd-namespace.yaml\n\n## changes to config maps\npatches:\n- path: argocd-cm-patch.yaml\n\nnamespace: argocd\n```\n\nUnder the `resources` parameter you can see a link to a ArgoCD installation manifest, followed by the ArgoCD version tag. This is a great way of enabling [Renovate](https://github.com/renovatebot/renovate) to keep our setup up-to-date automatically.\n\nAs Kustomize has the ability to use patch files, we also create a file [`argocd-cm-patch.yaml`](argocd/install/argocd-cm-patch.yaml). Here we can configure the annotation based resource tracking mode and exclude the Crossplane generated ProviderConfigUsage CRDs from ArgoCD:\n\n```yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: argocd-cm\ndata:\n  # Set Resource Tracking Method (see https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/#set-resource-tracking-method)\n  application.resourceTrackingMethod: annotation\n  # Set Resource Exclusion (see https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/#set-resource-exclusion)\n  resource.exclusions: |\n    - apiGroups:\n      - \"*\"\n      kinds:\n      - ProviderConfigUsage\n```\n\nAdditionally to our ConfigMap patch we create another file [argocd-namespace.yaml](argocd/install/argocd-namespace.yaml), that will automatically create the namespace `argocd` for us:\n\n```yaml\napiVersion: v1\nkind: Namespace\nmetadata:\n  name: argocd\n```\n\nWith this simple manifest and it's integration into our [`kustomization.yaml`](argocd/install/kustomization.yaml), we don't need to explicitely run `kubectl create namespace argocd` anymore.\n\nNow we have everything prepared to install ArgoCD via Kustomize. Simply run a `kubectl apply -k` aimed to our previously created directory:\n\n```shell\nkubectl apply -k argocd/install\n```\n\n\n\n\n### Accessing ArgoCD GUI\n\nSince we're using ArgoCD, we should also be able to access it's fantastic UI in our browser. Therefore we first need to obtain the initial password for the `admin` user on the command line:\n\n```shell\nkubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=\"{.data.password}\" | base64 -d; echo\n```\n\nIn order to make the `argocd-server` available outside of our management cluster we have multiple options. One of the simplest might be a `port-forward`:\n\n```shell\nkubectl port-forward -n argocd --address='0.0.0.0' service/argocd-server 8080:80\n```\n\nNow we can access the ArgoCD UI inside your Browser at http://localhost:8080 using `admin` user and the obtained password.\n\n\n\n### Login ArgoCD CLI into our argocd-server installed in kind\n\nhttps://argo-cd.readthedocs.io/en/stable/getting_started/#4-login-using-the-cli\n\nIn order to be able to add applications to Argo, we should login our ArgoCD CLI into our `argocd-server` Pod installed in kind:\n\n```shell\nargocd login localhost:8080 --username admin --password $(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=\"{.data.password}\" | base64 -d; echo) --insecure\n```\n\nRemember to change the initial password in production environments!\n\n\n\n\n# Let ArgoCD install Crossplane\n\nIs it possible to already use the GitOps approach right from here on to install crossplane? Let's try it.\n\nAs already used from https://github.com/jonashackt/crossplane-aws-azure and explained in https://stackoverflow.com/a/71765472/4964553 we have a simple Helm chart, which is able to be managed by RenovateBot - and thus kept up-to-date. Our Chart lives in [`crossplane/Chart.yaml`](crossplane/Chart.yaml):\n\n```yaml\napiVersion: v2\ntype: application\nname: crossplane-argocd\nversion: 0.0.0 # unused\nappVersion: 0.0.0 # unused\ndependencies:\n  - name: crossplane\n    repository: https://charts.crossplane.io/stable\n    version: 1.16.0\n```\n\n__This Helm chart needs to be picked up by Argo in a declarative GitOps way (not through the UI).__\n\nBut as this is a non-standard Helm Chart, we need to define a `Secret` first [as the docs state](https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#helm-chart-repositories):\n\n\u003e \"Non standard Helm Chart repositories have to be registered explicitly. Each repository must have url, type and name fields.\"\n\nSo we first create [`crossplane-helm-secret.yaml`](argocd/crossplane-bootstrap/crossplane-helm-secret.yaml):\n\n```yaml\napiVersion: v1\nkind: Secret\nmetadata:\n  name: crossplane-helm-repo\n  namespace: argocd\n  labels:\n    argocd.argoproj.io/secret-type: repository\nstringData:\n  name: crossplane\n  url: https://charts.crossplane.io/stable\n  type: helm \n```\n\nWe need to apply it via:\n\n```shell\nkubectl apply -f argocd/crossplane-bootstrap/crossplane-helm-secret.yaml\n```\n\n\nNow telling ArgoCD where to find our simple Crossplane Helm Chart, we use Argo's `Application` manifest in [argocd/crossplane-bootstrap/crossplane.yaml](argocd/crossplane-bootstrap/crossplane.yaml):\n\n```yaml\n# The ArgoCD Application for crossplane core components themselves\n---\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: crossplane\n  namespace: argocd\n  finalizers:\n    - resources-finalizer.argocd.argoproj.io\nspec:\n  project: default\n  source:\n    repoURL: https://github.com/jonashackt/crossplane-argocd\n    targetRevision: HEAD\n    path: crossplane\n  destination:\n    server: https://kubernetes.default.svc\n    namespace: crossplane-system\n  syncPolicy:\n    automated:\n      prune: true    \n    syncOptions:\n    - CreateNamespace=true\n    retry:\n      limit: 1\n      backoff:\n        duration: 5s \n        factor: 2 \n        maxDuration: 1m\n```\n\nAs the docs state https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#crossplane-bootstrap\n\n\u003e \"Without the `resources-finalizer.argocd.argoproj.io finalizer`, deleting an application will not delete the resources it manages. To perform a cascading delete, you must add the finalizer. See [App Deletion](https://argo-cd.readthedocs.io/en/stable/user-guide/app_deletion/#about-the-deletion-finalizer).\"\n\nIn other words, if we would run `kubectl delete -n argocd -f argocd/crossplane-bootstrap/crossplane.yaml`, Crossplane wouldn't be undeployed as we may think. Only the ArgoCD `Application` would be deleted, but Crossplane Pods etc. would be still running.\n\nOur `Application` configures Crossplane core componentes to be automatically pruned https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/#automatic-pruning via `automated: prune: true`.\n\nWe also use `syncOptions: - CreateNamespace=true` here [to let Argo create the crossplane `crossplane-system` namespace for us automatically](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#create-namespace).\n\n\n\n\n```shell\nkubectl apply -n argocd -f argocd/crossplane-bootstrap/crossplane.yaml\n```\n\nNow ArgoCD deploys our core crossplane components for us :)\n\nJust have a look into Argo UI:\n\n![](docs/argocd-deploys-crossplane.png)\n\nWe can double check everything is there on the command line via:\n\n```shell\nkubectl get all -n crossplane-system\n```\n                               \n\n### Create aws-creds.conf file \u0026 create AWS Provider secret\n\nhttps://docs.crossplane.io/latest/getting-started/provider-aws/#generate-an-aws-key-pair-file\n\nI assume here that you have [aws CLI installed and configured](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). So that the command `aws configure` should work on your system. With this prepared we can create an `aws-creds.conf` file:\n\n```shell\necho \"[default]\naws_access_key_id = $(aws configure get aws_access_key_id)\naws_secret_access_key = $(aws configure get aws_secret_access_key)\n\" \u003e aws-creds.conf\n```\n\n\u003e Don't ever check this file into source control - it holds your AWS credentials! For this repository I added `*-creds.conf` to the [.gitignore](.gitignore) file. \n\n\nNow we need to use the `aws-creds.conf` file to create the Crossplane AWS Provider secret:\n\n```shell\nkubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./aws-creds.conf\n```\n\n\n\n### Install crossplane's AWS provider with ArgoCD\n\nOur crossplane AWS provider for S3 resides in [upbound/provider-aws/provider/upbound-provider-aws-s3.yaml](upbound/provider-aws/provider/upbound-provider-aws-s3.yaml):\n\n```yaml\napiVersion: pkg.crossplane.io/v1\nkind: Provider\nmetadata:\n  name: upbound-provider-aws-s3\nspec:\n  package: xpkg.upbound.io/upbound/provider-aws-s3:v1.12.0\n  packagePullPolicy: Always\n  revisionActivationPolicy: Automatic\n  revisionHistoryLimit: 1\n```\n\nHow do we let ArgoCD manage and deploy this to our cluster? The simple way of [defining a directory containing k8s manifests](https://argo-cd.readthedocs.io/en/stable/user-guide/directory/) is what we're looking for. Therefore we create a new ArgoCD `Application` CRD at [argocd/crossplane-bootstrap/crossplane-provider-aws.yaml](argocd/crossplane-bootstrap/crossplane-provider-aws.yaml), which tells Argo to look in the directory path `upbound/provider-aws/config`:\n\n```yaml\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: crossplane-provider-aws-s3\n  namespace: argocd\n  finalizers:\n    - resources-finalizer.argocd.argoproj.io\nspec:\n  project: default\n  source:\n    path: upbound/provider-aws/config\n    repoURL: https://github.com/jonashackt/crossplane-argocd\n    targetRevision: HEAD\n  destination:\n    namespace: default\n    server: https://kubernetes.default.svc\n  # Using syncPolicy.automated here, otherwise the deployement of our Crossplane provider will fail with\n  # 'Resource not found in cluster: pkg.crossplane.io/v1/Provider:provider-aws-s3'\n  syncPolicy:\n    automated: \n      prune: true     \n```\n\nThe crucial point here is to use the `syncPolicy.automated` flag as described in the docs: https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/. Otherwise the deployment of the Crossplane `upbound-provider-aws-s3` will give the following error:\n\n```shell\nResource not found in cluster: pkg.crossplane.io/v1/Provider:upbound-provider-aws-s3\n```\n\nThe automated syncPolicy makes sure that child apps are automatically created, synced, and deleted when the manifest is changed.\n\n\u003e This flag enables ArgoCD's \"true\" GitOps feature, where the CI/CD pipeline doesn't deploy themselfes (Push-based GitOps) but only makes a git commit. Then the GitOps operator inside the Kubernetes cluster (here ArgoCD) recognizes the change in the Git repository and deploys the changes to match the state of the repository in the cluster.\n\nWe also use the finalizer `resources-finalizer.argocd.argoproj.io finalizer` like we did with the Crossplane core components so that a `kubectl delete -f` would also undeploy all components of our Provider `provider-aws-s3`.\n\nLet's apply this `Application` to our cluster also:\n\n```shell\nkubectl apply -n argocd -f argocd/crossplane-bootstrap/crossplane-provider-aws.yaml \n```\n\n\nWe run into the following error while syncing in Argo:\n\n```shell\nThe Kubernetes API could not find aws.upbound.io/ProviderConfig for requested resource default/default. Make sure the \"ProviderConfig\" CRD is installed on the destination cluster.\n```\n\n![](docs/argocd-crossplane-provider-sync-failed.png)\n\n\n\n\n### Install crossplane's AWS provider ProviderConfig with ArgoCD\n\nTo get our Provider finally working we also need to create a `ProviderConfig` accordingly that will tell the Provider where to find it's AWS credentials. Therefore we create a [upbound/provider-aws/config/provider-aws-config.yaml](upbound/provider-aws/config/provider-aws-config.yaml):\n\n```yaml\napiVersion: aws.upbound.io/v1beta1\nkind: ProviderConfig\nmetadata:\n  name: default\nspec:\n  credentials:\n    source: Secret\n    secretRef:\n      namespace: crossplane-system\n      name: aws-creds\n      key: creds\n```\n\n\u003e Crossplane resources use the `ProviderConfig` named `default` if no specific ProviderConfig is specified, so this ProviderConfig will be the default for all AWS resources.\n\nThe `secretRef.name` and `secretRef.key` has to match the fields of the already created Secret.\n\n\nTo let ArgoCD manage and deploy our `ProviderConfig` we again create a new ArgoCD `Application` CRD at [argocd/crossplane-bootstrap/crossplane-provider-aws-config.yaml](argocd/crossplane-bootstrap/crossplane-provider-aws-config.yaml) [defining a directory containing k8s manifests](https://argo-cd.readthedocs.io/en/stable/user-guide/directory/), which tells Argo to look in the directory path `upbound/provider-aws/config`:\n\n\n```yaml\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: provider-aws-config\n  namespace: argocd\n  finalizers:\n    - resources-finalizer.argocd.argoproj.io\nspec:\n  project: default\n  source:\n    path: upbound/provider-aws/config\n    repoURL: https://github.com/jonashackt/crossplane-argocd\n    targetRevision: HEAD\n  destination:\n    namespace: default\n    server: https://kubernetes.default.svc\n  # Using syncPolicy.automated here, otherwise the deployement of our Crossplane provider will fail with\n  # 'Resource not found in cluster: pkg.crossplane.io/v1/Provider:provider-aws-s3'\n  syncPolicy:\n    automated: \n      prune: true    \n```\n\n\n\n```shell\nkubectl apply -n argocd -f argocd/crossplane-bootstrap/crossplane-provider-aws-config.yaml \n```\n\n\n\nWe finally managed to let Argo deploy the Crossplane core components together with the AWS Provider and ProviderConfig correctly:\n\n![](docs/crossplane-core-provider-providerconfig-successfully-deployed.png)\n\n\n\n\n\n# Using ArgoCD's AppOfApps pattern to deploy Crossplane components\n\n### Why our current setup is sub optimal\n\nWhile our setup works now and also fully implements the GitOps way, we have a lot of `Application` files, that need to be applied in a specific order.\n\n\u003e __Our goal should be a single manifest defining the whole Crossplane setup incl. core, Provider, ProviderConfig etc. in ArgoCD__\n\nIf we would use [an Application that points to a directory](https://argo-cd.readthedocs.io/en/stable/user-guide/directory/) with multiple manifests, we'll run into errors like this:\n\n```shell\nThe Kubernetes API could not find aws.upbound.io/ProviderConfig for requested resource default/default. Make sure the \"ProviderConfig\" CRD is installed on the destination cluster.\n```\n\nSince deployment order wouldn't be clear and the `Provider` manifests need to be fully deployed before the `ProviderConfig`. Otherwise the deployment fails because of missing CRDs. \n\n__Wouldn't be Argo's SyncWaves feature a great match for that issue?__\n\n\u003e The ArgoCD docs have a great video explaining SyncWaves and Hooks: https://www.youtube.com/watch?v=zIHe3EVp528\n\n\u003e Another great SyncWave tutorial can be found here https://redhat-scholars.github.io/argocd-tutorial/argocd-tutorial/04-syncwaves-hooks.html\n\nSadly using Argo's [`SyncWaves` feature](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/) alone doesn't really help here, if we use them at the `Application` level. I had a hard time figuring that one out, but to really use the SyncWaves feature, we would need to use the annotations like `metadata: annotations: argocd.argoproj.io/sync-wave: \"2\"` on every of the Crossplane Provider's Kubernetes objects (and thus alter the manifests to add the annotation).\n\n\n\n### App of Apps Pattern vs. ApplicationSets\n\nNow there are multiple patterns you can use to manage multiple ArgoCD application. You can for example go with [the App of Apps Pattern](https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/#app-of-apps-pattern) or with [`ApplicationSets`](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/), which moved into the ArgoCD main project around version 2.6.\n\nYou'd might say: ApplicationSets is the way to go today. But __App of Apps is not deprecated__ https://github.com/argoproj/argo-cd/discussions/11892#discussioncomment-6765089 The exact same GitHub issue shows our discussion:\n\n\u003e To be super clear: app-of-apps is not deprecated. The idea of deploying Applications (which are just Kubernetes resources) from another Application is fundamental to how Argo CD works. It would be difficult to remove even if we wanted to.\n\n\u003e As for the hackiness: yes, it does have limitations, bugs, and idiosyncrasies. And ApplicationSets (or something else) may be better for some use cases. But all tools have limitations, bugs, and idiosyncrasies.\n\nFrom that I would extract the following TLDR: If you want to bootstrap a cluster (e.g. installing tools like Crossplane), the App of Apps feature together with it's support for SyncWaves is pretty handsome. That might be the reason, the feature is described inside the `operator-manual/cluster-bootstrapping` part of the docs: https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/#app-of-apps-pattern\n\nIf you want to get your teams enabled to deploy their apps in a GitOps fashion (incl. self-service) and want a great way to use multiple manifests in apps also from within monorepos (e.g. backend, frontend, db), then [the `ApplicationSet` feature](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/) is match for you. It also generates the `Application` manifests automatically leveraging it's many generators, like `Git Generator: Directories`, `Git Generator: Files` and so on. My colleague Daniel Häcker [wrote a great post about that topic](https://www.codecentric.de/wissens-hub/blog/gitops-argocd). \n\nAs we're focussing on bootstrapping our cluster with ArgoCD and Crossplane, let's go with the App of Apps Pattern here.\n\n\n\n### Implementing the App of Apps Pattern for Crossplane deployment\n\nArgoCD Applications can be used in ArgoCD Applications - since they are normal Kubernetes CRDs. \n\nTherefore let's define a new top level `Application` that manages the whole Crossplane setup incl. core, Provider, ProviderConfig etc.\n\nI created my App of Apps definition in [argocd/crossplane-bootstrap.yaml](argocd/crossplane-bootstrap.yaml):\n\n```yaml\n# The ArgoCD App of Apps for all Crossplane components\n---\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: crossplane-bootstrap\n  namespace: argocd\n  finalizers:\n    - resources-finalizer.argocd.argoproj.io\nspec:\n  project: default\n  source:\n    repoURL: https://github.com/jonashackt/crossplane-argocd\n    targetRevision: HEAD\n    path: argocd/crossplane-bootstrap\n  destination:\n    server: https://kubernetes.default.svc\n    namespace: crossplane-system\n  syncPolicy:\n    automated:\n      prune: true    \n    syncOptions:\n    - CreateNamespace=true\n    retry:\n      limit: 1\n      backoff:\n        duration: 5s \n        factor: 2 \n        maxDuration: 1m\n```\n\nThis `Application` will look for manifests at `argocd/crossplane-bootstrap` in our repository https://github.com/jonashackt/crossplane-argocd. And there all our Crossplane components are already defined as ArgoCD `Application` manifests. \n\nAlso don't forget to define the finalizers `finalizers: - resources-finalizer.argocd.argoproj.io`. Otherwise the Applications managed by this App of Apps won't be deleted and will still be running, if you delete just the App of Apps!\n\nVoilá. Now we need to use Argo's [`SyncWaves` feature](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/) as already mentioned to define, which ArgoCD Application (representing a Crossplane component each) needs to be deployed by Argo in which exact order.\n\nFirst we need to deploy the [Crossplane Helm Secret](argocd/crossplane-bootstrap/crossplane-helm-secret.yaml), so we add the `annotations: argocd.argoproj.io/sync-wave` configuration to it's `metadata`:\n\n```yaml\nmetadata:\n  annotations:\n    argocd.argoproj.io/sync-wave: \"0\"\n```\n\nWe use `sync-wave: \"0\"` here, to define it as the earliest stage of Argo deployment (you could use negative numbers though, but for simplicity we start at zero).\n\nThen we need to deploy the Crossplane core components, defined in [`argocd/crossplane-bootstrap/crossplane.yaml`](argocd/crossplane-bootstrap/crossplane.yaml). There we add the next SyncWave as `sync-wave: \"1\"`:\n\n```yaml\nmetadata:\n  annotations:\n    argocd.argoproj.io/sync-wave: \"1\"\n```\n\nYou get the point! We also add the `sync-wave` annotation to the AWS Provider in [`argocd/crossplane-bootstrap/crossplane-provider-aws.yaml`](argocd/crossplane-bootstrap/crossplane-provider-aws.yaml) and the ProviderConfig at [`argocd/crossplane-bootstrap/crossplane-provider-config-aws.yaml`](argocd/crossplane-bootstrap/crossplane-provider-config-aws.yaml).\n\nNow we should be able to finally apply our Crossplane App of Apps in Argo:\n\n```shell\nkubectl apply -n argocd -f argocd/crossplane-bootstrap.yaml \n```\n\nAnd like magic all our Crossplane components get deployed step by step in correct order:\n\n![](docs/crossplane-app-of-apps-successful-deployed.png)\n\n\nNow if we have a look into `crossplane` App of Apps we see all the needed components to deploy a running Crossplane installation using ArgoCD (which I found is super nice):\n\n![](docs/crossplane-app-of-apps-detail-view.png)\n\n\n\n\n\n\n## Doing it all with GitHub Actions\n\nOk, enough theory :)) Let's create a pipeline that shows stuff works. Let's introduce a [.github/workflows/crossplane-argocd.yml](.github/workflows/crossplane-argocd.yml):\n\n```yaml\nname: crossplane-argocd\n\non: [push]\n\nenv:\n  KIND_NODE_VERSION: v1.30.4\n  # AWS\n  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}\n  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n  AWS_DEFAULT_REGION: 'eu-central-1'\n\njobs:\n  provision:\n    runs-on: ubuntu-latest\n    steps:\n      - name: Checkout\n        uses: actions/checkout@master\n\n      - name: Spin up kind\n        run: |          \n          echo \"--- Create kind cluster\"\n          kind create cluster --image \"kindest/node:$KIND_NODE_VERSION\" --wait 5m\n\n          echo \"--- Let's try to access our kind cluster via kubectl\"\n          kubectl get nodes\n\n      - name: Install ArgoCD into kind\n        run: |\n          echo \"--- Create argo namespace and install it\"\n          kubectl create namespace argocd\n\n          echo \" Install \u0026 configure ArgoCD via Kustomize - see https://stackoverflow.com/a/71692892/4964553\"\n          kubectl apply -k argocd/install\n          \n          echo \"--- Wait for Argo to become ready\"\n          kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-server --namespace argocd --timeout=300s\n\n      - name: Prepare crossplane AWS Secret\n        run: |\n          echo \"--- Create aws-creds.conf file\"\n          echo \"[default]\n          aws_access_key_id = $AWS_ACCESS_KEY_ID\n          aws_secret_access_key = $AWS_SECRET_ACCESS_KEY\n          \" \u003e aws-creds.conf\n          \n          echo \"--- Create a namespace for crossplane\"\n          kubectl create namespace crossplane-system\n\n          echo \"--- Create AWS Provider secret\"\n          kubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./aws-creds.conf\n\n      - name: Use ArgoCD's AppOfApps pattern to deploy all Crossplane components\n        run: |\n          echo \"--- Let Argo do it's magic installing all Crossplane components\"\n          kubectl apply -n argocd -f argocd/crossplane-bootstrap.yaml \n\n      - name: Check crossplane status\n        run: |\n          echo \"--- Wait for crossplane to become ready (now prefaced with until as described in https://stackoverflow.com/questions/68226288/kubectl-wait-not-working-for-creation-of-resources)\"\n          until kubectl wait --for=condition=PodScheduled pod -l app=crossplane --namespace crossplane-system --timeout=120s \u003e /dev/null 2\u003e\u00261; do : ; done\n          kubectl wait --for=condition=ready pod -l app=crossplane --namespace crossplane-system --timeout=120s\n\n          echo \"--- Wait until AWS Provider is up and running (now prefaced with until to prevent Error from server (NotFound): providers.pkg.crossplane.io 'provider-aws-s3' not found)\"\n          until kubectl get provider/provider-aws-s3 \u003e /dev/null 2\u003e\u00261; do : ; done\n          kubectl wait --for=condition=healthy --timeout=180s provider/provider-aws-s3\n\n          kubectl get all -n crossplane-system\n```\n\nBe sure to create both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` configured as GitHub Repository Secrets:\n\n![github-actions-secrets](docs/github-actions-secrets.png)\n\nAlso make sure to have your `Default region` configured as a `env:` variable.\n\n\n\n\n\n## Finally provisioning Cloud resources with Crossplane and Argo\n\nLet's create a simple S3 Bucket in AWS. [The docs tell us](https://marketplace.upbound.io/providers/upbound/provider-aws-s3/v0.47.1/resources/s3.aws.upbound.io/Bucket/v1beta1), which config we need. [`infrastructure/s3/simple-bucket.yaml`](infrastructure/s3/simple-bucket.yaml) features a super simply example:\n\n```yaml\napiVersion: s3.aws.upbound.io/v1beta1\nkind: Bucket\nmetadata:\n  name: crossplane-argocd-s3-bucket\nspec:\n  forProvider:\n    region: eu-central-1\n  providerConfigRef:\n    name: default\n```\n\n\nSince we're using Argo, we should deploy our Bucket as Argo Application too. I created a new folder `argocd/infrastructure`\nhere, since the Crossplane provisioned infrastructure may not automatically be part of the bootstrap App of Apps.\n\nSo here's our Argo Application for all the Crossplane managed infrastructure that may come: [`argocd/infrastructure/aws-s3.yaml`](argocd/infrastructure/aws-s3.yaml):\n\n```yaml\n# The ArgoCD Application for all Crossplane Managed Resources\n---\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: aws-s3\n  namespace: argocd\n  finalizers:\n    - resources-finalizer.argocd.argoproj.io\nspec:\n  project: default\n  source:\n    repoURL: https://github.com/jonashackt/crossplane-argocd\n    targetRevision: HEAD\n    path: infrastructure\n  destination:\n    namespace: default\n    server: https://kubernetes.default.svc\n  syncPolicy:\n    automated:\n      prune: true    \n    retry:\n      limit: 5\n      backoff:\n        duration: 5s \n        factor: 2 \n        maxDuration: 1m\n```\n\nApply it with:\n\n```shell\nkubectl apply -f argocd/infrastructure/aws-s3.yaml\n```\n\nIf everything went fine, the Argo app should look `Healthy` like this:\n\n![](docs/first-s3-bucket-provisioned-with-argo-crossplane.png)\n\nAnd inside the AWS console, there should be a new S3 Bucket provisioned:\n\n![](docs/aws-console-s3-bucket-provisioned.png)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n# Getting rid of the manual K8s Secrets creation\n\nCI pushes Secrets into the cluster via `kubectl apply`...\n\nThis is an anti-GitOps pattern - so let's do something different!\n\nTODO: Insert why here :) GitOps Pull instead of Push...\n\n\nAfter reading through lot's of \"How to manage Secrets with GitOps articles\" (like [this](https://www.redhat.com/en/blog/a-guide-to-secrets-management-with-gitops-and-kubernetes), [this](https://betterprogramming.pub/why-you-should-avoid-sealed-secrets-in-your-gitops-deployment-e50131d360dd) and [this](https://akuity.io/blog/how-to-manage-kubernetes-secrets-gitops ) to name a few), I found that there's currently no widly accepted way of doing it. But there are some recommendations. E.g. checking Secrets into Git (although encrypted) using [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) or [SOPS](https://github.com/getsops/sops)/[KSOPS](https://github.com/viaduct-ai/kustomize-sops) might seem like the kind of easiest solution in the first place. But they have their own caveats in the long therm. Think of multiple secrets defined in multiple projects used by multiple teams all over your Git repositories - and now do a secret or key rotation...\n\n\nThe TLDR; of most (recent) articles and [GitHub discussions](https://github.com/argoproj/argo-cd/issues/1364) I distilled for me is: Use an external secret store and connect that one to your ArgoCD managed cluster. With an external secret store you get key rotation, support for serving secrets as symbolic references, usage audits and so on. Even in the case of secret or key compromisation you mostly get proven mitigations paths. \n\n\n\n## Which tooling to integrate ArgoCD with the external secret store\n\nThere is a huge list of possible plugins or operators helping to integrate your ArgoCD managed cluster with an external secret store. You can for example have a look onto [the list featured in the Argo docs](https://argo-cd.readthedocs.io/en/stable/operator-manual/secret-management/). I had a look on some promising candidates:\n\nA lightweight solution could be https://github.com/argoproj-labs/argocd-vault-plugin / https://argocd-vault-plugin.readthedocs.io/en/stable/. It supports multiple backends like AWS Secrets Manager, Azure Key Vault, Hashicorp Vault etc. But the installation [isn't that lightweight](https://argocd-vault-plugin.readthedocs.io/en/stable/installation/), because we need to download the Argo Vault Plugin in a volume and inject it into the `argocd-repo-server` ([although there are pre-build Kustomize manifests available](https://github.com/argoproj-labs/argocd-vault-plugin/blob/main/manifests/cmp-configmap)) by creating a custom argocd-repo-server image with the plugin and supporting tools pre-installed... Also [a newer sidecar option](https://argocd-vault-plugin.readthedocs.io/en/stable/installation/) is available, which nevertheless has [it's own caveats](https://argocd-vault-plugin.readthedocs.io/en/stable/usage/#running-argocd-vault-plugin-in-a-sidecar-container).\n\nThere's also [Hashicorps own Vault Agent](https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent) and the [Secrets Store CSI Driver](https://github.com/kubernetes-sigs/secrets-store-csi-driver), who both handle secrets without the need for Kubernetes Secrets. The first works with a per-pod based sidecar approach to connect to Vault via the agent and the latter uses the Container Storage Interface.\n\nBoth look nice, but I found the following the most promising solution right now: The [External Secrets Operator (ESO)](https://external-secrets.io/latest/). Featuring also a lot of GitHub stars External Secrets simply creates a Kubernetes Secret for each external secret. According to the docs:\n\n\u003e \"ExternalSecret, SecretStore and ClusterSecretStore that provide a user-friendly abstraction for the external API that stores and manages the lifecycle of the secrets for you.\"\n\nAnd what's also promising, [the community seems to be growing rapidly](https://github.com/external-secrets/external-secrets):\n\n\u003e \"Multiple people and organizations are joining efforts to create a single External Secrets solution based on existing projects.\"\n\n\n\n## Using External Secrets together with Doppler\n\nThe External Secrets Operator supports a multitude of tools for secret management! Just have a look at the docs \u0026 [you'll see more than 20 tools supported](https://external-secrets.io/latest/provider/aws-secrets-manager/), featuring the well known AWS Secretes Manager, Azure Key Vault, Hashicorp Vault, Akeyless and so on. \n\nAnd as I like to show solutions that are fully cromprehensible - ideally without a creditcard - I was on the lookout for a tool, that had a small free plan. But without the need to selfhost the solution, since that would be out of scope for this project. At first glance I thought that [Hashicorp's Vault Secrets](https://developer.hashicorp.com/hcp/docs/vault-secrets) as part of the Hashicorp Cloud Platform (HCP) would be a great choice since so many projects love and use Vault. But sadly External Secrets Operator currently doesn't support HCP Vault Secrets and I would have been forced to [switch to Hashicorp Vault Secrets Operator (VSO)](https://developer.hashicorp.com/hcp/docs/vault-secrets/integrations/kubernetes), which is for sure also an interesting project. But I wanted to stick with the External Secrets Operator since it's wide support for providers and it looks as it could develop into the defacto standard in external secrets integration in Kubernetes.\n\nSo I thought the exact secret management tool I use in this case is not that important and I trust my readers that they will choose the provider that suites them the most. That beeing said I chose [Doppler](https://www.doppler.com/) with their [generous free Developer plan](https://www.doppler.com/pricing).\n\nAs External-Secrets introduce more complexity to our setup, I decided to divide the crossplane only solution from the more advanced using External Secrets Operator. Therefore the `argocd` directory now looks like this:\n\n```shell\n$ tree\n.\n├── crossplane-bootstrap\n│   ├── crossplane.yaml\n│   ├── crossplane-helm-secret.yaml\n│   └── crossplane-provider-aws.yaml\n├── crossplane-eso-bootstrap\n│   ├── crossplane.yaml\n│   ├── crossplane-helm-secret.yaml\n│   ├── crossplane-provider-aws.yaml\n│   ├── external-secrets-config.yaml\n│   └── external-secrets-operator.yaml\n├── crossplane-bootstrap.yaml\n├── crossplane-eso-app-of-apps.yaml\n...\n```\n\nWhere `crossplane-bootstrap` and the corresponding `crossplane-bootstrap.yaml` feature the crossplane only solution - and `crossplane-eso-bootstrap` with it's `crossplane-eso-app-of-apps.yaml` App-of-Apps counterpart feature the more advanced ESO solution.\n\n\n\n### Create multiline Secret in Doppler\n\nSo let's create our first secret in Doppler. If you haven't already done so sign up at https://dashboard.doppler.com (e.g. with your GitHub account). Then click on `Projects` on the left navigation bar and on the `+` to create a new project. In this example I named it according to this example project: `crossplane-argocd`.\n\n![](docs/doppler-project-stages.png)\n\nDoppler automatically creates well known environments for us: development, staging and production. To create a new Secret, choose a environment and click on `Add First Secret`. Now give it the key `CREDS`. The value will be a multiline value. Just like [it is stated in the crossplane docs](https://docs.crossplane.io/latest/getting-started/provider-aws/#generate-an-aws-key-pair-file), we should have an `aws-creds.conf` file created already (that we don't want to check into source control):\n\n```shell\necho \"[default]\naws_access_key_id = $(aws configure get aws_access_key_id)\naws_secret_access_key = $(aws configure get aws_secret_access_key)\n\" \u003e aws-creds.conf\n```\n\nCopy the contents of the `aws-creds.conf` into the value field in Doppler. The Crossplane AWS Provider or rather it's ProviderConfig will later consume the secret just like it is as multiline text:\n\n![](docs/doppler-aws-creds-multiline.png)\n\nDon't forget so click on `save`. \n\n\n\n### Create Service Token in Doppler project environment\n\n[As stated in the External Secrets docs](https://external-secrets.io/latest/provider/doppler/), we need to create a Doppler Service Token in order to be albe to connect to Doppler later on.\n\nIn Doppler Service Tokens are created on project level - inside a specific environment, where we already created our secrets. As I created my secrets in the `dev` environment, I create the Service Token also there. Simply head over to your Doppler project, select the environment you created your secrets in and click on `Access`. Here you should find a button called `+ Generate` to create a new Service Token. Click the button and create a Service Token with `read` access and no expiration and copy it somewhere locally.\n\n![](docs/doppler-project-service-token.png)\n\n\n\n\n### Create Kubernetes Secret with the Doppler Service Token\n\nIn order to be able to let the External Secrets Operator access Doppler, we need to create a Kubernetes `Secret` containing the Doppler Service Token:\n\n```shell\nkubectl create secret generic \\\n    doppler-token-auth-api \\\n    --from-literal dopplerToken=\"dp.st.xxxx\"\n```\n\n\n\n\n### Install External Secrets Operator as ArgoCD Application\n\nhttps://external-secrets.io/latest/introduction/getting-started/\n\nInstalling External Secrets Operator in a GitOps fashion \u0026 have updates managed by Renovate, we can use the method already applied to Crossplane and explained in https://stackoverflow.com/a/71765472/4964553. Therefore we create a simple Helm chart at [`external-secrets/Chart.yaml`](external-secrets/Chart.yaml):\n\n```yaml\napiVersion: v2\ntype: application\nname: external-secrets\nversion: 0.0.0 # unused\nappVersion: 0.0.0 # unused\ndependencies:\n  - name: external-secrets\n    repository: https://charts.external-secrets.io\n    version: 0.9.11\n```\n\nNow telling ArgoCD where to find our simple external-secrets Helm Chart, we again use Argo's `Application` manifest in [argocd/crossplane-eso-bootstrap/external-secrets-operator.yaml](argocd/crossplane-eso-bootstrap/external-secrets-operator.yaml):\n\n```yaml\n# The ArgoCD Application for external-secrets-operator\n---\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: external-secrets-operator\n  namespace: argocd\n  finalizers:\n    - resources-finalizer.argocd.argoproj.io\n  annotations:\n    argocd.argoproj.io/sync-wave: \"0\"\nspec:\n  project: default\n  source:\n    repoURL: https://github.com/jonashackt/crossplane-argocd\n    targetRevision: HEAD\n    path: external-secrets/install\n  destination:\n    server: https://kubernetes.default.svc\n    namespace: external-secrets\n  syncPolicy:\n    automated:\n      prune: true    \n    syncOptions:\n    - CreateNamespace=true\n    retry:\n      limit: 1\n      backoff:\n        duration: 5s \n        factor: 2 \n        maxDuration: 1m\n```\n\nWe define the SyncWave to deploy external-secrets before every other Crossplane component via `annotations: argocd.argoproj.io/sync-wave: \"-1\"`.\n\nJust for checking if it works, we can use a `kubectl apply -f argocd/crossplane-bootstrap/external-secrets.yaml` to apply it to our cluster. If everything went correctly, there should be a new ArgoCD Application featuring a bunch of CRDs, some roles and three Pods: `external-secrets`, `external-secrets-webhook` and `external-secrets-cert-controller`:\n\n![](docs/external-secrets-argo-app.png)\n\n\n\n\n\n### Create ClusterSecretStore that manages access to Doppler\n\nhttps://external-secrets.io/latest/provider/doppler/#authentication\n\nhttps://external-secrets.io/latest/introduction/overview/#secretstore\n\n\u003e The idea behind the `SecretStore` resource is to separate concerns of authentication/access and the actual Secret and configuration needed for workloads. The ExternalSecret specifies what to fetch, the SecretStore specifies how to access. \n\nIn this project I opted for the similar `ClusterSecretStore`. As [the docs state](https://external-secrets.io/latest/introduction/overview/#clustersecretstore):\n\n\u003e \"The ClusterSecretStore is a global, cluster-wide SecretStore that can be referenced from all namespaces. You can use it to provide a central gateway to your secret provider.\"\n\nSounds like a good fit for our setup. But you can also opt for [the namespaced `SecretStore`](https://external-secrets.io/latest/introduction/overview/#secretstore) too. Our ClusterSecretStore reside here [`external-secrets/config/cluster-secret-store.yaml`](external-secrets/config/cluster-secret-store.yaml):\n\n```yaml\napiVersion: external-secrets.io/v1beta1\nkind: ClusterSecretStore\nmetadata:\n  name: doppler-auth-api\nspec:\n  provider:\n    doppler:\n      auth:\n        secretRef:\n          dopplerToken:\n            name: doppler-token-auth-api\n            key: dopplerToken\n            namespace: default\n```\n\nDon't forget to configure a `namespace` for the `doppler-token-auth-api` Secret we created earlier. Otherwise we'll run into errors like:\n\n```shell\nadmission webhook \"validate.clustersecretstore.external-secrets.io\" denied the request: invalid store: cluster scope requires namespace (retried 1 times).\n```\n\nThe External Secrets Operator will create a Secret that's similar to the one mentioned in the Crossplane docs (if you decode it), but with the uppercase `CREDS` key we used in Doppler:\n\n```shell\nCREDS: |+\n[default] \naws_access_key_id = yourAccessKeyIdHere\naws_secret_access_key = yourSecretAccessKeyHere\n```\n\n\n\n\n### Create ExternalSecret to access AWS credentials\n\nhttps://external-secrets.io/latest/introduction/overview/#externalsecret\n\nhttps://external-secrets.io/latest/provider/doppler/#use-cases\n\nAs we already defined how the external secret store (Doppler) could be accessed (using our `ClusterSecretStore` CRD) should now specify which secrets to fetch using the `ExternalSecret` CRD. Therefore let's create a [`external-secrets/config/external-secret.yaml`](external-secrets/config/external-secret.yaml):\n\n```yaml\napiVersion: external-secrets.io/v1beta1\nkind: ExternalSecret\nmetadata:\n  name: auth-api-db-url\nspec:\n  secretStoreRef:\n    kind: ClusterSecretStore\n    name: doppler-auth-api\n\n  # access our 'CREDS' key in Doppler\n  dataFrom:\n    - find:\n        path: CREDS\n\n  # Create a Kubernetes Secret just as we're used to without External Secrets Operator\n  target:\n    name: aws-secrets-from-doppler\n```\n\nWe created a `CREDS` secret in Doppler, so the `ExternalSecret` looks for this exact path.\n\n\nWe also need to create a ArgoCD Application so that Argo will deploy both `ClusterSecretStore` and `ExternalSecret` for us :) Therefore I created [`argocd/crossplane-eso-bootstrap/external-secrets-config.yaml`](argocd/crossplane-eso-bootstrap/external-secrets-config.yaml):\n\n```yaml\n# The ArgoCD Application for external-secrets-operator\n---\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: external-secrets-config\n  namespace: argocd\n  finalizers:\n    - resources-finalizer.argocd.argoproj.io\n  annotations:\n    argocd.argoproj.io/sync-wave: \"1\"\nspec:\n  project: default\n  source:\n    repoURL: https://github.com/jonashackt/crossplane-argocd\n    targetRevision: HEAD\n    path: external-secrets\n  destination:\n    server: https://kubernetes.default.svc\n    namespace: external-secrets\n  syncPolicy:\n    automated:\n      prune: true    \n    syncOptions:\n    - CreateNamespace=true\n    retry:\n      limit: 1\n      backoff:\n        duration: 5s \n        factor: 2 \n        maxDuration: 1m\n```\n\n\nOur ClusterSecretStore and ExternalSecrets deployment in Argo looks like this:\n\n![](docs/external-secrets-configuration-in-argo.png)\n\nBut the deployment doesn't run flawless, although configured as `argocd.argoproj.io/sync-wave: \"-1\"` right AFTER the `external-secrets` Argo Application, which deployes the External Secrets components:\n\n```shell\nFailed sync attempt to 603cce3949c2a916f51f3917e87aa814698e5f92: one or more objects failed to apply, reason: Internal error occurred: failed calling webhook \"validate.externalsecret.external-secrets.io\": failed to call webhook: Post \"https://external-secrets-webhook.external-secrets.svc:443/validate-external-secrets-io-v1beta1-externalsecret?timeout=5s\": dial tcp 10.96.42.44:443: connect: connection refused,Internal error occurred: failed calling webhook \"validate.clustersecretstore.external-secrets.io\": failed to call webhook: Post \"https://external-secrets-webhook.external-secrets.svc:443/validate-external-secrets-io-v1beta1-clustersecretstore?timeout=5s\": dial tcp 10.96.42.44:443: connect: connection refused (retried 1 times).\n```\n\nIt seems that our `external-secrets-webhook` isn't healthy already, but the `ClusterSecretStore` \u0026 the `ExternalSecret` already want to access the webhook. So we may need to wait for the `external-secrets-webhook` to be really available before we deploy our `external-secrets-config`?!\n\nTherefore let's give our `external-secrets-config` more `syncPolicy.retry.limit`:\n\n```yaml\n  syncPolicy:\n    ...\n    retry:\n      limit: 5\n      backoff:\n        duration: 5s \n        factor: 2 \n        maxDuration: 1m\n```\n\n\n### Point the Crossplane AWS ProviderConfig to our External Secret created Secret from Doppler\n\nWe need to change our `ProviderConfig` at [`upbound/provider-aws/provider-eos/provider-config-aws.yaml`](upbound/provider-aws/provider-eos/provider-config-aws.yaml) to use another Secret name and namespace: \n\n```yaml\napiVersion: aws.upbound.io/v1beta1\nkind: ProviderConfig\nmetadata:\n  name: default\nspec:\n  credentials:\n    source: Secret\n    secretRef:\n      namespace: external-secrets\n      name: aws-secrets-from-doppler\n      key: CREDS\n```\n\nWith this final piece our setup should be complete to be able to provision some infrastructure with ArgoCD and Crossplane!\n\nHere are all components together we deployed so far using Argo:\n\n![](docs/bootstrap-finalized-argo-crossplane-eso.png)\n\nDeploying our [`argocd/infrastructure/aws-s3.yaml`](argocd/infrastructure/aws-s3.yaml) should also work as expected:\n\n```shell\nkubectl apply -f argocd/infrastructure/aws-s3.yaml\n```\n\nIf everything went fine, the Argo app should look `Healthy` like this:\n\n![](docs/first-s3-bucket-provisioned-with-argo-crossplane.png)\n\nAnd inside the AWS console, there should be a new S3 Bucket provisioned:\n\n![](docs/aws-console-s3-bucket-provisioned.png)\n\n\n\n## Adding External Secrets Deployment to GitHub Actions\n\nLet's create another pipeline that shows what differences are to the deployment without External Secrets Operator. Let's introduce a [.github/workflows/crossplane-argocd-external-secrets](.github/workflows/crossplane-argocd-external-secrets):\n\n```yaml\nname: crossplane-argocd-external-secrets\n\non: [push]\n\nenv:\n  KIND_NODE_VERSION: v1.32.4\n  # Doppler\n  DOPPLER_SERVICE_TOKEN: ${{ secrets.DOPPLER_SERVICE_TOKEN }}\n\njobs:\n  provision:\n    runs-on: ubuntu-latest\n    steps:\n      - name: Checkout\n        uses: actions/checkout@master\n\n      - name: Spin up kind via brew\n        run: |          \n          echo \"--- Create kind cluster\"\n          kind create cluster --image \"kindest/node:$KIND_NODE_VERSION\" --wait 5m\n\n          echo \"--- Let's try to access our kind cluster via kubectl\"\n          kubectl get nodes\n\n      - name: Install ArgoCD into kind\n        run: |\n          echo \"--- Create argo namespace and install it\"\n          kubectl create namespace argocd\n\n          echo \" Install \u0026 configure ArgoCD via Kustomize - see https://stackoverflow.com/a/71692892/4964553\"\n          kubectl apply -k argocd/install\n          \n          echo \"--- Wait for Argo to become ready\"\n          kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-server --namespace argocd --timeout=300s\n\n      - name: Create Secret with the Doppler Service Token for External Secrets Operator\n        run: kubectl create secret generic doppler-token-auth-api --from-literal dopplerToken=\"$DOPPLER_SERVICE_TOKEN\"\n\n      - name: Use ArgoCD's AppOfApps pattern to deploy all Crossplane components\n        run: |\n          echo \"--- Let Argo do it's magic installing all Crossplane components\"\n          kubectl apply -n argocd -f argocd/crossplane-eso-app-of-apps.yaml \n\n      - name: Check crossplane status\n        run: |\n          echo \"--- Wait for crossplane to become ready (now prefaced with until as described in https://stackoverflow.com/questions/68226288/kubectl-wait-not-working-for-creation-of-resources)\"\n          until kubectl wait --for=condition=PodScheduled pod -l app=crossplane --namespace crossplane-system --timeout=120s \u003e /dev/null 2\u003e\u00261; do : ; done\n          kubectl wait --for=condition=ready pod -l app=crossplane --namespace crossplane-system --timeout=120s\n\n          echo \"--- Wait until AWS Provider is up and running (now prefaced with until to prevent Error from server (NotFound): providers.pkg.crossplane.io 'provider-aws-s3' not found)\"\n          until kubectl get provider/provider-aws-s3 \u003e /dev/null 2\u003e\u00261; do : ; done\n          kubectl wait --for=condition=healthy --timeout=180s provider/provider-aws-s3\n\n          kubectl get all -n crossplane-system\n```\n\nBe sure to create `DOPPLER_SERVICE_TOKEN` as GitHub Repository Secrets.\n\n\n\n\n\n# App Deployment\n\nLet's create a publicly accessible S3 bucket in our infrastructure/bucket.yaml:\n\n```yaml\napiVersion: s3.aws.upbound.io/v1beta1\nkind: Bucket\nmetadata:\n  name: crossplane-argocd-s3-bucket\nspec:\n  forProvider:\n    region: eu-central-1\n  providerConfigRef:\n    name: default\n---\napiVersion: s3.aws.upbound.io/v1beta1\nkind: BucketPublicAccessBlock\nmetadata:\n  name: crossplane-argocd-s3-bucket-pab\nspec:\n  forProvider:\n    blockPublicAcls: false\n    blockPublicPolicy: false\n    ignorePublicAcls: false\n    restrictPublicBuckets: false\n    bucketRef: \n      name: crossplane-argocd-s3-bucket\n    region: eu-central-1\n---\napiVersion: s3.aws.upbound.io/v1beta1\nkind: BucketOwnershipControls\nmetadata:\n  name: crossplane-argocd-s3-bucket-osc\nspec:\n  forProvider:\n    rule:\n      - objectOwnership: ObjectWriter\n    bucketRef: \n      name: crossplane-argocd-s3-bucket\n    region: eu-central-1\n---\napiVersion: s3.aws.upbound.io/v1beta1\nkind: BucketACL\nmetadata:\n  name: crossplane-argocd-s3-bucket-acl\nspec:\n  forProvider:\n    acl: \"public-read\"\n    bucketRef: \n      name: crossplane-argocd-s3-bucket\n    region: eu-central-1\n---\napiVersion: s3.aws.upbound.io/v1beta1\nkind: BucketWebsiteConfiguration\nmetadata:\n  name: crossplane-argocd-s3-bucket-websiteconf\nspec:\n  forProvider:\n    indexDocument:\n      - suffix: index.html\n    bucketRef: \n      name: crossplane-argocd-s3-bucket\n    region: eu-central-1\n```\n\n\nAlso let's sync the Nuxt.js project https://github.com/jonashackt/microservice-ui-nuxt-js via the used `aws s3 sync`:\n\n```shell\naws s3 sync .output/public/ s3://crossplane-argocd-s3-bucket --acl public-read\n```\n\nAnd we should be able to access our via http://crossplane-argocd-s3-bucket.s3-website.eu-central-1.amazonaws.com\n\n\n## Deploy a static website with ArgoCD?\n\nApplication sources are generally Kubernetes manifests in Argo https://argo-cd.readthedocs.io/en/stable/user-guide/application_sources/\n\nSo how do we actually deploy our static website to S3?\n\nhttps://www.reddit.com/r/kubernetes/comments/17qsi5b/is_there_a_kubernetes_way_of_deploying_static_web/\n\nAccording to https://github.com/argoproj/argo-cd/discussions/5052 there's the way to use custom Config Management Plugins https://argo-cd.readthedocs.io/en/stable/operator-manual/config-management-plugins/\n\n\nProposal for Parameterized Configuration Management Plugins in Argo: https://argo-cd.readthedocs.io/en/latest/proposals/parameterized-config-management-plugins/\n\n\nBut maybe we should simply deploy our static website to K8s as well? https://gimlet.io/blog/hosting-static-sites-on-kubernetes\n\nhttps://thenewstack.io/gitops-as-an-evolution-of-kubernetes/\n\n\n\n## Deploy an EKS Cluster\n\n### Multiple AWS Providers as ArgoCD Application\n\nTo be able to deploy a [nested Composition like this for EKS](https://github.com/jonashackt/crossplane-eks-cluster) we need to install multiple Crossplane Providers: `provider-aws-ec2`, `provider-aws-eks`, `provider-aws-iam` additionally to our already installed `provider-aws-s3`. Therefore we should enhance our concept on how to install a Provider with ArgoCD!\n\nSince every Upbound provider family has one ProviderConfig to access the credentials, but multiple providers, it would make sense to enhance the Argo Application `argocd/crossplane-bootstrap/crossplane-provider-aws.yaml` to support multiple providers:\n\n```yaml\n# The ArgoCD Application for all Crossplane AWS providers incl. it's ProviderConfig\n---\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: crossplane-provider-aws\n  namespace: argocd\n  labels:\n    crossplane.jonashackt.io: crossplane\n  finalizers:\n    - resources-finalizer.argocd.argoproj.io\n  annotations:\n    argocd.argoproj.io/sync-wave: \"2\"\nspec:\n  project: default\n  source:\n    repoURL: https://github.com/jonashackt/crossplane-argocd\n    targetRevision: HEAD\n    path: upbound/provider-aws/provider\n  destination:\n    namespace: default\n    server: https://kubernetes.default.svc\n  # Using syncPolicy.automated here, otherwise the deployement of our Crossplane provider will fail with\n  # 'Resource not found in cluster: pkg.crossplane.io/v1/Provider:provider-aws-s3'\n  syncPolicy:\n    automated:\n      prune: true    \n    retry:\n      limit: 5\n      backoff:\n        duration: 5s \n        factor: 2 \n        maxDuration: 1m\n```\n\nThus this Application simply references the folder `upbound/provider-aws/provider`, where all the `Provider` manifests can be stored:\n\n```shell\n└── provider-aws\n    ...\n    ├── config\n    │   └── provider-config-aws.yaml\n    ...\n    └── provider\n        ├── provider-aws-ec2.yaml\n        ├── provider-aws-eks.yaml\n        ├── provider-aws-iam.yaml\n        └── provider-aws-s3.yaml\n```\n\nNow in Argo, the Application shows all available Crossplane providers:\n\n![](docs/multiple-crossplane-provider.png)\n\n\n#### Provider Upgrade problems: 'Only one reference can have Controller set to true'\n\nIf new Provider versions get released, you can watch Argo trying to deploy the old version vs. Crossplane deploying the new one, which leads to a `degraded` status of the Providers:\n\n![](docs/degraded-aws-providers.png)\n\nThe problem is this error: `Only one reference can have Controller set to true. Found \"true\" in references for Provider/provider-aws-ec2 and Provider/provider-aws-ec2`:\n\n```shell\ncannot apply package revision: cannot create object: ProviderRevision.pkg.crossplane.io \"provider-aws-ec2-150095bdd614\" is invalid: metadata.ownerReferences: Invalid value: []v1.OwnerReference{v1.OwnerReference{APIVersion:\"pkg.crossplane.io/v1\", Kind:\"Provider\", Name:\"provider-aws-ec2\", UID:\"30bda236-6c12-412c-a647-b96368eff8b6\", Controller:(*bool)(0xc02afeb38c), BlockOwnerDeletion:(*bool)(0xc02afeb38d)}, v1.OwnerReference{APIVersion:\"pkg.crossplane.io/v1\", Kind:\"Provider\", Name:\"provider-aws-ec2\", UID:\"ee890f53-7590-4957-8f81-e92b931c4e8d\", Controller:(*bool)(0xc02afeb38e), BlockOwnerDeletion:(*bool)(0xc02afeb38f)}}: Only one reference can have Controller set to true. Found \"true\" in references for Provider/provider-aws-ec2 and Provider/provider-aws-ec2\n```\n\nTherefore we should change some options regarding the Provider upgrades in our Provider configurations:\n\n```yaml\napiVersion: pkg.crossplane.io/v1\nkind: Provider\nmetadata:\n  name: upbound-provider-aws-ec2\nspec:\n  package: xpkg.upbound.io/upbound/provider-aws-ec2:v1.1.1\n  packagePullPolicy: IfNotPresent # Only download the package if it isn’t in the cache.\n  revisionActivationPolicy: Automatic # Otherwise our Provider never gets activate \u0026 healthy\n  revisionHistoryLimit: 1\n```\n\nAs we're doing GitOpsified Crossplane with ArgoCD, we should configure the `packagePullPolicy` to `IfNotPresent` instead of `Always` (which means \" Check for new packages every minute and download any matching package that isn’t in the cache\", see https://docs.crossplane.io/master/concepts/packages/#configuration-package-pull-policy) - BUT leave the `revisionActivationPolicy` to `Automatic`! Since otherwise, the Provider will never get active and healty! See https://docs.crossplane.io/master/concepts/packages/#revision-activation-policy), but I didn't find it documented that way!\n\n\n#### GitOpsified Provider Upgrade\n\nSee also https://stackoverflow.com/a/78230499/4964553\n\nNow with `packagePullPolicy: IfNotPresent` \u0026 `revisionActivationPolicy: Automatic` to do a Provider version upgrade, we simply need to upgrade the `spec.package` version number:\n\n```yaml\nspec:\n  package: xpkg.upbound.io/upbound/provider-aws-ec2:v1.2.1 # --\u003e Upgraded to 1.2.1\n  packagePullPolicy: IfNotPresent # Only download the package if it isn’t in the cache.\n  revisionActivationPolicy: Automatic # Otherwise our Provider never gets activate \u0026 healthy\n  revisionHistoryLimit: 1\n```\n\nWe need to commit the change as always, but also be a bit patient here with Argo and Crossplane to initiate and do the update for us. Look at a `kubectl get providerrevisions`. Even after the update commited and registered by Argo, Crossplane will take it's time. First it looks like this:\n\n```shell\nk get providerrevisions\nNAME                                       HEALTHY   REVISION   IMAGE                                                STATE      DEP-FOUND   DEP-INSTALLED   AGE\nprovider-aws-ec2-3d66ea2d7903              True      1          xpkg.upbound.io/upbound/provider-aws-ec2:v1.2.1      Active     1           1               5m31s\nprovider-aws-eks-5021e69b327c              True      2          xpkg.upbound.io/upbound/provider-aws-eks:v1.2.1      Inactive   1           1               4m11s\nprovider-aws-eks-fbb6768e46c0              True      3          xpkg.upbound.io/upbound/provider-aws-eks:v1.1.1      Active     1           1               30m\nprovider-aws-iam-9565c6312cd0              True      1          xpkg.upbound.io/upbound/provider-aws-iam:v1.1.1      Active     1           1               30m\nprovider-aws-s3-6ca829a5198b               True      1          xpkg.upbound.io/upbound/provider-aws-s3:v1.1.1       Active     1           1               30m\nupbound-provider-family-aws-7cc64a779806   True      1          xpkg.upbound.io/upbound/provider-family-aws:v1.2.1   Active                                 30m\n```\n\nNow after a while and some events (look at them in `k9s` for example):\n\n![](docs/upgrade-provider-k9s-events.png)\n\nSome time later the new Provider version should be the `Active` one:\n\n```shell\nk get providerrevisions\nNAME                                       HEALTHY   REVISION   IMAGE                                                STATE      DEP-FOUND   DEP-INSTALLED   AGE\nprovider-aws-ec2-3d66ea2d7903              True      1          xpkg.upbound.io/upbound/provider-aws-ec2:v1.2.1      Active     1           1               6m52s\nprovider-aws-eks-5021e69b327c              True      4          xpkg.upbound.io/upbound/provider-aws-eks:v1.2.1      Active     1           1               5m32s\nprovider-aws-eks-fbb6768e46c0              True      3          xpkg.upbound.io/upbound/provider-aws-eks:v1.1.1      Inactive   1           1               31m\nprovider-aws-iam-9565c6312cd0              True      1          xpkg.upbound.io/upbound/provider-aws-iam:v1.1.1      Active     1           1               31m\nprovider-aws-s3-6ca829a5198b               True      1          xpkg.upbound.io/upbound/provider-aws-s3:v1.1.1       Active     1           1               31m\nupbound-provider-family-aws-7cc64a779806   True      1          xpkg.upbound.io/upbound/provider-family-aws:v1.2.1   Active                                 31m\n```\n\nAnd luckily without any errors like mentioned above!\n\n\n\n### Using the EKS Nested Composition as Configuration Package\n\nI offloaded all the EKS Nested Composition as a separate repository, which publishes a Crossplane Configuration Package as OCI image: https://github.com/jonashackt/crossplane-eks-cluster\n\nWe should be able to use it via the following Configuration:\n\n```yaml\napiVersion: pkg.crossplane.io/v1\nkind: Configuration\nmetadata:\n  name: crossplane-eks-cluster\nspec:\n  package: ghcr.io/jonashackt/crossplane-eks-cluster:v0.0.2\n```\n\nLet's try to apply it to our cluster and use it:\n\n```shell\nkubectl apply -f upbound/provider-aws/apis/crossplane-eks-cluster.yaml\n```\n\n\n\n### GitOpsify API installation: Use EKS Cluster Configuration in Argo Application\n\nWe should create an Argo Application for our EKS Configuration package to make Argo manage it's versions for us (which also makes the EKS Configuration viewable in Argo UI)!\n\nTherefore let's create a new folder `argocd/crossplane-apis` and a new `Application` [`argocd/crossplane-apis/crossplane-apis.yaml`](argocd/crossplane-apis/crossplane-apis.yaml):\n\n```yaml\n# The ArgoCD Application for all Crossplane Managed Resources\n---\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: crossplane-apis\n  namespace: argocd\n  finalizers:\n    - resources-finalizer.argocd.argoproj.io\nspec:\n  project: default\n  source:\n    repoURL: https://github.com/jonashackt/crossplane-argocd\n    targetRevision: app-deployment\n    path: upbound/provider-aws/apis\n  destination:\n    namespace: default\n    server: https://kubernetes.default.svc\n  syncPolicy:\n    automated:\n      prune: true    \n    retry:\n      limit: 5\n      backoff:\n        duration: 5s \n        factor: 2 \n        maxDuration: 1m\n```\n\nNow we can apply this `crossplane-apis` Application to our ArgoCD:\n\n```shell\nkubectl apply -f argocd/crossplane-apis/crossplane-apis.yaml\n```\n\nThat's pretty cool: Now we see all of our installed APIs as Argo Apps:\n\n![](docs/eks-apis-as-argo-app.png)\n\n\n\n### Craft a Composite Resource Claim (XRC) to provision an EKS cluster\n\nNow we use our installed APIs to create a Claim in [`infrastructure/eks/deploy-target-eks.yaml`](infrastructure/eks/deploy-target-eks.yaml):\n\n```yaml\n# Use the spec.group/spec.versions[0].name defined in the XRD\napiVersion: k8s.crossplane.jonashackt.io/v1alpha1\n# Use the spec.claimName or spec.name specified in the XRD\nkind: KubernetesCluster\nmetadata:\n  namespace: default\n  name: deploy-target-eks\nspec:\n  id: deploy-target-eks\n  parameters:\n    region: eu-central-1\n    nodes:\n      count: 3\n  # Crossplane creates the secret object in the same namespace as the Claim\n  # see https://docs.crossplane.io/latest/concepts/claims/#claim-connection-secrets\n  writeConnectionSecretToRef:\n    name: eks-cluster-kubeconfig\n```\n\nDon't apply it directly, we'll create a Argo App in a second.\n\n\n### Crossplane Composite Resource Claims (XRCs) as Argo Application\n\nWe should also create a Argo App for our EKS cluster Composite Resource Claim to see our infrastructure beeing deployed visually :)\n\nTherefore we create the Application [`argocd/infrastructure/aws-eks.yaml`](argocd/infrastructure/aws-eks.yaml):\n\n```yaml\n# The ArgoCD Application for all Crossplane Managed Resources\n---\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: crossplane-eks\n  namespace: argocd\n  finalizers:\n    - resources-finalizer.argocd.argoproj.io\nspec:\n  project: default\n  source:\n    repoURL: https://github.com/jonashackt/crossplane-argocd\n    targetRevision: app-deployment\n    path: infrastructure/eks\n  destination:\n    namespace: default\n    server: https://kubernetes.default.svc\n  syncPolicy:\n    automated:\n      prune: true    \n    retry:\n      limit: 5\n      backoff:\n        duration: 5s \n        factor: 2 \n        maxDuration: 1m\n```\n\nNow **this** will deploy our EKS cluster using ArgoCD and our EKS Configuration Package based Nested EKS Composition https://github.com/jonashackt/crossplane-eks-cluster:\n\n```shell\nkubectl apply -f argocd/infrastructure/aws-eks.yaml\n```\n\n\n\n### Add the new EKS cluster as a new ArgoCD deploy target\n\n![](docs/add-crossplane-created-cluster-to-argocd.png)\n\n\nhttps://dev.to/thenjdevopsguy/registering-a-new-cluster-with-argocd-12mn\n\nhttps://www.padok.fr/en/blog/argocd-eks\n\nhttps://itnext.io/argocd-setup-external-clusters-by-name-d3d58a53acb0\n\n\nBefore using `argocd` CLI, be sure to have logged the CLI into the current argocd-server instance. Therefore have a port forward ready\n\n```shell\n$ kubectl port-forward -n argocd --address='0.0.0.0' service/argocd-server 8080:80\n\n$ argocd login localhost:8080 --username admin --password $(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=\"{.data.password}\" | base64 -d; echo) --insecure\n'admin:login' logged in successfully\nContext 'localhost:8080' updated\n```\n\nhttps://argo-cd.readthedocs.io/en/stable/user-guide/commands/argocd_cluster_add/\n\n```shell\nargocd cluster add deploy-target-eks\n```\n\nThis will add a few resources to the Target cluster like `ServiceAccount`, `ClusterRole` and `ClusterRoleBinding`:\n\n```shell\n$ argocd cluster add deploy-target-eks\nWARNING: This will create a service account `argocd-manager` on the cluster referenced by context `deploy-target-eks` with full cluster level privileges. Do you want to continue [y/N]? y\nINFO[0002] ServiceAccount \"argocd-manager\" already exists in namespace \"kube-system\" \nINFO[0002] ClusterRole \"argocd-manager-role\" updated    \nINFO[0002] ClusterRoleBinding \"argocd-manager-role-binding\" updated \nCluster 'https://736F91649BD7B7A70846AD9F8363EDA8.yl4.eu-central-1.eks.amazonaws.com' added\n```\n\nThe new cluster becomes visible in the Argo web ui also:\n\n![](docs/argocd-added-new-deploy-target-cluster.png)\n\n\n\n### Add new EKS clusters declaratively to ArgoCD\n\nIs there only the `argocd cluster add` command or could we achieve that using a manifest?\n\nhttps://github.com/argoproj/argo-cd/issues/8107\n\nMaybe the Crossplane ArgoCD Provider has the crucial Manifest for us? See https://github.com/crossplane-contrib/provider-argocd/issues/18 and https://marketplace.upbound.io/providers/crossplane-contrib/provider-argocd/v0.6.0/resources/cluster.argocd.crossplane.io/Cluster/v1alpha1\n\n\nYou might already wondered, what the Crossplane ArgoCD provider is about: https://marketplace.upbound.io/providers/crossplane-contrib/provider-argocd\n\nThats what the project README says https://github.com/crossplane-contrib/provider-argocd about it's purpose:\n\n\u003e Custom Resource Definitions (CRDs) that model Argo CD resources\n\nWith this we can create a [`Cluster`](https://marketplace.upbound.io/providers/crossplane-contrib/provider-argocd/v0.6.0/resources/cluster.argocd.crossplane.io/Cluster/v1alpha1) which is able to represent the EKS cluster we just created. This Cluster itself can be referenced again by an ArgoCD Application managing for example our Spring Boot application we finally want to deploy.\n\n\n#### Install Crossplane ArgoCD Provider\n\n\u003e The whole process might become more straightforward in the future: https://github.com/crossplane-contrib/provider-argocd/issues/14#issuecomment-1879101376\n\nSo let's install the Crossplane ArgoCD provider, which is a community contribution project. Thus we create the `crossplane-contrib` folder containing a `provider-argocd` folder, where the new Provider should reside as `provider-argocd.yaml` in the `provider` dir:\n\n```yaml\napiVersion: pkg.crossplane.io/v1\nkind: Provider\nmetadata:\n  name: provider-argocd\nspec:\n  package: xpkg.upbound.io/crossplane-contrib/provider-argocd:v0.6.0\n  packagePullPolicy: IfNotPresent # Only download the package if it isn’t in the cache.\n  revisionActivationPolicy: Automatic # Otherwise our Provider never gets activate \u0026 healthy\n  revisionHistoryLimit: 1\n```\n\nAs we want to manage the Provider also using Argo, we need to create a new Argo Application. It get's the same `argocd.argoproj.io/sync-wave: \"4\"` as the other providers in our setup:\n\n```yaml\n# The ArgoCD Application for all Crossplane Community contribution Providers needed in the setup\n---\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: crossplane-provider-contrib\n  namespace: argocd\n  labels:\n    crossplane.jonashackt.io: crossplane\n  finalizers:\n    - resources-finalizer.argocd.argoproj.io\n  annotations:\n    argocd.argoproj.io/sync-wave: \"4\"\nspec:\n  project: default\n  source:\n    repoURL: https://github.com/jonashackt/crossplane-argocd\n    targetRevision: app-deployment\n    path: crossplane-contrib\n  destination:\n    namespace: default\n    server: https://kubernetes.default.svc\n  # Using syncPolicy.automated here, otherwise the deployement of our Crossplane provider will fail with\n  # 'Resource not found in cluster: pkg.crossplane.io/v1/Provider:provider-aws-s3'\n  syncPolicy:\n    automated:\n      prune: true    \n    retry:\n      limit: 5\n      backoff:\n        duration: 5s \n        factor: 2 \n        maxDuration: 1m\n```\n\nApply it via the ususal bootstrap setup:\n\n```shell\nkubectl apply -f argocd/crossplane-eso-bootstrap.yaml\n```\n\nArgo should now list our new Provider:\n\n![](docs/crossplane-contrib-argocd-provider-installed-by-argo.png)\n\n\n\n#### Create ArgoCD user \u0026 RBAC role for Crossplane ArgoCD Provider\n\nAs stated in the docs https://github.com/crossplane-contrib/provider-argocd?tab=readme-ov-file#create-a-new-argo-cd-user we need to create an API token for the `ProviderConfig` of the Crossplane ArgoCD provider to use. To create the API token, we first need to create a new ArgoCD user.\n\nTherefore we enhance [the ConfigMap `argocd-cm`](argocd/install/argocd-cm-patch.yaml) again:\n\n```yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: argocd-cm\ndata:\n  ...\n  # add an additional local user with apiKey capabilities for provider-argocd\n  # see https://github.com/crossplane-contrib/provider-argocd?tab=readme-ov-file#getting-started-and-documentation\n  accounts.provider-argocd: apiKey      \n```\n\nAs [the ArgoCD docs about user management](https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/#local-usersaccounts) state this is not enough:\n\n\u003e \"each of those users will need additional RBAC rules set up, otherwise they will fall back to the default policy specified by policy.default field of the `argocd-rbac-cm` ConfigMap.\"\n\nSo we need to create another Kustomization patch for [the `argocd-rbac-cm` ConfigMap](argocd/install/argocd-rbac-cm-patch.yaml):\n\n```yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: argocd-rbac-cm\ndata:\n  # For the provider-argocd user we need to add an additional rbac-rule\n  # see https://github.com/crossplane-contrib/provider-argocd?tab=readme-ov-file#create-a-new-argo-cd-user\n  policy.csv: \"g, provider-argocd, role:admin\"      \n```\n\nDon't forget to add this patch into the []`kustomization.yaml`](argocd/install/kustomization.yaml)!\n\n\n#### Create API Token for Crossplane ArgoCD Provider\n\nFirst we need to access the `argocd-server` Service somehow. In the simplest manner we create a port forward:\n\n```shell\nkubectl port-forward -n argocd --address='0.0.0.0' service/argocd-server 8443:443\n```\n\nWe also need to have the ArgoCD password ready:\n\n```shell\nARGOCD_ADMIN_SECRET=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=\"{.data.password}\" | base64 -d; echo)\n```\n\nNow we create a temporary JWT token for the `provider-argocd` user we just created (we need to have [`jq`](https://jqlang.github.io/jq/) installed for this command to work):\n\n```shell\n# be sure to have jq installed via 'brew install jq' or 'pamac install jq' etc.\n\nARGOCD_ADMIN_TOKEN=$(curl -s -X POST -k -H \"Content-Type: application/json\" --data '{\"username\":\"admin\",\"password\":\"'$ARGOCD_ADMIN_SECRET'\"}' https://localhost:8443/api/v1/session | jq -r .token)\n```\n\nNow we finally create an API token without expiration that can be used by `provider-argocd`:\n\n```shell\nARGOCD_API_TOKEN=$(curl -s -X POST -k -H \"Authorization: Bearer $ARGOCD_ADMIN_TOKEN\" -H \"Content-Type: application/json\" https://localhost:8443/api/v1/account/provider-argocd/token | jq -r .token)\n```\n\nYou can double check in the ArgoCD UI at `Settings/Accounts` if the Token got created:\n\n![](docs/provider-argocd-api-token-created.png)\n\n\n\n#### Create Secret containing the ARGOCD_API_TOKEN\n\nhttps://github.com/crossplane-contrib/provider-argocd?tab=readme-ov-file#setup-crossplane-provider-argocd\n\nThe `ARGOCD_API_TOKEN` can be used to create a Kubernetes Secret for the Crossplane ArgoCD Provider:\n\n```shell\nkubectl create secret generic argocd-credentials -n crossplane-system --from-literal=authToken=\"$ARGOCD_API_TOKEN\"\n```\n\nI also added all these steps to a script [`create-argocd-api-token-secret.sh`](create-argocd-api-token-secret.sh) so that we're able to run all the steps without much thinking:\n\n```shell\n#!/usr/bin/env bash\nset -euo pipefail\n\necho \"### This Script will prepare a K8s Secret with a ArgoCD API Token for Crossplane ArgoCD Provider (be sure to have a service/argocd-server 8443:443 running before)\"\n\necho \"--- Extract ArgoCD password\"\nARGOCD_ADMIN_SECRET=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=\"{.data.password}\" | base64 -d; echo)\n\necho \"--- Create temporary JWT token for the provider-argocd user\"\nARGOCD_ADMIN_TOKEN=$(curl -s -X POST -k -H \"Content-Type: application/json\" --data '{\"username\":\"admin\",\"password\":\"'$ARGOCD_ADMIN_SECRET'\"}' https://localhost:8443/api/v1/session | jq -r .token)\n\necho \"--- Create ArgoCD API Token\"\nARGOCD_API_TOKEN=$(curl -s -X POST -k -H \"Authorization: Bearer $ARGOCD_ADMIN_TOKEN\" -H \"Content-Type: application/json\" https://localhost:8443/api/v1/account/provider-argocd/token | jq -r .token)\n\necho \"--- Already create a namespace for crossplane for the Secret (if not already exist, see https://stackoverflow.com/a/65411733/4964553)\"\nkubectl create namespace crossplane-system --dry-run=client -o yaml | kubectl apply -f -\n\necho \"--- Create Secret containing the ARGOCD_API_TOKEN for Crossplane ArgoCD Provider\"\nkubectl create secret generic argocd-credentials -n crossplane-system --from-literal=authToken=\"$ARGOCD_API_TOKEN\"\n```\n\nNow all the steps to create the Secret for the Crossplane argocd-provider can be run via a simple:\n\n```shell\nkubectl port-forward -n argocd --address='0.0.0.0' service/argocd-server 8443:443\nbash create-argocd-api-token-secret.sh\n```\n\n\u003e The `kubectl port-forward` command can be run in subshell appending ` \u0026` + `Ctrl-C` and beeing deleted after running create-argocd-api-token-secret.sh via `fg 1%` (where 1 is the subshell id, obtain via `jobs` command) + `Ctrl-C` (see https://stackoverflow.com/a/72983554/4964553 \u0026 https://www.baeldung.com/linux/foreground-background-process).\n\nOur GitHub Actions workflow now also integrates the Secret creation:\n\n```yaml\n      - name: Prepare Secret with ArgoCD API Token for Crossplane ArgoCD Provider\n        run: |\n          echo \"--- Access the ArgoCD server with a port-forward in the background, see https://stackoverflow.com/a/72983554/4964553\"\n          kubectl port-forward -n argocd --address='0.0.0.0' service/argocd-server 8443:443 \u0026\n          \n          echo \"--- Wait shortly to let the port forward come available\"\n          sleep 5\n\n          bash create-argocd-api-token-secret.sh\n```\n\nAs you can see we use a `sleep 5` timer here in order to let the `kubectl port-forward` to become ready. Otherwise will run into a `Error: Process completed with exit code 7.` like this::\n\n```shell\n--- Access the ArgoCD server with a port-forward in the background, see https://stackoverflow.com/a/72983554/4964553\n### This Script will prepare a K8s Secret with a ArgoCD API Token for Crossplane ArgoCD Provider (be sure to have a service/argocd-server 8443:443 running before)\n--- Extract ArgoCD password\n--- Create temporary JWT token for the provider-argocd user\nForwarding from 0.0.0.0:8443 -\u003e 8080\nError: Process completed with exit code 7.\n```\n\n\n#### Configure Crossplane ArgoCD Provider\n\nNow finally we're able to tell our Crossplane ArgoCD Provider where it should obtain the ArgoCD API Token from. Let's create a ProviderConfig at [`crossplane-contrib/provider-argocd/config/provider-config-argocd.yaml`](crossplane-contrib/provider-argocd/config/provider-config-argocd.yaml):\n\n\n```yaml\napiVersion: argocd.crossplane.io/v1alpha1\nkind: ProviderConfig\nmetadata:\n  name: argocd-provider\nspec:\n  credentials:\n    secretRef:\n      key: authToken\n      name: argocd-credentials\n      namespace: crossplane-system\n    source: Secret\n  insecure: true\n  plainText: false\n  serverAddr: argocd-server.argocd.svc:443\n```\n\nWe should also create [a ArgoCD Application for the ProviderConfig](argocd/crossplane-eso-bootstrap/crossplane-provider-argocd-config.yaml):\n\n```yaml\n# The ArgoCD Application for the Crossplane ArgoCD providers ProviderConfig\n---\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: crossplane-provider-argocd-config\n  namespace: argocd\n  labels:\n    crossplane.jonashackt.io: crossplane\n  finalizers:\n    - resources-finalizer.argocd.argoproj.io\n  annotations:\n    argocd.argoproj.io/sync-wave: \"5\"\nspec:\n  project: default\n  source:\n    repoURL: https://github.com/jonashackt/crossplane-argocd\n    targetRevision: app-deployment\n    path: crossplane-contrib/provider-argocd/config\n  destination:\n    namespace: default\n    server: https://kubernetes.default.svc\n  syncPolicy:\n    automated:\n      prune: true    \n    retry:\n      limit: 5\n      backoff:\n        duration: 5s \n        factor: 2 \n        maxDuration: 1m\n```\n\n\n#### Create a Cluster in ArgoCD referencing our Crossplane created EKS cluster\n\nNow we're where we wanted to be: We can finally create a Cluster in ArgoCD referencing the Crossplane created EKS cluster. Therefore we make use of the [Crossplane ArgoCD Providers Cluster CRD](https://marketplace.upbound.io/providers/crossplane-contrib/provider-argocd/v0.6.0/resources/cluster.argocd.crossplane.io/Cluster/v1alpha1) in our [`infrastructure/eks/cluster.yaml`](infrastructure/eks/cluster.yaml):\n\n```yaml\napiVersion: cluster.argocd.crossplane.io/v1alpha1\nkind: Cluster\nmetadata:\n  name: argo-reference-deploy-target-eks\n  labels:\n    purpose: dev\nspec:\n  forProvider:\n    config:\n      kubeconfigSecretRef:\n        key: kubeconfig\n        name: eks-cluster-kubeconfig # Secret containing our kubeconfig to access the Crossplane created EKS cluster\n        namespace: default\n    name: deploy-target-eks # name of the Cluster registered in ArgoCD\n  providerConfigRef:\n    name: argocd-provider\n```\n\n\u003e **Be sure** to provide the `forProvider.name` **AFTER** the `forProvider.config`, otherwise the name of the Cluster *will we overwritten by the EKS server address from the kubeconfig*!\n\nThe `providerConfigRef.name.argocd-provider` references our `ProviderConfig`, which gives the Crossplane ArgoCD Provider the rights (via our API Token) to change the ArgoCD Server configuration (and thus add a new Cluster).\n\nAs the docs state https://marketplace.upbound.io/providers/crossplane-contrib/provider-argocd/v0.6.0/resources/cluster.argocd.crossplane.io/Cluster/v1alpha1\n\n`kubeconfigSecretRef' is described at what we need: \n\n\u003e KubeconfigSecretRef contains a reference to a Kubernetes secret entry that contains a raw kubeconfig in YAML or JSON. \n\nThe Secret containing the exact EKS kubeconfig is named `eks-cluster-kubeconfig` by our EKS Configuration and resides in the `default` namespace.\n\nLet's create the Cluster manually for now:\n\n```shell\nkubectl apply -f infrastructure/eks/cluster.yaml\n```\n\nIf everything went correctly, a `kubectl get cluster` should state READY and SYNCED as `True`:\n\n```shell\nkubectl get cluster\nNAME                               READY   SYNCED   AGE\nargo-reference-deploy-target-eks   True    True     21s\n```\n\nAnd also in the ArgoCD UI you should find the newly registerd Cluster now at `Settings/Clusters`:\n\n![](docs/cluster-in-argocd-referencing-crossplane-created-eks-cluster.png)\n\n\nTo also have the ArgoCD Cluster configuration available as Argo Application, it's enough to have the `cluster.yaml` be placed together with the `deploy-target-eks.yaml` in `infrastructure/eks` directory. The Argo Application argocd/infrastructure/aws-eks.yaml will pick it up:\n\n![](docs/crossplane-argocd-provider-cluster-part-of-eks-argo-application.png)\n\nIt won't be available until the EKS cluster is fully deployed, thus producing some `CannotCreateExternalResource` events:\n\n![](docs/argocd-provider-cluster-cannotcreateexternalresource-events.png)\n\n\n### Deploy a app to the newly added target cluster\n\nNow we finally finally have the cluster dynamically referencable via the Crossplane ArgoCD Provider created Cluster object with the name `deploy-target-eks`! Let's try to use that in an Application deployment.\n\nIn order to deploy our example app https://github.com/jonashackt/microservice-api-spring-boot\n\nwe need the corresponding Kubernetes deployment manifests, provided by https://github.com/jonashackt/microservice-api-spring-boot-config\n\nHaving both in place, we can craft a matching ArgoCD Application:\n\n```yaml\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: microservice-api-spring-boot\n  namespace: argocd\n  labels:\n    crossplane.jonashackt.io: application\n  finalizers:\n    - resources-finalizer.argocd.argoproj.io\nspec:\n  project: default\n  source:\n    repoURL: https://github.com/jonashackt/microservice-api-spring-boot-config\n    targetRevision: HEAD\n    path: deployment\n  destination:\n    namespace: default\n    server: deploy-target-eks\n  syncPolicy:\n    automated:\n      prune: true    \n    retry:\n      limit: 5\n      backoff:\n        duration: 5s \n        factor: 2 \n        maxDuration: 1m\n```\n\nAs you can see we use our Cluster name `deploy-target-eks` as `spec.destination.server`.\n\nNow let's finally deploy our app via:\n\n```shell\nkubectl apply -f argocd/applications/microservice-api-spring-boot.yaml\n```\n\n\nBut we get the following error in Argo: \n\n```shell\ncluster 'deploy-target-eks' has not been configured\n```\n\nLooking [into the docs](https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#applications) we get the point we're missing:\n\n\u003e `destination` reference to the target cluster and namespace. For the cluster one of server or name can be used, [...] Under the hood when the server is missing, it is calculated based on the name and used for any operations.\n\nThus we need to use `spec.destination.name` instead of `spec.destination.server`. This will then look into Argo's Cluster list and should find our `deploy-target-eks`.\n\nNow the working manifest looks like this:\n\n```yaml\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n  name: microservice-api-spring-boot\n  namespace: argocd\n  labels:\n    crossplane.jonashackt.io: application\n  finalizers:\n    - resources-finalizer.argocd.argoproj.io\nspec:\n  project: default\n  source:\n    repoURL: https://github.com/jonashackt/microservice-api-spring-boot-config\n    targetRevision: HEAD\n    path: deployment\n  destination:\n    namespace: default\n    name: deploy-target-eks\n  syncPolicy:\n    automated:\n      prune: true    \n    retry:\n      limit: 5\n      backoff:\n        duration: 5s \n        factor: 2 \n        maxDuration: 1m\n```\n\n```shell\nkubectl apply -f argocd/applications/microservice-api-spring-boot.yaml\n```\n\n\nIf everything went fine, our App should be deployed by ArgoCD:\n\n![](docs/first-successful-application-deployment-to-target-eks-cluster.png)\n\n\nFinally a full cycle is possible - from full bootstrap of ArgoCD \u0026 Crossplane Managed cluster to target EKS cluster creation in AWS via Crossplane to configuring that one in Argo and finally deploying an App dynamically referencing this Cluster! \n\n\n\n\n\n# Links\n\nhttps://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/\n\nhttps://blog.upbound.io/argo-crossplane-managing-application-stack\n\nhttps://docs.upbound.io/concepts/mcp/control-plane-connector/\n\nhttps://blog.upbound.io/2023-09-26-product-updates\n\nhttps://morningspace.medium.com/using-crossplane-in-gitops-what-to-check-in-git-76c08a5ff0c4\n\nInfrastructure-as-Apps https://codefresh.io/blog/infrastructure-as-apps-the-gitops-future-of-infra-as-code/\n\nhttps://docs.upbound.io/spaces/git-integration/\n\nhttps://codefresh.io/blog/using-gitops-infrastructure-applications-crossplane-argo-cd/\n\nConfiguration drift in Tf: Terraform horror stories about incomplete/invalid state https://www.youtube.com/watch?v=ix0Tw8uinWs\n\n\n\n\n\nBADGES :\n\nhttps://argo-cd.readthedocs.io/en/stable/user-guide/status-badge/\n\n\n## App of Apps and ApplicationSets\n\nhttps://codefresh.io/blog/argo-cd-application-dependencies/\n\nhttps://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/#app-of-apps-pattern\n\nhttps://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/\n\nhttps://github.com/argoproj/argo-cd/discussions/11892\n\nhttps://github.com/christianh814/golist\n\n\n\n## Crossplane producer of Secrets\n\nhttps://docs.crossplane.io/knowledge-base/integrations/vault-as-secret-store/\n\n\u003e External Secret Stores are an alpha feature. They’re not recommended for production use. Crossplane disables External Secret Stores by default.\n\nhttps://github.com/crossplane/crossplane/blob/master/design/design-doc-external-secret-stores.md\n\n\u003e storing sensitive information in external secret stores is a common practice. Since applications running on K8S need this information as well, it is also quite common to sync data from external secret stores to K8S. There are quite a few tools out there that are trying to resolve this exact same problem. **However, Crossplane, as a producer of infrastructure credentials, needs the opposite, which is storing sensitive information to external secret stores.**\n\n--\u003e So this feature is NOT for retrieving secrets FROM external secret providers, BUT for storing secrets IN external secret providers!\n\nBut the External Secrets Operator has also PushSecrets https://external-secrets.io/latest/api/pushsecret/ which seem to do the same\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonashackt%2Fcrossplane-argocd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonashackt%2Fcrossplane-argocd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonashackt%2Fcrossplane-argocd/lists"}