{"id":26445588,"url":"https://github.com/cisco-open/cluster-registry-controller","last_synced_at":"2025-03-18T11:19:32.841Z","repository":{"id":37824520,"uuid":"478789018","full_name":"cisco-open/cluster-registry-controller","owner":"cisco-open","description":"An operator that automatically synchronizes Kubernetes resources across multiple clusters","archived":false,"fork":false,"pushed_at":"2024-06-26T23:18:50.000Z","size":3273,"stargazers_count":22,"open_issues_count":3,"forks_count":8,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-06-27T02:58:55.500Z","etag":null,"topics":["golang","kubernetes","kubernetes-operator","multicluster"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cisco-open.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-04-07T01:49:46.000Z","updated_at":"2024-06-26T23:18:53.000Z","dependencies_parsed_at":"2024-02-08T02:28:39.487Z","dependency_job_id":"5e9bd170-3201-4ecd-92cb-01df62f20072","html_url":"https://github.com/cisco-open/cluster-registry-controller","commit_stats":null,"previous_names":[],"tags_count":104,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cisco-open%2Fcluster-registry-controller","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cisco-open%2Fcluster-registry-controller/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cisco-open%2Fcluster-registry-controller/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cisco-open%2Fcluster-registry-controller/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cisco-open","download_url":"https://codeload.github.com/cisco-open/cluster-registry-controller/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244208595,"owners_count":20416110,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","kubernetes","kubernetes-operator","multicluster"],"created_at":"2025-03-18T11:19:32.031Z","updated_at":"2025-03-18T11:19:32.832Z","avatar_url":"https://github.com/cisco-open.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cluster registry controller\n\nThe cluster registry controller helps to form a group of Kubernetes clusters and synchronize\nany K8s resources across those clusters arbitrarily.\n\n## Cluster Registry API\n\nThe `api` directory defines a lightweight Kubernetes Custom Resource Definition API\nfor defining a list of clusters and associated metadata in a K8s environment.\n\n## Defined CRDs\n\n1. `Cluster`: defines a Kubernetes cluster.\n2. `ResourceSyncRule`: defines a sync rule based on which Kubernetes resources are synced across clusters.\n3. `ClusterFeature`: defines a feature name, which can be used by a `ResourceSyncRule` resource to define which clusters\n   to sync from a given Kubernetes resource.\n\n## Overview\n\nThe `Cluster` resource represents a Kubernetes cluster.\nThe cluster registry controller fills the status with cluster related metadata for the `Cluster` CR.\n\nThe controller is mostly useful in multi-cluster scenarios.\nIn this scenario, the cluster registry controller is deployed to all Kubernetes clusters.\nThe same `Cluster` CRs should be placed on all participating Kubernetes clusters as well.\nAlso, the credentials for all clusters should be distributed to all clusters (these are usually stored in k8s secrets).\n\n\u003e The cluster registry controller syncs the `Cluster` CR and related secret resources across clusters\n\u003e to help bootstrap the cluster group itself.\n\nYou can define your own `ResourceSyncRule` resources to sync k8s resources between these clusters.\n\nIn such a multi-cluster setup, here is how the cluster registry controller works:\n- The controller only writes to the local cluster where it is deployed to\n- The controller only reads from peer clusters\n\nBy default, the required resources are kept in sync between all clusters.\nIt can be further adjusted, from which clusters and to which clusters certain resource should be synced.\n\nThe cluster registry controller works in a fully-distributed topology, there is no leader or single point of failure in the system.\n\n## Networking requirements\n\nThe cluster registry controller instances running on the clusters must be able to reach the API server of every other\ncluster in the cluster group, so every cluster can read the relevant resources from the other clusters.\n\nThe cluster registry controller pod connects directly to Kubernetes API server of the peer clusters.\nThis works automatically, if the API servers are publicly available.\nOtherwise, configure a reachable endpoint for them in the [Cluster CR spec](https://github.com/cisco-open/cluster-registry-controller/blob/master/api/v1alpha1/cluster_types.go#L60-L63).\n(For security reasons, we recommend making the API server addresses available only from the IP ranges of the peer clusters.)\n\n## Quickstart\n\n### Form cluster group with two clusters\n\n1. Install cluster registry controller on the first cluster. The following command installs the cluster registry controller on your cluster, creates a Cluster CR with the name `FIRST-CLUSTER-NAME`, and it also creates a secret that holds a Kubeconfig with read access to this cluster.\n\n    ```\n    helm install --namespace=cluster-registry --create-namespace cluster-registry-controller deploy/charts/cluster-registry --set localCluster.name=\u003cFIRST-CLUSTER-NAME\u003e --kube-context \u003cFIRST-CLUSTER-CONTEXT\u003e\n    ```\n\n    \u003e Tip: Use the `--set controller.apiServerEndpointAddress=\u003cPUBLIC-API-SERVER-ADDRESS\u003e` flag, if your Kubernetes cluster API returns private ip for the api server.\n\n2. Install cluster registry controller on the second cluster. This command installs the cluster registry controller on your cluster, creates a Cluster CR with the name `SECOND-CLUSTER-NAME`, and it also creates a secret that holds a Kubeconfig with read access to this cluster.\n\n    ```\n    helm install --namespace=cluster-registry --create-namespace cluster-registry-controller deploy/charts/cluster-registry --set localCluster.name=\u003cSECOND-CLUSTER-NAME\u003e --kube-context \u003cSECOND-CLUSTER-CONTEXT\u003e\n    ```\n\n3. Copy/paste Cluster and secret resources from first-\u003esecond and second-\u003efirst cluster. The secret is needed so that the cluster registry controller of one cluster can read from the other cluster.\n\n    From first cluster to second cluster:\n    ```\n    kubectl get cluster \u003cFIRST-CLUSTER-NAME\u003e --context \u003cFIRST-CLUSTER-CONTEXT\u003e -o yaml | pbcopy \u0026\u0026 pbpaste | kubectl apply --context \u003cSECOND-CLUSTER-CONTEXT\u003e -f -\n\n    kubectl get secret -n cluster-registry \u003cFIRST-CLUSTER-NAME\u003e --context \u003cFIRST-CLUSTER-CONTEXT\u003e -o yaml | pbcopy \u0026\u0026 pbpaste | kubectl apply --context \u003cSECOND-CLUSTER-CONTEXT\u003e -f -\n    ```\n\n    From second cluster to first cluster:\n    ```\n    kubectl get cluster \u003cSECOND-CLUSTER-NAME\u003e --context \u003cSECOND-CLUSTER-CONTEXT\u003e -o yaml | pbcopy \u0026\u0026 pbpaste | kubectl apply --context \u003cFIRST-CLUSTER-CONTEXT\u003e -f -\n\n    kubectl get secret -n cluster-registry \u003cSECOND-CLUSTER-NAME\u003e --context \u003cSECOND-CLUSTER-CONTEXT\u003e -o yaml | pbcopy \u0026\u0026 pbpaste | kubectl apply --context \u003cFIRST-CLUSTER-CONTEXT\u003e -f -\n    ```\n\n4. Check the status of the Cluster CRs. Note the following points:\n\n    - Both Cluster CRs should show `Synced` state.\n    - On the first cluster, the `\u003cFIRST-CLUSTER-NAME\u003e` Cluster CR should be type local in the status, the\n    `\u003cSECOND-CLUSTER-NAME\u003e` should be peer.\n    - On the second cluster, the `\u003cSECOND-CLUSTER-NAME\u003e` Cluster CR should be type local in the status, the\n    `\u003cFIRST-CLUSTER-NAME\u003e` should be peer.\n    \n    \u003e The type in the Cluster status is determined by the clusterID field in the Cluster spec and by the \n      `kube-system` namespace uid. If they match, the cluster is local, otherwise it is a peer cluster.\n\nThe cluster group is successfully formed at this point.\n\n### Attach additional clusters to the group\n\n1. Install the cluster registry controller on the new cluster as shown above.\n\n2. Choose one cluster from the existing cluster group and perform the Cluster and secret resource swap with the new cluster:\n\n   From first cluster to third cluster:\n    ```\n    kubectl get cluster \u003cFIRST-CLUSTER-NAME\u003e --context \u003cFIRST-CLUSTER-CONTEXT\u003e -o yaml | pbcopy \u0026\u0026 pbpaste | kubectl apply --context \u003cTHIRD-CLUSTER-CONTEXT\u003e -f -\n\n    kubectl get secret -n cluster-registry \u003cFIRST-CLUSTER-NAME\u003e --context \u003cFIRST-CLUSTER-CONTEXT\u003e -o yaml | pbcopy \u0026\u0026 pbpaste | kubectl apply --context \u003cTHIRD-CLUSTER-CONTEXT\u003e -f -\n    ```\n\n   From third cluster to first cluster:\n    ```\n    kubectl get cluster \u003cTHIRD-CLUSTER-NAME\u003e --context \u003cTHIRD-CLUSTER-CONTEXT\u003e -o yaml | pbcopy \u0026\u0026 pbpaste | kubectl apply --context \u003cFIRST-CLUSTER-CONTEXT\u003e -f -\n    \n    kubectl get secret -n cluster-registry \u003cTHIRD-CLUSTER-NAME\u003e --context \u003cTHIRD-CLUSTER-CONTEXT\u003e -o yaml | pbcopy \u0026\u0026 pbpaste | kubectl apply --context \u003cFIRST-CLUSTER-CONTEXT\u003e -f -\n    ```\n   \n    **You don't need to do the Cluster and secret resource swap between the new cluster and \n    any other cluster in the cluster group.**\n    The cluster registry controller automatically syncs these Cluster and secret resources between the clusters in the \n    cluster group and once a new cluster is added to the cluster group in any of the clusters, then it will be synced\n    between the other clusters automatically.\n\n    So, if you have 10 clusters in a cluster group, you'll still only need to do the Cluster and secret resource swap once\n    with one cluster from the cluster group, the rest should synchronize automatically.\n\n5. Check the status of the Cluster CRs:\n\n   All Cluster CRs should show `Synced` state.\n   If so, then the cluster group is successfully expanded.\n\n### ResourceSyncRule example usage\n\n#### Sync everywhere\n\n1. Create a sample secret on the third cluster, which will be copied around:\n\n    ````yaml\n    apiVersion: v1\n    kind: Secret\n    metadata:\n      name: test-secret\n      namespace: cluster-registry\n    data: {}\n    ````\n\n2. Create a `ResourceSyncRule` on the first cluster to synchronize the secret to all clusters:\n\n    ```yaml\n    apiVersion: clusterregistry.k8s.cisco.com/v1alpha1\n    kind: ResourceSyncRule\n    metadata:\n      name: test-secret-sink\n    spec:\n      groupVersionKind:\n        kind: Secret\n        version: v1\n      rules:\n      - match:\n        - objectKey:\n            name: test-secret\n            namespace: cluster-registry\n    ```\n   \n    This `ResourceSyncRule` resource itself and the `secret` resource as well should appear shortly on all \n    clusters of the cluster group.\n    \n    At this point, if a secret from any of the clusters (except from the one where it originates from) is deleted\n    or modified, it will be synced back immediately by the cluster registry controller.\n\n#### Sync to a set of clusters\n\nCluster registry controller can be configured to sync only to specific clusters in the cluster group (instead of all\nof them). To do that, you must add an annotation to the cluster where you don't want to sync to.\n\n1. Add the following annotation to the `ResourceSyncRule` on the first cluster:\n\n    ```yaml\n    annotations:\n      cluster-registry.k8s.cisco.com/resource-sync-disabled: \"true\"\n    ```\n\n2. Delete the `ResourceSyncRule` from the second cluster.\n\n    The `ResourceSyncRule` resource will not be recreated because of the annotation, which was just added.\n\n    \u003e If the annotation is not added as described in the previous step, then the `ResourceSyncRule` will be recreated.\n\n3. Delete the `test-secret` from the second cluster.\n\n    The secret will not be recreated because the `ResourceSyncRule` resource does not exist on the second cluster.\n\n#### Sync from a set of clusters\n\nCluster registry controller can be configured, to only sync from specific clusters in the cluster group (instead of all\nof them). To do that, you must create a `ClusterFeature` resource on the clusters where you want to sync from and add a\n`clusterFeatureMatch` field to the `ResourceSyncRule` resources on the clusters where you want to sync to.\n\n1. Add the following field to the `ResourceSyncRule` spec on the first cluster:\n\n    ```yaml\n    clusterFeatureMatch:\n    - featureName: test-secret-feature\n    ```\n   \n    This causes that the secret will only be synced from clusters where there are `ClusterFeature` resources defined.\n\n    \u003e At this point, there is no `ClusterFeature` present on any cluster, so if the secret would be deleted now from \n      the first cluster, it would not be recreated.\n\n2. Apply the following `ClusterFeature` to the third cluster:\n\n    ```yaml\n    apiVersion: clusterregistry.k8s.cisco.com/v1alpha1\n    kind: ClusterFeature\n    metadata:\n      name: test-secret-source\n    spec:\n      featureName: test-secret-feature\n    ```\n\n3. Delete the `test-secret` from the first cluster.\n\n    It should be recreated now, because it can sync the secret from the third cluster.\n\n## RBAC considerations\n\nThe cluster registry controller only writes to local clusters and only reads from peer clusters.\nBy default, it has access to read `namespace`, `node` and `secret` resources.\nThe quickstart example worked, because the controller was allowed to read the secret from the remote cluster.\n\nIf other resources should be synced, then the RBAC rules of the operator should be expanded.\nThe [ClusterRole aggregation](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#aggregated-clusterroles) \nfeature is used to achieve this conveniently.\n\n- On the cluster, where the resources are read from (usually where `ClusterFeature` resources are present)\n  a ClusterRole should be defined with the correct read roles and the following label should be added:\n  \n  ```\n  labels:\n    cluster-registry.k8s.cisco.com/reader-aggregated: \"true\"\n  ```\n  \n- On the cluster, where the resources are written to (usually where `ResourceSyncRule` resources are present)\n  a ClusterRole should be defined with the correct write roles and the following label should be added:\n\n  ```\n  labels:\n    cluster-registry.k8s.cisco.com/controller-aggregated: \"true\"\n  ```\n\n## Contributing\n\nIf you find this project useful, help us:\n\n- Support the development of this project and star this repo! :star:\n- If you use Cluster registry controller, add yourself to the list of [adopters](ADOPTERS.md).:metal: \u003cbr\u003e\n- Help new users with issues they may encounter :muscle:\n- Send a pull request with your new features and bug fixes :rocket: Check out the [developer docs](docs/development.md) for that.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcisco-open%2Fcluster-registry-controller","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcisco-open%2Fcluster-registry-controller","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcisco-open%2Fcluster-registry-controller/lists"}