https://github.com/openshift/cloud-credential-operator
Manage cloud provider credentials as Kubernetes CRDs
https://github.com/openshift/cloud-credential-operator
Last synced: 15 days ago
JSON representation
Manage cloud provider credentials as Kubernetes CRDs
- Host: GitHub
- URL: https://github.com/openshift/cloud-credential-operator
- Owner: openshift
- License: apache-2.0
- Created: 2018-12-21T19:50:29.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2024-10-21T23:20:30.000Z (6 months ago)
- Last Synced: 2024-10-22T02:05:04.269Z (6 months ago)
- Language: Go
- Size: 132 MB
- Stars: 62
- Watchers: 20
- Forks: 145
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# OpenShift Cloud Credential Operator
The cloud credential operator is a controller that will sync on
CredentialsRequest custom resources. CredentialsRequests allow OpenShift
components to request fine-grained credentials for a particular cloud provider
(as opposed to using the admin credentials, or elevated permissions granted via
instance roles).
- [Design Principles](#design-principles)
- [Cloud Providers](#cloud-providers)
- [Credentials Root Secret Formats](#credentials-root-secret-formats)
- [AWS](#aws)
- [Azure](#azure)
- [GCP](#gcp)
- [Kubevirt](#kubevirt)
- [OpenStack](#openstack)
- [Ovirt](#ovirt)
- [VSphere:](#vsphere)
- [Modes of Operation](#modes-of-operation)
- [1. Mint Mode](#1-mint-mode)
- [1.1 Mint Mode With Removal/Rotation Of Admin Credential](#11-mint-mode-with-removalrotation-of-admin-credential)
- [2. Passthrough Mode](#2-passthrough-mode)
- [3. Manual Credentials Management](#3-manual-credentials-management)
- [4. Short Lived Tokens](#4-short-lived-tokens)
- [Support Matrix](#support-matrix)
- [Developer Instructions](#developer-instructions)
- [Deploying in cluster](#deploying-in-cluster)
- [Credentials Requests](#credentials-requests)
- [AWS Sample](#aws-sample)
- [Azure Sample](#azure-sample)
- [Instructions to add new cloud provider](#instructions-to-add-new-cloud-provider)
- [For OpenShift Second Level Operators](#for-openshift-second-level-operators)
## Design Principles
* The controller should be able to run in either a cluster itself, or in a centralized management cluster, most likely alongside [Hive](https://github.com/openshift/hive).
* Controller expects access to a set of credentials we refer to as the "admin" credentials.
* If the admin credentials are missing, but all credentials requests are fulfilled and valid, this is considered a [valid state](#11-mint-mode-with-removalrotation-of-admin-credential) (i.e. the admin creds were removed from the cluster after use).
* If the admin credentials are able to create additional credentials, we will create fine grained permissions as defined in the credentials request (best practice).
* If the admin credentials cannot create additional credentials, but do themselves fulfill the requirements of the credentials request, they will be used (with logged warnings and a condition on the credentials request).
* If the admin credentials fulfill neither of the above requirements, the controller will fail to generate the credentials, report failure back to the Cluster Version Operator, and thus block upgrading. The installer will also perform this check early to inform the user their cluster will not function.
## Building binary from source
```bash
~[cloud-credential-operator] (master)$ go build -o $(go env GOPATH)/bin/ccoctl cmd/ccoctl/main.go
```
## Cloud Providers
Currently the operator supports AWS, Azure, GCP, KubeVirt, OpenStack. oVirt and VMWare.
### Credentials Root Secret Formats
Each cloud provider utilizes a credentials root secret in the kube-system
namespace (by convention), which is then used to satisfy all
CredentialsRequests and create their respective Secrets. (either by minting new
credentials (mint mode), or by copying the credentials root secret (passthrough
mode))
The format for the secret varies by cloud, and is also used for each CredentialsRequest Secret.
### AWS
```yaml
apiVersion: v1
kind: Secret
metadata:
namespace: kube-system
name: aws-creds
data:
aws_access_key_id: Base64encodeAccessKeyID
aws_secret_access_key: Base64encodeSecretAccessKey
```
### Azure
```yaml
apiVersion: v1
kind: Secret
metadata:
namespace: kube-system
name: azure-credentials
data:
azure_subscription_id: Base64encodeSubscriptionID
azure_client_id: Base64encodeClientID
azure_client_secret: Base64encodeClientSecret
azure_tenant_id: Base64encodeTenantID
azure_resource_prefix: Base64encodeResourcePrefix
azure_resourcegroup: Base64encodeResourceGroup
azure_region: Base64encodeRegion
```
### GCP
```yaml
apiVersion: v1
kind: Secret
metadata:
namespace: kube-system
name: gcp-credentials
data:
service_account.json: Base64encodeServiceAccount
```
### Kubevirt
```yaml
apiVersion: v1
kind: Secret
metadata:
namespace: kube-system
name: kubevirt-credentials
data:
kubeconfig: Base64encodeKubeconfig
```
### OpenStack
```yaml
apiVersion: v1
kind: Secret
metadata:
namespace: kube-system
name: openstack-credentials
data:
clouds.yaml: Base64encodeCloudCreds
clouds.conf: Base64encodeCloudCredsINI
cacert: Base64encodeCACert
```
### Ovirt
```yaml
apiVersion: v1
kind: Secret
metadata:
namespace: kube-system
name: ovirt-credentials
data:
ovirt_url: Base64encodeURL
ovirt_username: Base64encodeUsername
ovirt_password: Base64encodePassword
ovirt_insecure: Base64encodeInsecure
ovirt_ca_bundle: Base64encodeCABundle
```
### VSphere:
```yaml
apiVersion: v1
kind: Secret
metadata:
namespace: kube-system
name: vsphere-creds
data:
{{VCenter.username}}: Base64encodeUsername
{{VCenter.password}}: Base64encodePassword
```
Source of templates:
* https://github.com/openshift/installer/blob/master/data/data/manifests/openshift/cloud-creds-secret.yaml.template
## Modes of Operation
### 1. Mint Mode
The default and recommended best practice for running OpenShift is to run the installer with an admin level cloud credential. The admin credential is stored in kube-system namespace and then used by the cloud credential operator to process the CredentialRequests in the cluster and create new users for each with fine grained permissions.
Pros:
* Each cluster component has only the permissions it needs.
* Automatic ongoing reconciliation for cloud credentials including upgrades, which may require additional credentials or permissions.
Cons:
* Requires admin credential storage in a cluster kube-system secret. (However, if a user has access to all secrets in your cluster, you are severely compromised regardless.)
Supported clouds: AWS, GCP
#### 1.1 Mint Mode With Removal/Rotation Of Admin Credential
In this mode a user installs OpenShift with an admin credential per the normal mint mode, but removes the admin credential Secret from the cluster after installation. The cloud credential operator makes its own request for a read-only credential that allows it to verify if all CredentialsRequests have their required permissions, thus the admin credential is not needed unless something needs to be changed (e.g. on upgrade). Once removed the associated credential could then be destroyed on the underlying cloud if desired.
Prior to upgrade, the admin credential should be restored. In the future upgrade may be blocked if the credential is not present (see the Secret formats above).
Pros:
* Admin credential is not stored in the cluster permanently and does not need to be long-lived.
Cons:
* Still requires admin credential in the cluster for brief periods of time.
* Requires manually reinstating the Secret with admin credentials for each upgrade.
Supported clouds: AWS, GCP
### 2. Passthrough Mode
In this mode a user installs OpenShift with a single credential that *is not* an admin and cannot mint additional credentials, but itself has enough permissions to perform the installation as well as all operations needed by all components in the cluster. The cloud credential operator then shares this credential to each component.
Your passthrough mode credential will need to be manually maintained if CredentialsRequests change over time as the cluster is upgraded. This should be checked prior to every upgrade, and in the future you may be required to confirm you have done so if a change in CredentialsRequests is detected.
By default the permissions needed only for installation are required, however it is possible to reduce the permissions on your credential after install to just what is needed to run the cluster (as defined by the CredentialsRequests in the current release image). See the [secret formats](#credentials-root-secret-formats) above for details on how to do this.
Pros:
* Does not require installing or running with an admin credential.
Cons:
* Includes broad permissions only needed at install time, unless manual action is taken to reduce permissions after install.
* Credential permissions may need to be manually updated prior to any upgrade.
* Each component has permissions used by all other components.
Supported clouds: AWS, GCP, Azure, VMWare, OpenStack, oVirt, KubeVirt
### 3. Manual Credentials Management
In this mode a user manually performs the job of the cloud credential operator. This requires examining the CredentialsRequests in an OpenShift 4 release image, creating credentials in the underlying cloud provider, and finally creating Kubernetes Secrets in the correct namespaces to satisfy all CredentialsRequests for the cluster's cloud provider.
Pros:
* Admin credential never stored in the cluster.
* Each cluster component has only the permissions it needs.
Cons:
* Manual process required for install and every upgrade to reconcile permissions with the new release image.
Supported clouds: AWS
[Documentation](./docs/mode-manual-creds.md)
### 4. Short Lived Tokens
OpenShift can be configured to use short-lived credentials for different in-cluster components. It enables an authentication flow allowing a component to assume a cloud role resulting in short-lived credentials. It also automates requesting and refreshing of credentials using an OpenID Connect (OIDC) Identity Provider. OpenShift can sign ServiceAccount tokens trusted by the OIDC provider, which can be projected into a Pod and used for authentication.
Pros:
* Admin credentials are never stored in the cluster.
* Each cluster component has only the permissions it needs.
* Credentials for each cluster component are rotated periodically.
Cons:
* Requires an additional cloud infrastructure setup from the user. The [ccoctl](./docs/ccoctl.md) tool can assist in the setup process.
* Push-button upgrades will not work as the cluster no longer has the admin credentials to mint credentials
Read more about supported clouds by clicking on the links below:
* [AWS](./docs/sts.md)
* [Azure](./docs/azure_workload_identity.md)
* [GCP](./docs/gcp_workload_identity.md)
## Support Matrix
Cloud | Mint | Mint + Remove Admin Cred | Passthrough | Manual | Token
--- | --- | --- | --- | --- | ---
AWS | Y | 4.4+ | Y | 4.3+ | 4.8+
Azure | N1 | N | Y | Y | 4.14+
GCP | Y | 4.7+ | Y | Y | 4.10+
IBMCloud | N | N | N | Y | N
KubeVirt | N | N | Y | N | N
Nutanix | N | N | N | Y | N
OpenStack | N | N | Y | N | N
oVirt | N | N | Y | N | N
VMWare | N | N | Y | N | N
1 Mint mode was previously supported, but with the sunsetting of the Azure Active Directory Graph API, Mint mode support on Azure has since been [removed](./docs/azure-mint-mode-removal.md).
## Developer Instructions
Login to a cluster with admin credentials:
```
$ make install
$ make run
```
NOTE: To keep the in-cluster versions of the code from conflicting with your local copy, you should scale down the deployments for cloud-credential-operator and cluster-version-operator
```
$ kubectl scale -n openshift-cluster-version deployment.v1.apps/cluster-version-operator --replicas=0
$ kubectl scale -n openshift-cloud-credential-operator deployment.v1.apps/cloud-credential-operator --replicas=0
```
As an alternative to disabling the cluster verison operator entirely, you can add the CCO Deployment as an unmanaged object into the clusterversion resource:
```yaml
spec:
overrides:
- kind: Deployment
group: apps/v1
name: cloud-credential-operator
namespace: openshift-cloud-credential-operator
unmanaged: true
```
### Deploying in cluster
1. `export IMG=quay.io/dgoodwin/cloud-credential-operator:latest`
* You can upload to a personal repo if you wish to build images from source.
1. `make buildah-push`
1. `make deploy`
Cred Minter should now be running in the `openshift-cloud-credential-operator` namespace.
## Credentials Requests
The primary custom resource used by this operator is the CredentialsRequest, which allows cluster components to request fine-grained credentials.
A CredentialRequest spec consists of:
1. `secretRef` - Points to the secret where the credentials should be stored once generated. Can be in a separate namespace from the CredentialsRequest where it can be used by pods. If that namespace does not yet exist, the controller will immediately sync when it is created.
2. `providerSpec` - Contains the [cloud provider specific credentials specification](pkg/apis/cloudcredential/v1).
Once created, assuming admin credentials are available, the controller will provision e.g. a user, access key, and user policy in AWS. The access and secret key will be stored in the target secret specified above.
You can freely edit a CredentialsRequest to adjust permissions and the controller will reconcile those changes out to the respective user policy (assuming valid admin credentials still exist).
### AWS Sample
```yaml
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
name: openshift-image-registry
namespace: openshift-cloud-credential-operator
spec:
secretRef:
name: installer-cloud-credentials
namespace: openshift-image-registry
providerSpec:
apiVersion: cloudcredential.openshift.io/v1
kind: AWSProviderSpec
statementEntries:
- effect: Allow
action:
- s3:CreateBucket
- s3:DeleteBucket
resource: "*"
```
### Azure Sample
```yaml
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
name: openshift-image-registry
namespace: openshift-cloud-credential-operator
spec:
secretRef:
name: installer-cloud-credentials
namespace: openshift-image-registry
providerSpec:
apiVersion: cloudcredential.openshift.io/v1
kind: AzureProviderSpec
roleBindings:
- role: Storage Account Contributor
- role: Storage Blob Data Contributor
```
List of Azure built-in roles: https://docs.microsoft.com/en-us/azure/role-based-access-control/built-in-roles
## Instructions to add new cloud provider
Please refer [this](./docs/mode-manual-creds.md) documentation for adding a new provider.
## For OpenShift Second Level Operators
1. Add CredentialsRequests objects to your CVO manifests and deployed via the release payload. Please do not create them in operator code as we want to use the release manifest for auditing and dynamically checking permissions.
1. The cred operator launches early (runlevel 30) so should be available when your component's manifests are applied.
1. Your CredentialsRequests should be created in the `openshift-cloud-credential-operator` namespace.
1. Your component should tolerate the credentials secret not existing immediately.
1. Your component should tolerate the credentials secret periodically being rotated.