Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/dominodatalab/restorepvc_blue_green_eks

Last synced: 30 days ago
JSON representation

Host: GitHub
URL: https://github.com/dominodatalab/restorepvc_blue_green_eks
Owner: dominodatalab
Created: 2023-07-12T15:07:09.000Z (over 1 year ago)
Default Branch: main
Last Pushed: 2023-07-13T01:19:27.000Z (over 1 year ago)
Last Synced: 2023-08-07T03:05:39.265Z (over 1 year ago)
Size: 10.7 KB
Stars: 0
Watchers: 4
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

## Introduction

The intent of this page is to demonstrate the following on EKS based Domino installation

### What

Restore the PVC of an un-committed and stopped workspace in Domino A into Domino B and mount it on a Pod in Domino B. The result should be that the changes in the workspace made in Domino A are visible in pod of Domino B.

### Why

FINRA has been requesting Blue/Green deployments. Currently, when they upgrade, they make a fresh (Green) Domino install and migrate the metadata over. Except the workspace meta-data. This PoC aims to demonstrates a pre-requisite for successfully moving a pvc of a stopped workspace from Blue Domino to the Green version of Domino.

### Limitations

While we will demonstrate that the PVC can be successfully moved (even if its is encrypted by different KMS Keys), and mounted into a Pod in the compute namespace of Green Domino, more metadata is needed to restore the workspace.

## Installation

Start two Domino installations using the Fleet Command. For example

- https://blue.cs.domino.tech

- https://green.cs.domino.tech

In both installations of Domino follow the steps in this Amazon article to install the Amazon EBS CSI driver feature: Kubernetes Volume Snapshots . 0The steps are ambiguous and incomplete. The missing parts are clarified below

On a machine where you can connect to both K8s instances using kubectl perform a git pull on the following git repos

```
cd $INSTALL_FOLDER
git clone https://github.com/kubernetes-csi/external-snapshotter.git
git clone https://github.com/kubernetes-sigs/aws-ebs-csi-driver.git
```

This gives us to sub-folders
```
$INSTALL_FOLDER/aws-ebs-csi-driver
$INSTALL_FOLDER/external-snapshotter
```

Install the snapshot functionality
```
cd $INSTALL_FOLDER/external-snapshotter/client/config/crd
kubectl apply -k .
cd $INSTALL_FOLDER/external-snapshotter/deploy/kubernetes/snapshot-controller
kubectl apply -k .
```
The gist of how this process works is below:

> The volume snapshot controller watches Kubernetes Volume Snapshot CRD objects and triggers CreateSnapshot/DeleteSnapshot against a CSI endpoint. For more information, see CSI Snapshotter on GitHub.

The process of installing the CSI driver includes creating an Identity and Access Management (IAM) policy and a role that to use as part of the creation of a service account. AWS provides a managed policy for this called AmazonEBSCSIDriverPolicy. Details on how to create an IAM role, service account, and attach the required AWS managed policy can be found here.

```
eksctl create iamserviceaccount \
--name ebs-csi-controller-sa \
--namespace kube-system \
--cluster $my-cluster-id \
--role-name $AmazonEKS_EBS_CSI_DriverRole \
--role-only \
--attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
--approve
```

When you run this step you will be asked to add the OIDC provider attached to your EKS clusters to IAM.

Once the IAM Role and Service Accounts have been created, you can continue the installation Amazon EKS add-on support for the Amazon EBS CSI following the documentation. Missing parts

Do not forget to attach the Role you created above to the add-on

In this step you are doing the following:

a. Creating an AWS role

b. Attaching a managed policy AmazonEBSCSIDriverPolicy to this role

c. Attaching a custom policy which provides this role with the ability to use the KMS keys of both Domino installations. An example of such a policy is sameerw_AmazonEBSCSIDriverPolicy

At this point we are using [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) . Because the above process will create a K8s SA bound to this role

The full policy is
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kms:CreateGrant",
"kms:ListGrants",
"kms:RevokeGrant"
],
"Resource": [
"arn:aws:kms:us-west-2::key/b85b982b-527f-4018-a5bb-xxxxxxx"
],
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
},
{
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:us-west-2::key/b85b982b-527f-4018-a5bb-xxxxxxx"
]
},
{
"Effect": "Allow",
"Action": [
"kms:CreateGrant",
"kms:ListGrants",
"kms:RevokeGrant"
],
"Resource": [
"arn:aws:kms:us-west-2::key/f255c64f-a773-49bd-9b66-yyyyyyyy"
],
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
},
{
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:us-west-2::key/f255c64f-a773-49bd-9b66-yyyyyyyy"
]
},
{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKeyWithoutPlaintext",
"kms:CreateGrant"
],
"Resource": "*"
}
]
}
```
The part missing in the Amazon docs is adding this section which could be arguably tightened
```
{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKeyWithoutPlaintext",
"kms:CreateGrant"
],
"Resource": "*"
}
```
And note that there are two sections similar to the snippet below, one for each Domino installation participating in this snapshot to pvc migrations. You can add as many participating Domino’s here
```
{
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:us-west-2::key/b85b982b-527f-4018-a5bb-xxxxxxx"
]
},
{
"Effect": "Allow",
"Action": [
"kms:CreateGrant",
"kms:ListGrants",
"kms:RevokeGrant"
],
"Resource": [
"arn:aws:kms:us-west-2::key/f255c64f-a773-49bd-9b66-yyyyyyyy"
],
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
},
{
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:us-west-2::key/f255c64f-a773-49bd-9b66-yyyyyyyy"
]
}
```
5. Install the volume snapshot class
```
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/examples/kubernetes/snapshot/manifests/classes/snapshotclass.yaml
```

or
```
kubectl apply -f $INSTALL_FOLDER/aws-ebs-csi-driver/examples/kubernetes/snapshot/manifests/classes/
```
This completes the installation.

## Demo Walkthrough

1. Create an unsaved workspace in the blue deployment

2. In the Blue Domino installation start a workspace in the quick-start project using the standard DAD.

3. Inside the workspace create a file /mnt/bluegreen.txt and add a line to the file

```
echo 'This is from blue' > /mnt/bluegreen.txt
```

But do not commit the changes and STOP the workspace. This is important. Sometimes the unsaved contents are not flushed until the workspace is stopped.

4. Create a Snapshot of this pvc in the Blue Domino

There are many ways to identify the pvc of the stopped workspace. One method is described here. But it should have the same is as your workspace. Ex. `workspace-64ac594fbe2c046835547935`

Identify the un-commited workspace pvc in the blue deployment

Open `vi aws-ebs-csi-driver/examples/kubernetes/snapshot/manifests/snapshot/yaml` and modify and apply it or run the following script
```
export pvc=workspace-64ac594fbe2c046835547935
cat <> /data/out.txt; sleep 10000000; done"]
volumeMounts:
- name: persistent-storage
mountPath: /data
volumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: workspace-64ac594fbe2c046835547935
```
and apply
```
kubectl -n domino-compute apply -f pod.yaml
## When the pod is running
kubectl -n domino-compute exec -it app -- cat /data/mnt/bluegreen.txt
#You should see the content from the blue domino here
```

## This is not enough

We have the pvc in the Green Domino and can see the changes from the Blue Domino. We have been able to successfully import a KMS encrypted PVC from Blue to Green Domino.

However, this is not enough. There is metadata and possibly internal git artifacts that need to be changed to fully restore the workspace.

However, this was the first step