https://github.com/link-/actions-at-scale-ghoh
GitHub Office Hours - Adopting GitHub Actions at scale in the Enterprise
https://github.com/link-/actions-at-scale-ghoh
aks azure github-actions helm kubernetes tutorial-demos
Last synced: 7 months ago
JSON representation
GitHub Office Hours - Adopting GitHub Actions at scale in the Enterprise
- Host: GitHub
- URL: https://github.com/link-/actions-at-scale-ghoh
- Owner: Link-
- License: mit
- Archived: true
- Created: 2021-12-27T09:33:49.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2023-04-08T09:31:47.000Z (over 2 years ago)
- Last Synced: 2025-02-14T05:24:50.292Z (8 months ago)
- Topics: aks, azure, github-actions, helm, kubernetes, tutorial-demos
- Language: Dockerfile
- Homepage:
- Size: 41 KB
- Stars: 54
- Watchers: 8
- Forks: 36
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# ⚠️ IMPORTANT NOTICE
**This project has not been updated since its release. Some of the instructions shared below (and in the videos) might not be applicable any more. Use your best judgement when following these instructions!**
With the release of the [Autoscaling Runner Scale Set mode](https://github.com/actions/actions-runner-controller/tree/master/docs/preview/gha-runner-scale-set-controller) you are highly encouraged to use it as opposed to these legacy options.
# Adopting GitHub Actions at scale in the Enterprise
[](./LICENSE)
> GitHub Office Hours: Adopting GitHub Actions at scale in the Enterprise
This repository contains the scripts and configuration files for the GitHub Actions at scale in the Enterprise office hours video series.
## Agenda
*All episodes were followed by a live Q&A.*
**Episode 1:**
- Setup and configure AKS
- Deploy and attach an Application Gateway as our Ingress Controller
**Episode 2:**
- Configure cert-manager for TLS termination
- Create a GitHub App
- Install & configure actions-runner-controller
- Demonstrate auto-scaling
**Episode 3:**
- Configure our Web Application Firewall (WAF)
- Enable and use Docker in Docker
- Create a custom self-hosted runner image
- Install multiple actions-runner-controllers for different namespaces
## Reference Architecture
## Pre-Requisites
- Azure Subscription with at least [Contributor](https://docs.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#contributor) + [User Access Administrator](https://docs.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#user-access-administrator) built-in roles. You will be performing role assignments when using the Azure Application Gateway with AKS and when integrating Azure Container Registry with AKS.
## Folder Structure
```text
.
├── LICENSE
├── README.md
├── actions-runner-controller
│ ├── alt-namespace
│ │ ├── autoscale_webhook.yaml
│ │ └── values.yaml.example
│ ├── autoscale_webhook.yaml
│ ├── dind_deployment.yaml
│ ├── go-runners-autoscale_webhook.yaml
│ ├── multi_namespace_values.yaml
│ └── values.yaml.example
├── apps
│ └── test-app.yaml
├── cert-manager
│ ├── cluster-issuer-prod.yaml
│ └── cluster-issuer-staging.yaml
├── custom-runners
│ └── Dockerfile
├── ingress
│ ├── altns-ingress.yaml
│ ├── ingress-tls-runners.yaml
│ ├── ingress-tls.yaml
│ ├── ingress.yaml
│ └── multi-namespaces-ingress.yaml
└── sample-workflows
├── custom-runner.yaml
├── docker_job.yaml
├── matrix_jobs.yaml
├── multi_job.yaml
└── single_job.yml
```
- `actions-runner-controller/`: contains the actions-runner-controller configuration and helm chart values file for the default namespace
- `actions-runner-controller/alt-namespace/`: contains the actions-runner-controller configuration and helm chart values file for the alternate namespace
- `apps/`: contains the sample applications used for sanity checks
- `cert-manager/`: contains the cert-manager configuration
- `custom-runners/`: contains the Dockerfile of a custom runner image
- `ingress/`: contains the ingress controller configuration
- `sample-workflows/`: contains the sample workflows used for sanity checks
## Setup
:warning: *All the below assumes you are running `Bash`.*
### Install az cli
```bash
# Refresh packages
apt-get update
apt-get upgrade
# From:
# https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt
sudo apt-get update
sudo apt-get install ca-certificates curl apt-transport-https lsb-release gnupg
# Download the microsoft signing keys
curl -sL https://packages.microsoft.com/keys/microsoft.asc |
gpg --dearmor |
sudo tee /etc/apt/trusted.gpg.d/microsoft.gpg > /dev/null
# Add the Azure CLI software repository:
AZ_REPO=$(lsb_release -cs)
echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" |
sudo tee /etc/apt/sources.list.d/azure-cli.list
# Update repository information and install the azure-cli package:
sudo apt-get update
sudo apt-get install azure-cli
```
### Install kubectl (latest stable version)
```bash
# Download the latest release
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
# Download the kubectl checksum file:
curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
# Validate the kubectl binary against the checksum file:
echo "$( -k
# !!! IMPORTANT !!!
#
# Update the values.yaml file with the appropriate values
#
# !!! IMPORTANT !!!
# Install actions-runner-controller
# Add the actions-runner-controller Helm chart repository
helm repo add \
actions-runner-controller \
https://actions-runner-controller.github.io/actions-runner-controller
# Update your local Helm chart repository cache
helm repo update
# Install the actions-runner-controller Helm chart
helm upgrade --install \
-f actions-runner-controller/values.yaml \
--namespace default \
--create-namespace \
--wait \
actions-runner-controller \
actions-runner-controller/actions-runner-controller
#
# Update the ingress/ingress-tls-runners.yaml with the appropriate
# hostname and the actions-runner-controller service name
#
# Update the ingress controller
kubectl apply -f ingress/ingress-tls-runners.yaml --namespace default
# !!! IMPORTANT !!!
#
# Update the actions-runner-controller/autoscale_webhook.yaml file with your organization's name
#
# !!! IMPORTANT !!!
# Create a new runner deployment
kubectl apply -f actions-runner-controller/autoscale_webhook.yaml --namespace default
# Execute some sample runs
```
### Start / Stop AKS
```bash
# Stop AKS
az aks stop \
--resource-group GitHubActionsRunners \
--name GitHubActionsRunnersK8sCluster
# Stop application gateway
az network application-gateway stop \
--resource-group GitHubActionsRunners \
--name GitHubActionsRunnersAPGW
# Start AKS
az aks start \
--resource-group GitHubActionsRunners \
--name GitHubActionsRunnersK8sCluster
# Start application gateway
az network application-gateway start \
--resource-group GitHubActionsRunners \
--name GitHubActionsRunnersAPGW
# !!! IMPORTANT !!!
#
# Remember, when you start the application gateway, you need to
# reapply the Ingress configuration, otherwise you'll get 502 errors
#
# !!! IMPORTANT !!!
```
## Advanced Configuration
### Configuring our Web Application Firewall (WAF)
Described in the video
### Enable and use Docker in Docker
This is as simple as updating the runner deployment with these properties:
```yaml
spec:
replicas: 0
template:
spec:
organization: Inner-Sanctum
labels:
- azure
- docker
image: summerwind/actions-runner-dind
dockerdWithinRunnerContainer: true
```
Then create the new DinD enabled deployment:
```bash
kubectl apply -f actions-runner-controller/dind_deployment.yaml --namespace default
```
### Creating custom self-hosted runner images
Start by editing `custom-runners/Dockerfile` to include the dependencies you need in your runners:
```Dockerfile
FROM summerwind/actions-runner:latest
# This will be a good place to add your CA bundle if you're using
# a custom CA.
# If you have proxy configurations, you can also add them here
# Change the work dir to tmp because these are disposable files
WORKDIR /tmp
# EXAMPLE
# Install a stable version of Go
# and verify checksum of the tarball
#
# Go releases URL: https://go.dev/dl/
#
RUN curl -OL https://go.dev/dl/go1.17.6.linux-amd64.tar.gz && \
echo "231654bbf2dab3d86c1619ce799e77b03d96f9b50770297c8f4dff8836fc8ca2 go1.17.6.linux-amd64.tar.gz" | sha256sum -c - && \
sudo tar -C /usr/local -xvf go1.17.6.linux-amd64.tar.gz && \
export PATH=$PATH:/usr/local/go/bin && \
go version
```
Then we need to tag and push the image to our Azure Container Registry:
```bash
# Fetch ACR's FQDN
ACR_URL=$(az acr show \
--resource-group GitHubActionsRunners \
--name GitHubActionsOHACR \
--query loginServer \
--output tsv) \
&& echo $ACR_URL
# Login to ACR
az acr login --name GitHubActionsOHACR
# Verify we're logged in
cat ~/.docker/config.json | jq ".auths"
# You need to be in the root directory of this repository for this to work
# Build and tag the new runner image
docker build --tag $ACR_URL/runner-image:go1.17.6 --file $(pwd)/custom-runners/Dockerfile .
# List the image and verify the tag
docker image list
# Push the image to ACR
docker push $ACR_URL/runner-image:go1.17.6
# !!! IMPORTANT !!!
#
# Edit the actions-runner-controller/go-runners-autoscale_webhook.yaml to point
# to the correct container image and tag
#
# !!! IMPORTANT !!!
# Now we need to create a new deployment for the custom runners:
kubectl apply -f actions-runner-controller/go-runners-autoscale_webhook.yaml --namespace default
# Run a test with the custom-runner.yaml workflow
```
### Setup multiple actions-runner-controllers in different namespaces
```bash
# Create the new namespace
kubectl create namespace altns
# !!! IMPORTANT !!!
#
# In order to configure multiple actions-runner-controllers in different
# namesapces we have to introduce changes to these keys in the values.yaml
# Replace "altns" with the name of your namespace
#
# - nameOverride: "altns"
# - fullnameOverride: "altns-actions-runner-controller"
# - scope.singleNamespace: true
# - scope.watchNamespace: "altns"
# - githubWebhookServer.nameOverride: "altns"
# - githubWebhookServer.fullnameOverride: "altns-github-webhook-server"
#
# !!! IMPORTANT !!!
# Update the previous installation of actions-runner-controller in the
# default namespace to support multi-namespace installations
helm upgrade --install \
-f actions-runner-controller/multi_namespace_values.yaml \
--namespace default \
--wait \
actions-runner-controller \
actions-runner-controller/actions-runner-controller
# Install a new actions-runner-controller in the altns namespace
helm upgrade --install \
-f actions-runner-controller/alt-namespace/values.yaml \
--namespace altns \
--wait \
actions-runner-controller \
actions-runner-controller/actions-runner-controller
# !!! IMPORTANT !!!
#
# Update enterprise and organization settings to allow the "Default" group
# to be used by all organizations and repositories
#
# !!! IMPORTANT !!!
# Deploy new ingress configurations
kubectl apply -f ingress/multi-namespaces-ingress.yaml --namespace default
# altns ingress configuration
kubectl apply -f ingress/altns-ingress.yaml --namespace altns
# !!! IMPORTANT !!!
#
# Configure the Enterprise webhooks manually
# https://github.com/enterprises/:ENTERPRISE_NAME/settings/hooks
#
# !!! IMPORTANT !!!
# Deploy actions-runner-controllers
kubectl apply -f actions-runner-controller/alt-namespace/autoscale_webhook.yaml
```
### NUKE THE SETUP
This will destroy the resource group and all the services associated with it (i.e. everything created above).
```bash
az group delete --name GitHubActionsRunners
```
## References
- **Adopting GitHub Actions for Enterprise Guide:**
- GitHub Enterprise Cloud:
- GitHub Enterprise Server:
- **Azure AKS docs:**
- **Azure Application Gateway docs:**
- **Azure CLI docs:**