{"id":15672795,"url":"https://github.com/developer-guy/kcd-munich-2024-demo","last_synced_at":"2025-05-05T21:45:57.436Z","repository":{"id":246665454,"uuid":"820810302","full_name":"developer-guy/kcd-munich-2024-demo","owner":"developer-guy","description":"No More YAML Soup: Taking Control with Dagger's Pipeline-as-Code Philosophy","archived":false,"fork":false,"pushed_at":"2024-07-25T18:40:25.000Z","size":8576,"stargazers_count":17,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-03T09:14:33.384Z","etag":null,"topics":["crane","dagger","flux","go","k3d","kubernetes","registry"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/developer-guy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-27T08:15:02.000Z","updated_at":"2025-02-01T19:02:07.000Z","dependencies_parsed_at":"2024-06-29T14:29:23.028Z","dependency_job_id":"4588987e-616f-423e-9d91-e8b7adf46417","html_url":"https://github.com/developer-guy/kcd-munich-2024-demo","commit_stats":null,"previous_names":["developer-guy/kcd-munich-2024-demo"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developer-guy%2Fkcd-munich-2024-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developer-guy%2Fkcd-munich-2024-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developer-guy%2Fkcd-munich-2024-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developer-guy%2Fkcd-munich-2024-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/developer-guy","download_url":"https://codeload.github.com/developer-guy/kcd-munich-2024-demo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252582366,"owners_count":21771647,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crane","dagger","flux","go","k3d","kubernetes","registry"],"created_at":"2024-10-03T15:32:01.542Z","updated_at":"2025-05-05T21:45:57.384Z","avatar_url":"https://github.com/developer-guy.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kcd-munich-2024-demo\n\nNo More YAML Soup: Taking Control with Dagger's Pipeline-as-Code Philosophy\n\n\u003e https://www.kcdmunich.de/schedule/\n\n## Goal\n\nThe goal of this repository is to demonstrate how to use Dagger in a Kubernetes environment to manage the deployment of applications.\n\nFor this demonstration, we are going to use a simple Go application that exposes a REST API. The application is going to be deployed in a Kubernetes cluster using Dagger with a pipeline-as-code approach. \n\nFor the sake of simplicity, we are going to use a local Kubernetes cluster managed by `k3d` and `Flux` to manage the GitOps workflow.\n\nTalk is cheap - show me the code!\n\n### Prerequisites\n\n* [flux CLI](https://fluxcd.io/flux/installation/#install-the-flux-cli)\n* Kubernetes (_we are using [minikube](https://minikube.sigs.k8s.io/docs/start/) in this demonstration but any Kubernetes distribution would be fine_)\n* [dagger](https://docs.dagger.io/quickstart/cli/#install-the-dagger-cli)\n* [crane](https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md#installation)\n\n### Setup\n\nFirst things first, we are going to create a new Kubernetes cluster using `minikube`:\n\n```bash\nmake minikube\n```\n\nVerify the cluster is up and running:\n\n```bash\nkubectl cluster-info\n```\n\nIf everything is working as expected, we can proceed to the next step.\n\nAfter that, we are going to install `flux` in the cluster:\n\n```bash\nflux bootstrap github \\\n  --owner=developer-guy \\\n  --repository=kcd-munich-2024-demo \\\n  --branch=master \\\n  --path=./clusters/dagger-in-action \\\n  --personal\n```\n\n\u003e **Note:** The `--personal` flag is used to create a personal access token for the GitHub repository, this command is going to ask you to provide your PAT (Personal Access Token), to fix this you can use `gh auth token |export GITHUB_TOKEN=$(cat /dev/stdin)` trick to set the token, btw, gh is the GitHub CLI where you can install it from [here](https://cli.github.com).\n\n\u003e **BONUS:** There is an alternative way to bootstrap Flux with Git-less approach using OCI Registry,  [@stefanprodan](https://x.com/stefanprodan) who is the maintainer of the Flux started an RFC for this feature, you can find more details [here](https://github.com/fluxcd/flux2/pull/4749)\n\nAfter the `flux` is installed, you will see a bunch of files in the `clusters/dagger-in-action` directory created, these files are the Kubernetes manifests that are going to be deployed in the cluster by Flux. So, we are going to deploy the Dagger in the cluster by creating the manifests in that folder., now its time to deploy the Dagger in the cluster, as we follow the GitOps approach, we are going to use the `flux` to deploy the Dagger in the cluster.\n\nDagger provides a Helm chart to deploy the Dagger in the cluster, you can find more information how to set up a Dagger in Kubernetes environment [here](https://docs.dagger.io/integrations/kubernetes/).\n\nNow, lets create a Helm repository for the Dagger Helm chart:\n\n```bash\nflux create source helm dagger-repo \\\n  --url=oci://registry.dagger.io \\\n  --export \u003e ./clusters/dagger-in-action/dagger-oci-source.yaml\n```\n\nAfter that, we are going to create a HelmRelease to deploy the Dagger in the cluster:\n\n```bash\n flux create helmrelease dagger \\\n  --source=HelmRepository/dagger-repo.flux-system \\\n  --chart=dagger-helm \\\n  --target-namespace=dagger \\\n  --create-target-namespace \\\n  --export \u003e ./clusters/dagger-in-action/dagger-helm-release.yaml\n```\n\nThen run reconcile command to apply the changes immediately without waiting the default interval:\n\n```bash\n$ flux reconcile source git flux-system\n► annotating GitRepository flux-system in flux-system namespace\n✔ GitRepository annotated\n◎ waiting for GitRepository reconciliation\n...\n```\n\nAfter a few seconds, you should see the Dagger is deployed in the cluster:\n\n```bash\n$ kubectl get pods -n dagger\nNAME                                     READY   STATUS    RESTARTS   AGE\ndagger-dagger-dagger-helm-engine-5bmrz   1/1     Running   0          100s\n```\n\nNow, we have the Dagger deployed in the cluster, we can proceed to the next step.\n\nLet's connect to the Dagger engine locally:\n\n```bash\nDAGGER_ENGINE_POD_NAME=\"$(kubectl get pod \\\n    --selector=name=dagger-dagger-dagger-helm-engine --namespace=dagger \\\n    --output=jsonpath='{.items[0].metadata.name}')\"\nexport DAGGER_ENGINE_POD_NAME\n\n_EXPERIMENTAL_DAGGER_RUNNER_HOST=\"kube-pod://$DAGGER_ENGINE_POD_NAME?namespace=dagger\"\nexport _EXPERIMENTAL_DAGGER_RUNNER_HOST\n```\n\nLet's ensure that the Dagger engine connection is working:\n\n```bash\n$ echo $_EXPERIMENTAL_DAGGER_RUNNER_HOST\nkube-pod://dagger-dagger-dagger-helm-engine-5bmrz?namespace=dagger\n```\n\nThen, call the simple hello module by [@solomonstre](https://x.com/solomonstre):\n\n```bash\ndagger -m github.com/shykes/daggerverse/hello@v0.1.2 call hello\n```\n\nIf you see the output of the above command as `hello, world!`, then the connection is working as expected and we can proceed to the next step.\n\nLet's deploy the application:\n\n\u003e **Note:** We already built the `0.1.0` version of the application for the demonstration purposes.\n\n```shell\nflux create kustomization hello-server \\\n  --source=GitRepository/flux-system \\\n  --path=\"./kustomize\" \\\n  --prune=true \\\n  --interval=10m \\\n  --timeout=1m \\\n  --target-namespace=default \\\n  --namespace=flux-system --export \u003e ./clusters/dagger-in-action/hello-server-kustomization.yaml\n```\n\nThen it will be creating a deployment for the application:\n\n```shell\n$ kubectl port-forward pod/$(kubectl get pods -l app=hello-server -o jsonpath='{.items[0].metadata.name}') 8080\nForwarding from 127.0.0.1:8080 -\u003e 8080\nForwarding from [::1]:8080 -\u003e 8080\n```\n\nThen, you can test the application:\n\n```shell\n$ http :8080\nHTTP/1.1 200 OK\nContent-Length: 18\nContent-Type: text/plain; charset=utf-8\nDate: Sat, 29 Jun 2024 08:29:18 GMT\n\nHello World 0.1.0!\n```\n\nNoice!\n\n### Deploy the Go application\n\nNow, we are going to deploy the Go application in the cluster using Dagger modules `apko` and `melange` where you can find the details of these modules in [@tuananh_org](https://x.com/tuananh_org)'s [daggerverse](https://github.com/tuananh/daggerverse).\n\n\u003e [Daggerverse](https://daggerverse.dev) is a place where you can discover and share modules full of Dagger functions encapsulating the community's devops knowledge.\n\nBut let me show you the code of the module to basically show you what is happening under the hood:\n\n\u003cdetails\u003e\n\n\u003csummary\u003eMelange module\u003c/summary\u003e\n\n```go\ntype Melange struct{}\n\nfunc (m *Melange) Build(\n\tctx context.Context,\n\tmelangeFile *File,\n\tworkspaceDir *Directory,\n// +default=\"amd64\"\n\tarch string,\n// +default=\"latest\"\n\timageTag string,\n) *Directory {\n\t// generate public/private key pair\n\tcli := dag.Pipeline(\"melange-build\")\n\n\tctr := cli.Container().From(fmt.Sprintf(\"cgr.dev/chainguard/melange:%s\", imageTag)).\n\t\tWithWorkdir(\"/workspace\").\n\t\tWithExec([]string{\n\t\t\t\"keygen\"})\n\n\tf, _ := melangeFile.Name(ctx)\n\n\tc := cli.Container().\n\t\tFrom(fmt.Sprintf(\"cgr.dev/chainguard/melange:%s\", imageTag)).\n\t\tWithMountedDirectory(\"/workspace\", workspaceDir).\n\t\tWithDirectory(\"/workspace\", ctr.Directory(\"/workspace\")).\n\t\tWithWorkdir(\"/workspace\").\n\t\tWithExec([]string{\n\t\t\t\"build\", fmt.Sprintf(\"%s\", f), \"--arch\", arch, \"--signing-key=melange.rsa\"},\n\t\t\tContainerWithExecOpts{\n\t\t\t\tExperimentalPrivilegedNesting: true,\n\t\t\t\tInsecureRootCapabilities:      true,\n\t\t\t})\n\n\tpk := c.File(filepath.Join(\"/workspace\", \"melange.rsa.pub\"))\n\n\treturn c.Directory(\"/workspace\").WithFile(\".\", pk)\n}\n```\n\n\u003c/details\u003e\n\n\u003e If you would like to learn how to develop your own Dagger modules, please check the [Dagger documentation](https://docs.dagger.io/quickstart/daggerize).\n\nFor the the who don't know what is `apko` and `melange`, these are the newest tools by Chainguard that are used to create an OCI images. `apko` is a tool to create an OCI image from a apks you created with `melange` from scracth.\n\nI highly recommend to check the `apko` and `melange` tools, they are really cool tools to create an OCI images, [here](https://edu.chainguard.dev/open-source/build-tools/), also, [@adrianmouat](https://x.com/adrianmouat) who is a DevRel from Chainguard gave a presentation about `Building Container Images the Modern Way` where he talked about the tools that you can use to create OCI images today, you can find the video [here](https://www.youtube.com/watch?v=nZLz0o4duRs).\n\n\nFirst, we need to create an apk for the go application with melange:\n\n```shell\ndagger -m \"github.com/tuananh/daggerverse/melange@5e6b42cb28fc18757def43ef0997adf752b329b1\" call build --melange-file melange.yaml --workspace-dir=. --arch=aarch64 directory --path=. export --path=.\n```\n\nYou will see that `packages` folder, `melange.rsa` and `melange.rsa.pub` are created:\n\n```shell\n$ ls -latr\n...\nmelange.rsa\nmelange.rsa.pub\npackages\n$ tree -L5 packages\npackages\n└── aarch64\n    ├── APKINDEX.json\n    ├── APKINDEX.tar.gz\n    └── hello-server-0.1.0-r0.apk\n```\n\nNow its time to build the OCI image with apko module:\n\n```shell\ndagger call -m \"github.com/tuananh/daggerverse/apko@5e6b42cb28fc18757def43ef0997adf752b329b1\" build --apko-file apko.yaml --source=. --keyring-append=melange.rsa.pub --arch=arm64 --packages-append=packages --image=ghcr.io/developer-guy/hello-server --tag v2 export --path=\".\" --allowParentDirPath\n```\n\nYou should see `apko.tar` is getting created, then let's push it to the registry:\n\n```shell\n$ dagger -m \"../ci/\" call container-push --source=. --container-as-tarball apko.tar\n```\n\u003e **Note:** The `REGISTRY_PASSWORD` is the password of the registry, you can set it as an environment variable ie. `pbpaste|export REGISTRY_PASSWORD=$(cat /dev/stdin)`.\n\nOnce we pushed the image to the registry, let's test it before we can deploy it in the cluster:\n\n```shell\n$ crane ls ghcr.io/developer-guy/hello-server\n0.1.0\n\n$ docker container run --rm -p 8080:8080 ghcr.io/developer-guy/hello-server:0.1.0\n[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.\n\n[GIN-debug] [WARNING] Running in \"debug\" mode. Switch to \"release\" mode in production.\n - using env:   export GIN_MODE=release\n - using code:  gin.SetMode(gin.ReleaseMode)\n\n[GIN-debug] GET    /                         --\u003e main.main.func1 (3 handlers)\n[GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.\nPlease check https://pkg.go.dev/github.com/gin-gonic/gin#readme-don-t-trust-all-proxies for details.\n[GIN-debug] Environment variable PORT is undefined. Using port :8080 by default\n[GIN-debug] Listening and serving HTTP on :8080\n```\n\nIn a second terminal window, let's test the application:\n\n```shell\n$ http :8080\nHTTP/1.1 200 OK\nContent-Length: 18\nContent-Type: text/plain; charset=utf-8\nDate: Sat, 29 Jun 2024 08:14:45 GMT\n\nHello World 0.1.0!\n```\n\nNoice! The application is working as expected, now we can deploy it in the cluster.\n\nTo deploy the application in the cluster, we are going to use our own Dagger module, let's see what we have as functions in our module:\n\n```shell\n$ dagger -m \"ci/\" functions\nName             Description\ncommit-push      CommitPush local changes to the Git repository using the SSH Key.\ncontainer-push   ContainerPush pushes the container tarball to the Docker daemon using the Docker CLI.\nedit             Edit the kustomization file in the source directory with the given image tag\n```\n\nWe are going to use the `edit` function to edit the `kustomization.yaml` file with the image tag:\n\n```shell\ndagger-m \"ci/\" call edit --source=manifests --tag=\"ghcr.io/developer-guy/hello-server:$(VERSION)\" export --path=\"manifests\"\n```\n\nThis command will edit the `kustomization.yaml` file with the image tag, then we can deploy the application in the cluster by commiting and pushing these changes:\n\n```shell\ndagger -m \"ci/\" call commit-push --source=. --key=/Users/batuhanapaydin/.ssh/id_ed25519 export --path=.git/\n```\n\nThen, you should see the changes are getting applied in the cluster right after you trigger re-conciliation:\n\n```shell\n$ flux reconcile source git flux-system\n```\n\nAfter a few seconds, you should see the new version of the application is deployed in the cluster, do the same test as we did before:\n\n```shell\n$ kube port-forward pod/$(kubectl get pods -l app=hello-server -o jsonpath='{.items[0].metadata.name}') 8080\nForwarding from 127.0.0.1:8080 -\u003e 8080\nForwarding from [::1]:8080 -\u003e 8080\n```\n\nThen, you can test the application:\n\n```shell\n$ http :8080\nHTTP/1.1 200 OK\nContent-Length: 18\nContent-Type: text/plain; charset=utf-8\nDate: Sat, 29 Jun 2024 08:29:18 GMT\n\nHello World 0.2.0!\n```\n\nYay!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeveloper-guy%2Fkcd-munich-2024-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeveloper-guy%2Fkcd-munich-2024-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeveloper-guy%2Fkcd-munich-2024-demo/lists"}