{"id":15725882,"url":"https://github.com/ndrean/k8_cluster","last_synced_at":"2026-05-18T02:32:15.906Z","repository":{"id":124425338,"uuid":"492110653","full_name":"ndrean/k8_cluster","owner":"ndrean","description":"Demo for making a cluster of Elixir nodes (releases \u0026 iex)","archived":false,"fork":false,"pushed_at":"2022-05-16T03:20:26.000Z","size":28706,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-20T22:36:51.297Z","etag":null,"topics":["cluster","distributed","elixir","kubernetes"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ndrean.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-14T04:22:27.000Z","updated_at":"2022-05-14T22:59:28.000Z","dependencies_parsed_at":"2023-08-07T16:39:01.261Z","dependency_job_id":null,"html_url":"https://github.com/ndrean/k8_cluster","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ndrean/k8_cluster","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrean%2Fk8_cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrean%2Fk8_cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrean%2Fk8_cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrean%2Fk8_cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ndrean","download_url":"https://codeload.github.com/ndrean/k8_cluster/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrean%2Fk8_cluster/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33162610,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-17T22:39:12.733Z","status":"online","status_checked_at":"2026-05-18T02:00:06.436Z","response_time":71,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cluster","distributed","elixir","kubernetes"],"created_at":"2024-10-03T22:24:48.344Z","updated_at":"2026-05-18T02:32:15.890Z","avatar_url":"https://github.com/ndrean.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# K8Cluster\n\n[A word](http://blog.plataformatec.com.br/2019/10/kubernetes-and-the-erlang-vm-orchestration-on-the-large-and-the-small/)\n\nSetup of a cluster of connected Erlang nodes within a Kubernetes cluster with `libcluster`.\nWe have no endpoint, no HTTP communication in this app. The only communication will be within the TCP connection between e-nodes encapsulated in a k8-pod, and k8 itself via it's TCP connections.\n\nTwo dockerfiles will produce respectively a release (**Dockerfile.rel**, image of 20 Mb) and an IEX shell (**Dockerfile.mix**, image of 150Mb) from the same app.\nWe spin-up several pods of the release, and one shell to interact with the releases via `:rpc.call(remote_node, Module, function, args)`.\n\nThe library `libcluster` will give automatic connection of the releases with the topology `Cluster.Strategy.Kubernetes`, with:\n\n- `kubernetes_ip_lookup_mode: :pods`\n- `mod: :ip`,\n- and a `ServiceAccount` set.\n\n## Namespace\n\nAdded a namespace to isolate the whole. The namespace has to be added in the app for `libcluster` and set by `kubectl`. Create the ns with `kubectl` and set the namespace with `kubens`:\n\n```bash\n\u003ekubectl create namespace stage\nnamespace/stage created\n\n\u003e kubens stage\nContext \"minikube\" modified.\nActive namespace is \"stage\".\n\n\u003e kubens\nkube-node-lease\nkube-public\nkube-system\n*stage\n```\n\n## Launch\n\n```bash\n\u003e ctlptl apply -f k8/ctlptl-minikube-cluster-reg.yml\nSwitched to context \"minikube\".\ncluster.ctlptl.dev/minikube created\n\n\u003e ctlptl get\nCURRENT   NAME       PRODUCT    AGE   REGISTRY\n*         minikube   minikube   13m   localhost:64612\n\n\u003e ctlptl get registry\nNAME                       HOST ADDRESS      CONTAINER ADDRESS   AGE\nctlptl-minikube-registry   127.0.0.1:64612   172.17.0.2:5000     16m\n\n\u003e The shell has just a `CMD [\"sh\"]`. The pod will automatically connect once we `kubectl exec -it runner-pod-name -- sh` and the following in the pod shell:\n\n```bash\n\u003e kubectl get pods -o wide\n\nNAME                    STATUS    IP            NODE\nmyapp-d4444db6d-m9m82   Running   10.244.0.48   minikube\n\n\u003e kubectl exec -it runner-5696b7f8cf-4k9g9 -- sh\n\nbash# iex --cookie \"$(echo $ERLANG_COOKIE)\" --name \"$(echo k8_cluster@$(echo $POD_IP))\" -S mix\n```\n\n## Cookies\n\n[Nice post](https://blog.differentpla.net/blog/2022/01/09/erlang-cookies-and-kubernetes/)\n\nBy default, Erlang reads the cookie from **~/.erlang.cookie** (created if not exist).\n\nElixir releases\nIf you’re using mix release, a random cookie is created at build time and written to _build/prod/rel/myapp/releases/COOKIE. This means that the cookie changes every time you build from clean, which will break your cluster.\n\nYou can specify a cookie with the :cookie option in mix.exs. I would avoid doing this because it means that the cookie is now visible in source control history.\n\nThe other way to set the cookie is to set the RELEASE_COOKIE environment variable before starting the release. You can do this in rel/env.sh.eex, or from a Kubernetes secret.\n\nSecrets:\n\n```bash\n\u003e ERLANG_COOKIE=$(head -c 40 \u003c /dev/random | base64 | tr '/+' '_-')\n# or in Elixir:\niex\u003e Base.url_encode64(:crypto.strong_rand_bytes(40))\n\n\u003ekubectl --namespace myapp create secret generic erlang-cookie --from-literal=cookie=\"$ERLANG_COOKIE\"\n```\n\nRemember: secrets are scoped to the namespace, so you might want to put your app name as a prefix, unless you’re using a dedicated namespace. A secret can contain multiple items. The example above uses cookie as the key. Secrets aren’t actually that secret. Fortunately, Erlang cookies aren’t actually that secret either.\nSet the environmental variables for a deployment:\n\n```yml\ncontainers:\n- name: myapp\n  env:\n  - name: RELEASE_COOKIE  # or RELX_COOKIE (or just set both)\n    valueFrom:\n      secretKeyRef:\n        name: erlang-cookie\n        key: cookie\n```\n\n## Not to forget\n\n`alias tilt=/usr/local/bin/tilt`\n\n[Access k8 API from within a pod](https://blog.differentpla.net/blog/2022/01/16/k8s-api-elixir-container/) using `:httpc` so simply (compared to Ruby!!), and the [k8s](https://hexdocs.pm/k8s/readme.html) package.\n\n## Local registry: `ctlplt`\n\nCreate a local registry for Docker images to be used by `k3d` or `minikube`. No more needed to push Docker images to the Docker hub. The images wil lbe registred under `172.17.0.2:5000:my-image`\n\n- Create registry: `ctlptl apply -f ctlptl-registry.yml`\n- Delete cluster: `ctlptl delete -f ctlptl-registry.yml`\n- Check registries: `ctlptl get registry`\n\n```txt\nNAME              HOST ADDRESS     CONTAINER ADDRESS   AGE\nctlptl-registry   127.0.0.1:5000   172.17.0.2:5000     6m\n```\n\nThen Tilt will use this local registry and not go to the Docker hub.\n\n## Tiltfile\n\nRun `tilt up` and `tilt down`.\n\n## Libcluster\n\n[Source for k8](https://blog.differentpla.net/blog/2022/01/08/libcluster-kubernetes/)\n\n```bash\nkubectl get pods -l app=myapp -o json | \\\n  jq '.items[] | {ip: .status.podIP, namespace: .metadata.namespace}'\n\n```\n\n```json\n{\n  \"ip\": \"10.244.0.33\",\n  \"namespace\": \"default\"\n}\n{\n  \"ip\": \"10.244.0.34\",\n  \"namespace\": \"default\"\n}\n```\n\n## Docker comamnds\n\n```bash\ndocker build  -t elixcluster -f Dockerfile.ex\n\ndocker network create erl-cluster\n\ndocker run -it --rm --network erl-cluster --name bnode  elixcluster  iex --sname b -S mix\n\ndocker build -t localhost:5000/k8cluster\n\ndocker run --rm  -it --user=root caching:test\n\n# If bare elixir, do\ndocker run --mount type=bind,src=$(pwd),dst=/app  --rm localhost:5000/elixir-min mix deps.get \u0026\u0026 mix deps.compile \u0026\u0026 iex --sname a -S mix\n\ndocker run -v \"$(pwd)\":/app --rm elixir-min mix deps.get \u0026\u0026 mix deps.compile \u0026\u0026 iex --sname a -S mix\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndrean%2Fk8_cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fndrean%2Fk8_cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndrean%2Fk8_cluster/lists"}