{"id":26797617,"url":"https://github.com/kameshsampath/rp-k3d","last_synced_at":"2025-06-27T23:39:55.568Z","repository":{"id":212097797,"uuid":"730523448","full_name":"kameshsampath/rp-k3d","owner":"kameshsampath","description":"Scripts to provision Kubernetes(k3s) cluster using https://k3d.io and deploy https://redpanda.com cluster on to it","archived":false,"fork":false,"pushed_at":"2024-03-27T11:38:04.000Z","size":94,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-12-22T20:03:14.510Z","etag":null,"topics":["k3d","redpanda"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kameshsampath.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-12-12T05:35:25.000Z","updated_at":"2024-03-27T11:38:25.000Z","dependencies_parsed_at":"2023-12-12T15:03:35.572Z","dependency_job_id":null,"html_url":"https://github.com/kameshsampath/rp-k3d","commit_stats":null,"previous_names":["kameshsampath/rp-k3d"],"tags_count":0,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kameshsampath%2Frp-k3d","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kameshsampath%2Frp-k3d/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kameshsampath%2Frp-k3d/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kameshsampath%2Frp-k3d/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kameshsampath","download_url":"https://codeload.github.com/kameshsampath/rp-k3d/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246230520,"owners_count":20744349,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["k3d","redpanda"],"created_at":"2025-03-29T19:15:19.273Z","updated_at":"2025-03-29T19:15:19.778Z","avatar_url":"https://github.com/kameshsampath.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Setup Redpanda on k3s\n\nScripts to setup the [Redpanda](https://redpanda.com) dev cluster on a developer laptop using [k3d](https:/k3d.io).\n\n## Required Tools\n\n- [Docker for Desktop](https://www.docker.com/products/docker-desktop/)\n- [k3d](https://k3d.io)\n- [kubectl](https://kubernetes.io/docs/tasks/tools/)\n- [yq](https://github.com/mikefarah/yq)\n- [direnv](https://direnv.net)\n\n## Environment\n\nThe scripts expects the following environment variables to be set, Create `.envrc` file:\nCreate and load the sane defaults,\n\n```shell\ncp \"${PROJECT_HOME}/etc/templates/.envrc.template\" \"${PROJECT_HOME}/.envrc\"\n```\n\nLoad the environment variables,\n\n```shell\ndirenv allow .\n```\n\n## Create Cluster\n\nThe following script creates the [k3s](https://k3s.io)Kubernetes cluster using k3d and deploys the basic single node Redpanda cluster on to it.\n\n```shell\n$PROJECT_HOME/bin/setup.sh\n```\n\nAll the manifests in the features are applied on to the cluster via the [cluster.yml](./config/k3d/cluster.yml).\n\nLet us inspect the `redpanda` namespace,\n\n```shell\nkubectl get pods,svc -n redpanda\n```\n\nShould show an output like,\n\n```text\nNAME                                     READY   STATUS      RESTARTS       AGE\npod/redpanda-operator-6659c776dd-r2pdw   2/2     Running     0              2m44s\npod/redpanda-0                           2/2     Running     0              2m23s\npod/redpanda-console-6649f84d9c-h7btb    1/1     Running     1 (113s ago)   2m23s\npod/redpanda-configuration-tl454         0/1     Completed   0              101s\n\nNAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                       AGE\nservice/operator-metrics-service   ClusterIP   10.43.100.13    \u003cnone\u003e        8443/TCP                                                      2m44s\nservice/redpanda                   ClusterIP   None            \u003cnone\u003e        9644/TCP,8082/TCP,9093/TCP,33145/TCP,8081/TCP                 2m23s\nservice/redpanda-console           NodePort    10.43.111.128   \u003cnone\u003e        8080:30080/TCP                                                2m23s\nservice/redpanda-external          NodePort    10.43.214.220   \u003cnone\u003e        9645:31644/TCP,9094:31092/TCP,8083:30082/TCP,8084:30081/TCP   2m23s\n```\n\nThe k3d configuration [cluster.yml](./config/k3d/cluster.yml) has exposed the following `NodePort` to the host interface:\n\n- `Kafka Broker` - localhost:31902\n- `Schema Registry` - localhost:30081\n- `PandaProxy` - localhost:30082\n- `Admin API` - localhost:31644\n\nAnd the console is accessible using the url \u003chttp://localhost:30080/\u003e\n\n## Test the Setup\n\n### Cluster Authentication\n\nThe cluster is configured with `SASL` authentication with superuser `$RPK_SUPER_ADMIN` and password `$RPK_SUPER_ADMIN_PASS` with SASL mechanism as `$RPK_SASL_MECHANISM`.\n\n\u003e **IMPORTANT**: It is recommended we don't use the super user for normal day to day operations.\n\n### Create Topic Admin\n\nLet us create user called `$RPK_TOPIC_ADMIN` and give the user admin operations on topic,\n\n```shell\nrpk acl user create \"${RPK_TOPIC_ADMIN}\" -p \"${RPK_TOPIC_ADMIN_PASS}\" \\\n--mechanism \"${RPK_SASL_MECHANISM}\" \\\n-X user=\"${RPK_SUPER_ADMIN}\" \\\n-X pass=\"${RPK_SUPER_ADMIN_PASS}\" \\\n-X sasl.mechanism=\"${RPK_SASL_MECHANISM}\"\n```\n\nAssign the user the required permissions,\n\n```shell\nrpk acl create --allow-principal \"User:${RPK_TOPIC_ADMIN}\" \\\n--operation all --topic '*' --resource-pattern-type 'match' \\\n-X user=\"${RPK_SUPER_ADMIN}\" \\\n-X pass=\"${RPK_SUPER_ADMIN_PASS}\" \\\n-X sasl.mechanism=\"${RPK_SASL_MECHANISM}\"\n```\n\n### rpk Profile\n\n`rpk` profile is convenient way to switch Redpanda settings for different cluster environment. Let us setup one for `k3s` setup\n\n```shell\nrpk profile create k3d\n```\n\nNow let us set some profile properties,\n\n```shell\nrpk profile set brokers \"${RPK_BROKERS}\"\nrpk profile set admin.hosts \"${RPK_ADMIN_HOSTS}\"\nrpk profile set registry.hosts \"${RPK_SCHEMA_REGISTRY}\"\nrpk profile set sasl.mechanism \"${RPK_SASL_MECHANISM}\"\nrpk profile set user \"${RPK_TOPIC_ADMIN}\"\nrpk profile set pass \"${RPK_SUPER_ADMIN_PASS}\"\n```\n\nNow running the command to display the cluster status,\n\n```shell\nrpk cluster status\n```\n\nShould show the an output like\n\n```text\nCLUSTER\n=======\nredpanda.58b01085-1072-4ea1-8225-78fcc18238a5\n\nBROKERS\n=======\nID    HOST                  PORT\n0*    redpanda-0.localhost  31092\n\nTOPICS\n======\nNAME      PARTITIONS  REPLICAS\n_schemas  1           1\n```\n\n### List Topics\n\n```shell\nrpk topic list\n```\n\nShould show the following output,\n\n```shell\nNAME      PARTITIONS  REPLICAS\n_schemas  1           1\n```\n\nLet us try creating a new topic,\n\n```shell\nrpk topic create greetings\n```\n\nThe command should fail with following error,\n\n```text\nunable to create topics [greetings]: unable to dial: dial tcp: lookup redpanda-0.localhost: no such host\n```\n\n### Resolving `.localhost` domains\n\nWe don't have a resolver to route our requests to `redpanda-0.localhost`. There are many ways to do it and very simple of all is to add an entry to `/etc/hosts` file. But to make it more clean and neat, with ability to support other domain names than `.localhost` we will use [dnsmasq](https://dnsmasq.org).\n\nRun the following command to install `dnsmasq`\n\n```shell\nbrew install dnsmasq\n```\n\nConfigure the DNS server on `12.0.0.1` and make `.localhost` to be resolved using that DNS server,\n\n```shell\necho 'address=/.localhost/127.0.0.1' \u003e\u003e \"$(brew --prefix)/etc/dnsmasq.conf\"\necho 'listen-address=127.0.0.1' \u003e\u003e \"$(brew --prefix)/etc/dnsmasq.conf\"\n```\n\nRestart the `dnsmasq` service,\n\n```shell\nsudo brew services restart dnsmasq\n```\n\nAdd a resolver to be used by dnsmaq to resolve `.localhost`,\n\n```shell\nsudo mkdir -pv /etc/resolver\necho 'nameserver 127.0.0.1' | sudo tee -a /etc/resolver/localhost\n```\n\nNow when you try to ping the Redpanda broker address `redpanda-0.localhost` it should be reachable,\n\n```shell\nping -c3 redpanda-0.localhost\n```\n\nShould output\n\n```text\nPING redpanda-0.localhost (127.0.0.1): 56 data bytes\n64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.053 ms\n64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.092 ms\n64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.058 ms\n```\n\nNow we are all set to create new topics using the command,\n\n```shell\nrpk topic create greetings\n```\n\nWhich should return,\n\n```shell\nTOPIC      STATUS\ngreetings  OK\n```\n\n## Cleanup\n\n```shell\n$PROJECT_HOME/bin/destroy.sh\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkameshsampath%2Frp-k3d","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkameshsampath%2Frp-k3d","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkameshsampath%2Frp-k3d/lists"}