{"id":46843040,"url":"https://github.com/flamarion/ceph-cluster-kubernetes-ceph-csi","last_synced_at":"2026-03-10T13:39:20.540Z","repository":{"id":205474642,"uuid":"714321544","full_name":"flamarion/ceph-cluster-kubernetes-ceph-csi","owner":"flamarion","description":"This repo contain a basic configuration of a Ceph cluster and the Ceph CSI on Kubernetes.","archived":false,"fork":false,"pushed_at":"2024-02-10T15:05:54.000Z","size":26,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-02-10T16:23:52.965Z","etag":null,"topics":["ceph","ceph-csi","csi","kubernetes","persistent-volume"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flamarion.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-11-04T15:10:45.000Z","updated_at":"2024-02-10T16:23:52.966Z","dependencies_parsed_at":null,"dependency_job_id":"7bea7efb-1bf2-4055-8c0a-8ff9e2c92a13","html_url":"https://github.com/flamarion/ceph-cluster-kubernetes-ceph-csi","commit_stats":null,"previous_names":["flamarion/ceph-cluster-kubernetes-ceph-csi"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/flamarion/ceph-cluster-kubernetes-ceph-csi","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flamarion%2Fceph-cluster-kubernetes-ceph-csi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flamarion%2Fceph-cluster-kubernetes-ceph-csi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flamarion%2Fceph-cluster-kubernetes-ceph-csi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flamarion%2Fceph-cluster-kubernetes-ceph-csi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flamarion","download_url":"https://codeload.github.com/flamarion/ceph-cluster-kubernetes-ceph-csi/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flamarion%2Fceph-cluster-kubernetes-ceph-csi/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30334854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T12:41:07.687Z","status":"ssl_error","status_checked_at":"2026-03-10T12:41:06.728Z","response_time":106,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ceph","ceph-csi","csi","kubernetes","persistent-volume"],"created_at":"2026-03-10T13:39:18.240Z","updated_at":"2026-03-10T13:39:20.519Z","avatar_url":"https://github.com/flamarion.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ceph Cluster and Ceph CSI on Kubernetes\n\nI have a home lab running in some Intel NUC where I have a Kubernetes cluster that does almost everything I need (I still need an external DNS... this is in progress), and I needed something to provide dynamic volume provisioning.\n\nSo far, I have created a generic storage class and used a no-provisioner to map persistent volumes to empty directories. Still, it only works well in the short run because the volumes are not replicated and not dynamically provisioned and some other problems of empty dir as a PV.\n\nOn top of that, I worked with Ceph a few years ago and wanted to remember some stuff, so I merged my need with my will to return to Ceph.\n\nNow, I have a Ceph Cluster + Kubernetes cluster, and I can provision persistent volumes dynamically.\n\nAbout the External DNS, I have the PDNS already configured, so when I have some time, I will give it a try: https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/pdns.md.\n\n- [Ceph Cluster and Ceph CSI on Kubernetes](#ceph-cluster-and-ceph-csi-on-kubernetes)\n  - [Cluster Configuration (VMs)](#cluster-configuration-vms)\n  - [Repository setup (Al nodes)](#repository-setup-al-nodes)\n  - [Node 1 (ceph1):](#node-1-ceph1)\n  - [Configure Kubernetes to use Ceph to provisioning Persistent volumes](#configure-kubernetes-to-use-ceph-to-provisioning-persistent-volumes)\n    - [ConfigMap with CSI configuration:](#configmap-with-csi-configuration)\n    - [ConfigMap with KMS configuration.](#configmap-with-kms-configuration)\n    - [Another config map required for Ceph CSI (I don't remember why):](#another-config-map-required-for-ceph-csi-i-dont-remember-why)\n    - [Secret with the user and key created before.](#secret-with-the-user-and-key-created-before)\n    - [Create the Roles](#create-the-roles)\n    - [Deploy the provisioner and the node plugin](#deploy-the-provisioner-and-the-node-plugin)\n    - [Create the StorageClass](#create-the-storageclass)\n  - [Testing the configuration](#testing-the-configuration)\n  - [For those lazy (I love copy and paste too)](#for-those-lazy-i-love-copy-and-paste-too)\n  - [Helm Charts](#helm-charts)\n  - [References:](#references)\n  - [TODO](#todo)\n\n## Cluster Configuration (VMs)\n\nOS Version:\n```bash\nDistributor ID: Ubuntu\nDescription:    Ubuntu 22.04.3 LTS\nRelease:        22.04\nCodename:       jammy\n```\n\nCPU(s): `4`\nRAM: `4GB`\nDISKS:\n```bash\n/dev/sda  20G -\u003e OS\n/dev/sdb  50G -\u003e Ceph OSD\n```\n\n## Repository setup (Al nodes)\n```bash\ncurl https://download.ceph.com/keys/release.gpg -o /etc/apt/keyrings/ceph.gpg\napt-add-repository \"deb https://download.ceph.com/debian-reef/ $(lsb_release -sc) main\"\n```\nEdit the file `/etc/apt/sources.list.d/ceph.list` and add the `signed-by=/etc/apt/keyrings/ceph.gpg`:\n```bash\ndeb [signed-by=/etc/apt/keyrings/ceph.gpg] https://download.ceph.com/debian-reef/ jammy main\n```\n\nInstall Podman (used by Cephadm)\n```bash\napt update\napt install podman -y\n```\n\n## Node 1 (ceph1):\n```bash\napt update\napt install cephadm\ncephadm bootstrap --mon-ip 192.168.10.56\ncephadm add-repo --release reef\ncephadm install ceph-common\n```\nOptional (but why not help the community):\n```bash\nceph telemetry on --license sharing-1-0\n```\nContinue with the cluster configuration:\n```bash\nceph status\nssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph2.lab.local\nssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph3.lab.local\nceph status\nceph orch host add ceph2 --labels _admin\nceph orch host add ceph3 --labels _admin\nceph orch daemon add mon ceph2\nceph orch daemon add mon ceph3\nceph orch apply mon --placement=\"ceph1,ceph2,ceph3\"\nceph orch device ls\nceph orch apply osd --all-available-devices\n```\n\nAdd the Kubernetes pool:\n\n```bash\nceph osd pool create kubernetes\nrbd pool init kubernetes\n```\n\nCreate the user for Kubernetes (save this information, you will need it later):\n\n```bash\nceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'\n[client.kubernetes]\n        key = AQClQkZlHuZYLRAA0JuPOkWGtNujPrLwNoNcXQ==\n```\n\nDump the Ceph configuration (take note of FSID and monitors, you will need it later):\n\n```bash\nceph mon dump\nepoch 3\nfsid 233a71ec-7b10-11ee-a5c8-f76cb62122db\nlast_changed 2023-11-04T13:00:43.847269+0000\ncreated 2023-11-04T12:46:35.895367+0000\nmin_mon_release 18 (reef)\nelection_strategy: 1\n0: [v2:192.168.10.56:3300/0,v1:192.168.10.56:6789/0] mon.ceph1\n1: [v2:192.168.10.58:3300/0,v1:192.168.10.58:6789/0] mon.ceph2\n2: [v2:192.168.10.59:3300/0,v1:192.168.10.59:6789/0] mon.ceph3\ndumped monmap epoch 3\n```\n\nIn case the `ceph health` or `ceph status` command return a WARN because OSD performance, you can disable the scrub and deep-scrub and re-enable it after some time:\n\n```bash\n# ceph osd set nodeep-scrub\n# ceph osd set nodeep-scrub\n\n# ceph osd unset noscrub\n# ceph osd unset nodeep-scrub\n```\n\n## Configure Kubernetes to use Ceph to provisioning Persistent volumes\n\n### ConfigMap with CSI configuration:\n\n```bash\ncat \u003c\u003cEOF \u003e csi-config-map.yaml\n---\napiVersion: v1\nkind: ConfigMap\ndata:\n  config.json: |-\n    [\n      {\n        \"clusterID\": \"233a71ec-7b10-11ee-a5c8-f76cb62122db\",  #FSID (remember?)\n        \"monitors\": [\n          \"192.168.10.56:6789\",\n          \"192.168.10.58:6789\",\n          \"192.168.10.59:6789\"\n        ]\n      }\n    ]\nmetadata:\n  name: ceph-csi-config\nEOF\n\nkubectl create -f csi-config-map.yaml\n```\n\n### ConfigMap with KMS configuration. \n\nIt's mandatory even if you don't use KMS. If you don't use KMS, just leave the config.json empty (my case):\n```bash\ncat \u003c\u003cEOF \u003e csi-kms-config-map.yaml\n---\napiVersion: v1\nkind: ConfigMap\ndata:\n  config.json: |-\n    {}\nmetadata:\n  name: ceph-csi-encryption-kms-config\nEOF\n\nkubectl create -f csi-kms-config-map.yaml\n```\n\n### Another config map required for Ceph CSI (I don't remember why):\n```bash\ncat \u003c\u003cEOF \u003e ceph-config-map.yaml\n---\napiVersion: v1\nkind: ConfigMap\ndata:\n  ceph.conf: |\n    [global]\n    auth_cluster_required = cephx\n    auth_service_required = cephx\n    auth_client_required = cephx\n  # keyring is a required key and its value should be empty\n  keyring: |\nmetadata:\n  name: ceph-config\nEOF\n\nkubectl create -f ceph-config-map.yaml\n```\n\n### Secret with the user and key created before. \n\nThis will be used to create the images/blocks in the pool that we created before:\n```bash\ncat \u003c\u003cEOF \u003e csi-rbd-secret.yaml\n---\napiVersion: v1\nkind: Secret\nmetadata:\n  name: csi-rbd-secret\n  namespace: default\nstringData:\n  userID: kubernetes\n  userKey: AQClQkZlHuZYLRAA0JuPOkWGtNujPrLwNoNcXQ== #key (remember?)\nEOF\n\nkubectl create -f csi-rbd-secret.yaml\n```\n\n###  Create the Roles\nWe need to let Ceph CSI work, so let's give it the permissions that it needs:\n```bash\nkubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml\nkubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml\n```\n\n### Deploy the provisioner and the node plugin\nBefore apply, edit the yaml before and fix the pod applyingn and replace `quay.io/cephcsi/cephcsi:canary` with `quay.io/cephcsi/cephcsi:v3.8.1`.\n\n```bash\nwget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml\nkubectl apply -f csi-rbdplugin-provisioner.yaml\nwget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml\nkubectl apply -f csi-rbdplugin.yaml\n```\n\n### Create the StorageClass\n```bash\ncat \u003c\u003cEOF \u003e csi-rbd-sc.yaml\n---\napiVersion: storage.k8s.io/v1\nkind: StorageClass\nmetadata:\n   name: csi-rbd-sc\nprovisioner: rbd.csi.ceph.com\nparameters:\n   clusterID: 233a71ec-7b10-11ee-a5c8-f76cb62122db #FSID (remember?)\n   pool: kubernetes\n   imageFeatures: layering\n   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret\n   csi.storage.k8s.io/provisioner-secret-namespace: default\n   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret\n   csi.storage.k8s.io/controller-expand-secret-namespace: default\n   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret\n   csi.storage.k8s.io/node-stage-secret-namespace: default\nreclaimPolicy: Delete\nallowVolumeExpansion: true\nmountOptions:\n   - discard\nEOF\nkubectl apply -f csi-rbd-sc.yaml\n```\n\nOnce this first part is done, you can use the new storage class to provision persistent volumes directly from Ceph.\nYou may want to create a new namespace for this configuration so you can delete it later and don't mess with your current environment.\n\n## Testing the configuration \n\nSome tests validate the whole configuration (I assume you're familiar with Kubernetes, and you know how to check if the PVC is bound or not). Help yourself after each PVC is created.\n\nCreate a PVC to be used as raw block device (`volumeMode: Block`):\n```bash\ncat \u003c\u003cEOF \u003e raw-block-pvc.yaml\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: raw-block-pvc\nspec:\n  accessModes:\n    - ReadWriteOnce\n  volumeMode: Block\n  resources:\n    requests:\n      storage: 1Gi\n  storageClassName: csi-rbd-sc\nEOF\nkubectl apply -f raw-block-pvc.yaml\n```\n\nCreate a pod with the PVC as a raw block device:\n```bash\ncat \u003c\u003cEOF \u003e raw-block-pod.yaml\n---\napiVersion: v1\nkind: Pod\nmetadata:\n  name: pod-with-raw-block-volume\nspec:\n  containers:\n    - name: fc-container\n      image: fedora:26\n      command: [\"/bin/sh\", \"-c\"]\n      args: [\"tail -f /dev/null\"]\n      volumeDevices:\n        - name: data\n          devicePath: /dev/xvda\n  volumes:\n    - name: data\n      persistentVolumeClaim:\n        claimName: raw-block-pvc\nEOF\nkubectl apply -f raw-block-pod.yaml\n```\n\nCreate a new PVC now to be used as FS (`volumeMode: Filesystem`):\n```bash\ncat \u003c\u003cEOF \u003e pvc.yaml\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: rbd-pvc\nspec:\n  accessModes:\n    - ReadWriteOnce\n  volumeMode: Filesystem\n  resources:\n    requests:\n      storage: 1Gi\n  storageClassName: csi-rbd-sc\nEOF\nkubectl apply -f pvc.yaml\n```\n\nCreate a pod and mount the PVC as a FS:\n```bash\ncat \u003c\u003cEOF \u003e pod.yaml\n---\napiVersion: v1\nkind: Pod\nmetadata:\n  name: csi-rbd-demo-pod\nspec:\n  containers:\n    - name: web-server\n      image: nginx\n      volumeMounts:\n        - name: mypvc\n          mountPath: /var/lib/www/html\n  volumes:\n    - name: mypvc\n      persistentVolumeClaim:\n        claimName: rbd-pvc\n        readOnly: false\nEOF\nkubectl apply -f pod.yaml\n```\n\n## For those lazy (I love copy and paste too)\n\nAll files used to configure the CSI are in this repo, so make sure to gather all information (kubernetes client key, fsid, mons...), fix in the files and apply in the following order:\n\n```bash \nkubectl create -f csi-config-map.yaml\nkubectl create -f csi-kms-config-map.yaml\nkubectl create -f ceph-config-map.yaml\nkubectl create -f csi-rbd-secret.yaml\nkubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml\nkubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml\nkubectl apply -f csi-rbd-sc.yaml\nkubectl apply -f raw-block-pvc.yaml\nkubectl apply -f raw-block-pod.yaml\nkubectl apply -f pvc.yaml\nkubectl apply -f pod.yaml\n```\n\n## Helm Charts\n\nAdd the Ceph CSI Helm Charts Repo\n\n```bash\nhelm repo add ceph-csi https://ceph.github.io/csi-charts\nhelm repo update\n```\n\nThe file `values.yaml` does everything that all the other files to install and configure the Ceph CSI above would do.\n\nSo to acomplish the same result, you can simply run the following command:\n\n```bash\nhelm install --namespace \"ceph-csi-rbd\" --create-namespace \"ceph-csi-rbd\" ceph-csi/ceph-csi-rbd -f values.yaml\n```\n\nThe command above will organize everything in the namespace `ceph-csi-rbd` and set the `csi-rbd-sc` as default storage class in your cluster.\n\nIf you don't want the Ceph as default storage class, comment the following configuration in `values.yaml`\n\n```\n  annotations:\n    storageclass.kubernetes.io/is-default-class: \"true\"\n```\n\nTo test you can run one of the previous tests like `kubectl apply -f pvc.yaml`, below an example:\n\n```bash\n$ kubectl apply pvc.y^C\n$ kubectl get pv\nNo resources found\n$ kubectl get pvc\nNo resources found in default namespace.\n$ kubectl apply -f pvc.yaml\npersistentvolumeclaim/rbd-pvc created\n$ kubectl get pv\nNAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS   REASON   AGE\npvc-c4e16a77-3801-4861-aee1-d93bac9ca393   1Gi        RWO            Delete           Bound    default/rbd-pvc   csi-rbd-sc              2s\n$ kubectl get pvc\nNAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE\nrbd-pvc   Bound    pvc-c4e16a77-3801-4861-aee1-d93bac9ca393   1Gi        RWO            csi-rbd-sc     5s\n$ kubectl delete -f pvc.yaml\npersistentvolumeclaim \"rbd-pvc\" deleted\n```\n\n## References:\n\nhttps://docs.ceph.com/en/reef/cephadm/install/\nhttps://docs.ceph.com/en/reef/cephadm/host-management/#cephadm-adding-hosts\nhttps://docs.ceph.com/en/reef/cephadm/services/osd/#cephadm-deploy-osds\nhttps://docs.ceph.com/en/reef/rbd/rbd-kubernetes/\nhttps://github.com/ceph/ceph-csi/tree/devel/charts/ceph-csi-rbd\nhttps://github.com/ceph/ceph-csi/tree/devel/charts/ceph-csi-cephfs\n\n## TODO\n\n- [x] Make all the Kubernetes part of this via Helm Charts for the sake of our menthal health.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflamarion%2Fceph-cluster-kubernetes-ceph-csi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflamarion%2Fceph-cluster-kubernetes-ceph-csi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflamarion%2Fceph-cluster-kubernetes-ceph-csi/lists"}