Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/travisghansen/freenas-iscsi-provisioner
FreeNAS iscsi provisioner
https://github.com/travisghansen/freenas-iscsi-provisioner
csi freenas iscsi kubernetes provisioner
Last synced: 13 days ago
JSON representation
FreeNAS iscsi provisioner
- Host: GitHub
- URL: https://github.com/travisghansen/freenas-iscsi-provisioner
- Owner: travisghansen
- License: apache-2.0
- Created: 2018-07-12T03:27:09.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2021-03-18T22:01:27.000Z (over 3 years ago)
- Last Synced: 2024-08-02T15:48:54.021Z (3 months ago)
- Topics: csi, freenas, iscsi, kubernetes, provisioner
- Language: Go
- Homepage:
- Size: 121 KB
- Stars: 29
- Watchers: 6
- Forks: 6
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Deprecation Notice
Non-csi drivers are no longer receiving updates from the Kubernetes team.
As such this driver has been deprecated and replaced with a newer `csi` based
implementation - https://github.com/democratic-csi/democratic-csi
The new implementation works with all the fancy `csi` features such as snapshots,
resizing, etc. Enjoy!
Note that a conversion script has been created to facilitate migrating volumes
created using this project to `democratic-csi`. Please visit the project for
more details.
# What is freenas-provisioner
FreeNAS-iscsi-provisioner is a Kubernetes external provisioner.
When a `PersisitentVolumeClaim` appears on a Kube cluster, the provisioner will
make the corresponding calls to the configured FreeNAS API to create an iscsi
target/lun usable by the claim. When the claim or the persistent volume is
deleted, the provisioner deletes the previously created resources.
See this for more info on external provisioner:
https://github.com/kubernetes-incubator/external-storage
Unless you have a very specific use-case for iscsi/block devices, it is
recommended to use the NFS variant of this project available here:
https://github.com/nmaupu/freenas-provisioner
# Usage
The scope of the provisioner allows for a single instance to service multiple
classes (and/or FreeNAS servers). The provisioner itself can be deployed into
the cluster or ran out of cluster, for example, directly on a FreeNAS server.
Each `StorageClass` should have a corresponding `Secret` created which contains
the credentials and host information used to communicate with with FreeNAS API.
In essence each `Secret` corresponds to a FreeNAS server.
The `Secret` namespace and name may be customized using the appropriate
`StorageClass` `parameters`. By default `kube-system` and `freenas-iscsi` are
used. While multiple `StorageClass` resources may point to the same server
and hence same `Secret`, it is recommended to create a new `Secret` for each
`StorageClass` resource.
It is **highly** recommended to read `deploy/class.yaml` to review available
`parameters` and gain a better understanding of functionality and behavior.
## FreeNAS Setup
You must manually create a dataset. You may simply use a pool as the parent
dataset but it's recommended to create a dedicated dataset.
Additionally, you need to enable the iscsi service with it's corresponding
resources such as portal, initiator, and group.
## Provision the provisioner
Run it on the cluster:
```
kubectl apply -f deploy/rbac.yaml -f deploy/deployment.yaml
```
Alternatively, for advanced use-cases you may run the provisioner out of cluster
including directly on the FreeNAS server if desired. Running out of cluster is
not currently recommended.
```
./bin/freenas-iscsi-provisioner-freebsd --kubeconfig=/path/to/kubeconfig.yaml
```
## Create `StorageClass` and `Secret`
All the necessary resources are available in the `deploy` folder. At a minimum
`secret.yaml` must be modified (remember to `base64` the values) to reflect the
server details. You may also want to read `class.yaml` to review available
`parameters` of the storage class. For instance to set the `datasetParentName`.
```
kubectl apply -f deploy/secret.yaml -f deploy/class.yaml
```
## Example usage
Next, create a `PersistentVolumeClaim` using the storage class
(`deploy/test-claim.yaml`):
```
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: freenas-test-iscsi-pvc
spec:
storageClassName: freenas-iscsi
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Mi
```
Use that claim on a testing pod (`deploy/test-pod.yaml`):
```
---
kind: Pod
apiVersion: v1
metadata:
name: freenas-test-iscsi-pod
spec:
containers:
- name: freenas-test-isci-pod
image: gcr.io/google_containers/busybox:1.24
command:
- "/bin/sh"
- "-c"
- "--"
args: [ "date >> /mnt/file.log && while true; do sleep 30; done;" ]
volumeMounts:
- name: freenas-test-volume
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: freenas-test-volume
persistentVolumeClaim:
claimName: freenas-test-iscsi-pvc
```
The underlying zvol, target, extent, etc should be quickly appearing on the
FreeNAS side. In case of issue, follow the provisioner's logs using:
```
kubectl -n kube-system logs -f freenas-iscsi-provisioner-
```
## CHAP settings
You should create a secret which holds CHAP authentication credentials based on `deploy/freenas-iscsi-chap.yaml`.
- If you have authentication enabled for the portal (discovery) then set `discovery*` parameters in the secret, and in StorageClass you should set `targetDiscoveryCHAPAuth` to `true`.
- If you want authentication for the targets, then set `node*` parameters in the secret, and in StorageClass you should set `targetGroupAuthtype` and `targetGroupAuthgroup` accordingly, and also set `targetSessionCHAPAuth` to `true`.
# Performance
100 10MiB PVCs
Creating took ~10 minutes
Deleting took ~6 minutes
# Testing
Choas testing has been performed to ensure the various actions are idempotent.
# Development
```
make vendor && make
```
Binary is located into `bin/freenas-iscis-provisioner`. It is compiled to be
run on `linux-amd64` by default, but you may run the following for different
builds:
```
make vendor && make darwin
# OR
make vendor && make freebsd
```
To run locally with an appropriate `$KUBECONFIG` you may run:
```
./local-start.sh
```
To format code before committing:
```
make fmt
```
- https://github.com/kubernetes-incubator/external-storage/blob/v4.0.0/lib/controller/controller.go
- https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/blob/master/controller/controller.go
- https://gianarb.it/blog/kubernetes-shared-informer
- https://engineering.bitnami.com/articles/a-deep-dive-into-kubernetes-controllers.html
## Docs
- https://github.com/kubernetes/community/tree/master/contributors/design-proposals/storage
- https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/volume-provisioning.md
- https://kubernetes.io/docs/concepts/storage/storage-classes/
- https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/#-strong-api-overview-strong-
- https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_iscsi.html
- http://api.freenas.org
- https://doc.freenas.org/11/sharing.html#block-iscsi
- https://github.com/kubernetes-incubator/external-storage/blob/master/iscsi/targetd/provisioner/iscsi-provisioner.go
- https://github.com/dghubble/sling
## TODO
- volume resizing - https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/grow-volume-size.md
- volume snapshots - https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/volume-snapshotting.md
- mount options - https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/mount-options.md
- ~~CHAP~~
- fsType
- properly handle `zvol` API differences with `volsize` getting sent as string and returned as int
- loop GetBy requests that require `limit` param
- ~~recursive zvol delete in v1 api~~
## Notes
To sniff API traffic between host and server:
```
sudo tcpdump -i any -A -s 0 'host and tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
```