zfs-provisioner is an out of cluster external provisioner for Kubernetes. It creates ZFS datasets and shares them via NFS to make them mountable to pods. Currently all ZFS attributes are inherited from the parent dataset, different storage classes for e.g. cached/non-cached datasets or manually setting attributes via annotations should follow in the future. This provisioner is considered highly experimental and is still under development.
For more information about external storage in kubernetes, see kubernetes-incubator/external-storage.
The provisioner can be configured via the following environment variables:
| Variable | Description | Default |
|---|---|---|
ZFS_PROVISIONER_NAME |
Name of the provisioner. Change only if you want to run multiple instances. | creamfinance.com/zfs |
ZFS_KUBE_CONF |
Path to the kubernetes config file which will be used to connect to the cluster. | kube.conf |
ZFS_METRICS_PORT |
Port on which to export Prometheus metrics. | 8080 |
The storage class needs to be defined with a provisioner, an example would be:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: zfs
provisioner: creamfinance.com/zfs
reclaimPolicy: Delete
parameters:
parentDataset: storage/cluster
shareOptions: [email protected]/24
serverHostname: "10.5.3.1"
overProvision: "true"
Whereas:
| Parameter | Description |
|---|---|
| parentDataset | defines where the newly provisioned volumes will be places |
| shareOptions | defines the options for the sharenfs property of zfs |
| serverHostname | defines how nodes can actually reach the nfs server |
| overProvision | defines if refreservation (overProvision = false) is set or only refquota (overProvision = true) |
The dataset that was created is saved in the persistent volume as a annotation creamfinance.com/zfs-dataset.
On delete the dataset name is checked against the PV Name - so only datasets that match exactly the name of the volume will actually be deleted to hinder manipulation.
The second annotation that is available is creamfinance.com/zfs-owner which can be added to a pvc to set the ownership of the root folder.
A third annotation creamfinance.com/zfs-clone can be specified to always clone the specified volume into a new volume, and destroy the clone and snapshot on pvc delete.
The provisioner uses the reflimit and refquota ZFS attributes to limit storage space for volumes.
The overProvision property in the storage class controls if reflimit is set or not, refquota is always set.
The tests need to manage ZFS datasets, create a testing pool on a disk image:
# Create a 10GB disk image
dd if=/dev/zero bs=1024m count=10 of=disk.img
truncate --size 1G disk1.img
sudo zpool create pool1 $PWD/disk1.img -m $PWD/test
# Mount the image as a block device, MacOS way
hdiutil attach -imagekey diskimage-class=CRawDiskImage -nomount disk.img
# Create zpool with mount in current directory
sudo zpool create -m $PWD/test -f test /dev/disk2
For development under other operating systems, adapt mount command and block device.
You need GO and go-dep
sudo apt install golang-go go-dep
# If $GOPATH is empty
mkdir -p ~/go
export GOPATH=$HOME/go
mkdir -p $GOPATH/src
ln -s $PATH_TO_REPO $GOPATH/src/kubernetes-zfs-provisioner
cd $GOPATH/src/kubernetes-zfs-provisioner
# Install dependencies
dep ensure
# Build
make build