Integration with XigmaNAS and Cozystack to get a hyper-converged cloud-native HA storage

[a-prokopyev-resume]

Hello,

What do you think about possibility of integrating your bhyve orchestrator into XygmaNAS to produce something similar to SmartOS immutable OS if using only LiveCD version of XygmaNAS?

Then we could create a VM with Talos Linux on several XygmaNAS nodes, install Cozystack into this K8s cluster and get a HA hyperconverged cloud-native HA storage managed by Linstor Piraeus K8s operator.

Actually Cozystack already includes many cloud-native components like Piraeus operator and easily installed into a ready Talos K8s cluster bootstraping of which could be automated by some tools like Ansible.

If having a VM guest with two block devices:

1. Give first block device to Talos for its local storage for K8s
2. Give second block device directly to DRBD managed by Piraeus in Cozystack WITHOUT creating LVM or ZFS pool for Linstor.

This VM is going to be run in a bhyve on FreeBSD XygmaNAS, which will pass through two ZFS zvols into VM as block devices to be used for Talos (first) and for DRBD (second).
I just need to enable HA storage using 2-3 nodes with a such VM on each of them.
It would be a so called hyperconvereged cloud-native storage on the same hosts (but in VM guests) where XygmaNAS itself runs.

[yaroslav-gwit]

Hi, thanks for submitting this issue/request.

I am investigating MooseFS and HAST as potential hyper-converged options for the hoster storage itself, and there are already CSI adapters that work with ZFS and MooseFS.

The setup you describe is very cool on paper/in theory, but overly complicated (at least it is to me), as troubleshooting any issues from that setup might become a nightmare in the long run. I am a really big fan of simple, but feature-complete systems, that's why I chose to develop under FreeBSD in a first place.

Here are my thoughts on how to improve and simplify your setup, but still keep the core concept - K8s with hyper-converged and highly available storage:

• Use FreeBSD, not XigmaNAS (you'll have a freedom to release your own customised live images, instead of relying on XigmaNAS release cycles, + if something doesn't work on one release you can try another one or compile the kernel yourself to bake-in a missing feature)
• If you can tolerate external storage (outside of your K8s cluster): use NFS, ZFS or MooseFS CSI storage adapters
• If you can't tolerate external storage: use OpenEBS, Longhorn, RookFS or Linstor (which keeps the data on the 2nd hard-drive attached to the VM)
• Just keep in mind what everyone in this industry says (this is true for any production-grade cluster): do not run K8s-based storage under K8s itself - because if your cluster breaks so does your storage
• Hoster already handles most of the heavy-lifting in terms of VM management: fast VM deployments (under 1 second), CloudInit template generation, network and disk management for VMs, process supervision, WebUI, RestAPI, etc etc
• I've been testing alternative K8s distros (k3s, k0s and microk8s) under Hoster, but if need be I can release a Hoster-specific Talos image for easier PoC testing

All in all, I recommend keeping things simple - such setups are easier to maintain in the long run and easier to troubleshoot in case you are having issues.