-
Notifications
You must be signed in to change notification settings - Fork 21
FAQ
This involves ZFSOnLinux, which provides out-of-tree kernel modules. It has no dependency on FUSE.
ZFS support is in the portage tree. The sys-kernel/spl-9999 and sys-fs/zfs-9999 ebuilds are required to use ZFS at this time.
Booting off ZFS currently requires a separate /boot with an initramfs. Initial ZFS support was introduced in genkernel 3.4.25.1. genkernel 3.4.31 dramatically improved upon ZFS support.
Data stored using ZFS should be as safe as the disk configuration (e.g single disk, mirrored, raidz) permits. However, Gentoo ZFS support is not in a state in which it is ready for all users. The current ZFS support is suitable for advanced users who are willing to encounter minor issues. Those who wish to use ZFS as their root filesystem should use a separate /boot until a proper bootloader is available.
ZFS should be suitable for use by all Gentoo Linux users after a few more milestones are reached. In specific, this requires both that I reach my short term milestones for Gentoo and that upstream reach its milestones so that the ZFS kernel modules can be stabilized.
I intend to replace the mdadm/LVM/ext4 stack with ZFS on my systems, which requires making ZFS a first class filesystem on Gentoo. This project currently occupies a significant portion of my time, which I expect to continue for the foreseeable future.
In the long term, I plan to use ZFS exclusively on my physical systems, using ebuilds from the portage tree. As such, all improvements I make to it will be available to everyone, and not just myself. I will also handle bug reports in the Gentoo bug tracker as the maintainer for the kernel modules.
The ZFS Overlay currently hosts ebuilds that pull from my forks of upstream ZFS and SPL. Updates to the ebuilds that depend on upstream pull requests are made here. They enter the main tree as upstream merges pull requests.
Doing that would expose my work to a rather large audience. Until I have ebuilds in the portage tree keyworded, I would rather restrict this information to a smaller audience. I intend to gradually introduce information into the Official Gentoo Wiki as I work toward keywording ebuilds.
The current 0.6.0-rc8 releases have known issues. The current GIT code is in good shape and the amount of regression testing performed before commits are made to it is significant enough that it should be suitable for most purposes.
I plan to provide snapshots with patches that I know to improve things, but I have decided to work toward having improvements merged upstream first.
ZFS provides snapshots, checksumming and the elimination of fsck. It also integrates logical volume management and RAID, so it can heal itself automatically upon encountering a checksum failure if you use raidz or mirroring with it.
The following is based on a list that I provided to the Sabayon Linux developers in #sabayon-dev on freenode:
- All writes are atomic transactions, so you can disconnect the power cable on a system using ZFS and the disk format will not be corrupted. You will never need fsck.
- ZFS utilizes the ARC algorithm instead of Linux's LRU cache algorithm. I have observed ARC to have a 97% hit rate on my hardware.
- ZFS supports L2ARC, which lets you use a SSD as cache, like flashcache, but it limits writes to 8MB/sec. That ensures that SSDs will last a few years rather than 16 days, which is the time it took for Google's flash cache to destroy a SSD on a server administered by a Gentoo Linux user in #gentoo-chat on freenode. A benchmark by Anandtech shows two Intel SSDs to increase IOPS performance to 20873. For comparison, Anandtech observed a comparable hardware RAID array to support 2532 IOPS. A typical hard drive is unable to exceed 200 IOPS.
- ZFS supports using disks as a write cache through SLOG devices. All writes are stored in memory so that they can die without causing corruption. The advantage of this is that you get write sequentialization and lower latencies. You can use a SSD as a slog device and it will make writes very fast.
- ZFS has raidz, which is like raid 5/6, except it doesn't have a write hole because it leverages COW and a variable stripe size. raidz has 3 parity levels, so you can elect to have up to 3 levels of redundancy. The performance of raidz is far higher than that of hardware RAID 5. I have 6x 5400RPM 2TB disks in raidz2 that can sustain sequential write performance of 220MB/sec.
- ZFS has a feature called zvols, which are virtual block devices. You can make as many of them as you want and use them as storage for virtual machines and other filesystems.
- ZFS supports data deduplication.
- ZFS snapshots support incremental backup.
- Cross Platform Compatibility. ZFS originates from Solaris and has been ported to Linux, FreeBSD, Mac OS X and Windows.
- Swap support. It is possible to use swap on zvols. (in-overlay)
Advanced Format drives use 4K physical sectors while ZFS uses legacy 512 byte sectors by default. You can change the default by creating your ZFS pools with ashift=12, which will guarantee the use of 4K sectors.
ZFS lacks TRIM support at this time, so its performance will be negatively impacted. Recent SSD models implement garbage collection that should minimize the impact of this.
Furthermore, most SSDs use 4K sectors, so the use of ashift=12 is necessary to ensure peak performance when using them. Some newer models use 8K sectors, which will require ashift=13.
There needs to be a way to load the modules before the root filesystem is mounted, and unfortunately, the modules are out-of-tree, so without patching them into the kernel tree, it isn't possible to avoid an initramfs. Patching the ZFSOnLinux modules into the kernel tree is unsupported, so users will need to use an initramfs.
The current way to do this in Gentoo is to use genkernel. Some Gentoo are under the impression that genkernel automates everything and they dislike this because they like to do things manually. genkernel supports --menuconfig, so such users can use genkernel and still configure their kernels as they wish. genkernel also supports --integrated-initramfs, which will cause it to integrate the initramfs into the kernel binary with some minor changes to the .config file. This permits users to treat the kernel as if it did not use an initramfs, although they will still need to pass the kernel commandline parameters that genkernel needs to boot off ZFS.
Alternatively, users can use dracut and/or write an initramfs by hand. The adventurous can patch the modules into their kernel trees, but there is no support for this.
At the moment, Gentoo is ahead in some respects and Ubuntu is ahead in others.
The portage tree contains sys-kernel/spl and sys-fs/zfs, which provides the modules without the need for an overlay. They also have a maintainer.
In comparison, Ubuntu has no support for ZFS in its official repositories. Ubuntu users must use the ZFSOnLinux PPA to use ZFS while Gentoo users need no external overlay.
The Ubuntu community has had a significant head start in integrating ZFS support with GRUB2 and all ZFSOnLinux upstream developers working toward booting off ZFS are collaborating with them. They are also working to send their work to GRUB2 upstream, which should eventually come to Gentoo.
Gentoo's preferred bootloader is GRUB 0.97, but limited support for ZFS is available in sys-boot/grub:2. Efforts toward a ZFS-capable bootloader in Gentoo are fairly recent and as such, they are significantly behind the Ubuntu efforts.
The Ubuntu community relies on Fedora's dracut tool to generate initramfs images while the Gentoo community prefers its own genkernel tool. ZFSOnLinux upstream provides out-of-tree support to dracut while Gentoo has support in genkernel with the option of using dracut. The ZFS support in genkernel is much better integrated with genkernel as a consequence. However, as long as a working initramfs exists, the technology used to create it is irrelevant.
The Ubuntu community has documentation that permits Ubuntu Linux to be installed on ZFS using the official Ubuntu Linux CD with the ZFSOnLinux PPA, much like how Gentoo Linux would be installed. Gentoo Linux currently has an unofficial LiveCD by Zachary Bendell, but it is not as mature of the Ubuntu Live CD with the ZFSOnLinux PPA. However, Gentoo Linux users are able to use the Ubuntu LiveCD to do installations, much like they can use any LiveCD.
Notes on the installation process are in the zfs-install document at the root of the zfs overlay. The Ubuntu community has analogous documentation for installing Ubuntu on ZFS. Neither include raidz.
One Gentoo developer is working on ZFS support in Gentoo.
A few ZFSOnLinux contributors are working on ZFS support in Ubuntu independently of Canonical.
Short term milestones include:
-
ZFS support in sys-boot/grub(sys-boot/grub:2) -
An official Gentoo ZFS LiveCD(Gentoo 12.1 LiveDVD) -
Documenting the installation procedure for Gentoo on ZFS(zfs-install document) - Keyword the ZFS kernel modules in portage
Long term milestones include:
- Merging the Gentoo GRUB and Illumos GRUB codebases.
- Implement update pre-staging support (ZFS snapshot support) into Gentoo Portage.
- Implement support in useradd for ZFS dataset-based home directories.
- Implement support for thin provisioning in app-emulation/qemu-kvm.
There is a clear plan for reaching each short term milestone, but there is no current plan for the long term mile stones. Reaching them will likely require collaboration from others.
I plan to provide an ebuild for a snapshot of GIT HEAD with some patches and keyword it for the testing tree after I have finished testing some things. Initially, only amd64 will be keyworded, but additional 64-bit architectures will be keyworded after users report success with them. 32-bit architectures will not be keyworded unless upstream implements a workaround for vmalloc limitations in 32-bit Linux kernels.
With that said, the 0.6.0-rc7 and any other releases in that series will not be stabilized. The first official stable release by upstream is 0.7.0 and my plan as the Gentoo maintainer is to consider stabilization after it has been thoroughly vetted by those using the testing tree.
The zfs-install document contains some notes on how to install Gentoo on ZFS. An official procedure will be produced from it after I am happy with the procedure.
Compiling the modules with debug support can cause deadlocks in the kernel.- ZFS zvols currently cannot currently be safely used for ~~swap, dm-crypt, btrfs or`` ZFS (as a nested filesystem).
-
ZFS zvol devices can disappear upon reboot, although you can make them reappear by exporting and then importing the pool. This workaround is being tested in gentoofan/genkernel-zfs.(can no longer reproduce) -
The kernel modules currently cannot be compiled with CONFIG_KERNEL_PREEMPT set in the kernel config, although CONFIG_KERNEL_PREEMPT_VOLUNTARY is a viable substitute.(fixed in overlay) - The Linux kernel's virtual memory space is limited to about 100MB on 32-bit architectures, which will likely cause significant problems on such systems. You can workaround this by setting vmalloc to a larger figure.
- The kernel modules have not been properly optimized, so their CPU and memory overhead is larger than necessary.
- Some duplication occurs between ZFS ARC and the Linux file cache.
- The ZFSOnLinux regression tests currently fail on Gentoo Linux.
Efforts are being made to address these issues by ZFSOnLinux upstream.