FEMU

We want to emulate ZNS for which we can easily utilize FEMU to emulate the ZNS for us.

FEMU with 5.19 Kernel

When booting we do not have the nvme controller start due to some timeout in probing, however the device shows up on pci. From the Kernel log after booting we get:

[   1.213689] nvme nvme0: pci function 0000:00:05.0
[   9.238541] nvme nvme0: Device not ready; aborting initialisation, CSTS=0x2
[   9.312561] nvme nvme0: Removing after probe failure status: -19

It shows up on hardware info

root@stosys:/home/user# lshw | grep -i "storage" -A12
        *-storage
             description: Non-Volatile memory controller
             product: QEMU NVM Express Controller
             vendor: Intel Corporation
             physical id: 5
             bus info: pci@0000:00:05.0
             logical name: /dev/fb0
             version: 02
             width: 64 bits
             clock: 33MHz
             capabilities: storage msi msix pciexpress nvm_express cap_list fb
             configuration: depth=32 latency=0 mode=1280x800 visual=truecolor xres=1280 yres=800
             resources: iomemory:50-4f memory:540000000-540001fff memory:c0000000-c0000fff

Then we can remove the device and rescan pci

root@stosys:/home/user# echo 1 > /sys/bus/pci/devices/0000\:00\:05.0/remove
root@stosys:/home/user# sleep 1
root@stosys:/home/user# echo 1 > /sys/bus/pci/rescan

Where we can then see when comparing the kernel log that it times out again

[   84.246201] pci 0000:00:05.0: [8086:5845] type 00 class 0x010802
[   84.248858] pci 0000:00:05.0: reg 0x10: [mem 0xfebd0000-0xfebd1fff 64bit]
[   84.254718] pci 0000:00:05.0: reg 0x20: [mem 0xfebd5000-0xfebd5fff]
[   84.267550] pci 0000:00:05.0: BAR 0: assigned [mem 0x540000000-0x540001fff 64bit]
[   84.268411] pci 0000:00:05.0: BAR 4: assigned [mem 0xc0000000-0xc0000fff]
[   84.274786] nvme nvme0: pci function 0000:00:05.0
[   92.315218] nvme nvme0: Device not ready; aborting initialisation, CSTS=0x2
[   92.315376] nvme nvme0: Removing after probe failure status: -19

Fix

With a bit of debugging I tacked the issue down to the order of controller data and enable bit being sent when registering the controller. This only happened when I was using Kernel 5.19+. We have a fork with the fixed modifications (PR coming) available here: https://github.com/nicktehrany/FEMU The available code there also has a commit (72f9d82) that adds enforcing of 4KiB ZNS LBAF, which by default is 512B.

Running FEMU

This is the starting script that I use for running FEMU.

#!/bin/bash

set -e

U20_IMG="/mnt/sdb/nty/images/ubuntu-22.04-f2fs.qcow2"
# U20_IMG="/mnt/sdb/nty/images/ubuntu-22.04.qcow2"
# U20_IMG=/mnt/sdb/nty/images/femu-20.04.qcow2

sudo ${FEMU_BUILD}/qemu-system-x86_64 -name f2fs-vm -m 20G -cpu host -smp 12 -enable-kvm -nographic \
    -device virtio-scsi-pci,id=scsi0 \
    -device scsi-hd,drive=hd0 \
    -device femu,devsz_mb=81920,femu_mode=3,id=nvme0 \
    -drive file=${U20_IMG},if=none,aio=native,cache=none,format=qcow2,id=hd0 \
    -net nic,model=virtio -net user,hostfwd=tcp::8888-:22,hostfwd=tcp::3333-:3000 -device virtio-tablet-pci,id=tablet0,serial=virtio-tablet \