pstore doesn't persist panics correctly

[fionera]

While hacking around I found out that our pstore implementation doesn't correctly persist panics.

Metropolis: this is /dev/ttyS0. Verbose node logs follow.

        panichandler I Panic console: /dev/ttyS0
        panichandler I Panic console: /dev/ttyS1
                init I Starting Metropolis node init
                init I Version: v0.1.0-dev880.gc441f0b4.dirty
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2548be7]

goroutine 1 [running]:
main.main()
        metropolis/node/core/main.go:280 +0xc67

 Metropolis encountered an uncorrectable error and this node must be restarted.
core exit status: 2
  Disks synced, rebooting...

[    4.116511] reboot: Restarting system
=
BdsDxe: loading Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x4,0x0)
BdsDxe: starting Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x4,0x0)
[TRACE]: external/rules_rust~~crate~crate_index_efi__uefi-0.24.0/src/fs/file_system/fs.rs@327: Can't open file \EFI\metropolis\loader_state.pb: Error { status: NOT_FOUND, data: () }
Unable to load A/B loader state, using default slot A: while reading state file: IO error
Booting into Slot A

  Metropolis Cluster Operating System
  Copyright 2020-2024 The Monogon Project Authors


Metropolis: this is /dev/ttyS0. Verbose node logs follow.

        panichandler I Panic console: /dev/ttyS0
        panichandler I Panic console: /dev/ttyS1
                init I Starting Metropolis node init
                init I Version: v0.1.0-dev880.gc441f0b4.dirty
              pstore W �

[fionera]

With some more digging, it is already corrupted inside the EFI vars.

                init I reading "dump-type7-0-0-1734294273-D" in "cfc8fc79-be2e-4ddc-97f0-9f98bfe298a0"
                init I "\xcc": cc

[lorenz]

Weird, any idea why this is broken? Our code looks pretty bulletproof, maybe a kernel bug?

Anyways, we should probably move this into package and add a test if we want to make sure this keeps working as it is a pain to debug currently.