Unable to run Fedora (or large ramdiskfs) with Spike

[caizixian]

Impact: software (FireMarshal)| Spike

Tell us about your environment:
Chipyard Version: 1.5.0 (b5d013190d637e634113cb5179f8c8885df1945a)
OS: 5.4.0-80-generic #90~18.04.1-Ubuntu SMP Tue Jul 13 19:40:02 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux (Ubuntu 18.04 LTS with HWE kernel)
Other:

1. Clone chipyard repo
2. Build toolchain https://chipyard.readthedocs.io/en/latest/Chipyard-Basics/Initial-Repo-Setup.html#building-a-toolchain and source env.sh
3. In FireMarshal, run ./marshal -v -d build fedora-base.json and then ./marshal -d launch -s fedora-base.json.

What is the current behavior?
It displays the following and stuck.

OpenSBI v0.8
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name       : ucbbar,spike-bare
Platform Features   : timer,mfdeleg
Platform HART Count : 4
Boot HART ID        : 0
Boot HART ISA       : rv64imafdcsu
BOOT HART Features  : pmp,scounteren,mcounteren
BOOT HART PMP Count : 16
Firmware Base       : 0x80000000
Firmware Size       : 116 KB
Runtime SBI Version : 0.2

MIDELEG : 0x0000000000000222
MEDELEG : 0x000000000000b109
PMP0    : 0x0000000080000000-0x000000008001ffff (A)
PMP1    : 0x0000000000000000-0x01ffffffffffffff (A,R,W,X)

I ran the Spike command and step through the instructions where it gets stuck. I can see Spike is looping the following.

OpenSBI v0.8
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name       : ucbbar,spike-bare
Platform Features   : timer,mfdeleg
Platform HART Count : 1
Boot HART ID        : 0
Boot HART ISA       : rv64imafdcsu
BOOT HART Features  : pmp,scounteren,mcounteren
BOOT HART PMP Count : 16
Firmware Base       : 0x80000000
Firmware Size       : 92 KB
Runtime SBI Version : 0.2

MIDELEG : 0x0000000000000222
MEDELEG : 0x000000000000b109
PMP0    : 0x0000000080000000-0x000000008001ffff (A)
PMP1    : 0x0000000000000000-0x01ffffffffffffff (A,R,W,X)
^C: core   0: 0x00000000802000fc (0x0000bff5) c.j     pc - 4
: core   0: 0x00000000802000f8 (0x10500073) wfi
: core   0: 0x00000000802000fc (0x0000bff5) c.j     pc - 4
: core   0: 0x00000000802000f8 (0x10500073) wfi
: core   0: 0x00000000802000fc (0x0000bff5) c.j     pc - 4

What is the expected behavior?
The image boots.

Other information

• Vanilla br-base.json boots with Spike.
• Switching the bootloader from OpenSBI to bbl for fedora-base.json doesn't help.
• Fedora boots fine with QEMU.
• I create another image based on br-base, include something big (AdoptOpenJDK 11 RISC-V) via files, and then set rootfs-size to 0. That doesn't boot (similar issue with Fedora). But it does boot with QEMU.

[caizixian]

I'm not sure whether this is an upstream issue (i.e. FireMarshal or Spike), or it's due to a particular combination/usage.

[NathanTP]

I can reproduce on both Fedora and Buildroot. The limit seems to be around 128MB. It occurs on both Qemu and Spike for any no-disk build over that limit.

We're looking into this. It's strange because this used to work and I can't think of anything we changed recently that would break it. It seems similar to a Linux bug we had a while ago but @a0u fixed that here. Maybe @a0u has some idea.

[caizixian]

I can reproduce on both Fedora and Buildroot. The limit seems to be around 128MB. It occurs on both Qemu and Spike for any no-disk build over that limit.

We're looking into this. It's strange because this used to work and I can't think of anything we changed recently that would break it. It seems similar to a Linux bug we had a while ago but @a0u fixed that here. Maybe @a0u has some idea.

Thanks a lot for looking into this.

[NathanTP]

I've moved this bug over to firesim/FireMarshal#210 as this is a FireMarshal specific issue.

[a0u]

This is a limitation of the v5.7 Linux kernel. The provisional page tables created when virtual memory is first enabled can map at most a certain range, hardcoded to be only 128 MiB by default. If the size of the kernel image exceeds this limit, then setup_vm() simply BUG()s out and enters an infinite loop:
https://github.com/firesim/linux/blob/280191c0a6693bce79bec8ef235f58d5f3de4a47/arch/riscv/mm/init.c#L393

In software/firemarshal/boards/default/linux/arch/riscv/mm/init.c, increase MAX_EARLY_MAPPING_SIZE as appropriate for your initramfs:
https://github.com/firesim/linux/blob/280191c0a6693bce79bec8ef235f58d5f3de4a47/arch/riscv/mm/init.c#L190

#define MAX_EARLY_MAPPING_SIZE	0x200000000UL /* 8 GiB */

Fortunately, it appears this limitation has been fixed as of v5.12 (torvalds/linux@0f02de4).