Realm linux kernel booting log (stuck point)

[zpzigi754]

This is the latest log about the stuck point. I leave it for the reference.

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x00000000]
[    0.000000] Linux version 6.2.0-rc1-gc865d47546ee-dirty (changho@changho) (aarch64-none-linux-gnu-gcc (Arm GNU Toolchain 11.3.Rel1) 11.3.1 20220712, GNU ld (Arm GNU Toolchain 11.3.Rel1) 2.38.20220708) #49 SMP PREEMPT Thu Jul  6 18:42:23 KST 2023
[    0.000000] Machine model: linux,dummy-virt
[    0.000000] RME: RSI version 1.0 advertised
[    0.000000] earlycon: pl11 at MMIO 0x000000001c0a0000 (options '')
[    0.000000] printk: bootconsole [pl11] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x000000008fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x8ff73a00-0x8ff75fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x000000008fffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x000000008fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000008fffffff]
[    0.000000] cma: Reserved 32 MiB at 0x000000008c400000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] percpu: Embedded 20 pages/cpu s42728 r8192 d31000 u81920
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Hardware dirty bit management
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] alternatives: applying boot alternatives
[    0.000000] Fallback order for Node 0: 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 64512
[    0.000000] Policy zone: DMA
[    0.000000] Kernel command line:  console=ttyS0 root=/dev/vda rw  console=pl011,mmio,0x1c0a0000 console=ttyAMA0 printk.devkmsg=on
[    0.000000] random: crng init done
[    0.000000] Dentry cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.000000] Inode-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: area num 1.
[    0.000000] software IO TLB: mapped [mem 0x0000000088400000-0x000000008c400000] (64MB)
[    0.000000] Memory: 121156K/262144K available (9472K kernel code, 962K rwdata, 2732K rodata, 1600K init, 482K bss, 108220K reserved, 32768K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=1.
[    0.000000] 	Trampoline variant of Tasks RCU enabled.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: 32 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x000000003ffd0000
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 100.00MHz (virt).
[    0.000000] clocksource: arch_sys_counter: mask: 0x1ffffffffffffff max_cycles: 0x171024e7e0, max_idle_ns: 440795205315 ns
[    0.000004] sched_clock: 57 bits at 100MHz, resolution 10ns, wraps every 4398046511100ns
[    0.007598] Console: colour dummy device 80x25
[    0.008730] Calibrating delay loop (skipped), value calculated using timer frequency.. 200.00 BogoMIPS (lpj=400000)
[    0.008863] pid_max: default: 32768 minimum: 301
[    0.009295] LSM: initializing lsm=capability,integrity
[    0.011317] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.012200] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.014753] Spurious percpu IRQ12 on CPU0
[    0.070603] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    0.104326] cblist_init_generic: Setting adjustable number of callback queues.
[    0.104408] cblist_init_generic: Setting shift to 0 and lim to 1.
[    0.107874] cblist_init_generic: Setting shift to 0 and lim to 1.
[    0.120313] rcu: Hierarchical SRCU implementation.
[    0.120375] rcu: 	Max phase no-delay instances is 1000.
[    0.130800] EFI services will not be available.
[    0.141180] smp: Bringing up secondary CPUs ...
[    0.141238] smp: Brought up 1 node, 1 CPU
[    0.141317] SMP: Total of 1 processors activated.
[    0.141385] CPU features: detected: Branch Target Identification
[    0.141466] CPU features: detected: ARMv8.4 Translation Table Level
[    0.141542] CPU features: detected: Data cache clean to the PoU not required for I/D coherence
[    0.141637] CPU features: detected: Common not Private translations
[    0.141716] CPU features: detected: CRC32 instructions
[    0.141786] CPU features: detected: Data independent timing control (DIT)
[    0.141868] CPU features: detected: E0PD
[    0.141930] CPU features: detected: Enhanced Counter Virtualization
[    0.142020] CPU features: detected: RCpc load-acquire (LDAPR)
[    0.142095] CPU features: detected: LSE atomic instructions
[    0.142168] CPU features: detected: Privileged Access Never
[    0.142242] CPU features: detected: RAS Extension Support
[    0.142314] CPU features: detected: Random Number Generator
[    0.142388] CPU features: detected: Speculation barrier (SB)
[    0.142460] CPU features: detected: Stage-2 Force Write-Back
[    0.142537] CPU features: detected: TLB range maintenance instructions
[    0.143210] CPU: All CPU(s) started at EL1
[    0.143294] alternatives: applying system-wide alternatives
[    0.189495] devtmpfs: initialized
[    0.222596] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.222728] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[    0.237759] DMI not present or invalid.
[    0.249876] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.292676] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[    0.323034] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.353423] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.355272] audit: initializing netlink subsys (disabled)
[    0.368726] audit: type=2000 audit(0.000:1): state=initialized audit_enabled=0 res=1
[    0.368838] cpuidle: using governor menu
[    0.370915] hw-breakpoint: found 2 breakpoint and 2 watchpoint registers.
[    0.371050] ASID allocator initialised with 65536 entries
[    0.382105] Serial: AMBA PL011 UART driver
[    0.396964] KASLR enabled
[    0.414556] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[    0.414639] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
[    0.414735] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages
[    0.414817] HugeTLB: 508 KiB vmemmap can be freed for a 32.0 MiB page
[    0.414902] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    0.414994] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
[    0.415088] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages
[    0.415170] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page
[    0.452616] iommu: Default domain type: Translated 
[    0.452687] iommu: DMA domain TLB invalidation policy: strict mode 
[    0.459546] SCSI subsystem initialized
[    0.467988] pps_core: LinuxPPS API ver. 1 registered
[    0.468056] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.468214] PTP clock support registered
[    0.486697] clocksource: Switched to clocksource arch_sys_counter
[    0.518650] VFS: Disk quotas dquot_6.6.0
[    0.519647] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.620214] NET: Registered PF_INET protocol family
[    0.627724] IP idents hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.636044] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.636980] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.637122] TCP established hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.640546] TCP bind hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.653370] TCP: Hash tables configured (established 2048 bind 2048)
[    0.667503] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.669222] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.671176] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.691989] RPC: Registered named UNIX socket transport module.
[    0.692062] RPC: Registered udp transport module.
[    0.692127] RPC: Registered tcp transport module.
[    0.692188] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.702813] kvm [1]: HYP mode not available
[    0.710786] Initialise system trusted keyrings
[    0.718649] Unpacking initramfs...
[    1.024726] ------------[ cut here ]------------
[    1.024785] Unexpected interrupt (irqnr 75)
[    1.025160] WARNING: CPU: 0 PID: 9 at drivers/irqchip/irq-gic-v3.c:697 gic_handle_irq+0x118/0x124
[    1.025305] Modules linked in:
[    1.025375] CPU: 0 PID: 9 Comm: kworker/u2:0 Not tainted 6.2.0-rc1-gc865d47546ee-dirty #49
[    1.025500] Hardware name: linux,dummy-virt (DT)
[    1.025572] Workqueue: events_unbound async_run_entry_fn
[    1.025710] pstate: 614000c5 (nZCv daIF +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[    1.025832] pc : gic_handle_irq+0x118/0x124
[    1.025925] lr : gic_handle_irq+0x118/0x124
[    1.026017] sp : ffff800008003fe0
[    1.026079] x29: ffff800008003fe0 x28: ffff000001068000 x27: 0000000000000d30
[    1.026253] x26: 0000000000000000 x25: ffffd7c5d41ac000 x24: ffff8000080937b8
[    1.026427] x23: 0000000081400005 x22: ffffd7c5d35ec488 x21: ffff800008093730
[    1.026602] x20: 000000000000004b x19: 000000000000004b x18: ffffffffffffffff
[    1.026773] x17: ffff283a3bea5000 x16: ffff800008004000 x15: ffff800088003c57
[    1.026950] x14: 0000000000000000 x13: ffffd7c5d41bff98 x12: 00000000000004f2
[    1.027120] x11: 00000000000001a6 x10: ffffd7c5d4217f98 x9 : ffffd7c5d41bff98
[    1.027294] x8 : 00000000ffffefff x7 : ffffd7c5d4217f98 x6 : 0000000000000000
[    1.027464] x5 : 000000000000bff4 x4 : 0000000000000000 x3 : 0000000000000000
[    1.027630] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000001068000
[    1.027798] Call trace:
[    1.027848]  gic_handle_irq+0x118/0x124
[    1.027942]  call_on_irq_stack+0x2c/0x5c
[    1.028046]  do_interrupt_handler+0x80/0x84
[    1.028164]  el1_interrupt+0x34/0x70
[    1.028277]  el1h_64_irq_handler+0x18/0x2c
[    1.028398]  el1h_64_irq+0x64/0x68
[    1.028489]  release_pages+0x74/0x470
[    1.028614]  folio_batch_move_lru+0x100/0x1cc
[    1.028743]  folio_add_lru+0x7c/0xe0
[    1.028869]  filemap_add_folio+0x60/0xc4
[    1.028967]  __filemap_get_folio+0x210/0x410
[    1.029072]  pagecache_get_page+0x18/0x70
[    1.029194]  grab_cache_page_write_begin+0x18/0x24
[    1.029323]  simple_write_begin+0x2c/0xf4
[    1.029443]  generic_perform_write+0xa8/0x1e0
[    1.029578]  __generic_file_write_iter+0x11c/0x1a4
[    1.029688]  generic_file_write_iter+0x78/0xfc
[    1.029792]  __kernel_write_iter+0x94/0x200
[    1.029928]  kernel_write+0xd0/0x190
[    1.030018]  xwrite.constprop.0+0x4c/0xc4
[    1.030140]  do_copy+0xc8/0x124
[    1.030255]  write_buffer+0x38/0x58
[    1.030368]  flush_buffer+0x4c/0xbc
[    1.030487]  __gunzip+0x27c/0x30c
[    1.030573]  gunzip+0x1c/0x28
[    1.030661]  unpack_to_rootfs+0x154/0x278
[    1.030782]  do_populate_rootfs+0x68/0x12c
[    1.030906]  async_run_entry_fn+0x34/0xe0
[    1.031028]  process_one_work+0x1d0/0x320
[    1.031141]  worker_thread+0x14c/0x444
[    1.031247]  kthread+0x10c/0x110
[    1.031339]  ret_from_fork+0x10/0x20
[    1.031440] ---[ end trace 0000000000000000 ]---

[zpzigi754]

The below is another log with the latest commit (9d63ba7).

(the same with the above until the `Unpacking initramfs` is reached)
...
[    0.706113] Initialise system trusted keyrings
[    0.713935] Unpacking initramfs...
[   24.504593] Freeing initrd memory: 21360K
[   24.505640] workingset: timestamp_bits=42 max_order=16 bucket_order=0
[   24.512055] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[   24.520195] NFS: Registering the id_resolver key type
[   24.520308] Key type id_resolver registered
[   24.520373] Key type id_legacy registered
[   24.520507] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[   24.520588] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[   24.522683] 9p: Installing v9fs 9p2000 file system support
[   24.881148] Key type asymmetric registered
[   24.881214] Asymmetric key parser 'x509' registered
[   24.887639] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[   24.887723] io scheduler mq-deadline registered
[   24.887807] io scheduler kyber registered
[   24.948791] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[   24.962175] printk: console [ttyS0] disabled
[   24.962522] 1000000.U6_16550A: ttyS0 at MMIO 0x1000000 (irq = 13, base_baud = 115200) is a 16550A
[   24.974085] printk: console [ttyS0] enabled
[   86.458388] 1001000.U6_16550A: ttyS1 at MMIO 0x1001000 (irq = 14, base_baud = 115200) is a 16550A
[   88.668515] 1002000.U6_16550A: ttyS2 at MMIO 0x1002000 (irq = 15, base_baud = 115200) is a 16550A
[   90.879431] 1003000.U6_16550A: ttyS3 at MMIO 0x1003000 (irq = 16, base_baud = 115200) is a 16550A
[   91.265150] cacheinfo: Unable to detect cache hierarchy for CPU 0
[   91.694343] loop: module loaded
[   91.828884] tun: Universal TUN/TAP device driver, 1.6
[   92.000766] virtio_net virtio0: virtio: device refuses features: 0
[   92.195492] i2c_dev: i2c /dev entries driver
[   92.329975] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
[   92.533323] NET: Registered PF_PACKET protocol family
[   92.688551] 9pnet: Installing 9P2000 support
[   92.819701] Key type dns_resolver registered
[   93.024913] registered taskstats version 1
[   93.154813] Loading compiled-in X.509 certificates
[   93.511721] Freeing unused kernel memory: 1600K

[bokdeuk-jeong]

[ cause of the error ]
nw log

[ 4617.768991] [vcpu 0] Unhandled exit reason from realm (ESR: 0x7e00000)
KVM_RUN failed: No such device or address

islet log

[DEBUG]armv9a::exception::trap -- Synchronous: Other
[DEBUG]armv9a::exception::trap -- [ESR_EL2=7E00000 ELR_EL2=FFFF80000892AF28  // added by me
[TRACE]armv9a::smc -- cmd[C40001CF], args[0], ret[C4000151:GRANULE_DELEGATE, 88089B000, FFFFFF8000000000, FFFFFF887F809180, 0, 900000, 88089B, 0]

corresponding realm linux address

ffff80000892af20 <cpu_do_idle>:
ffff80000892af20:   d503233f    paciasp
ffff80000892af24:   d5033f9f    dsb sy
ffff80000892af28:   d503207f    wfi   // here
ffff80000892af2c:   d50323bf    autiasp
ffff80000892af30:   d65f03c0    ret

TODO: handle wfi.

[bokdeuk-jeong]

Status after applying #116:

...............[ Omitting the kernel startup log messages : same as above].......................
[    0.290784] RPC: Registered tcp transport module.
[    0.290809] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.293311] kvm [1]: HYP mode not available
[    0.297543] Unpacking initramfs...
[    0.305502] Initialise system trusted keyrings
[    0.309053] workingset: timestamp_bits=42 max_order=16 bucket_order=0
[    0.316387] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.322950] NFS: Registering the id_resolver key type
[    0.323017] Key type id_resolver registered
[    0.323046] Key type id_legacy registered
[    0.323231] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.323263] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.329398] 9p: Installing v9fs 9p2000 file system support
[    1.106433] Key type asymmetric registered
[    1.106462] Asymmetric key parser 'x509' registered
[    1.108511] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[    1.108545] io scheduler mq-deadline registered
[    1.108573] io scheduler kyber registered
[    1.163762] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    1.179666] printk: console [ttyS0] disabled
[    1.179941] 1000000.U6_16550A: ttyS0 at MMIO 0x1000000 (irq = 13, base_baud = 115200) is a 16550A
[    1.180929] printk: console [ttyS0] enabled
[    5.436760] 1001000.U6_16550A: ttyS1 at MMIO 0x1001000 (irq = 14, base_baud = 115200) is a 16550A
[    5.480759] 1002000.U6_16550A: ttyS2 at MMIO 0x1002000 (irq = 15, base_baud = 115200) is a 16550A
[    5.527892] 1003000.U6_16550A: ttyS3 at MMIO 0x1003000 (irq = 16, base_baud = 115200) is a 16550A
[    5.570894] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    5.750572] loop: module loaded
[    5.806686] tun: Universal TUN/TAP device driver, 1.6
Bus error

[bokdeuk-jeong]

The log from the realm kernel on the reference rmm project is different from the previsou log on ISLET that Changho left.

ISLET:

[   91.828884] tun: Universal TUN/TAP device driver, 1.6
[   92.000766] virtio_net virtio0: virtio: device refuses features: 0
[   92.195492] i2c_dev: i2c /dev entries driver

on the reference rmm:

[   10.637184] tun: Universal TUN/TAP device driver, 1.6
[   10.799974] software IO TLB: Memory encryption is active and system is using DMA bounce buffers
[   10.879901] net eth0: Fail to set guest offload.
[   10.918833] virtio_net virtio1 eth0: set_features() failed (-22); wanted 0x0000000000134829, left 0x0080000000134829
[   11.028899] i2c_dev: i2c /dev entries driver

[zpzigi754]

The below would be the log after applying #156.

...
[   31.686181] tun: Universal TUN/TAP device driver, 1.6
[   32.031429] software IO TLB: Memory encryption is active and system is using DMA bounce buffers
[   32.298157] net eth0: Fail to set guest offload.
[   32.445749] virtio_net virtio1 eth0: set_features() failed (-22); wanted 0x0000000000134829, left 0x0080000000134829
[   32.794106] i2c_dev: i2c /dev entries driver
[   32.935215] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
[   33.144430] NET: Registered PF_PACKET protocol family
[   33.302692] 9pnet: Installing 9P2000 support
[   33.506784] Key type dns_resolver registered
[   33.803595] registered taskstats version 1
[   33.945874] Loading compiled-in X.509 certificates
[   46.611991] Freeing initrd memory: 21360K
[   46.704189] Freeing unused kernel memory: 1600K
[   46.741478] Run /init as init process
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Saving 256 bits of creditable seed for next boot
Starting network: OK
Starting dhcpcd...
no such user dhcpcd
dhcpcd-9.4.1 starting
no interfaces have a carrier
forked to background, child pid 81

Welcome to Buildroot
buildroot login: