ArubaOS CX 10.12.0006 Not Booting

[aj-cruz]

Not sure if this is a problem with my environment or what, other VMs (Juniper, 9Kv) boot fine, but the 10.12.0006 Aruba image does not.
Below are the container logs.
I was able to extract the vmdk from the OVA and the make docker-image completed without error.

2024-02-20 20:57:06,643: launch     DEBUG    Environment variables: environ({'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'HOSTNAME': 'ARUBA', 'TERM': 'xterm', 'CLAB_INTFS': '2', 'DOCKER_NET_V6_ADDR': '2001:172:20:20::/64', 'CLAB_LABEL_CLAB_TOPO_FILE': '/home/rocky/labs/aruba/topology.clab.yml', 'CLAB_LABEL_CLAB_NODE_TYPE': '', 'DOCKER_NET_V4_ADDR': '172.20.20.0/24', 'CONNECTION_MODE': 'tc', 'USERNAME': 'admin', 'PASSWORD': 'admin', 'CLAB_LABEL_CLAB_NODE_GROUP': '', 'CLAB_LABEL_CLAB_NODE_LAB_DIR': '/home/rocky/labs/aruba/clab-aruba/ARUBA', 'CLAB_LABEL_CONTAINERLAB': 'aruba', 'CLAB_LABEL_CLAB_NODE_NAME': 'ARUBA', 'CLAB_LABEL_CLAB_NODE_KIND': 'vr-aoscx', 'DEBIAN_FRONTEND': 'noninteractive', 'HOME': '/root', 'LC_CTYPE': 'C.UTF-8'})
2024-02-20 20:57:06,643: vrnetlab   DEBUG    Creating overlay disk image
2024-02-20 20:57:06,713: vrnetlab   DEBUG    Starting vrnetlab AOSCX
2024-02-20 20:57:06,713: vrnetlab   DEBUG    VMs: [<__main__.AOSCX_vm object at 0x7f30e1c4cd60>]
2024-02-20 20:57:06,719: vrnetlab   DEBUG    VM not started; starting!
2024-02-20 20:57:06,719: vrnetlab   INFO     Starting AOSCX_vm
2024-02-20 20:57:06,719: vrnetlab   DEBUG    number of provisioned data plane interfaces is 2
2024-02-20 20:57:06,719: vrnetlab   DEBUG    waiting for provisioned interfaces to appear...
2024-02-20 20:57:11,725: vrnetlab   DEBUG    highest allocated interface id determined to be: 2...
2024-02-20 20:57:11,726: vrnetlab   DEBUG    interfaces provisioned, continuing...
2024-02-20 20:57:11,727: vrnetlab   DEBUG    qemu cmd: qemu-system-x86_64 -enable-kvm -display none -machine pc -monitor tcp:0.0.0.0:4000,server,nowait -m 4096 -serial telnet:0.0.0.0:5000,server,nowait -drive if=ide,file=/arubaoscx-disk-image-genericx86-p4-20230531220439-overlay.vmdk -cpu host,level=9 -smp 2 -device pci-bridge,chassis_nr=1,id=pci.1 -device virtio-net-pci,netdev=p00,mac=0C:00:10:8f:22:00 -netdev user,id=p00,net=10.0.0.0/24,tftp=/tftpboot,hostfwd=tcp::2022-10.0.0.15:22,hostfwd=udp::2161-10.0.0.15:161,hostfwd=tcp::2830-10.0.0.15:830,hostfwd=tcp::2080-10.0.0.15:80,hostfwd=tcp::2443-10.0.0.15:443 -device virtio-net-pci,netdev=p01,mac=0C:00:1e:81:85:01,bus=pci.1,addr=0x2 -netdev tap,id=p01,ifname=tap1,script=/etc/tc-tap-ifup,downscript=no -device virtio-net-pci,netdev=p02,mac=0C:00:6e:08:7f:02,bus=pci.1,addr=0x3 -netdev tap,id=p02,ifname=tap2,script=/etc/tc-tap-ifup,downscript=no
2024-02-20 20:57:12,784: vrnetlab   INFO     STDOUT: 
2024-02-20 20:57:12,784: vrnetlab   INFO     STDERR: qemu-system-x86_64: error: failed to set MSR 0x345 to 0x2000
qemu-system-x86_64: ../../target/i386/kvm.c:2701: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

2024-02-20 20:57:12,786: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 1)
2024-02-20 20:57:13,788: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 2)
2024-02-20 20:57:14,789: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 3)
2024-02-20 20:57:15,790: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 4)
2024-02-20 20:57:16,792: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 5)
2024-02-20 20:57:17,793: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 6)
2024-02-20 20:57:18,794: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 7)
2024-02-20 20:57:19,795: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 8)
2024-02-20 20:57:20,797: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 9)
2024-02-20 20:57:21,798: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 10)
2024-02-20 20:57:22,800: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 11)
2024-02-20 20:57:23,801: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 12)
2024-02-20 20:57:24,803: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 13)
2024-02-20 20:57:25,804: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 14)
2024-02-20 20:57:26,805: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 15)
2024-02-20 20:57:27,807: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 16)
2024-02-20 20:57:28,808: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 17)
2024-02-20 20:57:29,810: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 18)
2024-02-20 20:57:30,811: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 19)
2024-02-20 20:57:31,813: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 20)
2024-02-20 20:57:32,814: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 21)
2024-02-20 20:57:33,816: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 22)
2024-02-20 20:57:34,817: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 23)
2024-02-20 20:57:35,818: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 24)
2024-02-20 20:57:36,820: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 25)
2024-02-20 20:57:37,821: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 26)
2024-02-20 20:57:38,822: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 27)
2024-02-20 20:57:39,824: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 28)
2024-02-20 20:57:40,825: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 29)
2024-02-20 20:57:41,827: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 30)
2024-02-20 20:57:42,828: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 31)
2024-02-20 20:57:43,829: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 32)
2024-02-20 20:57:44,831: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 33)
2024-02-20 20:57:45,832: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 34)
2024-02-20 20:57:46,834: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 35)
2024-02-20 20:57:47,835: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 36)
2024-02-20 20:57:48,837: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 37)
2024-02-20 20:57:49,838: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 38)
2024-02-20 20:57:50,840: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 39)
2024-02-20 20:57:51,841: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 40)
2024-02-20 20:57:52,843: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 41)
2024-02-20 20:57:53,844: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 42)
2024-02-20 20:57:54,845: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 43)
2024-02-20 20:57:55,847: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 44)
2024-02-20 20:57:56,848: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 45)
2024-02-20 20:57:57,850: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 46)
2024-02-20 20:57:58,851: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 47)
2024-02-20 20:57:59,852: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 48)
2024-02-20 20:58:00,854: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 49)
2024-02-20 20:58:01,856: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 50)
2024-02-20 20:58:02,857: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 51)
2024-02-20 20:58:03,859: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 52)
2024-02-20 20:58:04,860: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 53)
2024-02-20 20:58:05,861: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 54)
2024-02-20 20:58:06,863: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 55)
2024-02-20 20:58:07,864: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 56)
2024-02-20 20:58:08,865: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 57)
2024-02-20 20:58:09,867: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 58)
2024-02-20 20:58:10,868: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 59)
2024-02-20 20:58:11,870: vrnetlab   INFO     Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 60)
Traceback (most recent call last):
  File "/launch.py", line 189, in <module>
    vr.start()
  File "/vrnetlab.py", line 745, in start
    vm.work()
  File "/vrnetlab.py", line 676, in work
    self.check_qemu()
  File "/vrnetlab.py", line 690, in check_qemu
    self.start()
  File "/vrnetlab.py", line 206, in start
    raise QemuBroken(
vrnetlab.QemuBroken: Unable to connect to qemu monitor on port 4000

[hellt]

this is a sign of a hw/sw incompatibility or severe resource shortage.
since you said you got other images running, I would say resource shortage is not the culprit.

[JJ-empowered]

I've tested this on a proxmox cluster and on a set of bare metal servers, given the system 4-8 cores and up to 16g of ram. Every run results in the same error regardless of where it's deployed.

2024-08-08 18:38:32,209: vrnetlab INFO STDERR: qemu-system-x86_64: error: failed to set MSR 0x345 to 0x2000
qemu-system-x86_64: ../../target/i386/kvm.c:2701: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

2024-08-08 18:38:32,211: vrnetlab INFO Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 1)
2024-08-08 18:38:33,213: vrnetlab INFO Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 2)
2024-08-08 18:38:34,214: vrnetlab INFO Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 3)
2024-08-08 18:38:35,216: vrnetlab INFO Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 4)
2024-08-08 18:38:36,218: vrnetlab INFO Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 5)
2024-08-08 18:38:37,220: vrnetlab INFO Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 6)
2024-08-08 18:38:38,222: vrnetlab INFO Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 7)
2024-08-08 18:38:39,224: vrnetlab INFO Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 8)
2024-08-08 18:38:40,226: vrnetlab INFO Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 9)
2024-08-08 18:38:41,228: vrnetlab INFO Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 10)
2024-08-08 18:38:42,230: vrnetlab INFO Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 11)
2024-08-08 18:38:43,231: vrnetlab INFO Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 12)

[hellt]

@JJ-empowered This is a hw issue (cpu not properly emulated or not all cpu instructions supported)

[JJ-empowered]

Can you provide the specs for where you got this working? I've tried several setups including baremetal with cpucounters and all other enhanced instruction sets enabled that I'm aware of.

Also had no issues deploying Arista, Junos and a few others with the same setup

[hellt]

@ssasso, do you have some recent experience with aos-cx?

[hellt]

@JJ-empowered I dont use aos myself, but the error is clearly indicating the issue with the hw/virtualization, as it comes from qemu itself.

[ssasso]

@JJ-empowered can you please try with the latest 10.13 release?

I use it daily, and never encountered such issue...

(Just fyi I use it both on "physical" servers as well as on VM with nested virtualization)

[ssasso]

Btw, the error looks similar to what reported in: GNS3/gns3-server#1774

There are some workarounds reported there, did you already hava a look at this?