Console shows Kernel crash during boot up with Mater debug image .339 on Dell S6000-ACS/S6000

[gechiang]

Description

Steps to reproduce the issue:

1. Install master.339 image to DUT
2. sudo reboot

Describe the results you received:
+----------------------------------------------------------------------------+
| SONiC-OS-201911.0-7d436b33 |
|*SONiC-OS-master.339-31baf381 |
| ONIE |
| |
| |
| |
| |
| |
| |
| |
| |
| |
+----------------------------------------------------------------------------+

  Use the ^ and v keys to select which entry is highlighted.
  Press enter to boot the selected OS, `e' to edit the commands
  before booting or `c' for a command-line.

Loading SONiC-OS OS kernel ...
Loading SONiC-OS OS initial ramdisk ...
[ 5.956688] pstore: crypto_comp_decompress failed, ret = -22!
[ 6.025465] pstore: decompression failed: -22
[ 8.015487] rc.local[338]: + sonic-cfggen -y /etc/sonic/sonic_version.yml -v build_version
[ 9.456310] kdump-tools[355]: Starting kdump-tools: no crashkernel= parameter in the kernel cmdline ... failed!
[ 10.882070] kvm: disabled by bios
[ 10.975780] kvm: disabled by bios
[ 11.052886] kvm: disabled by bios
[ 11.211011] kvm: disabled by bios
[ 13.274025] rc.local[338]: + SONIC_VERSION=master.339-31baf381
[ 13.352848] rc.local[338]: + FIRST_BOOT_FILE=/host/image-master.339-31baf381/platform/firsttime
[ 13.475293] rc.local[338]: + logger SONiC version master.339-31baf381 starting up...
[ 13.580693] rc.local[338]: + [ ! -e /host/machine.conf ]
[ 13.661107] rc.local[338]: + . /host/machine.conf
[ 13.720928] rc.local[338]: + onie_version=3.20.1.5
[ 13.780935] rc.local[338]: + onie_vendor_id=674
[ 13.842340] rc.local[338]: + onie_platform=x86_64-dell_s6000_s1220-r0
[ 13.928766] rc.local[338]: + onie_machine=dell_s6000_s1220
[ 14.000666] rc.local[338]: + onie_machine_rev=0
[ 14.061165] rc.local[338]: + onie_arch=x86_64
[ 14.124761] rc.local[338]: + onie_config_version=1
[ 14.188694] rc.local[338]: + onie_build_date=2016-02-09T17:14-0800
[ 14.268655] rc.local[338]: + onie_partition_type=gpt
[ 14.341208] rc.local[338]: + onie_kernel_version=3.15.10
[ 14.412676] rc.local[338]: + onie_firmware=auto
[ 14.472687] rc.local[338]: + program_console_speed
[ 14.536373] rc.local[338]: + cat /proc/cmdline
[ 14.606010] rc.local[338]: + cut -d , -f2
[ 14.665629] rc.local[338]: + grep -Eo console=ttyS[0-9]+,[0-9]+
[ 14.750144] rc.local[338]: + speed=9600
[ 14.805031] rc.local[338]: + [ -z 9600 ]
[ 14.860756] rc.local[338]: + CONSOLE_SPEED=9600
[ 14.920880] rc.local[338]: + sed -i s|--keep-baud .* %I| 9600 %I|g /lib/systemd/system/[email protected]
[ 15.048694] rc.local[338]: + systemctl daemon-reload
[ 15.120727] rc.local[338]: + [ -f /host/image-master.339-31baf381/platform/firsttime ]
[ 15.224707] rc.local[338]: + [ -f /var/log/fsck.log.gz ]
[ 15.312109] rc.local[338]: + logger -t FSCK
[ 15.373750] rc.local[338]: + gunzip -d -c /var/log/fsck.log.gz
[ 15.449362] rc.local[338]: + rm -f /var/log/fsck.log.gz
[ 15.520799] rc.local[338]: + exit 0

Debian GNU/Linux 10 str-s6000-acs-8 ttyS0

[ 33.176426] list_add corruption. prev->next should be next (ffff97342e094110), but was ffff97342e094118. (prev=ffff97342e094118).
[ 33.316220] kernel BUG at lib/list_debug.c:28!
[ 33.369561] invalid opcode: 0000 [#1] SMP NOPTI
[ 33.423853] CPU: 1 PID: 770 Comm: set-fan-speed Tainted: G OE 4.19.0-6-2-amd64 sonic-net/sonic-buildimage#1 Debian 4.19.67-2+deb10u2
[ 33.554057] Hardware name: Dell Inc S6000-ACS/S6000 CPU, BIOS 4.6.5 10/12/2015
[ 33.640599] RIP: 0010:__list_add_valid.cold.0+0x26/0x28
[ 33.703201] Code: 00 00 00 c3 48 89 d1 48 c7 c7 b8 b1 09 b2 48 89 c2 e8 c0 96 d0 ff 0f 0b 48 89 c1 4c 89 c6 48 c7 c7 10 b2 09 b2 e8 ac 96 d0 ff <0f> 0b 48 89 fe 48 89 c2 48 c7 c7 a0 b2 09 b2 e8 98 96 d0 ff 0f 0b
[ 33.928045] RSP: 0000:ffffb94a41307db0 EFLAGS: 00010246
[ 33.990645] RAX: 0000000000000075 RBX: ffffb94a41307e00 RCX: 0000000000000000
[ 34.076131] RDX: 0000000000000000 RSI: ffff973437a966b8 RDI: ffff973437a966b8
[ 34.161503] RBP: ffff97342e094110 R08: 0000000000000339 R09: 0000000000000004
[ 34.246887] R10: 000000000000072e R11: ffffffffb27f26ad R12: ffff97342e094118
[ 34.332270] R13: ffff97342e094100 R14: ffffb94a41307f08 R15: ffff97342e094100
[ 34.417657] FS: 00007fa40d4dc740(0000) GS:ffff973437a80000(0000) knlGS:0000000000000000
[ 34.514481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 34.583222] CR2: 00007f52592af024 CR3: 000000022e194000 CR4: 00000000000006e0
[ 34.668610] Call Trace:
[ 34.697843] __mutex_add_waiter+0x23/0x50
[ 34.745783] __mutex_lock.isra.8+0x1d1/0x4a0
[ 34.796846] ? __handle_mm_fault+0xa83/0x1270
[ 34.848948] ? _kstrtoull+0x3b/0x90
[ 34.890657] set_target+0x95/0xf0 [dni_dps460]
[ 34.943797] kernfs_fop_write+0x116/0x190
[ 34.991740] vfs_write+0xa5/0x1a0
[ 35.031363] ksys_write+0x57/0xd0
[ 35.070990] do_syscall_64+0x53/0x110
[ 35.114773] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 35.175195] RIP: 0033:0x7fa40d5c9504
[ 35.217939] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 f9 61 0d 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[ 35.442692] RSP: 002b:00007ffc1ff54c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 35.533268] RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00007fa40d5c9504
[ 35.618648] RDX: 0000000000000006 RSI: 00005648cb0ea6c0 RDI: 0000000000000001
[ 35.704032] RBP: 00005648cb0ea6c0 R08: 000000000000000a R09: 00007fa40d69aca0
[ 35.789415] R10: 000000000000000a R11: 0000000000000246 R12: 00007fa40d69b760
[ 35.874801] R13: 0000000000000006 R14: 00007fa40d696760 R15: 0000000000000006
[ 35.960186] Modules linked in: optoe(E) lm75(E) ltc4215(E) max6620(E) emc1403(E) jc42(E) dni_dps460(E) pmbus_core(E) at24(E) dell_s6000_platform(OE) i2c_mux_gpio(E) i2c_mux(E) nvram(E) nft_counter(E) xt_conntrack(E) nf_nat(E) nf_conntrack_netlink(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) libcrc32c(E) xfrm_user(E) xfrm_algo(E) intel_powerclamp(E) coretemp(E) bonding(E) kvm(E) evdev(E) dcdbas(E) efi_pstore(E) irqbypass(E) efivars(E) pcspkr(E) ie6xx_wdt(E) sg(E) linux_knet_cb(OE) linux_bcm_knet(OE) psample(OE) linux_user_bde(OE) linux_kernel_bde(OE) button(E) ebt_vlan(E) ebtable_broute(E) bridge(E) stp(E) llc(E) ebtable_nat(E) ebtable_filter(E) ebtables(E) nf_tables(E) nfnetlink(E) efivarfs(E) ip_tables(E) x_tables(E) autofs4(E) loop(E) ext4(E) crc16(E) mbcache(E) jbd2(E) crc32c_generic(E)
[ 36.808149] fscrypto(E) ecb(E) crypto_simd(E) cryptd(E) glue_helper(E) aes_x86_64(E) nls_utf8(E) nls_cp437(E) nls_ascii(E) vfat(E) fat(E) overlay(E) squashfs(E) zstd_decompress(E) xxhash(E) sd_mod(E) i2c_isch(E) gpio_sch(E) ohci_pci(E) ohci_hcd(E) ehci_pci(E) ehci_hcd(E) ahci(E) lpc_sch(E) libahci(E) i2c_ismt(E) e1000e(E) libata(E) usbcore(E) usb_common(E) scsi_mod(E) thermal(E)
[ 37.211027] ---[ end trace 0b237926c185882f ]---
[ 37.278141] RIP: 0010:__list_add_valid.cold.0+0x26/0x28
[ 37.340781] Code: 00 00 00 c3 48 89 d1 48 c7 c7 b8 b1 09 b2 48 89 c2 e8 c0 96 d0 ff 0f 0b 48 89 c1 4c 89 c6 48 c7 c7 10 b2 09 b2 e8 ac 96 d0 ff <0f> 0b 48 89 fe 48 89 c2 48 c7 c7 a0 b2 09 b2 e8 98 96 d0 ff 0f 0b
[ 37.565735] RSP: 0000:ffffb94a41307db0 EFLAGS: 00010246
[ 37.628361] RAX: 0000000000000075 RBX: ffffb94a41307e00 RCX: 0000000000000000
[ 37.713962] RDX: 0000000000000000 RSI: ffff973437a966b8 RDI: ffff973437a966b8
[ 37.799537] RBP: ffff97342e094110 R08: 0000000000000339 R09: 0000000000000004
[ 37.884991] R10: 000000000000072e R11: ffffffffb27f26ad R12: ffff97342e094118
[ 37.970487] R13: ffff97342e094100 R14: ffffb94a41307f08 R15: ffff97342e094100
[ 38.055977] FS: 00007fa40d4dc740(0000) GS:ffff973437a80000(0000) knlGS:0000000000000000
[ 38.152895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 38.221705] CR2: 00007f52592af024 CR3: 000000022e194000 CR4: 00000000000006e0
[ 38.390127] e1000e: eth0 NIC Link is Down
[ OK ] Started Dell S6000 Platform modules.
[ OK ] Started Dell S6000 fan speed regulator.
Starting Platform monitor container...
Stopping System Logging Service...
[ OK ] Stopped System Logging Service.
Starting System Logging Service...
[ OK ] Started System Logging Service.
[ OK ] Started Update rsyslog configuration.
[ OK ] Started BGP container.
[ OK ] Started Platform monitor container.

**Output of `show version`:**

```

admin@str-s6000-acs-8:~$ show vers

SONiC Software Version: SONiC.master.339-31baf381
Distribution: Debian 10.4
Kernel: 4.19.0-6-2-amd64
Build commit: 31baf381
Build date: Tue Jul 7 07:01:13 UTC 2020
Built by: johnar@jenkins-worker-8

Platform: x86_64-dell_s6000_s1220-r0
HwSKU: Force10-S6000
ASIC: broadcom
Serial Number: 1QBRX42
Uptime: 20:32:35 up 1:09, 1 user, load average: 1.12, 1.09, 1.12

Docker images:
REPOSITORY TAG IMAGE ID SIZE
docker-sonic-mgmt-framework-dbg latest 794af0888273 532MB
docker-sonic-mgmt-framework-dbg master.339-31baf381 794af0888273 532MB
docker-sonic-mgmt-framework latest 794af0888273 532MB
docker-sonic-mgmt-framework master.339-31baf381 794af0888273 532MB
docker-router-advertiser-dbg latest 955c10a491a8 409MB
docker-router-advertiser-dbg master.339-31baf381 955c10a491a8 409MB
docker-router-advertiser latest 955c10a491a8 409MB
docker-router-advertiser master.339-31baf381 955c10a491a8 409MB
docker-lldp-dbg latest 91902f23a0e2 440MB
docker-lldp-dbg master.339-31baf381 91902f23a0e2 440MB
docker-lldp latest 91902f23a0e2 440MB
docker-lldp master.339-31baf381 91902f23a0e2 440MB
docker-dhcp-relay-dbg latest 2e3649d1b65c 417MB
docker-dhcp-relay-dbg master.339-31baf381 2e3649d1b65c 417MB
docker-dhcp-relay latest 2e3649d1b65c 417MB
docker-dhcp-relay master.339-31baf381 2e3649d1b65c 417MB
docker-database-dbg latest dd1067175c65 409MB
docker-database-dbg master.339-31baf381 dd1067175c65 409MB
docker-database latest dd1067175c65 409MB
docker-database master.339-31baf381 dd1067175c65 409MB
docker-orchagent-dbg latest 6db66c4c240d 502MB
docker-orchagent-dbg master.339-31baf381 6db66c4c240d 502MB
docker-orchagent latest 6db66c4c240d 502MB
docker-orchagent master.339-31baf381 6db66c4c240d 502MB
docker-sonic-telemetry-dbg latest 7596d23edc61 472MB
docker-sonic-telemetry-dbg master.339-31baf381 7596d23edc61 472MB
docker-sonic-telemetry latest 7596d23edc61 472MB
docker-sonic-telemetry master.339-31baf381 7596d23edc61 472MB
docker-snmp-dbg latest 4c7efb40365c 453MB
docker-snmp-dbg master.339-31baf381 4c7efb40365c 453MB
docker-snmp latest 4c7efb40365c 453MB
docker-snmp master.339-31baf381 4c7efb40365c 453MB
docker-teamd latest 049b0865b5c8 420MB
docker-teamd master.339-31baf381 049b0865b5c8 420MB
docker-teamd-dbg latest 049b0865b5c8 420MB
docker-teamd-dbg master.339-31baf381 049b0865b5c8 420MB
docker-syncd-brcm-dbg latest 046236ae18c9 534MB
docker-syncd-brcm-dbg master.339-31baf381 046236ae18c9 534MB
docker-syncd-brcm latest 046236ae18c9 534MB
docker-syncd-brcm master.339-31baf381 046236ae18c9 534MB
docker-sflow-dbg latest ed900f1d0288 391MB
docker-sflow-dbg master.339-31baf381 ed900f1d0288 391MB
docker-sflow latest ed900f1d0288 391MB
docker-sflow master.339-31baf381 ed900f1d0288 391MB
docker-platform-monitor-dbg latest 18bcfbee046d 425MB
docker-platform-monitor-dbg master.339-31baf381 18bcfbee046d 425MB
docker-platform-monitor latest 18bcfbee046d 425MB
docker-platform-monitor master.339-31baf381 18bcfbee046d 425MB
docker-nat-dbg latest b143359092bd 421MB
docker-nat-dbg master.339-31baf381 b143359092bd 421MB
docker-nat latest b143359092bd 421MB
docker-nat master.339-31baf381 b143359092bd 421MB
docker-fpm-frr-dbg latest e68c02a1a8aa 449MB
docker-fpm-frr-dbg master.339-31baf381 e68c02a1a8aa 449MB
docker-fpm-frr latest e68c02a1a8aa 449MB
docker-fpm-frr master.339-31baf381 e68c02a1a8aa 449MB
```