Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crash: invalid kernel virtual address: 0 type: "module symbols" #174

Open
correabuscar opened this issue Mar 4, 2024 · 0 comments
Open

Comments

@correabuscar
Copy link

correabuscar commented Mar 4, 2024

I've compiled a kernel on Gentoo with genkernel and saved a version of it with debugging symbols (and even its System.map but it doesn't seem to be needed for 'crash' because the output is identical with/without the map) then,
I booted into this kernel with nokaslr kernel cmdline and triggered a manual crash (via sysrq+s,u,s then c), which triggered kexec to make a dump via makedumpfile and when done it rebooted back into the normal system.
(Same kernel was used for normal and kexec: the stripped symbols version of it from /boot.)

click me to expand
-rw-r--r-- 1 root root 6181973 Mar  4 15:10 /boot/System.map-6.7.6-gentoo-x86_64
lrwxrwxrwx 1 root root      30 Mar  4 15:11 /boot/System.map -> System.map-6.7.6-gentoo-x86_64
...
-rw-r--r-- 1 root root 11858944 Mar  4 15:11 /boot/vmlinuz-6.7.6-gentoo-x86_64
lrwxrwxrwx 1 root root       27 Mar  4 15:11 /boot/kernel -> vmlinuz-6.7.6-gentoo-x86_64
...
 -rw-r--r-- 1 root root   542623392 Mar  4 15:10 vmlinux-6.7.6-gentoo-x86_64-2024-03-04-15_14_26
 -rw------- 1 root root 62402000856 Mar  4 17:37 crashdump-2024-03-04-17_36_29

Now I try to examine the crash dump like this:

click me to expand (to see output of this command)
# CRASHPAGER=/bin/cat crash -d1 -i autoexec --CRASHPAGER --mod /lib/modules/6.7.6-gentoo-x86_64/ vmlinux-6.7.6-gentoo-x86_64-2024-03-04-15_14_26 crashdump-2024-03-04-17_36_29


crash 8.0.4++
Copyright (C) 2002-2022  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
compressed kdump: header->utsname.machine: x86_64
diskdump_data: 
          filename: crashdump-2024-03-04-17_36_29
             flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED) 
               dfd: 3
               ofp: 0
      machine_type: 62 (EM_X86_64)

            header: 5893c2d663e0
           signature: "KDUMP   "
      header_version: 6
             utsname:
               sysname: Linux
              nodename: i87k
               release: 6.7.6-gentoo-x86_64
               version: #1 SMP Mon Mar  4 15:10:10 CET 2024
               machine: x86_64
            domainname: (none)
           timestamp:
                tv_sec: 65e5f86b
               tv_usec: 0
              status: 0 ()
          block_size: 4096
        sub_hdr_size: 2
       bitmap_blocks: 1044
           max_mapnr: 17100800
    total_ram_blocks: 0
       device_blocks: 0
      written_blocks: 0
         current_cpu: 0
             nr_cpus: 12
      tasks[nr_cpus]: 0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0
                      0

        sub_header: 0 (n/a)

  sub_header_kdump: 5893c2d673f0 
           phys_base: 0
          dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO)
               split: 0
           start_pfn: (unused)
             end_pfn: (unused)
   offset_vmcoreinfo: 8496 (0x2130)
     size_vmcoreinfo: 3202 (0xc82)
                      OSRELEASE=6.7.6-gentoo-x86_64
                      BUILD-ID=582684599a3cc125e8a53f43ce93a8a30d0f7ab2
                      PAGESIZE=4096
                      SYMBOL(init_uts_ns)=ffffffff837ebf60
                      OFFSET(uts_namespace.name)=0
                      SYMBOL(node_online_map)=ffffffff837f1268
                      SYMBOL(swapper_pg_dir)=ffffffff8302c000
                      SYMBOL(_stext)=ffffffff81000000
                      SYMBOL(vmap_area_list)=ffffffff8374e630
                      SYMBOL(mem_map)=ffffffff841fd030
                      SYMBOL(contig_page_data)=ffffffff837ec1c0
                      SYMBOL(mem_section)=ffff88904a7f6000
                      LENGTH(mem_section)=4096
                      SIZE(mem_section)=32
                      OFFSET(mem_section.section_mem_map)=0
                      NUMBER(SECTION_SIZE_BITS)=27
                      NUMBER(MAX_PHYSMEM_BITS)=46
                      SIZE(page)=64
                      SIZE(pglist_data)=7360
                      SIZE(zone)=1600
                      SIZE(free_area)=104
                      SIZE(list_head)=16
                      SIZE(nodemask_t)=8
                      OFFSET(page.flags)=0
                      OFFSET(page._refcount)=52
                      OFFSET(page.mapping)=24
                      OFFSET(page.lru)=8
                      OFFSET(page._mapcount)=48
                      OFFSET(page.private)=40
                      OFFSET(page.compound_head)=8
                      OFFSET(pglist_data.node_zones)=0
                      OFFSET(pglist_data.nr_zones)=6480
                      OFFSET(pglist_data.node_start_pfn)=6488
                      OFFSET(pglist_data.node_spanned_pages)=6504
                      OFFSET(pglist_data.node_id)=6512
                      OFFSET(zone.free_area)=192
                      OFFSET(zone.vm_stat)=1472
                      OFFSET(zone.spanned_pages)=136
                      OFFSET(free_area.free_list)=0
                      OFFSET(list_head.next)=0
                      OFFSET(list_head.prev)=8
                      OFFSET(vmap_area.va_start)=0
                      OFFSET(vmap_area.list)=40
                      LENGTH(zone.free_area)=11
                      SYMBOL(prb)=ffffffff83042270
                      SYMBOL(printk_rb_static)=ffffffff83042280
                      SYMBOL(clear_seq)=ffffffff841f48e0
                      SIZE(printk_ringbuffer)=88
                      OFFSET(printk_ringbuffer.desc_ring)=0
                      OFFSET(printk_ringbuffer.text_data_ring)=48
                      OFFSET(printk_ringbuffer.fail)=80
                      SIZE(prb_desc_ring)=48
                      OFFSET(prb_desc_ring.count_bits)=0
                      OFFSET(prb_desc_ring.descs)=8
                      OFFSET(prb_desc_ring.infos)=16
                      OFFSET(prb_desc_ring.head_id)=24
                      OFFSET(prb_desc_ring.tail_id)=32
                      SIZE(prb_desc)=24
                      OFFSET(prb_desc.state_var)=0
                      OFFSET(prb_desc.text_blk_lpos)=8
                      SIZE(prb_data_blk_lpos)=16
                      OFFSET(prb_data_blk_lpos.begin)=0
                      OFFSET(prb_data_blk_lpos.next)=8
                      SIZE(printk_info)=88
                      OFFSET(printk_info.seq)=0
                      OFFSET(printk_info.ts_nsec)=8
                      OFFSET(printk_info.text_len)=16
                      OFFSET(printk_info.caller_id)=20
                      OFFSET(printk_info.dev_info)=24
                      SIZE(dev_printk_info)=64
                      OFFSET(dev_printk_info.subsystem)=0
                      LENGTH(printk_info_subsystem)=16
                      OFFSET(dev_printk_info.device)=16
                      LENGTH(printk_info_device)=48
                      SIZE(prb_data_ring)=32
                      OFFSET(prb_data_ring.size_bits)=0
                      OFFSET(prb_data_ring.data)=8
                      OFFSET(prb_data_ring.head_lpos)=16
                      OFFSET(prb_data_ring.tail_lpos)=24
                      SIZE(atomic_long_t)=8
                      OFFSET(atomic_long_t.counter)=0
                      SIZE(latched_seq)=24
                      OFFSET(latched_seq.val)=8
                      LENGTH(free_area.free_list)=6
                      NUMBER(NR_FREE_PAGES)=0
                      NUMBER(PG_lru)=5
                      NUMBER(PG_private)=15
                      NUMBER(PG_swapcache)=12
                      NUMBER(PG_swapbacked)=19
                      NUMBER(PG_slab)=11
                      NUMBER(PG_head_mask)=64
                      NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
                      NUMBER(PG_hugetlb)=8
                      NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
                      SYMBOL(kallsyms_names)=ffffffff82a891c0
                      SYMBOL(kallsyms_num_syms)=ffffffff82a891b8
                      SYMBOL(kallsyms_token_table)=ffffffff82c222d0
                      SYMBOL(kallsyms_token_index)=ffffffff82c22680
                      SYMBOL(kallsyms_offsets)=ffffffff82c22880
                      SYMBOL(kallsyms_relative_base)=ffffffff82cc2e90
                      NUMBER(phys_base)=0
                      SYMBOL(init_top_pgt)=ffffffff8302c000
                      NUMBER(pgtable_l5_enabled)=0
                      KERNELOFFSET=0
                      NUMBER(KERNEL_IMAGE_SIZE)=1073741824
                      NUMBER(sme_mask)=0
                      CRASHTIME=1709570155
         offset_note: 4200 (0x1068)
           size_note: 7500 (0x1d4c)
           notes_buf: 5893c2d5b440
  num_vmcoredd_notes: 0
  num_prstatus_notes: 12
            notes[0]: 5893c2d5b440 (NT_PRSTATUS)
                      si.signo: 0  si.code: 0  si.errno: 0
                      cursig: 0  sigpend: 0  sighold: 0
                      pid: 0  ppid: 0  pgrp: 0  sid:0
                      utime: 0.000000  stime: 0.000000
                      cutime: 0.000000  cstime: 0.000000
                      ORIG_RAX: ffffffffffffffff  fpvalid: 0
                           R15: 0000000000000000  R14: ffffffff83774660
                           R13: ffffffff83774678  R12: ffffffff83774320
                           RBP: 0000000000000008  RBX: ffffe8ffffa1f510
                           R11: 0000000000000000  R10: 0000000000000000
                            R9: 0000000000000000   R8: 0000000000000000
                           RAX: 0000000000000060  RCX: 0000000000000001
                           RDX: 0000000000000000  RSI: 0000000000000060
                           RDI: 0000000000000008  RIP: ffffffff822f6fa7
                        RFLAGS: 0000000000000046  RSP: ffffffff83003e38
                       FS_BASE: 0000000000000000
                       GS_BASE: 0000000000000000
                            CS: 0010  SS: 0018  DS: 0000
                            ES: 0000  FS: 0000  GS: 0000
            notes[1]: 5893c2d5b5a4 (NT_PRSTATUS)
                      si.signo: 0  si.code: 0  si.errno: 0
                      cursig: 0  sigpend: 0  sighold: 0
                      pid: 0  ppid: 0  pgrp: 0  sid:0
                      utime: 0.000000  stime: 0.000000
                      cutime: 0.000000  cstime: 0.000000
                      ORIG_RAX: ffffffffffffffff  fpvalid: 0
                           R15: 0000000000000000  R14: ffffffff83774660
                           R13: ffffffff83774678  R12: ffffffff83774320
                           RBP: 0000000000000008  RBX: ffffe8ffffa5f510
                           R11: 0000000000000000  R10: 0000000000000000
                            R9: 0000000000000000   R8: 0000000000000000
                           RAX: 0000000000000060  RCX: 0000000000000001
                           RDX: 0000000000000000  RSI: 0000000000000060
                           RDI: 0000000000000008  RIP: ffffffff822f6fa7
                        RFLAGS: 0000000000000046  RSP: ffffc900000a3e90
                       FS_BASE: 0000000000000000
                       GS_BASE: 0000000000000000
                            CS: 0010  SS: 0018  DS: 0000
                            ES: 0000  FS: 0000  GS: 0000
            notes[2]: 5893c2d5b708 (NT_PRSTATUS)
                      si.signo: 0  si.code: 0  si.errno: 0
                      cursig: 0  sigpend: 0  sighold: 0
                      pid: 0  ppid: 0  pgrp: 0  sid:0
                      utime: 0.000000  stime: 0.000000
                      cutime: 0.000000  cstime: 0.000000
                      ORIG_RAX: ffffffffffffffff  fpvalid: 0
                           R15: 0000000000000000  R14: ffffffff83774660
                           R13: ffffffff83774678  R12: ffffffff83774320
                           RBP: 0000000000000008  RBX: ffffe8ffffa9f510
                           R11: 0000000000000000  R10: 0000000000000000
                            R9: 0000000000000000   R8: 0000000000000000
                           RAX: 0000000000000060  RCX: 0000000000000001
                           RDX: 0000000000000000  RSI: 0000000000000060
                           RDI: 0000000000000008  RIP: ffffffff822f6fa7
                        RFLAGS: 0000000000000046  RSP: ffffc900000abe90
                       FS_BASE: 0000000000000000
                       GS_BASE: 0000000000000000
                            CS: 0010  SS: 0018  DS: 0000
                            ES: 0000  FS: 0000  GS: 0000
            notes[3]: 5893c2d5b86c (NT_PRSTATUS)
                      si.signo: 0  si.code: 0  si.errno: 0
                      cursig: 0  sigpend: 0  sighold: 0
                      pid: 0  ppid: 0  pgrp: 0  sid:0
                      utime: 0.000000  stime: 0.000000
                      cutime: 0.000000  cstime: 0.000000
                      ORIG_RAX: ffffffffffffffff  fpvalid: 0
                           R15: 0000000000000000  R14: ffffffff83774660
                           R13: ffffffff83774678  R12: ffffffff83774320
                           RBP: 0000000000000008  RBX: ffffe8ffffadf510
                           R11: 0000000000000000  R10: 0000000000000000
                            R9: 0000000000000000   R8: 0000000000000000
                           RAX: 0000000000000060  RCX: 0000000000000001
                           RDX: 0000000000000000  RSI: 0000000000000060
                           RDI: 0000000000000008  RIP: ffffffff822f6fa7
                        RFLAGS: 0000000000000046  RSP: ffffc900000b3e90
                       FS_BASE: 0000000000000000
                       GS_BASE: 0000000000000000
                            CS: 0010  SS: 0018  DS: 0000
                            ES: 0000  FS: 0000  GS: 0000
            notes[4]: 5893c2d5b9d0 (NT_PRSTATUS)
                      si.signo: 0  si.code: 0  si.errno: 0
                      cursig: 0  sigpend: 0  sighold: 0
                      pid: 0  ppid: 0  pgrp: 0  sid:0
                      utime: 0.000000  stime: 0.000000
                      cutime: 0.000000  cstime: 0.000000
                      ORIG_RAX: ffffffffffffffff  fpvalid: 0
                           R15: 0000000000000000  R14: ffffffff83774660
                           R13: ffffffff83774678  R12: ffffffff83774320
                           RBP: 0000000000000008  RBX: ffffe8ffffb1f510
                           R11: 0000000000000000  R10: 0000000000000000
                            R9: 0000000000000000   R8: 0000000000000000
                           RAX: 0000000000000060  RCX: 0000000000000001
                           RDX: 0000000000000000  RSI: 0000000000000060
                           RDI: 0000000000000008  RIP: ffffffff822f6fa7
                        RFLAGS: 0000000000000046  RSP: ffffc900000bbe90
                       FS_BASE: 0000000000000000
                       GS_BASE: 0000000000000000
                            CS: 0010  SS: 0018  DS: 0000
                            ES: 0000  FS: 0000  GS: 0000
            notes[5]: 5893c2d5bb34 (NT_PRSTATUS)
                      si.signo: 0  si.code: 0  si.errno: 0
                      cursig: 0  sigpend: 0  sighold: 0
                      pid: 0  ppid: 0  pgrp: 0  sid:0
                      utime: 0.000000  stime: 0.000000
                      cutime: 0.000000  cstime: 0.000000
                      ORIG_RAX: ffffffffffffffff  fpvalid: 0
                           R15: 0000000000000000  R14: ffffffff83774660
                           R13: ffffffff83774678  R12: ffffffff83774320
                           RBP: 0000000000000008  RBX: ffffe8ffffb5f510
                           R11: 0000000000000000  R10: 0000000000000000
                            R9: 0000000000000000   R8: 0000000000000000
                           RAX: 0000000000000060  RCX: 0000000000000001
                           RDX: 0000000000000000  RSI: 0000000000000060
                           RDI: 0000000000000008  RIP: ffffffff822f6fa7
                        RFLAGS: 0000000000000046  RSP: ffffc900000c3e90
                       FS_BASE: 0000000000000000
                       GS_BASE: 0000000000000000
                            CS: 0010  SS: 0018  DS: 0000
                            ES: 0000  FS: 0000  GS: 0000
            notes[6]: 5893c2d5bc98 (NT_PRSTATUS)
                      si.signo: 0  si.code: 0  si.errno: 0
                      cursig: 0  sigpend: 0  sighold: 0
                      pid: 0  ppid: 0  pgrp: 0  sid:0
                      utime: 0.000000  stime: 0.000000
                      cutime: 0.000000  cstime: 0.000000
                      ORIG_RAX: ffffffffffffffff  fpvalid: 0
                           R15: 0000000000000000  R14: ffffffff83774660
                           R13: ffffffff83774678  R12: ffffffff83774320
                           RBP: 0000000000000008  RBX: ffffe8ffffb9f510
                           R11: 0000000000000000  R10: 0000000000000000
                            R9: 0000000000000000   R8: 0000000000000000
                           RAX: 0000000000000060  RCX: 0000000000000001
                           RDX: 0000000000000000  RSI: 0000000000000060
                           RDI: 0000000000000008  RIP: ffffffff822f6fa7
                        RFLAGS: 0000000000000046  RSP: ffffc900000cbe90
                       FS_BASE: 0000000000000000
                       GS_BASE: 0000000000000000
                            CS: 0010  SS: 0018  DS: 0000
                            ES: 0000  FS: 0000  GS: 0000
            notes[7]: 5893c2d5bdfc (NT_PRSTATUS)
                      si.signo: 0  si.code: 0  si.errno: 0
                      cursig: 0  sigpend: 0  sighold: 0
                      pid: 0  ppid: 0  pgrp: 0  sid:0
                      utime: 0.000000  stime: 0.000000
                      cutime: 0.000000  cstime: 0.000000
                      ORIG_RAX: ffffffffffffffff  fpvalid: 0
                           R15: 0000000000000000  R14: ffffffff83774660
                           R13: ffffffff83774678  R12: ffffffff83774320
                           RBP: 0000000000000008  RBX: ffffe8ffffbdf510
                           R11: 0000000000000000  R10: 0000000000000000
                            R9: 0000000000000000   R8: 0000000000000000
                           RAX: 0000000000000060  RCX: 0000000000000001
                           RDX: 0000000000000000  RSI: 0000000000000060
                           RDI: 0000000000000008  RIP: ffffffff822f6fa7
                        RFLAGS: 0000000000000046  RSP: ffffc900000d3e90
                       FS_BASE: 0000000000000000
                       GS_BASE: 0000000000000000
                            CS: 0010  SS: 0018  DS: 0000
                            ES: 0000  FS: 0000  GS: 0000
            notes[8]: 5893c2d5bf60 (NT_PRSTATUS)
                      si.signo: 0  si.code: 0  si.errno: 0
                      cursig: 0  sigpend: 0  sighold: 0
                      pid: 0  ppid: 0  pgrp: 0  sid:0
                      utime: 0.000000  stime: 0.000000
                      cutime: 0.000000  cstime: 0.000000
                      ORIG_RAX: ffffffffffffffff  fpvalid: 0
                           R15: 0000000000000000  R14: ffffffff83774660
                           R13: ffffffff83774678  R12: ffffffff83774320
                           RBP: 0000000000000008  RBX: ffffe8ffffc1f510
                           R11: 0000000000000000  R10: 0000000000000000
                            R9: 0000000000000000   R8: 0000000000000000
                           RAX: 0000000000000060  RCX: 0000000000000001
                           RDX: 0000000000000000  RSI: 0000000000000060
                           RDI: 0000000000000008  RIP: ffffffff822f6fa7
                        RFLAGS: 0000000000000046  RSP: ffffc900000dbe90
                       FS_BASE: 0000000000000000
                       GS_BASE: 0000000000000000
                            CS: 0010  SS: 0018  DS: 0000
                            ES: 0000  FS: 0000  GS: 0000
            notes[9]: 5893c2d5c0c4 (NT_PRSTATUS)
                      si.signo: 0  si.code: 0  si.errno: 0
                      cursig: 0  sigpend: 0  sighold: 0
                      pid: 0  ppid: 0  pgrp: 0  sid:0
                      utime: 0.000000  stime: 0.000000
                      cutime: 0.000000  cstime: 0.000000
                      ORIG_RAX: ffffffffffffffff  fpvalid: 0
                           R15: 0000000000000000  R14: ffffffff83774660
                           R13: ffffffff83774678  R12: ffffffff83774320
                           RBP: 0000000000000008  RBX: ffffe8ffffc5f510
                           R11: 0000000000000000  R10: 0000000000000000
                            R9: 0000000000000000   R8: 0000000000000000
                           RAX: 0000000000000060  RCX: 0000000000000001
                           RDX: 0000000000000000  RSI: 0000000000000060
                           RDI: 0000000000000008  RIP: ffffffff822f6fa7
                        RFLAGS: 0000000000000046  RSP: ffffc900000e3e90
                       FS_BASE: 0000000000000000
                       GS_BASE: 0000000000000000
                            CS: 0010  SS: 0018  DS: 0000
                            ES: 0000  FS: 0000  GS: 0000
            notes[10]: 5893c2d5c228 (NT_PRSTATUS)
                      si.signo: 0  si.code: 0  si.errno: 0
                      cursig: 0  sigpend: 0  sighold: 0
                      pid: 0  ppid: 0  pgrp: 0  sid:0
                      utime: 0.000000  stime: 0.000000
                      cutime: 0.000000  cstime: 0.000000
                      ORIG_RAX: 0  fpvalid: 0
                           R15: ffff888141cf9a38  R14: 0000000000000063
                           R13: 0000000000000001  R12: ffffffff82d78a21
                           RBP: ffffc90000304ce0  RBX: ffffc90000304bb0
                           R11: 0000000000000000  R10: 0000000000000000
                            R9: 0000000000000000   R8: 0000000000000000
                           RAX: 0000000000000000  RCX: 0000000000000000
                           RDX: 0000000000000001  RSI: 0000000000000000
                           RDI: ffffc90000304c58  RIP: ffffffff811b9f8c
                        RFLAGS: 0000000000000046  RSP: ffffc90000304bb0
                       FS_BASE: 0000000000000000
                       GS_BASE: 0000000000000000
                            CS: 0010  SS: 0018  DS: 0000
                            ES: 0000  FS: 0000  GS: 0000
            notes[11]: 5893c2d5c38c (NT_PRSTATUS)
                      si.signo: 0  si.code: 0  si.errno: 0
                      cursig: 0  sigpend: 0  sighold: 0
                      pid: 0  ppid: 0  pgrp: 0  sid:0
                      utime: 0.000000  stime: 0.000000
                      cutime: 0.000000  cstime: 0.000000
                      ORIG_RAX: ffffffffffffffff  fpvalid: 0
                           R15: 0000000000000000  R14: ffffffff83774660
                           R13: ffffffff83774678  R12: ffffffff83774320
                           RBP: 0000000000000008  RBX: ffffe8ffffcdf510
                           R11: 0000000000000000  R10: 0000000000000000
                            R9: 0000000000000000   R8: 0000000000000000
                           RAX: 0000000000000060  RCX: 0000000000000001
                           RDX: 0000000000000000  RSI: 0000000000000060
                           RDI: 0000000000000008  RIP: ffffffff822f6fa7
                        RFLAGS: 0000000000000046  RSP: ffffc900000f3e90
                       FS_BASE: 0000000000000000
                       GS_BASE: 0000000000000000
                            CS: 0010  SS: 0018  DS: 0000
                            ES: 0000  FS: 0000  GS: 0000
       snapshot_task: 0 
      num_qemu_notes: 0
        NOTE offsets: 1068 (NT_PRSTATUS)
                      11cc (NT_PRSTATUS)
                      1330 (NT_PRSTATUS)
                      1494 (NT_PRSTATUS)
                      15f8 (NT_PRSTATUS)
                      175c (NT_PRSTATUS)
                      18c0 (NT_PRSTATUS)
                      1a24 (NT_PRSTATUS)
                      1b88 (NT_PRSTATUS)
                      1cec (NT_PRSTATUS)
                      1e50 (NT_PRSTATUS)
                      1fb4 (NT_PRSTATUS)
    offset_eraseinfo: 0 (0x0)
      size_eraseinfo: 0 (0x0)
        start_pfn_64: (unused)
          end_pfn_64: (unused)
        max_mapnr_64: 17100800 (0x104f000)

       data_offset: 417000
        block_size: 4096
       block_shift: 12
            bitmap: 75d08f803000
        bitmap_len: 4276224
         max_mapnr: 17100800 (0x104f000)
   dumpable_bitmap: 75d08fe00010
              byte: 0
               bit: 0
   compressed_page: 5893c2da9450
         curbufptr: 0

 page_cache_hdr[0]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2d99440
        pg_hit_count: 0
 page_cache_hdr[1]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2d9a440
        pg_hit_count: 0
 page_cache_hdr[2]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2d9b440
        pg_hit_count: 0
 page_cache_hdr[3]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2d9c440
        pg_hit_count: 0
 page_cache_hdr[4]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2d9d440
        pg_hit_count: 0
 page_cache_hdr[5]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2d9e440
        pg_hit_count: 0
 page_cache_hdr[6]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2d9f440
        pg_hit_count: 0
 page_cache_hdr[7]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2da0440
        pg_hit_count: 0
 page_cache_hdr[8]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2da1440
        pg_hit_count: 0
 page_cache_hdr[9]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2da2440
        pg_hit_count: 0
page_cache_hdr[10]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2da3440
        pg_hit_count: 0
page_cache_hdr[11]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2da4440
        pg_hit_count: 0
page_cache_hdr[12]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2da5440
        pg_hit_count: 0
page_cache_hdr[13]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2da6440
        pg_hit_count: 0
page_cache_hdr[14]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2da7440
        pg_hit_count: 0
page_cache_hdr[15]:
            pg_flags: 0 ()
             pg_addr: 0
           pg_bufptr: 5893c2da8440
        pg_hit_count: 0

    page_cache_buf: 5893c2d99440
       evict_index: 0
         evictions: 0
          accesses: 0
      cached_reads: 0 
       valid_pages: 5893c2d5d1a0
 total_valid_pages: 16348585
readmem: read_diskdump() 
VMCOREINFO: NUMBER(phys_base): 0 -> 0
     program_name: crash
     program_path: crash
  program_version: 8.0.4++
      gdb_version: 10.2
      program_pid: 88994
           prompt: "crash> "
            flags: 1800000200c22 
          (TTY|HASH|SCROLL|CMDLINE_IFILE|READLINE|DISKDUMP|DATADEBUG)           
           flags2: 100402 (ELF_NOTES|VMCOREINFO|REDZONE)
         namelist: vmlinux-6.7.6-gentoo-x86_64-2024-03-04-15_14_26
         dumpfile: crashdump-2024-03-04-17_36_29
      live_memsrc: (null)
       system_map: (null)
   namelist_debug: (null)
   debuginfo_file: (null)
    namelist_orig: (null)
namelist_dbg_orig: (null)
  kvmdump_mapfile: (null)
    memory_module: (null)
    memory_device: /dev/crash
     machine_type: X86_64
     editing_mode: vi
              nfd: -1
              mfd: -1
              kfd: -1
              dfd: -1
            confd: -2
             home: /root
     command_line: 
        orig_line: 
        eoc_index: 0
         readline: 0
           my_tty: pts/5
            debug: 1
       debug_save: 0
          console: (null)
 redhat_debug_loc: /usr/lib/debug/lib/modules
        pipefd[2]: 0,0
           nullfp: 5893c2d5b240
          stdpipe: 0
             pipe: 0
            ifile: 0
            ofile: 0
       ifile_pipe: 0
      ifile_ofile: 0
       args_ifile: 0
       input_file: autoexec
ifile_in_progress: 0 ()
     ifile_offset: 0
runtime_ifile_cmd: (unused)
   scroll_command: SCROLL_NONE
         redirect: 0 
      stdpipe_pid: 0
         pipe_pid: 0
   pipe_shell_pid: 0
     pipe_command: 
          symfile: 0 
         symfile2: 0 
          tmpfile: 0
         saved_fp: 0
           tmp_fp: 0
         tmpfile2: 0
        cmd_table: linux_command_table
           curcmd: crash
          lastcmd: (null)
      cur_gdb_cmd: 0  
     last_gdb_cmd: 0  
          cur_req: 0
        cmdgencur: 0
     curcmd_flags: 0 ()
   curcmd_private: 0
      cmd_cleanup: 0
  cmd_cleanup_arg: 0
       sigint_cnt: 0
        sigaction: 5893c297bca8
    gdb_sigaction: 5893c297bd40
    main_loop_env: 5893c297bdd8
 foreach_loop_env: 5893c297bea0
     termios_orig: 5893c297bf68
      termios_raw: 5893c297bfa4
            ncmds: 0
          cmdlist: 0
        cmdlistsz: 0
     output_radix: 0 (???)
           server: (null)
       server_pid: 0
             port: 0
           sockfd: 0
    server_memsrc: (null)
  server_namelist: (null)
             rmfd: 0
             rkfd: 0
       rcvbufsize: 0
          readmem: read_diskdump()
         writemem: write_diskdump()
  dumpfile memory: 0
           curext: 0
             sbrk: 5893c2dca000
          cleanup: (null)
            scope: 0 (not set)
   nr_hash_queues: 32768
  read_vmcoreinfo: 5893c2050bb0
         error_fp: 75d0905f75c0
       error_path: default
gdb vmlinux-6.7.6-gentoo-x86_64-2024-03-04-15_14_26 
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
State of pagination is off.

crash: CONFIG_NR_CPUS: 64
crash: CONFIG_HZ: 300
crash: # CONFIG_DEBUG_INFO_REDUCED is not set
cpu_possible_mask: cpus: 0 1 2 3 4 5 6 7 8 9 10 11 
 cpu_present_mask: cpus: 0 1 2 3 4 5 6 7 8 9 10 11 
  cpu_online_mask: cpus: 0 1 2 3 4 5 6 7 8 9 10 11 
  cpu_active_mask: cpus: 0 1 2 3 4 5 6 7 8 9 10 11 
xtime timespec.tv_sec: 65e5f86b: Mon Mar  4 17:35:55 CET 2024
utsname:
     sysname: Linux
    nodename: i87k
     release: 6.7.6-gentoo-x86_64
     version: #1 SMP Mon Mar  4 15:10:10 CET 2024
     machine: x86_64
  domainname: (none)
base kernel version: 6.7.6
verify_namelist:
dumpfile /proc/version:
Linux version 6.7.6-gentoo-x86_64 (_kernelbuilder@i87k) (gcc (Gentoo 13.2.1_p20240210 p13) 13.2.1 20240210, GNU ld (Gentoo 2.42 p3) 2.42.0) #1 SMP Mon Mar  4 15:10:10 CET 2024
vmlinux-6.7.6-gentoo-x86_64-2024-03-04-15_14_26:
Linux version 6.7.6-gentoo-x86_64 (_kernelbuilder@i87k) (gcc (Gentoo 13.2.1_p20240210 p13) 13.2.1 20240210, GNU ld (Gentoo 2.42 p3) 2.42.0) #1 SMP Mon Mar  4 15:10:10 CET 2024

crash: get_cpus_present: present: 12
hypervisor: (undetermined)
crash: per_cpu_symbol_search(hardirq_stack_ptr): NULL
crash: per_cpu_symbol_search(per_cpu__irq_stack_union): NULL
crash: per_cpu_symbol_search(per_cpu__cpu_number): NULL
crash: per_cpu_symbol_search(per_cpu__current_task): NULL
crash: get_cpus_present: present: 12
crash: per_cpu_symbol_search(per_cpu__init_tss): NULL
crash: per_cpu_symbol_search(per_cpu__cpu_tss): NULL
crash: per_cpu_symbol_search(per_cpu__orig_ist): NULL
exception IST:
  3: asm_exc_debug
  2: asm_exc_nmi
  1: asm_exc_double_fault
  4: asm_exc_machine_check
exception stacks:
  [0]: DOUBLEFAULT
  [1]: NMI
  [2]: DEBUG
  [3]: MCE
  [4]: (unknown)
  [5]: (unknown)
  [6]: (unknown)
kernel NR_CPUS: 64 
PAGESIZE=4096
mem_section_size = 32768
NR_SECTION_ROOTS = 4096
NR_MEM_SECTIONS = 524288
SECTIONS_PER_ROOT = 128
SECTION_ROOT_MASK = 0x7f
PAGES_PER_SECTION = 32768
node_online_map: [1] -> nodes online: 1
node_table[0]: 
             id: 0
          pgdat: ffffffff837ec1c0
           size: 17100799
        present: 16741801
        mem_map: ffffea0000000040
    start_paddr: 1000
    start_mapnr: 1
NOTE: page_hash_table does not exist in this kernel

please wait... (gathering kmem slab cache data)
kmem_cache_downsize: 208 to 208

                                                
pageflags from enum: 
  00000001 locked
  00000002 writeback
  00000004 referenced
  00000008 uptodate
  00000010 dirty
  00000020 lru
  00000040 head
  00000080 waiters
  00000100 active
  00000200 workingset
  00000400 error
  00000800 slab
  00001000 owner_priv_1
  00002000 arch_1
  00004000 reserved
  00008000 private
  00010000 private_2
  00020000 mappedtodisk
  00040000 reclaim
  00080000 swapbacked
  00100000 unevictable
  00200000 mlocked
  00400000 uncached
  00800000 young
  01000000 idle
NOTE: unwind_table structure has changed, or does not exist in this kernel
init_unwind_table: DWARF_UNWIND_EH_FRAME
crash: struct module_memory detected.

please wait... (gathering module symbol data)
crash: invalid kernel virtual address: 0  type: "module symbols"

(if I don't specify nokaslr that error looks like:
crash: invalid kernel virtual address: 19 type: "module gpl symbols", although, this was same version kernel it was a different compilation of it, maybe that's why 'gpl' is extra, unsure. I haven't tested without nokaslr in cmdline in this recompiled kernel that this whole issue is about)

that 'crash' I used was compiled on Gentoo from latest commit 5977936 and had a little patch(why the extra output):

show 'crash' args and stuff at startup, helps knowing what it sees.

Index: /var/tmp/portage/dev-util/crash-9999/work/crash-9999/main.c
===================================================================
--- .orig/var/tmp/portage/dev-util/crash-9999/work/crash-9999/main.c
+++ crash-9999/main.c
@@ -713,6 +713,7 @@ main(int argc, char **argv)
 	machdep_init(PRE_GDB);
         datatype_init();
 
+        dump_program_context();
 	/*
 	 *  gdb_main_loop() modifies "command_loop_hook" to point to the 
          *  main_loop() function below, and then calls gdb's main() function.

I'm not sure if --mod dir is required, but the modules are here:

click to expand
/lib/modules/6.7.6-gentoo-x86_64/:
total 560
drwxr-xr-x 1 root root     68 Mar  4 15:11 kernel
drwxr-xr-x 1 root root   2786 Mar  4 15:14 ..
drwxr-xr-x 1 root root     72 Mar  4 15:15 misc
drwxr-xr-x 1 root root    460 Mar  4 15:15 .
-rw-r--r-- 1 root root   4876 Mar  4 15:11 modules.order
-rw-r--r-- 1 root root 236449 Mar  4 15:11 modules.builtin.modinfo
-rw-r--r-- 1 root root  26143 Mar  4 15:11 modules.builtin
lrwxrwxrwx 1 root root     21 Mar  4 15:11 build -> /tmp/genkernel.outdir
-rw-r--r-- 1 root root  61961 Mar  4 15:15 modules.symbols
-rw-r--r-- 1 root root    269 Mar  4 15:15 modules.softdep
-rw-r--r-- 1 root root  15400 Mar  4 15:15 modules.dep.bin
-rw-r--r-- 1 root root   9759 Mar  4 15:15 modules.dep
-rw-r--r-- 1 root root  31304 Mar  4 15:15 modules.alias.bin
-rw-r--r-- 1 root root  27328 Mar  4 15:15 modules.alias
-rw-r--r-- 1 root root  69065 Mar  4 15:15 modules.symbols.bin
-rw-r--r-- 1 root root  30997 Mar  4 15:15 modules.builtin.bin
-rw-r--r-- 1 root root  25146 Mar  4 15:15 modules.builtin.alias.bin
-rw-r--r-- 1 root root    126 Mar  4 15:15 modules.devname

autoexec file contents:

bt
quit

To get rid of the error, I've found that I can add --no_modules but this is just a mitigation, doesn't solve the problem, and it shows that the issue isn't just for the modules, as bt shows another similar error about something else:

# CRASHPAGER=/bin/cat crash -i autoexec --CRASHPAGER --no_modules --mod /lib/modules/6.7.6-gentoo-x86_64/ vmlinux-6.7.6-gentoo-x86_64-2024-03-04-15_14_26 crashdump-2024-03-04-17_36_29
click me to expand

crash 8.0.4++
Copyright (C) 2002-2022  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
     program_name: crash
     program_path: crash
  program_version: 8.0.4++
      gdb_version: 10.2
      program_pid: 87323
           prompt: "crash> "
            flags: 1800000200c22 
          (TTY|HASH|SCROLL|CMDLINE_IFILE|READLINE|DISKDUMP|DATADEBUG)           
           flags2: 100402 (ELF_NOTES|VMCOREINFO|REDZONE)
         namelist: vmlinux-6.7.6-gentoo-x86_64-2024-03-04-15_14_26
         dumpfile: crashdump-2024-03-04-17_36_29
      live_memsrc: (null)
       system_map: (null)
   namelist_debug: (null)
   debuginfo_file: (null)
    namelist_orig: (null)
namelist_dbg_orig: (null)
  kvmdump_mapfile: (null)
    memory_module: (null)
    memory_device: /dev/crash
     machine_type: X86_64
     editing_mode: vi
              nfd: -1
              mfd: -1
              kfd: -1
              dfd: -1
            confd: -2
             home: /root
     command_line: 
        orig_line: 
        eoc_index: 0
         readline: 0
           my_tty: pts/5
            debug: 0
       debug_save: 0
          console: (null)
 redhat_debug_loc: /usr/lib/debug/lib/modules
        pipefd[2]: 0,0
           nullfp: 60fa3203f240
          stdpipe: 0
             pipe: 0
            ifile: 0
            ofile: 0
       ifile_pipe: 0
      ifile_ofile: 0
       args_ifile: 0
       input_file: autoexec
ifile_in_progress: 0 ()
     ifile_offset: 0
runtime_ifile_cmd: (unused)
   scroll_command: SCROLL_NONE
         redirect: 0 
      stdpipe_pid: 0
         pipe_pid: 0
   pipe_shell_pid: 0
     pipe_command: 
          symfile: 0 
         symfile2: 0 
          tmpfile: 0
         saved_fp: 0
           tmp_fp: 0
         tmpfile2: 0
        cmd_table: linux_command_table
           curcmd: crash
          lastcmd: (null)
      cur_gdb_cmd: 0  
     last_gdb_cmd: 0  
          cur_req: 0
        cmdgencur: 0
     curcmd_flags: 0 ()
   curcmd_private: 0
      cmd_cleanup: 0
  cmd_cleanup_arg: 0
       sigint_cnt: 0
        sigaction: 60fa319eaca8
    gdb_sigaction: 60fa319ead40
    main_loop_env: 60fa319eadd8
 foreach_loop_env: 60fa319eaea0
     termios_orig: 60fa319eaf68
      termios_raw: 60fa319eafa4
            ncmds: 0
          cmdlist: 0
        cmdlistsz: 0
     output_radix: 0 (???)
           server: (null)
       server_pid: 0
             port: 0
           sockfd: 0
    server_memsrc: (null)
  server_namelist: (null)
             rmfd: 0
             rkfd: 0
       rcvbufsize: 0
          readmem: read_diskdump()
         writemem: write_diskdump()
  dumpfile memory: 0
           curext: 0
             sbrk: 60fa320ae000
          cleanup: (null)
            scope: 0 (not set)
   nr_hash_queues: 32768
  read_vmcoreinfo: 60fa310bfbb0
         error_fp: 7a1ade7f75c0
       error_path: default
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
State of pagination is off.


please wait... (gathering kmem slab cache data)
                                                
WARNING: no kernel module access


please wait... (gathering task table data)
                                           

please wait... (determining panic task)
                                        
      KERNEL: vmlinux-6.7.6-gentoo-x86_64-2024-03-04-15_14_26  [TAINTED]
    DUMPFILE: crashdump-2024-03-04-17_36_29  [PARTIAL DUMP]
        CPUS: 12
        DATE: Mon Mar  4 17:35:55 CET 2024
      UPTIME: 00:01:06
LOAD AVERAGE: 2.30, 0.85, 0.30
       TASKS: 12
    NODENAME: i87k
     RELEASE: 6.7.6-gentoo-x86_64
     VERSION: #1 SMP Mon Mar  4 15:10:10 CET 2024
     MACHINE: x86_64  (3700 Mhz)
      MEMORY: 63.9 GB
       PANIC: "Kernel panic - not syncing: sysrq triggered crash"
         PID: 0
     COMMAND: ""
        TASK: ffff888100d85f40  (1 of 12)  [THREAD_INFO: ffff888100d85f40]
         CPU: 10
       STATE: EXIT_DEAD|EXIT_ZOMBIE|TASK_NOLOAD (PANIC)

crash> bt
PID: 0        TASK: ffff888100d85f40  CPU: 10   COMMAND: ""
bt: invalid kernel virtual address: 100000001  type: "stack contents"
bt: read of stack at 100000001 failed
crash> quit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant