Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Link down/up cycling occasionally fails to re-establish connection #45

Open
MWP opened this issue Jan 9, 2023 · 1 comment
Open

Link down/up cycling occasionally fails to re-establish connection #45

MWP opened this issue Jan 9, 2023 · 1 comment

Comments

@MWP
Copy link

MWP commented Jan 9, 2023 •
edited
Loading

I'll try to keep this short...
I have two PC's each with a "Aquantia Corp. AQC107" PCI-E card and direct eth connection between the two (no switches, etc).
One PC runs Windows 10 x64 (desktop), the other Debian x64 bookworm/sid on the 6.0.0 debian kernel (server).
The server is up 24/7, the deskop is on/off every day.

driver: atlantic
version: 6.0.0-6-amd64
firmware-version: 3.1.58
expansion-rom-version:
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: yes

Occasionally (say 25% of the time), the Linux server end of the Ethernet link is not reestablished after booting the Windows PC.
The only way ive found to re-establish the connection (without re-booting the Linux server) is to run the following script:

brctl delif br0 enp2s0
sleep 5s
ip link set enp2s0 down
sleep 20s
rmmod atlantic
sleep 15s
modprobe atlantic
sleep 15s
ip link set enp2s0 up
sleep 10s
brctl addif br0 enp2s0
brctl show

When I do so, most of the time the eth link then becomes operational.
Sometimes when I do run the script, i get segfaults in the kernel log, and the link does not come up.
It's pretty random.

Below is a section of the kernel log over a few days showing the link going up and down successfully, a fail, the segfaults that sometimes occur when running that script, and then the link coming up ok again.

Please let me know if i can provide an information that would help diagnose the issue.

[Jan 6 07:46] atlantic 0000:02:00.0 enp2s0: atlantic: link change old 0 new 10000
[  +0.001392] br0: port 1(enp2s0) entered blocking state
[  +0.000007] br0: port 1(enp2s0) entered forwarding state
[Jan 6 23:44] atlantic 0000:02:00.0 enp2s0: atlantic: link change old 10000 new 0
[  +0.000356] br0: port 1(enp2s0) entered disabled state
[Jan 7 06:33] XFS (sdb1): Mounting V5 Filesystem
[  +0.247521] XFS (sdb1): Ending clean mount
[Jan 7 06:36] XFS (sdb1): Unmounting Filesystem
[Jan 7 11:49] device enp2s0 left promiscuous mode
[  +0.000090] br0: port 1(enp2s0) entered disabled state
[Jan 7 11:50] atlantic 0000:02:00.0 enp2s0: renamed from eth0
[  +6.152662] atlantic 0000:02:00.0 enp2s0: atlantic: link change old 0 new 10000
[  +0.001463] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0: link becomes ready
[ +18.838382] br0: port 1(enp2s0) entered blocking state
[  +0.000010] br0: port 1(enp2s0) entered disabled state
[  +0.000130] device enp2s0 entered promiscuous mode
[  +0.000118] br0: port 1(enp2s0) entered blocking state
[  +0.000005] br0: port 1(enp2s0) entered forwarding state
[Jan 8 01:11] atlantic 0000:02:00.0 enp2s0: atlantic: link change old 10000 new 0
[  +0.000222] br0: port 1(enp2s0) entered disabled state
[Jan 9 09:16] device enp2s0 left promiscuous mode
[  +0.000072] br0: port 1(enp2s0) entered disabled state
[Jan 9 09:17] atlantic 0000:02:00.0 enp2s0: renamed from eth0
[  +0.626151] ip: page allocation failure: order:6, mode:0x40dc0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),[email protected],mems_allowed=0
[  +0.000014] CPU: 3 PID: 2675215 Comm: ip Not tainted 6.0.0-6-amd64 #1  Debian 6.0.12-1
[  +0.000003] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z68 Extreme3 Gen3, BIOS P2.30 06/29/2012
[  +0.000002] Call Trace:
[  +0.000003]  <TASK>
[  +0.000003]  dump_stack_lvl+0x44/0x5c
[  +0.000007]  warn_alloc+0x151/0x170
[  +0.000004]  ? __alloc_pages_direct_compact+0x221/0x330
[  +0.000004]  __alloc_pages_slowpath.constprop.0+0xd4f/0xdb0
[  +0.000005]  __alloc_pages+0x305/0x330
[  +0.000003]  kmalloc_order+0x2a/0xa0
[  +0.000003]  kmalloc_order_trace+0x19/0x90
[  +0.000004]  aq_ring_alloc+0x22/0xb0 [atlantic]
[  +0.000011]  aq_vec_ring_alloc+0xea/0x1a0 [atlantic]
[  +0.000009]  aq_nic_init+0x114/0x1d0 [atlantic]
[  +0.000009]  aq_ndev_open+0x16/0x70 [atlantic]
[  +0.000007]  __dev_open+0xf4/0x1a0
[  +0.000003]  __dev_change_flags+0x1d1/0x240
[  +0.000003]  dev_change_flags+0x22/0x60
[  +0.000002]  do_setlink+0x802/0x12b0
[  +0.000004]  ? __nla_validate_parse+0x5f/0xbc0
[  +0.000004]  __rtnl_newlink+0x61d/0x9b0
[  +0.000004]  rtnl_newlink+0x43/0x70
[  +0.000003]  rtnetlink_rcv_msg+0x14e/0x3b0
[  +0.000002]  ? copy_user_generic_string+0x2c/0x50
[  +0.000003]  ? rtnl_calcit.isra.0+0x140/0x140
[  +0.000002]  netlink_rcv_skb+0x51/0x100
[  +0.000003]  netlink_unicast+0x242/0x390
[  +0.000003]  netlink_sendmsg+0x250/0x4c0
[  +0.000002]  sock_sendmsg+0x5f/0x70
[  +0.000003]  ____sys_sendmsg+0x267/0x2b0
[  +0.000003]  ? copy_msghdr_from_user+0x7d/0xc0
[  +0.000003]  ___sys_sendmsg+0x9a/0xe0
[  +0.000006]  __sys_sendmsg+0x76/0xc0
[  +0.000003]  do_syscall_64+0x3a/0xc0
[  +0.000003]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  +0.000004] RIP: 0033:0x7f1ef032a980
[  +0.000002] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 66 2e 0f 1f 84 00 00 00 00 00 90 80 3d 61 fc 0c 00 00 74 17 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 48 83 ec 28 89 54
[  +0.000002] RSP: 002b:00007ffcd4d14b88 EFLAGS: 00000202 ORIG_RAX: 000000000000002e
[  +0.000002] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f1ef032a980
[  +0.000002] RDX: 0000000000000000 RSI: 00007ffcd4d14bf0 RDI: 0000000000000003
[  +0.000001] RBP: 0000000000000003 R08: 0000000063bb4804 R09: 000055b3a271a2a0
[  +0.000001] R10: 00007ffcd4d14614 R11: 0000000000000202 R12: 0000000063bb4805
[  +0.000002] R13: 000055b3a23cd020 R14: 0000000000000001 R15: 0000000000000000
[  +0.000002]  </TASK>
[  +0.000001] Mem-Info:
[  +0.000002] active_anon:261668 inactive_anon:260792 isolated_anon:0
               active_file:1009753 inactive_file:1700165 isolated_file:0
               unevictable:12496 dirty:1628 writeback:0
               slab_reclaimable:457046 slab_unreclaimable:98768
               mapped:40468 shmem:3580 pagetables:3633 bounce:0
               kernel_misc_reclaimable:0
               free:105615 free_pcp:0 free_cma:0
[  +0.000004] Node 0 active_anon:1046672kB inactive_anon:1043168kB active_file:4039012kB inactive_file:6800660kB unevictable:49984kB isolated(anon):0kB isolated(file):0kB mapped:161872kB dirty:6512kB writeback:0kB shmem:14320kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1232896kB writeback_tmp:0kB kernel_stack:7444kB pagetables:14532kB all_unreclaimable? no
[  +0.000005] Node 0 DMA free:13312kB boost:0kB min:60kB low:72kB high:84kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  +0.000004] lowmem_reserve[]: 0 3404 15873 15873 15873
[  +0.000004] Node 0 DMA32 free:108820kB boost:0kB min:14480kB low:18100kB high:21720kB reserved_highatomic:0KB active_anon:32100kB inactive_anon:165536kB active_file:734420kB inactive_file:1723652kB unevictable:6476kB writepending:636kB present:3588416kB managed:3522880kB mlocked:6476kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  +0.000005] lowmem_reserve[]: 0 0 12469 12469 12469
[  +0.000003] Node 0 Normal free:300328kB boost:0kB min:53036kB low:66292kB high:79548kB reserved_highatomic:0KB active_anon:1014572kB inactive_anon:877904kB active_file:3304804kB inactive_file:5076852kB unevictable:43508kB writepending:5876kB present:13096960kB managed:12776472kB mlocked:43500kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  +0.000004] lowmem_reserve[]: 0 0 0 0 0
[  +0.000004] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 13312kB
[  +0.000010] Node 0 DMA32: 8954*4kB (UME) 3797*8kB (UME) 1108*16kB (UME) 683*32kB (UME) 48*64kB (UME) 4*128kB (ME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 109360kB
[  +0.000012] Node 0 Normal: 14237*4kB (UME) 16737*8kB (UME) 4860*16kB (UME) 980*32kB (UME) 12*64kB (UME) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 300860kB
[  +0.000012] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[  +0.000002] 2720017 total pagecache pages
[  +0.000002] 5572 pages in swap cache
[  +0.000001] Free swap  = 3838088kB
[  +0.000001] Total swap = 4095996kB
[  +0.000002] 4175340 pages RAM
[  +0.000001] 0 pages HighMem/MovableOnly
[  +0.000001] 96662 pages reserved
[  +0.000001] 0 pages hwpoisoned
[ +14.840069] ip: page allocation failure: order:6, mode:0x40dc0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),cpuset=user.slice,mems_allowed=0
[  +0.000012] CPU: 5 PID: 2675424 Comm: ip Not tainted 6.0.0-6-amd64 #1  Debian 6.0.12-1
[  +0.000003] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z68 Extreme3 Gen3, BIOS P2.30 06/29/2012
[  +0.000001] Call Trace:
[  +0.000003]  <TASK>
[  +0.000003]  dump_stack_lvl+0x44/0x5c
[  +0.000006]  warn_alloc+0x151/0x170
[  +0.000005]  ? __alloc_pages_direct_compact+0x221/0x330
[  +0.000003]  __alloc_pages_slowpath.constprop.0+0xd4f/0xdb0
[  +0.000004]  __alloc_pages+0x305/0x330
[  +0.000003]  kmalloc_order+0x2a/0xa0
[  +0.000003]  kmalloc_order_trace+0x19/0x90
[  +0.000004]  aq_ring_alloc+0x22/0xb0 [atlantic]
[  +0.000010]  aq_vec_ring_alloc+0xea/0x1a0 [atlantic]
[  +0.000008]  aq_nic_init+0x114/0x1d0 [atlantic]
[  +0.000008]  aq_ndev_open+0x16/0x70 [atlantic]
[  +0.000006]  __dev_open+0xf4/0x1a0
[  +0.000003]  __dev_change_flags+0x1d1/0x240
[  +0.000003]  dev_change_flags+0x22/0x60
[  +0.000002]  do_setlink+0x802/0x12b0
[  +0.000003]  ? __nla_validate_parse+0x5f/0xbc0
[  +0.000003]  __rtnl_newlink+0x61d/0x9b0
[  +0.000004]  rtnl_newlink+0x43/0x70
[  +0.000002]  rtnetlink_rcv_msg+0x14e/0x3b0
[  +0.000002]  ? copy_user_generic_string+0x2c/0x50
[  +0.000003]  ? rtnl_calcit.isra.0+0x140/0x140
[  +0.000002]  netlink_rcv_skb+0x51/0x100
[  +0.000003]  netlink_unicast+0x242/0x390
[  +0.000002]  netlink_sendmsg+0x250/0x4c0
[  +0.000003]  sock_sendmsg+0x5f/0x70
[  +0.000003]  ____sys_sendmsg+0x267/0x2b0
[  +0.000002]  ? copy_msghdr_from_user+0x7d/0xc0
[  +0.000003]  ___sys_sendmsg+0x9a/0xe0
[  +0.000005]  __sys_sendmsg+0x76/0xc0
[  +0.000003]  do_syscall_64+0x3a/0xc0
[  +0.000002]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  +0.000003] RIP: 0033:0x7fefd7eb2980
[  +0.000003] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 66 2e 0f 1f 84 00 00 00 00 00 90 80 3d 61 fc 0c 00 00 74 17 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 48 83 ec 28 89 54
[  +0.000001] RSP: 002b:00007ffdf12afaa8 EFLAGS: 00000202 ORIG_RAX: 000000000000002e
[  +0.000003] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fefd7eb2980
[  +0.000001] RDX: 0000000000000000 RSI: 00007ffdf12afb10 RDI: 0000000000000003
[  +0.000001] RBP: 0000000000000002 R08: 0000000063bb4813 R09: 000056170f2aa2a0
[  +0.000001] R10: 00007ffdf12af534 R11: 0000000000000202 R12: 0000000063bb4814
[  +0.000001] R13: 000056170e47b020 R14: 0000000000000001 R15: 0000000000000000
[  +0.000002]  </TASK>
[  +0.000001] Mem-Info:
[  +0.000002] active_anon:261706 inactive_anon:256596 isolated_anon:0
               active_file:1010126 inactive_file:1701493 isolated_file:0
               unevictable:12496 dirty:2098 writeback:1
               slab_reclaimable:457045 slab_unreclaimable:98575
               mapped:42662 shmem:3587 pagetables:3399 bounce:0
               kernel_misc_reclaimable:0
               free:108996 free_pcp:0 free_cma:0
[  +0.000004] Node 0 active_anon:1046824kB inactive_anon:1026384kB active_file:4040504kB inactive_file:6805972kB unevictable:49984kB isolated(anon):0kB isolated(file):0kB mapped:170648kB dirty:8392kB writeback:4kB shmem:14348kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1232896kB writeback_tmp:0kB kernel_stack:7104kB pagetables:13596kB all_unreclaimable? no
[  +0.000003] Node 0 DMA free:13312kB boost:0kB min:60kB low:72kB high:84kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  +0.000005] lowmem_reserve[]: 0 3404 15873 15873 15873
[  +0.000003] Node 0 DMA32 free:114984kB boost:0kB min:14480kB low:18100kB high:21720kB reserved_highatomic:0KB active_anon:32100kB inactive_anon:159360kB active_file:735196kB inactive_file:1723228kB unevictable:6476kB writepending:452kB present:3588416kB managed:3522880kB mlocked:6476kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  +0.000004] lowmem_reserve[]: 0 0 12469 12469 12469
[  +0.000003] Node 0 Normal free:307688kB boost:0kB min:53036kB low:66292kB high:79548kB reserved_highatomic:0KB active_anon:1014724kB inactive_anon:867024kB active_file:3305504kB inactive_file:5082744kB unevictable:43508kB writepending:7684kB present:13096960kB managed:12776472kB mlocked:43500kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  +0.000004] lowmem_reserve[]: 0 0 0 0 0
[  +0.000003] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 13312kB
[  +0.000009] Node 0 DMA32: 8943*4kB (UME) 4106*8kB (UME) 1182*16kB (UME) 726*32kB (UME) 60*64kB (UME) 6*128kB (ME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115372kB
[  +0.000011] Node 0 Normal: 15461*4kB (UME) 17014*8kB (UME) 4880*16kB (UME) 966*32kB (UME) 14*64kB (UME) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 307844kB
[  +0.000010] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[  +0.000001] 2721748 total pagecache pages
[  +0.000001] 5572 pages in swap cache
[  +0.000001] Free swap  = 3838088kB
[  +0.000000] Total swap = 4095996kB
[  +0.000001] 4175340 pages RAM
[  +0.000001] 0 pages HighMem/MovableOnly
[  +0.000000] 96662 pages reserved
[  +0.000001] 0 pages hwpoisoned
[ +10.003121] br0: port 1(enp2s0) entered blocking state
[  +0.000008] br0: port 1(enp2s0) entered disabled state
[  +0.000088] device enp2s0 entered promiscuous mode
[Jan 9 09:55] device enp2s0 left promiscuous mode
[  +0.000048] br0: port 1(enp2s0) entered disabled state
[Jan 9 09:56] atlantic 0000:02:00.0 enp2s0: renamed from eth0
[  +0.596643] ip: page allocation failure: order:6, mode:0x40dc0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),[email protected],mems_allowed=0
[  +0.000014] CPU: 1 PID: 2745610 Comm: ip Not tainted 6.0.0-6-amd64 #1  Debian 6.0.12-1
[  +0.000002] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z68 Extreme3 Gen3, BIOS P2.30 06/29/2012
[  +0.000002] Call Trace:
[  +0.000002]  <TASK>
[  +0.000003]  dump_stack_lvl+0x44/0x5c
[  +0.000006]  warn_alloc+0x151/0x170
[  +0.000005]  ? __alloc_pages_direct_compact+0x221/0x330
[  +0.000003]  __alloc_pages_slowpath.constprop.0+0xd4f/0xdb0
[  +0.000004]  __alloc_pages+0x305/0x330
[  +0.000003]  kmalloc_order+0x2a/0xa0
[  +0.000003]  kmalloc_order_trace+0x19/0x90
[  +0.000003]  aq_ring_alloc+0x22/0xb0 [atlantic]
[  +0.000012]  aq_vec_ring_alloc+0xea/0x1a0 [atlantic]
[  +0.000008]  aq_nic_init+0x114/0x1d0 [atlantic]
[  +0.000008]  aq_ndev_open+0x16/0x70 [atlantic]
[  +0.000006]  __dev_open+0xf4/0x1a0
[  +0.000003]  __dev_change_flags+0x1d1/0x240
[  +0.000002]  dev_change_flags+0x22/0x60
[  +0.000002]  do_setlink+0x802/0x12b0
[  +0.000004]  ? __nla_validate_parse+0x5f/0xbc0
[  +0.000004]  __rtnl_newlink+0x61d/0x9b0
[  +0.000003]  rtnl_newlink+0x43/0x70
[  +0.000002]  rtnetlink_rcv_msg+0x14e/0x3b0
[  +0.000002]  ? copy_user_generic_string+0x2c/0x50
[  +0.000003]  ? rtnl_calcit.isra.0+0x140/0x140
[  +0.000002]  netlink_rcv_skb+0x51/0x100
[  +0.000003]  netlink_unicast+0x242/0x390
[  +0.000002]  netlink_sendmsg+0x250/0x4c0
[  +0.000002]  sock_sendmsg+0x5f/0x70
[  +0.000003]  ____sys_sendmsg+0x267/0x2b0
[  +0.000003]  ? copy_msghdr_from_user+0x7d/0xc0
[  +0.000002]  ___sys_sendmsg+0x9a/0xe0
[  +0.000005]  __sys_sendmsg+0x76/0xc0
[  +0.000004]  do_syscall_64+0x3a/0xc0
[  +0.000002]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  +0.000003] RIP: 0033:0x7fe0a60af980
[  +0.000003] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 66 2e 0f 1f 84 00 00 00 00 00 90 80 3d 61 fc 0c 00 00 74 17 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 48 83 ec 28 89 54
[  +0.000001] RSP: 002b:00007ffc9cacd3e8 EFLAGS: 00000202 ORIG_RAX: 000000000000002e
[  +0.000003] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007fe0a60af980
[  +0.000001] RDX: 0000000000000000 RSI: 00007ffc9cacd450 RDI: 0000000000000003
[  +0.000001] RBP: 0000000000000003 R08: 0000000063bb512b R09: 000055ea3e2442a0
[  +0.000001] R10: 00007ffc9cacce74 R11: 0000000000000202 R12: 0000000063bb512c
[  +0.000001] R13: 000055ea3c64c020 R14: 0000000000000001 R15: 0000000000000000
[  +0.000002]  </TASK>
[  +0.000001] Mem-Info:
[  +0.000002] active_anon:261537 inactive_anon:263183 isolated_anon:0
               active_file:1024560 inactive_file:1655471 isolated_file:0
               unevictable:12495 dirty:2065 writeback:0
               slab_reclaimable:456177 slab_unreclaimable:99423
               mapped:42225 shmem:3637 pagetables:3727 bounce:0
               kernel_misc_reclaimable:0
               free:120782 free_pcp:194 free_cma:0
[  +0.000004] Node 0 active_anon:1046148kB inactive_anon:1052732kB active_file:4098240kB inactive_file:6621884kB unevictable:49980kB isolated(anon):0kB isolated(file):0kB mapped:168900kB dirty:8260kB writeback:0kB shmem:14548kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1232896kB writeback_tmp:0kB kernel_stack:7440kB pagetables:14908kB all_unreclaimable? no
[  +0.000004] Node 0 DMA free:13312kB boost:0kB min:60kB low:72kB high:84kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  +0.000004] lowmem_reserve[]: 0 3404 15873 15873 15873
[  +0.000003] Node 0 DMA32 free:166032kB boost:4096kB min:18576kB low:22196kB high:25816kB reserved_highatomic:0KB active_anon:25796kB inactive_anon:165464kB active_file:748612kB inactive_file:1668024kB unevictable:0kB writepending:576kB present:3588416kB managed:3522880kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  +0.000004] lowmem_reserve[]: 0 0 12469 12469 12469
[  +0.000003] Node 0 Normal free:304288kB boost:4096kB min:57132kB low:70388kB high:83644kB reserved_highatomic:0KB active_anon:1020352kB inactive_anon:887268kB active_file:3349308kB inactive_file:4953940kB unevictable:49980kB writepending:7684kB present:13096960kB managed:12776472kB mlocked:49972kB bounce:0kB free_pcp:328kB local_pcp:0kB free_cma:0kB
[  +0.000004] lowmem_reserve[]: 0 0 0 0 0
[  +0.000003] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 13312kB
[  +0.000010] Node 0 DMA32: 10406*4kB (UME) 4740*8kB (UME) 1879*16kB (UME) 1031*32kB (UME) 285*64kB (UME) 44*128kB (ME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 166472kB
[  +0.000010] Node 0 Normal: 11311*4kB (UME) 13227*8kB (UME) 6031*16kB (UME) 1281*32kB (UME) 184*64kB (UME) 30*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 304164kB
[  +0.000011] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[  +0.000001] 2690187 total pagecache pages
[  +0.000001] 5576 pages in swap cache
[  +0.000001] Free swap  = 3838088kB
[  +0.000000] Total swap = 4095996kB
[  +0.000001] 4175340 pages RAM
[  +0.000001] 0 pages HighMem/MovableOnly
[  +0.000000] 96662 pages reserved
[  +0.000001] 0 pages hwpoisoned
[ +14.882583] ip: page allocation failure: order:6, mode:0x40dc0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),cpuset=user.slice,mems_allowed=0
[  +0.000018] CPU: 7 PID: 2747081 Comm: ip Not tainted 6.0.0-6-amd64 #1  Debian 6.0.12-1
[  +0.000004] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z68 Extreme3 Gen3, BIOS P2.30 06/29/2012
[  +0.000003] Call Trace:
[  +0.000003]  <TASK>
[  +0.000003]  dump_stack_lvl+0x44/0x5c
[  +0.000008]  warn_alloc+0x151/0x170
[  +0.000005]  ? __alloc_pages_direct_compact+0x221/0x330
[  +0.000003]  __alloc_pages_slowpath.constprop.0+0xd4f/0xdb0
[  +0.000004]  __alloc_pages+0x305/0x330
[  +0.000004]  kmalloc_order+0x2a/0xa0
[  +0.000003]  kmalloc_order_trace+0x19/0x90
[  +0.000003]  aq_ring_alloc+0x22/0xb0 [atlantic]
[  +0.000018]  aq_vec_ring_alloc+0xea/0x1a0 [atlantic]
[  +0.000014]  aq_nic_init+0x114/0x1d0 [atlantic]
[  +0.000008]  aq_ndev_open+0x16/0x70 [atlantic]
[  +0.000007]  __dev_open+0xf4/0x1a0
[  +0.000003]  __dev_change_flags+0x1d1/0x240
[  +0.000003]  dev_change_flags+0x22/0x60
[  +0.000002]  do_setlink+0x802/0x12b0
[  +0.000003]  ? __nla_validate_parse+0x5f/0xbc0
[  +0.000004]  __rtnl_newlink+0x61d/0x9b0
[  +0.000004]  rtnl_newlink+0x43/0x70
[  +0.000002]  rtnetlink_rcv_msg+0x14e/0x3b0
[  +0.000002]  ? copy_user_generic_string+0x2c/0x50
[  +0.000003]  ? rtnl_calcit.isra.0+0x140/0x140
[  +0.000002]  netlink_rcv_skb+0x51/0x100
[  +0.000003]  netlink_unicast+0x242/0x390
[  +0.000002]  netlink_sendmsg+0x250/0x4c0
[  +0.000003]  sock_sendmsg+0x5f/0x70
[  +0.000003]  ____sys_sendmsg+0x267/0x2b0
[  +0.000002]  ? copy_msghdr_from_user+0x7d/0xc0
[  +0.000003]  ___sys_sendmsg+0x9a/0xe0
[  +0.000005]  __sys_sendmsg+0x76/0xc0
[  +0.000004]  do_syscall_64+0x3a/0xc0
[  +0.000002]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  +0.000004] RIP: 0033:0x7fc568eba980
[  +0.000003] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 66 2e 0f 1f 84 00 00 00 00 00 90 80 3d 61 fc 0c 00 00 74 17 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 48 83 ec 28 89 54
[  +0.000002] RSP: 002b:00007fff0016d388 EFLAGS: 00000202 ORIG_RAX: 000000000000002e
[  +0.000002] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fc568eba980
[  +0.000001] RDX: 0000000000000000 RSI: 00007fff0016d3f0 RDI: 0000000000000003
[  +0.000001] RBP: 0000000000000002 R08: 0000000063bb513a R09: 0000562b3303e2a0
[  +0.000002] R10: 00007fff0016ce14 R11: 0000000000000202 R12: 0000000063bb513b
[  +0.000001] R13: 0000562b3172b020 R14: 0000000000000001 R15: 0000000000000000
[  +0.000002]  </TASK>
[  +0.000001] Mem-Info:
[  +0.000002] active_anon:261305 inactive_anon:257049 isolated_anon:0
               active_file:1024437 inactive_file:1647173 isolated_file:0
               unevictable:12495 dirty:1081 writeback:0
               slab_reclaimable:456150 slab_unreclaimable:99102
               mapped:40820 shmem:3388 pagetables:3325 bounce:0
               kernel_misc_reclaimable:0
               free:137390 free_pcp:48 free_cma:0
[  +0.000004] Node 0 active_anon:1045220kB inactive_anon:1028196kB active_file:4097748kB inactive_file:6588692kB unevictable:49980kB isolated(anon):0kB isolated(file):0kB mapped:163280kB dirty:4324kB writeback:0kB shmem:13552kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1232896kB writeback_tmp:0kB kernel_stack:6816kB pagetables:13300kB all_unreclaimable? no
[  +0.000004] Node 0 DMA free:13312kB boost:0kB min:60kB low:72kB high:84kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  +0.000004] lowmem_reserve[]: 0 3404 15873 15873 15873
[  +0.000004] Node 0 DMA32 free:176200kB boost:0kB min:14480kB low:18100kB high:21720kB reserved_highatomic:0KB active_anon:25796kB inactive_anon:165448kB active_file:748216kB inactive_file:1658996kB unevictable:0kB writepending:256kB present:3588416kB managed:3522880kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  +0.000004] lowmem_reserve[]: 0 0 12469 12469 12469
[  +0.000003] Node 0 Normal free:360048kB boost:0kB min:53036kB low:66292kB high:79548kB reserved_highatomic:0KB active_anon:1019424kB inactive_anon:862748kB active_file:3349532kB inactive_file:4929612kB unevictable:49980kB writepending:4068kB present:13096960kB managed:12776472kB mlocked:49972kB bounce:0kB free_pcp:192kB local_pcp:0kB free_cma:0kB
[  +0.000005] lowmem_reserve[]: 0 0 0 0 0
[  +0.000003] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 13312kB
[  +0.000010] Node 0 DMA32: 10736*4kB (UME) 4803*8kB (UME) 1958*16kB (UME) 1071*32kB (UME) 358*64kB (UME) 50*128kB (UME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 176280kB
[  +0.000011] Node 0 Normal: 16734*4kB (UME) 16134*8kB (UME) 6423*16kB (UME) 1378*32kB (UME) 209*64kB (UME) 31*128kB (ME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 360216kB
[  +0.000015] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[  +0.000003] 2681616 total pagecache pages
[  +0.000001] 5577 pages in swap cache
[  +0.000001] Free swap  = 3838088kB
[  +0.000001] Total swap = 4095996kB
[  +0.000001] 4175340 pages RAM
[  +0.000002] 0 pages HighMem/MovableOnly
[  +0.000001] 96662 pages reserved
[  +0.000001] 0 pages hwpoisoned
[Jan 9 09:57] br0: port 1(enp2s0) entered blocking state
[  +0.000010] br0: port 1(enp2s0) entered disabled state
[  +0.000159] device enp2s0 entered promiscuous mode
[Jan 9 09:58] device enp2s0 left promiscuous mode
[  +0.000034] br0: port 1(enp2s0) entered disabled state
[ +40.788192] atlantic 0000:02:00.0 enp2s0: renamed from eth0
[Jan 9 09:59] br0: port 1(enp2s0) entered blocking state
[  +0.000009] br0: port 1(enp2s0) entered disabled state
[  +0.000154] device enp2s0 entered promiscuous mode
[ +31.499798] atlantic 0000:02:00.0 enp2s0: atlantic: link change old 0 new 10000
[  +0.001499] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0: link becomes ready
[  +0.000075] br0: port 1(enp2s0) entered blocking state
[  +0.000005] br0: port 1(enp2s0) entered forwarding state
@cail
Copy link
Member

cail commented Jan 9, 2023

Two issues here.

  1. linkup failure between the nodes. That could be cable or autoneg issues. Hard to say at the moment.
  2. Panic in linux. That seems to come from out of memory condition. Could be a mem leak.
    Could you try running in your system "ifconfig down enp2s0; ifconfig up enp2s0" in a loop? If memory leak, you should get the same issue immediately.
    Please also try installing the driver from this repository and check the same.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cail @MWP