Help with understanding IF Drops and Fragment issues

[nirhenn]

Hi all,

Need help with guide how to troubleshoot IF Drops and Fragment issues.
I am running with Opencontrail 3.0.2 w/DPDK. When running a load in L2 inside a single node (all VMs are located on the same compute node) I can notice that once I pass the 18 Gbps traffic I can see IF Drops and Fragment errors and connection to VM get staggering.

I use isolcpu=0-3 on the first NUMA 0 for vrouter-dpdk and hugepages 1G in grub. The strange thing is that i do see the first 4 cores in 100%, almost constantly, but in the process lines I see 10 to 11 routers-dpdk process, but only 4 are in 100 and the rest are 0. x only, is it normal? Also, those processes that are 0.x are not running on core 0-3, is it normal, even if the vrouter-dpdk.ini taskset is set to -c 0,1,2,3?

What are the steps to identify from where the issues source from?

Nir.

[nirhenn]

Some more info on env:

root@node-36:~# dropstats
GARP 0
ARP no where to go 0
Invalid ARPs 0

Invalid IF 4
Trap No IF 0
IF TX Discard 0
IF Drop 1651839472
IF RX Discard 0

Flow Unusable 4
Flow No Memory 0
Flow Table Full 0
Flow NAT no rflow 0
Flow Action Drop 421044
Flow Action Invalid 0
Flow Invalid Protocol 0
Flow Queue Limit Exceeded 3027684

Discards 8014
TTL Exceeded 0
Mcast Clone Fail 0
Cloned Original 16

Invalid NH 352
Invalid Label 0
Invalid Protocol 0
Rewrite Fail 0
Invalid Mcast Source 0

Push Fails 0
Pull Fails 0
Duplicated 0
Head Alloc Fails 0
Head Space Reserve Fails 0
PCOW fails 0
Invalid Packets 0

Misc 0
Nowhere to go 0
Checksum errors 0
No Fmd 0
Invalid VNID 0
Fragment errors 359772754
Invalid Source 0
Jumbo Mcast Pkt with DF Bit 0
ARP No Route 0
ARP Reply No Route 0
No L2 Route 15976

VLAN fwd intf failed TX 0
VLAN fwd intf failed enq 0

most if not all errors comes from
Statistics for core 10

GARP 0
ARP no where to go 0
Invalid ARPs 0

Invalid IF 0
Trap No IF 0
IF TX Discard 0
IF Drop 852489867
IF RX Discard 0

Flow Unusable 0
Flow No Memory 0
Flow Table Full 0
Flow NAT no rflow 0
Flow Action Drop 1183
Flow Action Invalid 0
Flow Invalid Protocol 0
Flow Queue Limit Exceeded 0

Discards 0
TTL Exceeded 0
Mcast Clone Fail 0
Cloned Original 0

Invalid NH 0
Invalid Label 0
Invalid Protocol 0
Rewrite Fail 0
Invalid Mcast Source 0

Push Fails 0
Pull Fails 0
Duplicated 0
Head Alloc Fails 0
Head Space Reserve Fails 0
PCOW fails 0
Invalid Packets 0

Misc 0
Nowhere to go 0
Checksum errors 0
No Fmd 0
Invalid VNID 0
Fragment errors 0
Invalid Source 0
Jumbo Mcast Pkt with DF Bit 0
ARP No Route 0
ARP Reply No Route 0
No L2 Route 289

VLAN fwd intf failed TX 0
VLAN fwd intf failed enq 0

root@node-36:~# vif -l
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/0 PCI: 0:0:0.0 (Speed 10000, Duplex 1)
Type:Physical HWaddr:90:e2:ba:8f:9d:f0 IPaddr:0
Vrf:0 Flags:L3L2Vp MTU:1514 Ref:17
RX device packets:117822 bytes:25471899 errors:1
RX port packets:117822 errors:0
RX queue packets:2698 errors:1108
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 1108 0
RX packets:116714 bytes:24893783 errors:0
TX packets:18969979 bytes:29044258968 errors:0
TX port packets:37567365 errors:0
TX device packets:37567365 bytes:30197296900 errors:0

vif0/1 Virtual: vhost0
Type:Host HWaddr:90:e2:ba:8f:9d:f0 IPaddr:0
Vrf:0 Flags:L3L2 MTU:1514 Ref:10
RX port packets:215901 errors:0
RX queue packets:209491 errors:6362
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 788 5574 0
RX packets:209539 bytes:230319764 errors:0
TX packets:114016 bytes:24632043 errors:0
TX queue packets:99643 errors:0
TX port packets:114016 errors:0

vif0/2 Socket: unix
Type:Agent HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:65535 Flags:L3 MTU:1514 Ref:2
RX port packets:2796 errors:0
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
RX packets:2796 bytes:240456 errors:4
TX packets:8270 bytes:8052699 errors:0
TX queue packets:8270 errors:0
TX port packets:8270 errors:0 syscalls:8278

vif0/4 PMD: tap8dfb4f0d-b4
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:8 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:1552 errors:0
RX queue packets:1373 errors:11
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 11 0
RX packets:1541 bytes:1482486 errors:0
TX packets:149 bytes:6258 errors:0
TX port packets:149 errors:0 syscalls:149

vif0/5 PMD: tapc9db1a03-ff
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:10 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:1548 errors:0
RX queue packets:1176 errors:209
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 209 0 0
RX packets:1339 bytes:1212282 errors:0
TX packets:143 bytes:6006 errors:0
TX port packets:143 errors:0 syscalls:143

vif0/6 Ethernet: vetheaa774c7-c
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:6 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:10 errors:0
RX queue packets:2 errors:0
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
RX packets:10 bytes:476 errors:0
TX packets:10 bytes:420 errors:2
TX queue packets:8 errors:0
TX port packets:8 errors:0

vif0/7 PMD: tap49823e4f-1b
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:10 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:1019377032 errors:0 syscalls:2048
RX queue packets:166893433 errors:852483430
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 852483422 8 0
RX packets:166893602 bytes:242893359607 errors:0
TX packets:1488 bytes:102392 errors:0
TX port packets:1488 errors:0 syscalls:1427

vif0/8 PMD: tapc6f25dc7-d3
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:10 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:997707640 errors:0 syscalls:3754
RX queue packets:198359542 errors:799347927
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 799347927 0
RX packets:198359713 bytes:277502123415 errors:0
TX packets:1454 bytes:100652 errors:0
TX port packets:1454 errors:0 syscalls:1397

vif0/9 Ethernet: vethb82ae1d8-8
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:1 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:10 errors:0
RX queue packets:2 errors:0
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
RX packets:10 bytes:476 errors:0
TX packets:10 bytes:420 errors:2
TX queue packets:8 errors:0
TX port packets:8 errors:0

vif0/10 PMD: tapdf63a5cb-16
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:8 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:1552 errors:0
RX queue packets:1333 errors:20
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 16 4 0
RX packets:1532 bytes:1472004 errors:0
TX packets:146 bytes:6132 errors:0
TX port packets:146 errors:0 syscalls:146

vif0/11 PMD: tapb05a7fdd-04
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:8 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:1556 errors:0
RX queue packets:1374 errors:30
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 18 12 0
RX packets:1526 bytes:1478750 errors:0
TX packets:87508103 bytes:131943575895 errors:0
TX port packets:87475103 errors:33000 syscalls:7818654

vif0/13 PMD: tap084f2567-c9
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:8 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:1545 errors:0
RX queue packets:1297 errors:86
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 86 0
RX packets:1459 bytes:1384826 errors:0
TX packets:143 bytes:6006 errors:0
TX port packets:143 errors:0 syscalls:143

vif0/14 PMD: tape3bc7866-3e
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:9 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:1 errors:0
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
RX packets:1 bytes:42 errors:0
TX packets:1 bytes:42 errors:0
TX port packets:1 errors:0 syscalls:1

vif0/15 PMD: tapccc14950-83
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:10 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:1541 errors:0
RX queue packets:1269 errors:112
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 19 93 0
RX packets:1429 bytes:1363553 errors:0
TX packets:141 bytes:5982 errors:0
TX port packets:141 errors:0 syscalls:141

vif0/16 Ethernet: veth24265d20-d
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:3 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:11 errors:0
RX queue packets:0 errors:3
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 3 0 0
RX packets:8 bytes:336 errors:0
TX packets:8 bytes:336 errors:0
TX queue packets:8 errors:0
TX port packets:8 errors:0

vif0/17 Ethernet: vethed5dc57a-1
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:3 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:11 errors:0
RX queue packets:2 errors:1
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 1 0
RX packets:10 bytes:476 errors:0
TX packets:8 bytes:336 errors:0
TX queue packets:8 errors:0
TX port packets:8 errors:0

vif0/18 PMD: tap89771556-14
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:10 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:1546 errors:0
RX queue packets:1327 errors:23
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 23 0 0
RX packets:1523 bytes:1471125 errors:0
TX packets:142 bytes:5964 errors:0
TX port packets:142 errors:0 syscalls:142

vif0/19 PMD: tap02760b58-df
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:8 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:1542 errors:0
RX queue packets:1287 errors:95
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 24 71 0
RX packets:1447 bytes:1385399 errors:0
TX packets:139 bytes:5838 errors:0
TX port packets:139 errors:0 syscalls:139

vif0/20 PMD: tapa5b35a02-3b
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:8 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:1586 errors:0
RX queue packets:1375 errors:42
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 31 11 0
RX packets:1544 bytes:1481533 errors:0
TX packets:29620743 bytes:44740006989 errors:0
TX port packets:29594077 errors:26667 syscalls:1763294

vif0/21 PMD: tap595588eb-33
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
Vrf:10 Flags:PL3L2D MTU:9160 Ref:13
RX port packets:1555 errors:0
RX queue packets:1370 errors:9
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 9 0
RX packets:1546 bytes:1478840 errors:0
TX packets:147 bytes:6174 errors:0
TX port packets:147 errors:0 syscalls:147

root@node-36:~#

root@node-36:~# vrfstats --dump
Vrf: 0
Discards 0, Resolves 0, Receives 130769, L2 Receives 0, Vrf Translates 0, Unknown Unicast Floods 0
Ecmp Composites 0, L2 Mcast Composites 0, Fabric Composites 0, Encap Composites 0, Evpn Composites 0
Udp Tunnels 0, Udp Mpls Tunnels 0, Gre Mpls Tunnels 0, Vxlan Tunnels 0
L2 Encaps 0, Encaps 234346
GROs 0, Diags 0
Arp Virtual Proxys 0, Arp Virtual Stitchs 0, Arp Virtual Floods 0, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Floods 0

Vrf: 1
Discards 0, Resolves 0, Receives 0, L2 Receives 9, Vrf Translates 0, Unknown Unicast Floods 0
Ecmp Composites 0, L2 Mcast Composites 2, Fabric Composites 0, Encap Composites 0, Evpn Composites 0
Udp Tunnels 0, Udp Mpls Tunnels 0, Gre Mpls Tunnels 0, Vxlan Tunnels 0
L2 Encaps 0, Encaps 0
GROs 0, Diags 0
Arp Virtual Proxys 0, Arp Virtual Stitchs 0, Arp Virtual Floods 0, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Floods 0

Vrf: 3
Discards 5397, Resolves 0, Receives 0, L2 Receives 19, Vrf Translates 0, Unknown Unicast Floods 0
Ecmp Composites 0, L2 Mcast Composites 2, Fabric Composites 0, Encap Composites 0, Evpn Composites 0
Udp Tunnels 0, Udp Mpls Tunnels 0, Gre Mpls Tunnels 18758961, Vxlan Tunnels 0
L2 Encaps 0, Encaps 0
GROs 0, Diags 0
Arp Virtual Proxys 0, Arp Virtual Stitchs 0, Arp Virtual Floods 0, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Floods 0

Vrf: 6
Discards 0, Resolves 0, Receives 0, L2 Receives 13, Vrf Translates 0, Unknown Unicast Floods 0
Ecmp Composites 0, L2 Mcast Composites 2, Fabric Composites 0, Encap Composites 0, Evpn Composites 0
Udp Tunnels 0, Udp Mpls Tunnels 0, Gre Mpls Tunnels 0, Vxlan Tunnels 0
L2 Encaps 0, Encaps 0
GROs 0, Diags 0
Arp Virtual Proxys 0, Arp Virtual Stitchs 0, Arp Virtual Floods 0, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Floods 0

Vrf: 8
Discards 0, Resolves 0, Receives 0, L2 Receives 1266, Vrf Translates 0, Unknown Unicast Floods 0
Ecmp Composites 0, L2 Mcast Composites 6, Fabric Composites 0, Encap Composites 0, Evpn Composites 0
Udp Tunnels 0, Udp Mpls Tunnels 0, Gre Mpls Tunnels 0, Vxlan Tunnels 0
L2 Encaps 977, Encaps 283
GROs 0, Diags 0
Arp Virtual Proxys 977, Arp Virtual Stitchs 0, Arp Virtual Floods 0, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Floods 0

Vrf: 9
Discards 0, Resolves 0, Receives 0, L2 Receives 244, Vrf Translates 0, Unknown Unicast Floods 0
Ecmp Composites 0, L2 Mcast Composites 0, Fabric Composites 0, Encap Composites 0, Evpn Composites 0
Udp Tunnels 0, Udp Mpls Tunnels 0, Gre Mpls Tunnels 0, Vxlan Tunnels 0
L2 Encaps 1, Encaps 389
GROs 0, Diags 0
Arp Virtual Proxys 1, Arp Virtual Stitchs 0, Arp Virtual Floods 0, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Floods 0

Vrf: 10
Discards 2617, Resolves 0, Receives 0, L2 Receives 127021923, Vrf Translates 0, Unknown Unicast Floods 0
Ecmp Composites 0, L2 Mcast Composites 6, Fabric Composites 0, Encap Composites 0, Evpn Composites 0
Udp Tunnels 0, Udp Mpls Tunnels 0, Gre Mpls Tunnels 0, Vxlan Tunnels 0
L2 Encaps 948, Encaps 127021557
GROs 0, Diags 0
Arp Virtual Proxys 948, Arp Virtual Stitchs 0, Arp Virtual Floods 0, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Floods 0

[nirhenn]

Another update, from looking on the flow table query in contrail web i notice that i see allot of drops on the traffic, the common to all is that destination VN is unknown from some reason.
deny2017-01-08 16:04:06:956:248node-36default-domain:Proj1:proj1_1__UNKNOWN__192.168.11.104192.168.100.5372415044TCP74 B1
deny2017-01-08 16:04:05:322:569node-36default-domain:Proj1:proj1_1__UNKNOWN__192.168.11.102192.168.100.54050325826UDP2.69 KB2
deny2017-01-08 16:04:05:265:420node-36default-domain:Proj1:proj1_1__UNKNOWN__192.168.11.103192.168.100.53978625826UDP0 B0
deny2017-01-08 16:04:04:438:845node-36default-domain:Proj1:proj1_1__UNKNOWN__192.168.11.105192.168.100.53691225826UDP1.35 KB1

[kiss9]

I face the same problem, a lot of packets dropped because of "IF Drop" of vif of VM. Any clues to further debug?