Arista EOS MTU Behavior (3) - L3 Routed Ethernet Ports, IPv4

Now we switched to L3 routed ethernet (IPv4) ports:

• MTU enforcement is for egress traffic
• Hardware programs the MTU value (default value is 1500, in this case, 2000) and checks against forwarding packet length (in this case, it is ipv4 packet). 
• For IPv4 packets, if the length is more than MTU, the packets will be fragmented by CPU. 
• And the number of fragmented and dropped packets are reflected by CoPPEgressTrap counter in the output of "show cpu counters queue summary | nz"

Arista EOS MTU Behavior (2) - L2 Switched Ports

Below is a quick lab using 7280CR which is Jericho-based system, 

• Use EOS tool - Ethxmit on the generator to send packets
• --ip-dst pointing to the destination address on the receiver device
• --size specifies the packet size. Please the real ethernet packet size is this number minus 4 (I believe it is ethernet trailer). So 1504 means 1500-byte packet is received on sender ingress port
• Run tcpdump + mirror on R3/receiver to check MTU behavior

Conclusion:

• For switched traffic, no MTU enforcement, ingress nor egress

Some other observations:

• For using ethxmit:
• --size parameter for ethxmit includes the 4-byte CRC/ethernet trailer
• If egress is access port, the ethernet frame size = L3 MTU + 18-byte including 14-byte ethernet header and 4-byte CRC. 
• If egress port is trunk, ethernet frame size = L3 MTU + 22-byte including 14-byte ethernet, 4-byte vlan and 4-byte CRC
• No matter the mtu size under L2 interfaces, the forwarding MTU is set to maximum value 10178 (even show interface says Ethernet MTU, but it is not actual ethernet frame size, but forwarding MTU before ethernet header). Look at the ip header total length value in tcpdump output. 
• Trident-based system would use 9214. 

R2#sh int e47/1 | grep -i mtu
  Ethernet MTU 10178 bytes , BW 10000000 kbit
R2#sh run int e47/1
interface Ethernet47/1
   mtu 100
   switchport access vlan 100

Arista EOS MTU Behavior (1)

Summary:

In this series of blogs, I will cover the MTU (Maximum Transmission Unit) behavior on Arista Jericho-based routers. Here is the summary:

• 1. MTU enforcement happens on the egress interfaces. 
• So MTU is not MRU (maximum receive unit), large size packets are NOT dropped at ingress interfaces
• 2. Arista MTU is layer-3 MTU on Jericho chipset:
• 2.1 The MTU enforcement is only for routed packets, no action on switched traffic
• 2.2 The actual packet size is MTU size plus L2 header size, which is 14-byte for L3 routed interface or 18-byte for the trunk interface
• 3. IPv4 packets are fragmented by router CPU and capped by CoPP

Reference:

• Arista EOS Manual, Section 19.4 MTU
• Arista 4.23.1F TOI - Support for L3 MTU on 7280R3/7500R3/7800R3

MTU value in EOS interop with Cisco XR

Summary:

• Arista mtu value under interface is IP MTU value, while Cisco XR's mtu is L2 MTU
• So when ISIS peering fails to come up if both sides have same MTU value
• XR's mtu = EOS's mtu + 14 (Ethernet header length)

https://eos.arista.com/forum/isis-between-arista-and-ios-xr/

Arista MPLS VPN Step by Step (2c) L2VPN - Type 4 PW, MTU

In the LDP PW configuration, one thing needs to be careful is the MTU. You have to ensure the MTU along the underlying LSP matched. Here is an example:

So we have OSPF and ISIS enabled on VLAN 301/302 on CEs. But only OSPF up, no ISIS neighbor at all

CE1#sh ip os neighbor vrf ce1-pw4-301
Neighbor ID     VRF      Pri State                  Dead Time   Address         Interface
101.255.2.2     ce1-pw4-301 1   FULL/DR                00:00:32    101.101.101.2   Vlan301
CE1#sh isis neighbors vrf ce1-pw4-302
CE1#

Well the ping works as expected, CE1 can ping CE2's vlan 302 address - 102.102.102.2

CE1#ping vrf ce1-pw4-302 102.102.102.2
PING 102.102.102.2 (102.102.102.2) 72(100) bytes of data.
80 bytes from 102.102.102.2: icmp_seq=1 ttl=64 time=0.276 ms
80 bytes from 102.102.102.2: icmp_seq=2 ttl=64 time=0.119 ms
80 bytes from 102.102.102.2: icmp_seq=3 ttl=64 time=0.112 ms
80 bytes from 102.102.102.2: icmp_seq=4 ttl=64 time=0.111 ms
80 bytes from 102.102.102.2: icmp_seq=5 ttl=64 time=0.110 ms

--- 102.102.102.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.110/0.145/0.276/0.066 ms, ipg/ewma 0.202/0.208 ms

Both sides configured with MTU 9000

CE1#sh run int vlan 302
interface Vlan302
   mtu 9000
   vrf forwarding ce1-pw4-302
   ip address 102.102.102.1/24
   isis enable isis-ce1-pw4-302

CE2#sh run int vlan 302
interface Vlan302
   mtu 9000
   vrf forwarding ce2-pw4-302
   ip address 102.102.102.2/24
   isis enable isis-ce2-pw4-302

Let's try ping the destination with size of data plus DF-bit

CE1#ping vrf ce1-pw4-302 102.102.102.2 size 9000 df-bit
PING 102.102.102.2 (102.102.102.2) 8972(9000) bytes of data.
--- 102.102.102.2 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 40ms

Ok, it fails but where is the culprit?

PE1 and PE2 are good:

PE1(config-if-Et47/2)#sh ip int brief | grep 88
Ethernet3/1            88.1.11.11/24      up         up              9082
Ethernet45/3           88.3.11.11/24      down       down            9082

PE2(config-if-Et51/1)#sh ip int brief | grep 88
Ethernet33/1           88.4.22.22/24      up         up              9082
Ethernet51/1           88.2.22.22/24      up         up              9082

Let's move on to P routers:

P1#sh ip int brief | grep 88
Ethernet35/1           88.1.4.1/24        up         up              8000
Ethernet51/1           88.1.2.1/24        up         up              8000
Ethernet52/1           88.1.3.1/24        up         up              9082
Ethernet54/1           88.1.11.1/24       up         up              9082

Ok, here is the issue. Let's correct them and ISIS neighbor is up!

CE1#sh isis neighbors vrf ce1-pw4-302

Instance  VRF      System Id        Type Interface          SNPA              State Hold time   Circuit Id
isis-ce1- ce1-pw4- CE2              L2   Vlan302            44:4c:a8:11:6b:e1 UP    8           CE2.02

So, the reason is, the ISIS pads hello packet upto MTU and the oversized packets will be dropped siliently on underlying P routers. Hence ISIS fails to form neighbor. 

The OSPF just negotiates the MTU of connecting interfaces, which are 9000 in both sides, so ospf neighbor is up w/o any issues. But if you have a large LSA database, and ospf utilize the full MTU to transmit it, you will hit this MTU mismatch later, but not in the neigobor discoverage stage. 

Wireshark Tips (2)

packetbomb.com - troubleshooting MTU issue

• Following the previous post, need column - tcpLen, Seq, nextSeq, Ack, BytesInFlight
• Statistics -> conversation, find the biggest stream 
• couple of TCP retransmission packets in jumbo frame with DF bit, because no ACK
• later sender starts with 512B -> 1024B, then conversation on. 
• clearly, 1 or multiple middle routers can't handle jumbo frames
• filter = icmp, doesn't show any icmp unreachable
• per RFC 1191 - path MTU discovery. The middle routers should return a ICMP destination Unreachable message with code = "frag needed and DF set"
• So these pkts are dropped by either firewall or disabled on the router

IP fragmentation on Arad platform

• Fragmented/MTU violated packet is sw forwarded and shaped by CoPP or PDP;
• CoPP = 2~2.5 Mbps
• "no shape" under copp-system-mtu can make it 100Mbps
• PDP = 100 Mbps
• But not a good idea to do it.