Arista Sand-based System - MPLS Label Usage

If you see the following error message on Arista Jericho-based system associated with MPLS traffic loss, most likely the MPLS label hardware resource is exhausted. 

Feb  4 15:12:32 ASBR SandL3Unicast: %SAND-3-ROUTING_MPLS_TUNNEL_RESOURCE_FULL: Hardware resources are insufficient to program all mpls tunnel entries

Look deeply into a problematic prefix:

ASBR#sh bgp vpn-ipv4 101.1.1.0/24
BGP routing table information for VRF default
Router identifier 172.16.0.55, local AS number 8073
BGP routing table entry for IPv4 prefix 101.1.1.0/24, Route Distinguisher: 65000:1
 Paths: 2 available
  65000 65101
    192.168.66.7 from 192.168.0.7 (192.168.0.7)
      Origin IGP, metric -, localpref 100, weight 0, valid, external, best
      Extended Community: Route-Target-AS:65000:1
      Remote MPLS label: 116396
      Local MPLS label (allocated for received VPN routes): 186827
...

The local label for this vpnv4 prefix is 186827, but hw programming is DROP

ASBR#sh platform fap mpls route | egrep '186827|Dest'
| Label |  Action  |     Destination    | VID |Outlif |   MAC / CPU Code  |Index| Index| Action
|186827 |Forward   | DROP               |0    |  -    |                   |16434|358104|   -
|186827 |Forward   | DROP               |0    |  -    |                   |16434|358105|   -

Check MPLS route table, and it shows totally 93206 labels used

ASBR#sh mpls lfib route | grep "B3" | wc -l
93206

And check hardware Check MPLS route table, and it shows EEDB/MplsTunnel usage is almost 99%

ASBR#sh hardware capacity | egrep -i 'chip|mplstunn'
Table             Feature           Chip              Used     Used        Free     Committed     Best Case        High
EEDB              MplsTunnel        Jericho0          93206       99%       1002          4096         98304        93206

Arista EOS default label mode is per-vrf, so this high # of label is probably caused by other vendors. Now let's check how to check which neighbor sending most labels (this is another system running w/o any issue)

ASBR1#sh bgp vpn-ipv4 summary
BGP summary information for VRF default
Router identifier 1.1.1.1, local AS number 65000
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State   PfxRcd PfxAcc
  5.5.5.5          4  65000            421       405    0    0 05:20:01 Estab   801    801
  192.158.115.11   4  8073             381       439    0    0 05:30:45 Estab   800    800
  192.168.115.11   4  8073             410       439    0    0 05:29:12 Estab   802    802

Use the following command, you can see neighbor 5.5.5.5 totally sending 8 labels. 

ASBR1#show bgp neighbors 5.5.5.5 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l
8

How to know the maximum number of labels supported on the device? You can check it by running this command - "show hardware capacity" and check the line of "EEDB MplsTunnel":

7280CR#sh hardware capacity | egrep -i 'chip|eedb' | egrep -i 'chip|mpls'
Table             Feature           Chip              Used     Used        Free     Committed     Best Case        High
EEDB              MplsTunnel        Jericho0             0       0%       77824             0         81920           0

Here is a list of the Jericho-family models:
7280R/Jericho  : 80K
7280R2/Jericho+: 96K

Arista EOS MTU Behavior (3) - L3 Routed Ethernet Ports, IPv4

Now we switched to L3 routed ethernet (IPv4) ports:

• MTU enforcement is for egress traffic
• Hardware programs the MTU value (default value is 1500, in this case, 2000) and checks against forwarding packet length (in this case, it is ipv4 packet). 
• For IPv4 packets, if the length is more than MTU, the packets will be fragmented by CPU. 
• And the number of fragmented and dropped packets are reflected by CoPPEgressTrap counter in the output of "show cpu counters queue summary | nz"

Arista EOS MTU Behavior (2) - L2 Switched Ports

Below is a quick lab using 7280CR which is Jericho-based system, 

• Use EOS tool - Ethxmit on the generator to send packets
• --ip-dst pointing to the destination address on the receiver device
• --size specifies the packet size. Please the real ethernet packet size is this number minus 4 (I believe it is ethernet trailer). So 1504 means 1500-byte packet is received on sender ingress port
• Run tcpdump + mirror on R3/receiver to check MTU behavior

Conclusion:

• For switched traffic, no MTU enforcement, ingress nor egress

Some other observations:

• For using ethxmit:
• --size parameter for ethxmit includes the 4-byte CRC/ethernet trailer
• If egress is access port, the ethernet frame size = L3 MTU + 18-byte including 14-byte ethernet header and 4-byte CRC. 
• If egress port is trunk, ethernet frame size = L3 MTU + 22-byte including 14-byte ethernet, 4-byte vlan and 4-byte CRC
• No matter the mtu size under L2 interfaces, the forwarding MTU is set to maximum value 10178 (even show interface says Ethernet MTU, but it is not actual ethernet frame size, but forwarding MTU before ethernet header). Look at the ip header total length value in tcpdump output. 
• Trident-based system would use 9214. 

R2#sh int e47/1 | grep -i mtu
  Ethernet MTU 10178 bytes , BW 10000000 kbit
R2#sh run int e47/1
interface Ethernet47/1
   mtu 100
   switchport access vlan 100

Arista EOS MTU Behavior (1)

Summary:

In this series of blogs, I will cover the MTU (Maximum Transmission Unit) behavior on Arista Jericho-based routers. Here is the summary:

• 1. MTU enforcement happens on the egress interfaces. 
• So MTU is not MRU (maximum receive unit), large size packets are NOT dropped at ingress interfaces
• 2. Arista MTU is layer-3 MTU on Jericho chipset:
• 2.1 The MTU enforcement is only for routed packets, no action on switched traffic
• 2.2 The actual packet size is MTU size plus L2 header size, which is 14-byte for L3 routed interface or 18-byte for the trunk interface
• 3. IPv4 packets are fragmented by router CPU and capped by CoPP

Reference:

• Arista EOS Manual, Section 19.4 MTU
• Arista 4.23.1F TOI - Support for L3 MTU on 7280R3/7500R3/7800R3