Arista EOS/Networking Tips by Solomon Yang

5/21/2020

Arista EVPN VXLAN Configuration Example (2b) - Single-homing, L2 EVPN, Vlan-aware

The VLAN-based MAC VRF has RD/RT values per VLAN/VNI. But most of the time, one tenant customer uses multiple VLANs. In this case, we can use just 1 RD/RT to mark the EVPN routes, which is called VLAN-aware bundle MAC VRF, and is illustrated as below:

Explanations:

Configuration is much like the VLAN-based MAC VRF
Configure VLAN-aware MAC-VRF under router BGP with RD/RT and it can have multiple VLANs
"redistribute learned" is to advertised the learnt MAC under VLAN as type-2 routes to remote EVPN peers.
Under interface Vxlan 1, configure VNI values for above VLANs

Control Plane Check-up:

1. IMET, almost same as VLAN-based, but under 1 RD/RT with 2 VNIs/VTEP

snp261-eVtep1#sh bgp evpn route-type imet rd 160.255.255.20:610 detail
BGP routing table information for VRF default
Router identifier 160.255.255.10, local AS number 65161
BGP routing table entry for imet 611 160.255.255.20, Route Distinguisher: 160.255.255.20:610
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:610 TunnelEncap:tunnelTypeVxlan
VNI: 611
PMSI Tunnel: Ingress Replication, MPLS Label: 611, Leaf Information Required: false, Tunnel ID: 160.255.255.20
BGP routing table entry for imet 612 160.255.255.20, Route Distinguisher: 160.255.255.20:610
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:610 TunnelEncap:tunnelTypeVxlan
VNI: 612
PMSI Tunnel: Ingress Replication, MPLS Label: 612, Leaf Information Required: false, Tunnel ID: 160.255.255.20

2. MAC-IP, under 1 RD/RT, but 2 VNI for 2 VLANs

snp261-eVtep1#sh bgp evpn route-type mac-ip rd 160.255.255.20:610 detail
BGP routing table information for VRF default
Router identifier 160.255.255.10, local AS number 65161
BGP routing table entry for mac-ip 611 444c.a8a5.1141, Route Distinguisher: 160.255.255.20:610
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:610 TunnelEncap:tunnelTypeVxlan
VNI: 611 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip 612 444c.a8a5.1141, Route Distinguisher: 160.255.255.20:610
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:610 TunnelEncap:tunnelTypeVxlan
VNI: 612 ESI: 0000:0000:0000:0000:0000

Ping check-up:

Host1#ping vrf EvpnHost1 160.61.1.201
PING 160.61.1.201 (160.61.1.201) 72(100) bytes of data.
80 bytes from 160.61.1.201: icmp_seq=1 ttl=64 time=0.790 ms
80 bytes from 160.61.1.201: icmp_seq=2 ttl=64 time=0.134 ms
80 bytes from 160.61.1.201: icmp_seq=3 ttl=64 time=0.102 ms
80 bytes from 160.61.1.201: icmp_seq=4 ttl=64 time=0.107 ms
80 bytes from 160.61.1.201: icmp_seq=5 ttl=64 time=0.117 ms

--- 160.61.1.201 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4ms
rtt min/avg/max/mdev = 0.102/0.250/0.790/0.270 ms, ipg/ewma 1.000/0.510 ms

Host2#ping vrf EvpnHost1 160.61.2.201
PING 160.61.2.201 (160.61.2.201) 72(100) bytes of data.
80 bytes from 160.61.2.201: icmp_seq=1 ttl=64 time=0.884 ms
80 bytes from 160.61.2.201: icmp_seq=2 ttl=64 time=0.105 ms
80 bytes from 160.61.2.201: icmp_seq=3 ttl=64 time=0.106 ms
80 bytes from 160.61.2.201: icmp_seq=4 ttl=64 time=0.097 ms
80 bytes from 160.61.2.201: icmp_seq=5 ttl=64 time=0.096 ms

--- 160.61.2.201 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4ms
rtt min/avg/max/mdev = 0.096/0.257/0.884/0.313 ms, ipg/ewma 1.001/0.559 ms

Arista EVPN VXLAN Configuration Example (2a) - Single-homing, L2 EVPN, Vlan-based

As of May 2020, the latest EOS 4.24.0F supports 2 types of L2 EVPN MAC-VRF:

Vlan-based

1 VRF (1 RD): 1 VLAN
ESI = 0

Vlan-aware Bundled

1 VRF (1 RD): n VLANs
ESI = VNI

Here is the configuration of Vlan-Based L2 EVPN

Explanations:

Configure VLAN aka MAC-VRF under router BGP with RD/RT
"redistribute learned" is to advertised the learnt MAC under VLAN as type-2 routes to remote EVPN peers.
Under interface Vxlan 1, configure VNI values for above VLANs

Control Plane Checkup:

EVPN uses 2 types routes for L2EVPN, 1) type-3 IMET, 2) type-2 MAC-IP

1) IMET

snp261-eVtep1#show bgp evpn route-type imet vni 601 detail
BGP routing table information for VRF default
Router identifier 160.255.255.10, local AS number 65161
BGP routing table entry for imet 160.255.255.10, Route Distinguisher: 160.255.255.10:601
Paths: 1 available
Local
- from - (0.0.0.0)
Origin IGP, metric -, localpref -, weight 0, valid, local, best
Extended Community: Route-Target-AS:65100:601 TunnelEncap:tunnelTypeVxlan
VNI: 601
PMSI Tunnel: Ingress Replication, MPLS Label: 601, Leaf Information Required: false, Tunnel ID: 160.255.255.10
BGP routing table entry for imet 160.255.255.20, Route Distinguisher: 160.255.255.20:601
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:601 TunnelEncap:tunnelTypeVxlan << RT to control import
VNI: 601 << VNI for this VLAN
PMSI Tunnel: Ingress Replication, MPLS Label: 601, Leaf Information Required: false, Tunnel ID: 160.255.255.20 << VTEP ID

2) Flood-set, the above IMET prefix is used to form the flood-set

snp261-eV1.16:27:32#show vxlan flood vtep vlan 601
VXLAN Flood VTEP Table
--------------------------------------------------------------------------------
VLANS Ip Address
----------------------------- ------------------------------------------------
601 160.255.255.20

snp261-eV1.16:23:04#show l2rib output floodset vlan 601
L2 RIB Output flood set:
Source: Local Dynamic, Local Static, BGP, VXLAN Static, VXLAN Dynamic
Vlan Address Type Destination
---------- -------------------- ---------- -------------------
601 0000.0000.0000 All VTEP 160.255.255.20

snp261-eV1.16:27:02#show l2rib input bgp floodset vlan 601
L2 RIB EVPN Input flood set:
Vlan Address Type Destination
---------- -------------------- ---------- -------------------
601 0000.0000.0000 All VTEP 160.255.255.20

3) Type-2 MAC-IP EVPN Route

snp261-eVtep1#show bgp evpn route-type mac-ip vni 601 detail
BGP routing table information for VRF default
Router identifier 160.255.255.10, local AS number 65161
BGP routing table entry for mac-ip 444c.a8a5.1140, Route Distinguisher: 160.255.255.10:601
Paths: 1 available
Local
- from - (0.0.0.0)
Origin IGP, metric -, localpref -, weight 0, valid, local, best
Extended Community: Route-Target-AS:65100:601 TunnelEncap:tunnelTypeVxlan
VNI: 601 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip 444c.a8a5.1141, Route Distinguisher: 160.255.255.20:601 << mac address
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:601 TunnelEncap:tunnelTypeVxlan << RT to control import
VNI: 601 ESI: 0000:0000:0000:0000:0000 << VNI

4) MAC table vs EVPN prefixes

snp261-eV1.16:29:56#show mac address-table interface vxlan 1 vlan 601
Mac Address Table
------------------------------------------------------------------

Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------
601 444c.a8a5.1141 DYNAMIC Vx1 1 4:53:10 ago

5) Clear MAC on remote VTEP to simulate MAC aging out

wa464-eVtep2#clear mac address-table dynamic vlan 601 << clear MAC

snp261-eVtep1#show bgp evpn route-type mac-ip vni 601 detail << NO evpn type-2 prefix
BGP routing table information for VRF default
Router identifier 160.255.255.10, local AS number 65161
BGP routing table entry for mac-ip 444c.a8a5.1140, Route Distinguisher: 160.255.255.10:601
Paths: 1 available
Local
- from - (0.0.0.0)
Origin IGP, metric -, localpref -, weight 0, valid, local, best
Extended Community: Route-Target-AS:65100:601 TunnelEncap:tunnelTypeVxlan
VNI: 601 ESI: 0000:0000:0000:0000:0000

snp261-eVtep1#show mac address-table interface vxlan 1 vlan 601 << no MAC entry
Mac Address Table
------------------------------------------------------------------

Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------

Data Plane Checkup:

host1 under VTEP1 ping host2 behind VTEP2

Host1#ping vrf EvpnHost1 160.60.1.201
PING 160.60.1.201 (160.60.1.201) 72(100) bytes of data.
80 bytes from 160.60.1.201: icmp_seq=1 ttl=64 time=0.135 ms
80 bytes from 160.60.1.201: icmp_seq=2 ttl=64 time=0.100 ms
80 bytes from 160.60.1.201: icmp_seq=3 ttl=64 time=0.092 ms
80 bytes from 160.60.1.201: icmp_seq=4 ttl=64 time=0.088 ms
80 bytes from 160.60.1.201: icmp_seq=5 ttl=64 time=0.089 ms

--- 160.60.1.201 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.088/0.100/0.135/0.021 ms, ipg/ewma 0.128/0.117 ms

VLAN-based: RD/RT vs VNI = 1:1

From the below output, the different VLANs have different RD and RT values, so 1:1 relationship. (In our case, only one host simulates multiple hosts under different VLANs).

snp261-eV1.18:18:45#sh bgp evpn route-type mac-ip vni 601 detail
BGP routing table entry for mac-ip 444c.a8a5.1141, Route Distinguisher: 160.255.255.20:601
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:601 TunnelEncap:tunnelTypeVxlan
VNI: 601 ESI: 0000:0000:0000:0000:0000

snp261-eV1.18:18:49#sh bgp evpn route-type mac-ip vni 602 detail
BGP routing table entry for mac-ip 444c.a8a5.1141, Route Distinguisher: 160.255.255.20:602
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:602 TunnelEncap:tunnelTypeVxlan
VNI: 602 ESI: 0000:0000:0000:0000:0000

5/20/2020

Arista EVPN VXLAN Configuration Example (1c) - eBGP Overlay

After the eBGP Undelay is up, we can move on to the eBGP Overlay.

Topology and configuration:

Explanations:

The BGP AS# of spine and leaf routes are different, so need to use BGP local-as feature to form iBGP peering
The benefit of BGP RR is that, the NH of EVPN updates are unchanged
Also need to disable the RR under address family ipv4 unicast or configure "no bgp default ipv4-unicast" under router bgp, otherwise you will see the following failed neighbor

bn303-eSpine1#show ip bgp summary

BGP summary information for VRF default

Router identifier 180.255.255.1, local AS number 65100

Neighbor Status Codes: m - Under maintenance

Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

....

eOverL-V1 160.255.255.10 4 65100 552 545 0 0 07:40:16 Estab(NotNegotiated)

eOverL-V2 160.255.255.20 4 65100 555 543 0 0 07:40:16 Estab(NotNegotiated)

And in the following blogs of various L2/L3 vxlan evpn setup, we don't need to touch spine routers anymore

From control plane point of view, spine1 only reflects BGP EVPN updates among leaf routers w/o VXLAN interface or VRF, which means it doesn't need to understand or import the content.
From data plane point of view, spine1 only forwards the IPv4/VXLAN packets by the source/destination address are leafs' loopback address.

Verifications:

1) BGP EVPN peering to VTEP1/2 are up

bn303-eSpine1#show bgp evpn summary
BGP summary information for VRF default
Router identifier 180.255.255.1, local AS number 65100
Neighbor Status Codes: m - Under maintenance
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
eOverL-V1 160.255.255.10 4 65100 581 573 0 0 00:20:21 Estab 2 2
eOverL-V2 160.255.255.20 4 65100 585 571 0 0 00:20:21 Estab 2 2

Arista EVPN VXLAN Configuration Example (1b) - eBGP Underlay

Before start, let's talk a bit regarding the underlay vs overlay. This topic is very well covered in the EVPN Deployment Guide. Here is my easy understanding:

The underlay is:

EBGP between Spine and Leafs over P2P links. Of course, IGP is an option.
To advertise the routing information of the VTEPs' and Spines' loopbacks for

EVPN peering sessions
The source/destination address Vxlan data traffic

You can use different loopback for EVPN and Vxlan

In EVPN Deployment Guide (page 20), lo0 for EVPN peering, lo1 for VxLAN
In this blog, I use the same loopback160 for both

Overlay control plane is BGP EVPN, to

Register the VTEP (type 3), it is like L2vpn autodiscovery.
Carry EVPN prefixes L2 and L3

Topology and Configuration:

Explanation:

EVPN is only supported in BGP multi-agent mode.
Simple EBGP peering over b2b ethernet interfaces
All routers advertise the loopback160 /32 address
Please note, the spine routers/RR have a route-map to control, so that only /32 loopback prefixes within 160.255.255.0/24 are advertised.

The b2b interface addresses are not needed in the data plane, so unnecessary for routing protocol.
And this eBGP setup implies that interface address can be duplicated in different POP/DC. A big plus for automation

Control Plane Verification:
1) show ip bgp sum on VTEP1, and please note only 2 bgp routes

snp261-eVtep1#sh ip bgp sum
BGP summary information for VRF default
Router identifier 160.255.255.10, local AS number 65161
Neighbor Status Codes: m - Under maintenance
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
eUnder-Sp1 160.1.10.1 4 65100 23840 23848 0 0 13d17h Estab 2 2

2) show ip bgp on VTEP1 to check the bgp prefixes, 1 for spine, 1 from VTEP2

snp261-eVtep1#show ip bgp
Network Next Hop Metric LocPref Weight Path
* > 160.255.255.1/32 160.1.10.1 0 100 0 65100 i
* > 160.255.255.10/32 - - - 0 i
* > 160.255.255.20/32 160.1.10.1 0 100 0 65100 65162 i

3) show bgp prefix of VTEP2's loopback on VTEP1

snp261-eVtep1#sh ip bgp 160.255.255.20
BGP routing table information for VRF default
Router identifier 160.255.255.10, local AS number 65161
BGP routing table entry for 160.255.255.20/32
Paths: 1 available
65100 65162
160.1.10.1 from 160.1.10.1 (180.255.255.1)
Origin IGP, metric 0, localpref 100, weight 0, received 23:22:11 ago, valid, external, best
Rx SAFI: Unicast

4) show ip route on VTEP1 to ensure bgp route in routing table

snp261-eVtep1#show ip route 160.255.255.20/32
B E 160.255.255.20/32 [200/0] via 160.1.10.1, Ethernet13

5) VTEP1 pings VTEP2's lo160 - 160.255.255.20

snp261-eVtep1#ping 160.255.255.20 source lo160
PING 160.255.255.20 (160.255.255.20) from 160.255.255.10 : 72(100) bytes of data.
80 bytes from 160.255.255.20: icmp_seq=1 ttl=63 time=0.248 ms
80 bytes from 160.255.255.20: icmp_seq=2 ttl=63 time=0.128 ms
80 bytes from 160.255.255.20: icmp_seq=3 ttl=63 time=0.087 ms
80 bytes from 160.255.255.20: icmp_seq=4 ttl=63 time=0.082 ms
80 bytes from 160.255.255.20: icmp_seq=5 ttl=63 time=0.091 ms

--- 160.255.255.20 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.082/0.127/0.248/0.062 ms, ipg/ewma 0.179/0.184 ms

At this step, we are pretty sure the underlay is ready because the overlay bgp evpn peering is based on VTEPs' loopback interfaces.

Arista EVPN VXLAN Configuration Example (1a) - Overview

I am starting a series of blogs on the Arista EVPN Vxlan configuration and will cover the following topics:

Underlay/Overlay BGP Configuration (No IGP involved)
L2 EVPN vs L3

L2 VLAN-based vs L2 VLAN-aware
Symmetric and asymmetric routing

Single-homing and multi-homing

Multi-homing: MLAG vs EVPN active/active

2 major Arista platforms: Jericho and Trident families
Router reflector and router server
Inter-VPN solution
EVPN inter VRF route leak
Some other advanced features

Dynamic BGP peering

Pre-requisites: 2 things need to be taken care of before EVPN configuration:

BGP multi-agent mode, EVPN is ONLY supported with this mode

service routing protocols model multi-agent
And need to reboot device to make this effective

Hardware setup:

For Trident 2 (7050*X), Tomahawk (7060*X) family devices like , recirculation must be enabled
For Arad (7280E, 7500E)/Jericho (7280R*, 7500R*) series device, select vxlan-routing TCAM profile

List of reference:

Arista BGP EVPN - Configuration Example @ Overlaid

What's the difference between this series of blog and the above official documents?

Configuration and trouble-shooting focused, no theory. The above links did a good job on the theory explanation, so I don't need to waste time and effort here.
Simplest topology and step by step configuration

Starting with 2 single-homing VTEPs + 1 Spine + RR
L2 EVPN (vlan-based, vlan-bundle-aware) and L3 EVPN
2 dual-homing VTEPs by MLAG or EVPN A/A
IPv4/v6 overlay
Adding 1 Spine for ECMP and RS
Only necessary and user-input configuration (no BGP max routes or link speed)

Hardware and scale information and consideration
Convergence/switchover time

4/26/2020

Create a vEOS-lab vm with vmware Esx

Before starting, download vmdk and aboot.iso from arista.com. This example uses vEOS-lab-4.23.M.vmdk and Aboot-veso-8.0.0.iso.

Based on my experiences. vEOS-lab has more features enabled than vEOS-cloud, for example, IGP/ISIS.

1. Click "Create/Register VM", then "Creat a new VM"

2. Set ESX and OS values

3. Customize Setting. 2 vCPU and 4GB memory should be enough for lab usage. Need to be careful about HD and CD/DVD.

3.1 HD1 must use IDE0:slave. (IDE0:master is saved for CD/DVD to bootup)

3.2 Add 2nd HD to use the download vmdk file. And uses IDE1:Master. (ISCSI doesn't work)

3.3 Set CD/DVD to use the downloaded Aboot.iso file. By default it uses IDE0:Master.

4. Now all setting is done. You can power up the vEOS. It would boot up with ZTP enabled and would broadcast DHCP requests to get initial setup. You can login by using id - admin, and type "zerotouch disable", then router would reboot and change the behavior to normal router

4/25/2020

ECMP in MPLS L3 VPN

Consider the above topology

CE2 advertises prefix - 12.0.0.0/24 to both PE2 and PE3;
On PE2/PE3, this prefix fails in the same VRF but with different RD, says 65100:2 and 65100:3
Both PE2 and PE3 advertise this vpnv4 prefix along the path, ASBR2/3 - ASBR1 - PE1.

1. On the PE1, we could see, no ECMP unde vpnv4, because different RD

PE1#show bgp vpn-ipv4 12.0.0.0/24

BGP routing table information for VRF default

Router identifier 5.5.5.5, local AS number 65000

BGP routing table entry for IPv4 prefix 12.0.0.0/24, Route Distinguisher: 65100:2

Paths: 1 available

65100 65101

11.11.11.11 from 1.1.1.1 (1.1.1.1)

Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

Extended Community: Route-Target-AS:65000:1

Remote MPLS label: 118012

BGP routing table entry for IPv4 prefix 12.0.0.0/24, Route Distinguisher: 65100:3

Paths: 1 available

65100 65101

11.11.11.11 from 1.1.1.1 (1.1.1.1)

Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

Extended Community: Route-Target-AS:65000:4364

Remote MPLS label: 116707

2. In the VRF routing table, there is ECMP to this destination. Because at PE, the ECMP entry is by ASBR NH + remote ASBR Label. In this case, 11.11.11.11:11607 and 11.11.11.11:118012

PE1#show ip route vrf cust_1 12.0.0.0/24

VRF: cust_1

......

B I 12.0.0.0/24 [200/0] via 11.11.11.11/32, LDP tunnel index 1, label 116707

via 1.0.0.8, Ethernet2/1, label 100000

via 11.11.11.11/32, LDP tunnel index 1, label 118012

via 1.0.0.8, Ethernet2/1, label 100000

3. Hw routing table shows the ECMP index

PE1#show ip hardware ale vrf | egrep 'cust_1|VRF'

VRF Name VRF ID Table ID

cust_1 15 65535

PE1#show platform jericho ip route 12.0.0.0/24

Tunnel Type: M(mpls), G(gre), MoG(mpls-over-gre),

vxlan-o(vxlan outer-rewrite info), vxlan-i(vxlan inner-rewrite info)

CW - Control word

FL - Flow label

* - Routes in LEM

D - ECMP is divergent across switching chips

---------------------------------------------------------------------------------------------------------

| Routing Table | |

|---------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------

|15 |12.0.0.0/24 |ROUTE| FEC 32831 |0 | - | |300 | D |M 116707

|15 |12.0.0.0/24 |ROUTE| FEC 32831 |0 | - | |300 | D |M 118012

4. But this behavior could exhaust the ECMP resource. From the below output, the 1600 vpnv4 prefixes use up 875 of 4096 ECMP entries.

PE1#show hardware capacity | grep -i ECMP

ECMP 875 21% 3220 0 4095 875

PE1#show bgp vpn-ipv4 summary

BGP summary information for VRF default

Router identifier 5.5.5.5, local AS number 65000

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

1.1.1.1 4 65000 1693 1673 0 0 23:25:15 Estab 1602 1602

5. So there is a workaround to disable fib ecmp and lower the ECMP usage.

PE1#conf term

PE1(config)#no ip hardware fib hierarchical next-hop disabled << default config, but have to flip

PE1(config)#router general

PE1(config-router-general)#rib fib fec ecmp emulated

PE1-lp402.17:32:38(config-router-general)#show hardware capacity | grep -i ECMP

ECMP 0 0% 4095 0 4095 875

ECMP Mpls 0 0% 4095 0 4095 0

ECMP Routing 0 0% 4095 0 4095 875

ECMP VxlanOverlay 0 0% 4095 0 4095 0

ECMP VxlanTunnel 0 0% 3891 0 3891 0

Now no ECMP anymore in the sw/hw routing table.

PE1#show ip route vrf cust_1 12.0.0.0/24

VRF: cust_1

B I 12.0.0.0/24 [200/0] via 11.11.11.11/32, LDP tunnel index 1, label 116707

via 1.0.0.8, Ethernet2/1, label 100000

via 11.11.11.11/32, LDP tunnel index 1, label 118012, backup

via 1.0.0.8, Ethernet2/1, label 100000

PE1#show platform jericho ip route 12.0.0.0/24

---------------------------------------------------------------------------------------------------------

| Routing Table | |

|---------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------

|15 |12.0.0.0/24 |ROUTE| FEC 32830 |0 | - | | - |49624 |M 116707

6. You probably want to take one step further to ask why the ECMP. Both PE2/PE3 are Arista EOS device which allocates labels per VRF. In this setup, only 8 VRFs but why the PE sees 800+ labels.

Now let's check ASBR1, which receives 800 from 2 ASBRs as expected

ASBR1#sh bgp vpn-ipv4 summary

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 65000

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

....

192.158.115.11 4 65100 1568 1819 0 0 1d00h Estab 800 800

192.168.115.11 4 65100 1749 1827 0 0 1d00h Estab 802 802

But different number of unique labels

ASBR1#sh bgp neighbors 192.168.115.11 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l

ASBR1#sh bgp neighbors 192.158.115.11 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l

800

Neighbor 192.158.115.11 which is Cisco XR device, sends 800 vpnv4 prefixes with per-prefix labels! Even it receives per-VRF labels, it still re-assign per-prefix labels.

RP/0/RSP1/CPU0:ASBR3#show bgp vpnv4 unicast rd 65100:101 12.0.0.0/24 detail | inc bel

Sat Apr 25 16:55:59.312 UTC

Local Label: 16694 (with rewrite);

Received Label 116390

RP/0/RSP1/CPU0:ASBR3#show bgp vpnv4 unicast rd 65100:101 12.0.1.0/24 detail | inc Label

Sat Apr 25 16:56:02.058 UTC

Local Label: 16695 (with rewrite);

Received Label 116390

4/24/2020

Arista Sand-based System - MPLS Label Usage

If you see the following error message on Arista Jericho-based system associated with MPLS traffic loss, most likely the MPLS label hardware resource is exhausted.

Feb 4 15:12:32 ASBR SandL3Unicast: %SAND-3-ROUTING_MPLS_TUNNEL_RESOURCE_FULL: Hardware resources are insufficient to program all mpls tunnel entries

Look deeply into a problematic prefix:

ASBR#sh bgp vpn-ipv4 101.1.1.0/24
BGP routing table information for VRF default
Router identifier 172.16.0.55, local AS number 8073
BGP routing table entry for IPv4 prefix 101.1.1.0/24, Route Distinguisher: 65000:1
Paths: 2 available
65000 65101
192.168.66.7 from 192.168.0.7 (192.168.0.7)
Origin IGP, metric -, localpref 100, weight 0, valid, external, best
Extended Community: Route-Target-AS:65000:1
Remote MPLS label: 116396
Local MPLS label (allocated for received VPN routes): 186827
...

The local label for this vpnv4 prefix is 186827, but hw programming is DROP

ASBR#sh platform fap mpls route | egrep '186827|Dest'
| Label | Action | Destination | VID |Outlif | MAC / CPU Code |Index| Index| Action
|186827 |Forward | DROP |0 | - | |16434|358104| -
|186827 |Forward | DROP |0 | - | |16434|358105| -

Check MPLS route table, and it shows totally 93206 labels used

ASBR#sh mpls lfib route | grep "B3" | wc -l
93206

And check hardware Check MPLS route table, and it shows EEDB/MplsTunnel usage is almost 99%

ASBR#sh hardware capacity | egrep -i 'chip|mplstunn'
Table Feature Chip Used Used Free Committed Best Case High
EEDB MplsTunnel Jericho0 93206 99% 1002 4096 98304 93206

Arista EOS default label mode is per-vrf, so this high # of label is probably caused by other vendors. Now let's check how to check which neighbor sending most labels (this is another system running w/o any issue)

ASBR1#sh bgp vpn-ipv4 summary
BGP summary information for VRF default
Router identifier 1.1.1.1, local AS number 65000
Neighbor Status Codes: m - Under maintenance
Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
5.5.5.5 4 65000 421 405 0 0 05:20:01 Estab 801 801
192.158.115.11 4 8073 381 439 0 0 05:30:45 Estab 800 800
192.168.115.11 4 8073 410 439 0 0 05:29:12 Estab 802 802

Use the following command, you can see neighbor 5.5.5.5 totally sending 8 labels.

ASBR1#show bgp neighbors 5.5.5.5 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l
8

How to know the maximum number of labels supported on the device? You can check it by running this command - "show hardware capacity" and check the line of "EEDB MplsTunnel":

7280CR#sh hardware capacity | egrep -i 'chip|eedb' | egrep -i 'chip|mpls'
Table Feature Chip Used Used Free Committed Best Case High
EEDB MplsTunnel Jericho0 0 0% 77824 0 81920 0

Here is a list of the Jericho-family models:
7280R/Jericho : 80K
7280R2/Jericho+: 96K

3/07/2020

Arista EOS MTU Behavior (3) - L3 Routed Ethernet Ports, IPv4

Now we switched to L3 routed ethernet (IPv4) ports:

MTU enforcement is for egress traffic
Hardware programs the MTU value (default value is 1500, in this case, 2000) and checks against forwarding packet length (in this case, it is ipv4 packet).
For IPv4 packets, if the length is more than MTU, the packets will be fragmented by CPU.
And the number of fragmented and dropped packets are reflected by CoPPEgressTrap counter in the output of "show cpu counters queue summary | nz"

Arista EOS MTU Behavior (2) - L2 Switched Ports

Below is a quick lab using 7280CR which is Jericho-based system,

Use EOS tool - Ethxmit on the generator to send packets

--ip-dst pointing to the destination address on the receiver device
--size specifies the packet size. Please the real ethernet packet size is this number minus 4 (I believe it is ethernet trailer). So 1504 means 1500-byte packet is received on sender ingress port

Run tcpdump + mirror on R3/receiver to check MTU behavior

Conclusion:

For switched traffic, no MTU enforcement, ingress nor egress

Some other observations:

For using ethxmit:

--size parameter for ethxmit includes the 4-byte CRC/ethernet trailer
If egress is access port, the ethernet frame size = L3 MTU + 18-byte including 14-byte ethernet header and 4-byte CRC.
If egress port is trunk, ethernet frame size = L3 MTU + 22-byte including 14-byte ethernet, 4-byte vlan and 4-byte CRC

No matter the mtu size under L2 interfaces, the forwarding MTU is set to maximum value 10178 (even show interface says Ethernet MTU, but it is not actual ethernet frame size, but forwarding MTU before ethernet header). Look at the ip header total length value in tcpdump output.

Trident-based system would use 9214.

R2#sh int e47/1 | grep -i mtu
Ethernet MTU 10178 bytes , BW 10000000 kbit
R2#sh run int e47/1
interface Ethernet47/1
mtu 100
switchport access vlan 100

3/03/2020

Arista EOS MTU Behavior (1)

Summary:

In this series of blogs, I will cover the MTU (Maximum Transmission Unit) behavior on Arista Jericho-based routers. Here is the summary:

1. MTU enforcement happens on the egress interfaces.

So MTU is not MRU (maximum receive unit), large size packets are NOT dropped at ingress interfaces

2. Arista MTU is layer-3 MTU on Jericho chipset:

2.1 The MTU enforcement is only for routed packets, no action on switched traffic
2.2 The actual packet size is MTU size plus L2 header size, which is 14-byte for L3 routed interface or 18-byte for the trunk interface

3. IPv4 packets are fragmented by router CPU and capped by CoPP

Reference:

2/18/2020

MTU value in EOS interop with Cisco XR

Summary:

Arista mtu value under interface is IP MTU value, while Cisco XR's mtu is L2 MTU
So when ISIS peering fails to come up if both sides have same MTU value
XR's mtu = EOS's mtu + 14 (Ethernet header length)

https://eos.arista.com/forum/isis-between-arista-and-ios-xr/