Arista EOS/Networking Tips by Solomon Yang: April 2020

4/26/2020

Create a vEOS-lab vm with vmware Esx

Before starting, download vmdk and aboot.iso from arista.com. This example uses vEOS-lab-4.23.M.vmdk and Aboot-veso-8.0.0.iso.

Based on my experiences. vEOS-lab has more features enabled than vEOS-cloud, for example, IGP/ISIS.

1. Click "Create/Register VM", then "Creat a new VM"

2. Set ESX and OS values

3. Customize Setting. 2 vCPU and 4GB memory should be enough for lab usage. Need to be careful about HD and CD/DVD.

3.1 HD1 must use IDE0:slave. (IDE0:master is saved for CD/DVD to bootup)

3.2 Add 2nd HD to use the download vmdk file. And uses IDE1:Master. (ISCSI doesn't work)

3.3 Set CD/DVD to use the downloaded Aboot.iso file. By default it uses IDE0:Master.

4. Now all setting is done. You can power up the vEOS. It would boot up with ZTP enabled and would broadcast DHCP requests to get initial setup. You can login by using id - admin, and type "zerotouch disable", then router would reboot and change the behavior to normal router

4/25/2020

ECMP in MPLS L3 VPN

Consider the above topology

CE2 advertises prefix - 12.0.0.0/24 to both PE2 and PE3;
On PE2/PE3, this prefix fails in the same VRF but with different RD, says 65100:2 and 65100:3
Both PE2 and PE3 advertise this vpnv4 prefix along the path, ASBR2/3 - ASBR1 - PE1.

1. On the PE1, we could see, no ECMP unde vpnv4, because different RD

PE1#show bgp vpn-ipv4 12.0.0.0/24

BGP routing table information for VRF default

Router identifier 5.5.5.5, local AS number 65000

BGP routing table entry for IPv4 prefix 12.0.0.0/24, Route Distinguisher: 65100:2

Paths: 1 available

65100 65101

11.11.11.11 from 1.1.1.1 (1.1.1.1)

Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

Extended Community: Route-Target-AS:65000:1

Remote MPLS label: 118012

BGP routing table entry for IPv4 prefix 12.0.0.0/24, Route Distinguisher: 65100:3

Paths: 1 available

65100 65101

11.11.11.11 from 1.1.1.1 (1.1.1.1)

Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

Extended Community: Route-Target-AS:65000:4364

Remote MPLS label: 116707

2. In the VRF routing table, there is ECMP to this destination. Because at PE, the ECMP entry is by ASBR NH + remote ASBR Label. In this case, 11.11.11.11:11607 and 11.11.11.11:118012

PE1#show ip route vrf cust_1 12.0.0.0/24

VRF: cust_1

......

B I 12.0.0.0/24 [200/0] via 11.11.11.11/32, LDP tunnel index 1, label 116707

via 1.0.0.8, Ethernet2/1, label 100000

via 11.11.11.11/32, LDP tunnel index 1, label 118012

via 1.0.0.8, Ethernet2/1, label 100000

3. Hw routing table shows the ECMP index

PE1#show ip hardware ale vrf | egrep 'cust_1|VRF'

VRF Name VRF ID Table ID

cust_1 15 65535

PE1#show platform jericho ip route 12.0.0.0/24

Tunnel Type: M(mpls), G(gre), MoG(mpls-over-gre),

vxlan-o(vxlan outer-rewrite info), vxlan-i(vxlan inner-rewrite info)

CW - Control word

FL - Flow label

* - Routes in LEM

D - ECMP is divergent across switching chips

---------------------------------------------------------------------------------------------------------

| Routing Table | |

|---------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------

|15 |12.0.0.0/24 |ROUTE| FEC 32831 |0 | - | |300 | D |M 116707

|15 |12.0.0.0/24 |ROUTE| FEC 32831 |0 | - | |300 | D |M 118012

4. But this behavior could exhaust the ECMP resource. From the below output, the 1600 vpnv4 prefixes use up 875 of 4096 ECMP entries.

PE1#show hardware capacity | grep -i ECMP

ECMP 875 21% 3220 0 4095 875

PE1#show bgp vpn-ipv4 summary

BGP summary information for VRF default

Router identifier 5.5.5.5, local AS number 65000

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

1.1.1.1 4 65000 1693 1673 0 0 23:25:15 Estab 1602 1602

5. So there is a workaround to disable fib ecmp and lower the ECMP usage.

PE1#conf term

PE1(config)#no ip hardware fib hierarchical next-hop disabled << default config, but have to flip

PE1(config)#router general

PE1(config-router-general)#rib fib fec ecmp emulated

PE1-lp402.17:32:38(config-router-general)#show hardware capacity | grep -i ECMP

ECMP 0 0% 4095 0 4095 875

ECMP Mpls 0 0% 4095 0 4095 0

ECMP Routing 0 0% 4095 0 4095 875

ECMP VxlanOverlay 0 0% 4095 0 4095 0

ECMP VxlanTunnel 0 0% 3891 0 3891 0

Now no ECMP anymore in the sw/hw routing table.

PE1#show ip route vrf cust_1 12.0.0.0/24

VRF: cust_1

B I 12.0.0.0/24 [200/0] via 11.11.11.11/32, LDP tunnel index 1, label 116707

via 1.0.0.8, Ethernet2/1, label 100000

via 11.11.11.11/32, LDP tunnel index 1, label 118012, backup

via 1.0.0.8, Ethernet2/1, label 100000

PE1#show platform jericho ip route 12.0.0.0/24

---------------------------------------------------------------------------------------------------------

| Routing Table | |

|---------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------

|15 |12.0.0.0/24 |ROUTE| FEC 32830 |0 | - | | - |49624 |M 116707

6. You probably want to take one step further to ask why the ECMP. Both PE2/PE3 are Arista EOS device which allocates labels per VRF. In this setup, only 8 VRFs but why the PE sees 800+ labels.

Now let's check ASBR1, which receives 800 from 2 ASBRs as expected

ASBR1#sh bgp vpn-ipv4 summary

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 65000

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

....

192.158.115.11 4 65100 1568 1819 0 0 1d00h Estab 800 800

192.168.115.11 4 65100 1749 1827 0 0 1d00h Estab 802 802

But different number of unique labels

ASBR1#sh bgp neighbors 192.168.115.11 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l

ASBR1#sh bgp neighbors 192.158.115.11 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l

800

Neighbor 192.158.115.11 which is Cisco XR device, sends 800 vpnv4 prefixes with per-prefix labels! Even it receives per-VRF labels, it still re-assign per-prefix labels.

RP/0/RSP1/CPU0:ASBR3#show bgp vpnv4 unicast rd 65100:101 12.0.0.0/24 detail | inc bel

Sat Apr 25 16:55:59.312 UTC

Local Label: 16694 (with rewrite);

Received Label 116390

RP/0/RSP1/CPU0:ASBR3#show bgp vpnv4 unicast rd 65100:101 12.0.1.0/24 detail | inc Label

Sat Apr 25 16:56:02.058 UTC

Local Label: 16695 (with rewrite);

Received Label 116390

4/24/2020

Arista Sand-based System - MPLS Label Usage

If you see the following error message on Arista Jericho-based system associated with MPLS traffic loss, most likely the MPLS label hardware resource is exhausted.

Feb 4 15:12:32 ASBR SandL3Unicast: %SAND-3-ROUTING_MPLS_TUNNEL_RESOURCE_FULL: Hardware resources are insufficient to program all mpls tunnel entries

Look deeply into a problematic prefix:

ASBR#sh bgp vpn-ipv4 101.1.1.0/24
BGP routing table information for VRF default
Router identifier 172.16.0.55, local AS number 8073
BGP routing table entry for IPv4 prefix 101.1.1.0/24, Route Distinguisher: 65000:1
Paths: 2 available
65000 65101
192.168.66.7 from 192.168.0.7 (192.168.0.7)
Origin IGP, metric -, localpref 100, weight 0, valid, external, best
Extended Community: Route-Target-AS:65000:1
Remote MPLS label: 116396
Local MPLS label (allocated for received VPN routes): 186827
...

The local label for this vpnv4 prefix is 186827, but hw programming is DROP

ASBR#sh platform fap mpls route | egrep '186827|Dest'
| Label | Action | Destination | VID |Outlif | MAC / CPU Code |Index| Index| Action
|186827 |Forward | DROP |0 | - | |16434|358104| -
|186827 |Forward | DROP |0 | - | |16434|358105| -

Check MPLS route table, and it shows totally 93206 labels used

ASBR#sh mpls lfib route | grep "B3" | wc -l
93206

And check hardware Check MPLS route table, and it shows EEDB/MplsTunnel usage is almost 99%

ASBR#sh hardware capacity | egrep -i 'chip|mplstunn'
Table Feature Chip Used Used Free Committed Best Case High
EEDB MplsTunnel Jericho0 93206 99% 1002 4096 98304 93206

Arista EOS default label mode is per-vrf, so this high # of label is probably caused by other vendors. Now let's check how to check which neighbor sending most labels (this is another system running w/o any issue)

ASBR1#sh bgp vpn-ipv4 summary
BGP summary information for VRF default
Router identifier 1.1.1.1, local AS number 65000
Neighbor Status Codes: m - Under maintenance
Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
5.5.5.5 4 65000 421 405 0 0 05:20:01 Estab 801 801
192.158.115.11 4 8073 381 439 0 0 05:30:45 Estab 800 800
192.168.115.11 4 8073 410 439 0 0 05:29:12 Estab 802 802

Use the following command, you can see neighbor 5.5.5.5 totally sending 8 labels.

ASBR1#show bgp neighbors 5.5.5.5 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l
8

How to know the maximum number of labels supported on the device? You can check it by running this command - "show hardware capacity" and check the line of "EEDB MplsTunnel":

7280CR#sh hardware capacity | egrep -i 'chip|eedb' | egrep -i 'chip|mpls'
Table Feature Chip Used Used Free Committed Best Case High
EEDB MplsTunnel Jericho0 0 0% 77824 0 81920 0

Here is a list of the Jericho-family models:
7280R/Jericho : 80K
7280R2/Jericho+: 96K