Arista EOS/Networking Tips by Solomon Yang

5/20/2020

Arista EVPN VXLAN Configuration Example (1c) - eBGP Overlay

After the eBGP Undelay is up, we can move on to the eBGP Overlay.

Topology and configuration:

Explanations:

The BGP AS# of spine and leaf routes are different, so need to use BGP local-as feature to form iBGP peering
The benefit of BGP RR is that, the NH of EVPN updates are unchanged
Also need to disable the RR under address family ipv4 unicast or configure "no bgp default ipv4-unicast" under router bgp, otherwise you will see the following failed neighbor

bn303-eSpine1#show ip bgp summary

BGP summary information for VRF default

Router identifier 180.255.255.1, local AS number 65100

Neighbor Status Codes: m - Under maintenance

Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

....

eOverL-V1 160.255.255.10 4 65100 552 545 0 0 07:40:16 Estab(NotNegotiated)

eOverL-V2 160.255.255.20 4 65100 555 543 0 0 07:40:16 Estab(NotNegotiated)

And in the following blogs of various L2/L3 vxlan evpn setup, we don't need to touch spine routers anymore

From control plane point of view, spine1 only reflects BGP EVPN updates among leaf routers w/o VXLAN interface or VRF, which means it doesn't need to understand or import the content.
From data plane point of view, spine1 only forwards the IPv4/VXLAN packets by the source/destination address are leafs' loopback address.

Verifications:

1) BGP EVPN peering to VTEP1/2 are up

bn303-eSpine1#show bgp evpn summary
BGP summary information for VRF default
Router identifier 180.255.255.1, local AS number 65100
Neighbor Status Codes: m - Under maintenance
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
eOverL-V1 160.255.255.10 4 65100 581 573 0 0 00:20:21 Estab 2 2
eOverL-V2 160.255.255.20 4 65100 585 571 0 0 00:20:21 Estab 2 2

Arista EVPN VXLAN Configuration Example (1b) - eBGP Underlay

Before start, let's talk a bit regarding the underlay vs overlay. This topic is very well covered in the EVPN Deployment Guide. Here is my easy understanding:

The underlay is:

EBGP between Spine and Leafs over P2P links. Of course, IGP is an option.
To advertise the routing information of the VTEPs' and Spines' loopbacks for

EVPN peering sessions
The source/destination address Vxlan data traffic

You can use different loopback for EVPN and Vxlan

In EVPN Deployment Guide (page 20), lo0 for EVPN peering, lo1 for VxLAN
In this blog, I use the same loopback160 for both

Overlay control plane is BGP EVPN, to

Register the VTEP (type 3), it is like L2vpn autodiscovery.
Carry EVPN prefixes L2 and L3

Topology and Configuration:

Explanation:

EVPN is only supported in BGP multi-agent mode.
Simple EBGP peering over b2b ethernet interfaces
All routers advertise the loopback160 /32 address
Please note, the spine routers/RR have a route-map to control, so that only /32 loopback prefixes within 160.255.255.0/24 are advertised.

The b2b interface addresses are not needed in the data plane, so unnecessary for routing protocol.
And this eBGP setup implies that interface address can be duplicated in different POP/DC. A big plus for automation

Control Plane Verification:
1) show ip bgp sum on VTEP1, and please note only 2 bgp routes

snp261-eVtep1#sh ip bgp sum
BGP summary information for VRF default
Router identifier 160.255.255.10, local AS number 65161
Neighbor Status Codes: m - Under maintenance
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
eUnder-Sp1 160.1.10.1 4 65100 23840 23848 0 0 13d17h Estab 2 2

2) show ip bgp on VTEP1 to check the bgp prefixes, 1 for spine, 1 from VTEP2

snp261-eVtep1#show ip bgp
Network Next Hop Metric LocPref Weight Path
* > 160.255.255.1/32 160.1.10.1 0 100 0 65100 i
* > 160.255.255.10/32 - - - 0 i
* > 160.255.255.20/32 160.1.10.1 0 100 0 65100 65162 i

3) show bgp prefix of VTEP2's loopback on VTEP1

snp261-eVtep1#sh ip bgp 160.255.255.20
BGP routing table information for VRF default
Router identifier 160.255.255.10, local AS number 65161
BGP routing table entry for 160.255.255.20/32
Paths: 1 available
65100 65162
160.1.10.1 from 160.1.10.1 (180.255.255.1)
Origin IGP, metric 0, localpref 100, weight 0, received 23:22:11 ago, valid, external, best
Rx SAFI: Unicast

4) show ip route on VTEP1 to ensure bgp route in routing table

snp261-eVtep1#show ip route 160.255.255.20/32
B E 160.255.255.20/32 [200/0] via 160.1.10.1, Ethernet13

5) VTEP1 pings VTEP2's lo160 - 160.255.255.20

snp261-eVtep1#ping 160.255.255.20 source lo160
PING 160.255.255.20 (160.255.255.20) from 160.255.255.10 : 72(100) bytes of data.
80 bytes from 160.255.255.20: icmp_seq=1 ttl=63 time=0.248 ms
80 bytes from 160.255.255.20: icmp_seq=2 ttl=63 time=0.128 ms
80 bytes from 160.255.255.20: icmp_seq=3 ttl=63 time=0.087 ms
80 bytes from 160.255.255.20: icmp_seq=4 ttl=63 time=0.082 ms
80 bytes from 160.255.255.20: icmp_seq=5 ttl=63 time=0.091 ms

--- 160.255.255.20 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.082/0.127/0.248/0.062 ms, ipg/ewma 0.179/0.184 ms

At this step, we are pretty sure the underlay is ready because the overlay bgp evpn peering is based on VTEPs' loopback interfaces.

Arista EVPN VXLAN Configuration Example (1a) - Overview

I am starting a series of blogs on the Arista EVPN Vxlan configuration and will cover the following topics:

Underlay/Overlay BGP Configuration (No IGP involved)
L2 EVPN vs L3

L2 VLAN-based vs L2 VLAN-aware
Symmetric and asymmetric routing

Single-homing and multi-homing

Multi-homing: MLAG vs EVPN active/active

2 major Arista platforms: Jericho and Trident families
Router reflector and router server
Inter-VPN solution
EVPN inter VRF route leak
Some other advanced features

Dynamic BGP peering

Pre-requisites: 2 things need to be taken care of before EVPN configuration:

BGP multi-agent mode, EVPN is ONLY supported with this mode

service routing protocols model multi-agent
And need to reboot device to make this effective

Hardware setup:

For Trident 2 (7050*X), Tomahawk (7060*X) family devices like , recirculation must be enabled
For Arad (7280E, 7500E)/Jericho (7280R*, 7500R*) series device, select vxlan-routing TCAM profile

List of reference:

Arista BGP EVPN - Configuration Example @ Overlaid

What's the difference between this series of blog and the above official documents?

Configuration and trouble-shooting focused, no theory. The above links did a good job on the theory explanation, so I don't need to waste time and effort here.
Simplest topology and step by step configuration

Starting with 2 single-homing VTEPs + 1 Spine + RR
L2 EVPN (vlan-based, vlan-bundle-aware) and L3 EVPN
2 dual-homing VTEPs by MLAG or EVPN A/A
IPv4/v6 overlay
Adding 1 Spine for ECMP and RS
Only necessary and user-input configuration (no BGP max routes or link speed)

Hardware and scale information and consideration
Convergence/switchover time

4/26/2020

Create a vEOS-lab vm with vmware Esx

Before starting, download vmdk and aboot.iso from arista.com. This example uses vEOS-lab-4.23.M.vmdk and Aboot-veso-8.0.0.iso.

Based on my experiences. vEOS-lab has more features enabled than vEOS-cloud, for example, IGP/ISIS.

1. Click "Create/Register VM", then "Creat a new VM"

2. Set ESX and OS values

3. Customize Setting. 2 vCPU and 4GB memory should be enough for lab usage. Need to be careful about HD and CD/DVD.

3.1 HD1 must use IDE0:slave. (IDE0:master is saved for CD/DVD to bootup)

3.2 Add 2nd HD to use the download vmdk file. And uses IDE1:Master. (ISCSI doesn't work)

3.3 Set CD/DVD to use the downloaded Aboot.iso file. By default it uses IDE0:Master.

4. Now all setting is done. You can power up the vEOS. It would boot up with ZTP enabled and would broadcast DHCP requests to get initial setup. You can login by using id - admin, and type "zerotouch disable", then router would reboot and change the behavior to normal router

4/25/2020

ECMP in MPLS L3 VPN

Consider the above topology

CE2 advertises prefix - 12.0.0.0/24 to both PE2 and PE3;
On PE2/PE3, this prefix fails in the same VRF but with different RD, says 65100:2 and 65100:3
Both PE2 and PE3 advertise this vpnv4 prefix along the path, ASBR2/3 - ASBR1 - PE1.

1. On the PE1, we could see, no ECMP unde vpnv4, because different RD

PE1#show bgp vpn-ipv4 12.0.0.0/24

BGP routing table information for VRF default

Router identifier 5.5.5.5, local AS number 65000

BGP routing table entry for IPv4 prefix 12.0.0.0/24, Route Distinguisher: 65100:2

Paths: 1 available

65100 65101

11.11.11.11 from 1.1.1.1 (1.1.1.1)

Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

Extended Community: Route-Target-AS:65000:1

Remote MPLS label: 118012

BGP routing table entry for IPv4 prefix 12.0.0.0/24, Route Distinguisher: 65100:3

Paths: 1 available

65100 65101

11.11.11.11 from 1.1.1.1 (1.1.1.1)

Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

Extended Community: Route-Target-AS:65000:4364

Remote MPLS label: 116707

2. In the VRF routing table, there is ECMP to this destination. Because at PE, the ECMP entry is by ASBR NH + remote ASBR Label. In this case, 11.11.11.11:11607 and 11.11.11.11:118012

PE1#show ip route vrf cust_1 12.0.0.0/24

VRF: cust_1

......

B I 12.0.0.0/24 [200/0] via 11.11.11.11/32, LDP tunnel index 1, label 116707

via 1.0.0.8, Ethernet2/1, label 100000

via 11.11.11.11/32, LDP tunnel index 1, label 118012

via 1.0.0.8, Ethernet2/1, label 100000

3. Hw routing table shows the ECMP index

PE1#show ip hardware ale vrf | egrep 'cust_1|VRF'

VRF Name VRF ID Table ID

cust_1 15 65535

PE1#show platform jericho ip route 12.0.0.0/24

Tunnel Type: M(mpls), G(gre), MoG(mpls-over-gre),

vxlan-o(vxlan outer-rewrite info), vxlan-i(vxlan inner-rewrite info)

CW - Control word

FL - Flow label

* - Routes in LEM

D - ECMP is divergent across switching chips

---------------------------------------------------------------------------------------------------------

| Routing Table | |

|---------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------

|15 |12.0.0.0/24 |ROUTE| FEC 32831 |0 | - | |300 | D |M 116707

|15 |12.0.0.0/24 |ROUTE| FEC 32831 |0 | - | |300 | D |M 118012

4. But this behavior could exhaust the ECMP resource. From the below output, the 1600 vpnv4 prefixes use up 875 of 4096 ECMP entries.

PE1#show hardware capacity | grep -i ECMP

ECMP 875 21% 3220 0 4095 875

PE1#show bgp vpn-ipv4 summary

BGP summary information for VRF default

Router identifier 5.5.5.5, local AS number 65000

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

1.1.1.1 4 65000 1693 1673 0 0 23:25:15 Estab 1602 1602

5. So there is a workaround to disable fib ecmp and lower the ECMP usage.

PE1#conf term

PE1(config)#no ip hardware fib hierarchical next-hop disabled << default config, but have to flip

PE1(config)#router general

PE1(config-router-general)#rib fib fec ecmp emulated

PE1-lp402.17:32:38(config-router-general)#show hardware capacity | grep -i ECMP

ECMP 0 0% 4095 0 4095 875

ECMP Mpls 0 0% 4095 0 4095 0

ECMP Routing 0 0% 4095 0 4095 875

ECMP VxlanOverlay 0 0% 4095 0 4095 0

ECMP VxlanTunnel 0 0% 3891 0 3891 0

Now no ECMP anymore in the sw/hw routing table.

PE1#show ip route vrf cust_1 12.0.0.0/24

VRF: cust_1

B I 12.0.0.0/24 [200/0] via 11.11.11.11/32, LDP tunnel index 1, label 116707

via 1.0.0.8, Ethernet2/1, label 100000

via 11.11.11.11/32, LDP tunnel index 1, label 118012, backup

via 1.0.0.8, Ethernet2/1, label 100000

PE1#show platform jericho ip route 12.0.0.0/24

---------------------------------------------------------------------------------------------------------

| Routing Table | |

|---------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------

|15 |12.0.0.0/24 |ROUTE| FEC 32830 |0 | - | | - |49624 |M 116707

6. You probably want to take one step further to ask why the ECMP. Both PE2/PE3 are Arista EOS device which allocates labels per VRF. In this setup, only 8 VRFs but why the PE sees 800+ labels.

Now let's check ASBR1, which receives 800 from 2 ASBRs as expected

ASBR1#sh bgp vpn-ipv4 summary

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 65000

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

....

192.158.115.11 4 65100 1568 1819 0 0 1d00h Estab 800 800

192.168.115.11 4 65100 1749 1827 0 0 1d00h Estab 802 802

But different number of unique labels

ASBR1#sh bgp neighbors 192.168.115.11 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l

ASBR1#sh bgp neighbors 192.158.115.11 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l

800

Neighbor 192.158.115.11 which is Cisco XR device, sends 800 vpnv4 prefixes with per-prefix labels! Even it receives per-VRF labels, it still re-assign per-prefix labels.

RP/0/RSP1/CPU0:ASBR3#show bgp vpnv4 unicast rd 65100:101 12.0.0.0/24 detail | inc bel

Sat Apr 25 16:55:59.312 UTC

Local Label: 16694 (with rewrite);

Received Label 116390

RP/0/RSP1/CPU0:ASBR3#show bgp vpnv4 unicast rd 65100:101 12.0.1.0/24 detail | inc Label

Sat Apr 25 16:56:02.058 UTC

Local Label: 16695 (with rewrite);

Received Label 116390

4/24/2020

Arista Sand-based System - MPLS Label Usage

If you see the following error message on Arista Jericho-based system associated with MPLS traffic loss, most likely the MPLS label hardware resource is exhausted.

Feb 4 15:12:32 ASBR SandL3Unicast: %SAND-3-ROUTING_MPLS_TUNNEL_RESOURCE_FULL: Hardware resources are insufficient to program all mpls tunnel entries

Look deeply into a problematic prefix:

ASBR#sh bgp vpn-ipv4 101.1.1.0/24
BGP routing table information for VRF default
Router identifier 172.16.0.55, local AS number 8073
BGP routing table entry for IPv4 prefix 101.1.1.0/24, Route Distinguisher: 65000:1
Paths: 2 available
65000 65101
192.168.66.7 from 192.168.0.7 (192.168.0.7)
Origin IGP, metric -, localpref 100, weight 0, valid, external, best
Extended Community: Route-Target-AS:65000:1
Remote MPLS label: 116396
Local MPLS label (allocated for received VPN routes): 186827
...

The local label for this vpnv4 prefix is 186827, but hw programming is DROP

ASBR#sh platform fap mpls route | egrep '186827|Dest'
| Label | Action | Destination | VID |Outlif | MAC / CPU Code |Index| Index| Action
|186827 |Forward | DROP |0 | - | |16434|358104| -
|186827 |Forward | DROP |0 | - | |16434|358105| -

Check MPLS route table, and it shows totally 93206 labels used

ASBR#sh mpls lfib route | grep "B3" | wc -l
93206

And check hardware Check MPLS route table, and it shows EEDB/MplsTunnel usage is almost 99%

ASBR#sh hardware capacity | egrep -i 'chip|mplstunn'
Table Feature Chip Used Used Free Committed Best Case High
EEDB MplsTunnel Jericho0 93206 99% 1002 4096 98304 93206

Arista EOS default label mode is per-vrf, so this high # of label is probably caused by other vendors. Now let's check how to check which neighbor sending most labels (this is another system running w/o any issue)

ASBR1#sh bgp vpn-ipv4 summary
BGP summary information for VRF default
Router identifier 1.1.1.1, local AS number 65000
Neighbor Status Codes: m - Under maintenance
Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
5.5.5.5 4 65000 421 405 0 0 05:20:01 Estab 801 801
192.158.115.11 4 8073 381 439 0 0 05:30:45 Estab 800 800
192.168.115.11 4 8073 410 439 0 0 05:29:12 Estab 802 802

Use the following command, you can see neighbor 5.5.5.5 totally sending 8 labels.

ASBR1#show bgp neighbors 5.5.5.5 vpn-ipv4 received-routes detail | grep Remote | awk '{print $4}' | sort | uniq | wc -l
8

How to know the maximum number of labels supported on the device? You can check it by running this command - "show hardware capacity" and check the line of "EEDB MplsTunnel":

7280CR#sh hardware capacity | egrep -i 'chip|eedb' | egrep -i 'chip|mpls'
Table Feature Chip Used Used Free Committed Best Case High
EEDB MplsTunnel Jericho0 0 0% 77824 0 81920 0

Here is a list of the Jericho-family models:
7280R/Jericho : 80K
7280R2/Jericho+: 96K

3/07/2020

Arista EOS MTU Behavior (3) - L3 Routed Ethernet Ports, IPv4

Now we switched to L3 routed ethernet (IPv4) ports:

MTU enforcement is for egress traffic
Hardware programs the MTU value (default value is 1500, in this case, 2000) and checks against forwarding packet length (in this case, it is ipv4 packet).
For IPv4 packets, if the length is more than MTU, the packets will be fragmented by CPU.
And the number of fragmented and dropped packets are reflected by CoPPEgressTrap counter in the output of "show cpu counters queue summary | nz"

Arista EOS MTU Behavior (2) - L2 Switched Ports

Below is a quick lab using 7280CR which is Jericho-based system,

Use EOS tool - Ethxmit on the generator to send packets

--ip-dst pointing to the destination address on the receiver device
--size specifies the packet size. Please the real ethernet packet size is this number minus 4 (I believe it is ethernet trailer). So 1504 means 1500-byte packet is received on sender ingress port

Run tcpdump + mirror on R3/receiver to check MTU behavior

Conclusion:

For switched traffic, no MTU enforcement, ingress nor egress

Some other observations:

For using ethxmit:

--size parameter for ethxmit includes the 4-byte CRC/ethernet trailer
If egress is access port, the ethernet frame size = L3 MTU + 18-byte including 14-byte ethernet header and 4-byte CRC.
If egress port is trunk, ethernet frame size = L3 MTU + 22-byte including 14-byte ethernet, 4-byte vlan and 4-byte CRC

No matter the mtu size under L2 interfaces, the forwarding MTU is set to maximum value 10178 (even show interface says Ethernet MTU, but it is not actual ethernet frame size, but forwarding MTU before ethernet header). Look at the ip header total length value in tcpdump output.

Trident-based system would use 9214.

R2#sh int e47/1 | grep -i mtu
Ethernet MTU 10178 bytes , BW 10000000 kbit
R2#sh run int e47/1
interface Ethernet47/1
mtu 100
switchport access vlan 100

3/03/2020

Arista EOS MTU Behavior (1)

Summary:

In this series of blogs, I will cover the MTU (Maximum Transmission Unit) behavior on Arista Jericho-based routers. Here is the summary:

1. MTU enforcement happens on the egress interfaces.

So MTU is not MRU (maximum receive unit), large size packets are NOT dropped at ingress interfaces

2. Arista MTU is layer-3 MTU on Jericho chipset:

2.1 The MTU enforcement is only for routed packets, no action on switched traffic
2.2 The actual packet size is MTU size plus L2 header size, which is 14-byte for L3 routed interface or 18-byte for the trunk interface

3. IPv4 packets are fragmented by router CPU and capped by CoPP

Reference:

2/18/2020

MTU value in EOS interop with Cisco XR

Summary:

Arista mtu value under interface is IP MTU value, while Cisco XR's mtu is L2 MTU
So when ISIS peering fails to come up if both sides have same MTU value
XR's mtu = EOS's mtu + 14 (Ethernet header length)

https://eos.arista.com/forum/isis-between-arista-and-ios-xr/

1/29/2020

Tunnel RIB related features

Self-Defined Tunnel Rib TOI (4.22.1F)
Pseudowire Resolution Over User-defined Tunnel RIBs (4.22.1F)

May provide a sample later on how to use these 2 features.

1/17/2020

Arista BFD related TOIs

4.23*

HW BFD

https://eos.arista.com/eos-4-23-1f/hardware-accelerated-bfd-transmit/

4.22*

BFD C-bit

https://eos.arista.com/eos-4-22-1f/bfd-control-plane-independent-interoperability/

BFD Authentication

https://eos.arista.com/eos-4-22-0f/bfd-authentication/

4.20*

BFD SSO:

https://eos.arista.com/eos-4-18-2f/bfd-sso/
4.18.2F, starting with Arad Modular
4.20.5: Add Jericho

BFD for static routes

4.17*

BFD Enhancement:

https://eos.arista.com/eos-4-17-0f/bfd-enchancements/
BFD under VRF

OSPFv3 BFD:

https://eos.arista.com/eos-4-17-0f/ospfv3-bfd

4.15*

ISIS BFD

https://eos.arista.com/eos-4-15-2f/isis-bfd

BFD NSX Monitoring:

https://eos.arista.com/eos-4-15-0f/nsx-sn-health-monitoring/

12/28/2019

%SAND-6-VXLAN_ROUTE_IGNORED error message

If you see the syslog error message - %SAND-6-VXLAN_ROUTE_IGNORED like below (on arista 7280E/R series) , that's because your hardware TCAM profile is not correct.

Dec 28 16:02:16 eVtep2 SandL3Unicast: %SAND-6-VXLAN_ROUTE_IGNORED: VXLAN routes cannot be programmed in the data path with the current TCAM profile

Just change the configuration like:

hardware tcam profile vxlan-routing

11/21/2019

A dummy error - low MTU vs ipv6

After configuring IPv6 under an interface, see the ipv6 neighbor or host resolution doesn't work.

Router1.10:36:22(config-if-Po5)#sh ipv6 interface po5
Port-Channel5 is up, line protocol is up (connected)
IPv6 is stalled, link-local is unassigned
Global unicast address(es):
2000:170:1:170::1, subnet is 2000:170:1:170::/64 [INACTIVE]
No joined group addresses
ND DAD status is unavailable
ND Reachable time is 30000 milliseconds
ND retransmit interval is 1000 milliseconds

The reason is that, I have a line of a left-over configuration of smaller-than-normal mtu size - 1200. The lowest MTU size of IPv6 is 1280.

interface Port-Channel5
description Cdn_Test_upLink
mtu 1200
no switchport
vrf IxCdn
ip address 170.1.170.1/24
ipv6 nd cache expire 60
ipv6 enable
ipv6 address 2000:170:1:170::1/64
ipv6 nd ra disabled all
no mpls ip
no qos trust
spanning-tree bpdufilter enable

After removing this line, everything is fine.

Port-Channel5 is up, line protocol is up (connected)
IPv6 is enabled, link-local is fe80::464c:a8ff:fea5:1140/64
Global unicast address(es):
2000:170:1:170::1, subnet is 2000:170:1:170::/64
Joined group address(es):
ff02::1
ff02::1:ff00:1
ff02::1:ffa5:1140

11/05/2019

Arista EOS Segment Routing (3) - SR Routing #1, Static Route + NHG

The Arista MPLS Segment Routing White Paper(Page 8) lists 4 SR routing solutions:
1) Static label push via NHG
2) Controller via EOS SDK
3) BGP LU with SR
4) SR-TE

This blog gives an example of solution #1.

The configuration is straightforward:

Instead of running a routing protocol between PE1 and PE2, a static route of remote destination (100.255.4.1/32 on PE1) is configured and pointing to NHG.
The NHG defines:

MPLS encapsulation
Push a label of 900004
The NH is 10.1.2.2 for out-interface and destination MAC.
Obviously, 2 NHGs are needed to 2 way traffic.

Not like the regular NHG setup, decap group is NOT needed.
P1 and P2 have no idea of destination ip - 100.255.4.1, but use label to forward traffic

PE1#sh ip route 100.255.4.1

VRF: default

Codes: C - connected, S - static, K - kernel,

O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,

E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,

N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,

R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,

O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,

NG - Nexthop Group Static Route, V - VXLAN Control Service,

DH - DHCP client installed default route, M - Martian,

DP - Dynamic Policy Route, L - VRF Leaked

NG 100.255.4.1/32 [1/0] via sr-1-push-label-900004, Nexthop Group ID 1

The output of tcpdump on P2's et42/4 (the interface facing to P1) shows MPLS with the correct label in both directions.

23:10:17.713412 44:4c:a8:97:84:5f > 44:4c:a8:97:8c:51, ethertype MPLS unicast (0x8847), length 118: MPLS (label 900001, exp 0, [S], ttl 63)

(tos 0x0, ttl 64, id 6381, offset 0, flags [none], proto ICMP (1), length 100)

100.255.4.1 > 100.255.1.1: ICMP echo request, id 9258, seq 5, length 80

23:10:17.713473 44:4c:a8:97:8c:51 > 44:4c:a8:97:84:5f, ethertype MPLS unicast (0x8847), length 118: MPLS (label 900004, exp 0, [S], ttl 63)

(tos 0x0, ttl 64, id 2480, offset 0, flags [none], proto ICMP (1), length 100)

100.255.1.1 > 100.255.4.1: ICMP echo reply, id 9258, seq 5, length 80

Now you have the simplest, but completed ISIS segment-routing solution, control plane + data plane :-) In the real network, you can complete the puzzle by adding,

Have a controller participate in the ISIS domain via the ISIS over GRE tunnel.
As an ISIS neighbor, it can fetch the full ISIS LSDB to have a global view of the network including the segment.
Then it can program the edge router vi CLI/capi to steer the traffic. Or use the EOS SDK for faster program speed (this is the solution #2)

Arista EOS Segment Routing (2) - MPLS Ping to verify SR data plane

With the same topology, to verify the SR data plane, we can use the command - "ping mpls" to have PE1 send a UDP packet with the correct label.

PE1#ping mpls segment-routing ip 4.4.4.4/32 repeat 1
LSP ping to Segment-Routing route 4.4.4.4/32
timeout is 5000ms, interval is 1000ms
Via 10.1.2.2, Ethernet2/4, label stack (top label first): [900004]
Reply from 10.3.4.4: seq=1, time=0.53ms, success: egress ok

--- Segment-Routing target fec 4.4.4.4/32 : lspping statistics ---
Via 10.1.2.2, Ethernet2/4, label stack (top label first): [900004]
1 packets transmitted, 1 received, 0% packet loss, time 150ms
1 received from 10.3.4.4, rtt min/max/avg 0.530/0.530/0.530 ms

Turning on tcpdump on P1, it shows the following packets:

P1#sh run | grep moni
monitor session sr source Ethernet54/4
monitor session sr destination Cpu

P1#bash tcpdump -nvvvi mirror1 udp or mpls
tcpdump: listening on mirror1, link-type EN10MB (Ethernet), capture size 262144 bytes
13:48:15.934459 44:4c:a8:97:77:21 > 44:4c:a8:97:8c:51, ethertype MPLS unicast (0x8847), length 98: MPLS (label 900004, exp 0, [S], ttl 255)
(tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 80, options (RA))
10.1.2.1.36260 > 127.0.0.1.lsp-ping:
LSP-PINGv1, msg-type: MPLS Echo Request (1), length: 48
reply-mode: Reply via an IPv4/IPv6 UDP packet (2)
Return Code: No return code or return code contained in the Error Code TLV (0)
Return Subcode: (0)
Sender Handle: 0x00000000, Sequence: 1
Sender Timestamp: -16:-39:-45.54962696 Receiver Timestamp: no timestamp
Target FEC Stack TLV (1), length: 12
Unknown subTLV (31744), length: 6
0x0000: 0404 0404 2002
0x0000: 7c00 0006 0404 0404 2002 0000
13:48:15.934737 44:4c:a8:97:8c:51 > 44:4c:a8:97:77:21, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 36448, offset 0, flags [DF], proto UDP (17), length 60)
10.3.4.4.lsp-ping > 10.1.2.1.36260:
LSP-PINGv1, msg-type: MPLS Echo Reply (2), length: 32
reply-mode: Reply via an IPv4/IPv6 UDP packet (2)
Return Code: Replying router is an egress for the FEC at stack depth 1 (3)
Return Subcode: (1)
Sender Handle: 0x00000000, Sequence: 1
Sender Timestamp: -16:-39:-45.54962696 Receiver Timestamp: -16:-40:-1.401478080

From the above packet capture,

The icmp echo request is encap'ed in MPLS packet with label 90004 as expected.
The dest address of inner ip is 127.0.0.1. With destination address on network 127, the packet can't be routed if LSP is broken before the final destination and the packet is decap'ed prematurely.
The icmp echo reply is just a native ipv4 packet.

Arista EOS Segment Routing (1) - Simplest Setup/Configuration

Here is the simplest SR setup,
1) Only IPv4, only ipv4 node-segment;
2) No IPv6, no ISIS multi-topology

From the above configuration, we can observe:

The basic ISIS-SR configuration is quite simple:

enable "segment-routing mpls" under router isis to attach SR info to ISIS LSA;
Specify the loopback0 interface with a globally unique node-segment index.
ISIS-SR calculates the node-segment label = mpls label range base + index, so the routers in whole domain use the same label. This is an important characteristic of SR, for anycast.

To verify the SR setup, you can use the following CLI commands:

show mpls label ranges
show isis segment-routing
show isis database PE2.00-00 detail !PE2.00-00 is id
show mpls lfib route
show mpls segment-routing bindings
show platform jericho mpls route

Detailed output of PE1:

PE1#show mpls label ranges

Start End Size Usage

------------------------------------------------

0 15 16 reserved

16 99999 99984 static mpls

100000 116383 16384 ldp (dynamic)

116384 132767 16384 pseudowire (dynamic)

132768 149151 16384 bgp (dynamic)

149152 165535 16384 isis (dynamic)

165536 362143 196608 free (dynamic)

362144 899999 537856 unassigned

900000 965535 65536 isis-sr <<< default ISIS-SR range

900000 965535 65536 bgp-sr

965536 1031071 65536 srlb

1031072 1036287 5216 unassigned

1036288 1048575 12288 l2evpn

PE1#show isis segment-routing

System ID: PE1 Instance: sr

SR supported Data-plane: MPLS SR Router ID: 1.1.1.1

SR Global Block( SRGB ): Base: 900000 Size: 65536

Adj-SID allocation mode: SR-adjacencies

Adj-SID allocation pool: Base: 149152 Size: 16384

All Prefix Segments have : P:0 E:0 V:0 L:0

IS-IS Reachability Algorithm : SPF (0)

Number of IS-IS segment routing capable peers: 3

Self-Originated Segment Statistics:

Node-Segments : 1

Prefix-Segments : 0

Proxy-Node-Segments : 0

Adjacency Segments : 1

PE1#show isis database PE2.00-00 detail

IS-IS Instance: sr VRF: default

IS-IS Level 2 Link State Database

LSPID Seq Num Cksum Life IS Flags

PE2.00-00 12 57553 1049 L2 <>

NLPID: 0xCC(IPv4)

Hostname: PE2

Area address: 49.0001

Interface address: 4.4.4.4

Interface address: 10.3.4.4

IS Neighbor : P2.11 Metric: 10

LAN-Adj-sid: 100000 flags: [ L V ] weight: 0 system ID: 0000.0000.3333

Reachability : 4.4.4.4/32 Metric: 10 Type: 1 Up

SR Prefix-SID: 4 Flags: [ N ] Algorithm: 0

Reachability : 10.3.4.0/24 Metric: 10 Type: 1 Up

Router Capabilities: Router Id: 4.4.4.4 Flags: [ ]

SR Local Block:

SRLB Base: 965536 Range: 65536

Area leader priority: 250 algorithm: 0

SR Capability: Flags: [ I ]

SRGB Base: 900000 Range: 65536

PE1#show mpls lfib route

MPLS forwarding table (Label [metric] Vias) - 4 routes

MPLS next-hop resolution allow default route: False

Via Type Codes:

M - MPLS via, P - Pseudowire via,

I - IP lookup via, V - VLAN via,

VA - EVPN VLAN aware via, ES - EVPN ethernet segment via,

VF - EVPN VLAN flood via, AF - EVPN VLAN aware flood via,

NG - Nexthop group via

Source Codes:

G - gRIBI, S - Static MPLS route,

B2 - BGP L2 EVPN, B3 - BGP L3 VPN,

R - RSVP, LP - LDP pseudowire,

L - LDP, M - MLDP,

IP - IS-IS SR prefix segment, IA - IS-IS SR adjacency segment,

IL - IS-IS SR segment to LDP, LI - LDP to IS-IS SR segment,

BL - BGP LU, ST - SR TE policy,

DE - Debug LFIB

IA 149152 [1]

via M, 10.1.2.2, pop

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900002 [1], 2.2.2.2/32

via M, 10.1.2.2, pop

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900003 [1], 3.3.3.3/32

via M, 10.1.2.2, forward

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900004 [1], 4.4.4.4/32

via M, 10.1.2.2, forward

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

PE1#show mpls segment-routing bindings

1.1.1.1/32

Local binding: Label: imp-null

Remote binding: Peer ID: 0000.0000.2222, Label: 900001

2.2.2.2/32

Local binding: Label: 900002

Remote binding: Peer ID: 0000.0000.2222, Label: imp-null

3.3.3.3/32

Local binding: Label: 900003

Remote binding: Peer ID: 0000.0000.2222, Label: 900003

4.4.4.4/32

Local binding: Label: 900004

Remote binding: Peer ID: 0000.0000.2222, Label: 900004

PE1#show platform jericho mpls route

D - ECMP is divergent across switching chips

---------------------------------------------------------------------------------------------

| Mpls Table |

|---------------------------------------------------------------------------------------------|

|---------------------------------------------------------------------------------------------|

|149152 |Et2/4 |1027 | 44:4c:a8:97:8c:51 | PopE pipe auto|32773 |8200 |8 | 0 |

Egress Action Codes:

M - Mpls Tunnel, G - GRE Tunnel, MoG - Mpls-over-GRE Tunnel

ECMP Codes:

D - ECMP is divergent across switching chips

-----------------------------------------------------------------------------------------------

| Mpls Table

|-----------------------------------------------------------------------------------------------

|-----------------------------------------------------------------------------------------------

|900002 |Pop | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -

|900003 |Forward | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -

|900004 |Forward | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -