Arista EOS/Networking Tips by Solomon Yang: EVPN

Showing posts with label EVPN. Show all posts

2/21/2022

EVPN VxLAN Centralized Routing: redistribute VARP MAC to prevent unnecessary flooded traffic

https://www.arista.com/en/um-eos/eos-evpn-vxlan-single-gateway-centralized-routing

In the below EVPN VxLAN Centralized Routing topology

There are 2 centralized routers, both have default gateway SVI - Vlan100 (100.1.0.1/24) with a virtual MAC (00dc.0000.0001)

interface Vlan100

mtu 9214

ip address virtual 100.1.0.1/24

ip virtual-router mac-address 00:dc:00:00:00:01

The hostA (100.1.0.201) pings the default GW address and receives duplicated ICMP response like below:

HostA.10:07:37#ping vrf Vlan100 100.1.0.1

PING 100.1.0.1 (100.1.0.1) 72(100) bytes of data.

80 bytes from 100.1.0.1: icmp_seq=1 ttl=64 time=0.181 ms

80 bytes from 100.1.0.1: icmp_seq=1 ttl=64 time=0.318 ms (DUP!)

80 bytes from 100.1.0.1: icmp_seq=1 ttl=64 time=0.345 ms (DUP!)

80 bytes from 100.1.0.1: icmp_seq=2 ttl=64 time=0.103 ms

80 bytes from 100.1.0.1: icmp_seq=2 ttl=64 time=0.119 ms (DUP!)

80 bytes from 100.1.0.1: icmp_seq=2 ttl=64 time=0.135 ms (DUP!)

80 bytes from 100.1.0.1: icmp_seq=2 ttl=64 time=0.149 ms (DUP!)

And the tcpdump shows the same, duplicated replies from both gateways:

[admin@HostA ~]$ sudo ip netns exec ns-Vlan100 tcpdump -nvvi vlan100

10:07:46.165380 44:4c:a8:80:c1:c8 > 00:dc:00:00:00:01, ethertype IPv4 (0x0800), length 114: (tos 0x0, ttl 64, id 20298, offset 0, flags [none], proto ICMP (1), length 100) <<< req to VARP MAC 00dc.0000.0001

100.1.0.201 > 100.1.0.1: ICMP echo request, id 6062, seq 2, length 80

10:07:46.165446 44:4c:a8:a5:11:41 > 44:4c:a8:80:c1:c8, ethertype IPv4 (0x0800), length 114: (tos 0x0, ttl 64, id 8443, offset 0, flags [none], proto ICMP (1), length 100) <<< reply1 from 444c.a8a5.1141

100.1.0.1 > 100.1.0.201: ICMP echo reply, id 6062, seq 2, length 80

10:07:46.165467 44:4c:a8:a5:11:40 > 44:4c:a8:80:c1:c8, ethertype IPv4 (0x0800), length 114: (tos 0x0, ttl 64, id 60002, offset 0, flags [none], proto ICMP (1), length 100) <<< reply2 from 444c.a8a5.1140

100.1.0.1 > 100.1.0.201: ICMP echo reply, id 6062, seq 2, length 80

Why does it happen? It is because the L2 VTEP doesn't learn the MAC address

L2VTEPa.10:07:17#sh mac address-table vlan 100

Mac Address Table

------------------------------------------------------------------

Vlan Mac Address Type Ports Moves Last Move

---- ----------- ---- ----- ----- ---------

100 444c.a86c.6dd9 DYNAMIC Vx1 1 0:00:49 ago

100 444c.a87c.d809 DYNAMIC Vx1 1 0:34:40 ago

100 444c.a880.c1c8 DYNAMIC Po1 1 0:05:44 ago

Thus, no type-2 update for 00dc.0000.0001

L2VTEPa.10:07:22#show bgp evpn route-type mac-ip 00dc.0000.0001

BGP routing table information for VRF default

Router identifier 10.0.0.11, local AS number 65011

Route status codes: * - valid, > - active, S - Stale, E - ECMP head, e - ECMP

c - Contributing to ECMP, % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric LocPref Weight Path

L2VTEPa.10:09:42#

Per https://www.arista.com/en/um-eos/eos-evpn-vxlan-single-gateway-centralized-routing, CLI - "redistribute router-mac next-hop vtep primary" to generate type-2 udpate for VARP MAC.

router bgp 65017

vlan 100

rd 10.0.0.17:100

route-target both 100:100

redistribute learned

redistribute router-mac next-hop vtep primary

Now the pings are not dup'ed anymore.

HostA.10:07:46#ping vrf Vlan100 100.1.0.1

PING 100.1.0.1 (100.1.0.1) 72(100) bytes of data.

80 bytes from 100.1.0.1: icmp_seq=1 ttl=64 time=0.208 ms

80 bytes from 100.1.0.1: icmp_seq=2 ttl=64 time=0.155 ms

80 bytes from 100.1.0.1: icmp_seq=3 ttl=64 time=0.111 ms

80 bytes from 100.1.0.1: icmp_seq=4 ttl=64 time=0.093 ms

80 bytes from 100.1.0.1: icmp_seq=5 ttl=64 time=0.160 ms

L2VTEPa.10:19:14#sh mac address-table vlan 100

Mac Address Table

------------------------------------------------------------------

Vlan Mac Address Type Ports Moves Last Move

---- ----------- ---- ----- ----- ---------

100 00dc.0000.0001 STATIC Vx1

L2VTEPa#show bgp evpn route-type mac-ip 00dc.0000.0001

BGP routing table information for VRF default

Router identifier 10.0.0.11, local AS number 65011

Route status codes: * - valid, > - active, S - Stale, E - ECMP head, e - ECMP

c - Contributing to ECMP, % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric LocPref Weight Path

* >Ec RD: 10.0.0.17:100 mac-ip 00dc.0000.0001

10.0.0.17 - 100 0 65001 65017 i

* ec RD: 10.0.0.17:100 mac-ip 00dc.0000.0001

10.0.0.17 - 100 0 65001 65017 i

* ec RD: 10.0.0.17:100 mac-ip 00dc.0000.0001

10.0.0.17 - 100 0 65001 65017 i

* ec RD: 10.0.0.17:100 mac-ip 00dc.0000.0001

10.0.0.17 - 100 0 65001 65017 i

* >Ec RD: 10.0.0.18:100 mac-ip 00dc.0000.0001

10.0.0.18 - 100 0 65001 65018 i

* ec RD: 10.0.0.18:100 mac-ip 00dc.0000.0001

10.0.0.18 - 100 0 65001 65018 i

* ec RD: 10.0.0.18:100 mac-ip 00dc.0000.0001

10.0.0.18 - 100 0 65001 65018 i

* ec RD: 10.0.0.18:100 mac-ip 00dc.0000.0001

10.0.0.18 - 100 0 65001 65018 i

6/13/2021

MPLS Hashing: Control Word and Entropy Label

A good presentation on this topic: https://ripe66.ripe.net/presentations/141-RIPE66_Understanding_MPLS_Hashing-Snijders.pdf

MPLS Lable doesn't have encapsulated protocol information. So after the bottom label, 2 possibilities:

IPv4/v6 packets if L3VPN, the 1st 4-bit is ipVersion, 4=ipv4, 6=ipv6
Ethernet frames if L2VPN, the 1st 48-bit is dstMAC

Along the LSP, if P routers like to do ECMP by hashing, it has to guess like above to do hashing the 5-tuple fields (src/dstAddr, src/dstPort, protoNum)
So the problem happens if the dstMac starts with 4 or 6

The solution for the above issue is MPLS control word (RFC 4385, Feb 2006), a 4-byte all-zero (same length as a label) control word after the bottom label.

Arista EOS 4.21.3F: Configurable L2 EVPN MPLS control word

mpls evpn; label-stack control-word

Arista EOS 4.22.1F: Control Word support on LDP Pseudowire

mpls ldp; pseudowire <name>; control-word

So the MPLS control word is to prevent incorrect hashing. How does the LSR do the load-balancing? The answer is Entropy Label (RFC 6790, Nov 2012)

Increase the label stack by 2: 1 for ELI (entropy label indicator = 7), 1 for EL (entropy label)
Aristea EOS 4.26.0F: LDP Entropy Label Support

4.25.2F: LSR support
4.26.0F: LER support
mpls ldp; entropy-label

This feature has a limitation: PHP LSR can't pop ELI + EL labels, so has to rely on LER to pop up [ELI, ELI, VPN] labels.

Another solution is FAT - Flow Aware Transport on LDP pseudowires (4.24.0F) based on RFC 6391

mpls ldp; pseudowire; pseudowire <name>; label flow

5/27/2021

Troubleshooting Arista EOS EVPN VPWS (1)

This blog provides troubleshooting information for common problems with Arista EVPN MPLS VPWS. It covers the following error:

Status: CLI conflict
Status: Admin down
Status: Interface unavailable
Status: No remote
Status: Unprogrammed local connector

Most of the issues start with down state in the output of "show patch panel" like below:

RR1-PE5-wa465.11:42:56#sh patch panel
Patch Connector Status
---------------------------- ------------------------------------------------------- ------
pbVpws-PE5:e471--PE1:e1312 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312 Down
2: Ethernet47/1

And command - "show patch panel <name> details" shows the detailed error reason:

RR1-PE5-wa465.14:58:02#sh patch panel pbVpws-PE5:e471--PE1:e1312 detail

......

Patch: pbVpws-PE5:e471--PE1:e1312, Status: Down

Connector 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312

Status: Interface unavailable

Local MPLS label: 100004

EVPN VPWS type: port-based

Connector 2: Ethernet47/1

Status: Interface mode

The following error status means a local misconfiguration most likely.

Ethernet side:

Status: CLI conflict

Possible reason: duplicated patch panel entries for the same ethernet interface

Status: Admin down

Check the interface status, to make sure is up

Pseudowire side:

Status: Interface unavailable

Possible reason: make sure "no switchport" under interface

After the local configuration is corrected, check the output of "show bgp evpn route-type auto-dis next-hop 0.0.0.0". This is to check if the locally originated BGP EVPN prefixes are advertised out to BGP peers.

RR1-PE5-wa465.14:58:14#show bgp evpn route-type auto-discovery next-hop 0.0.0.0

....

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric LocPref Weight Path

* > RD: 100.250.250.5:100 auto-discovery 50471 0000:0000:0000:0000:0000

- - - 0 i

RR1-PE5-wa465.15:07:10#show bgp evpn route-type auto-discovery next-hop 0.0.0.0 detail

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

BGP routing table entry for auto-discovery 50471 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.5:100

Paths: 1 available

Local

- from - (0.0.0.0)

Origin IGP, metric -, localpref -, weight 0, valid, local, best

Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

MPLS label: 100004

The most important information as highlighted above is the ETID/PW ID. And compare it with the definition of PW PE5:e471--PE1:e1312

RR1-PE5-wa465.15:06:29#show bgp evpn instance vpws custAB

...

Pseudowire PE5:e471--PE1:e1312

Status: up

VPWS label: 100004

Local VPWS ID: 50471

Remote VPWS ID: 101312

After this point, you are pretty sure this end of PW is good.

But if you are seeing the error status as "No remote", the device may have an issue with remote PW.

Patch: pbVpws-PE5:e471--PE1:e1312, Status: Down

Connector 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312

Status: No remote

Local MPLS label: 100004

EVPN VPWS type: port-based

Connector 2: Ethernet47/1

Status: Unprogrammed local connector

Now let's check the router knows how to reach PW with VPWS ID 101312.

RR1-PE5-wa465.15:13:06#show bgp evpn route-type auto-discovery | grep 101312

RR1-PE5-wa465.15:13:10#

Clearly, the router has no EVPN AD prefix with ETID 101312, so it doesn't how to reach. The next step is to log in to the other end of PW to figure out why by following the above steps. For example, the remote PW should be from NH 100.250.250.11

PE3-lp232.22:00:03#sh bgp evpn route-type auto-discovery next-hop 100.250.250.11

...

Network Next Hop Metric LocPref Weight Path

* >Ec RD: 100.250.250.11:100 auto-discovery 101331 0000:0000:0000:0000:0000

100.250.250.11 - 100 0 i Or-ID: 100.250.250.11 C-LST: 100.250.250.6

From the above output, the PE1(100.250.250.11) sends out an AD prefix with ETID 101331, not 101311.

And sometimes, you may face traffic issues even all the PW and patch panels are UP!! That's because the BGP EVPN AD prefixes only contain the local ETID/PW id, no the expected remote one. There is no way to ensure consistency.

Reference:

https://eos.arista.com/eos-4-24-2f/evpn-mpls-virtual-private-wire-service-vpws/

5/20/2021

"Pending BGP Convergence" and invalid on MPLS/SR/EVPN BGP RR

On a BGP EVPN RR, I am seeing some EVPN routes are marked as "Pending BGP Convergence" as below:

RR1.12:01:04#sh bgp evpn
BGP routing table information for VRF default
Router identifier 100.250.250.5, local AS number 65000
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
Network Next Hop Metric LocPref Weight Path
% RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000
100.250.250.11 - 100 0 i
% RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000
100.250.250.12 - 100 0 i

The reason is "invalid" in the output of show bgp evpn detail

RR1.12:10:27#sh bgp evpn route-type auto-discovery detail

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

BGP routing table entry for auto-discovery 110100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.11:100

Paths: 1 available

Local (Received from a RR-client)

100.250.250.11 from 100.250.250.11 (100.250.250.11)

Origin IGP, metric -, localpref 100, weight 0, invalid, internal, pending resolution

Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

BGP routing table entry for auto-discovery 120100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.12:100

Paths: 1 available

Local (Received from a RR-client)

100.250.250.12 from 100.250.250.12 (100.250.250.12)

Origin IGP, metric -, localpref 100, weight 0, invalid, internal, pending resolution

Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

Here is the BGP configuration:

RR1.11:59:52#sh run sec router bgp

router bgp 65000

router-id 100.250.250.5

no bgp default ipv4-unicast

neighbor RRC peer group

neighbor RRC remote-as 65000

neighbor RRC update-source Loopback0

neighbor RRC route-reflector-client

neighbor RRC send-community extended

neighbor 100.250.250.11 peer group RRC

neighbor 100.250.250.12 peer group RRC

neighbor 100.250.250.21 peer group RRC

neighbor 100.250.250.22 peer group RRC

address-family evpn

neighbor RRC activate

Add "neighbor default encapsulation mpls next-hop-self source-interface Loopback0" under EVPN AF

RR1.12:20:00(config)#router bgp 65000

RR1.12:20:02(config-router-bgp)#address-family evpn

RR1.12:20:04(config-router-bgp-af)#neighbor default encapsulation mpls next-hop-self source-interface Loopback0

One prefix shows "* >" - valid. One shows no code

RR1.12:20:10#sh bgp evpn

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP

S - Stale, c - Contributing to ECMP, b - backup

% - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric LocPref Weight Path

RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000

100.250.250.11 - 100 0 i

* > RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000

100.250.250.12 - 100 0 i

Details still shows "invalid"

RR1.12:22:09#sh bgp evpn route-type auto-discovery rd 100.250.250.11:100 detail

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

BGP routing table entry for auto-discovery 110100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.11:100

Paths: 1 available

Local (Received from a RR-client)

100.250.250.11 from 100.250.250.11 (100.250.250.11)

Origin IGP, metric -, localpref 100, weight 0, invalid, internal

Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

MPLS label: 132768

No tunnel endpoint in "show tunnel rib brief"

RR1.12:22:50#show tunnel rib brief | grep 250.250.1

100.250.250.1/32 IS-IS SR IPv4 6 65 115 20

100.250.250.12/32 IS-IS SR IPv4 4 65 115 40

The cause is the PE's config, missing node-segment ipv4 configuration

PEa1.12:10:13#sh run int lo0

interface Loopback0

ip address 100.250.250.11/32

ipv6 address 2000:100:250:250::11/128

node-segment ipv6 index 11

isis enable MplsSR

isis passive

PEa1.12:25:24#c

PEa1.12:26:08(config)#int lo0

PEa1.12:26:10(config-if-Lo0)#node-segment ipv4 index 11

On RR1, NH emerges in tunnel rib.

RR1.12:22:56#show tunnel rib brief | grep 250.250.1

100.250.250.1/32 IS-IS SR IPv4 6 65 115 20

100.250.250.11/32 IS-IS SR IPv4 2 65 115 30

100.250.250.12/32 IS-IS SR IPv4 4 65 115 40

And bgp table is good

RR1.12:24:51#sh bgp evpn

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP

S - Stale, c - Contributing to ECMP, b - backup

% - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric LocPref Weight Path

* > RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000

100.250.250.11 - 100 0 i

* > RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000

100.250.250.12 - 100 0 i

12/02/2020

Arista EOS: % - Pending BGP convergence when "show bgp evpn"

1. Router(RR) receives 4 bgp evpn prefixes from RRC

wa465-PD.P1-JHM.SR.14:35:52(config-router-bgp-af)#show bgp evpn summary
BGP summary information for VRF default
Router identifier 100.250.250.30, local AS number 65000
Neighbor Status Codes: m - Under maintenance
Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
100.250.250.11 4 65000 500 487 0 0 00:02:15 Estab 4 4
100.250.250.12 4 65000 504 488 0 0 00:02:15 Estab 4 4

2. But "show bgp evpn" complains about "Pending BGP convergence"

wa465-PD.P1-JHM.SR.14:36:48(config-router-bgp-af)#show bgp evpn

BGP routing table information for VRF default

Router identifier 100.250.250.30, local AS number 65000

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP

S - Stale, c - Contributing to ECMP, b - backup

% - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric LocPref Weight Path

% RD: 100.250.250.12:100 auto-discovery 0 0000:0000:0000:1111:0000

100.250.250.12 - 100 0 i

% RD: 100.250.250.12:1 auto-discovery 0000:0000:0000:1111:0000

100.250.250.12 - 100 0 i

3. Check the NH, it was resolved in the tunnel rib

wa465-PD.P1-JHM.SR.14:37:33(config-router-bgp-af)#show tunnel rib brief | grep 100.250.250.12

100.250.250.12/32 IS-IS SR IPv4 3 65 115 20

4. The cause is the mis-configuration under router bgp evpn address family. The EOS needs to know to resolve the NH by using the mpls NH tunnel rib.

wa465-PD.P1-JHM.SR.14:36:53(config-router-bgp-af)#router bgp 65000

wa465-PD.P1-JHM.SR.14:36:57(config-router-bgp)#add evpn

wa465-PD.P1-JHM.SR.14:37:00(config-router-bgp-af)#neighbor default encapsulation mpls next-hop-self source-interface loopback 0

wa465-PD.P1-JHM.SR.14:37:25(config-router-bgp-af)#show bgp evpn

BGP routing table information for VRF default

Router identifier 100.250.250.30, local AS number 65000

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP

S - Stale, c - Contributing to ECMP, b - backup

% - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric LocPref Weight Path

* > RD: 100.250.250.12:100 auto-discovery 0 0000:0000:0000:1111:0000

100.250.250.12 - 100 0 i

* > RD: 100.250.250.12:1 auto-discovery 0000:0000:0000:1111:0000

5/31/2020

Arista EVPN VXLAN Configuration Example (3c) - Single-homing, L3 EVPN, Symmetric IRB

One of the purposes of symmetric IRB is to address the scale issue of asymmetric IRB solution. And here is the list of differences compared with asymmetric IRB:

VTEPs only need to hold the VLANs and SVIs of directly connected subnets
An intermediate IP-VRF to carry the remote subnets

5/30/2020

Arista EVPN VXLAN Configuration Example (3b) - Single-homing, L3 EVPN, Asymmetric IRB

To solve the sub-optimal routing pattern in the solution of centralized routing, the IRB EVPN Draft proposes 2 solutions, asymmetric IRB and symmetric IRB. Because the local VTEP does both inter-VLAN routing and intra-VLAN switching, it is called IRB (Integrated Routing and Bridging).

The asymmetric IRB is illustrated as below

Explanations:

VTEP on has 1 directly connected VLAN:

VTEP1 - VLAN 641
VTEP2 - VLAN 642

But the VTEPs must have

2 x SVI interface, VLAN 641 and 642
2 x VLANs under MAC VRF
2 x VLAN/VNI bindings under Vxlan interfce

Routing is performed on the ingress VTEP, and egress VTEP only decapsulates the Vxlan header and switches into destination VLANs.
The returning traffic does the same, so routing is done on different VTEPs, hence the term Asymmetric IRB
Advantage:

Optimal traffic path and no traffic trombone

Disadvantage:

VTEPs must have all SVIs and VLANs configured, even not locally connected.
That means ALL VTEPs hold ALL MAC and ARP of hosts for source and destination VLANs.
So the scale is the biggest issue. To make things worse, TOR devices normally don't much high capacity.

From the below output, the VTEP1 has 6 ARP entries, 3 local VLANs and 3 remote VLANs

snp261-eVtep1.23:11:00#sh arp

Address Age (sec) Hardware Addr Interface

160.64.1.101 0:02:54 444c.a8a5.1140 Vlan641, Ethernet78

160.64.1.102 0:01:08 444c.a8a5.1140 Vlan641, Ethernet78

160.64.1.103 0:01:04 444c.a8a5.1140 Vlan641, Ethernet78

160.64.2.201 - 444c.a8a5.1141 Vlan642, Vxlan1

160.64.2.202 - 444c.a8a5.1141 Vlan642, Vxlan1

160.64.2.203 - 444c.a8a5.1141 Vlan642, Vxlan1

Arista EVPN VXLAN Configuration Example (3a) - Single-homing, L3 EVPN, Centralized Routing

In traditional DC design, the most common inter-VLAN routing is centralized routing, and illustrated as below,

Explanation:

A dedicated router - up506/cenRtr is used to route the traffic between vlans;
All gateway SVIs on the cenRtr
Advantages:

Low resource requirement on VTEPs, which only need to know how to reach gateway. So fewer MAC and no ARP
Easy managed.

Disadvantages:

Sub-optimal traffic flow.
Single point failure

Control Plane Check-up:

1. IMET:

On centralized router, under vlan 631, only 2 VTEPs - local and VTEP1

up506-CentRtr#show bgp evpn route-type imet vni 631
Network Next Hop Metric LocPref Weight Path
* > RD: 160.255.255.10:630 imet 631 160.255.255.10
160.255.255.10 - 100 0 i Or-ID: 160.255.255.10 C-LST: 180.255.255.1
* > RD: 160.255.255.20:630 imet 631 160.255.255.100
- - - 0 i

Similarly, under vlan 632, only 2 VTEPs - local and VTEP2

up506-CentRtr#show bgp evpn route-type imet vni 632
Network Next Hop Metric LocPref Weight Path
* > RD: 160.255.255.20:630 imet 632 160.255.255.20
160.255.255.20 - 100 0 i Or-ID: 160.255.255.20 C-LST: 180.255.255.1
* > RD: 160.255.255.20:630 imet 632 160.255.255.100
- - - 0 i

2. MAC-IP:

up506-CentRtr#show bgp evpn route-type mac-ip vni 631
Network Next Hop Metric LocPref Weight Path
* > RD: 160.255.255.10:630 mac-ip 631 444c.a8a5.1140
160.255.255.10 - 100 0 i Or-ID: 160.255.255.10 C-LST: 180.255.255.1

up506-CentRtr#show bgp evpn route-type mac-ip vni 632
Network Next Hop Metric LocPref Weight Path
* > RD: 160.255.255.20:630 mac-ip 632 444c.a8a5.1141
160.255.255.20 - 100 0 i Or-ID: 160.255.255.20 C-LST: 180.255.255.1

Ping check-up:

Host1#ping vrf EvpnHost1 160.63.2.202
PING 160.63.2.202 (160.63.2.202) 72(100) bytes of data.
80 bytes from 160.63.2.202: icmp_seq=1 ttl=63 time=0.157 ms
80 bytes from 160.63.2.202: icmp_seq=2 ttl=63 time=0.112 ms
80 bytes from 160.63.2.202: icmp_seq=3 ttl=63 time=0.144 ms
80 bytes from 160.63.2.202: icmp_seq=4 ttl=63 time=0.106 ms
80 bytes from 160.63.2.202: icmp_seq=5 ttl=63 time=0.133 ms

--- 160.63.2.202 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.106/0.130/0.157/0.021 ms, ipg/ewma 0.156/0.143 ms

5/21/2020

Arista EVPN VXLAN Configuration Example (2c) - Single-homing, L2 EVPN, Vlan-aware vs Vlan-based

According to draft-krattiger-evpn-modes-interop-0, the vlan-based should interop with vlan-aware bundled MAC VRF. But I did a quick test on 4.24.0F EOS, it doesn't work obviously

And the problem is that, flood set is not correct.

snp261#sh l2rib input bgp floodset
L2 RIB EVPN Input flood set:
Vlan Address Type Destination
---------- -------------------- ---------- -------------------
601 0000.0000.0000 All VTEP 160.255.255.20
602 0000.0000.0000 All VTEP 160.255.255.20
611 0000.0000.0000 All VTEP 160.255.255.20
612 0000.0000.0000 All VTEP 160.255.255.20
<<<< there is no VTEP flood set for VLAN 621 and 622

Even the "show bgp evpn route-type imet <prefix>" shows correct RT values, the but import doesn't work here.

wa464#sh bgp evpn route-type imet rd 160.255.255.10:621 detail

BGP routing table information for VRF default

Router identifier 160.255.255.20, local AS number 65162

BGP routing table entry for imet 160.255.255.10, Route Distinguisher: 160.255.255.10:621

Paths: 1 available

Local

160.255.255.10 from 160.255.255.1 (180.255.255.1)

Origin IGP, metric -, localpref 100, weight 0, valid, internal, best

Originator: 160.255.255.10, Cluster list: 180.255.255.1

Extended Community: Route-Target-AS:65100:620 TunnelEncap:tunnelTypeVxlan

VNI: 621

PMSI Tunnel: Ingress Replication, MPLS Label: 621, Leaf Information Required: false, Tunnel ID: 160.255.255.10

Arista EVPN VXLAN Configuration Example (2b) - Single-homing, L2 EVPN, Vlan-aware

The VLAN-based MAC VRF has RD/RT values per VLAN/VNI. But most of the time, one tenant customer uses multiple VLANs. In this case, we can use just 1 RD/RT to mark the EVPN routes, which is called VLAN-aware bundle MAC VRF, and is illustrated as below:

Explanations:

Configuration is much like the VLAN-based MAC VRF
Configure VLAN-aware MAC-VRF under router BGP with RD/RT and it can have multiple VLANs
"redistribute learned" is to advertised the learnt MAC under VLAN as type-2 routes to remote EVPN peers.
Under interface Vxlan 1, configure VNI values for above VLANs

Control Plane Check-up:

1. IMET, almost same as VLAN-based, but under 1 RD/RT with 2 VNIs/VTEP

snp261-eVtep1#sh bgp evpn route-type imet rd 160.255.255.20:610 detail
BGP routing table information for VRF default
Router identifier 160.255.255.10, local AS number 65161
BGP routing table entry for imet 611 160.255.255.20, Route Distinguisher: 160.255.255.20:610
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:610 TunnelEncap:tunnelTypeVxlan
VNI: 611
PMSI Tunnel: Ingress Replication, MPLS Label: 611, Leaf Information Required: false, Tunnel ID: 160.255.255.20
BGP routing table entry for imet 612 160.255.255.20, Route Distinguisher: 160.255.255.20:610
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:610 TunnelEncap:tunnelTypeVxlan
VNI: 612
PMSI Tunnel: Ingress Replication, MPLS Label: 612, Leaf Information Required: false, Tunnel ID: 160.255.255.20

2. MAC-IP, under 1 RD/RT, but 2 VNI for 2 VLANs

snp261-eVtep1#sh bgp evpn route-type mac-ip rd 160.255.255.20:610 detail
BGP routing table information for VRF default
Router identifier 160.255.255.10, local AS number 65161
BGP routing table entry for mac-ip 611 444c.a8a5.1141, Route Distinguisher: 160.255.255.20:610
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:610 TunnelEncap:tunnelTypeVxlan
VNI: 611 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip 612 444c.a8a5.1141, Route Distinguisher: 160.255.255.20:610
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:610 TunnelEncap:tunnelTypeVxlan
VNI: 612 ESI: 0000:0000:0000:0000:0000

Ping check-up:

Host1#ping vrf EvpnHost1 160.61.1.201
PING 160.61.1.201 (160.61.1.201) 72(100) bytes of data.
80 bytes from 160.61.1.201: icmp_seq=1 ttl=64 time=0.790 ms
80 bytes from 160.61.1.201: icmp_seq=2 ttl=64 time=0.134 ms
80 bytes from 160.61.1.201: icmp_seq=3 ttl=64 time=0.102 ms
80 bytes from 160.61.1.201: icmp_seq=4 ttl=64 time=0.107 ms
80 bytes from 160.61.1.201: icmp_seq=5 ttl=64 time=0.117 ms

--- 160.61.1.201 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4ms
rtt min/avg/max/mdev = 0.102/0.250/0.790/0.270 ms, ipg/ewma 1.000/0.510 ms

Host2#ping vrf EvpnHost1 160.61.2.201
PING 160.61.2.201 (160.61.2.201) 72(100) bytes of data.
80 bytes from 160.61.2.201: icmp_seq=1 ttl=64 time=0.884 ms
80 bytes from 160.61.2.201: icmp_seq=2 ttl=64 time=0.105 ms
80 bytes from 160.61.2.201: icmp_seq=3 ttl=64 time=0.106 ms
80 bytes from 160.61.2.201: icmp_seq=4 ttl=64 time=0.097 ms
80 bytes from 160.61.2.201: icmp_seq=5 ttl=64 time=0.096 ms

--- 160.61.2.201 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4ms
rtt min/avg/max/mdev = 0.096/0.257/0.884/0.313 ms, ipg/ewma 1.001/0.559 ms

Arista EVPN VXLAN Configuration Example (2a) - Single-homing, L2 EVPN, Vlan-based

As of May 2020, the latest EOS 4.24.0F supports 2 types of L2 EVPN MAC-VRF:

Vlan-based

1 VRF (1 RD): 1 VLAN
ESI = 0

Vlan-aware Bundled

1 VRF (1 RD): n VLANs
ESI = VNI

Here is the configuration of Vlan-Based L2 EVPN

Explanations:

Configure VLAN aka MAC-VRF under router BGP with RD/RT
"redistribute learned" is to advertised the learnt MAC under VLAN as type-2 routes to remote EVPN peers.
Under interface Vxlan 1, configure VNI values for above VLANs

Control Plane Checkup:

EVPN uses 2 types routes for L2EVPN, 1) type-3 IMET, 2) type-2 MAC-IP

1) IMET

snp261-eVtep1#show bgp evpn route-type imet vni 601 detail
BGP routing table information for VRF default
Router identifier 160.255.255.10, local AS number 65161
BGP routing table entry for imet 160.255.255.10, Route Distinguisher: 160.255.255.10:601
Paths: 1 available
Local
- from - (0.0.0.0)
Origin IGP, metric -, localpref -, weight 0, valid, local, best
Extended Community: Route-Target-AS:65100:601 TunnelEncap:tunnelTypeVxlan
VNI: 601
PMSI Tunnel: Ingress Replication, MPLS Label: 601, Leaf Information Required: false, Tunnel ID: 160.255.255.10
BGP routing table entry for imet 160.255.255.20, Route Distinguisher: 160.255.255.20:601
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:601 TunnelEncap:tunnelTypeVxlan << RT to control import
VNI: 601 << VNI for this VLAN
PMSI Tunnel: Ingress Replication, MPLS Label: 601, Leaf Information Required: false, Tunnel ID: 160.255.255.20 << VTEP ID

2) Flood-set, the above IMET prefix is used to form the flood-set

snp261-eV1.16:27:32#show vxlan flood vtep vlan 601
VXLAN Flood VTEP Table
--------------------------------------------------------------------------------
VLANS Ip Address
----------------------------- ------------------------------------------------
601 160.255.255.20

snp261-eV1.16:23:04#show l2rib output floodset vlan 601
L2 RIB Output flood set:
Source: Local Dynamic, Local Static, BGP, VXLAN Static, VXLAN Dynamic
Vlan Address Type Destination
---------- -------------------- ---------- -------------------
601 0000.0000.0000 All VTEP 160.255.255.20

snp261-eV1.16:27:02#show l2rib input bgp floodset vlan 601
L2 RIB EVPN Input flood set:
Vlan Address Type Destination
---------- -------------------- ---------- -------------------
601 0000.0000.0000 All VTEP 160.255.255.20

3) Type-2 MAC-IP EVPN Route

snp261-eVtep1#show bgp evpn route-type mac-ip vni 601 detail
BGP routing table information for VRF default
Router identifier 160.255.255.10, local AS number 65161
BGP routing table entry for mac-ip 444c.a8a5.1140, Route Distinguisher: 160.255.255.10:601
Paths: 1 available
Local
- from - (0.0.0.0)
Origin IGP, metric -, localpref -, weight 0, valid, local, best
Extended Community: Route-Target-AS:65100:601 TunnelEncap:tunnelTypeVxlan
VNI: 601 ESI: 0000:0000:0000:0000:0000
BGP routing table entry for mac-ip 444c.a8a5.1141, Route Distinguisher: 160.255.255.20:601 << mac address
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:601 TunnelEncap:tunnelTypeVxlan << RT to control import
VNI: 601 ESI: 0000:0000:0000:0000:0000 << VNI

4) MAC table vs EVPN prefixes

snp261-eV1.16:29:56#show mac address-table interface vxlan 1 vlan 601
Mac Address Table
------------------------------------------------------------------

Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------
601 444c.a8a5.1141 DYNAMIC Vx1 1 4:53:10 ago

5) Clear MAC on remote VTEP to simulate MAC aging out

wa464-eVtep2#clear mac address-table dynamic vlan 601 << clear MAC

snp261-eVtep1#show bgp evpn route-type mac-ip vni 601 detail << NO evpn type-2 prefix
BGP routing table information for VRF default
Router identifier 160.255.255.10, local AS number 65161
BGP routing table entry for mac-ip 444c.a8a5.1140, Route Distinguisher: 160.255.255.10:601
Paths: 1 available
Local
- from - (0.0.0.0)
Origin IGP, metric -, localpref -, weight 0, valid, local, best
Extended Community: Route-Target-AS:65100:601 TunnelEncap:tunnelTypeVxlan
VNI: 601 ESI: 0000:0000:0000:0000:0000

snp261-eVtep1#show mac address-table interface vxlan 1 vlan 601 << no MAC entry
Mac Address Table
------------------------------------------------------------------

Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------

Data Plane Checkup:

host1 under VTEP1 ping host2 behind VTEP2

Host1#ping vrf EvpnHost1 160.60.1.201
PING 160.60.1.201 (160.60.1.201) 72(100) bytes of data.
80 bytes from 160.60.1.201: icmp_seq=1 ttl=64 time=0.135 ms
80 bytes from 160.60.1.201: icmp_seq=2 ttl=64 time=0.100 ms
80 bytes from 160.60.1.201: icmp_seq=3 ttl=64 time=0.092 ms
80 bytes from 160.60.1.201: icmp_seq=4 ttl=64 time=0.088 ms
80 bytes from 160.60.1.201: icmp_seq=5 ttl=64 time=0.089 ms

--- 160.60.1.201 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.088/0.100/0.135/0.021 ms, ipg/ewma 0.128/0.117 ms

VLAN-based: RD/RT vs VNI = 1:1

From the below output, the different VLANs have different RD and RT values, so 1:1 relationship. (In our case, only one host simulates multiple hosts under different VLANs).

snp261-eV1.18:18:45#sh bgp evpn route-type mac-ip vni 601 detail
BGP routing table entry for mac-ip 444c.a8a5.1141, Route Distinguisher: 160.255.255.20:601
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:601 TunnelEncap:tunnelTypeVxlan
VNI: 601 ESI: 0000:0000:0000:0000:0000

snp261-eV1.18:18:49#sh bgp evpn route-type mac-ip vni 602 detail
BGP routing table entry for mac-ip 444c.a8a5.1141, Route Distinguisher: 160.255.255.20:602
Paths: 1 available
Local
160.255.255.20 from 160.255.255.1 (180.255.255.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Originator: 160.255.255.20, Cluster list: 180.255.255.1
Extended Community: Route-Target-AS:65100:602 TunnelEncap:tunnelTypeVxlan
VNI: 602 ESI: 0000:0000:0000:0000:0000