VXLAN Routing with MLAG

VXLAN Routing with MLAG
https://eos.arista.com/vxlan-routing-with-mlag/

• VXLAN routing routes pkt based on IP address in inner header, not outer VXLAN header. 
• 有个示意图，可以帮助理解
• 在SW-1其实就是简单的Inter VLAN SVI routing
• 只是SVI-VLAN 20有个VTEP／VNI，所以可以Learn到DEST MAC of Svr2
• Srv-2‘s GW SVI-VLAN-20 is on VTEP-1/SW-1
• VNI 1020 链接左右的L2 Domain
• Routing Topologies
• Direct Routing - routing at 1st-hop leaf node for ALL subnets. 
• Indirect Routing - only route for ONE subnet, reduce amt of ARP/MAC resource on leaf
• Indirect is a derivative of direct
• Direct Routing
• works by creating anycast IP address:
• Leaf acts as GW, owns and responds ARP req
• 所有Leaf Config same "ip address virtual" and "ip virtual mac"
• ip address virtual 10.10.10.254/24
• No routing over an VLAN interface w/ "ip address virtual"
• VTEP w/ "ip address virtual" will fwd any ARP responses to virtual router MAC to all neighbor VTEPs via HER(head-replication). So neighbor VTEPs host same ARP tables. 
• in MLAG, ARP res to "virtual ip addr" are sync'ed with MLAG peer. 
• Note: ARP sync between MLAG is done via VXLAN agent, hence "ip virtual address" is ONLY supported with VXLAN config
• virtual VTEP:
• 每个Leaf都有Virtual IP addr + MAC, 所以都可以response ARP req. 
• 所以建立一个vVTEP. 不太明白
• ARP Timer
• Serv1 sends ARP req to VTEP1. By routing, VTEP1 would learn MAC of Serv4 via initial ARP req. 
• But not via subsequent bi-directional data traffic, because returning traffic could be ECMP'd to VTEP2, which also routes and rewrite SrcMAc of  inner pkt by VTEP2 mac. 
• To avoid MAC being flush (default timeout is 5 min), it is advised to config ARP aging timeout (default 4 hours) less than MAC timeout. 
• So force a ARP refresh and re-learning MAC. 
• Direct Routing Config:
• VTEP only needs to announce its loopback/end-point into BGP. 
• Then tenant subnets exit only on the leafs, NOT in BGP or on spines. 
• show vxlan address-table
• show mac address-table

VXLAN (2) - RFC 7348

• 明白一个概念, Overlay - overlay L2 connectivity over L3 network
• Inter-VM 需要L2 access mode
• 但是DC Infra都是 L3/IP, 因为ECMP，
• 所以Overlay = provide L2 network over L3 infra
• Bcast/Unknown traffic via Mcast
• 这个在EOS里面没有implement，客户不喜欢this approach; 
• 现在就是简单Flood, 所以Mcast/Bcast/Unknown traffic会被复制多份
• Pkt @ IP/UDP (dest port 4789)

VXLAN Bridging with MLAG

VXLAN Bridging with MLAG

• Key takeaways:
• FH VTEP Encap/Decap
• Routing between MLAG peers
• MLAG peers share the same loopback/VTI address
• https://eos.arista.com/vxlan-with-mlag-configuration-guide/
• Provides remote L2 connectivity between racks or DC;
• Each MLAG domain(2 MLAG peers) has ONE logical VTEP
• Same virtual tunnel ip address (VTI)
• 因为两个MLAG Peer work as ONE physical switch
• MAC Sync:
• For encap/decap traffic, both local and remote MAC address need to be sync'ed between peers via peer-link
• remote = remote MAC associated with remote VTEP ip address. 

Configuration (same on both Mlag peers)

interface loopback1
 ip address 192.168.0.1/32
interface vxlan1
 vlan source-interface loopback 1
 vxlan udp-port 4789
 vxlan vlan 10 vni 10
 vxlan vlan 10 flood 192.168.0.2

MAC, ARP, Traffic例子

• serverA (macA) under MLAG domain 1 (Peer1A和1B)，比方说VLAN 10，sends ARP request
• ARP Req 会被Hash over 1 link of 2-port LAG. 
• Peer1A受到这个ARP req, 有4个Actions
• Act#1: peer1A floods this ARP Req所有本地VLAN 10的端口，因为是Bcast Pkt
• Act#2: peer1A floods it to peer1B，这是给peer1B上面的Singly端口
• peer1B只会flood singly ports，而不会flood dual-home ports
• Act#3: peer1A sync with 1B，peer1B知道 macA 是在Port-channel上面
• 这个Sync是另外的 MLAG signaling, 
• Act#4: peer1A ENCAP ARP in VXLAN and floods all VTEP
• FH GW 负责encap/decap vxlan traffic
• VXLAN pkt is ECMP'ed to spine then to remote Peer2A/B，
• Pear2A/B 和 1A/B一样，share一个VTI address，所以逻辑上是一个
• peer1A ECMP to one spine;
• This spine 有2个path to VTEP 192.168.0.2, 比方说ECMP to peer2A
• Peer2A首到ARP req, DECAP VXLAN pkt and learns MACa in from VTEP 192.168.0.1, 以下是标准的MLAG流程 和 Peer1A很类似
• Act#1: Peer2A flood ARP req all local ports
• Act#2: Peer2A flood it via peer-link for those singly ports on Peer2B
• Act#3: Peer2A sync‘s with Peer2B, MACa from VTEP 192.168.0.1
• peerRemoteDynamic
• ServerB unicasts ARP response to ServerA
• dstMAC = MAC.AAA; srcMAC = MAC.BBB
• Now both peer2A/2B know MAC.AAA is on VTEP 192.168.0.1, and ARP response is encap into VXLAN and routed to peer1B
• ENCAP ARP reply on FH device
• 如果peer1B 收到这个ARP reply, 
• Learns MAC.BBB from VTEP 192.168.0.2, remoteDynamic;
• sync with peer1A
• and pkt fwd down to port-ch 10

Useful CLIs:

• show mac address, Ports里面有Vx1
• show vxlan address-table, 有Mac/Vtep/Port

Switch over 例子，例如Peer1A lose all uplinks

• First Hop MLAG Peer/VTEP，负责encap/decap pkts, 这个是Principle
• 例如MLAG peer1A lost all uplinks，但是device is up running
• 还是Peer1A encap/decap pkts
• 需要Routing between peer via Peerlink
• Best Practise是建议routing on a dedicated VLAN而不是Peerlink VLAN 

Arista EOS - "ip virtual mac-address mlag-peer"

Say, in a mlag environment, 

• The hosts can not understand the vMAC in the ARP packets. For example, some F5 and Netapp devices only check the srcMAC of ARP reply, instead of the srcHwMAC inside the ARP.
• These host hashes the traffic with gateway's system mac to 2 mlag peers. 
• When mlagPeer1 receives packets with dstMAC = mlagPeer2 system MAC, it should forward it to peer2 via peerlink, not good, a totally waste of peerlink
• We can configure "ip virtual-router mac-address mlag-peer", which enables the peer to consume packets destined to peer and route them directly. 

psp111.14:56:21#sh platform trident l3  shadow my-station

My Station Tcam:

--------------------------------------------------------------------------------

Id    Vlan/Mask                                  Mac/Mask       VVVVMACD         T/       ModId/    IngPort/

                                                                4646PRPS       Mask         Mask        Mask

                                                                UUMMLPUC

                                                                CCCCS  D

7         0/0x0       44:4c:a8:93:22:9b/ff:ff:ff:ff:ff:ff       00000000        0/1        0/0x0      0/0x7f

8         0/0x0       44:4c:a8:93:22:9b/ff:ff:ff:ff:ff:ff       11001000        0/0        0/0x0       0/0x0

9         0/0x0       01:00:5e:00:00:00/ff:ff:ff:00:00:00       00000100        0/0        0/0x0       0/0x0

10        0/0x0       44:4c:a8:93:29:d5/ff:ff:ff:ff:ff:ff       11001100        0/0        0/0x0       0/0x0

11        0/0x0       00:dc:00:02:00:01/ff:ff:ff:ff:ff:ff       11001100        0/0        0/0x0       0/0x0

Arista EOS - BGP maintenance mode

Basically the BGP maintenance mode on Eos is an implementation of BGP G-SHUT in RFC 8326. The mechanism is quite simple and effective:

• Add an outbound policy to attach GSHUT community to all prefixes, and it triggers a re-advertisement;
• Add an inbound policy to set LOCAL_PRF = 0 to all incoming prefixes. 
• Wait bgp convergence then shut bgp session

The issue to be solved here is: if backup path is hidden by RR or nodes of an AS, it will trigger relearn routes and put them effective. 

Configuration: (system-level in Arista)

config

maintenance

   unit System

      profile unit System

install source scp:solomon@server/export/images/EOS.swi destination flash:

copy runn start

quiese

reload now force

show ip bgp summary

show ip bgp 0.0.0.0/0 detail (in any leaf to verify GSHUT)

show output (before quiesce)

!! Gshut initiator (dut to reload)

ck421.15:08:47(config-builtin-unit-System)#sh ip bgp neighbors | egrep '^BGP|Updates:'
BGP neighbor is 100.1.11.1, remote AS 65110, external link
                         Sent      Rcvd
    Updates:             2810      2574

!! BGP neighbor

pts321.15:06:07(config)#sh ip bgp 2.2.2.2/32 detail
pts321.15:06:14#sh ip bgp 2.2.2.2/32 det
BGP routing table information for VRF default
Router identifier 100.1.11.1, local AS number 65110
BGP routing table entry for 2.2.2.2/32
 Paths: 5 available
....
  65100 65120
    192.1.0.0 from 192.1.0.0 (169.169.169.1) <<< Initiator 
      Origin IGP, metric -, localpref 100, weight 0, valid, external, ECMP, ECMP contributor
      Not best: ECMP-Fast configured

show output (after quiesce)

ck421.15:11:28(config-builtin-unit-System)#sh ip bgp sum
BGP summary information for VRF default
Router identifier 169.169.169.1, local AS number 65100
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc
m 192.1.0.1        4  65110            100       103    0    0 01:15:55 Estab  11     11
m 192.1.0.3        4  65110            100       104    0    0 01:15:55 Estab  11     11

ck421.15:11:13(config-builtin-unit-System)#sh ip bgp neighbors | egrep '^BGP|Updates:|Sent.*Rcvd'
BGP neighbor is 100.1.11.1, remote AS 65110, external link
                         Sent      Rcvd
    Updates:             4233      2574  <<< 2810 vs 4233 (resend)

pts321.15:06:20#sh ip bgp 2.2.2.2/32 det
BGP routing table information for VRF default
Router identifier 100.1.11.1, local AS number 65110
BGP routing table entry for 2.2.2.2/32
 Paths: 5 available
....
  65100 65120
    192.1.0.0 from 192.1.0.0 (169.169.169.1)
      Origin IGP, metric -, localpref 0, weight 0, valid, external
      Not best: Local preference
      Community: GSHUT <<<<< 

Arista EOS: tcpdump the VRF interface

Use the Linux name space: 

[admin@dc7050 ~]$ sudo ip netns exec ns-<vrf> tcpdump -i vlan2101 arp

Arista: BGP neighbor next-hop-unchanged doesn't work in gated mode

A bit background, in Arista EOS, there is 2 implementations of BGP process. Default is gated, and late one is multi-agent, which can be enabled via cli - service routing protocols model multi-agent. 

BGP neighbor next-hop-unchanged is only supported in multi-agent mode, not in gated. This feature works in route-map with both modes. 

Arista - traffic disruption during LAG reprogramming

No traffic disruption is expected during LAG reprogramming like adding/removing member ports. There is only one exception - on Sand(Arad/Jericho) platforms, if software Lag is enabled, traffic loss will be seen when # of Lag is changed from 1 to 2,  and vice versa. And software Lag can be disabled by knob - "platform sand lag hardware-only".