FHRP, VRRP and VARP

VARP (Z636)

• "ip virtual mac <mac>" 
• = treat <mac> as own mac;
• 和MLAG合用=poor-man's VRRP，
• 好处是active-active，
• 坏处是static ARP on neighbor hosts，为啥坏啊？
• 如果SVI有"ip virtual addr"
• respond ARP req for vIP + vMAC, 但是srcMAC还是phyMAC;
• GARP, srcMAC = vMAC, 刷switch MAC table
• under bash, 还有command?
• varp vlan3 1.2.3.4 00:1c:73:00:00:01

VRRP over MLAG (Z1223)

• 传统上，MLAG最好的选择的是VARP，用VRRP is kinda dumb(DE's comments)
• 最大的问题是，VRRP Backup不fwd traffic，而且peerlink上不学MAC，结果哪？
• hash到backup的traffic，会被flood，连switch都没有，因为peerlink不学mac
• 纪录在Y31356
• Solution是write vrrp mac address into mlag host
• 还有mlag reload delay + VRRP
• peerlink先起来，vrrp prempt所以newly up peer becomes master
• 可是the new peer还在reload delay，black hole traffic!!
• 纪录在Y30494
• Workaround: config preempt delay reload #1 > reload-delay #2
• 这个Z1223最后没有做

FHRP: HSRP, VRRP and VARP

• HSRP, VRRP and VARP use vMAC. GLBP uses phyMAC for LB; 
• HSRP vMAC = 0000:0c07:ac**
• VRRP vMAC = 0000:5e00:01xx, xx = VRID (1-256)
• VARP vMAC = self-configured
• GLBP其实也用vMAC，应该是不同的vMAC = phyMAC
• Assigned MAC address (side note)
• 00-00-5e, IANA (internet assign num association) ucast
• 00-00-5e-00-01/02-xx, VRRP v4/v6
• 01-00-5e, IANA mcast
• 00-00-00 to 7f-ff-ff: v4 mcast
• 90-00-01: bfd on LAG
• 最大的区别是，Active-Active vs Active-Standby, 如何做到的？
• 都用vMAC, hosts send packets with dstMAC = vMAC
• VARP是active-active，直接route out. 
• 而VRRP是bridge to peer via peerLink 
• GARP是刷switch mac table + 通知全部的hosts, ip/MAC mapping of vMAC = vIP
• GARP和普通ARP Reply一样，只是dstMAC = FF or hostMAC
• 只有GARP pkt里面的srcMAC是vMAC. 这是唯一pkt!!. 其他data甚至ARP的srcMAC都是phyMAC. 
• 需要phyIP吗？
• 'ip virtual address' 无论有没有mask，都需要phyIP. 不过w/ mask可以是dummy ip. 
• 'ip address virtual' 不要phyIP
• VARP = ip virtual address  - IVA
• GARP和ARP Reply一样，srcMAC, arp.sndMAC = vMAC
• ARP request里面, 里外Eth/ARP都是Switch System MAC，arp.sndIP = phyIP，因为要确保Arp reply回到Src Mlag Peer!!!
• VARP w/mask = ip virtual address w/ mask - IVAM
• GARP + ARP Reply = VARP way
• 关键是ARP Req, 没有phyIP under this subnet, 所以里外都是vMAC/vIP. 好了有问题了，如果Host ARP Reply hashed到里外一个Peer，咋办？
• 所以这个VxLAN VARP必须有 ARP Sync!
• VxLAN Anycast = ip address virtual + vMac
• both Mlag peer都是一个Addr. 不需要phyIP, vIP就可以
• 没有GARP，为啥？因为ARP reply另外都是vMAC，不需要GARP刷switch
• host知道GW，必须靠ARP reply by mlag peer. 里外都是vMac + vIP. 
• ARP Req = VARP w/mask, 因为没有phyIP；
• 所以也需要ARP Sync. 

Python Tips

1. Is vs ==, mutable and immutable

• == is to compare value, is is compare address
• == is to compare by calling object.__eq__()
• list1 = list2, address is assigned, not value. need list1=list2[:]

>>> a = [1,2,3]
>>> id(a)
4463753048  <<<< address is 048
>>> a[0] = 11
>>> id(a)
4463753048
>>> a
[11, 2, 3]
>>> a.append(44)
>>> a
[11, 2, 3, 44]
>>> id(a)   <<<< can change item and append, address is same
4463753048
>>> b = a
>>> id(b)   <<<< list= is point to same address
4463753048
>>> b[0] = 111
>>> a
[111, 2, 3, 44]  <<<< change b = change a
>>> c = a[:]
>>> id(c)
4463919328
>>> c[0] = 1234
>>> a
[111, 2, 3, 44]
>>> b
[111, 2, 3, 44]
>>> c
[1234, 2, 3, 44]
>>>

2. keyword argument

>>> def ff(*args, **kwargs):
...     print args
...     print kwargs
...

>>> ff(1, '22', k1=333, k2='4444')
(1, '22')
{'k2': '4444', 'k1': 333}

>>> d = {'ka':'aaa', 'kb':123}
>>> ff(**d)
()
{'kb': 123, 'ka': 'aaa'}

3. Regexp \符号

https://docs.python.org/3/howto/regex.html

Remove全部的None-Alphanumeric char from string. 应该是

s = re.sub('\W', '', s)

还有记住以下的

\d, \D = [0-9]
\s, \S = [ \t\n\r\v\f]

\w, \W = alphanumeric,[0-9a-bA-Z_]

4. arstCli Script模版

from arstCliLib import *
import sys

dut = sys.argv[1]
openSshOnDut( dut )
setAccessMethod(dut, 'ssh')
cmd = ['show ip int brief | grep " 10\." | grep Vlan']
output = sendCmd(dut, cmd, prompt='enable', raw=True)

print '\n'.join(output)

[solomonyang@syscon] ~ $ python testCli.py
Traceback (most recent call last):
  File "testCli.py", line 4, in <module>
    from arstCliLib import *
ImportError: No module named arstCliLib

PATH=$PATH:$HOME/bin:$HOME/py
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/git/systest-infra/lib/
PYTHONPATH=$PYTHONPATH:$HOME/git/systest-infra/lib/

export PATH
export LD_LIBRARY_PATH
export PYTHONPATH

5. List vs Tuple, based on stackoverflow post

1. Literal/语法

>>> t = (1,2); l = [1,2]; t[1]; l[0]
2
1

2. Size/内存量，区别大概list +12%

>>> t = tuple(range(100000)); l =  list(range(100000)); t.__sizeof__(); l.__sizeof__()
800024
900088

3. Mutable vs Inmutable/可变 vs 不可变
>>> t = (1,2); l = [1,2]
>>> l[0] = 11; print l
[11, 2]
>>> t[0] = 11; print t
TypeError: 'tuple' object does not support item assignment

4. 都可以增加，但是不同，tuple += (3)是返回一个新的obj，而list.add()是改原来的
>>> t = (1,2); l = [1,2]; id(t); id(l); t+=(3,); l+=[3]; id(t); id(l); l.append(3); id(l)
4306859000 <<<<< t
4307014720 <<<<< 1, 都一个地址
4306913952 <<<<< t +=(3,)，新地址
4307014720

4307014720

5. 因为不可变，所以tuple (1,2)可以是Dict[key]
>>> d = {}; d[t] = '1,2'
>>> d = {}; d[l] = '1,2'
TypeError: unhashable type: 'list'

6. 用途
比方说，(10,11)是个bookmark，第10页的第11行，一般没有必要改；而list of bookmark = [ (1,10), (10,11), (22, 1)] 

7. Tuple != constant list
http://news.e-scribe.com/397
但是这么了解哪，好像也撮合:-) 好像更多的是字面的了解，tuple = lightweight record，比方说：DB API's fetchmany() 返回的是List of tuple. 每个tuple是一个record，不能改其中一项，改了就没有意义了。

List vs Set

1. Literal/语法/内存量
>>> s = set(range(100000)); s.__sizeof__(); l = list(range(100000)); l.__sizeof__()
4194504

900088

2. Set没有Index, 没有重复，可以数学操作 &, -, ^
>>> s1=set(range(1,20,2)); s2=set(range(10,30)); print 's1->', s1; print 's2->', s2; print '-:', s1 - s2; print '&:', s1&s2; s1^s2
s1-> set([1, 3, 5, 7, 9, 11, 13, 15, 17, 19])
s2-> set([10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29])
-: set([1, 3, 9, 5, 7])
&: set([19, 17, 11, 13, 15])

set([1, 3, 5, 7, 9, 10, 12, 14, 16, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29])

ISIS Segment Routing

CCIE SR discussion

• 说起来不难，就是用IGP(ISIS)来signal labels，而不是LDP或者RSVP，更简单
• 但是还是用MPLS，所以软件升级就可以了
• 有Node Segment和Adj Segment，Ingress Router构建Label Stack就可以控制LSP

Arista MPLS SR

• 网络的难点是：
• 如何Classify traffic 和 Engineer path. 
• SDN如何control traffic path/flow，特别是native solution for ipv6
• MPLS解决了一些，但是太复杂，例如TE，而且还没有ipv6 native support. 
• Segment是Arista认为的solution
• SR operation:
• 网络分成Segment, 给一个SID - segment id. 利用BGP/ISIS/OSPF extension来distribute. 而不在需要Label protocols - LDP/RSVP
• SID有Global Unique或者Local significant, Base + Index
• 3种 global SID: 
• 1) prefix;  
• 2) node; loopback of node
• 3) anycast; loopback shared by a set of routers, ecmp
• Prefix SID: 全网都是same value, 非常关键，reducing DP state;
• 图上的例子是, 所有的router SRGB(SR Global Block)都是900,000-965,535, Rtr5's 5.5.5.5/32, prefix-SID是10，所有的router全部assign label 900,000+10 = 900,010
• Adj Sid是locally significant, 只给neighbor，只installed at neighbor
• 最简单应用，用到ECMP，ingress LER就push一个SR label. 或者是push a set of label
• SR vs LDP:
• 相同的地方：
• easy configured, "plug and play"
• Both form stateless Mp-to-pt LSP
• 不同：
• LDP全是local signficant label, SR是global unique labels, 减少DP state; scale 好；
• SR有TE，LDP TE没有流行，v6也没有
• SR vs RSVP-TE
• RSVP-TE的特点：
• constraints routing like b/w, shared link risk group and explicit paths，可以不按照IGP shortest path;
• 有b/w
• FRR，有pre-computed backup paths
• 不好的地方：
• full-mesh p2p TE tunnels, 没有ECMP
• failure后有churn，需要re-signaling. 
• scalability issue，所以不那么流行
• SR利用SDN，在head加入stack of mpls label，而不需要中间router纪录state。还有scale
• 什么都好，没有Multicast?
• Arista SR:
• 必须是R-series with FlexRoute?
• 三种SR Solutions:
• Static MPLS push + NHG
• 在ingress LER上，configure a route pointing to a label stack via CLI;
• ECMP = multiple tunnels
• Class-based service policy 
• Easy start
• Controller based using Eos SDK
• BGP-LU (labeled unicast/RFC 3107)
• 应用：Cloud-WAN, CDN, NFV

Arista ISIS SR (from 4-17-0f)

Tutorial: Segment Routing

https://www.youtube.com/watch?v=BEo5MdB3o3Y

Take-away points:

• SR中最重要的是TE
• Segment (1-hop or n-hop) vs SID
• SR是hybrid TE approaches: path info in packet and network
• CSPF - attribute(b/w, color), requirement and shortest path
• CP = controller

What's SR?

• A tunneling tech - nothing fancy
• A TE! - steer packet instead of routing path. ~= RSVP TE

Terminology:

• Domain
• SR path
• Segment = SID, can be multiple hops

Traditional TE Approaches

• path info in packet
• like IPv4 strict routing option
• path info in the network
• RSVP-signaled MPLS

SR is hybrid TE approach

• Segment types:
• 1 router hop
• multiple router hops
• Types:
• Adjacency (IGP adjacency, 1-hop)
• Prefix = IGP least cost path to a prefix

SR Encapsulation

• MPLS
• Label = segment
• IPv6:
• Segment Routing Extension Header (SRH)
• A list of ipv6 addr
• each ipv6 addr = a segment

Local Labels

• Some SIDs have node-local significance
• Adjancency
• Why important
• Stack can be too big, ASIC cannot handle
• MTU

Global Labels

• Some have domain-wide significance
• Each node reserves a block of labels. 
• SRGB base

IPv6 forwarding

• SRH, segment routing hdr
• A list of ipv6 address as SID

Binding SIDs

• Pointing to another SR paths or tunnels
• Reasons:
• Label stack
• MTU size too big

Control Plane

• Path computation: on SR ingress or central controller. 

CSPF, 这个讲的清楚！

• Attributes to segment: color, b/w, SRLG
• Req to each path
• Shortest path meeting req.
• LSDB, TED = extension to carry info. 
• Alternative path to protect
• All info in LSDB, no need for RSVP or LDP

SR convergence after failures

• Fast recovery by IGP reconvergence. 
• TI-FLA, speed up convergence if not fast enuf
• Use anycast SID (ecmp)

SR benefits:

• If using SR to reserve b/w, MUST go for controller
• Central controller has global view 

Controller Protocol options:

• pull LSDB:
• controller: a passive mbr in IGP
• BGP-LS
• push segment list to ingress SR
• PCEP
• BGP
• push policy
• binding what traffic to which path
• PCEP or BGP

Conclusion:

• SR moves state from network to packet - simplified
• Some open issues: OAM, Fast Reroute
• Need  experience

Arista EOS - hardware counter feature

Since on Sand platform, Ingress/Egress ipv4 ACL, Qos and PBR counters occupy the same counter engineer, so they will not be able to work together. You have to specify which counter could use this shared engineer. 

yo411.mlagB.profA.11:49:40(config)#hardware counter feature ?
  acl            ACL counter feature
  mpls           MPLS LFIB counter feature
  nexthop        Nexthop counter feature
  pdp            PDP counter feature
  subinterface   Subinterface counter feature
  traffic-class  Traffic-class counter feature
  vlan           VLAN counter feature

Arista/EOS, MLAG ipv6 partial traffic loss

Topology (a typical MLAG network):

[mlagA] ======= [mlagB]
    \              /
     +---[Leaf]---+

Symptom:
1) about 10% L3 ipv6 traffic, WE and SN
2) No L2 or L3 ipv4 traffic loss

Get one problem destination - 2000:120:4d:d::1

1) show ipv6 route is good,
bn303.mlagA.profA.16:17:04(config)#sh ipv6 route 2000:120:4d:d::1
 C    2000:120:4d::/48 [0/1]

       via Vlan2077, directly connected

bn302.mlagB.profA.16:17:03(config-if-Vl2199)#sh ipv6 route 2000:120:4d:d::1
 C    2000:120:4d::/48 [0/1]
       via Vlan2077, directly connected

2) show ipv6 route host is NOT right
bn303.mlagA.profA.16:18:20(config)#sh ipv6 route host | grep 2000:120:4d:d
 A  2000:120:4d:d::1 on Vlan2077
 A  2000:120:4d:d::3 on Vlan2077 <<<<<< missing ::2
 A  2000:120:4d:d::4 on Vlan2077

bn302.mlagB.profA.16:18:20(config-if-Vl2199)#sh ipv6 route host | grep 2000:120:4d:d
 A  2000:120:4d:d::2 on Vlan2077 <<<<<< missing ::1
 A  2000:120:4d:d::3 on Vlan2077
 A  2000:120:4d:d::4 on Vlan2077

Root cause: 
missing configuration - "ip virtual-router mac-address mlag-peer"

Arista EOS Debug Tips on Traffic Loss

1. check what/how many pkts punted to CPU?

- "show cpu counter queue", this is the place we see the software drop

- output is by switch ASIC and cpu queues

- CpuQueueL3DstMiss: pkt destinated to unknown address, like unARP'ed host address

- CpuQueueL3LpmOverflow - ?

2. hardward drop

- show hardware counter drop

3. show platform fap interrup

- check hw interrupts

4. Check drops on which interfaces

- show interface counter discard | nz

Arista EOS: %QOS-3-POLICY_HW_RESOURCE_FULL

When applying Qos policy under port-channel, the system doesn't accept it. 

wa463.bug228215.16:12:21(config-if-Po20)#service-policy type qos input SPTest
% Error: Cannot apply service-policy to Port-Channel20 ()

And show logg displays an error msg:
Mar  8 16:01:25 wa463 SandAcl: %QOS-3-POLICY_HW_RESOURCE_FULL: Insufficient hardware resources to program the input policy-map SPTest.

It is because the TCAM is running out. One possible reason is PDP (per-port data policy) which uses up quite some TCAM. So try the EOS-Int image. 

wa462.bug228215.16:17:04#sh platform jericho acl tcam summary
The total number of TCAM lines per Jericho bank is 2048

========================================================
Jericho0:
========================================================
   Bank   Used           Used %          Used By
      0   2046               99         IP RACLs
      1   1554               75         IP RACLs
   2, 3   2048              100       IPv6 RACLs
   4, 5   2048              100       IPv6 RACLs
   6, 7   2048              100       IPv6 RACLs
   8, 9   2046               99       IPv6 RACLs
  10,11    210               10       IPv6 RACLs
     14     79               61 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls

Total Number of TCAM lines used is: 20479

After changing to INT image, this issue is gone. 

wa462.bug228215.16:28:09(config)#int po20
wa462.bug228215.16:28:11(config-if-Po20)#service-policy type qos input SPTest
wa462.bug228215.16:28:13(config-if-Po20)#show ver
Arista DCS-7280CR-48-F
Hardware version:    11.01
Serial number:       JPE16473148
System MAC address:  444c.a897.8c51

Software image version: 4.20.0F-INT-7767198.bloomingtonrel (engineering build)

iptables in EOS

iptables is a Linux firewall utility program, which is leveraged by Arista EOS to control protocol control packets. For example:

Example: sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
-S : List the rules
-A : Append rule
-p : protocol
-j : jump = action

[admin@ck478 ~]$ sudo iptables -S
.....
-A INPUT -p tcp -m tcp --dport 4432 -m ttl --ttl-eq 255 -j SERVICE   <<< MLAG control pkt
-A INPUT -p udp -m udp --dport 4432 -m ttl --ttl-eq 255 -j SERVICE

! add a bgp neighbor 1.1.1.1
ck478.17:51:26(config)#router bgp 65500
ck478.17:51:43(config-router-bgp)#nei 1.1.1.1 remote 65001

[admin@ck478 ~]$ sudo iptables -S | grep -i BGP | grep 1.1.1.1
-A BGP -s 1.1.1.1/32 -j ACCEPT   <<< a new rule added for bgp nei 1.1.1.1

! configure bgp ttl security rule
ck478.17:51:51(config-router-bgp)#nei 1.1.1.1 ttl maximum-hops 2

[admin@ck478 ~]$ sudo iptables -S | grep -i BGP | grep 1.1.1.1
-A BGP -s 1.1.1.1/32 -m ttl --ttl-lt 253 -j DROP <<< all bgp pkts w/ ttl <253 droped!
-A BGP -s 1.1.1.1/32 -j ACCEPT

strace in EOS

"strace" is a powerful linux debug command and it can be used on Arista EOS. 

Some useful arguments:
-c -- count time and calls
-p pid -- trace process with pid#
-T -- print time spent

Example:
[admin@bn303 ~]$ ps -ef | grep Bgp
root     14926  2044  1 09:55 ?        00:00:09 Bgp

[admin@bn303 ~]$ sudo strace -c -p 14926
Process 14926 attached
^CProcess 14926 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  -nan    0.000000           0         1           ioctl
  -nan    0.000000           0         3         3 stat64
  -nan    0.000000           0        46           epoll_ctl
  -nan    0.000000           0        45           epoll_wait
  -nan    0.000000           0         4           send
  -nan    0.000000           0         1           sendmsg
  -nan    0.000000           0         1           recvmsg
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000                   101         3 total

Arista EOS 4.21 : L2 subinterface

Topology:

[rtr3]
  |
  | et2
  |    eth1.1
[rtr1]==============[rtr2]
  |    eth1.2
  | eth3
  |
[rtr4]

Configuration

interface et1.100
  encap dot1q vlan 100
  vlan id 1000
interface et1.200
  encap dot1q vlan 200
  vlan id 1000
interface eth2
  switch access vlan 1000
interface eth3
  switch mode trunk

L2 bridge domain

• Above configuration is to create a L2 bridge domain of 4 end points - et1.100, et1.200, eth2 (access), eth3 (trunk). 
• 1 bcast from et1.100, will be flood to et1.200, et2, et3
• Vlan mapping is different, which is 1:1 relation. So no vlan 1000 on both et1.100 and et1.200
• et1 must be "no switch". 
• Feature is supported from 4.21.*?

Arista Eos 4.21: BGP DSCP Configuration

By default, the BGP packets' DSCP value is 0x0. Now customer wants to have it configurable. A global value is enough. New configuration/change doesn't tear down existing sessions. This feature is started from 4.21.*?

The configuration is quite simple:

router bgp 1
  bgp transport qos dscp 48

Arista EOS CLIs

Platform Dependent

• Trident hw programming
• show platform trident counter | egrep 'card|drop' | nz
• show platform trident tcam detail | grep -i "LAG E" -A2
• show platform trident L3 shadow my-station
• Trident agent logs:
• qtcat strataL3.qt | grep <prefix> 

Platform Independent:

• evpn:
• show l2rib input bgp
• System:
• show event-monitor mac

Software:

• CVP:
• management api http-commands
•   no shutdown
•   user cvpadmin privilege 15 role admin secret eosuper

Ixia: Custom view of traffic

Custom view:
Filters: 
select Traffic -》flow detective
Traffic Item Equals "Traffic Name"
Show 50 "Worst Performers"

Then select Stat

"ip directed-broadcast" in VxLAN

1. What's the feature of "ip directed-broadcast" for?

• One application is Wake-on-Lan(WOL). A host device like PC can be powered on/resumed remotely. 
• Need hw/BIOS support. When host receives a WOL magic packet, it turns on. 
• Enable it in OS, linux - "sudo ethtool -s eth0 wol g"
• So server (20.1.1.1) sends a bcast packet destined to remote subnet like 10.1.1.255 to 10.1.1.0/24 network. 
• With this feature enabled under SVI, this bcast pkt will be fwded to remote subnet like a ucast pkt. 
• By default is disabled, because of the security concern. 
• This is a legacy feature starting from 2011/12?

2. Configuration and details
interface vlan 2001
  ip directed-broadcast

Says the topology with Vxlan is like

vlan 1001
10.1.1.1/24 [host2]-----+
                        |
vlan 2001               |
20.1.1.1/24 [host1]---[l2vtep]---[l2vtep]---[gw of svi1001/2001]

* host1 sends pkts dstIp=10.1.1.255
* pkts follows vxlan to reach gw
* gw routes pkts back l2vtep as a ucast pkt

3. Caveats

• Arista EOS has a bug - 217001, when the vlan is included vxlan interface, the directed-bcast traffic will be shaped by PDP. 

yr252.23:14:18#sh pl trident l3 software host-table | beg 10.50.51.255

   Entry: 0, HwEntry: 0x6002a50, Type: v4Uc, Vrf:  0, Host: 10.50.51.255/32

Bucket: 687, state: 0x00011

VXLAN Routing with MLAG

VXLAN Routing with MLAG
https://eos.arista.com/vxlan-routing-with-mlag/

• VXLAN routing routes pkt based on IP address in inner header, not outer VXLAN header. 
• 有个示意图，可以帮助理解
• 在SW-1其实就是简单的Inter VLAN SVI routing
• 只是SVI-VLAN 20有个VTEP／VNI，所以可以Learn到DEST MAC of Svr2
• Srv-2‘s GW SVI-VLAN-20 is on VTEP-1/SW-1
• VNI 1020 链接左右的L2 Domain
• Routing Topologies
• Direct Routing - routing at 1st-hop leaf node for ALL subnets. 
• Indirect Routing - only route for ONE subnet, reduce amt of ARP/MAC resource on leaf
• Indirect is a derivative of direct
• Direct Routing
• works by creating anycast IP address:
• Leaf acts as GW, owns and responds ARP req
• 所有Leaf Config same "ip address virtual" and "ip virtual mac"
• ip address virtual 10.10.10.254/24
• No routing over an VLAN interface w/ "ip address virtual"
• VTEP w/ "ip address virtual" will fwd any ARP responses to virtual router MAC to all neighbor VTEPs via HER(head-replication). So neighbor VTEPs host same ARP tables. 
• in MLAG, ARP res to "virtual ip addr" are sync'ed with MLAG peer. 
• Note: ARP sync between MLAG is done via VXLAN agent, hence "ip virtual address" is ONLY supported with VXLAN config
• virtual VTEP:
• 每个Leaf都有Virtual IP addr + MAC, 所以都可以response ARP req. 
• 所以建立一个vVTEP. 不太明白
• ARP Timer
• Serv1 sends ARP req to VTEP1. By routing, VTEP1 would learn MAC of Serv4 via initial ARP req. 
• But not via subsequent bi-directional data traffic, because returning traffic could be ECMP'd to VTEP2, which also routes and rewrite SrcMAc of  inner pkt by VTEP2 mac. 
• To avoid MAC being flush (default timeout is 5 min), it is advised to config ARP aging timeout (default 4 hours) less than MAC timeout. 
• So force a ARP refresh and re-learning MAC. 
• Direct Routing Config:
• VTEP only needs to announce its loopback/end-point into BGP. 
• Then tenant subnets exit only on the leafs, NOT in BGP or on spines. 
• show vxlan address-table
• show mac address-table

VXLAN (2) - RFC 7348

• 明白一个概念, Overlay - overlay L2 connectivity over L3 network
• Inter-VM 需要L2 access mode
• 但是DC Infra都是 L3/IP, 因为ECMP，
• 所以Overlay = provide L2 network over L3 infra
• Bcast/Unknown traffic via Mcast
• 这个在EOS里面没有implement，客户不喜欢this approach; 
• 现在就是简单Flood, 所以Mcast/Bcast/Unknown traffic会被复制多份
• Pkt @ IP/UDP (dest port 4789)

VXLAN Bridging with MLAG

VXLAN Bridging with MLAG

• Key takeaways:
• FH VTEP Encap/Decap
• Routing between MLAG peers
• MLAG peers share the same loopback/VTI address
• https://eos.arista.com/vxlan-with-mlag-configuration-guide/
• Provides remote L2 connectivity between racks or DC;
• Each MLAG domain(2 MLAG peers) has ONE logical VTEP
• Same virtual tunnel ip address (VTI)
• 因为两个MLAG Peer work as ONE physical switch
• MAC Sync:
• For encap/decap traffic, both local and remote MAC address need to be sync'ed between peers via peer-link
• remote = remote MAC associated with remote VTEP ip address. 

Configuration (same on both Mlag peers)

interface loopback1
 ip address 192.168.0.1/32
interface vxlan1
 vlan source-interface loopback 1
 vxlan udp-port 4789
 vxlan vlan 10 vni 10
 vxlan vlan 10 flood 192.168.0.2

MAC, ARP, Traffic例子

• serverA (macA) under MLAG domain 1 (Peer1A和1B)，比方说VLAN 10，sends ARP request
• ARP Req 会被Hash over 1 link of 2-port LAG. 
• Peer1A受到这个ARP req, 有4个Actions
• Act#1: peer1A floods this ARP Req所有本地VLAN 10的端口，因为是Bcast Pkt
• Act#2: peer1A floods it to peer1B，这是给peer1B上面的Singly端口
• peer1B只会flood singly ports，而不会flood dual-home ports
• Act#3: peer1A sync with 1B，peer1B知道 macA 是在Port-channel上面
• 这个Sync是另外的 MLAG signaling, 
• Act#4: peer1A ENCAP ARP in VXLAN and floods all VTEP
• FH GW 负责encap/decap vxlan traffic
• VXLAN pkt is ECMP'ed to spine then to remote Peer2A/B，
• Pear2A/B 和 1A/B一样，share一个VTI address，所以逻辑上是一个
• peer1A ECMP to one spine;
• This spine 有2个path to VTEP 192.168.0.2, 比方说ECMP to peer2A
• Peer2A首到ARP req, DECAP VXLAN pkt and learns MACa in from VTEP 192.168.0.1, 以下是标准的MLAG流程 和 Peer1A很类似
• Act#1: Peer2A flood ARP req all local ports
• Act#2: Peer2A flood it via peer-link for those singly ports on Peer2B
• Act#3: Peer2A sync‘s with Peer2B, MACa from VTEP 192.168.0.1
• peerRemoteDynamic
• ServerB unicasts ARP response to ServerA
• dstMAC = MAC.AAA; srcMAC = MAC.BBB
• Now both peer2A/2B know MAC.AAA is on VTEP 192.168.0.1, and ARP response is encap into VXLAN and routed to peer1B
• ENCAP ARP reply on FH device
• 如果peer1B 收到这个ARP reply, 
• Learns MAC.BBB from VTEP 192.168.0.2, remoteDynamic;
• sync with peer1A
• and pkt fwd down to port-ch 10

Useful CLIs:

• show mac address, Ports里面有Vx1
• show vxlan address-table, 有Mac/Vtep/Port

Switch over 例子，例如Peer1A lose all uplinks

• First Hop MLAG Peer/VTEP，负责encap/decap pkts, 这个是Principle
• 例如MLAG peer1A lost all uplinks，但是device is up running
• 还是Peer1A encap/decap pkts
• 需要Routing between peer via Peerlink
• Best Practise是建议routing on a dedicated VLAN而不是Peerlink VLAN 

Arista EOS - "ip virtual mac-address mlag-peer"

Say, in a mlag environment, 

• The hosts can not understand the vMAC in the ARP packets. For example, some F5 and Netapp devices only check the srcMAC of ARP reply, instead of the srcHwMAC inside the ARP.
• These host hashes the traffic with gateway's system mac to 2 mlag peers. 
• When mlagPeer1 receives packets with dstMAC = mlagPeer2 system MAC, it should forward it to peer2 via peerlink, not good, a totally waste of peerlink
• We can configure "ip virtual-router mac-address mlag-peer", which enables the peer to consume packets destined to peer and route them directly. 

psp111.14:56:21#sh platform trident l3  shadow my-station

My Station Tcam:

--------------------------------------------------------------------------------

Id    Vlan/Mask                                  Mac/Mask       VVVVMACD         T/       ModId/    IngPort/

                                                                4646PRPS       Mask         Mask        Mask

                                                                UUMMLPUC

                                                                CCCCS  D

7         0/0x0       44:4c:a8:93:22:9b/ff:ff:ff:ff:ff:ff       00000000        0/1        0/0x0      0/0x7f

8         0/0x0       44:4c:a8:93:22:9b/ff:ff:ff:ff:ff:ff       11001000        0/0        0/0x0       0/0x0

9         0/0x0       01:00:5e:00:00:00/ff:ff:ff:00:00:00       00000100        0/0        0/0x0       0/0x0

10        0/0x0       44:4c:a8:93:29:d5/ff:ff:ff:ff:ff:ff       11001100        0/0        0/0x0       0/0x0

11        0/0x0       00:dc:00:02:00:01/ff:ff:ff:ff:ff:ff       11001100        0/0        0/0x0       0/0x0

Arista EOS - BGP maintenance mode

Basically the BGP maintenance mode on Eos is an implementation of BGP G-SHUT in RFC 8326. The mechanism is quite simple and effective:

• Add an outbound policy to attach GSHUT community to all prefixes, and it triggers a re-advertisement;
• Add an inbound policy to set LOCAL_PRF = 0 to all incoming prefixes. 
• Wait bgp convergence then shut bgp session

The issue to be solved here is: if backup path is hidden by RR or nodes of an AS, it will trigger relearn routes and put them effective. 

Configuration: (system-level in Arista)

config

maintenance

   unit System

      profile unit System

install source scp:solomon@server/export/images/EOS.swi destination flash:

copy runn start

quiese

reload now force

show ip bgp summary

show ip bgp 0.0.0.0/0 detail (in any leaf to verify GSHUT)

show output (before quiesce)

!! Gshut initiator (dut to reload)

ck421.15:08:47(config-builtin-unit-System)#sh ip bgp neighbors | egrep '^BGP|Updates:'
BGP neighbor is 100.1.11.1, remote AS 65110, external link
                         Sent      Rcvd
    Updates:             2810      2574

!! BGP neighbor

pts321.15:06:07(config)#sh ip bgp 2.2.2.2/32 detail
pts321.15:06:14#sh ip bgp 2.2.2.2/32 det
BGP routing table information for VRF default
Router identifier 100.1.11.1, local AS number 65110
BGP routing table entry for 2.2.2.2/32
 Paths: 5 available
....
  65100 65120
    192.1.0.0 from 192.1.0.0 (169.169.169.1) <<< Initiator 
      Origin IGP, metric -, localpref 100, weight 0, valid, external, ECMP, ECMP contributor
      Not best: ECMP-Fast configured

show output (after quiesce)

ck421.15:11:28(config-builtin-unit-System)#sh ip bgp sum
BGP summary information for VRF default
Router identifier 169.169.169.1, local AS number 65100
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc
m 192.1.0.1        4  65110            100       103    0    0 01:15:55 Estab  11     11
m 192.1.0.3        4  65110            100       104    0    0 01:15:55 Estab  11     11

ck421.15:11:13(config-builtin-unit-System)#sh ip bgp neighbors | egrep '^BGP|Updates:|Sent.*Rcvd'
BGP neighbor is 100.1.11.1, remote AS 65110, external link
                         Sent      Rcvd
    Updates:             4233      2574  <<< 2810 vs 4233 (resend)

pts321.15:06:20#sh ip bgp 2.2.2.2/32 det
BGP routing table information for VRF default
Router identifier 100.1.11.1, local AS number 65110
BGP routing table entry for 2.2.2.2/32
 Paths: 5 available
....
  65100 65120
    192.1.0.0 from 192.1.0.0 (169.169.169.1)
      Origin IGP, metric -, localpref 0, weight 0, valid, external
      Not best: Local preference
      Community: GSHUT <<<<< 

Arista EOS: tcpdump the VRF interface

Use the Linux name space: 

[admin@dc7050 ~]$ sudo ip netns exec ns-<vrf> tcpdump -i vlan2101 arp

Arista: BGP neighbor next-hop-unchanged doesn't work in gated mode

A bit background, in Arista EOS, there is 2 implementations of BGP process. Default is gated, and late one is multi-agent, which can be enabled via cli - service routing protocols model multi-agent. 

BGP neighbor next-hop-unchanged is only supported in multi-agent mode, not in gated. This feature works in route-map with both modes. 

Arista - traffic disruption during LAG reprogramming

No traffic disruption is expected during LAG reprogramming like adding/removing member ports. There is only one exception - on Sand(Arad/Jericho) platforms, if software Lag is enabled, traffic loss will be seen when # of Lag is changed from 1 to 2,  and vice versa. And software Lag can be disabled by knob - "platform sand lag hardware-only".