Nexthop Group (2)

NHG supported features are different per Arista platforms:

This is DCS-7060CX based on Tomahwak/Strata, only supports IP next hop, ip-in-ip and mpls

DCS-7060CX(config)#nexthop-group test type ?
  ip        ecmp next hop
  ip-in-ip  ip-in-ip tunnel type
  mpls      mpls tunnel type

This is DCS-7512N based on Jericho/Sand, which supports GRE, ip next hop, mpls and mpls-over-gre.

DCS-7512N(s1)(config)#nexthop-group test type ?
  gre            gre tunnel type
  ip             ecmp next hop
  mpls           mpls tunnel type
  mpls-over-gre  mpls-over-gre tunnel type

Nexthop Group (1)

为什么需要一个这样的功能？比方说，以下一个典型的JY网络

           [internet]<---- user: 200.1.1.1
           /        \
          /          \
   [borderLeaf]  [borderLeaf]
       |||          ||||
[...........spine nodes..........]
       |||          ||||
     [Leaf]        [Leaf  ]
     /    \        /      \
  [LB1]  [LB2] [SVR1]     [SVR2]
[customer.com] [10.1.1.1]
[100.1.1.1   ]            [10.1.1.1]
    
数据包流是这样的

• 每个LB advertise 一组LB IP给用户，应该是internet public ip 
• 用户的请求先到LB，LB再通过IP-in-IP，把数据包发给服务器，extDstIP = SvrIP
• 服务器Decap pkts，处理完之后直接返回给用户
• 所以数据流是: internet --> LB --> Svr --> internet

这个功能做啥哪？主要是switch offload SLB workload

• 第一个是 IPinIP encap
• 在LB上面的Leaf上，create static routes to Svr by NHG-IPinIP-tunnel。LB直接就转发native ip pkts, switch负责encap ip-in-ip tunnel header，因为是hw，比LB高速有效
• Server LB - 上图里面的SVR1/2都有DstIP = 10.1.1.1，ECMP hashing会选一个final dest. 
• 可以在NHG-IPinIP-tunnel增加 ECMP，从而达到network LB

需要啥？

• T+ platform
• LB using ECMP
• v4-in-v4, v6-in-v6，以后的releae应该可以是v4-over-v6 or verse vice
• Scale: 512 v4/256 v6 tunnels
• EAPI: 1) config NHG; 2) config static route pointing to NHG

CLI:

• size = # of NH entries in NHG, default = 256
• entry <index> tunnel dest <dstIP>, dstIP = svrIP，这里在做LB!
• index# = 0..size-1, 如果缺失就是drop, 注意！
• 所以hashing很少啊，srcIP唯一, proto#=4(IPinIP)也是唯一，全靠dstIP
• ip route 10.1.1.1/32 nexthop-group NHG
• 如果route to dstIP本身就是multipath, 那么就选第一条

注意的几点

• 这个NHG不是interface，什么意思哪？
• 他不会像正常的interface去解析Nexthop
• 正常的情况下，解析是ASIC fwd to software due to missing NH info. 在Tunnel情况，就是software自己去解析
• 为啥有drop entry，主要就不会generate ICMP unreachable msg
• multiple NHG可以share tunnels
• 如果多个routes to host ip，gated show多个，但是hw里面只有NHG
• 可以VRF吗？不知道
• IP routing on/off，会重写hw，改变interface ip也是
• MTU必须注意，增加了20B (v4) or 40B (v6)， 需要config correct MTU on ALL interfaces
• RPF: srcIP随意配置，所以注意不要被RPF给drop了
• Linux Kernel has a drop entry for each IP, how to check?

为啥不叫IPinIP tunneling? 为啥取个新名字

• NHG != tunnel interface, 3个区别:
• 没有路由协议
• 不会接受数据包
• 而且NHG可以有不同种类的Encap: LSP/MPLS, GRE, IPinIP
• 而且这是一个比interface更底层的概念，更像Adj/NHG = fib entry, 或者via = ECMP adj 

(Z1446)

EOS.swi vs /mnt/flash issue

就是 mount /dev/sda1 /mnt/flash, 然后vi boot-config, show boot, 其他没啥

Loading an engineering EOS.swi, the dut stuck at "wait for /mnt/flash timeout". Somehow, this image failed to mount /mnt/flash. The easiest way is to mount manually and change /mnt/flash/boot-config to load up a stable image.

mkdir /mnt/flash
mount /dev/sda1 /mnt/flash
vi /mnt/flash/boot-config

Mlag reload-delay timers tunning

General rules:

• No.1 rule - side-way up first
• Peer-link up first. Recommended (if not must) to have an iBGP peering between MLAG peers. So always have a side-way. 
• Why need a side-way? Because there is for sure a gap between non-mlag (uplinks) and mlag (downlinks) ready to forward, in this case, there is a fire exit to avoid traffic drop. 
• No.2 rule - non-mlag relay < mlag delay
• "In a topology where non-mlag links are only L3 ports that want to reach hosts behind MLAG interface, then it is preferable to keep non-mlag reload-delay smaller compared to mlag reload-delay to have L3 protocols converge before the L2 links ( MLAG interfaces ) are brought up."
• Why? don't overwhelm the peerLink. Most of the time, servers's ingress << egress. 
• No.3 rule - if "lacp standby" enabled, mlag delay < non-mlag
• Bring up interfaces to allow hw programing. So non-mlag (uplinks) up first, N-S traffic in but downlinks not ready. 

Primary和Secondary最大的区别是STP，Sec上的STP is held.

MLAG reboot/SSO, all ports except peerlink are in err-disabled state。这个期间，系统需要bring up hardware，Mlag peer可以sync states.  From 4.15.2F (released Sep 2015), the default-relay is automatically adjusted based on platform, 1800s in Sand(Arad/J/J+) modular, 1200s in Strata(Trident/TH) modular and 300s fixed.

如果non-mlag links都是L3 ports，最好是non-mlag delay < mlag delay，所以uplink先起来，等Mlag converged了，S-N已经有routes ready了。但是这个时候，会Attract N-S traffic，但是我们peerlink已经up了，就走peerlink, no loss.

LACP-standby就是可以更新LAG成员表，MAC地址表，ports appear as linkActive. 如果这个是ON，那么Non-Mlag delay >= Mlag delay. 不如N-S traffic进来，以为Mlag ports up，就drop了

如果使用VRRP，就需要configure VRRP reload daly time higher than MLAG reload delay.

Z1554

Disable Mlag Peerlink

An EOS engineering command to shutdown the peerLink port-channel:

errordisable test interface po2000

FHRP, VRRP and VARP

VARP (Z636)

• "ip virtual mac <mac>" 
• = treat <mac> as own mac;
• 和MLAG合用=poor-man's VRRP，
• 好处是active-active，
• 坏处是static ARP on neighbor hosts，为啥坏啊？
• 如果SVI有"ip virtual addr"
• respond ARP req for vIP + vMAC, 但是srcMAC还是phyMAC;
• GARP, srcMAC = vMAC, 刷switch MAC table
• under bash, 还有command?
• varp vlan3 1.2.3.4 00:1c:73:00:00:01

VRRP over MLAG (Z1223)

• 传统上，MLAG最好的选择的是VARP，用VRRP is kinda dumb(DE's comments)
• 最大的问题是，VRRP Backup不fwd traffic，而且peerlink上不学MAC，结果哪？
• hash到backup的traffic，会被flood，连switch都没有，因为peerlink不学mac
• 纪录在Y31356
• Solution是write vrrp mac address into mlag host
• 还有mlag reload delay + VRRP
• peerlink先起来，vrrp prempt所以newly up peer becomes master
• 可是the new peer还在reload delay，black hole traffic!!
• 纪录在Y30494
• Workaround: config preempt delay reload #1 > reload-delay #2
• 这个Z1223最后没有做

FHRP: HSRP, VRRP and VARP

• HSRP, VRRP and VARP use vMAC. GLBP uses phyMAC for LB; 
• HSRP vMAC = 0000:0c07:ac**
• VRRP vMAC = 0000:5e00:01xx, xx = VRID (1-256)
• VARP vMAC = self-configured
• GLBP其实也用vMAC，应该是不同的vMAC = phyMAC
• Assigned MAC address (side note)
• 00-00-5e, IANA (internet assign num association) ucast
• 00-00-5e-00-01/02-xx, VRRP v4/v6
• 01-00-5e, IANA mcast
• 00-00-00 to 7f-ff-ff: v4 mcast
• 90-00-01: bfd on LAG
• 最大的区别是，Active-Active vs Active-Standby, 如何做到的？
• 都用vMAC, hosts send packets with dstMAC = vMAC
• VARP是active-active，直接route out. 
• 而VRRP是bridge to peer via peerLink 
• GARP是刷switch mac table + 通知全部的hosts, ip/MAC mapping of vMAC = vIP
• GARP和普通ARP Reply一样，只是dstMAC = FF or hostMAC
• 只有GARP pkt里面的srcMAC是vMAC. 这是唯一pkt!!. 其他data甚至ARP的srcMAC都是phyMAC. 
• 需要phyIP吗？
• 'ip virtual address' 无论有没有mask，都需要phyIP. 不过w/ mask可以是dummy ip. 
• 'ip address virtual' 不要phyIP
• VARP = ip virtual address  - IVA
• GARP和ARP Reply一样，srcMAC, arp.sndMAC = vMAC
• ARP request里面, 里外Eth/ARP都是Switch System MAC，arp.sndIP = phyIP，因为要确保Arp reply回到Src Mlag Peer!!!
• VARP w/mask = ip virtual address w/ mask - IVAM
• GARP + ARP Reply = VARP way
• 关键是ARP Req, 没有phyIP under this subnet, 所以里外都是vMAC/vIP. 好了有问题了，如果Host ARP Reply hashed到里外一个Peer，咋办？
• 所以这个VxLAN VARP必须有 ARP Sync!
• VxLAN Anycast = ip address virtual + vMac
• both Mlag peer都是一个Addr. 不需要phyIP, vIP就可以
• 没有GARP，为啥？因为ARP reply另外都是vMAC，不需要GARP刷switch
• host知道GW，必须靠ARP reply by mlag peer. 里外都是vMac + vIP. 
• ARP Req = VARP w/mask, 因为没有phyIP；
• 所以也需要ARP Sync. 

Python Tips

1. Is vs ==, mutable and immutable

• == is to compare value, is is compare address
• == is to compare by calling object.__eq__()
• list1 = list2, address is assigned, not value. need list1=list2[:]

>>> a = [1,2,3]
>>> id(a)
4463753048  <<<< address is 048
>>> a[0] = 11
>>> id(a)
4463753048
>>> a
[11, 2, 3]
>>> a.append(44)
>>> a
[11, 2, 3, 44]
>>> id(a)   <<<< can change item and append, address is same
4463753048
>>> b = a
>>> id(b)   <<<< list= is point to same address
4463753048
>>> b[0] = 111
>>> a
[111, 2, 3, 44]  <<<< change b = change a
>>> c = a[:]
>>> id(c)
4463919328
>>> c[0] = 1234
>>> a
[111, 2, 3, 44]
>>> b
[111, 2, 3, 44]
>>> c
[1234, 2, 3, 44]
>>>

2. keyword argument

>>> def ff(*args, **kwargs):
...     print args
...     print kwargs
...

>>> ff(1, '22', k1=333, k2='4444')
(1, '22')
{'k2': '4444', 'k1': 333}

>>> d = {'ka':'aaa', 'kb':123}
>>> ff(**d)
()
{'kb': 123, 'ka': 'aaa'}

3. Regexp \符号

https://docs.python.org/3/howto/regex.html

Remove全部的None-Alphanumeric char from string. 应该是

s = re.sub('\W', '', s)

还有记住以下的

\d, \D = [0-9]
\s, \S = [ \t\n\r\v\f]

\w, \W = alphanumeric,[0-9a-bA-Z_]

4. arstCli Script模版

from arstCliLib import *
import sys

dut = sys.argv[1]
openSshOnDut( dut )
setAccessMethod(dut, 'ssh')
cmd = ['show ip int brief | grep " 10\." | grep Vlan']
output = sendCmd(dut, cmd, prompt='enable', raw=True)

print '\n'.join(output)

[solomonyang@syscon] ~ $ python testCli.py
Traceback (most recent call last):
  File "testCli.py", line 4, in <module>
    from arstCliLib import *
ImportError: No module named arstCliLib

PATH=$PATH:$HOME/bin:$HOME/py
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/git/systest-infra/lib/
PYTHONPATH=$PYTHONPATH:$HOME/git/systest-infra/lib/

export PATH
export LD_LIBRARY_PATH
export PYTHONPATH

5. List vs Tuple, based on stackoverflow post

1. Literal/语法

>>> t = (1,2); l = [1,2]; t[1]; l[0]
2
1

2. Size/内存量，区别大概list +12%

>>> t = tuple(range(100000)); l =  list(range(100000)); t.__sizeof__(); l.__sizeof__()
800024
900088

3. Mutable vs Inmutable/可变 vs 不可变
>>> t = (1,2); l = [1,2]
>>> l[0] = 11; print l
[11, 2]
>>> t[0] = 11; print t
TypeError: 'tuple' object does not support item assignment

4. 都可以增加，但是不同，tuple += (3)是返回一个新的obj，而list.add()是改原来的
>>> t = (1,2); l = [1,2]; id(t); id(l); t+=(3,); l+=[3]; id(t); id(l); l.append(3); id(l)
4306859000 <<<<< t
4307014720 <<<<< 1, 都一个地址
4306913952 <<<<< t +=(3,)，新地址
4307014720

4307014720

5. 因为不可变，所以tuple (1,2)可以是Dict[key]
>>> d = {}; d[t] = '1,2'
>>> d = {}; d[l] = '1,2'
TypeError: unhashable type: 'list'

6. 用途
比方说，(10,11)是个bookmark，第10页的第11行，一般没有必要改；而list of bookmark = [ (1,10), (10,11), (22, 1)] 

7. Tuple != constant list
http://news.e-scribe.com/397
但是这么了解哪，好像也撮合:-) 好像更多的是字面的了解，tuple = lightweight record，比方说：DB API's fetchmany() 返回的是List of tuple. 每个tuple是一个record，不能改其中一项，改了就没有意义了。

List vs Set

1. Literal/语法/内存量
>>> s = set(range(100000)); s.__sizeof__(); l = list(range(100000)); l.__sizeof__()
4194504

900088

2. Set没有Index, 没有重复，可以数学操作 &, -, ^
>>> s1=set(range(1,20,2)); s2=set(range(10,30)); print 's1->', s1; print 's2->', s2; print '-:', s1 - s2; print '&:', s1&s2; s1^s2
s1-> set([1, 3, 5, 7, 9, 11, 13, 15, 17, 19])
s2-> set([10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29])
-: set([1, 3, 9, 5, 7])
&: set([19, 17, 11, 13, 15])

set([1, 3, 5, 7, 9, 10, 12, 14, 16, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29])

ISIS Segment Routing

CCIE SR discussion

• 说起来不难，就是用IGP(ISIS)来signal labels，而不是LDP或者RSVP，更简单
• 但是还是用MPLS，所以软件升级就可以了
• 有Node Segment和Adj Segment，Ingress Router构建Label Stack就可以控制LSP

Arista MPLS SR

• 网络的难点是：
• 如何Classify traffic 和 Engineer path. 
• SDN如何control traffic path/flow，特别是native solution for ipv6
• MPLS解决了一些，但是太复杂，例如TE，而且还没有ipv6 native support. 
• Segment是Arista认为的solution
• SR operation:
• 网络分成Segment, 给一个SID - segment id. 利用BGP/ISIS/OSPF extension来distribute. 而不在需要Label protocols - LDP/RSVP
• SID有Global Unique或者Local significant, Base + Index
• 3种 global SID: 
• 1) prefix;  
• 2) node; loopback of node
• 3) anycast; loopback shared by a set of routers, ecmp
• Prefix SID: 全网都是same value, 非常关键，reducing DP state;
• 图上的例子是, 所有的router SRGB(SR Global Block)都是900,000-965,535, Rtr5's 5.5.5.5/32, prefix-SID是10，所有的router全部assign label 900,000+10 = 900,010
• Adj Sid是locally significant, 只给neighbor，只installed at neighbor
• 最简单应用，用到ECMP，ingress LER就push一个SR label. 或者是push a set of label
• SR vs LDP:
• 相同的地方：
• easy configured, "plug and play"
• Both form stateless Mp-to-pt LSP
• 不同：
• LDP全是local signficant label, SR是global unique labels, 减少DP state; scale 好；
• SR有TE，LDP TE没有流行，v6也没有
• SR vs RSVP-TE
• RSVP-TE的特点：
• constraints routing like b/w, shared link risk group and explicit paths，可以不按照IGP shortest path;
• 有b/w
• FRR，有pre-computed backup paths
• 不好的地方：
• full-mesh p2p TE tunnels, 没有ECMP
• failure后有churn，需要re-signaling. 
• scalability issue，所以不那么流行
• SR利用SDN，在head加入stack of mpls label，而不需要中间router纪录state。还有scale
• 什么都好，没有Multicast?
• Arista SR:
• 必须是R-series with FlexRoute?
• 三种SR Solutions:
• Static MPLS push + NHG
• 在ingress LER上，configure a route pointing to a label stack via CLI;
• ECMP = multiple tunnels
• Class-based service policy 
• Easy start
• Controller based using Eos SDK
• BGP-LU (labeled unicast/RFC 3107)
• 应用：Cloud-WAN, CDN, NFV

Arista ISIS SR (from 4-17-0f)

Tutorial: Segment Routing

https://www.youtube.com/watch?v=BEo5MdB3o3Y

Take-away points:

• SR中最重要的是TE
• Segment (1-hop or n-hop) vs SID
• SR是hybrid TE approaches: path info in packet and network
• CSPF - attribute(b/w, color), requirement and shortest path
• CP = controller

What's SR?

• A tunneling tech - nothing fancy
• A TE! - steer packet instead of routing path. ~= RSVP TE

Terminology:

• Domain
• SR path
• Segment = SID, can be multiple hops

Traditional TE Approaches

• path info in packet
• like IPv4 strict routing option
• path info in the network
• RSVP-signaled MPLS

SR is hybrid TE approach

• Segment types:
• 1 router hop
• multiple router hops
• Types:
• Adjacency (IGP adjacency, 1-hop)
• Prefix = IGP least cost path to a prefix

SR Encapsulation

• MPLS
• Label = segment
• IPv6:
• Segment Routing Extension Header (SRH)
• A list of ipv6 addr
• each ipv6 addr = a segment

Local Labels

• Some SIDs have node-local significance
• Adjancency
• Why important
• Stack can be too big, ASIC cannot handle
• MTU

Global Labels

• Some have domain-wide significance
• Each node reserves a block of labels. 
• SRGB base

IPv6 forwarding

• SRH, segment routing hdr
• A list of ipv6 address as SID

Binding SIDs

• Pointing to another SR paths or tunnels
• Reasons:
• Label stack
• MTU size too big

Control Plane

• Path computation: on SR ingress or central controller. 

CSPF, 这个讲的清楚！

• Attributes to segment: color, b/w, SRLG
• Req to each path
• Shortest path meeting req.
• LSDB, TED = extension to carry info. 
• Alternative path to protect
• All info in LSDB, no need for RSVP or LDP

SR convergence after failures

• Fast recovery by IGP reconvergence. 
• TI-FLA, speed up convergence if not fast enuf
• Use anycast SID (ecmp)

SR benefits:

• If using SR to reserve b/w, MUST go for controller
• Central controller has global view 

Controller Protocol options:

• pull LSDB:
• controller: a passive mbr in IGP
• BGP-LS
• push segment list to ingress SR
• PCEP
• BGP
• push policy
• binding what traffic to which path
• PCEP or BGP

Conclusion:

• SR moves state from network to packet - simplified
• Some open issues: OAM, Fast Reroute
• Need  experience