Arista MPLS VPN Step by Step (1e) MPLS L3VPN PE/CE ISIS

Now it is the PE/CE routing option - ISIS. Here is the simplified topology



PE1 Configuration:

router bgp 65001

   !

   vrf pe1-isis

      rd 100:115

      route-target import vpn-ipv4 100:115

      route-target import vpn-ipv4 200:115 << import PE2's routes

      route-target export vpn-ipv4 100:115 << pe1=100:115, pe2=200:115

      maximum-paths 2 ecmp 2

      redistribute connected

      redistribute isis level-2

!

router isis isis-pe1 vrf pe1-isis

   net 47.0001.0011.0011.0011.00

   is-type level-2

   redistribute bgp

   !

   address-family ipv4 unicast

CE Configuration:

router isis isis-ce1 vrf ce1-isis

   net 47.0001.0001.0001.0001.00

   is-type level-2

   !

   address-family ipv4 unicast

Ping check:

CE1(vrf:ce1-isis)#sh ip route
.......
 C       100.100.115.0/24 is directly connected, Vlan104
 I L2    100.250.115.0/24 [115/20] via 100.100.115.11, Vlan104
 C       100.255.115.0/24 is directly connected, Loopback104
 I L2    200.200.115.0/24 [115/10] via 100.100.115.11, Vlan104
 I L2    200.250.115.0/24 [115/10] via 100.100.115.11, Vlan104
 I L2    200.255.115.0/24 [115/10] via 100.100.115.11, Vlan104

CE1(vrf:ce1-isis)#ping 200.250.115.22 << ping pe2's loopback
....
--- 200.250.115.22 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.117/0.181/0.335/0.083 ms, ipg/ewma 0.243/0.253 ms

CE1(vrf:ce1-isis)#ping 200.255.115.2  << ping ce2's loopback
.....
--- 200.255.115.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.120/0.156/0.291/0.069 ms, ipg/ewma 0.214/0.221 ms

CE1(vrf:ce1-isis)#ping 200.200.115.2 << ping pe2-ce2's ip @ ce2
....
--- 200.200.115.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.120/0.153/0.274/0.061 ms, ipg/ewma 0.210/0.211 ms

CE1(vrf:ce1-isis)#ping 200.200.115.22  << ping pe2-ce2's ip @ pe2
.....
--- 200.200.115.22 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.115/0.151/0.253/0.053 ms, ipg/ewma 0.208/0.199 ms

Arista MPLS VPN Step by Step (1d) MPLS L3VPN PE/CE OSPF

In this post I will discuss the PE/CE routing option - OSPF. Here is the simplified topology



PE Configuration:

router bgp 65001
....
   vrf pe1-ospf
      rd 100:110
      route-target import vpn-ipv4 100:110
      route-target import vpn-ipv4 200:110
      route-target export vpn-ipv4 100:110
      maximum-paths 2 ecmp 2
      redistribute connected
      redistribute ospf
!
router ospf 100 vrf pe1-ospf
   router-id 88.255.11.11
   passive-interface Loopback103
   redistribute bgp
   network 100.100.110.11/32 area 0.0.0.0
   network 100.250.110.11/32 area 0.0.0.0
   max-lsa 12000

CE Configuration:

router ospf 100 vrf ce1-ospf
   router-id 100.250.100.100
   passive-interface Loopback103
   network 100.100.110.1/32 area 0.0.0.0
   network 100.255.110.1/32 area 0.0.0.0
   max-lsa 12000

Arista MPLS VPN Step by Step (1c) MPLS L3VPN PE/CE eBGP

Following the previous blog, in this post I will discuss the PE/CE routing option - eBGP. Here is the simplified topology

PE Configuration

router bgp 65001

   router-id 88.255.22.22

   neighbor vpnv4-rr peer-group

   neighbor vpnv4-rr remote-as 65001

   neighbor vpnv4-rr update-source Loopback88

   neighbor vpnv4-rr send-community standard extended

   neighbor vpnv4-rr maximum-routes 12000 

   neighbor 88.255.1.1 peer-group vpnv4-rr

   neighbor 88.255.2.2 peer-group vpnv4-rr

   address-family vpn-ipv4

      neighbor vpnv4-rr activate

      neighbor default encapsulation mpls next-hop-self source-interface Loopback88

   !

   vrf pe2-ebgp

      rd 200:20

      route-target import vpn-ipv4 100:20

      route-target import vpn-ipv4 200:20

      route-target export vpn-ipv4 200:20

      maximum-paths 2 ecmp 2

      neighbor 200.200.20.2 remote-as 65500

      neighbor 200.200.20.2 maximum-routes 12000 

      redistribute connected

CE configuration is as below. In my lab, I use vrf on CE router to segregate and simulate different CEs. 

router bgp 65500

   router-id 100.255.100.100

   !

   vrf ce1-ebgp

      rd 100:20

      neighbor 100.100.20.11 remote-as 65001

      ! need alllowas-in to permit remote prefix with same AS#

      neighbor 100.100.20.11 allowas-in 3

      neighbor 100.100.20.11 maximum-routes 12000 

      redistribute connected

Arista MPLS VPN Step by Step (1b) MPLS L3VPN PE/CE Static

In this blog, I will discuss the PE/CE routing option - static. Here is the simplified topology

The configuration is quite simple. 

router bgp 65001
   router-id 88.255.11.11
   neighbor vpnv4-rr peer-group
   neighbor vpnv4-rr remote-as 65001
   neighbor vpnv4-rr update-source Loopback88
   neighbor vpnv4-rr send-community standard extended
   neighbor vpnv4-rr maximum-routes 12000 
   neighbor 88.255.1.1 peer-group vpnv4-rr
   neighbor 88.255.2.2 peer-group vpnv4-rr
   !
   address-family vpn-ipv4
      neighbor vpnv4-rr activate
      neighbor default encapsulation mpls next-hop-self source-interface Loopback88
   !
....
   vrf pe1-static
      !! pe/ce: static, intf:vlan101+lo101, addr:100.1.11.0/24+100.1.100.0/32
      rd 100:1
      !
      ! PE1 ex/import 100:1, and import PE2's 200:1 
      !
      route-target import vpn-ipv4 100:1
      route-target import vpn-ipv4 200:1
      route-target export vpn-ipv4 100:1
      maximum-paths 2 ecmp 2
      redistribute connected
      redistribute static

So on PE1, we can 6 vpn-ipv4 routes (doubled by 2 RRs)

PE1(config)#sh bgp vpn-ipv4 rd 200:1

BGP routing table information for VRF default

Router identifier 88.255.11.11, local AS number 65001

Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, e - ECMP

                    S - Stale, c - Contributing to ECMP, b - backup

                    % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

         Network                Next Hop            Metric  LocPref Weight  Path

 * >     RD: 200:1 IPv4 prefix 200.200.1.0/24

                                88.255.22.22          -       100     0        i Or-ID: 88.255.22.22 C-LST: 88.255.1.1

 *       RD: 200:1 IPv4 prefix 200.200.1.0/24

                                88.255.22.22          -       100     0        i Or-ID: 88.255.22.22 C-LST: 88.255.2.2

 * >     RD: 200:1 IPv4 prefix 200.250.1.0/24

                                88.255.22.22          -       100     0        i Or-ID: 88.255.22.22 C-LST: 88.255.1.1

 *       RD: 200:1 IPv4 prefix 200.250.1.0/24

                                88.255.22.22          -       100     0        i Or-ID: 88.255.22.22 C-LST: 88.255.2.2

 * >     RD: 200:1 IPv4 prefix 200.255.1.0/24

                                88.255.22.22          -       100     0        ? Or-ID: 88.255.22.22 C-LST: 88.255.1.1

 *       RD: 200:1 IPv4 prefix 200.255.1.0/24

                                88.255.22.22          -       100     0        ? Or-ID: 88.255.22.22 C-LST: 88.255.2.2

Arista MPLS VPN Step by Step (1a) MPLS LDP L3VPN Configuration

Above is the topology, in which we have:

• 2 PE + 4 P routers, all full mesh connections (we will leverage this setup for later configurations)
• IGP: OSPF process 71
• iBGP: AS 65001, peering with loopback 71 (71.#.#.#/32, # is the router-id)
• MPLS LDP is enabled on all interfaces

P router configuration:

mpls ip
!
mpls ldp
   router-id 88.255.1.1
   transport-address interface Loopback88
   fec filter prefix-list fec-pfx-list
   no shutdown
!
router bgp 65001
   router-id 88.255.1.1
   maximum-paths 4 ecmp 4
   neighbor pe peer-group
   neighbor pe remote-as 65001
   neighbor pe update-source Loopback88
   neighbor pe route-reflector-client
   neighbor pe send-community standard extended
   neighbor pe maximum-routes 12000 
   !
   ! RR configuration, peering to PE1/PE2
   !
   neighbor 88.255.11.11 peer-group pe
   neighbor 88.255.22.22 peer-group pe
   !
   address-family vpn-ipv4
   ! active vpn-ipv4 AF
      neighbor pe activate
!
router ospf 88
   router-id 88.255.1.1
   passive-interface Loopback88
   network 88.0.0.0/8 area 0.0.0.0
   max-lsa 12000

PE router configuration:

mpls ip
!
mpls ldp
router bgp 65001
   router-id 88.255.11.11
   neighbor vpnv4-rr peer-group
   neighbor vpnv4-rr remote-as 65001
   neighbor vpnv4-rr update-source Loopback88
   neighbor vpnv4-rr send-community standard extended
   neighbor vpnv4-rr maximum-routes 12000 
   neighbor 88.255.1.1 peer-group vpnv4-rr
   neighbor 88.255.2.2 peer-group vpnv4-rr
   !
   address-family vpn-ipv4
      neighbor vpnv4-rr activate
      neighbor default encapsulation mpls next-hop-self source-interface Loopback88

For the full router configuration, please go to the directory. 

Arista MPLS VPN Step by Step (0)

In this step-by-step series, I will cover the Arista MPLS VPN solution. All of the configuration is based on the following topology. For the technical details, please refer to "Arista MPLS IPv4/v6 MPLS VPN Configuration Guide" 

And some other references:

• Arista Software TOI - click TOI
• 4-15-0F:
• MPLS Label Encap
• Static MPLS Label Pop/Swap (T2)
• 4-17F:
• MPLS LDP (4-17-0F)
• ISIS Segment Routing
• MPLS over GRE
• LDP Pw
• 4-18-1F:
• Static MPLS Label Push (T2/TH)
• 4-20F:
• L3 EVPN MPLS
• LDP Pw type 5
• BGP LU/SR
• L2 EVPN MPLS
• 4-21F:
• MPLS Traceroute
• Inter-VRF Route Leaking
• Arista MPLS Manual
• Arista EVPN Manual

OSPF Fast Convergence

This blog is the best article on this subject, maybe a little old but very informational. 

Some takeaway notes:

• IGP has micro loop during convergence, because FIB update from near to far，EIGRP is an exception - explicit route due to diffusing computation. 
• Convergence = Detection + Propogation + SpfComp + FibUpdTime
• 1. Detection:
• Don't use IGP KA, instead use hw failure detection, like p2p Ethernet. 
• No debounce timers. SONET is 例外，有50 ms protection, can set 50ms+
• If P2P, OSPF does not need DR elections to save time. And no type2 LSA, reduces LSDB size
• If no hw failure detection, have to use fast protocol hello, like ospf fast timer - "ip ospf dead-interval minimal hello-multiplier 5" = 5 hello/sec
• Can do 1-sec failure detection, but cpu load is high
• Use BFD, workload is off to LC. 
• 2. Propagation:
• Out: ISIS/OPSF, using LSA flood. IGP has throttling mechanism, like Cisco has exceptional backoff algorithm. so set initial as low as 5-10 ms. don't use 0, because multiple link down event better in ONE LSA. 
• In: should be ok
• Processing: 
• ISIS has fast-flood, flood before SPF processing.
• Timer pacing flood, the interval between LSA. 
• OutIfQueue: Need Qos to reserve b/w for CP pkts
• Packet propagation: small not serious
• 3. SPF Computation
• Used to be a big impact before, but not now thanks for fast hw and some algorithm improvement like iSPF (incremental)
• 4. Fib update
• This stage contributes the MOST to convergence time
• 2 ways to speed up: 1) less prefixes; 2) sequence important first
• Edge routers don't need core transit prefixes. So, 
• (1) use ip unumbered so to put loopback0; 
• (2) ospf 2 LSA for p2p links, type-1 router; and type-3 network. "ip ospf prefix-suppression", only process type 1, no type-3
• (3) ISIS has priority driven installation

Arista EOS Hashing Features

Here are some Arista EOS hashing features: 

• 4.15.2F, hashing on ttl and hop-limit
• Trident2-based systems: 7300X, 7050X, 7250QX. All have 2 hash engines, 1 for port-channel, 1 for ECMP
• CLI: 
• port-channel load-balance trident fields ip ttl 
• port-channel load-balance trident fields ipv6 hop-limit
• show port-channel load-balance trident fields
• IP TTL hashing is ON
• Unlike other "port-ch load-balance trident fields" commands, this one applies both to both LAG and ECMP hash engines. 
• Be care of unexpected behavior,  for example,  
• Internet traffic goes to proxy or load-balance servers.
• But somehow traffic going a different path results into different TTL values, like VRRP or MLAG. 
• So if hashing based on ttl, the same traffic stream could end up at different destinations. 
• Hashing egress interface: 
• 4.17.0
• show load-balance destination ingress-interface Ethernet1/1 src-ipv4-address 10.0.0.93 dst-ipv4-address 17.0.0.11 ip-protocol 17 src-l4-port 11001 dst-l4-port 19872
• Output Interface: Port-Channel102
• Disable Ingress interface in hashing
• Arad/Jericho based systems: 7500E/R, 7280E/R
• CLI: 
• no ip load-sharing sand fields ingress-interface
• To avoid polarization, use different preset in different layers:
• ip load-sharing sand has preset <0-2>
• 好像是，不同interface会有不同random#
• 这个for ECMP
• LAG hashing on ingress interface
• 和上面差不多，区别是config放在LB profile里面
• for LAG
• Global LAG hashing profile
• Arad/Jericho, Sand-based systems
• CLI:
• load-balance policies
• load-balance sand profile myProfile
• hash seed 50
• Dynamic and Symmetric LAG Hashing
• From 4-15-0
• Arad/Jericho, Sand-based systems
• Bidirectional flows go thru same member port of LAG
• 做法不难，就是异或源和目标地址，再求哈希，当然LB就是不好
• CLI: 
• load-balance policies
•    load-balance sand profile sym
•    fields symmetric-hash

Arista L3 Leaf/Spine Step by Step (3a) - Vxlan Routing, Modes

The following couple of posts will cover the VxLAN routing. 

What's and why VxLAN Routing? With VxLAN bridging, the hosts in the same VLAN 10 can talk to each other. But we need VxLAN routing to have VLAN 10 address to reach the hosts in VLAN 11. 

How to enable VxLAN Routing? Creating a SVI interface on a VLAN which is associated to a VNI

We can achieve this in the following 5 modes:

• Router on a stick - Gateway only on dedicated routers
• Direct Routing - Gateway on all VTEPs
• Indirect Routing - Gateway only on few dedicated VTEPs
• Naked Routing - Regular routing w/o VxLAN encap
• * Hybrid - Combination of direct/indirect and naked routing based on case

Unbalanced Hashing vs DLB vs RECMP

Broadcom introduced Dynamic Load Balancing in Trident 3 product. 

https://www.youtube.com/watch?v=1y25OTyFFrs

里面提到了Unblanced Hashing的3个Reasons:
1. Elephant Flow
2. Link Failures, Why?
3. Hash Polarization 

可能based在MIT一个flowlet paper

• 这个原理不难，多根线路，计算Maximum和Minimum Delay，之差就是一个timeout value;
• 如果后续packet timeout之后到，就不必hash到固定的线路，而是严格Load Balancing. 

一个相关的Arista EOS feature - Resilient ECMP

• 说起来很简单，就是一根线路断了，不要re-hashing区别的sessions，这样会teardown TCP. 而是just re-hash 故障线路上的session
• CLI: ip hardware fib ecmp resilience 10.14.2.0/24 capacity 6 redundancy 4
• 解释:
• network 10.14.2.0/24有6个NextHop
• 但是建立一个24 Entries的ECMP
• 如果其中一个NH断了，就把表中这个NH用其他NH替换，其他ECMP NH没有变化
• Redundancy = 4，应该就是一根断了，有4个来LB

TCP Notes (2)

1. 如何计算 a TCP segment length?
A: ip.len - ip.hdr_len - tcp.hdr_len

2. TCP Flags
* 最常见的是SYN, ACK, FIN
* URG - 比方说，sender希望receiver stop，无论queue里面还有多少data. 
* PSH - 和URG很像，必须high priority的处理
* RST - 本意是，如果port没有open，host就返回一个RST，容易被hacker利用
* SYN - 只有3-way handshake时有
* FIN - FINISH，4-way close? ACK/FIN-ACK-ACK/FIN-ACK

3. TCP window size
* len = 2 Bytes, max = 64K, in Byte
* 还有一个window scale，就是window size左移多少位
* SYN packet里面没有scaled
* 两边可以有不同的scale#, 不需要negotiated

这个是SYN Packet


后面的Data Packet


4. TCP checksum
看下面的图，前面加上IP Header Info, srcAdr, dstArd, reserved, proto#, tcpLen(computed), + tcpHdr, tcpData. 可以cover ip header integrity



5. TCP Timestamps
10-Byte TCP option, type(1B=8); len(=10), TS(len=4), TSEcho(len=4)，例如：

1st packet, TS = 2216538



2nd packet, TSEcho = 2216538. 这样就可以计算RTT了



6. TCP MSS vs IP MTU

这个MSS有点confusing，因为Segment一般是指Header + Data，但是这个TCP MSS说的是Max Data Leng. 这个图很清楚，
* Ethernet Frame size = 1518, 
* IP MTU = 1500 = 1518 - 18, 
* TCP MSS = 1460 = 1500 - 20 (ip.hdr_len) - 20 (tcp.hdr_len)



Reference:
1. firewall.cx
2. tcpipguide.com

Link-debounce timer

https://www.manualslib.com/manual/546511/Dell-Force10-S4810p.html?page=501

Delay the notification of interface up/down events. 主要是防止快速而重复的interface flaps，对上层协议的影响。时间是milisec. 

Configuration:

bn303.spine.host3(s1)(config-if-Et3/1/1)#link-debounce time 1000 1000
bn303.spine.host3(s1)(config-if-Et3/1/1)#int e3/1/1
bn303.spine.host3(s1)(config-if-Et3/1/1)#link-debounce time 1000 1000
bn303.spine.host3(s1)(config-if-Et3/1/1)#

TCP Notes (1)

How TCP Works @ youtube

• TCP fundamentals Part 2
• 2 related TCP windows: 
• CWND - Congestion Window
• RWND - Receive Window
• Only RWND advertized in packets
• minimum of CWND and RWND is the cap of transmission
• CWND = n * MSS, n = 1,2,4...., slowstart
• not advertised, changing
• Example 1, slow backup between 2 local servers
• from the trace, recvr to sender 只发ACKs, (len = 60) 
• window size 每隔一段时间，会从64K一直降到1000 (小于MSS). 然后hold for 190ms. 然后window size back to 64K
• 显示chart - statistics -> TCP stream graph -> Time sequence (tcptrace)
• 足够的证据是server side问题
• Example 2, 是sender application无法fill CWND,
• Example 3, ip tcp mss-adjust问题
• server/client, 2个pcap
• client直接看到seq = 1461, 不是1
• server一直re-send seq = 1, size = 1460 pkts. 
• TCP NoOP option - 很简单，#TcpLen * 4 = Tcp Header Len，而TCP Options Len不全是4的倍数，所以TCP Header需要NoOp pad out header size to a multiple of 4B. 
• TCP option - timestamp
• TSval, EchoTSval, 为了计算RTT and RTO (retrans timeout)
• Some TCP options:
• 0 - end of TCP options
• 1 - NoOP
• 2 - MSS
• 3 - Window Scale
• 4/5 - SACK
• 8 - timestamp

Wireshark Tips (2)

packetbomb.com - troubleshooting MTU issue

• Following the previous post, need column - tcpLen, Seq, nextSeq, Ack, BytesInFlight
• Statistics -> conversation, find the biggest stream 
• couple of TCP retransmission packets in jumbo frame with DF bit, because no ACK
• later sender starts with 512B -> 1024B, then conversation on. 
• clearly, 1 or multiple middle routers can't handle jumbo frames
• filter = icmp, doesn't show any icmp unreachable
• per RFC 1191 - path MTU discovery. The middle routers should return a ICMP destination Unreachable message with code = "frag needed and DF set"
• So these pkts are dropped by either firewall or disabled on the router

Wireshark Tips (1)

Quick refresh:

• Where to capture有讲究
• Capture on host machine不好，不是真正packets on the wire
• Wireshark setup: 
• 必须有time delta, 
• TCP: seq + tcpLen = nextSeq, ACK
• TCP: winSize，必须有，scaleNum in 3-way handshake
• TCP: byteOnTheFlight

Sources:

• A youtube video - Using Wireshark to Solve Real Problems for Real People (by Kary Rogers)
• packetbomb.com 
• packetbomb.com's "Fundamental Course"

Takeaways:

• Packet capture:
• Where matters
• Close to the source of complaint, capture all traffic w/o filter. If doing this in the network, have to figure out a smart way because of too much traffic. 
• Near the servers, if out of the domain, on the edge where the traffic leave
• Both if possible
• How to capture:
• On the client machine: NOT a good idea
• wireshark, tcpdump, windump
• easy but not the real traffic on the wire, why?
• application -- kernel's tcp/ip stack -- pcap (where is wireshark) -- NIC -- network. 
• for example, the packet before segmentation
• SPAN/Mirror
• TAP
• Hub??
• Wireshark setup (on my MAC wireshark 2.14)
• Wireshark -> Preference
• Layout
• Name Resolution: MAC, protoName, IP addr. 
• Protocol -> TCP
• Column:
• [] means data not in packet but calculated 
• 1. Add [time delta from previous displayed frame] to column
• right-click == apply as column
• 2. Add TCP's seq, Len, nextSeq, ACK to column. 
• Easy to see if losing a packet
• 3. Add TCP's windowSize
• Scale # is ONLY in 3-way handshake, x 256
• 4. Add "Bytes in flight"

EOS - CPU monitor session

monitor session 1 source Ethernet3/1
monitor session 1 destination Cpu

wa463.vtepB2(config)#show monitor sess

Session 1
------------------------
Source Ports:
  Both:        Et3/1
Destination Ports:
    Cpu :  active (mirror0) <<< use this interface to tcpdump


wa462.vtepB1#bash tcpdump -nvvi mirror0 udp
tcpdump: listening on mirror0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:56:32.121346 44:4c:a8:a5:11:40 > 44:4c:a8:97:8c:51, ethertype IPv4 (0x0800), length 164: (tos 0x0, ttl 63, id 47642, offset 0, flags [DF], proto UDP (17), length 150)
    200.255.200.34.62729 > 200.255.200.56.4789: VXLAN, flags [I] (0x08), vni 2000
00:1c:73:ff:3c:30 > 44:4c:a8:97:72:b7, ethertype IPv4 (0x0800), length 114: (tos 0x0, ttl 64, id 32537, offset 0, flags [none], proto ICMP (1), length 100)

Arista EOS Tips - alias

1. Multiple line alias
alias ccc
10 clear arp
20 clear mac add dyn
30 bash clear

2. Alias argument

alias da bash tcpdump -nvvvi %1 arp

Arista EOS: VARP - GARP

If you have the following VARP configuration:

ip virtual-router mac-address 00:00:33:33:44:44
ip virtual-router mac-address advertisement-interval 1
interface Vlan2000
   ip address 20.0.134.14/16
   ip virtual-router address 20.0.134.1

You will see the incoming GARP every 1 sec (collecting by "bash tcpdump -nvvi vlan2000 -w arp.cap")



Arista EOS

• Use opcode=2 (reply) as GARP 
• SrcMAC = vMAC, DstMAC = all FF
• Send/Target MAC/IP = vMAC/vIP
• You see 2 GARP in 1 sec because of 2 mlag peers 

FB: A Billion user load balancer

https://www.usenix.org/conference/lisa16/conference-program/presentation/shuff

• A 2016 presentation by PE - Patrick Shuff
• Tb of egress traffic leaving FB routers, 85% from outside US

FB Req Flow vs Architecture

• Client - DNS AAAA req for facebook.com
• Client - Router
• Router - ECMP to L4LB(ipvs)
• L4LB(ipvs) - Http GET to L7LB(proxygen)
• L7LB(proxygen) - Http GET to HHVM (websever)
• Router + L4LB + L7LB + HHVM = a cluster/DC
• Data flow:
• router/ECMP --> L4LB (ipvs) --> L7LB(proxygen) --> HHVM
• L4LB = 10+             
• L7LB = 100+
• HHVM = 1000+
• And L4LB/L7LB/HHVM are NOT dedicted servers or devices. All x86 servers dynamically allocated. 

L4/L7 LB:

• L4LB (ipvs) + xBGP, a python BGP dameon to talk to TOR to announce 
• ipv4:/32, 
• ipv6:/64
• Router to L4LB, just ECMP hash
• L4LB to L7LB, hash + state table
• 要考虑两种failures
• 1) L4LB down, 新的L4LB用同样的hash，到原来的HHVM;
• 2) L7LB down, TCP breaks, L4LB hash到不同的L7LB, 如果L7LB back up, 因为有state table，不会影响TCP
• DSR = direct server access
• return traffic bypass the L4LB, L7LB to router. 
• L4LB ==ipInip== L7LB
• 然后L7LB直接 original ipv4 packet back to client. 和微软很像

Speed up 

• 用PoP early terminates客户TCP/SSL handshake，这样加快链接建立的速度，而不是到DC
• Pop has direct SSL connections to DC. 
• 需要看TCP (3-way), SSL (4-way)

DNS

• Real time monitor

Arista DCS-7050QX-32S-F enable eth1-4

On the DCS-7050QX-32S-F, there are 4x10G SFP on the front panel. This 4 ports share the hardware with Eth5/1-4. So to switch the hardware of eth5/1 to eth1-4 of 10G, use this command

ck475.vtepX4(config)#hardware port-group 1 select ?
  Et1-4    Activate ports Et1-4
  Et5/1-4  Activate ports Et5/1-4

Arista EOS MLAG - domain name must be matched

snp262.vtepA1#sh run | sec mlag config
mlag configuration
   domain-id vtepA
   local-interface Vlan4094
   peer-address 200.255.134.2
   peer-link Port-Channel2000

snp263.vtepA2(config-mlag)#sh run sec mlag config
mlag configuration
   domain-id vtepA2  <<<<< a typo in mlag domain-name
   local-interface Vlan4094
   peer-address 200.255.134.1
   peer-link Port-Channel2000
   reload-delay mode lacp standby

snp263.vtepA2#sh mlag det | grep State
State                           :            inactive
Peer State                      :             unknown
State changes                   :                   3

snp263.vtepA2(config)#mlag config
snp263.vtepA2(config-mlag)#domain-id vtepA
snp263.vtepA2(config-mlag)#end

After correcting it, the mlag is up

snp263.vtepA2#sh mlag det | grep State
State                           :             primary
Peer State                      :           secondary
State changes                   :                   4

Arista EOS BGP wait-for-install and wait-for-convergence

"update wait-for-install"

Quite straightforward feature, update the prefixes until they are installed in hardware

• Obviously, this prevents packet loss. Without this feature, routers start to advertise prefixes when hardware forwarding not ready yet. When its peers start to flood traffic to it, the only thing it can do is to drop them. 
• Very useful when working with MLAG, which requires the routers to put Mlag downstream interface to hold state during initialization. 

"update wait-for-convergence"

• https://aristaeos.blogspot.com/2018/03/arista-eos-bgp-update-wait-for.html
• This is a feature for spine routers. The key point behind it is, don't advertise prefixes until fully convergence (which means you have full routing info). 

Where to use them:

• On MLAG or L2/L3 boundary,  ONLY use wait-for-install
• On spine, use both

FB: Eng Egress with Edge Fabric

https://research.fb.com/wp-content/uploads/2017/08/sigcomm17-final177-2billion.pdf?

PR's BGP connection types:

• Transit: private link with dedicated b/w
• Peers:
• private peer: dedicated PN ?~= transit ?
• public: via public fabric
• route server: prefix redirected by RS and traffic via public fabric

How prefixes are preferred:

• Prefer peer routes over transit (via local_pref), as_path tiebreaker
• In/egress traffic over same path
• If still tie, private peer > public > route server peer, using MED
• to avoid cross-congestion over fabric

BGP multi-path vs. ECMP

• Cisco BGP multipath doc
• Requirement of path characteristics to be multipath
• Weight
• local_pref
• as_path length
• origin
• MED
• one of these:
• neighbor AS or sub-as
• as_path

BGP limitation

• Not capacity-aware + ECMP
• unbalances links get equal load
• Static bgp policy likely optimizes traffic, but
• as-path != performance

Avoid congestion

• Input:
• Prefix via BMP, BGP only has 1 best
• controller does best selection
• sFLOW, IPFIX, traffic info
• SNMP, interface info
• Output, via BGP update by using higher local_pref

Performance-based routing

• servers set DSCP
• PBR on PRs, 1 DSCP - 1 route (table?)
• PR ISIS-SR/MPLS to ASW
• eBPF - extended berkley packet filter
• change pkt egress this server

to be continued.....

How to check # of BGP attributes

Check BGP export/shadown table

First enable "sflow extension bgp" then run command - "show bgp export-table path-attribute" or "show bgp shadown-table bgp-attribute" if older releases

bn302.17:01:15#show bgp export-table path-attribute
Bgp Attribute Info Table
key             nextHop  origin  pathFlags  originatorId  aspType  med  localPref  asPathAttrInfoId  commListId  extCommListId
 64        210.100.22.1       0         16             0        2    0          0                 3  4294967295     4294967295
 80  2000:210:255:254::       0          0    3539992321        1    0        100        4294967295  4294967295     4294967295
 88        210.100.11.1       0         24    3539992321        1    0        100                 2  4294967295     4294967295
 56        210.100.22.1       2         16             0        2    0          0                 2  4294967295     4294967295
 20  2000:210:100:32::1       0         48             0        2    0          0                 1  4294967295     4294967295

A new command:
bn302.17:01:08#bash smash -p ar/Smash/routing/bgp/export/pathAttrTableInfo/default/pathAttrEntry | egrep '^bucket' | wc -l
23