OSPF Fast Convergence

This blog is the best article on this subject, maybe a little old but very informational. 

Some takeaway notes:

• IGP has micro loop during convergence, because FIB update from near to far，EIGRP is an exception - explicit route due to diffusing computation. 
• Convergence = Detection + Propogation + SpfComp + FibUpdTime
• 1. Detection:
• Don't use IGP KA, instead use hw failure detection, like p2p Ethernet. 
• No debounce timers. SONET is 例外，有50 ms protection, can set 50ms+
• If P2P, OSPF does not need DR elections to save time. And no type2 LSA, reduces LSDB size
• If no hw failure detection, have to use fast protocol hello, like ospf fast timer - "ip ospf dead-interval minimal hello-multiplier 5" = 5 hello/sec
• Can do 1-sec failure detection, but cpu load is high
• Use BFD, workload is off to LC. 
• 2. Propagation:
• Out: ISIS/OPSF, using LSA flood. IGP has throttling mechanism, like Cisco has exceptional backoff algorithm. so set initial as low as 5-10 ms. don't use 0, because multiple link down event better in ONE LSA. 
• In: should be ok
• Processing: 
• ISIS has fast-flood, flood before SPF processing.
• Timer pacing flood, the interval between LSA. 
• OutIfQueue: Need Qos to reserve b/w for CP pkts
• Packet propagation: small not serious
• 3. SPF Computation
• Used to be a big impact before, but not now thanks for fast hw and some algorithm improvement like iSPF (incremental)
• 4. Fib update
• This stage contributes the MOST to convergence time
• 2 ways to speed up: 1) less prefixes; 2) sequence important first
• Edge routers don't need core transit prefixes. So, 
• (1) use ip unumbered so to put loopback0; 
• (2) ospf 2 LSA for p2p links, type-1 router; and type-3 network. "ip ospf prefix-suppression", only process type 1, no type-3
• (3) ISIS has priority driven installation

MLAG Fast Convergence - MAC Redirection/Promption

https://eos.arista.com/eos-4-18-0f/mlag-unicast-convergence/

Problem Description:

Consider the following setup 

• mlagA and mlagB are 2 mlag peers with port-ch 2000;
• The host MAC - 0000:1111:2222 is learnt on MLAG 10 and A is the owner. 
• In another way, MAC 0000:1111:2222 is A's local MAC and B's remote
• A bit background:
• The MAC address and ARP information are all sync'ed during boot-up;
• After that, only MAC table is sync'ed, for example, A tells B that MAC a.b.c is from mlag po10, or singly interface, or remote vtep. 
• So in the MAC table, the MAC has at least 4 states:
• learnedDynamic (local mlag), 
• peerDynamic (remote mlag), 
• learnedRemoteDynamic (vxlan) 
• peerRemoteDynamic (remote vxlan)
• Get this information by command - show mac address mlag-peer

Now saying we have 

• link failure, which has 2 loss: down and up
• when B's po 10 is down, all MAC are re-programmed from po 10 to po 2000, so be MAC move is done one by one in old releases before 4.18.1F. (#1)
• when B's po1 10 is back up, the ACL on peerLink to block BUM traffic immediately to break L2 loop while the MAC move needs time to be completed. (#2)
• node failure, which introduces 3 loss, down, up and delay timeout
• node down, 100s msec loss depending on scale
• node up, 100s msec loss (#3)
• When the peerlink is up, why? At this time, peerB has no uplink or downlink up, which are all in reload-delay. 
• Remember the MAC sync mentioned above? A needs to sync up with B on the MAC, so on A these MAC learnt from B will be flushed!!
• A has to relearn these MAC, 50% all of sudden. Still ok for locally switched packets because the hw flooding kicks in. 
• But bad for Vxlan, which requires software flood for the head-end-replication. 
• reload-delay timeout, 100s msec, actually 2 times
• Need to have iBGP or IGP L3 routing between 2 peers.
• Since peerlink is up fast and much earlier than mlag/non-mlag interface. So when non-mlag or mlag interfaces are up, they can send the traffic to peer link before the optimal path converged. 

Feature and solution

So from 4.18F, a feature called MLAG fast MAC redirection is developed to address above issues. This feature has 2 aspects:

1. MAC redirect, for #1 and #2 loss

• With this feature, the the interface attribute of impacted MACs still point to MLAG po 10 in host table. So, no move at all. 
• Strata and Sand implementation are slightly different but same idea. 
• On Sand, it is to use a recirc channel on each Arad/Jericho chip to recycle the MLAG destined packets over to peer-link. 
• Requirements and limitation:
• Peerlink must be a LAG not Ethernet on Strata. 
• On Sand "platform sand lag hardware-only" must be enabled, I believe only hw LAG can share member port - the recirc channel. 
• MLAG ASU2 cannot co-exist. 

2. MAC address promotion, targeting #3. 

• When 1) peer reboot; 2) hitful restart of fwding plane, the remote MACs (learnt via peer switch) will be flushed, which cause
• 1) before the MAC is re-learnt, packets needs to flooded. Still ok in pure L2/L3 environment because done by hw;
• 2) software forward of Vxlan packets. That's a big issue because it results in drops by CoPP. 
• Why the MACs are flushed?
• when peerB is up, MAC are sync'ed from A to B. These remote MAC are flushed. No MAC, then flooding. 
• Solution: 
• when peerB is down, the ownership of MAC are transfered to peerA
• when peerB recovers, *ALL* MAC are sync'ed from A to B
• Details:
• when peerB is down, peerA enters failover state;
• peerDynamic, peerLearnedRemote to learnedDynamic, learnedRemoteDynamic
• NOT single-leg host

Misc:

• From AD1554:
• If all uplinks are L3 interfaces, then it is preferable to keep non-mlag reload-delay timer < mlag timer, so
• Upstream/L3 up first before downstream/L2;
• In this way, S-N traffic should be no loss. 
• Enabled "reload-delay mode lacp standby", need to have non-mlag timre >= mlag timer,
• MLAG interfaces with LACP are kept warm for LAG membership table, MAC table programming. 
• But upstream/L3 must be up after L2/downstream/mlag interfaces, otherwise S/N traffic are blackhole'd. 
• From AD3152:
• SandL3Unicast - managing NH and ensuring EEDB no change
• SandACL - programming DROP ACL on the peerLinkRecircPort avoid pkts from peerLink back to peerLink
• Assigning LagMemberID is interesting, this peerLinkRecircPort needs a member id. what about overflown?
• LAG member, C/D bit
• C = collecting, D = Distributing
• if static LAG, C/D=True, added to LAG
• if LACP enabled, only C=True, D=True, member can be added. 
• peerLinkRecircPort is always C/D=False/True
• LC removal event
• If all ports on this LC, 
• L3 will have some downtime since losing all ARP entries; 
• L2 should be fine once recirc port is programmed. 
• If at least 1 member on another LC, L2/L3 should be fine
• This is quite complicated! 
• 3/1, 3/36, 4/1 are local member of mlag Po 10
• all 3 ports down, all 3 members retained with C/D=False
• PeerLinkRecicFap is added from either 3/0, 3/2 or 4/0 
• If LC3 is pulled, peerLinkRecirFap 4/0 is added. 
• And a lot of combination of events, LAG config change, member port down...
• 4 events:
• LC removal
• LAG config change: unconfig and change config
• Member port down or cable unplug
• port-ch shutdown
• CLIs:
• show plat trident counter int e27/1
• Drops counts in Vlan boundary = Vlan ID missed. In another word, the VLAN id programming on this interface is not done yet. 

Reference:

• PeerOne Vxlan + MLAG
• AD3398, BG141435/96642
• AD3152
• AD1554

Arista EOS - BGP, update wait-for-convergence

Feature Name: 
BGP Update Wait-for-convergence

Purpose: 
"basically prevents BGP from programming routes into hardware and from advertising routes until a convergence event is resolved". The benefit of this feature is to reduce CPU/hw programming churn during convergence event

Reference:
http://aspiringnetworker.blogspot.com/2015/08/bgp-in-arista-data-center_90.html

Use case: 
For example on the spine routers, they have 128-way ECMP and multiple BGP sessions to neighbor routers. During convergence event, it could receive routing information piece by piece depending on how fast and the order of underlying physical interfaces become up. So this feature is to hold BGP prefix programming until the control plane is converged. 

What's the convergence event:

• Router reload, bgp first time to start
• BGP clear
• Rib agent restart, etc

How does it work?

• When BGP enters to converge, it will exit in 1 of 3 conditions
• ALL BGP peers converge
• Convergence timeout - default 5 min
• Slow peer timeout - default 1 min 30 sec
• ALL BGP peers are converged if 
• Neighbor established, and
• Receive a End-of-Rib or BGP KA if GR not enabled, 
• The following cases will slow or stop the convergence process, so need 2 timers
• If 1 neighbor has big tables to convey 
• If there is a dead neighbor, configured but never up
• If dynamic peer is configured
• Slow peer timeout
• default 1:30 after the first bgp neighbor up
• BGP convergence timer out:
• Default is 5 min
• Can be changed by "bgp convergence time xxx (sec)"

R2.15:01:44(config-router-bgp)#sh bgp convergence

BGP Convergence information for VRF: default

Configured convergence timeout: 00:05:00

Configured convergence slow peer timeout: 00:01:30

Convergence based update synchronization is enabled

Last Bgp convergence event 00:01:41 ago

Bgp convergence state : Timeout reached

   Time taken to converge 00:01:30

   First peer came up 00:01:41 ago

   Pending Peers:          1

       Total Peers:        3

       Established Peers:  2

       Disabled Peers:     0

   Peers that did not converge before local bgp convergence:

       IPv4 peers:

           24.24.24.4            (Session : Active)

       IPv6 peers:

           None

Limitation:
While this feature makes a big sense on spine, but DO NOT enable it on TOR/Leaf or First Hop switches. 

For example:
       [ebgp]
       /     \(uplink)
 [mlagA]=====[mlagB]
       \     /
     [server pool]

Say in above a common mlag setup. 

1. mlagB reboots, then all uplink, downlink and peerlink all down 
2. The peerlink between mlagA and mlagB is up (fast before mlag reload delay). 
3. iBGP between mlagA and B is up, B receives routes from A
4. But at this time, the ebgp via uplink still down, so these iBGP prefixes will be hold!!
5. the downlink(mlag port-ch) are up, servers start to forward traffic, then all the traffic are dropped at floor. 

Sample configuration:
router bgp 65500
   update wait-for-convergence
   update wait-install

Sample Topology:

[R1] <----1.1.1.0/24 prefix
 |
 | (ebgp)
 |
[R2]--(ibgp)--- [R4] <--- dead neighbor
 |
 | (ebgp)
 |
[R3]

1) we shut down the ebgp session between R2 and R4, which simulate a "dead" neighbor

! bgp prefix 1.1.1.0/24 in and best path is selected
R2.15:25:34(config-router-bgp)#sh ip bgp 1.1.1.0
BGP routing table information for VRF default
Router identifier 110.255.255.1, local AS number 2
BGP routing table entry for 1.1.1.0/24
 Paths: 1 available
  1
    12.12.12.1 from 12.12.12.1 (130.255.255.100)
      Origin IGP, metric 0, localpref 100, IGP metric 1, weight 0, received 00:01:55 ago, valid, external, best
      Rx SAFI: Unicast

! 1.1.1.0/24 in RIB
R2.15:25:49(config-router-bgp)#sh ip route 1.1.1.0
 B E    1.1.1.0/24 [200/0] via 12.12.12.1, Ethernet3/36/1

2) Now do a hard clear on R2

R2.15:26:58(config-router-bgp)#clear ip bgp *
! Clearing all IPv4 and IPv6 peering sessions

! bgp sessions up except the dead neigh - 24.24.24.4
R2.15:27:06(config-router-bgp)#bas
BGP summary information for VRF default
Router identifier 110.255.255.1, local AS number 2
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc
  12.12.12.1       4  1                 82        84    0    0 00:00:09 Estab  1      1
  23.23.23.3       4  3                 73        91    0    0 00:00:09 Estab  0      0
  24.24.24.4       4  2                 69        72    0    0 00:26:52 Active

! bgp best path here
R2.15:27:20(config-router-bgp)#sh ip bgp 1.1.1.0
BGP routing table entry for 1.1.1.0/24
 Paths: 1 available
  1
    12.12.12.1 from 12.12.12.1 (130.255.255.100)
      Origin IGP, metric 0, localpref 100, IGP metric 1, weight 0, received 00:00:23 ago, valid, external, best
      Rx SAFI: Unicast

! but route NOT in RIB
R2.15:27:34(config-router-bgp)#sh ip route 1.1.1.0
Gateway of last resort is not set

3) After slow peer timeout, the prefix shows up in RIB

R2.15:31:32(config-router-bgp)#sh ip route 1.1.1.0
Gateway of last resort is not set

R2.15:31:34(config-router-bgp)#show bgp conv
BGP Convergence information for VRF: default
Configured convergence timeout: 00:05:00
Configured convergence slow peer timeout: 00:01:30
Convergence based update synchronization is enabled
Last Bgp convergence event 00:01:29 ago
Bgp convergence state : Pending (Waiting for EORs/Keepalives from peer(s) and IGP convergence)
   Convergence timer running, will expire in 00:03:31
   Convergence timeout in use: 00:05:00
   Convergence slow peer timeout in use: 00:01:30
   First peer came up 00:01:29 ago
   All the expected peers are up: no
   All IGP protocols have converged: yes
   Outstanding EORs: 0, Outstanding Keepalives: 0
   Pending Peers:          1
       Total Peers:        3
       Established Peers:  2
       Disabled Peers:     0
   Peers that have not converged yet:
       IPv4 peers:
           24.24.24.4            (Session : Active)
       IPv6 peers:
           None

R2.15:31:35(config-router-bgp)#sh ip route 1.1.1.0
 B E    1.1.1.0/24 [200/0] via 12.12.12.1, Ethernet3/36/1

R2.15:31:37(config-router-bgp)#show bgp conv
BGP Convergence information for VRF: default
Configured convergence timeout: 00:05:00
Configured convergence slow peer timeout: 00:01:30
Convergence based update synchronization is enabled
Last Bgp convergence event 00:01:34 ago
Bgp convergence state : Timeout reached
   Time taken to converge 00:01:30
   First peer came up 00:01:34 ago
   Pending Peers:          1
       Total Peers:        3
       Established Peers:  2
       Disabled Peers:     0
   Peers that did not converge before local bgp convergence:
       IPv4 peers:
           24.24.24.4            (Session : Active)
       IPv6 peers:
           None