"Pending BGP Convergence" and invalid on MPLS/SR/EVPN BGP RR

On a BGP EVPN RR, I am seeing some EVPN routes are marked as "Pending BGP Convergence" as below:

RR1.12:01:04#sh bgp evpn
BGP routing table information for VRF default
Router identifier 100.250.250.5, local AS number 65000
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
                    S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
          Network                Next Hop              Metric  LocPref Weight  Path
   %     RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000
                                 100.250.250.11        -       100     0       i
   %     RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000
                                 100.250.250.12        -       100     0       i

The reason is "invalid" in the output of show bgp evpn detail

RR1.12:10:27#sh bgp evpn route-type auto-discovery detail

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

BGP routing table entry for auto-discovery 110100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.11:100

 Paths: 1 available

  Local (Received from a RR-client)

    100.250.250.11 from 100.250.250.11 (100.250.250.11)

      Origin IGP, metric -, localpref 100, weight 0, invalid, internal, pending resolution

      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

BGP routing table entry for auto-discovery 120100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.12:100

 Paths: 1 available

  Local (Received from a RR-client)

    100.250.250.12 from 100.250.250.12 (100.250.250.12)

      Origin IGP, metric -, localpref 100, weight 0, invalid, internal, pending resolution

      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

Here is the BGP configuration:

RR1.11:59:52#sh run sec router bgp

router bgp 65000

   router-id 100.250.250.5

   no bgp default ipv4-unicast

   neighbor RRC peer group

   neighbor RRC remote-as 65000

   neighbor RRC update-source Loopback0

   neighbor RRC route-reflector-client

   neighbor RRC send-community extended

   neighbor 100.250.250.11 peer group RRC

   neighbor 100.250.250.12 peer group RRC

   neighbor 100.250.250.21 peer group RRC

   neighbor 100.250.250.22 peer group RRC

   !

   address-family evpn

      neighbor RRC activate

Add "neighbor default encapsulation mpls next-hop-self source-interface Loopback0" under EVPN AF

RR1.12:20:00(config)#router bgp 65000

RR1.12:20:02(config-router-bgp)#address-family evpn

RR1.12:20:04(config-router-bgp-af)#neighbor default encapsulation mpls next-hop-self source-interface Loopback0

One prefix shows "* >" - valid. One shows no code

RR1.12:20:10#sh bgp evpn

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP

                    S - Stale, c - Contributing to ECMP, b - backup

                    % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path

         RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000

                                 100.250.250.11        -       100     0       i

 * >     RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000

                                 100.250.250.12        -       100     0       i

Details still shows "invalid"

RR1.12:22:09#sh bgp evpn route-type auto-discovery rd 100.250.250.11:100 detail

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

BGP routing table entry for auto-discovery 110100 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.11:100

 Paths: 1 available

  Local (Received from a RR-client)

    100.250.250.11 from 100.250.250.11 (100.250.250.11)

      Origin IGP, metric -, localpref 100, weight 0, invalid, internal

      Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

      MPLS label: 132768

No tunnel endpoint in "show tunnel rib brief"

RR1.12:22:50#show tunnel rib brief | grep 250.250.1

   100.250.250.1/32               IS-IS SR IPv4       6               65                      115               20

   100.250.250.12/32              IS-IS SR IPv4       4               65                      115               40

The cause is the PE's config, missing node-segment ipv4 configuration

PEa1.12:10:13#sh run int lo0

interface Loopback0

   ip address 100.250.250.11/32

   ipv6 address 2000:100:250:250::11/128

   node-segment ipv6 index 11

   isis enable MplsSR

   isis passive

PEa1.12:25:24#c

PEa1.12:26:08(config)#int lo0

PEa1.12:26:10(config-if-Lo0)#node-segment ipv4 index 11

On RR1, NH emerges in tunnel rib. 

RR1.12:22:56#show tunnel rib brief | grep 250.250.1

   100.250.250.1/32        IS-IS SR IPv4       6               65                      115               20

   100.250.250.11/32       IS-IS SR IPv4       2               65                      115               30

   100.250.250.12/32       IS-IS SR IPv4       4               65                      115               40

And bgp table is good

RR1.12:24:51#sh bgp evpn

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP

                    S - Stale, c - Contributing to ECMP, b - backup

                    % - Pending BGP convergence

Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path

 * >     RD: 100.250.250.11:100 auto-discovery 110100 0000:0000:0000:0000:0000

                                 100.250.250.11        -       100     0       i

 * >     RD: 100.250.250.12:100 auto-discovery 120100 0000:0000:0000:0000:0000

                                 100.250.250.12        -       100     0       i

EOS: % Not supported when show bgp summary

If you see the error message with EOS command - show bgp <AF> summary, it is probably caused the routing mode. To be more specific, you are probably running ribd mode and the CLI - "show bgp <AF> summary" is only supported in multi-agent mode

ghs259#show bgp ipv6 unicast summary
% Not supported

ghs259-CIN-DPA2.23:32:18#show ip route summary
Operating routing protocol model: ribd
Configured routing protocol model: multi-agent (will apply after next reboot)

Arista EOS - BGP Selective Route Download

 In this post, I will share my experience with a relatively old (was released back in 2015) but very useful Arista EOS feature - BGP Selective Route Download (SRD). 

The use cases are quite straightforward:

• Program the necessary routes on the routers with small hardware resources. In the above TOI link, only 30K prefixes of 520K (back in 2015) cover 99% traffic. The left small traffic can be directed by the default route. 
• Another useful case (for me) is to control what routes be programmed, or even not installed at all. At meanwhile the BGP runs transparently, which processes, receives and advertises the BGP prefixes. A good example is the RR which is not in the data path.  Or hardness router in the lab, it just sends bgp updates. The traffic can be handled by a couple of static routes. 

In the below example, BGP only installs /24 IPv4 routes within 110.0.0/8 range and /64 IPv6 routes in 2000:110:1::/48. 

router bgp 65501

   bgp route install-map part-peer-v46

!

route-map part-peer-v46 permit 10

   match ip address prefix-list part-peer-route

!

route-map part-peer-v46 permit 20

   match ipv6 address prefix-list part-peer-route-v6

!

ip prefix-list part-peer-route seq 10 permit 110.0.0.0/8 eq 24

!

ipv6 prefix-list part-peer-route-v6

    seq 10 permit 2000:110:1::/48 eq 64

bn309#show ip route summary

...

VRF: default

   Route Source                                Number Of Routes

------------------------------------- -------------------------

...

   ospfv3                                                     0

   bgp                                                     1814

     External: 1814 Internal: 0

...

   Total Routes                                            1871

Number of routes per mask-length:

   /8: 2         /12: 2        /16: 1        /24: 1816     /25: 1

   /30: 7        /32: 42

bn309#show ip bgp installed | egrep '^ \* ' | wc -l

1816

At this time actually, this router receives/accepts over 1.4M prefixes. 

bn309#show ip bgp summary

BGP summary information for VRF default

Router identifier 192.168.230.2, local AS number 65501

Neighbor Status Codes: m - Under maintenance

  Description              Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State   PfxRcd PfxAcc

  IpTransit#1-7504         100.101.1.1      4  12083          44303    119708    0    0 01:08:12 Estab   690049 690049

  Local-Aris-Simu          192.168.230.1    4  65510         129394         5    0    0 01:03:57 Estab   819857 819857

As of August 2020, this feature is only supported on RIBD (so multi-agent mode doesn't work)

EOS: alias to sum up total num of received bgp prefixes

EOS-R1#show ip bgp neighbors 

BGP neighbor is 100.101.1.2, remote AS 100, external link

  Prefix Statistics:

                                   Sent      Rcvd     Best Paths     Best ECMP Paths

    IPv4 Unicast:                688000    818876         808929                   0

    IPv6 Unicast:                     0         0              0                   0

If we like to know the total number of rcvd prefix from all bgp peers, here is the alias command could be useful

alias totbgp show ip bgp neighbors | grep "IPv4 Unicast: \s\s" | awk  '{s+=$4}END{print s}'

EOS-R1#totbgp

6001256

BGP Oscillation (RFC 3345)

BGP Oscillation (RFC 3345)

• BGP Oscillation is common in the context of MED + RR, because,
• BGP only advertise the best path and hide the full list. 
• Similar to RR, it is also with BGP confederation
• RFC 3345 has a very good depiction of how it happens
• And I converted it to a google slides for a better view
• in RFC, there lists a couple of design options to avoid this kind of churn
• In RFC 7964, advertises all available paths by using ADD-PATH
• Always compare MED even from different AS;
• Don't accept MED
• Utilize other BGP attributes higher in the decision process. (a little risky, because if any prefix leaked w/o higher attribute and tied)
• Assign a high IGP cost to inter-cluster-link

AS_PATH and Community regexp tips

1. match certain # of as_path

ip as-path access-list SixAS permit ^._._._._._.$ any

2. community list for some specific ending number, like ending 123. Because there is a space at the end of comm list, so need an underscore _, like

ip community-list regexp Ending123 permit ^.*:.*123_$

Router Server

References:

• https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/xe-3s/irg-xe-3s-book/irg-route-server.pdf
• https://tools.ietf.org/html/rfc7947

Router Server = Internet Exchange (IX) Route Server = eBGP RR

• Used in internet exchange
• eBGP RR, so no need a full mesh eBGP peering among all parties
• Reduces configuration complexity and CPU/memory overhead on border routers

Requirements:

• AS_PATH transparency:
• RS doesn't append its own AS# in the AS_PATH
• RS-client doesn't enforce first AS
• Nexthop transparency: 
• doesn't change NRLI's NH
• MED transparency: 
• doesn't change path MED
• Path hiding:
• Per RFC 4271, BGP only advertises the best path, and a later update will be considered as an implicit withdrawal of the existing path.  
• In RFC 7947, it does mention the "path hiding". So the RS only advertises the best path, which could slow down the convergence time

EOS configurations:

service routing protocols model multi-agent  <<  must multi-agent

!

route-map rtmap-no-change-med permit 10

   set metric +0     << keep metric unchanged

!

router bgp 10

   router-id 10.255.255.251

   neighbor RS-Client peer group

   neighbor RS-Client next-hop-unchanged   << NH unchanged

   neighbor RS-Client as-path prepend-own disabled  << no own as#

   neighbor RS-Client route-map rtmap-not-change-med out << rtmap

   neighbor RS-Client password 7 xLcnzAMGHkI=

   neighbor RS-Client send-community

   neighbor RS-Client maximum-routes 12000

   neighbor 10.10.10.11 peer group RS-Client

   neighbor 10.10.10.11 remote-as 11

!

How many prefixes received? Considering the following scenario:

• 3 peers advertising ONE prefix 
• 2 route servers
• And all 3 peers also have bilateral peering  
• Then should be 5

Router1.15:19:15#sh ip bgp 11.111.1.1

BGP routing table information for VRF default

Router identifier 180.255.255.1, local AS number 65100

BGP routing table entry for 11.111.1.1/32

 Paths: 5 available

  12 111

    10.10.10.12 from 10.10.10.12 (10.255.255.12)  << from R2

      Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP head, ECMP, best, ECMP contributor

      Rx SAFI: Unicast

  11 111

    10.10.10.11 from 10.10.10.11 (10.255.255.11)  << from R1

      Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP, ECMP contributor

      Rx SAFI: Unicast

  11 111

    10.10.10.11 from 10.10.10.251 (10.255.255.251) << R1 vis RS1

      Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP, ECMP contributor

      Rx SAFI: Unicast

  11 111

    10.10.10.11 from 10.10.10.252 (10.255.255.252) << R1 via RS2

      Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP, ECMP contributor

      Rx SAFI: Unicast

  13 333 111

    10.10.10.13 from 10.10.10.13 (10.255.255.13)    << R3

      Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external

      Rx SAFI: Unicast

Let's shut down the private peering to R2. So you can see the RS1 and RS2 don't advertise R2's path. 

Router.15:26:31(config-router-bgp)#sh ip bgp 11.111.1.1

BGP routing table information for VRF default

Router identifier 180.255.255.1, local AS number 65100

BGP routing table entry for 11.111.1.1/32

 Paths: 4 available

  11 111

    10.10.10.11 from 10.10.10.252 (10.255.255.252) << RS2

      Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external, best

      Rx SAFI: Unicast

  11 111

    10.10.10.11 from 10.10.10.251 (10.255.255.251) << RS1

      Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external

      Rx SAFI: Unicast

  11 111

    10.10.10.11 from 10.10.10.11 (10.255.255.11)   << R1

      Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external

      Rx SAFI: Unicast

  12 111

    10.10.10.12 from 10.10.10.12 (10.255.255.12)   << R2

      Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external

      Rx SAFI: Unicast

Ping bgp ipv6 link local address

With the feature of "BGP IPv6 link-local peering support", you can establish the bgp peering via ipv6 link-local address. To check the underlying reachability, we can use linux kernel command - ping6

For example,

interface Vlan4001
   vrf forwarding vrf3_BGP_v6P_v6Ll
   ipv6 address fe80::1/64 link-local
!
router bgp 65100
   vrf vrf3_BGP_v6P_v6Ll
      rd 65000:3
      bgp default ipv4-unicast
      bgp default ipv4-unicast transport ipv6
      bgp default ipv6-unicast
      neighbor fe80::100%Vl4001 remote-as 65500

bn303#bash sudo ip netns exec ns-vrf3_BGP_v6P_v6Ll ping6 -I vlan4001 fe80::100
PING fe80::100(fe80::100) from fe80::1%vlan4001 vlan4001: 56 data bytes
64 bytes from fe80::100%vlan4001: icmp_seq=1 ttl=64 time=0.483 ms
64 bytes from fe80::100%vlan4001: icmp_seq=2 ttl=64 time=0.238 ms
.....


Please note that this peering is under VRF, so you have to use the namespace command - "ip netns..."

Arista BGP Tips - ECMP, RR, Active Prefix

If the following 6 attributes of paths are identical, they are considered as equal paths:

• Weight
• Local_Pref
• AS_Path
• Origin
• MED
• IGP cost to NH

BGP RR

• ONLY RR knows who is RRC, RRC has no idea
• Originator ID is assigned by originating router
• RR changes nothing, but add Cluster ID

Active BGP Prefix

• show ip bgp vrf all, some prefixes are valid but not active
• the common reason is, this particular prefix is learned from other routing protocols
• that's why we need the knob - "bgp advertise-inactive"

BGP Origin Attribute

3 possible BGP origin attributes - Incomplete, IGP and EGP. EGP is never used. 

• If redistributed, the origin is Incomplete
• If network command, the origin is IGP

For example:

ip route 88.88.88.0/24 Loopback88 <<< static route

!

interface Loopback88

   ip address 88.88.88.1/32

!

router bgp 4

   redistribute static

   address-family ipv4

      network 88.88.88.1/32

R4#sh ip bgp 88.88.88.0/24

BGP routing table entry for 88.88.88.0/24

 Paths: 1 available

  Local

    - from - (0.0.0.0)

      Origin INCOMPLETE, metric -, localpref -, weight 0, valid, local, best, redistributed (Static)

R4#sh ip bgp 88.88.88.1/32

BGP routing table entry for 88.88.88.1/32

 Paths: 1 available

  Local

    - from - (0.0.0.0)

      Origin IGP, metric -, localpref -, weight 0, valid, local, best

In output of "show ip bgp sum", does PfxRcd mean prefix or path?

R1 #show ip bgp summary
BGP summary information for VRF default
Router identifier 10.1.255.1, local AS number 65001
Neighbor Status Codes: m - Under maintenance
  Description              Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc
  RRv4                        10.1.255.104  4  65010        2678462     57500    0    0  216d20h Estab  928963 925526

Before addpath, for one particular neighbor, it only sends ONE best path for one particular prefix. So PfxRcd = PathRcd. But after BGP addpath, the behavior changes. Multiple paths can be sent for one prefix to speed up the convergence time. So the meaning of this number is changed to "pathRcd". 

Trouble-shoot BGP peering issue over GRE tunnel

Starting from 4.21.1F, Arista EOS starts to support the hardware GRE tunnel interface and BGP session over the tunnel on Jericho platforms. Before the tunnel is implemented by nexthop-group and decap group. 

Here is a very simple and straightforward setup of eBGP over GRE tunnel. 

But BGP session fails to come up as shown below:

R1.gts425#sh ip bgp sum

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 1

Neighbor Status Codes: m - Under maintenance

  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

  10.100.100.4     4  4               4766       257    0    0 00:20:40 Connect 0      0

Tunnel interface is up and works fine. 

R1.gts425#sh int tunnel 100

Tunnel100 is up, line protocol is up (connected)

  Hardware is Tunnel, address is 0101.0101.0800

  Description: tunnel-gre-sand-to-sand

  Internet address is 10.100.100.1/24

  Broadcast address is 255.255.255.255

  Tunnel source 1.1.1.1, destination 4.4.4.4

  Tunnel protocol/transport GRE/IP

   Key disabled, sequencing disabled

   Checksumming of packets disabled

  Tunnel TTL 0, Hardware forwarding not supported

  Tunnel TOS 0

  Path MTU Discovery

  Tunnel transport MTU 1476 bytes

  Up 22 minutes, 2 seconds

Ping with MTU size works totally fine

R1.gts425#ping 10.100.100.4 size 1476

...

80 bytes from 10.100.100.4: icmp_seq=5 ttl=64 time=0.118 ms

--- 10.100.100.4 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.118/0.155/0.295/0.070 ms, ipg/ewma 0.223/0.222 ms

Now let's run the tcpdump on R1 to see if hello packet out

R1.gts425(config-router-bgp)#bash tcpdump -nvvi et21 host 1.1.1.1

tcpdump: listening on et21, link-type EN10MB (Ethernet), capture size 262144 bytes

11:19:19.774224 28:99:3a:8f:91:bf > 44:4c:a8:c1:78:69, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 1, id 39325, offset 0, flags [DF], proto GRE (47), length 84)

    1.1.1.1 > 4.4.4.4: GREv0, Flags [none], proto IPv4 (0x0800), length 64

(tos 0xc0, ttl 1, id 6538, offset 0, flags [DF], proto TCP (6), length 60)

    10.100.100.1.46931 > 10.100.100.4.bgp: Flags [S], seq 3552659475, win 28720, options [mss 1436,sackOK,TS val 18764185 ecr 0,nop,wscale 7], length 0

11:19:19.774420 44:4c:a8:c1:78:69 > 28:99:3a:8f:91:bf, ethertype IPv4 (0x0800), length 126: (tos 0xc0, ttl 64, id 29177, offset 0, flags [none], proto ICMP (1), length 112)

    10.1.2.2 > 1.1.1.1: ICMP time exceeded in-transit, length 92

(tos 0x0, ttl 1, id 39325, offset 0, flags [DF], proto GRE (47), length 84)

    1.1.1.1 > 4.4.4.4: GREv0, Flags [none], proto IPv4 (0x0800), length 64

(tos 0xc0, ttl 1, id 6538, offset 0, flags [DF], proto TCP (6), length 60)

    10.100.100.1.46931 > 10.100.100.4.bgp: Flags [S], seq 3552659475, win 28720, options [mss 1436,sackOK,TS val 18764185 ecr 0,nop,wscale 7], length 0

Now we can see the reason clearly. The eBGP TCP session is default with ttl 1 and copied to outer GRE packets, so the packets get TTL expired at 10.1.2.2 which is R2. 

To fix this issue, just need to set TTL under tunnel interface

R1.gts425(config-router-bgp)#int tu 100

R1.gts425(config-if-Tu100)#tunnel ttl 10

R1.gts425(config-router-bgp)#sh ip bgp sum

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 1

Neighbor Status Codes: m - Under maintenance

  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

  10.100.100.4     4  4                  4         4    0    0 00:00:01 Estab  0      0

Arista BGP Peering Options (5) - v6 link-local peering/NH

Arista TOI Link

Compared with "v4 over v6 NH", the configuration is quite similar, except the interface address and neighbor use link-local address. 

So the benefits here are obvious:

• Minimizing the configuration efforts, no need ipv4 and global ipv6 address. 
• IPv4 NLRI over IPv6 link-local NH. 

Because the link-local address is a local address and not accessible by remote devices, the bgp configure is considered to be a special one. The following regular bgp knobs don't work as usually:

• iBGP
• eBGP multi-hop
• confed BGP
• local AS if changing to iBGP
• Link-local eBGP learnt routes are not advertised to iBGP peers, because of lacking NH address

Please note that this is ONLY supported in Gated, not in multi agent mode. 

Arista BGP Peering Options (4) - IPv4 NRLI over IPv6 NH

This blog is about the v4 NRLI over v6 NH. Based on Arista 4.17.0F TOI, this feature comprises 2 parts:

• IPv4 forwarding on IPv6 interface
• IPv6 NRLI with IPv6 NH

Compared with previous configurations, 2 configures are needed:

1. IPv4 forwarding on IPv6 interface, which is enabled by "ip routing ipv6 interfaces  vrf v3"
2. neighbor <ipv6Nei> next-hop address-family ipv6 originate

Another way of 2nd part is to enable "bgp next-hop address-family ipv6". But in the above topology, the NRLI is originated directly, so have to put originate. 

From the output of "show ip route", you can see the v4 route's NH is the ipv6 address

R1.lp231.20:13:59(config-router-bgp-vrf-v3)#sh ip route bgp detail

VRF: v3

 B E   103.2.2.0/24 [200/0]

       via 12:12:103::2, Vlan103

R1.lp231.20:14:07(config-router-bgp-vrf-v3)#sh ip bgp 103.2.2.0

BGP routing table information for VRF v3

Router identifier 103.2.2.2, local AS number 1

BGP routing table entry for 103.2.2.0/24

 Paths: 1 available

  2

    12:12:103::2 from 12:12:103::2 (103.2.2.2)

      Origin IGP, metric 0, localpref 100, IGP metric 1, weight 0, received 00:18:46 ago, valid, external, best

      Rx SAFI: Unicast

Arista BGP Peering Options (3) - IPv6 NRLI over IPv4 Peering

This peering configuration is the reverse version of the previous one. So the configuration-wise, it is very similar. 

The benefit is clear, we only need 1 BGP peering to carry both address families. But we still need to keep a dual stack on the interfaces to be used as NH. In today data center network, ECMP is very popular and wide. Evening using /31, the finest network mask, a /24 can only afford 128 links. 

And this youtube explains very well on the requirement behind RFC 5549, ipv4 over v6 NH, which is shown in the next blog

Arista BGP Peering Options (2) - IPv4 NRLI over IPv6 Peering

Here is the TOI link

On the top of regular IPv6 BGP configuration, you need to configure 2 things:

1) Enable IPv4 NLRI over v6 peering and 2 ways to do it:

1. Under ipv4 address-family, activate it on ipv6 peering, which is shown in R1's configuration
2. Global configuration of "bgp default ipv4-unicast transport ipv6" in R2's show run

2) Specify the IPv4 NH of IPv4 prefixes, and we can do it 3 ways:

1. Global automatic NH address - "bgp auto-local-addr" in R'2 configuration. But this is a global knob, so if you have IPv6 prefixes over IPv4 NH, it will break peering silently. 
2. Neighbor-base automatic NH address - "neighbor 12:12:101::1 auto-local-addr"
3. Neighbor-base manual NH address in R1's configuration - "neighbor 12:12:101::2 local-v4-addr 12.12.101.1"

Here is the output of "sh ip bgp 101.2.2.0" (I enable "routing-context vrf v1", so no need vrf)

R1.lp231.16:28:58(config)#sh ip bgp 101.2.2.0

BGP routing table information for VRF v1

Router identifier 101.1.1.1, local AS number 1

BGP routing table entry for 101.2.2.0/24

 Paths: 1 available

  2

    12.12.101.2 from 12:12:101::2 (101.2.2.2)

      Origin IGP, metric 0, localpref 100, IGP metric 1, weight 0, received 00:02:03 ago, valid, external, best

      Rx SAFI: Unicast

And both ipv4/ipv6 ping work as expected. 

R1.lp231.16:29:11(config)#ping 101.2.2.2

PING 101.2.2.2 (101.2.2.2) 72(100) bytes of data.

80 bytes from 101.2.2.2: icmp_seq=1 ttl=64 time=0.242 ms

80 bytes from 101.2.2.2: icmp_seq=2 ttl=64 time=0.100 ms

80 bytes from 101.2.2.2: icmp_seq=3 ttl=64 time=0.093 ms

80 bytes from 101.2.2.2: icmp_seq=4 ttl=64 time=0.091 ms

80 bytes from 101.2.2.2: icmp_seq=5 ttl=64 time=0.093 ms

--- 101.2.2.2 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.091/0.123/0.242/0.060 ms, ipg/ewma 0.174/0.180 ms

R1.lp231.16:31:39(config)#ping ipv6 101:2:2::2

PING 101:2:2::2(101:2:2::2) 72 data bytes

80 bytes from 101:2:2::2: icmp_seq=1 ttl=64 time=0.262 ms

80 bytes from 101:2:2::2: icmp_seq=2 ttl=64 time=0.106 ms

80 bytes from 101:2:2::2: icmp_seq=3 ttl=64 time=0.103 ms

80 bytes from 101:2:2::2: icmp_seq=4 ttl=64 time=0.099 ms

80 bytes from 101:2:2::2: icmp_seq=5 ttl=64 time=0.103 ms

--- 101:2:2::2 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.099/0.134/0.262/0.064 ms, ipg/ewma 0.184/0.196 ms

Arista BGP Peering Options (1)

In this series of blogs, I will discuss a bit on the Arista EOS BGP peering options. As of March 2019, the EOS has the following peering options:



Here is the Arista official TOI links:

• BGP IPv4 NLRI over IPv6 Transport
• BGP IPv4 NLRI over IPv6 NH
• BGP IPv6 Link-local Peering

So, why do we need them, what's the difference and how to configure them?

1) Why
Today's network needs to support dual address spaces - IPv4 and IPv6, which requires providers to run 2 parallel control/forward planes. So the purpose of all the above features is to decrease the complexity by reducing control plane even forward plane.

For example, a data center with public ipv4/ipv6 address and needs support worldwide ipv4 and ipv6 access. Is that possible to use ipv4/ipv6 address space at the server pool and service routers, and only ipv6 infrastructure? The answer is yes.

2)  What's the difference between the above 3 options?

1. The basic way is to have dual ipv4/ipv6 infrastructures and control planes, which means, 2 BGP sessions and 2 address planes. 
2. IPv4 NRLI over IPv6 Transport is to send v4 prefixes over v6 peering. So it saves 1 just 1 control plane - ipv6 bgp but still needs ipv4 next hop as ipv4 prefixes, which means 2 data planes or dual v4/v6 stack. 
3. RFC 5549 - IPv4 NRLI over IPv6 NH, this feature is one step further, ipv4 prefixes can use ipv6 NH. So 1 BGP session and 1 data planes (v6 infrastructure)
4. BGP IPv6 link-local peering, this utilizes the ipv6 link-local address to establish BGP peering w/o assigning global IPv6 interface addresses. 

3) How to configure them? In the following blogs, I will show the configuration 1 by 1. (Please note that I use vrf under router bgp, in order to configure them together in one place. But it is not required and most of the time, you only need ONE)

Arista EOS BGP wait-for-install and wait-for-convergence

"update wait-for-install"

Quite straightforward feature, update the prefixes until they are installed in hardware

• Obviously, this prevents packet loss. Without this feature, routers start to advertise prefixes when hardware forwarding not ready yet. When its peers start to flood traffic to it, the only thing it can do is to drop them. 
• Very useful when working with MLAG, which requires the routers to put Mlag downstream interface to hold state during initialization. 

"update wait-for-convergence"

• https://aristaeos.blogspot.com/2018/03/arista-eos-bgp-update-wait-for.html
• This is a feature for spine routers. The key point behind it is, don't advertise prefixes until fully convergence (which means you have full routing info). 

Where to use them:

• On MLAG or L2/L3 boundary,  ONLY use wait-for-install
• On spine, use both

How to check # of BGP attributes

Check BGP export/shadown table

First enable "sflow extension bgp" then run command - "show bgp export-table path-attribute" or "show bgp shadown-table bgp-attribute" if older releases

bn302.17:01:15#show bgp export-table path-attribute
Bgp Attribute Info Table
key             nextHop  origin  pathFlags  originatorId  aspType  med  localPref  asPathAttrInfoId  commListId  extCommListId
 64        210.100.22.1       0         16             0        2    0          0                 3  4294967295     4294967295
 80  2000:210:255:254::       0          0    3539992321        1    0        100        4294967295  4294967295     4294967295
 88        210.100.11.1       0         24    3539992321        1    0        100                 2  4294967295     4294967295
 56        210.100.22.1       2         16             0        2    0          0                 2  4294967295     4294967295
 20  2000:210:100:32::1       0         48             0        2    0          0                 1  4294967295     4294967295

A new command:
bn302.17:01:08#bash smash -p ar/Smash/routing/bgp/export/pathAttrTableInfo/default/pathAttrEntry | egrep '^bucket' | wc -l
23

FB's back bone network - EBB

https://code.fb.com/networking-traffic/building-express-backbone-facebook-s-new-long-haul-network/

Why need EBB? Inter-DC traffic growth is much faster than internet egress traffic and WAN technology evolution.

What needed for the new EBB?

• Incremental deployment of software features. 不是一步到位，而且可以roll back. 
• NO RSVP-TE, because of inefficiency and complexity. 嗯，锤！
• Use MPLS SR so network state lean. 这倒是真的，SR分离CP和Local segment management.

How

• Use DC network ideas
• Small routing table - on the order of K routes
• 4 planes. 
• 3 Steps:
• IGP + full-mesh iBGP for packet forwarding
• TE + controller
• OPEN/R to replace IGP
• Components:
• 3 server-side:
• SFlow - read
• BGP route injector - write
• Controller - think
• 2 device-side:
• Open/R = IGP
• SR = forwarding
• 还有clean fault domain splitting, 这是咋做的？

Software Design:

• Hybrid approach: centralized controller + SR, best way
• path computation is at central level with global view
• network failure is handled at distributed level. 
• Open/R
• how to open/R on arista devices?
• Traffic estimator:
• SFlow, need src/dst pair
• what is the sample rate? 1 of 1000? how accurate
• LSP agent:
• With thrift-based API (eapi on eos) to program hw
• In future? need failure detection and fails over within hundreds of msec. 

Next step:

• per-service model
• scheduler for bulk transfer