How to debug ISIS/BFD neighbor issue

Topology:

[R1/DCS-7512N]:et11/36/1--{180.13.1.0/24}--et11/1:[R2/DCS-7050QX]

* both routers run 4.20.11M
* Between R1 and R2, ISIS/BFD is enabled shown as below

Router Configuration:

R1 configuration:

router isis isis.def
   net 49.0001.0000.0000.0001.00
   is-type level-1
   log-adjacency-changes
   redistribute connected
   spf-interval 1 50 100
   timers lsp generation 1 10 1000
   !
   address-family ipv4 unicast
      maximum-paths 32
   !
   address-family ipv6 unicast
      maximum-paths 32
!
interface Ethernet11/36/1
   description UpEt_A_3.1
   speed forced 10000full
   no switchport
   ip address 180.13.1.1/24
   ipv6 address 180:13:1::1/64
   isis enable isis.def
   isis bfd
   isis circuit-type level-1
   isis network point-to-point
   isis authentication mode md5 level-1
   isis authentication key 7 QI3r+/7nP3Q= level-1

R2 configuration:
router isis isis.def
   net 49.0001.0000.0000.0103.00
   is-type level-1
   log-adjacency-changes
   spf-interval 1 50 100
   timers lsp generation 1 10 1000
   !
   address-family ipv4 unicast
      maximum-paths 32
   !
   address-family ipv6 unicast
      maximum-paths 32
!
interface Ethernet11/1
   description UpEt_A_3.1
   speed forced 10000full
   no switchport
   ip address 180.13.1.100/24
   ipv6 address 180:13:1::100/64
   isis enable isis.def
   isis bfd
   isis circuit-type level-1
   isis network point-to-point
   isis authentication mode md5 level-1
   isis authentication key 7 QI3r+/7nP3Q= level-1

Symptom:

R1: No ISIS and BFD neighbor

R1.13:53:14(config)#show isis nei | grep uspine3
R1.13:53:16(config)#show bfd nei | grep 180.13
R1.13:53:18(config)#

R2: ISIS neighbor stuck in INIT, and BFD is down

R2#sh isis nei
Instance  VRF      System Id        Type Interface          SNPA              State Hold time   Circuit Id
isis.def  default  R1               L1   Ethernet11/1       P2P               INIT  26          2D

R2#sh bfd nei
VRF name: default
-----------------
DstAddr                    MyDisc        YourDisc             Interface         Type               LastUp             LastDown                LastDiag    State
-------------------- --------------- --------------- --------------------- ------------ -------------------- -------------------- ----------------------- -----
180.13.1.1             2628313401               0      Ethernet11/1(47)       normal       04/15/19 13:25       04/15/19 13:26       Nbr Signaled Down     Down

Step1: R1/R2 - "show lldp nei" good and ping works, so no connectivity issue but high latency

-- show lldp nei --
R2#sh lldp nei
Port       Neighbor Device ID               Neighbor Port ID           TTL
Et11/1     R2                               Ethernet11/36/1            120

-- ping -- R2#ping 180.13.1.1
PING 180.13.1.1 (180.13.1.1) 72(100) bytes of data.
80 bytes from 180.13.1.1: icmp_seq=1 ttl=64 time=390 ms
80 bytes from 180.13.1.1: icmp_seq=2 ttl=64 time=437 ms
....

-- show interface <intf> | grep error --
R1.13:46:01(config)#sh int et11/36/1 | grep error
     0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards

     0 output errors, 0 collisions

R2.13:45:34(config)#sh int et11/1 | grep error
     0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards

     0 output errors, 0 collisions

Step2: R2's Down BFD shows NO incoming bfd pkts

-- show bfd neighbor dest-ip <ip> detail --
R2#sh bfd neighbors dest-ip 180.13.1.1 detail
VRF name: default
-----------------
Peer Addr 180.13.1.1, Intf Ethernet11/1, Type normal, State Down
VRF default, LAddr 180.13.2.100, LD/RD 2628313401/0
Session state is Down and not using echo function
Last Up Apr 15 13:25:27 2019
Last Down Apr 15 13:26:23 2019
Last Diag: Nbr Signaled Down
TxInt: 1000, RxInt: 1000, Multiplier: 3
Received RxInt: 300, Received Multiplier: 3
Rx Count: 0, Rx Interval (ms) min/max/avg: 0/0/0 last: never
Tx Count: 87681, Tx Interval (ms) min/max/avg: 508/996/870 last: 627 ms ago
Detect Time: 3000
Sched Delay: 1*TxInt: 87898, 2*TxInt: 0, 3*TxInt: 0, GT 3*TxInt: 0

R2 sends out BFD but no receiving

-- bash tcpdump -ni <intf> udp --
[admin@R2 ~]$ tcpdump -ni et11_1 udp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:48:22.309536 28:99:3a:3b:15:39 > 44:4c:a8:a5:11:40, ethertype IPv4 (0x0800), length 66: 180.13.1.100.57657 > 180.13.1.1.bfd-control: BFDv1, Control, State Down, Flags: [none], length: 24


11:48:23.301574 28:99:3a:3b:15:39 > 44:4c:a8:a5:11:40, ethertype IPv4 (0x0800), length 66: 180.13.1.100.57657 > 180.13.1.1.bfd-control: BFDv1, Control, State Down, Flags: [none], length: 24

Same observation on R1, only receiving no sending

R1.13:53:18(config)#bash tcpdump -ni et11_36_2 udp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_36_2, link-type EN10MB (Ethernet), capture size 262144 bytes
13:55:10.568533 28:99:3a:3b:15:39 > 44:4c:a8:a5:11:40, ethertype IPv4 (0x0800), length 66: 180.13.2.100.57657 > 180.13.2.1.bfd-control: BFDv1, Control, State Down, Flags: [none], length: 24

13:55:11.445074 28:99:3a:3b:15:39 > 44:4c:a8:a5:11:40, ethertype IPv4 (0x0800), length 66: 180.13.2.100.57657 > 180.13.2.1.bfd-control: BFDv1, Control, State Down, Flags: [none], length: 24

ISIS should notify the BFD to establish session after ISIS neighbor up. So the symptom is narrowed down to ISIS neighbor issue. 

Step 3: Tcpdump ISIS IIH (isis hello) on both sides 

-- show ver | grep MAC to get system MAC --

R1#show ver | grep MAC
System MAC address:  444c.a8a5.1140

R2#show ver | grep MAC
System MAC address:  2899.3a3b.1539

R2: IIH 2-way and looks good, R1/R2 send IIH every 10 seconds

-- bash tcpdump -ni <intf> iih --
[admin@R2 ~]$ tcpdump -ni et11_1 iih
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:18:16.104275 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497
11:18:18.000062 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497
11:18:24.723990 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497
11:18:27.597320 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497

R1: IIH, only 1 way, from R1 to R2, no R2 to R1

[admin@bn303 ~]$ tcpdump -ni et11_36_2 iih
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_36_2, link-type EN10MB (Ethernet), capture size 262144 bytes
11:23:53.268387 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497

11:24:02.823894 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497

IIH packets are destinated to well-known LAN broadcast address (0900:2b00:0005). These packets can't reach from R2 to R1. 

Step 4: Arping

Arping shows similar, R1 to R2 is good, but seeing broadcast packets (all ff) drop on the direction of R2 to R1 even with arp ping

* Arp ping is to send our brocast L2 arp request to probe hosts

[admin@R1 ~]$ arping -b -I et11_36_1 180.13.1.100
ARPING 180.13.1.100 from 180.13.1.1 et11_36_1
Unicast reply from 180.13.1.100 [28:99:3A:3B:15:39]  419.967ms
Unicast reply from 180.13.1.100 [28:99:3A:3B:15:39]  409.205ms
Unicast reply from 180.13.1.100 [28:99:3A:3B:15:39]  401.616ms
Unicast reply from 180.13.1.100 [28:99:3A:3B:15:39]  410.797ms
^CSent 4 probes (4 broadcast(s))
Received 4 response(s)

[admin@R2 ~]$ arping -b -I et11_1 180.13.1.1
ARPING 180.13.1.1 from 180.13.1.100 et11_1
^CSent 9 probes (9 broadcast(s))

Received 0 response(s)

[admin@R2 ~]$ tcpdump -ni et11_1 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:46:04.859574 28:99:3a:3b:15:39 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 180.13.1.1 (Broadcast) tell 180.13.1.100, length 28

11:46:05.859645 28:99:3a:3b:15:39 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 180.13.1.1 (Broadcast) tell 180.13.1.100, length 28

At this point, the issue is narrowed down to, broadcast packets from R2 to R1 are dropped unexpectedly, which causes R1 not seeing R2's IIH packets, then no ISIS/BFD neighbor.  

Step 5: hardware issue, but who drops the packets?

Ok, now we can say, most likely the problem is hardware, but who? The best way to nail down the culprit is to mirror packets to cpu, to see if the broadcast packets leave the R2. 

* The feature of mirroring interface packets to CPU is only supported on Sand (Jericho/Arad) platform. 

R1.14:32:14(config)#monitor session 1 source ethernet 11/36/1 rx
R1.14:32:19(config)#monitor session 1 destination cpu

Seeing IIH from R2 to R1 on the mirror cpu. << This is the packets received on the wire of et11/36/1

R1.14:32:20(config)#bash tcpdump -ni mirror0 iih
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on mirror0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:33:04.105249 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497
14:33:13.257565 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497
14:33:21.554871 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497

But not on the physical ethernet port << This is the packets received by the kernel via et11/36/1

R1.14:33:22(config)#bash tcpdump -ni et11_36_1 iih
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_36_1, link-type EN10MB (Ethernet), capture size 262144 bytes
14:33:35.056748 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497

14:33:42.990686 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497

Now we know the broadcast packets are dropped on R1 between eth and kernel. 

Step 6: Counters

And this is approved by the counter output on both R1 and R2

R2/Trident - No loss

R2#clear counters
R2#clear platform trident counters
R2#sh cpu counters queue | nz

--------------------------------------------------------------------------------
                                 Linecard0/0
--------------------------------------------------------------------------------
Queue                                          Counter/pkts*          Drops/pkts
---------------                          ------------------- -------------------
OSPF/ISIS                                                  2                   0
BFD                                                       10                   0
R2#sh platform trident counters interface e11/1 | egrep -i 'card|drop' | nz
R2#

R1/Jericho - Loss seen

R1#clear hardware counter drop
R1#clear counters

R1#sh cpu counters queue | grep 'Et11/36/1' | nz

CoppSystemLldp             Et11/36/1                  4                924                  0                  0

R1#sh hardware counter drop | grep o11/1
A     Jericho11/1   DeqDeletePktCnt                :           39148 : 2019-04-16 11:59:39 : 2019-04-16 11:59:39
P     Jericho11/1   dropVoqInNullRoute             :      3800339040 : 2019-04-16 11:59:39 : 2019-04-16 11:59:39

R2#sh hardware counter drop | grep o11/1
A     Jericho11/1   DeqDeletePktCnt                :           39153 : 2019-04-16 11:59:39 : 2019-04-16 11:59:45
P     Jericho11/1   dropVoqInNullRoute             :      3800605701 : 2019-04-16 11:59:39 : 2019-04-16 11:59:45

So, should goes to DeqDeletePktCnt or dropVoqInNullRoute. What does they mean?

L2 Control Packet Fwding on J-based system

[Update, Nov 1, 2019] Starting from 4.22.0F, EOS can support multiple L2 protocols packets but NEED to create/apply a TCAM profile. 

The L2 control packets like LACP/LLDP/STP are trapped by a switch and sent to CPU for processing. But for some use cases, this behavior requires to be changed. For example, In the below diagram, the CE1 and CE2 want to form a port-channel across the service provider's MPLS cloud network. 



Another use case is the create an L2 harness switch. As shown below, R1 and R2 can establish a port-channel across a harness switch transparently. 




Starting from Arista EOS 4.21.3F, a feature called - L2 protocol forwarding was introduced. With that, above requirement can be achieved. 

The configurations are quite straightforward, 
1) Step 1 - create l2-protocol forwarding profile and enable lacp forward
2) Step 2 - apply this profile under all the client facing interfaces
3) Step 3 - configure LDP type 5 pw - local patch to stitch 2 local eth ports

1. The port-channel is up. 

R1.gts425.15:54:21#sh port-ch 100
Port Channel Port-Channel100:
  Active Ports: Ethernet21 Ethernet22

2. "show lldp neighbor" shows remote and intermediate devices

R1.gts425.16:15:26#sh lldp nei et21, 22
Port       Neighbor Device ID   Neighbor Port ID           TTL
Et21       lp232                Ethernet12/1               120
Et21       R2.hss457            Ethernet21                 120
Et22       lp232                Ethernet12/2               120
Et22       R2.hss457            Ethernet22                 120

Limitation:

• In 4.21.3F, only LACP/LLDP supported. STP and other L2 protocols will be supported in later releases. 
• From 4.22.0F, more protocols are supported, but need to create a tcam profile. 

Arista BGP Peering Options (5) - v6 link-local peering/NH

Arista TOI Link

Compared with "v4 over v6 NH", the configuration is quite similar, except the interface address and neighbor use link-local address. 

So the benefits here are obvious:

• Minimizing the configuration efforts, no need ipv4 and global ipv6 address. 
• IPv4 NLRI over IPv6 link-local NH. 

Because the link-local address is a local address and not accessible by remote devices, the bgp configure is considered to be a special one. The following regular bgp knobs don't work as usually:

• iBGP
• eBGP multi-hop
• confed BGP
• local AS if changing to iBGP
• Link-local eBGP learnt routes are not advertised to iBGP peers, because of lacking NH address

Please note that this is ONLY supported in Gated, not in multi agent mode. 

Arista BGP Peering Options (4) - IPv4 NRLI over IPv6 NH

This blog is about the v4 NRLI over v6 NH. Based on Arista 4.17.0F TOI, this feature comprises 2 parts:

• IPv4 forwarding on IPv6 interface
• IPv6 NRLI with IPv6 NH

Compared with previous configurations, 2 configures are needed:

1. IPv4 forwarding on IPv6 interface, which is enabled by "ip routing ipv6 interfaces  vrf v3"
2. neighbor <ipv6Nei> next-hop address-family ipv6 originate

Another way of 2nd part is to enable "bgp next-hop address-family ipv6". But in the above topology, the NRLI is originated directly, so have to put originate. 

From the output of "show ip route", you can see the v4 route's NH is the ipv6 address

R1.lp231.20:13:59(config-router-bgp-vrf-v3)#sh ip route bgp detail

VRF: v3

 B E   103.2.2.0/24 [200/0]

       via 12:12:103::2, Vlan103

R1.lp231.20:14:07(config-router-bgp-vrf-v3)#sh ip bgp 103.2.2.0

BGP routing table information for VRF v3

Router identifier 103.2.2.2, local AS number 1

BGP routing table entry for 103.2.2.0/24

 Paths: 1 available

  2

    12:12:103::2 from 12:12:103::2 (103.2.2.2)

      Origin IGP, metric 0, localpref 100, IGP metric 1, weight 0, received 00:18:46 ago, valid, external, best

      Rx SAFI: Unicast

Arista BGP Peering Options (3) - IPv6 NRLI over IPv4 Peering

This peering configuration is the reverse version of the previous one. So the configuration-wise, it is very similar. 

The benefit is clear, we only need 1 BGP peering to carry both address families. But we still need to keep a dual stack on the interfaces to be used as NH. In today data center network, ECMP is very popular and wide. Evening using /31, the finest network mask, a /24 can only afford 128 links. 

And this youtube explains very well on the requirement behind RFC 5549, ipv4 over v6 NH, which is shown in the next blog

Arista BGP Peering Options (2) - IPv4 NRLI over IPv6 Peering

Here is the TOI link

On the top of regular IPv6 BGP configuration, you need to configure 2 things:

1) Enable IPv4 NLRI over v6 peering and 2 ways to do it:

1. Under ipv4 address-family, activate it on ipv6 peering, which is shown in R1's configuration
2. Global configuration of "bgp default ipv4-unicast transport ipv6" in R2's show run

2) Specify the IPv4 NH of IPv4 prefixes, and we can do it 3 ways:

1. Global automatic NH address - "bgp auto-local-addr" in R'2 configuration. But this is a global knob, so if you have IPv6 prefixes over IPv4 NH, it will break peering silently. 
2. Neighbor-base automatic NH address - "neighbor 12:12:101::1 auto-local-addr"
3. Neighbor-base manual NH address in R1's configuration - "neighbor 12:12:101::2 local-v4-addr 12.12.101.1"

Here is the output of "sh ip bgp 101.2.2.0" (I enable "routing-context vrf v1", so no need vrf)

R1.lp231.16:28:58(config)#sh ip bgp 101.2.2.0

BGP routing table information for VRF v1

Router identifier 101.1.1.1, local AS number 1

BGP routing table entry for 101.2.2.0/24

 Paths: 1 available

  2

    12.12.101.2 from 12:12:101::2 (101.2.2.2)

      Origin IGP, metric 0, localpref 100, IGP metric 1, weight 0, received 00:02:03 ago, valid, external, best

      Rx SAFI: Unicast

And both ipv4/ipv6 ping work as expected. 

R1.lp231.16:29:11(config)#ping 101.2.2.2

PING 101.2.2.2 (101.2.2.2) 72(100) bytes of data.

80 bytes from 101.2.2.2: icmp_seq=1 ttl=64 time=0.242 ms

80 bytes from 101.2.2.2: icmp_seq=2 ttl=64 time=0.100 ms

80 bytes from 101.2.2.2: icmp_seq=3 ttl=64 time=0.093 ms

80 bytes from 101.2.2.2: icmp_seq=4 ttl=64 time=0.091 ms

80 bytes from 101.2.2.2: icmp_seq=5 ttl=64 time=0.093 ms

--- 101.2.2.2 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.091/0.123/0.242/0.060 ms, ipg/ewma 0.174/0.180 ms

R1.lp231.16:31:39(config)#ping ipv6 101:2:2::2

PING 101:2:2::2(101:2:2::2) 72 data bytes

80 bytes from 101:2:2::2: icmp_seq=1 ttl=64 time=0.262 ms

80 bytes from 101:2:2::2: icmp_seq=2 ttl=64 time=0.106 ms

80 bytes from 101:2:2::2: icmp_seq=3 ttl=64 time=0.103 ms

80 bytes from 101:2:2::2: icmp_seq=4 ttl=64 time=0.099 ms

80 bytes from 101:2:2::2: icmp_seq=5 ttl=64 time=0.103 ms

--- 101:2:2::2 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.099/0.134/0.262/0.064 ms, ipg/ewma 0.184/0.196 ms

Arista BGP Peering Options (1)

In this series of blogs, I will discuss a bit on the Arista EOS BGP peering options. As of March 2019, the EOS has the following peering options:



Here is the Arista official TOI links:

• BGP IPv4 NLRI over IPv6 Transport
• BGP IPv4 NLRI over IPv6 NH
• BGP IPv6 Link-local Peering

So, why do we need them, what's the difference and how to configure them?

1) Why
Today's network needs to support dual address spaces - IPv4 and IPv6, which requires providers to run 2 parallel control/forward planes. So the purpose of all the above features is to decrease the complexity by reducing control plane even forward plane.

For example, a data center with public ipv4/ipv6 address and needs support worldwide ipv4 and ipv6 access. Is that possible to use ipv4/ipv6 address space at the server pool and service routers, and only ipv6 infrastructure? The answer is yes.

2)  What's the difference between the above 3 options?

1. The basic way is to have dual ipv4/ipv6 infrastructures and control planes, which means, 2 BGP sessions and 2 address planes. 
2. IPv4 NRLI over IPv6 Transport is to send v4 prefixes over v6 peering. So it saves 1 just 1 control plane - ipv6 bgp but still needs ipv4 next hop as ipv4 prefixes, which means 2 data planes or dual v4/v6 stack. 
3. RFC 5549 - IPv4 NRLI over IPv6 NH, this feature is one step further, ipv4 prefixes can use ipv6 NH. So 1 BGP session and 1 data planes (v6 infrastructure)
4. BGP IPv6 link-local peering, this utilizes the ipv6 link-local address to establish BGP peering w/o assigning global IPv6 interface addresses. 

3) How to configure them? In the following blogs, I will show the configuration 1 by 1. (Please note that I use vrf under router bgp, in order to configure them together in one place. But it is not required and most of the time, you only need ONE)

Install Arista EOS Swix

Reference:

• Arista's - "Package RPM into an Arista swix"
• Arista Extension ylb.jp

In the Linux world, RPM package is used for software installation. Simply put, the swix (software image extension) is a software package which contains multiple RPM files and a manifest file, which is used to install software or software patch on Arista EOS. 

Generally, a swix is installed in the following steps:

1. copy to local extension directory - "scp, copy"

2. install - "extension"

3. copy them to boot-extension - "copy"

4. if a modular system, install on the standby supervisor - "session"

5. restart agents or reset hardware components if required - "agent terminate"

Additionally, 

1. uninstall - "no extension"

2. remove packet - "delete" 

3. If both sup running mismatched swix, login console and reset boot-extension file

Below is the step by step:

Step 1 - copy

[admin@eosrouter flash]$ scp solomonyang@server:/users/solomonyang/swix/*swix /mnt/flash

patch.swix                                                                            100%   11MB  11.4MB/s   00:00

eosrouter#copy flash:patch.swix extension:

eosrouter#show extensions

Name                          Version/Release      Status      Extension

----------------------------- -------------------- ----------- ---------

patch.swix                    1.0.4 .              A, NI       6

A: available | NA: not available | I: installed | NI: not installed | F: forced

eosrouter#show extensions detail

       Name: patch.swix

    Version: 1.0.4

    Release: blahblah

   Presence: available

     Status: not installed
    Summary: SandFap agent related libs

   Packages:

 Total size: 0 bytes

Description: SandFap agent related libs.

Step 2 - install

eosrouter#extension patch.swix

eosrouter#show extensions

Name                          Version/Release      Status      Extension

----------------------------- -------------------- ----------- ---------

patch.swix                    1.0.4                A, I        6

Step 3 - copy to boot extension

eosrouter#show installed-extensions

patch.swix

eosrouter#show boot-extensions

eosrouter#copy installed-extensions boot-extensions

Copy completed successfully.

eosrouter#show boot-extensions

patch.swix

Step 4 - install them on standby sup

eosrouter#copy flash:patch.swix supervisor-peer:/mnt/flash

Copy completed successfully.

eosrouter#session peer-supervisor

Warning: Permanently added '[127.1.0.2]:3601' (RSA) to the list of known hosts.

WARNING - you are currently logged in to the standby supervisor.

Not all cli commands are available or supported. Configuration mode

is disabled for SSO.

s1:eosrouter#copy flash:patch.swix extension:

Copy completed successfully.

s1:eosrouter#extension patch.swix

s1:eosrouter#copy installed-extensions boot-extensions

Copy completed successfully.

Step 5 - restart agents or reset hardware component

For some software patches, it is required to restart software agents or hardware components to make patch effective. 

eosrouter#show agent names | grep SandFap

SandFap-Linecard3

SandFap-Linecard4

SandFap-Linecard5

eosrouter#agent SandFap-Linecard3 SandFap-Linecard4 SandFap-Linecard5 terminate

SandFap-Linecard3 was terminated

SandFap-Linecard4 was terminated

SandFap-Linecard5 was terminated

Also you need to do the same on the standby sup

How to fix unmatched swix'ed sup

If you run into unmatched swix supervisor due to mis-configuration, the way to fix them is as follows:

1. Login in the console of both supervisor. Sometimes the standby sup is at bad shape, then you have to fix it in Aboot mode

Arista Networks EOS 4.20.5.2F

localhost login: admin

[PyServer ar.Aaa not responding, still trying -- is it running?]

2. Reboot both and press ctrl-C to enter Aboot mode

3. Vi /mnt/flash/boot-extensions to remove swix entries. Then reboot so you will have a clean EOS w/o any extensions. 

Arista EOS - dot1q tunnel

There are 2 common use cases for dot1q tunnel:
1) Service providers tunnel customer's VLAN-tag traffic across different sites;
2) Lab harness switch for flexible connectivity between devices

But with one limitation: L2 control packets can't traverse. For example, "show lldp neighbor" on cd644 indicates yo412 instead of remote router. 

Arista BGP-LU/SR

In this blog, talking a bit about BGP-LU/SR and its use cases.  Here are some References:

Arista BGP-LU

• By the RFC 3107, the equivalent Cisco feature is called labeled BGP. The concept is quite simple to advertise an IPv4 prefix with a label. 
• Use case 1) - replace RSVP-TE, cloud provider
• Central controllers push routes with multiple label via BGP-LU
• Replace RSVP-TE
• Use case 2) - CSC
• ce1--pe1---asbr1--[--cPE1-----cPE2--]--asbr2---pe2--ce2
• between asbr and cPE, eBGP-LU, so cPEs know the label to reach PEs
• cPEs switch packets based on labels of PEs' loopback
• Configuration:
• router bgp 100
•   bgp labeled-unitcast rib tunnel|ip
•   neighbor 77.255.11.11 remote-as 100
•   neighbor 77.255.11.11 update-source lo77
•   address-family ipv4 labeled-unicast
•     neighbor 77.255.11.11 act
• show platform jericho mpls route
• Tunnel RIB vs IP RIB
• Mutli-agent vs GateD

Arista MPLS Segment Routing WP

• 3 segment routing approaches
• 1) Static mpls push + NHG
• prefix based - 100.1.1.0/24 -> NHG1 3.3.3.3 {2, 3, 5, 8}
• class based - 100.1.1.0/24 EF -> NHG1 3.3.3.3 {1,2,3}
• 2) EOS SDK by controller
• Listen: controller leans ISIS topology SIDs (node, adj, prefix) via ISISoGRE or BGP-LS (not supported by Arista EOS yet as of Feb 2019)
• Write: program SR multi-label stack tunnel using NHG
• 3) BGP-LU
• Listen: same as 2)
• Write: use BGP-LU

Arista MPLS VPN Step by Step (2c) L2VPN - Tcpdump

One powerful tool in Arista/EOS is tcpdump. In this setup, for example, if you see some traffic issue and want to see if the PE sending out the packets, you can do the tcpdump this way:

If you want to check the type 5 pw between CE1 and CE2, 

Step 1: find out the mpls ldp tunnel

PE1(config)#sh patch panel forwarding
In/Out                     Type CW VLAN Status Patch
-------------------------- ---- -- ---- ------ --------------------
....
Et47/3                        5         Up     ce1-ce2-pw5-1
   Label 132770, LDP Tun 5  <<< tunnel 5

Step 2: check mpls ldp tunnel 5 fwding info so have the interface name

PE1(config)#sh mpls ldp tunnel 5
   Index       Endpoint              Nexthop         Interface      Labels
----------- --------------------- --------------- ----------------- ----------
   5           88.255.22.22/32       88.1.11.1       Ethernet3/1    [ 100005 ]

Step 3: create the monitor session to dump packet to cpu

monitor session ldp-et3-1 source Ethernet3/1

monitor session ldp-et3-1 destination Cpu

Step 4: tcpdump the packet

bash tcpdump -nevvi mirror0 ether proto 0x8847 -w /mnt/flash/ldp.pcp

Arista MPLS VPN Step by Step (2c) L2VPN - Type 4 PW, MTU

In the LDP PW configuration, one thing needs to be careful is the MTU. You have to ensure the MTU along the underlying LSP matched. Here is an example:

So we have OSPF and ISIS enabled on VLAN 301/302 on CEs. But only OSPF up, no ISIS neighbor at all

CE1#sh ip os neighbor vrf ce1-pw4-301
Neighbor ID     VRF      Pri State                  Dead Time   Address         Interface
101.255.2.2     ce1-pw4-301 1   FULL/DR                00:00:32    101.101.101.2   Vlan301
CE1#sh isis neighbors vrf ce1-pw4-302
CE1#

Well the ping works as expected, CE1 can ping CE2's vlan 302 address - 102.102.102.2

CE1#ping vrf ce1-pw4-302 102.102.102.2
PING 102.102.102.2 (102.102.102.2) 72(100) bytes of data.
80 bytes from 102.102.102.2: icmp_seq=1 ttl=64 time=0.276 ms
80 bytes from 102.102.102.2: icmp_seq=2 ttl=64 time=0.119 ms
80 bytes from 102.102.102.2: icmp_seq=3 ttl=64 time=0.112 ms
80 bytes from 102.102.102.2: icmp_seq=4 ttl=64 time=0.111 ms
80 bytes from 102.102.102.2: icmp_seq=5 ttl=64 time=0.110 ms

--- 102.102.102.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.110/0.145/0.276/0.066 ms, ipg/ewma 0.202/0.208 ms

Both sides configured with MTU 9000

CE1#sh run int vlan 302
interface Vlan302
   mtu 9000
   vrf forwarding ce1-pw4-302
   ip address 102.102.102.1/24
   isis enable isis-ce1-pw4-302

CE2#sh run int vlan 302
interface Vlan302
   mtu 9000
   vrf forwarding ce2-pw4-302
   ip address 102.102.102.2/24
   isis enable isis-ce2-pw4-302

Let's try ping the destination with size of data plus DF-bit

CE1#ping vrf ce1-pw4-302 102.102.102.2 size 9000 df-bit
PING 102.102.102.2 (102.102.102.2) 8972(9000) bytes of data.
--- 102.102.102.2 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 40ms

Ok, it fails but where is the culprit?

PE1 and PE2 are good:

PE1(config-if-Et47/2)#sh ip int brief | grep 88
Ethernet3/1            88.1.11.11/24      up         up              9082
Ethernet45/3           88.3.11.11/24      down       down            9082

PE2(config-if-Et51/1)#sh ip int brief | grep 88
Ethernet33/1           88.4.22.22/24      up         up              9082
Ethernet51/1           88.2.22.22/24      up         up              9082

Let's move on to P routers:

P1#sh ip int brief | grep 88
Ethernet35/1           88.1.4.1/24        up         up              8000
Ethernet51/1           88.1.2.1/24        up         up              8000
Ethernet52/1           88.1.3.1/24        up         up              9082
Ethernet54/1           88.1.11.1/24       up         up              9082

Ok, here is the issue. Let's correct them and ISIS neighbor is up!

CE1#sh isis neighbors vrf ce1-pw4-302

Instance  VRF      System Id        Type Interface          SNPA              State Hold time   Circuit Id
isis-ce1- ce1-pw4- CE2              L2   Vlan302            44:4c:a8:11:6b:e1 UP    8           CE2.02

So, the reason is, the ISIS pads hello packet upto MTU and the oversized packets will be dropped siliently on underlying P routers. Hence ISIS fails to form neighbor. 

The OSPF just negotiates the MTU of connecting interfaces, which are 9000 in both sides, so ospf neighbor is up w/o any issues. But if you have a large LSA database, and ospf utilize the full MTU to transmit it, you will hit this MTU mismatch later, but not in the neigobor discoverage stage.