Kill list of zombie process on Linux(Eos)

Due to a script bug, one EOS device has quite some zombie process like,

[admin@bn302 flash]$ ps aux  | grep 'CliShell -A'
root     28804  0.0  0.0  10504  8580 ?        S    10:53   0:00 CliShell -A -p 15 -c show port-channel | json
root     28809  0.0  0.0  10504  8688 ?        S    10:53   0:00 CliShell -A -p 15 -c show port-channel | json
root     28826  0.0  0.0  10504  8612 ?        S    10:54   0:00 CliShell -A -p 15 -c show port-channel | json
root     28835  0.0  0.0  10504  8580 ?        S    10:54   0:00 CliShell -A -p 15 -c show port-channel | json
root     28844  0.0  0.0  10504  8616 ?        S    10:54   0:00 CliShell -A -p 15 -c show port-channel | json
....

Instead of killing one by one manually, write a quick shell command to kill all. (killall is not supported in EOS/Linux)

[admin@bn302 flash]$ for pid in $(ps aux  | grep 'CliShell -A' | awk '{print $2, $11}' | grep CliShell | awk '{print $1}'); do sudo kill -9 $pid; done

ISIS multi-topology - interfaces with different IPv4/v6 AF enabled

As we know, the ISIS requires address family configuration must be matched between the ISIS router and interface configuration. So if you have both ipv4 and ipv6 AF configuration, you must do the same under interface.

Sometime you may have some of interfaces only enabled ipv4, but dual stack globally, in this case, you will need the ISIS multi-topology.

router isis VrfDefault
   net 49.0001.0000.0000.0002.00
   router-id ipv4 100.255.255.2
   is-type level-1
   !
   address-family ipv4 unicast
   !
   address-family ipv6 unicast
      multi-topology   <<< need multi-topology enabled here
!
interface Port-Channel211
   no switchport
   ip address 100.201.1.0/31
   ipv6 address 100:201:1::/127
   isis enable VrfDefault <<< this intf has both v4 and v6
!
interface Port-Channel251
   no switchport
   ip address 100.205.1.0/31
   isis enable VrfDefault <<< but this intf only v4
   isis multi-topology address-family ipv4 unicast <<<

And both neighbors are up!!

Router1(config-if-Po251)#show isis nei
Instance  VRF      System Id     Type Interface          SNPA              State Hold time   Circuit Id
VrfDef default  RouterDualStack  L1   Port-Channel211    P2P               UP    3           48
VrfDef default  RouterV4         L1   Port-Channel251    P2P               UP    2           63

MACSec

Arista EOS starts to support MACSec - 802.1AE from 4.15.4F which was released around 2016. Later related features:

• 4.17.0F - MACSec EAP-FAST (key server)
• 4.21.1F - MACSec Proxy
• Arista MACSec WP (a list of supported hw)
• Cisco MACSec WP (ipsec vs macsec)

Some key take-aways:

• Purpose: to protect data traffic from various types of attacks, 
• passive: snooping
• active: reply, man in the middle
• vs IPSec
• Level: IPSec is at the IP level, vs MACSec MAC/data link level.  
• End or Hop: IPSec provides end-to-end protection, MACsec secures Per-hop. (TLS is similar like IPSec, end-to-end)
• HW: IPSec doesn't rely on hw but with IPSec often, MACSec is on PHY level with special hw. For example, 
• 7500E-6CFPX-LC
• 7280SRAM-48C6
• 7280CR2M-30
• 7500R2M-36CQ-LC
• Throughput: IPSec has a ceiling, while MACsec is line-rate like 100G. 
• Key components:
• CAK - Conn Ass Key, master key. Either manual or from key server
• CAN - CAK's name
• SAK - Secu Ass Key, derived from CAK, used to data encryption. 
• Point-to-Point vs clear 802.1Q
• Clear 802.1Q means, move vlan tag ahead of MACSec Tag, a must for hub-spoke topology. Cisco ASR supports it. 
• Arista EOS only supports point-to-point. 
• Packet format:
• Add 16-byte SecTAG after srcMAC and VLAN Tag
• Add 16-byte ICV at end of ethernet frames
• Ethertype = 0x88e5 (why not use 0x85ec?, hahaha)
• MACSec proxy:
• Provides MACSec for VxLAN traffic
• Loopback traffic to a MACSec capable front panel port to process. 
• MISC:
• fallback key in case MACSec failed or during a configuration change
• L2-protocol pass thru, skipping LLDP packets

Configuration (EOS):

mac security

license DCI1 <license>

profile toISP1

  ! key <CAN> <type, 0-clear text> <key>

  key a1 0 a123456

interface eth4/1/1

  mac security profile toISP1

!

show mac secu counters

!

show mac secu interface

Arista Modular Switch - simulate LC remove/insert

!! bash cat /persist/sys/Linecard3.cfg
platform module Linecard3 remove       
platform module Linecard3 insert

7508E CPU Util% is high (1)

Saw quite slow response on one of my lab router, a 7508E with latest EOS release. 

mlagA.10:18:17(config)#show ver
Arista DCS-7508
Hardware version:    06.00
....
Software image version: 4.22.0.1F

CPU only has 64% idle cycles. Not low. 

mlagA.10:18:13(config)#show proc top once | more
%Cpu(s): 29.6 us,  3.9 sy,  0.0 ni, 64.0 id,  0.1 wa,  0.5 hi,  2.0 si,  0.0 st
  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
   23 root      20   0     0    0    0 R  89.6  0.0  26:04.70 ksoftirqd/2
   17 root      20   0     0    0    0 R  68.5  0.0  24:27.80 ksoftirqd/1

I suspect there may be some unexpected traffic hitting the CPU, so check the output of "show cpu couter queue | nz"

yo412.mlagA.10:22:38(config)#clear counters
!!! even a single command - clear counter, takes almost 10 sec to complete !!!


yo412.mlagA.10:22:47(config)#show cpu counters queue | nz | more
Arad3/0:
CoPP Class                 Queue                    Pkts             Octets           DropPkts         DropOctets
Aggregate
-----------------------------------------------------------------------------------------------------------------
CoppSystemL3LpmOverflow    Et3/6/1                  1753             473344              74945           21049856
CoppSystemL3LpmOverflow    Et3/6/2                  1112             307200              73605           20702976
CoppSystemL3LpmOverflow    Et3/6/3                   610             166912              86302           23954432
CoppSystemL3LpmOverflow    Et3/6/4                  1178             320256              77089           21414656

Looks like there is a lot of packets hitting the cpu, even the CoPP filters out most of them. But this is a full load chassis, the aggregated traffic is still too heavy to a x86 CPU. 

Try to tcpdump the incoming packets from et3/6/1 and punted to cpu. Surprisingly not many... 

mlagA.10:35:23(config)#bash tcpdump -nvvi et3_6_1
tcpdump: listening on et3_6_1, link-type EN10MB (Ethernet), capture size 262144 bytes
10:35:36.066689 00:1c:73:46:0d:b0 > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 124: LACPv1, length 110
10:35:39.530341 00:1c:73:3b:e0:22 > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 124: LACPv1, length 110
^C

2 packets captured

Try to mirror this port to cpu then tcpdump it. (This feature is only supported on 7500E/R or 7280R devices)

mlagA.10:37:46(config)#monitor session 1 source et3/6/1 rx
mlagA.10:39:12(config)#monitor session 1 destination cpu

mlagA.10:39:15(config)#bash tcpdump -nvi mirror0
tcpdump: listening on mirror0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:40:08.198512 1e:af:14:08:18:02 > 00:aa:aa:aa:bb:cc, ethertype 802.1Q (0x8100), length 252: vlan 1408, p 0, ethertype IPv4, 
    100.14.8.119.30485 > 220.200.16.1.24659: Flags [R.UW], seq 0:194, ack 0, win 61689, urg 0, length 194 
10:40:08.199078 1e:af:14:09:18:01 > 00:aa:aa:aa:bb:cc, ethertype 802.1Q (0x8100), length 252: vlan 1409, p 0, ethertype IPv4, 
    100.14.9.118.30504 > 220.200.17.1.24648: Flags [PUEW], seq 0:194, win 62028, urg 0, length 194

Do we have the route? No....

mlagA.10:40:08(config)#sh ip route 220.200.17.1
VRF: default
....
Gateway of last resort is not set

Create a null route for this prefix, response is better and "show cpu couter queue | nz" is back to normal now, no L3LPMOverflow anymore. 

mlagA.10:54:28(config)#ip route 220.200.0.0/16 null0

mlagA.10:54:59(config)#sh cpu counters queue | nz | more
Arad3/0:
CoPP Class                 Queue                    Pkts             Octets           DropPkts         DropOctets
Aggregate
-----------------------------------------------------------------------------------------------------------------
CoppSystemIgmp             Et3/1/2                   160              10240                  0                  0
CoppSystemIgmp             Et3/1/4                   160              10240                  0                  0

But cpu still high. And the busiest process is changed to SandFap instead of ksoftirqd. Hmmm.... why?

mlagA.10:57:19(config)#sh proc top once | more

%Cpu(s): 30.2 us,  4.1 sy,  0.0 ni, 62.1 id,  0.1 wa,  0.5 hi,  3.1 si,  0.0 st

...

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND

12874 root      20   0 1001m 357m 192m R 100.4  2.2 110:39.04 SandFap

13025 root      20   0 1001m 357m 192m S  69.6  2.2 110:54.53 SandFap

16765 root      20   0 1001m 359m 193m S  51.2  2.2  96:56.18 SandFap

Arista BGP Tips - ECMP, RR, Active Prefix

If the following 6 attributes of paths are identical, they are considered as equal paths:

• Weight
• Local_Pref
• AS_Path
• Origin
• MED
• IGP cost to NH

BGP RR

• ONLY RR knows who is RRC, RRC has no idea
• Originator ID is assigned by originating router
• RR changes nothing, but add Cluster ID

Active BGP Prefix

• show ip bgp vrf all, some prefixes are valid but not active
• the common reason is, this particular prefix is learned from other routing protocols
• that's why we need the knob - "bgp advertise-inactive"

BGP Origin Attribute

3 possible BGP origin attributes - Incomplete, IGP and EGP. EGP is never used. 

• If redistributed, the origin is Incomplete
• If network command, the origin is IGP

For example:

ip route 88.88.88.0/24 Loopback88 <<< static route

!

interface Loopback88

   ip address 88.88.88.1/32

!

router bgp 4

   redistribute static

   address-family ipv4

      network 88.88.88.1/32

R4#sh ip bgp 88.88.88.0/24

BGP routing table entry for 88.88.88.0/24

 Paths: 1 available

  Local

    - from - (0.0.0.0)

      Origin INCOMPLETE, metric -, localpref -, weight 0, valid, local, best, redistributed (Static)

R4#sh ip bgp 88.88.88.1/32

BGP routing table entry for 88.88.88.1/32

 Paths: 1 available

  Local

    - from - (0.0.0.0)

      Origin IGP, metric -, localpref -, weight 0, valid, local, best

Arista EOS Tunneling Mechanism (5) - Hw Tunnel

Starting from 4.21.1F, hardware GRE tunnel is supported on Jericho-based platforms. Below is a typical setup of hw tunnel and configuration



You can see, configuration-wise it is just like the software tunnel. How to tell which mode, check "show interface tunnel". 

R1#sh int tu200
Tunnel200 is up, line protocol is up (connected)
...
  Tunnel TTL 10, Hardware forwarding enabled

Compared with software tunnel, the biggest difference is the hardware programming. 

1. TCAM:

R2(config)#sh platform fap tcam summary
            Tcam Allocation (Jericho0)
   Bank                      Used By    Reserved By
---------- ---------------------------- -----------
      0                       dbVTT0              -
     12                     dbSystem              -
..

Now configuring an up tunnel interface

R2(config)#int tu100
R2(config-if-Tu100)#tunnel source 100.2.2.2
R2(config-if-Tu100)#tunnel destination 100.3.3.3
R2(config-if-Tu100)#show ip int brief
Interface              IP Address         Status     Protocol         MTU
....
Tunnel100              unassigned         up         up              1476

Now you can see a new Tcam allocation named dbGreTunnel

R2#sh platform fap tcam summary
            Tcam Allocation (Jericho0)
   Bank                      Used By    Reserved By
---------- ---------------------------- -----------
      0                       dbVTT0              -
      2                  dbGreTunnel              -
     12                     dbSystem              -

2. Tunnel interface

R1#show platform fap eedb ip-tunnel gre interface tunnel 200
....
| Bank/   | OutLIF  | Next    | VSI   | Encap | TOS  | TTL  | Source   | Destination     | OamLIF | OutLIF  | Drop |
| Offset  |         | OutLIF  | LSB   | Mode  |      |      | IP       | IP              | Set    | Profile |      |
|-----------------------------------------------------------------------------------------------------------|
| 2/0     | 0x6000  | 0x4012  | 0     | 2     | 0    | 10   | 100.1.1.1  | 100.4.4.4       | No     | 0       | No   |

* Bank should match the bank# in "show plat fap tcam summary"

3. Ip route

Below show 220.4.4.0/24 is learned via Tu200

R1(config-if-Tu200)#sh ip route

VRF: default
 I L2     220.4.4.0/24 [115/20] via 200.1.4.4, Tunnel200, Static Interface GRE tunnel index 200, dst 100.4.4.4, src 100.1.1.1, TTL 10

R1(config-if-Tu200)#show platform fap ip route | egrep 'VRF|ID|220.4'
|VRF|   Destination    |     |                    |     |       |                   | ECMP|  FEC | Tunnel
| ID|      Subnet      | Cmd |     Destination    | VID |Outlif |   MAC / CPU Code  |Index| Index|T Value
|0  |220.4.4.0/24      |ROUTE| FEC 32773          |0    |  -    |                   |  -  |49160 |   -

4. FEC

R1(config-if-Tu200)#show platform fap fec all
.....
|     |      |     |                    |     |       |                   |
| ECMP|  FEC |     |                    |     |       |                   | Tunnel
|Index| Index| Cmd |     Destination    | VID |Outlif |   MAC / CPU Code  |T Value
 ----------------------------------------------------------------------------------
.....
|  -  |49160 |ROUTE| FEC 32773          |0    |  -    |                   |   -
|  -  |32773 |ROUTE| Et21               |12   |  -    | 44:4c:a8:c1:78:69 |T 100.4.4.4

Limitations:

In above TOI link, there is a list of hw tunnel limitations, for example:

• Underlay endpoint (tunnel source/destination) must IPv4 address in default VRF. 
• Even there is tunnel source/destination ipv6 address option, but not supported on this platform
• Overlay can be IPv4/IPv6 under default or non-default VRF
• No GRE KA, if one side has incorrect GRE configuration like missing tunnel source, the other end of GRE interface is still up. 

In output of "show ip bgp sum", does PfxRcd mean prefix or path?

R1 #show ip bgp summary
BGP summary information for VRF default
Router identifier 10.1.255.1, local AS number 65001
Neighbor Status Codes: m - Under maintenance
  Description              Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc
  RRv4                        10.1.255.104  4  65010        2678462     57500    0    0  216d20h Estab  928963 925526

Before addpath, for one particular neighbor, it only sends ONE best path for one particular prefix. So PfxRcd = PathRcd. But after BGP addpath, the behavior changes. Multiple paths can be sent for one prefix to speed up the convergence time. So the meaning of this number is changed to "pathRcd". 

How to run RFC2544 thruput/latency testing with Ixia

Step 1 - Connect device in a snake way like below



Step 2 - Configure IxNetworks for RFC 2544

1. Click "Add QuickTest". After configuration, it should show up in the right side



2. Finish all options. Pretty straightforward, just need to configure like

• Ports to be used (Definitely need 2-way)
• Packet type (Eth/IPv4/IPv6)
• Packet size (make sure interface MTU matched)
• Traffic option, like how long each case last
• Stats parameters, like calculate latency (Cut through)
• 100% line rate (if for thruput)

Step 3 - Run the test and generate the report, like

Use MAC ACL to isolate the failure point

For L2 traffic, besides checking drops/discard counter, another way to isolate the failure point is to use the MAC ACL, like

mac access-list macCount
   counters per-entry
   10 permit 00:00:03:03:00:14 00:00:00:00:00:00 04:68:03:03:00:14 00:00:00:00:00:00 log
   20 permit any any log
!
interface Ethernet3/1
   switchport access vlan 3003
   mac access-group macCount in

The above MAC acl - macCount is count the number of packets with source MAC - 0000.0303.0014 and dest MAC - 0468.0303.0014. And it is applied on Eth3/1 ingress direction (egress ACL is not supported)

Router#show mac access-lists
MAC Access List macCount
        counters per-entry
        10 permit 00:00:03:03:00:14 00:00:00:00:00:00 04:68:03:03:00:14 00:00:00:00:00:00 log [match 216114288 packets, 0:00:00 ago]
        20 permit any any log

This is an Arista DCS-7280CR2A-60-F with 4.22.0F

Connect Ixia Novas Qsfp28 100G via 100GBASE-CR4 cable

The arista device is Arista DCS-7280CR2-60-F loading 4.21.2F. The other side is IxNetworks's NOVUS-M100GE8Q28 card, and using Arista 100G AOC cable. 

Et60/1              connected    1        full   100G   100GBASE-CR4

Have to disable the autoneg on Ixia side to bring up the link. As shown as below

1. First stop all traffic (if the option is gray, that's the reason) then Edit L1 Properties
2. Unclick "Use IEEE Media for 100BASE CR4"
3. Uncheck "Auto Negotiate"

Arista EOS Tunneling Mechanism (4) - Sw Tunnel vs Data

In the above topology, I show you that the data traffic doesn't go thru the sw tunnel. I use 2 routers to simulate hosts with default gateway pointing to R11 and R44. And we can see

• Ping between R11 and R44 works
• But the ping from host1 to host2 doesn't work

R11 has the correct route and ping to R44's ip address is good. 

R11#sh ip route 99.2.2.99
...
 I L1     99.2.2.0/24 [115/20] via 10.100.100.44, Tunnel100

R11.cd642.leaf18#ping 99.2.2.2

PING 99.2.2.2 (99.2.2.2) 72(100) bytes of data.

80 bytes from 99.2.2.2: icmp_seq=1 ttl=64 time=0.264 ms

80 bytes from 99.2.2.2: icmp_seq=2 ttl=64 time=0.119 ms

Host1 also has right route but ping failed, so data traffic can't pass thru. 

host1#sh ip route 99.2.2.99
....
 S        99.0.0.0/8 [1/0] via 99.1.1.1, Ethernet51/1

host1#ping 99.2.2.99
--- 99.2.2.99 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 40ms

Now, we add NHG + Decap group (will cover the details later) on both ends. 



!!!!! R11 !!!!!!
ip route 99.2.2.0/24 Nexthop-Group nhg-gre-99-net
!
nexthop-group nhg-gre-99-net type gre
   size 1
   ttl 64
   tunnel-source 11.11.11.11
   entry 0 tunnel-destination 44.44.44.44
!
ip decap-group decap-net-99
   tunnel type gre
   tunnel decap-ip 11.11.11.11

!!!!! R44 !!!!!!

ip route 99.1.1.0/24 Nexthop-Group nhg-gre-99-net

!

nexthop-group nhg-gre-99-net type gre

   size 1

   ttl 64

   tunnel-source 44.44.44.44

   entry 0 tunnel-destination 11.11.11.11

!

ip decap-group decap-net-99

   tunnel type gre

   tunnel decap-ip 44.44.44.44

Now the ping works well. 

host1#ping 99.2.2.99

PING 99.2.2.99 (99.2.2.99) 72(100) bytes of data.

80 bytes from 99.2.2.99: icmp_seq=1 ttl=62 time=0.264 ms

.....

--- 99.2.2.99 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.102/0.140/0.264/0.062 ms, ipg/ewma 0.184/0.200 ms

So basically that's how it works. 

• On each router has static nexthop + decap group configuration to other routers
• The application software works as a passive ISIS neighbor to establish a neighbor with one ISIS router over a GRE tunnel. 
• So it can fetch the whole LSA DB to get a whole view of the network. 
• By using CLI or eAPI, the software can program each router with a static route pointing to the nexthop group entry configured in step 1. 

Arista EOS Tunneling Mechanism (3) - Sw Tunnel + ISIS

 

Above is a simple setup to demonstrate the feature of ISIS over GRE tunnel. It is worthy to note that:

• This tunnel is a software tunnel without hardware programming. It ONLY works for the local packets generated by CPU and is capped by CoPP policy. So no data/traffic traffic is thru the tunnel. 
• The only valid use case per TOI is to advertise the ISIS routes to a remote application, like below
• The application can learn the whole view of a network topology
• Based application or business intelligence, it can program the routers to instruct the traffic flows. So a practical SDN solution to me. 

Limitation:

• IPv6 underlay endpoint is not supported
• Underlay VRF is not supported
• That says the tunnel source/destination must be ipv4 address under default VRF.

Some extras:

• Even in the above the TOI, it says it only supports ISIS, but in the lab, the BGP session is up and running w/o any issue. But one thing to be aware is to assign the tunnel interface TTL. Otherwise, the neighbor fails to come up

R11(config-router-bgp)#sh run sec router bgp

router bgp 11

   neighbor 10.100.100.44 remote-as 44

   neighbor 10.100.100.44 maximum-routes 12000

   redistribute connected

R11(config-router-bgp)#sh ip bgp sum

BGP summary information for VRF default

Router identifier 11.11.11.11, local AS number 11

Neighbor Status Codes: m - Under maintenance

  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State   PfxRcd PfxAcc

  10.100.100.44    4  44              4002      4013    0    0 01:29:18 Estab   11     11 <<<< session is up

• Overlay VRF is supported, so tunnel interface can belong to none-default VRF

R11(config)#sh run int tunnel 101

interface Tunnel101

   description sw-tunnel-gre-vrf-v1

   vrf forwarding v1

   ip address 10.101.101.11/24

   isis enable isis.over.GRE.v1

   isis bfd

   isis network point-to-point

   tunnel mode gre

   tunnel source 11.11.11.1

   tunnel destination 44.44.44.1

R11(config)#sh int tunnel 101

Tunnel101 is up, line protocol is up (connected)

  Hardware is Tunnel, address is 0b0b.0b01.0800

  Description: sw-tunnel-gre-vrf-v1

  Internet address is 10.101.101.11/24

  Broadcast address is 255.255.255.255

  Tunnel source 11.11.11.1, destination 44.44.44.1

....

R44(config)#sh run int tu101

interface Tunnel101

   description sw-tunnel-gre-vrf-v1

   vrf forwarding v1

   ip address 10.101.101.44/24

   isis enable isis.over.GRE.v1

   isis bfd

   isis network point-to-point

   tunnel mode gre

   tunnel source 44.44.44.1

   tunnel destination 11.11.11.1

R44(config)#sh int tu101

Tunnel101 is up, line protocol is up (connected)

  Hardware is Tunnel, address is 2c2c.2c01.0800

  Description: sw-tunnel-gre-vrf-v1

  Internet address is 10.101.101.44/24

  Broadcast address is 255.255.255.255

....

R11(config)#ping vrf v1 10.101.101.44

PING 10.101.101.44 (10.101.101.44) 72(100) bytes of data.

80 bytes from 10.101.101.44: icmp_seq=1 ttl=64 time=0.234 ms

80 bytes from 10.101.101.44: icmp_seq=2 ttl=64 time=0.153 ms

...

R11(config)#sh isis neighbors vrf v1

Instance  VRF      System Id        Type Interface          SNPA              State Hold time   Circuit Id

isis.over v1       R44              L1   Tunnel101          P2P               UP    25          87

R11(config)#sh ip route vrf v1 isis

VRF: v1

....

 I L1     101.101.44.1/32 [115/11] via 10.101.101.44, Tunnel101

 I L1     101.101.44.2/32 [115/11] via 10.101.101.44, Tunnel101

 I L1     101.101.44.3/32 [115/11] via 10.101.101.44, Tunnel101

 I L1     101.101.44.4/32 [115/11] via 10.101.101.44, Tunnel101

Arista EOS Tunneling Mechanism (2) - Hw/Sw Tunnel

We first talk a bit on GRE tunneling - "tunnel interface".

• Starting from 4.17.0F, you can configure a software GRE tunnel with ISIS over it. 
• Over the tunnel, it is ONLY for the control plane as specified in the feature limitation
• "The only supported use case is to allow IS-IS on the switch to advertise its routes over a GRE tunnel to a remote system that hosts IS-IS topology-learning applications."  
• So it is more like to establish isis neighbor and advertise ISIS LSDB to remote software. 
• In 4.21.1F, the hardware GRE tunnel is supported on Jericho platforms like 7280R, 7500R and 7020R. But as of now (Jun 2019), only supported overlay routing protocol is BGP/ISIS. 

Here is the detailed comparison between 2 types of tunnel interfaces:

Some configuration tips:

• The configuration is same, the system will automatically select hardware or software based on hardware platform and release. 
• "show interface tunnel #" shows the tunnel type. Please note that even this show interface output indicates the interface is up, it doesn't mean it is working. Better to verify by ping the tunnel interaface ip address. 
• Only GRE mode is supported, even there is other options like ipip or Ipsec in the CLI. 
• Underlay endpoint address family, only the ipv4 is supported, even there is options like "tunnel source/dest <ipv6Adr>"

Arista EOS Tunneling Mechanism (1) - Tunnel vs NHG/Decap

In Arista EOS, there are 2 types of tunnel mechanisms,

1. Regular tunnel interface - starting with "interface tunnel 10"
2. Next-hop-group + decap-group; 

The first one is easy to understand and other vendors have similar features. The second one is very unique. Based on the EOS manual, section 35.2 and 35.3 

• The decap group is a data structure that receives encapsulated packets and extracts the payload.
• A nexthop group is a data structure that defines a list of nexthop addresses and a tunnel type for packets routed to the specified address.

Comparison between Tunnel and next-hop-group/decap-group:

• The tunnel is 2-way vs NHG/Decap is a 1-way
• The tunnel is for both control plane and data plane, while NHG/Decap is ONLY for data plane, so no routing protocol over it
• Before 4.21.1F, the tunnel is software forwarding which is capped by the CoPP policy. NHG/Decap is hardware forwarding from day-1. 

Or you can think about it in this way:

• The tunnel interface is a traditional tunnel with a routing protocol, which is responsible for programing the forwarding information. 
• The nhg/decap is purely a data plane tunnel, and network admin can use EAPI to program the router's forwarding. 

In the following blog posts, will cover the details of configuration and troubleshooting for some different flavor of setup. Here is a (probably uncompleted) list of EOS NHG/DecapG and tunnel features

 

Trouble-shoot BGP peering issue over GRE tunnel

Starting from 4.21.1F, Arista EOS starts to support the hardware GRE tunnel interface and BGP session over the tunnel on Jericho platforms. Before the tunnel is implemented by nexthop-group and decap group. 

Here is a very simple and straightforward setup of eBGP over GRE tunnel. 

But BGP session fails to come up as shown below:

R1.gts425#sh ip bgp sum

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 1

Neighbor Status Codes: m - Under maintenance

  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

  10.100.100.4     4  4               4766       257    0    0 00:20:40 Connect 0      0

Tunnel interface is up and works fine. 

R1.gts425#sh int tunnel 100

Tunnel100 is up, line protocol is up (connected)

  Hardware is Tunnel, address is 0101.0101.0800

  Description: tunnel-gre-sand-to-sand

  Internet address is 10.100.100.1/24

  Broadcast address is 255.255.255.255

  Tunnel source 1.1.1.1, destination 4.4.4.4

  Tunnel protocol/transport GRE/IP

   Key disabled, sequencing disabled

   Checksumming of packets disabled

  Tunnel TTL 0, Hardware forwarding not supported

  Tunnel TOS 0

  Path MTU Discovery

  Tunnel transport MTU 1476 bytes

  Up 22 minutes, 2 seconds

Ping with MTU size works totally fine

R1.gts425#ping 10.100.100.4 size 1476

...

80 bytes from 10.100.100.4: icmp_seq=5 ttl=64 time=0.118 ms

--- 10.100.100.4 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.118/0.155/0.295/0.070 ms, ipg/ewma 0.223/0.222 ms

Now let's run the tcpdump on R1 to see if hello packet out

R1.gts425(config-router-bgp)#bash tcpdump -nvvi et21 host 1.1.1.1

tcpdump: listening on et21, link-type EN10MB (Ethernet), capture size 262144 bytes

11:19:19.774224 28:99:3a:8f:91:bf > 44:4c:a8:c1:78:69, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 1, id 39325, offset 0, flags [DF], proto GRE (47), length 84)

    1.1.1.1 > 4.4.4.4: GREv0, Flags [none], proto IPv4 (0x0800), length 64

(tos 0xc0, ttl 1, id 6538, offset 0, flags [DF], proto TCP (6), length 60)

    10.100.100.1.46931 > 10.100.100.4.bgp: Flags [S], seq 3552659475, win 28720, options [mss 1436,sackOK,TS val 18764185 ecr 0,nop,wscale 7], length 0

11:19:19.774420 44:4c:a8:c1:78:69 > 28:99:3a:8f:91:bf, ethertype IPv4 (0x0800), length 126: (tos 0xc0, ttl 64, id 29177, offset 0, flags [none], proto ICMP (1), length 112)

    10.1.2.2 > 1.1.1.1: ICMP time exceeded in-transit, length 92

(tos 0x0, ttl 1, id 39325, offset 0, flags [DF], proto GRE (47), length 84)

    1.1.1.1 > 4.4.4.4: GREv0, Flags [none], proto IPv4 (0x0800), length 64

(tos 0xc0, ttl 1, id 6538, offset 0, flags [DF], proto TCP (6), length 60)

    10.100.100.1.46931 > 10.100.100.4.bgp: Flags [S], seq 3552659475, win 28720, options [mss 1436,sackOK,TS val 18764185 ecr 0,nop,wscale 7], length 0

Now we can see the reason clearly. The eBGP TCP session is default with ttl 1 and copied to outer GRE packets, so the packets get TTL expired at 10.1.2.2 which is R2. 

To fix this issue, just need to set TTL under tunnel interface

R1.gts425(config-router-bgp)#int tu 100

R1.gts425(config-if-Tu100)#tunnel ttl 10

R1.gts425(config-router-bgp)#sh ip bgp sum

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 1

Neighbor Status Codes: m - Under maintenance

  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

  10.100.100.4     4  4                  4         4    0    0 00:00:01 Estab  0      0