Arista EOS Tunneling Mechanism (5) - Hw Tunnel

Starting from 4.21.1F, hardware GRE tunnel is supported on Jericho-based platforms. Below is a typical setup of hw tunnel and configuration



You can see, configuration-wise it is just like the software tunnel. How to tell which mode, check "show interface tunnel". 

R1#sh int tu200
Tunnel200 is up, line protocol is up (connected)
...
  Tunnel TTL 10, Hardware forwarding enabled

Compared with software tunnel, the biggest difference is the hardware programming. 

1. TCAM:

R2(config)#sh platform fap tcam summary
            Tcam Allocation (Jericho0)
   Bank                      Used By    Reserved By
---------- ---------------------------- -----------
      0                       dbVTT0              -
     12                     dbSystem              -
..

Now configuring an up tunnel interface

R2(config)#int tu100
R2(config-if-Tu100)#tunnel source 100.2.2.2
R2(config-if-Tu100)#tunnel destination 100.3.3.3
R2(config-if-Tu100)#show ip int brief
Interface              IP Address         Status     Protocol         MTU
....
Tunnel100              unassigned         up         up              1476

Now you can see a new Tcam allocation named dbGreTunnel

R2#sh platform fap tcam summary
            Tcam Allocation (Jericho0)
   Bank                      Used By    Reserved By
---------- ---------------------------- -----------
      0                       dbVTT0              -
      2                  dbGreTunnel              -
     12                     dbSystem              -

2. Tunnel interface

R1#show platform fap eedb ip-tunnel gre interface tunnel 200
....
| Bank/   | OutLIF  | Next    | VSI   | Encap | TOS  | TTL  | Source   | Destination     | OamLIF | OutLIF  | Drop |
| Offset  |         | OutLIF  | LSB   | Mode  |      |      | IP       | IP              | Set    | Profile |      |
|-----------------------------------------------------------------------------------------------------------|
| 2/0     | 0x6000  | 0x4012  | 0     | 2     | 0    | 10   | 100.1.1.1  | 100.4.4.4       | No     | 0       | No   |

* Bank should match the bank# in "show plat fap tcam summary"

3. Ip route

Below show 220.4.4.0/24 is learned via Tu200

R1(config-if-Tu200)#sh ip route

VRF: default
 I L2     220.4.4.0/24 [115/20] via 200.1.4.4, Tunnel200, Static Interface GRE tunnel index 200, dst 100.4.4.4, src 100.1.1.1, TTL 10

R1(config-if-Tu200)#show platform fap ip route | egrep 'VRF|ID|220.4'
|VRF|   Destination    |     |                    |     |       |                   | ECMP|  FEC | Tunnel
| ID|      Subnet      | Cmd |     Destination    | VID |Outlif |   MAC / CPU Code  |Index| Index|T Value
|0  |220.4.4.0/24      |ROUTE| FEC 32773          |0    |  -    |                   |  -  |49160 |   -

4. FEC

R1(config-if-Tu200)#show platform fap fec all
.....
|     |      |     |                    |     |       |                   |
| ECMP|  FEC |     |                    |     |       |                   | Tunnel
|Index| Index| Cmd |     Destination    | VID |Outlif |   MAC / CPU Code  |T Value
 ----------------------------------------------------------------------------------
.....
|  -  |49160 |ROUTE| FEC 32773          |0    |  -    |                   |   -
|  -  |32773 |ROUTE| Et21               |12   |  -    | 44:4c:a8:c1:78:69 |T 100.4.4.4

Limitations:

In above TOI link, there is a list of hw tunnel limitations, for example:

• Underlay endpoint (tunnel source/destination) must IPv4 address in default VRF. 
• Even there is tunnel source/destination ipv6 address option, but not supported on this platform
• Overlay can be IPv4/IPv6 under default or non-default VRF
• No GRE KA, if one side has incorrect GRE configuration like missing tunnel source, the other end of GRE interface is still up. 

Arista EOS Tunneling Mechanism (4) - Sw Tunnel vs Data

In the above topology, I show you that the data traffic doesn't go thru the sw tunnel. I use 2 routers to simulate hosts with default gateway pointing to R11 and R44. And we can see

• Ping between R11 and R44 works
• But the ping from host1 to host2 doesn't work

R11 has the correct route and ping to R44's ip address is good. 

R11#sh ip route 99.2.2.99
...
 I L1     99.2.2.0/24 [115/20] via 10.100.100.44, Tunnel100

R11.cd642.leaf18#ping 99.2.2.2

PING 99.2.2.2 (99.2.2.2) 72(100) bytes of data.

80 bytes from 99.2.2.2: icmp_seq=1 ttl=64 time=0.264 ms

80 bytes from 99.2.2.2: icmp_seq=2 ttl=64 time=0.119 ms

Host1 also has right route but ping failed, so data traffic can't pass thru. 

host1#sh ip route 99.2.2.99
....
 S        99.0.0.0/8 [1/0] via 99.1.1.1, Ethernet51/1

host1#ping 99.2.2.99
--- 99.2.2.99 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 40ms

Now, we add NHG + Decap group (will cover the details later) on both ends. 



!!!!! R11 !!!!!!
ip route 99.2.2.0/24 Nexthop-Group nhg-gre-99-net
!
nexthop-group nhg-gre-99-net type gre
   size 1
   ttl 64
   tunnel-source 11.11.11.11
   entry 0 tunnel-destination 44.44.44.44
!
ip decap-group decap-net-99
   tunnel type gre
   tunnel decap-ip 11.11.11.11

!!!!! R44 !!!!!!

ip route 99.1.1.0/24 Nexthop-Group nhg-gre-99-net

!

nexthop-group nhg-gre-99-net type gre

   size 1

   ttl 64

   tunnel-source 44.44.44.44

   entry 0 tunnel-destination 11.11.11.11

!

ip decap-group decap-net-99

   tunnel type gre

   tunnel decap-ip 44.44.44.44

Now the ping works well. 

host1#ping 99.2.2.99

PING 99.2.2.99 (99.2.2.99) 72(100) bytes of data.

80 bytes from 99.2.2.99: icmp_seq=1 ttl=62 time=0.264 ms

.....

--- 99.2.2.99 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.102/0.140/0.264/0.062 ms, ipg/ewma 0.184/0.200 ms

So basically that's how it works. 

• On each router has static nexthop + decap group configuration to other routers
• The application software works as a passive ISIS neighbor to establish a neighbor with one ISIS router over a GRE tunnel. 
• So it can fetch the whole LSA DB to get a whole view of the network. 
• By using CLI or eAPI, the software can program each router with a static route pointing to the nexthop group entry configured in step 1. 

Arista EOS Tunneling Mechanism (3) - Sw Tunnel + ISIS

 

Above is a simple setup to demonstrate the feature of ISIS over GRE tunnel. It is worthy to note that:

• This tunnel is a software tunnel without hardware programming. It ONLY works for the local packets generated by CPU and is capped by CoPP policy. So no data/traffic traffic is thru the tunnel. 
• The only valid use case per TOI is to advertise the ISIS routes to a remote application, like below
• The application can learn the whole view of a network topology
• Based application or business intelligence, it can program the routers to instruct the traffic flows. So a practical SDN solution to me. 

Limitation:

• IPv6 underlay endpoint is not supported
• Underlay VRF is not supported
• That says the tunnel source/destination must be ipv4 address under default VRF.

Some extras:

• Even in the above the TOI, it says it only supports ISIS, but in the lab, the BGP session is up and running w/o any issue. But one thing to be aware is to assign the tunnel interface TTL. Otherwise, the neighbor fails to come up

R11(config-router-bgp)#sh run sec router bgp

router bgp 11

   neighbor 10.100.100.44 remote-as 44

   neighbor 10.100.100.44 maximum-routes 12000

   redistribute connected

R11(config-router-bgp)#sh ip bgp sum

BGP summary information for VRF default

Router identifier 11.11.11.11, local AS number 11

Neighbor Status Codes: m - Under maintenance

  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State   PfxRcd PfxAcc

  10.100.100.44    4  44              4002      4013    0    0 01:29:18 Estab   11     11 <<<< session is up

• Overlay VRF is supported, so tunnel interface can belong to none-default VRF

R11(config)#sh run int tunnel 101

interface Tunnel101

   description sw-tunnel-gre-vrf-v1

   vrf forwarding v1

   ip address 10.101.101.11/24

   isis enable isis.over.GRE.v1

   isis bfd

   isis network point-to-point

   tunnel mode gre

   tunnel source 11.11.11.1

   tunnel destination 44.44.44.1

R11(config)#sh int tunnel 101

Tunnel101 is up, line protocol is up (connected)

  Hardware is Tunnel, address is 0b0b.0b01.0800

  Description: sw-tunnel-gre-vrf-v1

  Internet address is 10.101.101.11/24

  Broadcast address is 255.255.255.255

  Tunnel source 11.11.11.1, destination 44.44.44.1

....

R44(config)#sh run int tu101

interface Tunnel101

   description sw-tunnel-gre-vrf-v1

   vrf forwarding v1

   ip address 10.101.101.44/24

   isis enable isis.over.GRE.v1

   isis bfd

   isis network point-to-point

   tunnel mode gre

   tunnel source 44.44.44.1

   tunnel destination 11.11.11.1

R44(config)#sh int tu101

Tunnel101 is up, line protocol is up (connected)

  Hardware is Tunnel, address is 2c2c.2c01.0800

  Description: sw-tunnel-gre-vrf-v1

  Internet address is 10.101.101.44/24

  Broadcast address is 255.255.255.255

....

R11(config)#ping vrf v1 10.101.101.44

PING 10.101.101.44 (10.101.101.44) 72(100) bytes of data.

80 bytes from 10.101.101.44: icmp_seq=1 ttl=64 time=0.234 ms

80 bytes from 10.101.101.44: icmp_seq=2 ttl=64 time=0.153 ms

...

R11(config)#sh isis neighbors vrf v1

Instance  VRF      System Id        Type Interface          SNPA              State Hold time   Circuit Id

isis.over v1       R44              L1   Tunnel101          P2P               UP    25          87

R11(config)#sh ip route vrf v1 isis

VRF: v1

....

 I L1     101.101.44.1/32 [115/11] via 10.101.101.44, Tunnel101

 I L1     101.101.44.2/32 [115/11] via 10.101.101.44, Tunnel101

 I L1     101.101.44.3/32 [115/11] via 10.101.101.44, Tunnel101

 I L1     101.101.44.4/32 [115/11] via 10.101.101.44, Tunnel101

Arista EOS Tunneling Mechanism (2) - Hw/Sw Tunnel

We first talk a bit on GRE tunneling - "tunnel interface".

• Starting from 4.17.0F, you can configure a software GRE tunnel with ISIS over it. 
• Over the tunnel, it is ONLY for the control plane as specified in the feature limitation
• "The only supported use case is to allow IS-IS on the switch to advertise its routes over a GRE tunnel to a remote system that hosts IS-IS topology-learning applications."  
• So it is more like to establish isis neighbor and advertise ISIS LSDB to remote software. 
• In 4.21.1F, the hardware GRE tunnel is supported on Jericho platforms like 7280R, 7500R and 7020R. But as of now (Jun 2019), only supported overlay routing protocol is BGP/ISIS. 

Here is the detailed comparison between 2 types of tunnel interfaces:

Some configuration tips:

• The configuration is same, the system will automatically select hardware or software based on hardware platform and release. 
• "show interface tunnel #" shows the tunnel type. Please note that even this show interface output indicates the interface is up, it doesn't mean it is working. Better to verify by ping the tunnel interaface ip address. 
• Only GRE mode is supported, even there is other options like ipip or Ipsec in the CLI. 
• Underlay endpoint address family, only the ipv4 is supported, even there is options like "tunnel source/dest <ipv6Adr>"

Trouble-shoot BGP peering issue over GRE tunnel

Starting from 4.21.1F, Arista EOS starts to support the hardware GRE tunnel interface and BGP session over the tunnel on Jericho platforms. Before the tunnel is implemented by nexthop-group and decap group. 

Here is a very simple and straightforward setup of eBGP over GRE tunnel. 

But BGP session fails to come up as shown below:

R1.gts425#sh ip bgp sum

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 1

Neighbor Status Codes: m - Under maintenance

  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

  10.100.100.4     4  4               4766       257    0    0 00:20:40 Connect 0      0

Tunnel interface is up and works fine. 

R1.gts425#sh int tunnel 100

Tunnel100 is up, line protocol is up (connected)

  Hardware is Tunnel, address is 0101.0101.0800

  Description: tunnel-gre-sand-to-sand

  Internet address is 10.100.100.1/24

  Broadcast address is 255.255.255.255

  Tunnel source 1.1.1.1, destination 4.4.4.4

  Tunnel protocol/transport GRE/IP

   Key disabled, sequencing disabled

   Checksumming of packets disabled

  Tunnel TTL 0, Hardware forwarding not supported

  Tunnel TOS 0

  Path MTU Discovery

  Tunnel transport MTU 1476 bytes

  Up 22 minutes, 2 seconds

Ping with MTU size works totally fine

R1.gts425#ping 10.100.100.4 size 1476

...

80 bytes from 10.100.100.4: icmp_seq=5 ttl=64 time=0.118 ms

--- 10.100.100.4 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.118/0.155/0.295/0.070 ms, ipg/ewma 0.223/0.222 ms

Now let's run the tcpdump on R1 to see if hello packet out

R1.gts425(config-router-bgp)#bash tcpdump -nvvi et21 host 1.1.1.1

tcpdump: listening on et21, link-type EN10MB (Ethernet), capture size 262144 bytes

11:19:19.774224 28:99:3a:8f:91:bf > 44:4c:a8:c1:78:69, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 1, id 39325, offset 0, flags [DF], proto GRE (47), length 84)

    1.1.1.1 > 4.4.4.4: GREv0, Flags [none], proto IPv4 (0x0800), length 64

(tos 0xc0, ttl 1, id 6538, offset 0, flags [DF], proto TCP (6), length 60)

    10.100.100.1.46931 > 10.100.100.4.bgp: Flags [S], seq 3552659475, win 28720, options [mss 1436,sackOK,TS val 18764185 ecr 0,nop,wscale 7], length 0

11:19:19.774420 44:4c:a8:c1:78:69 > 28:99:3a:8f:91:bf, ethertype IPv4 (0x0800), length 126: (tos 0xc0, ttl 64, id 29177, offset 0, flags [none], proto ICMP (1), length 112)

    10.1.2.2 > 1.1.1.1: ICMP time exceeded in-transit, length 92

(tos 0x0, ttl 1, id 39325, offset 0, flags [DF], proto GRE (47), length 84)

    1.1.1.1 > 4.4.4.4: GREv0, Flags [none], proto IPv4 (0x0800), length 64

(tos 0xc0, ttl 1, id 6538, offset 0, flags [DF], proto TCP (6), length 60)

    10.100.100.1.46931 > 10.100.100.4.bgp: Flags [S], seq 3552659475, win 28720, options [mss 1436,sackOK,TS val 18764185 ecr 0,nop,wscale 7], length 0

Now we can see the reason clearly. The eBGP TCP session is default with ttl 1 and copied to outer GRE packets, so the packets get TTL expired at 10.1.2.2 which is R2. 

To fix this issue, just need to set TTL under tunnel interface

R1.gts425(config-router-bgp)#int tu 100

R1.gts425(config-if-Tu100)#tunnel ttl 10

R1.gts425(config-router-bgp)#sh ip bgp sum

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 1

Neighbor Status Codes: m - Under maintenance

  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

  10.100.100.4     4  4                  4         4    0    0 00:00:01 Estab  0      0

Arista EOS Tunneling (2) - GRE

Supported and Release:
The GRE Tunneling in EOS is supported from 4.15.0F (Apr 2015)

Not like the interface tunnel configuration in Cisco OS's, Arista uses nexthop-group and decap-group as the encap and decap endpoints. (EOS also has tunnel interfaces, but it is performed by software and ONLY for control plane purpose).

GRE Header Info:

• Protocol #: 47 (0x2F)
• Between Outer IP Header and Payload packet:
• GRE Flags (2B)
• Protocol Type (2B) = 0x0800 (ipv4)
• Key (4B) if enabled tunnel-key shown as below

Diagram and configuration:

Configuration details:

• nexthop-group <tunnel_intf_name> type gre
• on Arad platform: there is 4 encap - gre, ip-in-ip, mpls and mpls-over-gre
• size 1:
• # of ecmp path
• tunnel-source intf <intf, loopback most of time>
• Source ip of outer GRE/IP header
• entry 0 tunnel-destination <ip>
• Dest ip of GRE/IP header
• match above size #, starting from 0
• tunnel-key ingress-interface
• if enabled, will add 4B of GRE key value after 4B GRE header(flag + protocol#). This key is calculated from ingress interface + vlan id. 

Show output:

R1.yo412#sh ip decap-group

NOTE: "D" column indicates dynamic entries

D | Name               | Type       | Info              | Version | Address Type

--|--------------------|------------|-------------------|---------|-------------

  | gre-tunnel-1       | GRE        | 100.1.255.1       |         |

R1.yo412#sh platform arad decap-group

        DecapIp         |   LIF

------------------------+-------

100.1.255.1             |      1

R1.yo412#show nexthop-group

gre-tunnel-1

  Id         2

  Type       gre  (key: ingress-interface)

  Size       1

  TTL        64

  Source IP  100.1.255.1 (Loopback1)

  Counters   shared

  Entries (left most label is the top of the stack)

    0  100.1.255.2

         Via 100.1.12.2, Ethernet7/36/1  (100.1.255.2/32)

         00:1c:73:44:58:d0, Ethernet7/36/1

R1.yo412#sh platform arad ip nexthop-group

gre-tunnel-1

Type: GRE (key: ingress-interface)

Ecmp Fec Id: -

 ---------------------------------------------------------------------------------------------------------

|Pos|  Tunnel DIP   |Tunnel SIP [idx]    |TTL[idx]| FEC |OLIF |Cmd |     Intf      |VID | MAC / CPU Code  |

 ---------------------------------------------------------------------------------------------------------

|0  |100.1.255.2    |100.1.255.1    [0  ]|64 [0  ]|1047 |6144 |GRE |Et7/36/1       |3983|00:1c:73:44:58:d0|

R1.yo412#sh platform arad ip route | egrep '222|VRF|ID'

|VRF|   Destination    |     |                    |     |       |                   | ECMP|  FEC | Tunnel

| ID|      Subnet      | Cmd |     Destination    | VID |Outlif |   MAC / CPU Code  |Index| Index|T Value

|0  |222.100.0.0/16    |ROUTE| Et5/1/1            |3984 |4094   | 00:1c:73:3b:ce:c9 |  -  |1046  |   -

|0  |222.200.0.0/16    |ROUTE| Et7/36/1           |3983 |  -    | 00:1c:73:44:58:d0 |  -  |1047  |G 100.1.255.2

Arista EOS Tunneling (1)

Arista EOS Tunneling is a little different from other vendors' implmentations

• Encap and decap are separated, which are defined in nexthop-group and decap-group. 
• On 1 router configure nexthop-group and on other router only has decap group. 
• For example, you can have internet traffic to FW, which tunnels the packets to severs. And servers directly route the returning traffic to users. 
• nexthop-group does NOT equal to tunnel interface. It is not an interface, actually it is encap endpoint. 
• Starting from 4.21.1F, EOS support hw GRE tunnel interface. Only on Jericho platforms - 7020R, 7280R, 7500R. 

Tunneling vs platforms(chipset) and releases. 

• Sand platform - Arad/Jericho/J+/Qurman
• GRE encap + decap
• IPinIP decap only
• Strata platform - Trident 2/Thomahawk
• IPinIP encap + decap

In this quick lab, we will configure the GRE and IPinIP tunnels in the following topology:

• Use ONE router - RtrAB to simulate end hosts by dividing into VRFs.
• 222.10.10.1 -- 222.10.20.1 over GRE tunnel
• 222.20.10.1 -- 222.20.20.1 over IPinIP tunnel