Arista EOS Tunneling Mechanism (4) - Sw Tunnel vs Data

In the above topology, I show you that the data traffic doesn't go thru the sw tunnel. I use 2 routers to simulate hosts with default gateway pointing to R11 and R44. And we can see

• Ping between R11 and R44 works
• But the ping from host1 to host2 doesn't work

R11 has the correct route and ping to R44's ip address is good. 

R11#sh ip route 99.2.2.99
...
 I L1     99.2.2.0/24 [115/20] via 10.100.100.44, Tunnel100

R11.cd642.leaf18#ping 99.2.2.2

PING 99.2.2.2 (99.2.2.2) 72(100) bytes of data.

80 bytes from 99.2.2.2: icmp_seq=1 ttl=64 time=0.264 ms

80 bytes from 99.2.2.2: icmp_seq=2 ttl=64 time=0.119 ms

Host1 also has right route but ping failed, so data traffic can't pass thru. 

host1#sh ip route 99.2.2.99
....
 S        99.0.0.0/8 [1/0] via 99.1.1.1, Ethernet51/1

host1#ping 99.2.2.99
--- 99.2.2.99 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 40ms

Now, we add NHG + Decap group (will cover the details later) on both ends. 



!!!!! R11 !!!!!!
ip route 99.2.2.0/24 Nexthop-Group nhg-gre-99-net
!
nexthop-group nhg-gre-99-net type gre
   size 1
   ttl 64
   tunnel-source 11.11.11.11
   entry 0 tunnel-destination 44.44.44.44
!
ip decap-group decap-net-99
   tunnel type gre
   tunnel decap-ip 11.11.11.11

!!!!! R44 !!!!!!

ip route 99.1.1.0/24 Nexthop-Group nhg-gre-99-net

!

nexthop-group nhg-gre-99-net type gre

   size 1

   ttl 64

   tunnel-source 44.44.44.44

   entry 0 tunnel-destination 11.11.11.11

!

ip decap-group decap-net-99

   tunnel type gre

   tunnel decap-ip 44.44.44.44

Now the ping works well. 

host1#ping 99.2.2.99

PING 99.2.2.99 (99.2.2.99) 72(100) bytes of data.

80 bytes from 99.2.2.99: icmp_seq=1 ttl=62 time=0.264 ms

.....

--- 99.2.2.99 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.102/0.140/0.264/0.062 ms, ipg/ewma 0.184/0.200 ms

So basically that's how it works. 

• On each router has static nexthop + decap group configuration to other routers
• The application software works as a passive ISIS neighbor to establish a neighbor with one ISIS router over a GRE tunnel. 
• So it can fetch the whole LSA DB to get a whole view of the network. 
• By using CLI or eAPI, the software can program each router with a static route pointing to the nexthop group entry configured in step 1. 

Arista EOS Tunneling Mechanism (2) - Hw/Sw Tunnel

We first talk a bit on GRE tunneling - "tunnel interface".

• Starting from 4.17.0F, you can configure a software GRE tunnel with ISIS over it. 
• Over the tunnel, it is ONLY for the control plane as specified in the feature limitation
• "The only supported use case is to allow IS-IS on the switch to advertise its routes over a GRE tunnel to a remote system that hosts IS-IS topology-learning applications."  
• So it is more like to establish isis neighbor and advertise ISIS LSDB to remote software. 
• In 4.21.1F, the hardware GRE tunnel is supported on Jericho platforms like 7280R, 7500R and 7020R. But as of now (Jun 2019), only supported overlay routing protocol is BGP/ISIS. 

Here is the detailed comparison between 2 types of tunnel interfaces:

Some configuration tips:

• The configuration is same, the system will automatically select hardware or software based on hardware platform and release. 
• "show interface tunnel #" shows the tunnel type. Please note that even this show interface output indicates the interface is up, it doesn't mean it is working. Better to verify by ping the tunnel interaface ip address. 
• Only GRE mode is supported, even there is other options like ipip or Ipsec in the CLI. 
• Underlay endpoint address family, only the ipv4 is supported, even there is options like "tunnel source/dest <ipv6Adr>"

Arista EOS Tunneling Mechanism (1) - Tunnel vs NHG/Decap

In Arista EOS, there are 2 types of tunnel mechanisms,

1. Regular tunnel interface - starting with "interface tunnel 10"
2. Next-hop-group + decap-group; 

The first one is easy to understand and other vendors have similar features. The second one is very unique. Based on the EOS manual, section 35.2 and 35.3 

• The decap group is a data structure that receives encapsulated packets and extracts the payload.
• A nexthop group is a data structure that defines a list of nexthop addresses and a tunnel type for packets routed to the specified address.

Comparison between Tunnel and next-hop-group/decap-group:

• The tunnel is 2-way vs NHG/Decap is a 1-way
• The tunnel is for both control plane and data plane, while NHG/Decap is ONLY for data plane, so no routing protocol over it
• Before 4.21.1F, the tunnel is software forwarding which is capped by the CoPP policy. NHG/Decap is hardware forwarding from day-1. 

Or you can think about it in this way:

• The tunnel interface is a traditional tunnel with a routing protocol, which is responsible for programing the forwarding information. 
• The nhg/decap is purely a data plane tunnel, and network admin can use EAPI to program the router's forwarding. 

In the following blog posts, will cover the details of configuration and troubleshooting for some different flavor of setup. Here is a (probably uncompleted) list of EOS NHG/DecapG and tunnel features

 

Arista EOS Tunneling (3) - IPinIP

IP-in-IP format:

• Size = 20B, just an ip header, 
• Protocol# = 4

Digram and configuration:

Verification:

R4.cd681#sh ip decap-group
NOTE: "D" column indicates dynamic entries
D | Name         | Type           | Info               | Version | Address Type
--|--------------|----------------|--------------------|---------|-------------
  | ipinip       | IP-in-IP       | 100.20.255.4       | IPv4    |

R4.cd681#show platform trident l3 shadow tunnel encap raw
EGR IP Tunnel:
--------------------------------------------------------------------------------
maxEntries: 512, numEntries: 1
EGR_IP_TUNNEL Address: 0xd72817a4, magic:aeaeaeae, maxEntries: 512, numEntries: 1

EGR_IP_TUNNEL[2]:  0x01  | 0x1 0x0 0x0 0x100000 0x9053fc11 0x9053fc0d 0x1 0x0


R4.cd681#show platform trident l3 shadow tunnel encap
EGR IP Tunnel:
--------------------------------------------------------------------------------
Unit    Id       TnlType       EntType       TTL       DscpSel              SrcIp           DstIp
0        2             0          IPV4        64             1       100.20.255.4    100.20.255.3

R4.cd681#show nexthop-group
ipip
  Id         1
  Type       ipInIp
  Size       1
  TTL        64
  Source IP  100.20.255.4 (Loopback20)
  Entries (left most label is the top of the stack)
    0  100.20.255.3
         Via 100.20.34.3, Ethernet31  (100.20.255.3/32)
         00:1c:73:54:27:cb, Ethernet31

Arista EOS Tunneling (2) - GRE

Supported and Release:
The GRE Tunneling in EOS is supported from 4.15.0F (Apr 2015)

Not like the interface tunnel configuration in Cisco OS's, Arista uses nexthop-group and decap-group as the encap and decap endpoints. (EOS also has tunnel interfaces, but it is performed by software and ONLY for control plane purpose).

GRE Header Info:

• Protocol #: 47 (0x2F)
• Between Outer IP Header and Payload packet:
• GRE Flags (2B)
• Protocol Type (2B) = 0x0800 (ipv4)
• Key (4B) if enabled tunnel-key shown as below

Diagram and configuration:

Configuration details:

• nexthop-group <tunnel_intf_name> type gre
• on Arad platform: there is 4 encap - gre, ip-in-ip, mpls and mpls-over-gre
• size 1:
• # of ecmp path
• tunnel-source intf <intf, loopback most of time>
• Source ip of outer GRE/IP header
• entry 0 tunnel-destination <ip>
• Dest ip of GRE/IP header
• match above size #, starting from 0
• tunnel-key ingress-interface
• if enabled, will add 4B of GRE key value after 4B GRE header(flag + protocol#). This key is calculated from ingress interface + vlan id. 

Show output:

R1.yo412#sh ip decap-group

NOTE: "D" column indicates dynamic entries

D | Name               | Type       | Info              | Version | Address Type

--|--------------------|------------|-------------------|---------|-------------

  | gre-tunnel-1       | GRE        | 100.1.255.1       |         |

R1.yo412#sh platform arad decap-group

        DecapIp         |   LIF

------------------------+-------

100.1.255.1             |      1

R1.yo412#show nexthop-group

gre-tunnel-1

  Id         2

  Type       gre  (key: ingress-interface)

  Size       1

  TTL        64

  Source IP  100.1.255.1 (Loopback1)

  Counters   shared

  Entries (left most label is the top of the stack)

    0  100.1.255.2

         Via 100.1.12.2, Ethernet7/36/1  (100.1.255.2/32)

         00:1c:73:44:58:d0, Ethernet7/36/1

R1.yo412#sh platform arad ip nexthop-group

gre-tunnel-1

Type: GRE (key: ingress-interface)

Ecmp Fec Id: -

 ---------------------------------------------------------------------------------------------------------

|Pos|  Tunnel DIP   |Tunnel SIP [idx]    |TTL[idx]| FEC |OLIF |Cmd |     Intf      |VID | MAC / CPU Code  |

 ---------------------------------------------------------------------------------------------------------

|0  |100.1.255.2    |100.1.255.1    [0  ]|64 [0  ]|1047 |6144 |GRE |Et7/36/1       |3983|00:1c:73:44:58:d0|

R1.yo412#sh platform arad ip route | egrep '222|VRF|ID'

|VRF|   Destination    |     |                    |     |       |                   | ECMP|  FEC | Tunnel

| ID|      Subnet      | Cmd |     Destination    | VID |Outlif |   MAC / CPU Code  |Index| Index|T Value

|0  |222.100.0.0/16    |ROUTE| Et5/1/1            |3984 |4094   | 00:1c:73:3b:ce:c9 |  -  |1046  |   -

|0  |222.200.0.0/16    |ROUTE| Et7/36/1           |3983 |  -    | 00:1c:73:44:58:d0 |  -  |1047  |G 100.1.255.2

Arista EOS Tunneling (1)

Arista EOS Tunneling is a little different from other vendors' implmentations

• Encap and decap are separated, which are defined in nexthop-group and decap-group. 
• On 1 router configure nexthop-group and on other router only has decap group. 
• For example, you can have internet traffic to FW, which tunnels the packets to severs. And servers directly route the returning traffic to users. 
• nexthop-group does NOT equal to tunnel interface. It is not an interface, actually it is encap endpoint. 
• Starting from 4.21.1F, EOS support hw GRE tunnel interface. Only on Jericho platforms - 7020R, 7280R, 7500R. 

Tunneling vs platforms(chipset) and releases. 

• Sand platform - Arad/Jericho/J+/Qurman
• GRE encap + decap
• IPinIP decap only
• Strata platform - Trident 2/Thomahawk
• IPinIP encap + decap

In this quick lab, we will configure the GRE and IPinIP tunnels in the following topology:

• Use ONE router - RtrAB to simulate end hosts by dividing into VRFs.
• 222.10.10.1 -- 222.10.20.1 over GRE tunnel
• 222.20.10.1 -- 222.20.20.1 over IPinIP tunnel

Nexthop Group (2)

NHG supported features are different per Arista platforms:

This is DCS-7060CX based on Tomahwak/Strata, only supports IP next hop, ip-in-ip and mpls

DCS-7060CX(config)#nexthop-group test type ?
  ip        ecmp next hop
  ip-in-ip  ip-in-ip tunnel type
  mpls      mpls tunnel type

This is DCS-7512N based on Jericho/Sand, which supports GRE, ip next hop, mpls and mpls-over-gre.

DCS-7512N(s1)(config)#nexthop-group test type ?
  gre            gre tunnel type
  ip             ecmp next hop
  mpls           mpls tunnel type
  mpls-over-gre  mpls-over-gre tunnel type

Nexthop Group (1)

为什么需要一个这样的功能？比方说，以下一个典型的JY网络

           [internet]<---- user: 200.1.1.1
           /        \
          /          \
   [borderLeaf]  [borderLeaf]
       |||          ||||
[...........spine nodes..........]
       |||          ||||
     [Leaf]        [Leaf  ]
     /    \        /      \
  [LB1]  [LB2] [SVR1]     [SVR2]
[customer.com] [10.1.1.1]
[100.1.1.1   ]            [10.1.1.1]
    
数据包流是这样的

• 每个LB advertise 一组LB IP给用户，应该是internet public ip 
• 用户的请求先到LB，LB再通过IP-in-IP，把数据包发给服务器，extDstIP = SvrIP
• 服务器Decap pkts，处理完之后直接返回给用户
• 所以数据流是: internet --> LB --> Svr --> internet

这个功能做啥哪？主要是switch offload SLB workload

• 第一个是 IPinIP encap
• 在LB上面的Leaf上，create static routes to Svr by NHG-IPinIP-tunnel。LB直接就转发native ip pkts, switch负责encap ip-in-ip tunnel header，因为是hw，比LB高速有效
• Server LB - 上图里面的SVR1/2都有DstIP = 10.1.1.1，ECMP hashing会选一个final dest. 
• 可以在NHG-IPinIP-tunnel增加 ECMP，从而达到network LB

需要啥？

• T+ platform
• LB using ECMP
• v4-in-v4, v6-in-v6，以后的releae应该可以是v4-over-v6 or verse vice
• Scale: 512 v4/256 v6 tunnels
• EAPI: 1) config NHG; 2) config static route pointing to NHG

CLI:

• size = # of NH entries in NHG, default = 256
• entry <index> tunnel dest <dstIP>, dstIP = svrIP，这里在做LB!
• index# = 0..size-1, 如果缺失就是drop, 注意！
• 所以hashing很少啊，srcIP唯一, proto#=4(IPinIP)也是唯一，全靠dstIP
• ip route 10.1.1.1/32 nexthop-group NHG
• 如果route to dstIP本身就是multipath, 那么就选第一条

注意的几点

• 这个NHG不是interface，什么意思哪？
• 他不会像正常的interface去解析Nexthop
• 正常的情况下，解析是ASIC fwd to software due to missing NH info. 在Tunnel情况，就是software自己去解析
• 为啥有drop entry，主要就不会generate ICMP unreachable msg
• multiple NHG可以share tunnels
• 如果多个routes to host ip，gated show多个，但是hw里面只有NHG
• 可以VRF吗？不知道
• IP routing on/off，会重写hw，改变interface ip也是
• MTU必须注意，增加了20B (v4) or 40B (v6)， 需要config correct MTU on ALL interfaces
• RPF: srcIP随意配置，所以注意不要被RPF给drop了
• Linux Kernel has a drop entry for each IP, how to check?

为啥不叫IPinIP tunneling? 为啥取个新名字

• NHG != tunnel interface, 3个区别:
• 没有路由协议
• 不会接受数据包
• 而且NHG可以有不同种类的Encap: LSP/MPLS, GRE, IPinIP
• 而且这是一个比interface更底层的概念，更像Adj/NHG = fib entry, 或者via = ECMP adj 

(Z1446)