Arista EOS/Networking Tips by Solomon Yang

10/28/2019

EOS - Find the interfaces with the most flaps

bn302.12:40:24#show int phy | egrep '^Eth' | awk '{print $3, $1}' | sort -rn | head -10
479 Ethernet9/25/2
23 Ethernet9/11/1
5 Ethernet9/34/1
5 Ethernet3/34/1

BGP Oscillation (RFC 3345)

BGP Oscillation is common in the context of MED + RR, because,

BGP only advertise the best path and hide the full list.
Similar to RR, it is also with BGP confederation

RFC 3345 has a very good depiction of how it happens

And I converted it to a google slides for a better view

in RFC, there lists a couple of design options to avoid this kind of churn

In RFC 7964, advertises all available paths by using ADD-PATH
Always compare MED even from different AS;
Don't accept MED
Utilize other BGP attributes higher in the decision process. (a little risky, because if any prefix leaked w/o higher attribute and tied)
Assign a high IGP cost to inter-cluster-link

10/15/2019

AS_PATH and Community regexp tips

1. match certain # of as_path

ip as-path access-list SixAS permit ^._._._._._.$ any

2. community list for some specific ending number, like ending 123. Because there is a space at the end of comm list, so need an underscore _, like

ip community-list regexp Ending123 permit ^.*:.*123_$

10/08/2019

Router Server

References:

Router Server = Internet Exchange (IX) Route Server = eBGP RR

Used in internet exchange
eBGP RR, so no need a full mesh eBGP peering among all parties
Reduces configuration complexity and CPU/memory overhead on border routers

Requirements:

AS_PATH transparency:

RS doesn't append its own AS# in the AS_PATH
RS-client doesn't enforce first AS

Nexthop transparency:

doesn't change NRLI's NH

MED transparency:

doesn't change path MED

Path hiding:

Per RFC 4271, BGP only advertises the best path, and a later update will be considered as an implicit withdrawal of the existing path.
In RFC 7947, it does mention the "path hiding". So the RS only advertises the best path, which could slow down the convergence time

EOS configurations:

service routing protocols model multi-agent << must multi-agent

route-map rtmap-no-change-med permit 10

set metric +0 << keep metric unchanged

router bgp 10

router-id 10.255.255.251

neighbor RS-Client peer group

neighbor RS-Client next-hop-unchanged << NH unchanged

neighbor RS-Client as-path prepend-own disabled << no own as#

neighbor RS-Client route-map rtmap-not-change-med out << rtmap

neighbor RS-Client password 7 xLcnzAMGHkI=

neighbor RS-Client send-community

neighbor RS-Client maximum-routes 12000

neighbor 10.10.10.11 peer group RS-Client

neighbor 10.10.10.11 remote-as 11

How many prefixes received? Considering the following scenario:

3 peers advertising ONE prefix
2 route servers
And all 3 peers also have bilateral peering
Then should be 5

Router1.15:19:15#sh ip bgp 11.111.1.1

BGP routing table information for VRF default

Router identifier 180.255.255.1, local AS number 65100

BGP routing table entry for 11.111.1.1/32

Paths: 5 available

12 111

10.10.10.12 from 10.10.10.12 (10.255.255.12) << from R2

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP head, ECMP, best, ECMP contributor

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.11 (10.255.255.11) << from R1

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP, ECMP contributor

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.251 (10.255.255.251) << R1 vis RS1

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP, ECMP contributor

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.252 (10.255.255.252) << R1 via RS2

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP, ECMP contributor

Rx SAFI: Unicast

13 333 111

10.10.10.13 from 10.10.10.13 (10.255.255.13) << R3

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external

Rx SAFI: Unicast

Let's shut down the private peering to R2. So you can see the RS1 and RS2 don't advertise R2's path.

Router.15:26:31(config-router-bgp)#sh ip bgp 11.111.1.1

BGP routing table information for VRF default

Router identifier 180.255.255.1, local AS number 65100

BGP routing table entry for 11.111.1.1/32

Paths: 4 available

11 111

10.10.10.11 from 10.10.10.252 (10.255.255.252) << RS2

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external, best

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.251 (10.255.255.251) << RS1

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.11 (10.255.255.11) << R1

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external

Rx SAFI: Unicast

12 111

10.10.10.12 from 10.10.10.12 (10.255.255.12) << R2

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external

Rx SAFI: Unicast

10/03/2019

EOS: A quick alias to show the interfaces with most util%

bn302#bash Cli -p15 -c 'srnz' | awk '{print $5, $8, $1}' | sort -r | more
Mbps Kpps Port
1.6% 4.5% Et9/36/1
1.5% 4.3% Et3/36/1
0.7% 0.5% Et13/33/1
0.7% 0.4% Et12/36/2

or set it as an alias

srnzsort bash Cli -p15 -c 'srnz' | awk '{print $5, $8, $1}' | sort -r | head -n 6

Updated note: The above command works with interface names/description defined. If not, have to change awk '{print $5, $8, $1}' to awk '{print $4, $7, $1}'

hs486.11:32:15#srnz
Port Name Intvl In Mbps % In Kpps Out Mbps % Out Kpps
Et49/1 0:05 9870.4 100.0% 812 9860.3 99.9% 811

hs486.12:14:45#bash Cli -p15 -c 'srnz' | awk '{print $4, $7, $1}' | sort -r | more
In In Port
100.0% 99.9% Et50/1
100.0% 99.9% Et49/4

How about some interfaces with description, some don't? hm.... Let me think about it..... :-)

9/19/2019

Ping bgp ipv6 link local address

With the feature of "BGP IPv6 link-local peering support", you can establish the bgp peering via ipv6 link-local address. To check the underlying reachability, we can use linux kernel command - ping6

For example,

interface Vlan4001
vrf forwarding vrf3_BGP_v6P_v6Ll
ipv6 address fe80::1/64 link-local
!
router bgp 65100
vrf vrf3_BGP_v6P_v6Ll
rd 65000:3
bgp default ipv4-unicast
bgp default ipv4-unicast transport ipv6
bgp default ipv6-unicast
neighbor fe80::100%Vl4001 remote-as 65500

bn303#bash sudo ip netns exec ns-vrf3_BGP_v6P_v6Ll ping6 -I vlan4001 fe80::100
PING fe80::100(fe80::100) from fe80::1%vlan4001 vlan4001: 56 data bytes
64 bytes from fe80::100%vlan4001: icmp_seq=1 ttl=64 time=0.483 ms
64 bytes from fe80::100%vlan4001: icmp_seq=2 ttl=64 time=0.238 ms
.....

Please note that this peering is under VRF, so you have to use the namespace command - "ip netns..."

9/09/2019

Kill list of zombie process on Linux(Eos)

Due to a script bug, one EOS device has quite some zombie process like,

[admin@bn302 flash]$ ps aux | grep 'CliShell -A'
root 28804 0.0 0.0 10504 8580 ? S 10:53 0:00 CliShell -A -p 15 -c show port-channel | json
root 28809 0.0 0.0 10504 8688 ? S 10:53 0:00 CliShell -A -p 15 -c show port-channel | json
root 28826 0.0 0.0 10504 8612 ? S 10:54 0:00 CliShell -A -p 15 -c show port-channel | json
root 28835 0.0 0.0 10504 8580 ? S 10:54 0:00 CliShell -A -p 15 -c show port-channel | json
root 28844 0.0 0.0 10504 8616 ? S 10:54 0:00 CliShell -A -p 15 -c show port-channel | json
....

Instead of killing one by one manually, write a quick shell command to kill all. (killall is not supported in EOS/Linux)

[admin@bn302 flash]$ for pid in $(ps aux | grep 'CliShell -A' | awk '{print $2, $11}' | grep CliShell | awk '{print $1}'); do sudo kill -9 $pid; done

7/31/2019

ISIS multi-topology - interfaces with different IPv4/v6 AF enabled

As we know, the ISIS requires address family configuration must be matched between the ISIS router and interface configuration. So if you have both ipv4 and ipv6 AF configuration, you must do the same under interface.

Sometime you may have some of interfaces only enabled ipv4, but dual stack globally, in this case, you will need the ISIS multi-topology.

router isis VrfDefault
net 49.0001.0000.0000.0002.00
router-id ipv4 100.255.255.2
is-type level-1
!
address-family ipv4 unicast
!
address-family ipv6 unicast
multi-topology <<< need multi-topology enabled here
!
interface Port-Channel211
no switchport
ip address 100.201.1.0/31
ipv6 address 100:201:1::/127
isis enable VrfDefault <<< this intf has both v4 and v6
!
interface Port-Channel251
no switchport
ip address 100.205.1.0/31
isis enable VrfDefault <<< but this intf only v4
isis multi-topology address-family ipv4 unicast <<<

And both neighbors are up!!

Router1(config-if-Po251)#show isis nei
Instance VRF System Id Type Interface SNPA State Hold time Circuit Id
VrfDef default RouterDualStack L1 Port-Channel211 P2P UP 3 48
VrfDef default RouterV4 L1 Port-Channel251 P2P UP 2 63

7/21/2019

MACSec

Arista EOS starts to support MACSec - 802.1AE from 4.15.4F which was released around 2016. Later related features:

4.17.0F - MACSec EAP-FAST (key server)
4.21.1F - MACSec Proxy
Arista MACSec WP (a list of supported hw)
Cisco MACSec WP (ipsec vs macsec)

Some key take-aways:

Purpose: to protect data traffic from various types of attacks,

passive: snooping
active: reply, man in the middle

vs IPSec

Level: IPSec is at the IP level, vs MACSec MAC/data link level.
End or Hop: IPSec provides end-to-end protection, MACsec secures Per-hop. (TLS is similar like IPSec, end-to-end)
HW: IPSec doesn't rely on hw but with IPSec often, MACSec is on PHY level with special hw. For example,

7500E-6CFPX-LC
7280SRAM-48C6
7280CR2M-30
7500R2M-36CQ-LC

Throughput: IPSec has a ceiling, while MACsec is line-rate like 100G.

Key components:

CAK - Conn Ass Key, master key. Either manual or from key server
CAN - CAK's name
SAK - Secu Ass Key, derived from CAK, used to data encryption.

Point-to-Point vs clear 802.1Q

Clear 802.1Q means, move vlan tag ahead of MACSec Tag, a must for hub-spoke topology. Cisco ASR supports it.
Arista EOS only supports point-to-point.

Packet format:

Add 16-byte SecTAG after srcMAC and VLAN Tag
Add 16-byte ICV at end of ethernet frames
Ethertype = 0x88e5 (why not use 0x85ec?, hahaha)

MACSec proxy:

Provides MACSec for VxLAN traffic
Loopback traffic to a MACSec capable front panel port to process.

MISC:

fallback key in case MACSec failed or during a configuration change
L2-protocol pass thru, skipping LLDP packets

Configuration (EOS):

mac security

license DCI1 <license>

profile toISP1

! key <CAN> <type, 0-clear text> <key>

key a1 0 a123456

interface eth4/1/1

mac security profile toISP1

show mac secu counters

show mac secu interface

7/18/2019

Arista Modular Switch - simulate LC remove/insert

!! bash cat /persist/sys/Linecard3.cfg
platform module Linecard3 remove
platform module Linecard3 insert

7508E CPU Util% is high (1)

Saw quite slow response on one of my lab router, a 7508E with latest EOS release.

mlagA.10:18:17(config)#show ver
Arista DCS-7508
Hardware version: 06.00
....
Software image version: 4.22.0.1F

CPU only has 64% idle cycles. Not low.

mlagA.10:18:13(config)#show proc top once | more
%Cpu(s): 29.6 us, 3.9 sy, 0.0 ni, 64.0 id, 0.1 wa, 0.5 hi, 2.0 si, 0.0 st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23 root 20 0 0 0 0 R 89.6 0.0 26:04.70 ksoftirqd/2
17 root 20 0 0 0 0 R 68.5 0.0 24:27.80 ksoftirqd/1

I suspect there may be some unexpected traffic hitting the CPU, so check the output of "show cpu couter queue | nz"

yo412.mlagA.10:22:38(config)#clear counters
!!! even a single command - clear counter, takes almost 10 sec to complete !!!

yo412.mlagA.10:22:47(config)#show cpu counters queue | nz | more
Arad3/0:
CoPP Class Queue Pkts Octets DropPkts DropOctets
Aggregate
-----------------------------------------------------------------------------------------------------------------
CoppSystemL3LpmOverflow Et3/6/1 1753 473344 74945 21049856
CoppSystemL3LpmOverflow Et3/6/2 1112 307200 73605 20702976
CoppSystemL3LpmOverflow Et3/6/3 610 166912 86302 23954432
CoppSystemL3LpmOverflow Et3/6/4 1178 320256 77089 21414656

Looks like there is a lot of packets hitting the cpu, even the CoPP filters out most of them. But this is a full load chassis, the aggregated traffic is still too heavy to a x86 CPU.

Try to tcpdump the incoming packets from et3/6/1 and punted to cpu. Surprisingly not many...

mlagA.10:35:23(config)#bash tcpdump -nvvi et3_6_1
tcpdump: listening on et3_6_1, link-type EN10MB (Ethernet), capture size 262144 bytes
10:35:36.066689 00:1c:73:46:0d:b0 > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 124: LACPv1, length 110
10:35:39.530341 00:1c:73:3b:e0:22 > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 124: LACPv1, length 110
^C

2 packets captured

Try to mirror this port to cpu then tcpdump it. (This feature is only supported on 7500E/R or 7280R devices)

mlagA.10:37:46(config)#monitor session 1 source et3/6/1 rx
mlagA.10:39:12(config)#monitor session 1 destination cpu

mlagA.10:39:15(config)#bash tcpdump -nvi mirror0
tcpdump: listening on mirror0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:40:08.198512 1e:af:14:08:18:02 > 00:aa:aa:aa:bb:cc, ethertype 802.1Q (0x8100), length 252: vlan 1408, p 0, ethertype IPv4,
100.14.8.119.30485 > 220.200.16.1.24659: Flags [R.UW], seq 0:194, ack 0, win 61689, urg 0, length 194
10:40:08.199078 1e:af:14:09:18:01 > 00:aa:aa:aa:bb:cc, ethertype 802.1Q (0x8100), length 252: vlan 1409, p 0, ethertype IPv4,
100.14.9.118.30504 > 220.200.17.1.24648: Flags [PUEW], seq 0:194, win 62028, urg 0, length 194

Do we have the route? No....

mlagA.10:40:08(config)#sh ip route 220.200.17.1
VRF: default
....
Gateway of last resort is not set

Create a null route for this prefix, response is better and "show cpu couter queue | nz" is back to normal now, no L3LPMOverflow anymore.

mlagA.10:54:28(config)#ip route 220.200.0.0/16 null0

mlagA.10:54:59(config)#sh cpu counters queue | nz | more
Arad3/0:
CoPP Class Queue Pkts Octets DropPkts DropOctets
Aggregate
-----------------------------------------------------------------------------------------------------------------
CoppSystemIgmp Et3/1/2 160 10240 0 0
CoppSystemIgmp Et3/1/4 160 10240 0 0

But cpu still high. And the busiest process is changed to SandFap instead of ksoftirqd. Hmmm.... why?

mlagA.10:57:19(config)#sh proc top once | more

%Cpu(s): 30.2 us, 4.1 sy, 0.0 ni, 62.1 id, 0.1 wa, 0.5 hi, 3.1 si, 0.0 st

...

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

12874 root 20 0 1001m 357m 192m R 100.4 2.2 110:39.04 SandFap

13025 root 20 0 1001m 357m 192m S 69.6 2.2 110:54.53 SandFap

16765 root 20 0 1001m 359m 193m S 51.2 2.2 96:56.18 SandFap

6/28/2019

Arista BGP Tips - ECMP, RR, Active Prefix

If the following 6 attributes of paths are identical, they are considered as equal paths:

Weight
Local_Pref
AS_Path
Origin
MED
IGP cost to NH

BGP RR

ONLY RR knows who is RRC, RRC has no idea
Originator ID is assigned by originating router
RR changes nothing, but add Cluster ID

Active BGP Prefix

show ip bgp vrf all, some prefixes are valid but not active
the common reason is, this particular prefix is learned from other routing protocols
that's why we need the knob - "bgp advertise-inactive"

BGP Origin Attribute

3 possible BGP origin attributes - Incomplete, IGP and EGP. EGP is never used.

If redistributed, the origin is Incomplete
If network command, the origin is IGP

For example:

ip route 88.88.88.0/24 Loopback88 <<< static route

interface Loopback88

ip address 88.88.88.1/32

router bgp 4

redistribute static

address-family ipv4

network 88.88.88.1/32

R4#sh ip bgp 88.88.88.0/24

BGP routing table entry for 88.88.88.0/24

Paths: 1 available

Local

- from - (0.0.0.0)

Origin INCOMPLETE, metric -, localpref -, weight 0, valid, local, best, redistributed (Static)

R4#sh ip bgp 88.88.88.1/32

BGP routing table entry for 88.88.88.1/32

Paths: 1 available

Local

- from - (0.0.0.0)

Origin IGP, metric -, localpref -, weight 0, valid, local, best

Arista EOS Tunneling Mechanism (5) - Hw Tunnel

Starting from 4.21.1F, hardware GRE tunnel is supported on Jericho-based platforms. Below is a typical setup of hw tunnel and configuration

You can see, configuration-wise it is just like the software tunnel. How to tell which mode, check "show interface tunnel".

R1#sh int tu200
Tunnel200 is up, line protocol is up (connected)
...
Tunnel TTL 10, Hardware forwarding enabled

Compared with software tunnel, the biggest difference is the hardware programming.

1. TCAM:

R2(config)#sh platform fap tcam summary
Tcam Allocation (Jericho0)
Bank Used By Reserved By
---------- ---------------------------- -----------
0 dbVTT0 -
12 dbSystem -
..

Now configuring an up tunnel interface

R2(config)#int tu100
R2(config-if-Tu100)#tunnel source 100.2.2.2
R2(config-if-Tu100)#tunnel destination 100.3.3.3
R2(config-if-Tu100)#show ip int brief
Interface IP Address Status Protocol MTU
....
Tunnel100 unassigned up up 1476

Now you can see a new Tcam allocation named dbGreTunnel

R2#sh platform fap tcam summary
Tcam Allocation (Jericho0)
Bank Used By Reserved By
---------- ---------------------------- -----------
0 dbVTT0 -
2 dbGreTunnel -
12 dbSystem -

2. Tunnel interface

R1#show platform fap eedb ip-tunnel gre interface tunnel 200
....
| Bank/ | OutLIF | Next | VSI | Encap | TOS | TTL | Source | Destination | OamLIF | OutLIF | Drop |
| Offset | | OutLIF | LSB | Mode | | | IP | IP | Set | Profile | |
|-----------------------------------------------------------------------------------------------------------|
| 2/0 | 0x6000 | 0x4012 | 0 | 2 | 0 | 10 | 100.1.1.1 | 100.4.4.4 | No | 0 | No |

* Bank should match the bank# in "show plat fap tcam summary"

3. Ip route

Below show 220.4.4.0/24 is learned via Tu200

R1(config-if-Tu200)#sh ip route

VRF: default
I L2 220.4.4.0/24 [115/20] via 200.1.4.4, Tunnel200, Static Interface GRE tunnel index 200, dst 100.4.4.4, src 100.1.1.1, TTL 10

R1(config-if-Tu200)#show platform fap ip route | egrep 'VRF|ID|220.4'
|VRF| Destination | | | | | | ECMP| FEC | Tunnel
| ID| Subnet | Cmd | Destination | VID |Outlif | MAC / CPU Code |Index| Index|T Value
|0 |220.4.4.0/24 |ROUTE| FEC 32773 |0 | - | | - |49160 | -

4. FEC

R1(config-if-Tu200)#show platform fap fec all
.....
| | | | | | | |
| ECMP| FEC | | | | | | Tunnel
|Index| Index| Cmd | Destination | VID |Outlif | MAC / CPU Code |T Value
----------------------------------------------------------------------------------
.....
| - |49160 |ROUTE| FEC 32773 |0 | - | | -
| - |32773 |ROUTE| Et21 |12 | - | 44:4c:a8:c1:78:69 |T 100.4.4.4

Limitations:

In above TOI link, there is a list of hw tunnel limitations, for example:

Underlay endpoint (tunnel source/destination) must IPv4 address in default VRF.

Even there is tunnel source/destination ipv6 address option, but not supported on this platform

Overlay can be IPv4/IPv6 under default or non-default VRF
No GRE KA, if one side has incorrect GRE configuration like missing tunnel source, the other end of GRE interface is still up.

In output of "show ip bgp sum", does PfxRcd mean prefix or path?

R1 #show ip bgp summary
BGP summary information for VRF default
Router identifier 10.1.255.1, local AS number 65001
Neighbor Status Codes: m - Under maintenance
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
RRv4 10.1.255.104 4 65010 2678462 57500 0 0 216d20h Estab 928963 925526

Before addpath, for one particular neighbor, it only sends ONE best path for one particular prefix. So PfxRcd = PathRcd. But after BGP addpath, the behavior changes. Multiple paths can be sent for one prefix to speed up the convergence time. So the meaning of this number is changed to "pathRcd".

How to run RFC2544 thruput/latency testing with Ixia

Step 1 - Connect device in a snake way like below

Step 2 - Configure IxNetworks for RFC 2544

1. Click "Add QuickTest". After configuration, it should show up in the right side

2. Finish all options. Pretty straightforward, just need to configure like

Ports to be used (Definitely need 2-way)
Packet type (Eth/IPv4/IPv6)
Packet size (make sure interface MTU matched)
Traffic option, like how long each case last
Stats parameters, like calculate latency (Cut through)
100% line rate (if for thruput)

Step 3 - Run the test and generate the report, like

6/27/2019

Use MAC ACL to isolate the failure point

For L2 traffic, besides checking drops/discard counter, another way to isolate the failure point is to use the MAC ACL, like

mac access-list macCount
counters per-entry
10 permit 00:00:03:03:00:14 00:00:00:00:00:00 04:68:03:03:00:14 00:00:00:00:00:00 log
20 permit any any log
!
interface Ethernet3/1
switchport access vlan 3003
mac access-group macCount in

The above MAC acl - macCount is count the number of packets with source MAC - 0000.0303.0014 and dest MAC - 0468.0303.0014. And it is applied on Eth3/1 ingress direction (egress ACL is not supported)

Router#show mac access-lists
MAC Access List macCount
counters per-entry
10 permit 00:00:03:03:00:14 00:00:00:00:00:00 04:68:03:03:00:14 00:00:00:00:00:00 log [match 216114288 packets, 0:00:00 ago]
20 permit any any log

This is an Arista DCS-7280CR2A-60-F with 4.22.0F

6/19/2019

Connect Ixia Novas Qsfp28 100G via 100GBASE-CR4 cable

The arista device is Arista DCS-7280CR2-60-F loading 4.21.2F. The other side is IxNetworks's NOVUS-M100GE8Q28 card, and using Arista 100G AOC cable.

Et60/1 connected 1 full 100G 100GBASE-CR4

Have to disable the autoneg on Ixia side to bring up the link. As shown as below

1. First stop all traffic (if the option is gray, that's the reason) then Edit L1 Properties
2. Unclick "Use IEEE Media for 100BASE CR4"
3. Uncheck "Auto Negotiate"

6/17/2019

Arista EOS Tunneling Mechanism (4) - Sw Tunnel vs Data

In the above topology, I show you that the data traffic doesn't go thru the sw tunnel. I use 2 routers to simulate hosts with default gateway pointing to R11 and R44. And we can see

Ping between R11 and R44 works
But the ping from host1 to host2 doesn't work

R11 has the correct route and ping to R44's ip address is good.

R11#sh ip route 99.2.2.99
...
I L1 99.2.2.0/24 [115/20] via 10.100.100.44, Tunnel100

R11.cd642.leaf18#ping 99.2.2.2

PING 99.2.2.2 (99.2.2.2) 72(100) bytes of data.

80 bytes from 99.2.2.2: icmp_seq=1 ttl=64 time=0.264 ms

80 bytes from 99.2.2.2: icmp_seq=2 ttl=64 time=0.119 ms

Host1 also has right route but ping failed, so data traffic can't pass thru.

host1#sh ip route 99.2.2.99
....
S 99.0.0.0/8 [1/0] via 99.1.1.1, Ethernet51/1

host1#ping 99.2.2.99
--- 99.2.2.99 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 40ms

Now, we add NHG + Decap group (will cover the details later) on both ends.

!!!!! R11 !!!!!!
ip route 99.2.2.0/24 Nexthop-Group nhg-gre-99-net
!
nexthop-group nhg-gre-99-net type gre
size 1
ttl 64
tunnel-source 11.11.11.11
entry 0 tunnel-destination 44.44.44.44
!
ip decap-group decap-net-99
tunnel type gre
tunnel decap-ip 11.11.11.11

!!!!! R44 !!!!!!

ip route 99.1.1.0/24 Nexthop-Group nhg-gre-99-net

!

nexthop-group nhg-gre-99-net type gre

   size 1

   ttl 64

   tunnel-source 44.44.44.44

   entry 0 tunnel-destination 11.11.11.11

!

ip decap-group decap-net-99

   tunnel type gre

   tunnel decap-ip 44.44.44.44

Now the ping works well.

host1#ping 99.2.2.99

PING 99.2.2.99 (99.2.2.99) 72(100) bytes of data.

80 bytes from 99.2.2.99: icmp_seq=1 ttl=62 time=0.264 ms

.....

--- 99.2.2.99 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.102/0.140/0.264/0.062 ms, ipg/ewma 0.184/0.200 ms

So basically that's how it works.

On each router has static nexthop + decap group configuration to other routers
The application software works as a passive ISIS neighbor to establish a neighbor with one ISIS router over a GRE tunnel.
So it can fetch the whole LSA DB to get a whole view of the network.
By using CLI or eAPI, the software can program each router with a static route pointing to the nexthop group entry configured in step 1.

6/14/2019

Arista EOS Tunneling Mechanism (3) - Sw Tunnel + ISIS

Above is a simple setup to demonstrate the feature of ISIS over GRE tunnel. It is worthy to note that:

This tunnel is a software tunnel without hardware programming. It ONLY works for the local packets generated by CPU and is capped by CoPP policy. So no data/traffic traffic is thru the tunnel.
The only valid use case per TOI is to advertise the ISIS routes to a remote application, like below

The application can learn the whole view of a network topology
Based application or business intelligence, it can program the routers to instruct the traffic flows. So a practical SDN solution to me.

Limitation:

IPv6 underlay endpoint is not supported
Underlay VRF is not supported
That says the tunnel source/destination must be ipv4 address under default VRF.

Some extras:

Even in the above the TOI, it says it only supports ISIS, but in the lab, the BGP session is up and running w/o any issue. But one thing to be aware is to assign the tunnel interface TTL. Otherwise, the neighbor fails to come up

R11(config-router-bgp)#sh run sec router bgp

router bgp 11

neighbor 10.100.100.44 remote-as 44

neighbor 10.100.100.44 maximum-routes 12000

redistribute connected

R11(config-router-bgp)#sh ip bgp sum

BGP summary information for VRF default

Router identifier 11.11.11.11, local AS number 11

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

10.100.100.44 4 44 4002 4013 0 0 01:29:18 Estab 11 11 <<<< session is up

Overlay VRF is supported, so tunnel interface can belong to none-default VRF

R11(config)#sh run int tunnel 101

interface Tunnel101

description sw-tunnel-gre-vrf-v1

vrf forwarding v1

ip address 10.101.101.11/24

isis enable isis.over.GRE.v1

isis bfd

isis network point-to-point

tunnel mode gre

tunnel source 11.11.11.1

tunnel destination 44.44.44.1

R11(config)#sh int tunnel 101

Tunnel101 is up, line protocol is up (connected)

Hardware is Tunnel, address is 0b0b.0b01.0800

Description: sw-tunnel-gre-vrf-v1

Internet address is 10.101.101.11/24

Broadcast address is 255.255.255.255

Tunnel source 11.11.11.1, destination 44.44.44.1

....

R44(config)#sh run int tu101

interface Tunnel101

description sw-tunnel-gre-vrf-v1

vrf forwarding v1

ip address 10.101.101.44/24

isis enable isis.over.GRE.v1

isis bfd

isis network point-to-point

tunnel mode gre

tunnel source 44.44.44.1

tunnel destination 11.11.11.1

R44(config)#sh int tu101

Tunnel101 is up, line protocol is up (connected)

Hardware is Tunnel, address is 2c2c.2c01.0800

Description: sw-tunnel-gre-vrf-v1

Internet address is 10.101.101.44/24

Broadcast address is 255.255.255.255

....

R11(config)#ping vrf v1 10.101.101.44

PING 10.101.101.44 (10.101.101.44) 72(100) bytes of data.

80 bytes from 10.101.101.44: icmp_seq=1 ttl=64 time=0.234 ms

80 bytes from 10.101.101.44: icmp_seq=2 ttl=64 time=0.153 ms

...

R11(config)#sh isis neighbors vrf v1

Instance VRF System Id Type Interface SNPA State Hold time Circuit Id

isis.over v1 R44 L1 Tunnel101 P2P UP 25 87

R11(config)#sh ip route vrf v1 isis

VRF: v1

....

I L1 101.101.44.1/32 [115/11] via 10.101.101.44, Tunnel101

I L1 101.101.44.2/32 [115/11] via 10.101.101.44, Tunnel101

I L1 101.101.44.3/32 [115/11] via 10.101.101.44, Tunnel101

I L1 101.101.44.4/32 [115/11] via 10.101.101.44, Tunnel101