Arista EOS/Networking Tips by Solomon Yang

3/07/2020

Arista EOS MTU Behavior (2) - L2 Switched Ports

Below is a quick lab using 7280CR which is Jericho-based system,

Use EOS tool - Ethxmit on the generator to send packets

--ip-dst pointing to the destination address on the receiver device
--size specifies the packet size. Please the real ethernet packet size is this number minus 4 (I believe it is ethernet trailer). So 1504 means 1500-byte packet is received on sender ingress port

Run tcpdump + mirror on R3/receiver to check MTU behavior

Conclusion:

For switched traffic, no MTU enforcement, ingress nor egress

Some other observations:

For using ethxmit:

--size parameter for ethxmit includes the 4-byte CRC/ethernet trailer
If egress is access port, the ethernet frame size = L3 MTU + 18-byte including 14-byte ethernet header and 4-byte CRC.
If egress port is trunk, ethernet frame size = L3 MTU + 22-byte including 14-byte ethernet, 4-byte vlan and 4-byte CRC

No matter the mtu size under L2 interfaces, the forwarding MTU is set to maximum value 10178 (even show interface says Ethernet MTU, but it is not actual ethernet frame size, but forwarding MTU before ethernet header). Look at the ip header total length value in tcpdump output.

Trident-based system would use 9214.

R2#sh int e47/1 | grep -i mtu
Ethernet MTU 10178 bytes , BW 10000000 kbit
R2#sh run int e47/1
interface Ethernet47/1
mtu 100
switchport access vlan 100

3/03/2020

Arista EOS MTU Behavior (1)

Summary:

In this series of blogs, I will cover the MTU (Maximum Transmission Unit) behavior on Arista Jericho-based routers. Here is the summary:

1. MTU enforcement happens on the egress interfaces.

So MTU is not MRU (maximum receive unit), large size packets are NOT dropped at ingress interfaces

2. Arista MTU is layer-3 MTU on Jericho chipset:

2.1 The MTU enforcement is only for routed packets, no action on switched traffic
2.2 The actual packet size is MTU size plus L2 header size, which is 14-byte for L3 routed interface or 18-byte for the trunk interface

3. IPv4 packets are fragmented by router CPU and capped by CoPP

Reference:

2/18/2020

MTU value in EOS interop with Cisco XR

Summary:

Arista mtu value under interface is IP MTU value, while Cisco XR's mtu is L2 MTU
So when ISIS peering fails to come up if both sides have same MTU value
XR's mtu = EOS's mtu + 14 (Ethernet header length)

https://eos.arista.com/forum/isis-between-arista-and-ios-xr/

1/29/2020

Tunnel RIB related features

Self-Defined Tunnel Rib TOI (4.22.1F)
Pseudowire Resolution Over User-defined Tunnel RIBs (4.22.1F)

May provide a sample later on how to use these 2 features.

1/17/2020

Arista BFD related TOIs

4.23*

HW BFD

https://eos.arista.com/eos-4-23-1f/hardware-accelerated-bfd-transmit/

4.22*

BFD C-bit

https://eos.arista.com/eos-4-22-1f/bfd-control-plane-independent-interoperability/

BFD Authentication

https://eos.arista.com/eos-4-22-0f/bfd-authentication/

4.20*

BFD SSO:

https://eos.arista.com/eos-4-18-2f/bfd-sso/
4.18.2F, starting with Arad Modular
4.20.5: Add Jericho

BFD for static routes

4.17*

BFD Enhancement:

https://eos.arista.com/eos-4-17-0f/bfd-enchancements/
BFD under VRF

OSPFv3 BFD:

https://eos.arista.com/eos-4-17-0f/ospfv3-bfd

4.15*

ISIS BFD

https://eos.arista.com/eos-4-15-2f/isis-bfd

BFD NSX Monitoring:

https://eos.arista.com/eos-4-15-0f/nsx-sn-health-monitoring/

12/28/2019

%SAND-6-VXLAN_ROUTE_IGNORED error message

If you see the syslog error message - %SAND-6-VXLAN_ROUTE_IGNORED like below (on arista 7280E/R series) , that's because your hardware TCAM profile is not correct.

Dec 28 16:02:16 eVtep2 SandL3Unicast: %SAND-6-VXLAN_ROUTE_IGNORED: VXLAN routes cannot be programmed in the data path with the current TCAM profile

Just change the configuration like:

hardware tcam profile vxlan-routing

11/21/2019

A dummy error - low MTU vs ipv6

After configuring IPv6 under an interface, see the ipv6 neighbor or host resolution doesn't work.

Router1.10:36:22(config-if-Po5)#sh ipv6 interface po5
Port-Channel5 is up, line protocol is up (connected)
IPv6 is stalled, link-local is unassigned
Global unicast address(es):
2000:170:1:170::1, subnet is 2000:170:1:170::/64 [INACTIVE]
No joined group addresses
ND DAD status is unavailable
ND Reachable time is 30000 milliseconds
ND retransmit interval is 1000 milliseconds

The reason is that, I have a line of a left-over configuration of smaller-than-normal mtu size - 1200. The lowest MTU size of IPv6 is 1280.

interface Port-Channel5
description Cdn_Test_upLink
mtu 1200
no switchport
vrf IxCdn
ip address 170.1.170.1/24
ipv6 nd cache expire 60
ipv6 enable
ipv6 address 2000:170:1:170::1/64
ipv6 nd ra disabled all
no mpls ip
no qos trust
spanning-tree bpdufilter enable

After removing this line, everything is fine.

Port-Channel5 is up, line protocol is up (connected)
IPv6 is enabled, link-local is fe80::464c:a8ff:fea5:1140/64
Global unicast address(es):
2000:170:1:170::1, subnet is 2000:170:1:170::/64
Joined group address(es):
ff02::1
ff02::1:ff00:1
ff02::1:ffa5:1140

11/05/2019

Arista EOS Segment Routing (3) - SR Routing #1, Static Route + NHG

The Arista MPLS Segment Routing White Paper(Page 8) lists 4 SR routing solutions:
1) Static label push via NHG
2) Controller via EOS SDK
3) BGP LU with SR
4) SR-TE

This blog gives an example of solution #1.

The configuration is straightforward:

Instead of running a routing protocol between PE1 and PE2, a static route of remote destination (100.255.4.1/32 on PE1) is configured and pointing to NHG.
The NHG defines:

MPLS encapsulation
Push a label of 900004
The NH is 10.1.2.2 for out-interface and destination MAC.
Obviously, 2 NHGs are needed to 2 way traffic.

Not like the regular NHG setup, decap group is NOT needed.
P1 and P2 have no idea of destination ip - 100.255.4.1, but use label to forward traffic

PE1#sh ip route 100.255.4.1

VRF: default

Codes: C - connected, S - static, K - kernel,

O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,

E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,

N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,

R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,

O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,

NG - Nexthop Group Static Route, V - VXLAN Control Service,

DH - DHCP client installed default route, M - Martian,

DP - Dynamic Policy Route, L - VRF Leaked

NG 100.255.4.1/32 [1/0] via sr-1-push-label-900004, Nexthop Group ID 1

The output of tcpdump on P2's et42/4 (the interface facing to P1) shows MPLS with the correct label in both directions.

23:10:17.713412 44:4c:a8:97:84:5f > 44:4c:a8:97:8c:51, ethertype MPLS unicast (0x8847), length 118: MPLS (label 900001, exp 0, [S], ttl 63)

(tos 0x0, ttl 64, id 6381, offset 0, flags [none], proto ICMP (1), length 100)

100.255.4.1 > 100.255.1.1: ICMP echo request, id 9258, seq 5, length 80

23:10:17.713473 44:4c:a8:97:8c:51 > 44:4c:a8:97:84:5f, ethertype MPLS unicast (0x8847), length 118: MPLS (label 900004, exp 0, [S], ttl 63)

(tos 0x0, ttl 64, id 2480, offset 0, flags [none], proto ICMP (1), length 100)

100.255.1.1 > 100.255.4.1: ICMP echo reply, id 9258, seq 5, length 80

Now you have the simplest, but completed ISIS segment-routing solution, control plane + data plane :-) In the real network, you can complete the puzzle by adding,

Have a controller participate in the ISIS domain via the ISIS over GRE tunnel.
As an ISIS neighbor, it can fetch the full ISIS LSDB to have a global view of the network including the segment.
Then it can program the edge router vi CLI/capi to steer the traffic. Or use the EOS SDK for faster program speed (this is the solution #2)

Arista EOS Segment Routing (2) - MPLS Ping to verify SR data plane

With the same topology, to verify the SR data plane, we can use the command - "ping mpls" to have PE1 send a UDP packet with the correct label.

PE1#ping mpls segment-routing ip 4.4.4.4/32 repeat 1
LSP ping to Segment-Routing route 4.4.4.4/32
timeout is 5000ms, interval is 1000ms
Via 10.1.2.2, Ethernet2/4, label stack (top label first): [900004]
Reply from 10.3.4.4: seq=1, time=0.53ms, success: egress ok

--- Segment-Routing target fec 4.4.4.4/32 : lspping statistics ---
Via 10.1.2.2, Ethernet2/4, label stack (top label first): [900004]
1 packets transmitted, 1 received, 0% packet loss, time 150ms
1 received from 10.3.4.4, rtt min/max/avg 0.530/0.530/0.530 ms

Turning on tcpdump on P1, it shows the following packets:

P1#sh run | grep moni
monitor session sr source Ethernet54/4
monitor session sr destination Cpu

P1#bash tcpdump -nvvvi mirror1 udp or mpls
tcpdump: listening on mirror1, link-type EN10MB (Ethernet), capture size 262144 bytes
13:48:15.934459 44:4c:a8:97:77:21 > 44:4c:a8:97:8c:51, ethertype MPLS unicast (0x8847), length 98: MPLS (label 900004, exp 0, [S], ttl 255)
(tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 80, options (RA))
10.1.2.1.36260 > 127.0.0.1.lsp-ping:
LSP-PINGv1, msg-type: MPLS Echo Request (1), length: 48
reply-mode: Reply via an IPv4/IPv6 UDP packet (2)
Return Code: No return code or return code contained in the Error Code TLV (0)
Return Subcode: (0)
Sender Handle: 0x00000000, Sequence: 1
Sender Timestamp: -16:-39:-45.54962696 Receiver Timestamp: no timestamp
Target FEC Stack TLV (1), length: 12
Unknown subTLV (31744), length: 6
0x0000: 0404 0404 2002
0x0000: 7c00 0006 0404 0404 2002 0000
13:48:15.934737 44:4c:a8:97:8c:51 > 44:4c:a8:97:77:21, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 36448, offset 0, flags [DF], proto UDP (17), length 60)
10.3.4.4.lsp-ping > 10.1.2.1.36260:
LSP-PINGv1, msg-type: MPLS Echo Reply (2), length: 32
reply-mode: Reply via an IPv4/IPv6 UDP packet (2)
Return Code: Replying router is an egress for the FEC at stack depth 1 (3)
Return Subcode: (1)
Sender Handle: 0x00000000, Sequence: 1
Sender Timestamp: -16:-39:-45.54962696 Receiver Timestamp: -16:-40:-1.401478080

From the above packet capture,

The icmp echo request is encap'ed in MPLS packet with label 90004 as expected.
The dest address of inner ip is 127.0.0.1. With destination address on network 127, the packet can't be routed if LSP is broken before the final destination and the packet is decap'ed prematurely.
The icmp echo reply is just a native ipv4 packet.

Arista EOS Segment Routing (1) - Simplest Setup/Configuration

Here is the simplest SR setup,
1) Only IPv4, only ipv4 node-segment;
2) No IPv6, no ISIS multi-topology

From the above configuration, we can observe:

The basic ISIS-SR configuration is quite simple:

enable "segment-routing mpls" under router isis to attach SR info to ISIS LSA;
Specify the loopback0 interface with a globally unique node-segment index.
ISIS-SR calculates the node-segment label = mpls label range base + index, so the routers in whole domain use the same label. This is an important characteristic of SR, for anycast.

To verify the SR setup, you can use the following CLI commands:

show mpls label ranges
show isis segment-routing
show isis database PE2.00-00 detail !PE2.00-00 is id
show mpls lfib route
show mpls segment-routing bindings
show platform jericho mpls route

Detailed output of PE1:

PE1#show mpls label ranges

Start End Size Usage

------------------------------------------------

0 15 16 reserved

16 99999 99984 static mpls

100000 116383 16384 ldp (dynamic)

116384 132767 16384 pseudowire (dynamic)

132768 149151 16384 bgp (dynamic)

149152 165535 16384 isis (dynamic)

165536 362143 196608 free (dynamic)

362144 899999 537856 unassigned

900000 965535 65536 isis-sr <<< default ISIS-SR range

900000 965535 65536 bgp-sr

965536 1031071 65536 srlb

1031072 1036287 5216 unassigned

1036288 1048575 12288 l2evpn

PE1#show isis segment-routing

System ID: PE1 Instance: sr

SR supported Data-plane: MPLS SR Router ID: 1.1.1.1

SR Global Block( SRGB ): Base: 900000 Size: 65536

Adj-SID allocation mode: SR-adjacencies

Adj-SID allocation pool: Base: 149152 Size: 16384

All Prefix Segments have : P:0 E:0 V:0 L:0

IS-IS Reachability Algorithm : SPF (0)

Number of IS-IS segment routing capable peers: 3

Self-Originated Segment Statistics:

Node-Segments : 1

Prefix-Segments : 0

Proxy-Node-Segments : 0

Adjacency Segments : 1

PE1#show isis database PE2.00-00 detail

IS-IS Instance: sr VRF: default

IS-IS Level 2 Link State Database

LSPID Seq Num Cksum Life IS Flags

PE2.00-00 12 57553 1049 L2 <>

NLPID: 0xCC(IPv4)

Hostname: PE2

Area address: 49.0001

Interface address: 4.4.4.4

Interface address: 10.3.4.4

IS Neighbor : P2.11 Metric: 10

LAN-Adj-sid: 100000 flags: [ L V ] weight: 0 system ID: 0000.0000.3333

Reachability : 4.4.4.4/32 Metric: 10 Type: 1 Up

SR Prefix-SID: 4 Flags: [ N ] Algorithm: 0

Reachability : 10.3.4.0/24 Metric: 10 Type: 1 Up

Router Capabilities: Router Id: 4.4.4.4 Flags: [ ]

SR Local Block:

SRLB Base: 965536 Range: 65536

Area leader priority: 250 algorithm: 0

SR Capability: Flags: [ I ]

SRGB Base: 900000 Range: 65536

PE1#show mpls lfib route

MPLS forwarding table (Label [metric] Vias) - 4 routes

MPLS next-hop resolution allow default route: False

Via Type Codes:

M - MPLS via, P - Pseudowire via,

I - IP lookup via, V - VLAN via,

VA - EVPN VLAN aware via, ES - EVPN ethernet segment via,

VF - EVPN VLAN flood via, AF - EVPN VLAN aware flood via,

NG - Nexthop group via

Source Codes:

G - gRIBI, S - Static MPLS route,

B2 - BGP L2 EVPN, B3 - BGP L3 VPN,

R - RSVP, LP - LDP pseudowire,

L - LDP, M - MLDP,

IP - IS-IS SR prefix segment, IA - IS-IS SR adjacency segment,

IL - IS-IS SR segment to LDP, LI - LDP to IS-IS SR segment,

BL - BGP LU, ST - SR TE policy,

DE - Debug LFIB

IA 149152 [1]

via M, 10.1.2.2, pop

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900002 [1], 2.2.2.2/32

via M, 10.1.2.2, pop

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900003 [1], 3.3.3.3/32

via M, 10.1.2.2, forward

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900004 [1], 4.4.4.4/32

via M, 10.1.2.2, forward

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

PE1#show mpls segment-routing bindings

1.1.1.1/32

Local binding: Label: imp-null

Remote binding: Peer ID: 0000.0000.2222, Label: 900001

2.2.2.2/32

Local binding: Label: 900002

Remote binding: Peer ID: 0000.0000.2222, Label: imp-null

3.3.3.3/32

Local binding: Label: 900003

Remote binding: Peer ID: 0000.0000.2222, Label: 900003

4.4.4.4/32

Local binding: Label: 900004

Remote binding: Peer ID: 0000.0000.2222, Label: 900004

PE1#show platform jericho mpls route

D - ECMP is divergent across switching chips

---------------------------------------------------------------------------------------------

| Mpls Table |

|---------------------------------------------------------------------------------------------|

|---------------------------------------------------------------------------------------------|

|149152 |Et2/4 |1027 | 44:4c:a8:97:8c:51 | PopE pipe auto|32773 |8200 |8 | 0 |

Egress Action Codes:

M - Mpls Tunnel, G - GRE Tunnel, MoG - Mpls-over-GRE Tunnel

ECMP Codes:

D - ECMP is divergent across switching chips

-----------------------------------------------------------------------------------------------

| Mpls Table

|-----------------------------------------------------------------------------------------------

|-----------------------------------------------------------------------------------------------

|900002 |Pop | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -

|900003 |Forward | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -

|900004 |Forward | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -

10/28/2019

EOS - Find the interfaces with the most flaps

bn302.12:40:24#show int phy | egrep '^Eth' | awk '{print $3, $1}' | sort -rn | head -10
479 Ethernet9/25/2
23 Ethernet9/11/1
5 Ethernet9/34/1
5 Ethernet3/34/1

BGP Oscillation (RFC 3345)

BGP Oscillation is common in the context of MED + RR, because,

BGP only advertise the best path and hide the full list.
Similar to RR, it is also with BGP confederation

RFC 3345 has a very good depiction of how it happens

And I converted it to a google slides for a better view

in RFC, there lists a couple of design options to avoid this kind of churn

In RFC 7964, advertises all available paths by using ADD-PATH
Always compare MED even from different AS;
Don't accept MED
Utilize other BGP attributes higher in the decision process. (a little risky, because if any prefix leaked w/o higher attribute and tied)
Assign a high IGP cost to inter-cluster-link

10/15/2019

AS_PATH and Community regexp tips

1. match certain # of as_path

ip as-path access-list SixAS permit ^._._._._._.$ any

2. community list for some specific ending number, like ending 123. Because there is a space at the end of comm list, so need an underscore _, like

ip community-list regexp Ending123 permit ^.*:.*123_$

10/08/2019

Router Server

References:

Router Server = Internet Exchange (IX) Route Server = eBGP RR

Used in internet exchange
eBGP RR, so no need a full mesh eBGP peering among all parties
Reduces configuration complexity and CPU/memory overhead on border routers

Requirements:

AS_PATH transparency:

RS doesn't append its own AS# in the AS_PATH
RS-client doesn't enforce first AS

Nexthop transparency:

doesn't change NRLI's NH

MED transparency:

doesn't change path MED

Path hiding:

Per RFC 4271, BGP only advertises the best path, and a later update will be considered as an implicit withdrawal of the existing path.
In RFC 7947, it does mention the "path hiding". So the RS only advertises the best path, which could slow down the convergence time

EOS configurations:

service routing protocols model multi-agent << must multi-agent

route-map rtmap-no-change-med permit 10

set metric +0 << keep metric unchanged

router bgp 10

router-id 10.255.255.251

neighbor RS-Client peer group

neighbor RS-Client next-hop-unchanged << NH unchanged

neighbor RS-Client as-path prepend-own disabled << no own as#

neighbor RS-Client route-map rtmap-not-change-med out << rtmap

neighbor RS-Client password 7 xLcnzAMGHkI=

neighbor RS-Client send-community

neighbor RS-Client maximum-routes 12000

neighbor 10.10.10.11 peer group RS-Client

neighbor 10.10.10.11 remote-as 11

How many prefixes received? Considering the following scenario:

3 peers advertising ONE prefix
2 route servers
And all 3 peers also have bilateral peering
Then should be 5

Router1.15:19:15#sh ip bgp 11.111.1.1

BGP routing table information for VRF default

Router identifier 180.255.255.1, local AS number 65100

BGP routing table entry for 11.111.1.1/32

Paths: 5 available

12 111

10.10.10.12 from 10.10.10.12 (10.255.255.12) << from R2

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP head, ECMP, best, ECMP contributor

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.11 (10.255.255.11) << from R1

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP, ECMP contributor

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.251 (10.255.255.251) << R1 vis RS1

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP, ECMP contributor

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.252 (10.255.255.252) << R1 via RS2

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external, ECMP, ECMP contributor

Rx SAFI: Unicast

13 333 111

10.10.10.13 from 10.10.10.13 (10.255.255.13) << R3

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:19 ago, valid, external

Rx SAFI: Unicast

Let's shut down the private peering to R2. So you can see the RS1 and RS2 don't advertise R2's path.

Router.15:26:31(config-router-bgp)#sh ip bgp 11.111.1.1

BGP routing table information for VRF default

Router identifier 180.255.255.1, local AS number 65100

BGP routing table entry for 11.111.1.1/32

Paths: 4 available

11 111

10.10.10.11 from 10.10.10.252 (10.255.255.252) << RS2

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external, best

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.251 (10.255.255.251) << RS1

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external

Rx SAFI: Unicast

11 111

10.10.10.11 from 10.10.10.11 (10.255.255.11) << R1

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external

Rx SAFI: Unicast

12 111

10.10.10.12 from 10.10.10.12 (10.255.255.12) << R2

Origin IGP, metric 100, localpref 100, weight 0, received 00:00:02 ago, valid, external

Rx SAFI: Unicast

10/03/2019

EOS: A quick alias to show the interfaces with most util%

bn302#bash Cli -p15 -c 'srnz' | awk '{print $5, $8, $1}' | sort -r | more
Mbps Kpps Port
1.6% 4.5% Et9/36/1
1.5% 4.3% Et3/36/1
0.7% 0.5% Et13/33/1
0.7% 0.4% Et12/36/2

or set it as an alias

srnzsort bash Cli -p15 -c 'srnz' | awk '{print $5, $8, $1}' | sort -r | head -n 6

Updated note: The above command works with interface names/description defined. If not, have to change awk '{print $5, $8, $1}' to awk '{print $4, $7, $1}'

hs486.11:32:15#srnz
Port Name Intvl In Mbps % In Kpps Out Mbps % Out Kpps
Et49/1 0:05 9870.4 100.0% 812 9860.3 99.9% 811

hs486.12:14:45#bash Cli -p15 -c 'srnz' | awk '{print $4, $7, $1}' | sort -r | more
In In Port
100.0% 99.9% Et50/1
100.0% 99.9% Et49/4

How about some interfaces with description, some don't? hm.... Let me think about it..... :-)

9/19/2019

Ping bgp ipv6 link local address

With the feature of "BGP IPv6 link-local peering support", you can establish the bgp peering via ipv6 link-local address. To check the underlying reachability, we can use linux kernel command - ping6

For example,

interface Vlan4001
vrf forwarding vrf3_BGP_v6P_v6Ll
ipv6 address fe80::1/64 link-local
!
router bgp 65100
vrf vrf3_BGP_v6P_v6Ll
rd 65000:3
bgp default ipv4-unicast
bgp default ipv4-unicast transport ipv6
bgp default ipv6-unicast
neighbor fe80::100%Vl4001 remote-as 65500

bn303#bash sudo ip netns exec ns-vrf3_BGP_v6P_v6Ll ping6 -I vlan4001 fe80::100
PING fe80::100(fe80::100) from fe80::1%vlan4001 vlan4001: 56 data bytes
64 bytes from fe80::100%vlan4001: icmp_seq=1 ttl=64 time=0.483 ms
64 bytes from fe80::100%vlan4001: icmp_seq=2 ttl=64 time=0.238 ms
.....

Please note that this peering is under VRF, so you have to use the namespace command - "ip netns..."

9/09/2019

Kill list of zombie process on Linux(Eos)

Due to a script bug, one EOS device has quite some zombie process like,

[admin@bn302 flash]$ ps aux | grep 'CliShell -A'
root 28804 0.0 0.0 10504 8580 ? S 10:53 0:00 CliShell -A -p 15 -c show port-channel | json
root 28809 0.0 0.0 10504 8688 ? S 10:53 0:00 CliShell -A -p 15 -c show port-channel | json
root 28826 0.0 0.0 10504 8612 ? S 10:54 0:00 CliShell -A -p 15 -c show port-channel | json
root 28835 0.0 0.0 10504 8580 ? S 10:54 0:00 CliShell -A -p 15 -c show port-channel | json
root 28844 0.0 0.0 10504 8616 ? S 10:54 0:00 CliShell -A -p 15 -c show port-channel | json
....

Instead of killing one by one manually, write a quick shell command to kill all. (killall is not supported in EOS/Linux)

[admin@bn302 flash]$ for pid in $(ps aux | grep 'CliShell -A' | awk '{print $2, $11}' | grep CliShell | awk '{print $1}'); do sudo kill -9 $pid; done

7/31/2019

ISIS multi-topology - interfaces with different IPv4/v6 AF enabled

As we know, the ISIS requires address family configuration must be matched between the ISIS router and interface configuration. So if you have both ipv4 and ipv6 AF configuration, you must do the same under interface.

Sometime you may have some of interfaces only enabled ipv4, but dual stack globally, in this case, you will need the ISIS multi-topology.

router isis VrfDefault
net 49.0001.0000.0000.0002.00
router-id ipv4 100.255.255.2
is-type level-1
!
address-family ipv4 unicast
!
address-family ipv6 unicast
multi-topology <<< need multi-topology enabled here
!
interface Port-Channel211
no switchport
ip address 100.201.1.0/31
ipv6 address 100:201:1::/127
isis enable VrfDefault <<< this intf has both v4 and v6
!
interface Port-Channel251
no switchport
ip address 100.205.1.0/31
isis enable VrfDefault <<< but this intf only v4
isis multi-topology address-family ipv4 unicast <<<

And both neighbors are up!!

Router1(config-if-Po251)#show isis nei
Instance VRF System Id Type Interface SNPA State Hold time Circuit Id
VrfDef default RouterDualStack L1 Port-Channel211 P2P UP 3 48
VrfDef default RouterV4 L1 Port-Channel251 P2P UP 2 63

7/21/2019

MACSec

Arista EOS starts to support MACSec - 802.1AE from 4.15.4F which was released around 2016. Later related features:

4.17.0F - MACSec EAP-FAST (key server)
4.21.1F - MACSec Proxy
Arista MACSec WP (a list of supported hw)
Cisco MACSec WP (ipsec vs macsec)

Some key take-aways:

Purpose: to protect data traffic from various types of attacks,

passive: snooping
active: reply, man in the middle

vs IPSec

Level: IPSec is at the IP level, vs MACSec MAC/data link level.
End or Hop: IPSec provides end-to-end protection, MACsec secures Per-hop. (TLS is similar like IPSec, end-to-end)
HW: IPSec doesn't rely on hw but with IPSec often, MACSec is on PHY level with special hw. For example,

7500E-6CFPX-LC
7280SRAM-48C6
7280CR2M-30
7500R2M-36CQ-LC

Throughput: IPSec has a ceiling, while MACsec is line-rate like 100G.

Key components:

CAK - Conn Ass Key, master key. Either manual or from key server
CAN - CAK's name
SAK - Secu Ass Key, derived from CAK, used to data encryption.

Point-to-Point vs clear 802.1Q

Clear 802.1Q means, move vlan tag ahead of MACSec Tag, a must for hub-spoke topology. Cisco ASR supports it.
Arista EOS only supports point-to-point.

Packet format:

Add 16-byte SecTAG after srcMAC and VLAN Tag
Add 16-byte ICV at end of ethernet frames
Ethertype = 0x88e5 (why not use 0x85ec?, hahaha)

MACSec proxy:

Provides MACSec for VxLAN traffic
Loopback traffic to a MACSec capable front panel port to process.

MISC:

fallback key in case MACSec failed or during a configuration change
L2-protocol pass thru, skipping LLDP packets

Configuration (EOS):

mac security

license DCI1 <license>

profile toISP1

! key <CAN> <type, 0-clear text> <key>

key a1 0 a123456

interface eth4/1/1

mac security profile toISP1

show mac secu counters

show mac secu interface

7/18/2019

Arista Modular Switch - simulate LC remove/insert

!! bash cat /persist/sys/Linecard3.cfg
platform module Linecard3 remove
platform module Linecard3 insert

7508E CPU Util% is high (1)

Saw quite slow response on one of my lab router, a 7508E with latest EOS release.

mlagA.10:18:17(config)#show ver
Arista DCS-7508
Hardware version: 06.00
....
Software image version: 4.22.0.1F

CPU only has 64% idle cycles. Not low.

mlagA.10:18:13(config)#show proc top once | more
%Cpu(s): 29.6 us, 3.9 sy, 0.0 ni, 64.0 id, 0.1 wa, 0.5 hi, 2.0 si, 0.0 st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23 root 20 0 0 0 0 R 89.6 0.0 26:04.70 ksoftirqd/2
17 root 20 0 0 0 0 R 68.5 0.0 24:27.80 ksoftirqd/1

I suspect there may be some unexpected traffic hitting the CPU, so check the output of "show cpu couter queue | nz"

yo412.mlagA.10:22:38(config)#clear counters
!!! even a single command - clear counter, takes almost 10 sec to complete !!!

yo412.mlagA.10:22:47(config)#show cpu counters queue | nz | more
Arad3/0:
CoPP Class Queue Pkts Octets DropPkts DropOctets
Aggregate
-----------------------------------------------------------------------------------------------------------------
CoppSystemL3LpmOverflow Et3/6/1 1753 473344 74945 21049856
CoppSystemL3LpmOverflow Et3/6/2 1112 307200 73605 20702976
CoppSystemL3LpmOverflow Et3/6/3 610 166912 86302 23954432
CoppSystemL3LpmOverflow Et3/6/4 1178 320256 77089 21414656

Looks like there is a lot of packets hitting the cpu, even the CoPP filters out most of them. But this is a full load chassis, the aggregated traffic is still too heavy to a x86 CPU.

Try to tcpdump the incoming packets from et3/6/1 and punted to cpu. Surprisingly not many...

mlagA.10:35:23(config)#bash tcpdump -nvvi et3_6_1
tcpdump: listening on et3_6_1, link-type EN10MB (Ethernet), capture size 262144 bytes
10:35:36.066689 00:1c:73:46:0d:b0 > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 124: LACPv1, length 110
10:35:39.530341 00:1c:73:3b:e0:22 > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 124: LACPv1, length 110
^C

2 packets captured

Try to mirror this port to cpu then tcpdump it. (This feature is only supported on 7500E/R or 7280R devices)

mlagA.10:37:46(config)#monitor session 1 source et3/6/1 rx
mlagA.10:39:12(config)#monitor session 1 destination cpu

mlagA.10:39:15(config)#bash tcpdump -nvi mirror0
tcpdump: listening on mirror0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:40:08.198512 1e:af:14:08:18:02 > 00:aa:aa:aa:bb:cc, ethertype 802.1Q (0x8100), length 252: vlan 1408, p 0, ethertype IPv4,
100.14.8.119.30485 > 220.200.16.1.24659: Flags [R.UW], seq 0:194, ack 0, win 61689, urg 0, length 194
10:40:08.199078 1e:af:14:09:18:01 > 00:aa:aa:aa:bb:cc, ethertype 802.1Q (0x8100), length 252: vlan 1409, p 0, ethertype IPv4,
100.14.9.118.30504 > 220.200.17.1.24648: Flags [PUEW], seq 0:194, win 62028, urg 0, length 194

Do we have the route? No....

mlagA.10:40:08(config)#sh ip route 220.200.17.1
VRF: default
....
Gateway of last resort is not set

Create a null route for this prefix, response is better and "show cpu couter queue | nz" is back to normal now, no L3LPMOverflow anymore.

mlagA.10:54:28(config)#ip route 220.200.0.0/16 null0

mlagA.10:54:59(config)#sh cpu counters queue | nz | more
Arad3/0:
CoPP Class Queue Pkts Octets DropPkts DropOctets
Aggregate
-----------------------------------------------------------------------------------------------------------------
CoppSystemIgmp Et3/1/2 160 10240 0 0
CoppSystemIgmp Et3/1/4 160 10240 0 0

But cpu still high. And the busiest process is changed to SandFap instead of ksoftirqd. Hmmm.... why?

mlagA.10:57:19(config)#sh proc top once | more

%Cpu(s): 30.2 us, 4.1 sy, 0.0 ni, 62.1 id, 0.1 wa, 0.5 hi, 3.1 si, 0.0 st

...

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

12874 root 20 0 1001m 357m 192m R 100.4 2.2 110:39.04 SandFap

13025 root 20 0 1001m 357m 192m S 69.6 2.2 110:54.53 SandFap

16765 root 20 0 1001m 359m 193m S 51.2 2.2 96:56.18 SandFap