Arista EOS/Networking Tips by Solomon Yang: Troubleshooting

Showing posts with label Troubleshooting. Show all posts

8/12/2021

EOS device boots into a weird state - [PyServer ar.Aaa not responding, still trying -- is it running?]

One possible reason is that, there is an unknown registered toggle switch.....

Arista Networks EOS 4.22.4M
localhost login: admin
[PyServer ar.Aaa not responding, still trying -- is it running?]
[PyServer ar.Aaa not responding, still trying -- is it running?]

...<snip> reboot....

Welcome to Arista Networks EOS 4.22.4M

Failed(tg != toggles.end()) Toggle #BfdPeerDeferredDeletion not registered

.....

...reboot....

Press Control-C now to enter Aboot shell

...press ctrl-c here to enter aboot mode...

^CWelcome to Aboot.

Aboot# cd /mnt/flash

Aboot# mv toggle_override toggle_override.old

! get rid of /mnt/flash/toggle_override file

Aboot# reboot

....

.... now it is good....

5/27/2021

Troubleshooting Arista EOS EVPN VPWS (1)

This blog provides troubleshooting information for common problems with Arista EVPN MPLS VPWS. It covers the following error:

Status: CLI conflict
Status: Admin down
Status: Interface unavailable
Status: No remote
Status: Unprogrammed local connector

Most of the issues start with down state in the output of "show patch panel" like below:

RR1-PE5-wa465.11:42:56#sh patch panel
Patch Connector Status
---------------------------- ------------------------------------------------------- ------
pbVpws-PE5:e471--PE1:e1312 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312 Down
2: Ethernet47/1

And command - "show patch panel <name> details" shows the detailed error reason:

RR1-PE5-wa465.14:58:02#sh patch panel pbVpws-PE5:e471--PE1:e1312 detail

......

Patch: pbVpws-PE5:e471--PE1:e1312, Status: Down

Connector 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312

Status: Interface unavailable

Local MPLS label: 100004

EVPN VPWS type: port-based

Connector 2: Ethernet47/1

Status: Interface mode

The following error status means a local misconfiguration most likely.

Ethernet side:

Status: CLI conflict

Possible reason: duplicated patch panel entries for the same ethernet interface

Status: Admin down

Check the interface status, to make sure is up

Pseudowire side:

Status: Interface unavailable

Possible reason: make sure "no switchport" under interface

After the local configuration is corrected, check the output of "show bgp evpn route-type auto-dis next-hop 0.0.0.0". This is to check if the locally originated BGP EVPN prefixes are advertised out to BGP peers.

RR1-PE5-wa465.14:58:14#show bgp evpn route-type auto-discovery next-hop 0.0.0.0

....

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric LocPref Weight Path

* > RD: 100.250.250.5:100 auto-discovery 50471 0000:0000:0000:0000:0000

- - - 0 i

RR1-PE5-wa465.15:07:10#show bgp evpn route-type auto-discovery next-hop 0.0.0.0 detail

BGP routing table information for VRF default

Router identifier 100.250.250.5, local AS number 65000

BGP routing table entry for auto-discovery 50471 0000:0000:0000:0000:0000, Route Distinguisher: 100.250.250.5:100

Paths: 1 available

Local

- from - (0.0.0.0)

Origin IGP, metric -, localpref -, weight 0, valid, local, best

Extended Community: Route-Target-AS:65000:100 TunnelEncap:tunnelTypeMpls L2 Attributes: control word

MPLS label: 100004

The most important information as highlighted above is the ETID/PW ID. And compare it with the definition of PW PE5:e471--PE1:e1312

RR1-PE5-wa465.15:06:29#show bgp evpn instance vpws custAB

...

Pseudowire PE5:e471--PE1:e1312

Status: up

VPWS label: 100004

Local VPWS ID: 50471

Remote VPWS ID: 101312

After this point, you are pretty sure this end of PW is good.

But if you are seeing the error status as "No remote", the device may have an issue with remote PW.

Patch: pbVpws-PE5:e471--PE1:e1312, Status: Down

Connector 1: BGP VPWS custAB Pseudowire PE5:e471--PE1:e1312

Status: No remote

Local MPLS label: 100004

EVPN VPWS type: port-based

Connector 2: Ethernet47/1

Status: Unprogrammed local connector

Now let's check the router knows how to reach PW with VPWS ID 101312.

RR1-PE5-wa465.15:13:06#show bgp evpn route-type auto-discovery | grep 101312

RR1-PE5-wa465.15:13:10#

Clearly, the router has no EVPN AD prefix with ETID 101312, so it doesn't how to reach. The next step is to log in to the other end of PW to figure out why by following the above steps. For example, the remote PW should be from NH 100.250.250.11

PE3-lp232.22:00:03#sh bgp evpn route-type auto-discovery next-hop 100.250.250.11

...

Network Next Hop Metric LocPref Weight Path

* >Ec RD: 100.250.250.11:100 auto-discovery 101331 0000:0000:0000:0000:0000

100.250.250.11 - 100 0 i Or-ID: 100.250.250.11 C-LST: 100.250.250.6

From the above output, the PE1(100.250.250.11) sends out an AD prefix with ETID 101331, not 101311.

And sometimes, you may face traffic issues even all the PW and patch panels are UP!! That's because the BGP EVPN AD prefixes only contain the local ETID/PW id, no the expected remote one. There is no way to ensure consistency.

Reference:

https://eos.arista.com/eos-4-24-2f/evpn-mpls-virtual-private-wire-service-vpws/

9/19/2019

Ping bgp ipv6 link local address

With the feature of "BGP IPv6 link-local peering support", you can establish the bgp peering via ipv6 link-local address. To check the underlying reachability, we can use linux kernel command - ping6

For example,

interface Vlan4001
vrf forwarding vrf3_BGP_v6P_v6Ll
ipv6 address fe80::1/64 link-local
!
router bgp 65100
vrf vrf3_BGP_v6P_v6Ll
rd 65000:3
bgp default ipv4-unicast
bgp default ipv4-unicast transport ipv6
bgp default ipv6-unicast
neighbor fe80::100%Vl4001 remote-as 65500

bn303#bash sudo ip netns exec ns-vrf3_BGP_v6P_v6Ll ping6 -I vlan4001 fe80::100
PING fe80::100(fe80::100) from fe80::1%vlan4001 vlan4001: 56 data bytes
64 bytes from fe80::100%vlan4001: icmp_seq=1 ttl=64 time=0.483 ms
64 bytes from fe80::100%vlan4001: icmp_seq=2 ttl=64 time=0.238 ms
.....

Please note that this peering is under VRF, so you have to use the namespace command - "ip netns..."

7/18/2019

7508E CPU Util% is high (1)

Saw quite slow response on one of my lab router, a 7508E with latest EOS release.

mlagA.10:18:17(config)#show ver
Arista DCS-7508
Hardware version: 06.00
....
Software image version: 4.22.0.1F

CPU only has 64% idle cycles. Not low.

mlagA.10:18:13(config)#show proc top once | more
%Cpu(s): 29.6 us, 3.9 sy, 0.0 ni, 64.0 id, 0.1 wa, 0.5 hi, 2.0 si, 0.0 st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23 root 20 0 0 0 0 R 89.6 0.0 26:04.70 ksoftirqd/2
17 root 20 0 0 0 0 R 68.5 0.0 24:27.80 ksoftirqd/1

I suspect there may be some unexpected traffic hitting the CPU, so check the output of "show cpu couter queue | nz"

yo412.mlagA.10:22:38(config)#clear counters
!!! even a single command - clear counter, takes almost 10 sec to complete !!!

yo412.mlagA.10:22:47(config)#show cpu counters queue | nz | more
Arad3/0:
CoPP Class Queue Pkts Octets DropPkts DropOctets
Aggregate
-----------------------------------------------------------------------------------------------------------------
CoppSystemL3LpmOverflow Et3/6/1 1753 473344 74945 21049856
CoppSystemL3LpmOverflow Et3/6/2 1112 307200 73605 20702976
CoppSystemL3LpmOverflow Et3/6/3 610 166912 86302 23954432
CoppSystemL3LpmOverflow Et3/6/4 1178 320256 77089 21414656

Looks like there is a lot of packets hitting the cpu, even the CoPP filters out most of them. But this is a full load chassis, the aggregated traffic is still too heavy to a x86 CPU.

Try to tcpdump the incoming packets from et3/6/1 and punted to cpu. Surprisingly not many...

mlagA.10:35:23(config)#bash tcpdump -nvvi et3_6_1
tcpdump: listening on et3_6_1, link-type EN10MB (Ethernet), capture size 262144 bytes
10:35:36.066689 00:1c:73:46:0d:b0 > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 124: LACPv1, length 110
10:35:39.530341 00:1c:73:3b:e0:22 > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 124: LACPv1, length 110
^C

2 packets captured

Try to mirror this port to cpu then tcpdump it. (This feature is only supported on 7500E/R or 7280R devices)

mlagA.10:37:46(config)#monitor session 1 source et3/6/1 rx
mlagA.10:39:12(config)#monitor session 1 destination cpu

mlagA.10:39:15(config)#bash tcpdump -nvi mirror0
tcpdump: listening on mirror0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:40:08.198512 1e:af:14:08:18:02 > 00:aa:aa:aa:bb:cc, ethertype 802.1Q (0x8100), length 252: vlan 1408, p 0, ethertype IPv4,
100.14.8.119.30485 > 220.200.16.1.24659: Flags [R.UW], seq 0:194, ack 0, win 61689, urg 0, length 194
10:40:08.199078 1e:af:14:09:18:01 > 00:aa:aa:aa:bb:cc, ethertype 802.1Q (0x8100), length 252: vlan 1409, p 0, ethertype IPv4,
100.14.9.118.30504 > 220.200.17.1.24648: Flags [PUEW], seq 0:194, win 62028, urg 0, length 194

Do we have the route? No....

mlagA.10:40:08(config)#sh ip route 220.200.17.1
VRF: default
....
Gateway of last resort is not set

Create a null route for this prefix, response is better and "show cpu couter queue | nz" is back to normal now, no L3LPMOverflow anymore.

mlagA.10:54:28(config)#ip route 220.200.0.0/16 null0

mlagA.10:54:59(config)#sh cpu counters queue | nz | more
Arad3/0:
CoPP Class Queue Pkts Octets DropPkts DropOctets
Aggregate
-----------------------------------------------------------------------------------------------------------------
CoppSystemIgmp Et3/1/2 160 10240 0 0
CoppSystemIgmp Et3/1/4 160 10240 0 0

But cpu still high. And the busiest process is changed to SandFap instead of ksoftirqd. Hmmm.... why?

mlagA.10:57:19(config)#sh proc top once | more

%Cpu(s): 30.2 us, 4.1 sy, 0.0 ni, 62.1 id, 0.1 wa, 0.5 hi, 3.1 si, 0.0 st

...

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

12874 root 20 0 1001m 357m 192m R 100.4 2.2 110:39.04 SandFap

13025 root 20 0 1001m 357m 192m S 69.6 2.2 110:54.53 SandFap

16765 root 20 0 1001m 359m 193m S 51.2 2.2 96:56.18 SandFap

6/27/2019

Use MAC ACL to isolate the failure point

For L2 traffic, besides checking drops/discard counter, another way to isolate the failure point is to use the MAC ACL, like

mac access-list macCount
counters per-entry
10 permit 00:00:03:03:00:14 00:00:00:00:00:00 04:68:03:03:00:14 00:00:00:00:00:00 log
20 permit any any log
!
interface Ethernet3/1
switchport access vlan 3003
mac access-group macCount in

The above MAC acl - macCount is count the number of packets with source MAC - 0000.0303.0014 and dest MAC - 0468.0303.0014. And it is applied on Eth3/1 ingress direction (egress ACL is not supported)

Router#show mac access-lists
MAC Access List macCount
counters per-entry
10 permit 00:00:03:03:00:14 00:00:00:00:00:00 04:68:03:03:00:14 00:00:00:00:00:00 log [match 216114288 packets, 0:00:00 ago]
20 permit any any log

This is an Arista DCS-7280CR2A-60-F with 4.22.0F

6/08/2019

Trouble-shoot BGP peering issue over GRE tunnel

Starting from 4.21.1F, Arista EOS starts to support the hardware GRE tunnel interface and BGP session over the tunnel on Jericho platforms. Before the tunnel is implemented by nexthop-group and decap group.

Here is a very simple and straightforward setup of eBGP over GRE tunnel.

But BGP session fails to come up as shown below:

R1.gts425#sh ip bgp sum

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 1

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

10.100.100.4 4 4 4766 257 0 0 00:20:40 Connect 0 0

Tunnel interface is up and works fine.

R1.gts425#sh int tunnel 100

Tunnel100 is up, line protocol is up (connected)

Hardware is Tunnel, address is 0101.0101.0800

Description: tunnel-gre-sand-to-sand

Internet address is 10.100.100.1/24

Broadcast address is 255.255.255.255

Tunnel source 1.1.1.1, destination 4.4.4.4

Tunnel protocol/transport GRE/IP

Key disabled, sequencing disabled

Checksumming of packets disabled

Tunnel TTL 0, Hardware forwarding not supported

Tunnel TOS 0

Path MTU Discovery

Tunnel transport MTU 1476 bytes

Up 22 minutes, 2 seconds

Ping with MTU size works totally fine

R1.gts425#ping 10.100.100.4 size 1476

...

80 bytes from 10.100.100.4: icmp_seq=5 ttl=64 time=0.118 ms

--- 10.100.100.4 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.118/0.155/0.295/0.070 ms, ipg/ewma 0.223/0.222 ms

Now let's run the tcpdump on R1 to see if hello packet out

R1.gts425(config-router-bgp)#bash tcpdump -nvvi et21 host 1.1.1.1

tcpdump: listening on et21, link-type EN10MB (Ethernet), capture size 262144 bytes

11:19:19.774224 28:99:3a:8f:91:bf > 44:4c:a8:c1:78:69, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 1, id 39325, offset 0, flags [DF], proto GRE (47), length 84)

1.1.1.1 > 4.4.4.4: GREv0, Flags [none], proto IPv4 (0x0800), length 64

(tos 0xc0, ttl 1, id 6538, offset 0, flags [DF], proto TCP (6), length 60)

10.100.100.1.46931 > 10.100.100.4.bgp: Flags [S], seq 3552659475, win 28720, options [mss 1436,sackOK,TS val 18764185 ecr 0,nop,wscale 7], length 0

11:19:19.774420 44:4c:a8:c1:78:69 > 28:99:3a:8f:91:bf, ethertype IPv4 (0x0800), length 126: (tos 0xc0, ttl 64, id 29177, offset 0, flags [none], proto ICMP (1), length 112)

10.1.2.2 > 1.1.1.1: ICMP time exceeded in-transit, length 92

(tos 0x0, ttl 1, id 39325, offset 0, flags [DF], proto GRE (47), length 84)

1.1.1.1 > 4.4.4.4: GREv0, Flags [none], proto IPv4 (0x0800), length 64

(tos 0xc0, ttl 1, id 6538, offset 0, flags [DF], proto TCP (6), length 60)

10.100.100.1.46931 > 10.100.100.4.bgp: Flags [S], seq 3552659475, win 28720, options [mss 1436,sackOK,TS val 18764185 ecr 0,nop,wscale 7], length 0

Now we can see the reason clearly. The eBGP TCP session is default with ttl 1 and copied to outer GRE packets, so the packets get TTL expired at 10.1.2.2 which is R2.

To fix this issue, just need to set TTL under tunnel interface

R1.gts425(config-router-bgp)#int tu 100

R1.gts425(config-if-Tu100)#tunnel ttl 10

R1.gts425(config-router-bgp)#sh ip bgp sum

BGP summary information for VRF default

Router identifier 1.1.1.1, local AS number 1

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

10.100.100.4 4 4 4 4 0 0 00:00:01 Estab 0 0

4/16/2019

How to debug ISIS/BFD neighbor issue

Topology:

[R1/DCS-7512N]:et11/36/1--{180.13.1.0/24}--et11/1:[R2/DCS-7050QX]

* both routers run 4.20.11M
* Between R1 and R2, ISIS/BFD is enabled shown as below

Router Configuration:

R1 configuration:

router isis isis.def
net 49.0001.0000.0000.0001.00
is-type level-1
log-adjacency-changes
redistribute connected
spf-interval 1 50 100
timers lsp generation 1 10 1000
!
address-family ipv4 unicast
maximum-paths 32
!
address-family ipv6 unicast
maximum-paths 32
!
interface Ethernet11/36/1
description UpEt_A_3.1
speed forced 10000full
no switchport
ip address 180.13.1.1/24
ipv6 address 180:13:1::1/64
isis enable isis.def
isis bfd
isis circuit-type level-1
isis network point-to-point
isis authentication mode md5 level-1
isis authentication key 7 QI3r+/7nP3Q= level-1

R2 configuration:
router isis isis.def
net 49.0001.0000.0000.0103.00
is-type level-1
log-adjacency-changes
spf-interval 1 50 100
timers lsp generation 1 10 1000
!
address-family ipv4 unicast
maximum-paths 32
!
address-family ipv6 unicast
maximum-paths 32
!
interface Ethernet11/1
description UpEt_A_3.1
speed forced 10000full
no switchport
ip address 180.13.1.100/24
ipv6 address 180:13:1::100/64
isis enable isis.def
isis bfd
isis circuit-type level-1
isis network point-to-point
isis authentication mode md5 level-1
isis authentication key 7 QI3r+/7nP3Q= level-1

Symptom:

R1: No ISIS and BFD neighbor

R1.13:53:14(config)#show isis nei | grep uspine3
R1.13:53:16(config)#show bfd nei | grep 180.13
R1.13:53:18(config)#

R2: ISIS neighbor stuck in INIT, and BFD is down

R2#sh isis nei
Instance VRF System Id Type Interface SNPA State Hold time Circuit Id
isis.def default R1 L1 Ethernet11/1 P2P INIT 26 2D

R2#sh bfd nei
VRF name: default
-----------------
DstAddr MyDisc YourDisc Interface Type LastUp LastDown LastDiag State
-------------------- --------------- --------------- --------------------- ------------ -------------------- -------------------- ----------------------- -----
180.13.1.1 2628313401 0 Ethernet11/1(47) normal 04/15/19 13:25 04/15/19 13:26 Nbr Signaled Down Down

Step1: R1/R2 - "show lldp nei" good and ping works, so no connectivity issue but high latency

-- show lldp nei --
R2#sh lldp nei
Port Neighbor Device ID Neighbor Port ID TTL
Et11/1 R2 Ethernet11/36/1 120

-- ping -- R2#ping 180.13.1.1
PING 180.13.1.1 (180.13.1.1) 72(100) bytes of data.
80 bytes from 180.13.1.1: icmp_seq=1 ttl=64 time=390 ms
80 bytes from 180.13.1.1: icmp_seq=2 ttl=64 time=437 ms
....

-- show interface <intf> | grep error --
R1.13:46:01(config)#sh int et11/36/1 | grep error
0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards

0 output errors, 0 collisions

R2.13:45:34(config)#sh int et11/1 | grep error
0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards

0 output errors, 0 collisions

Step2: R2's Down BFD shows NO incoming bfd pkts

-- show bfd neighbor dest-ip <ip> detail --
R2#sh bfd neighbors dest-ip 180.13.1.1 detail
VRF name: default
-----------------
Peer Addr 180.13.1.1, Intf Ethernet11/1, Type normal, State Down
VRF default, LAddr 180.13.2.100, LD/RD 2628313401/0
Session state is Down and not using echo function
Last Up Apr 15 13:25:27 2019
Last Down Apr 15 13:26:23 2019
Last Diag: Nbr Signaled Down
TxInt: 1000, RxInt: 1000, Multiplier: 3
Received RxInt: 300, Received Multiplier: 3
Rx Count: 0, Rx Interval (ms) min/max/avg: 0/0/0 last: never
Tx Count: 87681, Tx Interval (ms) min/max/avg: 508/996/870 last: 627 ms ago
Detect Time: 3000
Sched Delay: 1*TxInt: 87898, 2*TxInt: 0, 3*TxInt: 0, GT 3*TxInt: 0

R2 sends out BFD but no receiving

-- bash tcpdump -ni <intf> udp --
[admin@R2 ~]$ tcpdump -ni et11_1 udp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:48:22.309536 28:99:3a:3b:15:39 > 44:4c:a8:a5:11:40, ethertype IPv4 (0x0800), length 66: 180.13.1.100.57657 > 180.13.1.1.bfd-control: BFDv1, Control, State Down, Flags: [none], length: 24

11:48:23.301574 28:99:3a:3b:15:39 > 44:4c:a8:a5:11:40, ethertype IPv4 (0x0800), length 66: 180.13.1.100.57657 > 180.13.1.1.bfd-control: BFDv1, Control, State Down, Flags: [none], length: 24

Same observation on R1, only receiving no sending

R1.13:53:18(config)#bash tcpdump -ni et11_36_2 udp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_36_2, link-type EN10MB (Ethernet), capture size 262144 bytes
13:55:10.568533 28:99:3a:3b:15:39 > 44:4c:a8:a5:11:40, ethertype IPv4 (0x0800), length 66: 180.13.2.100.57657 > 180.13.2.1.bfd-control: BFDv1, Control, State Down, Flags: [none], length: 24

13:55:11.445074 28:99:3a:3b:15:39 > 44:4c:a8:a5:11:40, ethertype IPv4 (0x0800), length 66: 180.13.2.100.57657 > 180.13.2.1.bfd-control: BFDv1, Control, State Down, Flags: [none], length: 24

ISIS should notify the BFD to establish session after ISIS neighbor up. So the symptom is narrowed down to ISIS neighbor issue.

Step 3: Tcpdump ISIS IIH (isis hello) on both sides

-- show ver | grep MAC to get system MAC --

R1#show ver | grep MAC
System MAC address: 444c.a8a5.1140

R2#show ver | grep MAC
System MAC address: 2899.3a3b.1539

R2: IIH 2-way and looks good, R1/R2 send IIH every 10 seconds

-- bash tcpdump -ni <intf> iih --
[admin@R2 ~]$ tcpdump -ni et11_1 iih
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:18:16.104275 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497
11:18:18.000062 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497
11:18:24.723990 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497
11:18:27.597320 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497

R1: IIH, only 1 way, from R1 to R2, no R2 to R1

[admin@bn303 ~]$ tcpdump -ni et11_36_2 iih
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_36_2, link-type EN10MB (Ethernet), capture size 262144 bytes
11:23:53.268387 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497

11:24:02.823894 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497

IIH packets are destinated to well-known LAN broadcast address (0900:2b00:0005). These packets can't reach from R2 to R1.

Step 4: Arping

Arping shows similar, R1 to R2 is good, but seeing broadcast packets (all ff) drop on the direction of R2 to R1 even with arp ping

* Arp ping is to send our brocast L2 arp request to probe hosts

[admin@R1 ~]$ arping -b -I et11_36_1 180.13.1.100
ARPING 180.13.1.100 from 180.13.1.1 et11_36_1
Unicast reply from 180.13.1.100 [28:99:3A:3B:15:39] 419.967ms
Unicast reply from 180.13.1.100 [28:99:3A:3B:15:39] 409.205ms
Unicast reply from 180.13.1.100 [28:99:3A:3B:15:39] 401.616ms
Unicast reply from 180.13.1.100 [28:99:3A:3B:15:39] 410.797ms
^CSent 4 probes (4 broadcast(s))
Received 4 response(s)

[admin@R2 ~]$ arping -b -I et11_1 180.13.1.1
ARPING 180.13.1.1 from 180.13.1.100 et11_1
^CSent 9 probes (9 broadcast(s))

Received 0 response(s)

[admin@R2 ~]$ tcpdump -ni et11_1 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:46:04.859574 28:99:3a:3b:15:39 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 180.13.1.1 (Broadcast) tell 180.13.1.100, length 28

11:46:05.859645 28:99:3a:3b:15:39 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 180.13.1.1 (Broadcast) tell 180.13.1.100, length 28

At this point, the issue is narrowed down to, broadcast packets from R2 to R1 are dropped unexpectedly, which causes R1 not seeing R2's IIH packets, then no ISIS/BFD neighbor.

Step 5: hardware issue, but who drops the packets?

Ok, now we can say, most likely the problem is hardware, but who? The best way to nail down the culprit is to mirror packets to cpu, to see if the broadcast packets leave the R2.

* The feature of mirroring interface packets to CPU is only supported on Sand (Jericho/Arad) platform.

R1.14:32:14(config)#monitor session 1 source ethernet 11/36/1 rx
R1.14:32:19(config)#monitor session 1 destination cpu

Seeing IIH from R2 to R1 on the mirror cpu. << This is the packets received on the wire of et11/36/1

R1.14:32:20(config)#bash tcpdump -ni mirror0 iih
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on mirror0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:33:04.105249 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497
14:33:13.257565 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497
14:33:21.554871 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497

But not on the physical ethernet port << This is the packets received by the kernel via et11/36/1

R1.14:33:22(config)#bash tcpdump -ni et11_36_1 iih
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_36_1, link-type EN10MB (Ethernet), capture size 262144 bytes
14:33:35.056748 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497

14:33:42.990686 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497

Now we know the broadcast packets are dropped on R1 between eth and kernel.

Step 6: Counters

And this is approved by the counter output on both R1 and R2

R2/Trident - No loss

R2#clear counters
R2#clear platform trident counters
R2#sh cpu counters queue | nz

--------------------------------------------------------------------------------
Linecard0/0
--------------------------------------------------------------------------------
Queue Counter/pkts* Drops/pkts
--------------- ------------------- -------------------
OSPF/ISIS 2 0
BFD 10 0
R2#sh platform trident counters interface e11/1 | egrep -i 'card|drop' | nz
R2#

R1/Jericho - Loss seen

R1#clear hardware counter drop
R1#clear counters

R1#sh cpu counters queue | grep 'Et11/36/1' | nz

CoppSystemLldp Et11/36/1 4 924 0 0

R1#sh hardware counter drop | grep o11/1
A Jericho11/1 DeqDeletePktCnt : 39148 : 2019-04-16 11:59:39 : 2019-04-16 11:59:39
P Jericho11/1 dropVoqInNullRoute : 3800339040 : 2019-04-16 11:59:39 : 2019-04-16 11:59:39

R2#sh hardware counter drop | grep o11/1
A Jericho11/1 DeqDeletePktCnt : 39153 : 2019-04-16 11:59:39 : 2019-04-16 11:59:45
P Jericho11/1 dropVoqInNullRoute : 3800605701 : 2019-04-16 11:59:39 : 2019-04-16 11:59:45

So, should goes to DeqDeletePktCnt or dropVoqInNullRoute. What does they mean?

3/05/2019

Install Arista EOS Swix

Reference:

Arista's - "Package RPM into an Arista swix"
Arista Extension ylb.jp

In the Linux world, RPM package is used for software installation. Simply put, the swix (software image extension) is a software package which contains multiple RPM files and a manifest file, which is used to install software or software patch on Arista EOS.

Generally, a swix is installed in the following steps:

1. copy to local extension directory - "scp, copy"

2. install - "extension"

3. copy them to boot-extension - "copy"

4. if a modular system, install on the standby supervisor - "session"

5. restart agents or reset hardware components if required - "agent terminate"

Additionally,

1. uninstall - "no extension"

2. remove packet - "delete"

3. If both sup running mismatched swix, login console and reset boot-extension file

Below is the step by step:

Step 1 - copy

[admin@eosrouter flash]$ scp solomonyang@server:/users/solomonyang/swix/*swix /mnt/flash

patch.swix 100% 11MB 11.4MB/s 00:00

eosrouter#copy flash:patch.swix extension:

eosrouter#show extensions

Name Version/Release Status Extension

----------------------------- -------------------- ----------- ---------

patch.swix 1.0.4 . A, NI 6

A: available | NA: not available | I: installed | NI: not installed | F: forced

eosrouter#show extensions detail

Name: patch.swix

Version: 1.0.4

Release: blahblah

Presence: available

Status: not installed
Summary: SandFap agent related libs

Packages:

Total size: 0 bytes

Description: SandFap agent related libs.

Step 2 - install

eosrouter#extension patch.swix

eosrouter#show extensions

Name Version/Release Status Extension

----------------------------- -------------------- ----------- ---------

patch.swix 1.0.4 A, I 6

Step 3 - copy to boot extension

eosrouter#show installed-extensions

patch.swix

eosrouter#show boot-extensions

eosrouter#copy installed-extensions boot-extensions

Copy completed successfully.

eosrouter#show boot-extensions

patch.swix

Step 4 - install them on standby sup

eosrouter#copy flash:patch.swix supervisor-peer:/mnt/flash

Copy completed successfully.

eosrouter#session peer-supervisor

Warning: Permanently added '[127.1.0.2]:3601' (RSA) to the list of known hosts.

WARNING - you are currently logged in to the standby supervisor.

Not all cli commands are available or supported. Configuration mode

is disabled for SSO.

s1:eosrouter#copy flash:patch.swix extension:

Copy completed successfully.

s1:eosrouter#extension patch.swix

s1:eosrouter#copy installed-extensions boot-extensions

Copy completed successfully.

Step 5 - restart agents or reset hardware component

For some software patches, it is required to restart software agents or hardware components to make patch effective.

eosrouter#show agent names | grep SandFap

SandFap-Linecard3

SandFap-Linecard4

SandFap-Linecard5

eosrouter#agent SandFap-Linecard3 SandFap-Linecard4 SandFap-Linecard5 terminate

SandFap-Linecard3 was terminated

SandFap-Linecard4 was terminated

SandFap-Linecard5 was terminated

Also you need to do the same on the standby sup

How to fix unmatched swix'ed sup

If you run into unmatched swix supervisor due to mis-configuration, the way to fix them is as follows:

1. Login in the console of both supervisor. Sometimes the standby sup is at bad shape, then you have to fix it in Aboot mode

Arista Networks EOS 4.20.5.2F

localhost login: admin

[PyServer ar.Aaa not responding, still trying -- is it running?]

2. Reboot both and press ctrl-C to enter Aboot mode

3. Vi /mnt/flash/boot-extensions to remove swix entries. Then reboot so you will have a clean EOS w/o any extensions.

2/11/2019

Arista MPLS VPN Step by Step (2c) L2VPN - Tcpdump

One powerful tool in Arista/EOS is tcpdump. In this setup, for example, if you see some traffic issue and want to see if the PE sending out the packets, you can do the tcpdump this way:

If you want to check the type 5 pw between CE1 and CE2,

Step 1: find out the mpls ldp tunnel

PE1(config)#sh patch panel forwarding
In/Out Type CW VLAN Status Patch
-------------------------- ---- -- ---- ------ --------------------
....
Et47/3 5 Up ce1-ce2-pw5-1
Label 132770, LDP Tun 5 <<< tunnel 5

Step 2: check mpls ldp tunnel 5 fwding info so have the interface name

PE1(config)#sh mpls ldp tunnel 5
Index Endpoint Nexthop Interface Labels
----------- --------------------- --------------- ----------------- ----------
5 88.255.22.22/32 88.1.11.1 Ethernet3/1 [ 100005 ]

Step 3: create the monitor session to dump packet to cpu

monitor session ldp-et3-1 source Ethernet3/1

monitor session ldp-et3-1 destination Cpu

Step 4: tcpdump the packet

bash tcpdump -nevvi mirror0 ether proto 0x8847 -w /mnt/flash/ldp.pcp