Arista EOS/Networking Tips by Solomon Yang: ISIS

Showing posts with label ISIS. Show all posts

1/02/2021

Why unprotected ISIS segment routing prefix with TiLFA enabled

In the above topology, the output of "show isis segment-routing prefix" on PE11 shows the SID of PE22 is not protected.

bn303-PEa1.23:58:00(config-router-isis-af)#show isis segment-routing prefix-segments

System ID: 0000.0000.0011 Instance: 'isis-sr'

SR supported Data-plane: MPLS SR Router ID: 200.250.250.1

Node: 10 Proxy-Node: 0 Prefix: 0 Total Segments: 10

Flag Descriptions: R: Re-advertised, N: Node Segment, P: no-PHP

E: Explicit-NULL, V: Value, L: Local

Segment status codes: * - Self originated Prefix, L1 - level 1, L2 - level 2

Prefix SID Type Flags System ID Level Protection

------------------------- ----- ---------- ----------------------- --------------- ----- ----------

......

100.250.250.22/32 22 Node R:0 N:1 P:0 E:0 V:0 L:0 0000.0000.0022 L2 unprotected

100.255.255.31/32 31 Node R:0 N:1 P:0 E:0 V:0 L:0 0000.0000.0031 L2 node

The reason the SID of PE22 is "unprotected" is that the PE11 has a 2-way ECMP to the destination.

bn303-PEa1.23:58:01(config-router-isis-af)#show ip route 100.250.250.22/32

I L2 100.250.250.22/32 [115/50] via 100.2.11.0, Ethernet3/1/1

via 100.1.11.0, Ethernet10/36/1

Now let's the shutdown the interface between PE11 and P2 to break this ECMP. Now the prefix-segment is proteced now.

bn303-PEa1.00:14:21(config)#int e3/1/1

bn303-PEa1.00:14:27(config-if-Et3/1/1)#shu

bn303-PEa1.00:14:31#show ip route 100.250.250.22/32

I L2 100.250.250.22/32 [115/50] via 100.1.11.0, Ethernet10/36/1

bn303-PEa1.00:14:34#show isis segment-routing prefix-segments

Segment status codes: * - Self originated Prefix, L1 - level 1, L2 - level 2

Prefix SID Type Flags System ID Level Protection

------------------------- ----- ---------- ----------------------- --------------- ----- ----------

...

100.250.250.22/32 22 Node R:0 N:1 P:0 E:0 V:0 L:0 0000.0000.0022 L2 node

100.255.255.31/32 31 Node R:0 N:1 P:0 E:0 V:0 L:0 0000.0000.0031 L2 node

2/18/2020

MTU value in EOS interop with Cisco XR

Summary:

Arista mtu value under interface is IP MTU value, while Cisco XR's mtu is L2 MTU
So when ISIS peering fails to come up if both sides have same MTU value
XR's mtu = EOS's mtu + 14 (Ethernet header length)

https://eos.arista.com/forum/isis-between-arista-and-ios-xr/

11/05/2019

Arista EOS Segment Routing (3) - SR Routing #1, Static Route + NHG

The Arista MPLS Segment Routing White Paper(Page 8) lists 4 SR routing solutions:
1) Static label push via NHG
2) Controller via EOS SDK
3) BGP LU with SR
4) SR-TE

This blog gives an example of solution #1.

The configuration is straightforward:

Instead of running a routing protocol between PE1 and PE2, a static route of remote destination (100.255.4.1/32 on PE1) is configured and pointing to NHG.
The NHG defines:

MPLS encapsulation
Push a label of 900004
The NH is 10.1.2.2 for out-interface and destination MAC.
Obviously, 2 NHGs are needed to 2 way traffic.

Not like the regular NHG setup, decap group is NOT needed.
P1 and P2 have no idea of destination ip - 100.255.4.1, but use label to forward traffic

PE1#sh ip route 100.255.4.1

VRF: default

Codes: C - connected, S - static, K - kernel,

O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,

E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,

N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,

R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,

O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,

NG - Nexthop Group Static Route, V - VXLAN Control Service,

DH - DHCP client installed default route, M - Martian,

DP - Dynamic Policy Route, L - VRF Leaked

NG 100.255.4.1/32 [1/0] via sr-1-push-label-900004, Nexthop Group ID 1

The output of tcpdump on P2's et42/4 (the interface facing to P1) shows MPLS with the correct label in both directions.

23:10:17.713412 44:4c:a8:97:84:5f > 44:4c:a8:97:8c:51, ethertype MPLS unicast (0x8847), length 118: MPLS (label 900001, exp 0, [S], ttl 63)

(tos 0x0, ttl 64, id 6381, offset 0, flags [none], proto ICMP (1), length 100)

100.255.4.1 > 100.255.1.1: ICMP echo request, id 9258, seq 5, length 80

23:10:17.713473 44:4c:a8:97:8c:51 > 44:4c:a8:97:84:5f, ethertype MPLS unicast (0x8847), length 118: MPLS (label 900004, exp 0, [S], ttl 63)

(tos 0x0, ttl 64, id 2480, offset 0, flags [none], proto ICMP (1), length 100)

100.255.1.1 > 100.255.4.1: ICMP echo reply, id 9258, seq 5, length 80

Now you have the simplest, but completed ISIS segment-routing solution, control plane + data plane :-) In the real network, you can complete the puzzle by adding,

Have a controller participate in the ISIS domain via the ISIS over GRE tunnel.
As an ISIS neighbor, it can fetch the full ISIS LSDB to have a global view of the network including the segment.
Then it can program the edge router vi CLI/capi to steer the traffic. Or use the EOS SDK for faster program speed (this is the solution #2)

Arista EOS Segment Routing (2) - MPLS Ping to verify SR data plane

With the same topology, to verify the SR data plane, we can use the command - "ping mpls" to have PE1 send a UDP packet with the correct label.

PE1#ping mpls segment-routing ip 4.4.4.4/32 repeat 1
LSP ping to Segment-Routing route 4.4.4.4/32
timeout is 5000ms, interval is 1000ms
Via 10.1.2.2, Ethernet2/4, label stack (top label first): [900004]
Reply from 10.3.4.4: seq=1, time=0.53ms, success: egress ok

--- Segment-Routing target fec 4.4.4.4/32 : lspping statistics ---
Via 10.1.2.2, Ethernet2/4, label stack (top label first): [900004]
1 packets transmitted, 1 received, 0% packet loss, time 150ms
1 received from 10.3.4.4, rtt min/max/avg 0.530/0.530/0.530 ms

Turning on tcpdump on P1, it shows the following packets:

P1#sh run | grep moni
monitor session sr source Ethernet54/4
monitor session sr destination Cpu

P1#bash tcpdump -nvvvi mirror1 udp or mpls
tcpdump: listening on mirror1, link-type EN10MB (Ethernet), capture size 262144 bytes
13:48:15.934459 44:4c:a8:97:77:21 > 44:4c:a8:97:8c:51, ethertype MPLS unicast (0x8847), length 98: MPLS (label 900004, exp 0, [S], ttl 255)
(tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 80, options (RA))
10.1.2.1.36260 > 127.0.0.1.lsp-ping:
LSP-PINGv1, msg-type: MPLS Echo Request (1), length: 48
reply-mode: Reply via an IPv4/IPv6 UDP packet (2)
Return Code: No return code or return code contained in the Error Code TLV (0)
Return Subcode: (0)
Sender Handle: 0x00000000, Sequence: 1
Sender Timestamp: -16:-39:-45.54962696 Receiver Timestamp: no timestamp
Target FEC Stack TLV (1), length: 12
Unknown subTLV (31744), length: 6
0x0000: 0404 0404 2002
0x0000: 7c00 0006 0404 0404 2002 0000
13:48:15.934737 44:4c:a8:97:8c:51 > 44:4c:a8:97:77:21, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 36448, offset 0, flags [DF], proto UDP (17), length 60)
10.3.4.4.lsp-ping > 10.1.2.1.36260:
LSP-PINGv1, msg-type: MPLS Echo Reply (2), length: 32
reply-mode: Reply via an IPv4/IPv6 UDP packet (2)
Return Code: Replying router is an egress for the FEC at stack depth 1 (3)
Return Subcode: (1)
Sender Handle: 0x00000000, Sequence: 1
Sender Timestamp: -16:-39:-45.54962696 Receiver Timestamp: -16:-40:-1.401478080

From the above packet capture,

The icmp echo request is encap'ed in MPLS packet with label 90004 as expected.
The dest address of inner ip is 127.0.0.1. With destination address on network 127, the packet can't be routed if LSP is broken before the final destination and the packet is decap'ed prematurely.
The icmp echo reply is just a native ipv4 packet.

Arista EOS Segment Routing (1) - Simplest Setup/Configuration

Here is the simplest SR setup,
1) Only IPv4, only ipv4 node-segment;
2) No IPv6, no ISIS multi-topology

From the above configuration, we can observe:

The basic ISIS-SR configuration is quite simple:

enable "segment-routing mpls" under router isis to attach SR info to ISIS LSA;
Specify the loopback0 interface with a globally unique node-segment index.
ISIS-SR calculates the node-segment label = mpls label range base + index, so the routers in whole domain use the same label. This is an important characteristic of SR, for anycast.

To verify the SR setup, you can use the following CLI commands:

show mpls label ranges
show isis segment-routing
show isis database PE2.00-00 detail !PE2.00-00 is id
show mpls lfib route
show mpls segment-routing bindings
show platform jericho mpls route

Detailed output of PE1:

PE1#show mpls label ranges

Start End Size Usage

------------------------------------------------

0 15 16 reserved

16 99999 99984 static mpls

100000 116383 16384 ldp (dynamic)

116384 132767 16384 pseudowire (dynamic)

132768 149151 16384 bgp (dynamic)

149152 165535 16384 isis (dynamic)

165536 362143 196608 free (dynamic)

362144 899999 537856 unassigned

900000 965535 65536 isis-sr <<< default ISIS-SR range

900000 965535 65536 bgp-sr

965536 1031071 65536 srlb

1031072 1036287 5216 unassigned

1036288 1048575 12288 l2evpn

PE1#show isis segment-routing

System ID: PE1 Instance: sr

SR supported Data-plane: MPLS SR Router ID: 1.1.1.1

SR Global Block( SRGB ): Base: 900000 Size: 65536

Adj-SID allocation mode: SR-adjacencies

Adj-SID allocation pool: Base: 149152 Size: 16384

All Prefix Segments have : P:0 E:0 V:0 L:0

IS-IS Reachability Algorithm : SPF (0)

Number of IS-IS segment routing capable peers: 3

Self-Originated Segment Statistics:

Node-Segments : 1

Prefix-Segments : 0

Proxy-Node-Segments : 0

Adjacency Segments : 1

PE1#show isis database PE2.00-00 detail

IS-IS Instance: sr VRF: default

IS-IS Level 2 Link State Database

LSPID Seq Num Cksum Life IS Flags

PE2.00-00 12 57553 1049 L2 <>

NLPID: 0xCC(IPv4)

Hostname: PE2

Area address: 49.0001

Interface address: 4.4.4.4

Interface address: 10.3.4.4

IS Neighbor : P2.11 Metric: 10

LAN-Adj-sid: 100000 flags: [ L V ] weight: 0 system ID: 0000.0000.3333

Reachability : 4.4.4.4/32 Metric: 10 Type: 1 Up

SR Prefix-SID: 4 Flags: [ N ] Algorithm: 0

Reachability : 10.3.4.0/24 Metric: 10 Type: 1 Up

Router Capabilities: Router Id: 4.4.4.4 Flags: [ ]

SR Local Block:

SRLB Base: 965536 Range: 65536

Area leader priority: 250 algorithm: 0

SR Capability: Flags: [ I ]

SRGB Base: 900000 Range: 65536

PE1#show mpls lfib route

MPLS forwarding table (Label [metric] Vias) - 4 routes

MPLS next-hop resolution allow default route: False

Via Type Codes:

M - MPLS via, P - Pseudowire via,

I - IP lookup via, V - VLAN via,

VA - EVPN VLAN aware via, ES - EVPN ethernet segment via,

VF - EVPN VLAN flood via, AF - EVPN VLAN aware flood via,

NG - Nexthop group via

Source Codes:

G - gRIBI, S - Static MPLS route,

B2 - BGP L2 EVPN, B3 - BGP L3 VPN,

R - RSVP, LP - LDP pseudowire,

L - LDP, M - MLDP,

IP - IS-IS SR prefix segment, IA - IS-IS SR adjacency segment,

IL - IS-IS SR segment to LDP, LI - LDP to IS-IS SR segment,

BL - BGP LU, ST - SR TE policy,

DE - Debug LFIB

IA 149152 [1]

via M, 10.1.2.2, pop

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900002 [1], 2.2.2.2/32

via M, 10.1.2.2, pop

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900003 [1], 3.3.3.3/32

via M, 10.1.2.2, forward

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

IP 900004 [1], 4.4.4.4/32

via M, 10.1.2.2, forward

payload autoDecide, ttlMode uniform, apply egress-acl

interface Ethernet2/4

PE1#show mpls segment-routing bindings

1.1.1.1/32

Local binding: Label: imp-null

Remote binding: Peer ID: 0000.0000.2222, Label: 900001

2.2.2.2/32

Local binding: Label: 900002

Remote binding: Peer ID: 0000.0000.2222, Label: imp-null

3.3.3.3/32

Local binding: Label: 900003

Remote binding: Peer ID: 0000.0000.2222, Label: 900003

4.4.4.4/32

Local binding: Label: 900004

Remote binding: Peer ID: 0000.0000.2222, Label: 900004

PE1#show platform jericho mpls route

D - ECMP is divergent across switching chips

---------------------------------------------------------------------------------------------

| Mpls Table |

|---------------------------------------------------------------------------------------------|

|---------------------------------------------------------------------------------------------|

|149152 |Et2/4 |1027 | 44:4c:a8:97:8c:51 | PopE pipe auto|32773 |8200 |8 | 0 |

Egress Action Codes:

M - Mpls Tunnel, G - GRE Tunnel, MoG - Mpls-over-GRE Tunnel

ECMP Codes:

D - ECMP is divergent across switching chips

-----------------------------------------------------------------------------------------------

| Mpls Table

|-----------------------------------------------------------------------------------------------

|-----------------------------------------------------------------------------------------------

|900002 |Pop | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -

|900003 |Forward | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -

|900004 |Forward | Et2/4 |1027 |8188 | 44:4c:a8:97:8c:51 | - |32771 | -

7/31/2019

ISIS multi-topology - interfaces with different IPv4/v6 AF enabled

As we know, the ISIS requires address family configuration must be matched between the ISIS router and interface configuration. So if you have both ipv4 and ipv6 AF configuration, you must do the same under interface.

Sometime you may have some of interfaces only enabled ipv4, but dual stack globally, in this case, you will need the ISIS multi-topology.

router isis VrfDefault
net 49.0001.0000.0000.0002.00
router-id ipv4 100.255.255.2
is-type level-1
!
address-family ipv4 unicast
!
address-family ipv6 unicast
multi-topology <<< need multi-topology enabled here
!
interface Port-Channel211
no switchport
ip address 100.201.1.0/31
ipv6 address 100:201:1::/127
isis enable VrfDefault <<< this intf has both v4 and v6
!
interface Port-Channel251
no switchport
ip address 100.205.1.0/31
isis enable VrfDefault <<< but this intf only v4
isis multi-topology address-family ipv4 unicast <<<

And both neighbors are up!!

Router1(config-if-Po251)#show isis nei
Instance VRF System Id Type Interface SNPA State Hold time Circuit Id
VrfDef default RouterDualStack L1 Port-Channel211 P2P UP 3 48
VrfDef default RouterV4 L1 Port-Channel251 P2P UP 2 63

6/14/2019

Arista EOS Tunneling Mechanism (3) - Sw Tunnel + ISIS

Above is a simple setup to demonstrate the feature of ISIS over GRE tunnel. It is worthy to note that:

This tunnel is a software tunnel without hardware programming. It ONLY works for the local packets generated by CPU and is capped by CoPP policy. So no data/traffic traffic is thru the tunnel.
The only valid use case per TOI is to advertise the ISIS routes to a remote application, like below

The application can learn the whole view of a network topology
Based application or business intelligence, it can program the routers to instruct the traffic flows. So a practical SDN solution to me.

Limitation:

IPv6 underlay endpoint is not supported
Underlay VRF is not supported
That says the tunnel source/destination must be ipv4 address under default VRF.

Some extras:

Even in the above the TOI, it says it only supports ISIS, but in the lab, the BGP session is up and running w/o any issue. But one thing to be aware is to assign the tunnel interface TTL. Otherwise, the neighbor fails to come up

R11(config-router-bgp)#sh run sec router bgp

router bgp 11

neighbor 10.100.100.44 remote-as 44

neighbor 10.100.100.44 maximum-routes 12000

redistribute connected

R11(config-router-bgp)#sh ip bgp sum

BGP summary information for VRF default

Router identifier 11.11.11.11, local AS number 11

Neighbor Status Codes: m - Under maintenance

Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc

10.100.100.44 4 44 4002 4013 0 0 01:29:18 Estab 11 11 <<<< session is up

Overlay VRF is supported, so tunnel interface can belong to none-default VRF

R11(config)#sh run int tunnel 101

interface Tunnel101

description sw-tunnel-gre-vrf-v1

vrf forwarding v1

ip address 10.101.101.11/24

isis enable isis.over.GRE.v1

isis bfd

isis network point-to-point

tunnel mode gre

tunnel source 11.11.11.1

tunnel destination 44.44.44.1

R11(config)#sh int tunnel 101

Tunnel101 is up, line protocol is up (connected)

Hardware is Tunnel, address is 0b0b.0b01.0800

Description: sw-tunnel-gre-vrf-v1

Internet address is 10.101.101.11/24

Broadcast address is 255.255.255.255

Tunnel source 11.11.11.1, destination 44.44.44.1

....

R44(config)#sh run int tu101

interface Tunnel101

description sw-tunnel-gre-vrf-v1

vrf forwarding v1

ip address 10.101.101.44/24

isis enable isis.over.GRE.v1

isis bfd

isis network point-to-point

tunnel mode gre

tunnel source 44.44.44.1

tunnel destination 11.11.11.1

R44(config)#sh int tu101

Tunnel101 is up, line protocol is up (connected)

Hardware is Tunnel, address is 2c2c.2c01.0800

Description: sw-tunnel-gre-vrf-v1

Internet address is 10.101.101.44/24

Broadcast address is 255.255.255.255

....

R11(config)#ping vrf v1 10.101.101.44

PING 10.101.101.44 (10.101.101.44) 72(100) bytes of data.

80 bytes from 10.101.101.44: icmp_seq=1 ttl=64 time=0.234 ms

80 bytes from 10.101.101.44: icmp_seq=2 ttl=64 time=0.153 ms

...

R11(config)#sh isis neighbors vrf v1

Instance VRF System Id Type Interface SNPA State Hold time Circuit Id

isis.over v1 R44 L1 Tunnel101 P2P UP 25 87

R11(config)#sh ip route vrf v1 isis

VRF: v1

....

I L1 101.101.44.1/32 [115/11] via 10.101.101.44, Tunnel101

I L1 101.101.44.2/32 [115/11] via 10.101.101.44, Tunnel101

I L1 101.101.44.3/32 [115/11] via 10.101.101.44, Tunnel101

I L1 101.101.44.4/32 [115/11] via 10.101.101.44, Tunnel101

4/16/2019

How to debug ISIS/BFD neighbor issue

Topology:

[R1/DCS-7512N]:et11/36/1--{180.13.1.0/24}--et11/1:[R2/DCS-7050QX]

* both routers run 4.20.11M
* Between R1 and R2, ISIS/BFD is enabled shown as below

Router Configuration:

R1 configuration:

router isis isis.def
net 49.0001.0000.0000.0001.00
is-type level-1
log-adjacency-changes
redistribute connected
spf-interval 1 50 100
timers lsp generation 1 10 1000
!
address-family ipv4 unicast
maximum-paths 32
!
address-family ipv6 unicast
maximum-paths 32
!
interface Ethernet11/36/1
description UpEt_A_3.1
speed forced 10000full
no switchport
ip address 180.13.1.1/24
ipv6 address 180:13:1::1/64
isis enable isis.def
isis bfd
isis circuit-type level-1
isis network point-to-point
isis authentication mode md5 level-1
isis authentication key 7 QI3r+/7nP3Q= level-1

R2 configuration:
router isis isis.def
net 49.0001.0000.0000.0103.00
is-type level-1
log-adjacency-changes
spf-interval 1 50 100
timers lsp generation 1 10 1000
!
address-family ipv4 unicast
maximum-paths 32
!
address-family ipv6 unicast
maximum-paths 32
!
interface Ethernet11/1
description UpEt_A_3.1
speed forced 10000full
no switchport
ip address 180.13.1.100/24
ipv6 address 180:13:1::100/64
isis enable isis.def
isis bfd
isis circuit-type level-1
isis network point-to-point
isis authentication mode md5 level-1
isis authentication key 7 QI3r+/7nP3Q= level-1

Symptom:

R1: No ISIS and BFD neighbor

R1.13:53:14(config)#show isis nei | grep uspine3
R1.13:53:16(config)#show bfd nei | grep 180.13
R1.13:53:18(config)#

R2: ISIS neighbor stuck in INIT, and BFD is down

R2#sh isis nei
Instance VRF System Id Type Interface SNPA State Hold time Circuit Id
isis.def default R1 L1 Ethernet11/1 P2P INIT 26 2D

R2#sh bfd nei
VRF name: default
-----------------
DstAddr MyDisc YourDisc Interface Type LastUp LastDown LastDiag State
-------------------- --------------- --------------- --------------------- ------------ -------------------- -------------------- ----------------------- -----
180.13.1.1 2628313401 0 Ethernet11/1(47) normal 04/15/19 13:25 04/15/19 13:26 Nbr Signaled Down Down

Step1: R1/R2 - "show lldp nei" good and ping works, so no connectivity issue but high latency

-- show lldp nei --
R2#sh lldp nei
Port Neighbor Device ID Neighbor Port ID TTL
Et11/1 R2 Ethernet11/36/1 120

-- ping -- R2#ping 180.13.1.1
PING 180.13.1.1 (180.13.1.1) 72(100) bytes of data.
80 bytes from 180.13.1.1: icmp_seq=1 ttl=64 time=390 ms
80 bytes from 180.13.1.1: icmp_seq=2 ttl=64 time=437 ms
....

-- show interface <intf> | grep error --
R1.13:46:01(config)#sh int et11/36/1 | grep error
0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards

0 output errors, 0 collisions

R2.13:45:34(config)#sh int et11/1 | grep error
0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards

0 output errors, 0 collisions

Step2: R2's Down BFD shows NO incoming bfd pkts

-- show bfd neighbor dest-ip <ip> detail --
R2#sh bfd neighbors dest-ip 180.13.1.1 detail
VRF name: default
-----------------
Peer Addr 180.13.1.1, Intf Ethernet11/1, Type normal, State Down
VRF default, LAddr 180.13.2.100, LD/RD 2628313401/0
Session state is Down and not using echo function
Last Up Apr 15 13:25:27 2019
Last Down Apr 15 13:26:23 2019
Last Diag: Nbr Signaled Down
TxInt: 1000, RxInt: 1000, Multiplier: 3
Received RxInt: 300, Received Multiplier: 3
Rx Count: 0, Rx Interval (ms) min/max/avg: 0/0/0 last: never
Tx Count: 87681, Tx Interval (ms) min/max/avg: 508/996/870 last: 627 ms ago
Detect Time: 3000
Sched Delay: 1*TxInt: 87898, 2*TxInt: 0, 3*TxInt: 0, GT 3*TxInt: 0

R2 sends out BFD but no receiving

-- bash tcpdump -ni <intf> udp --
[admin@R2 ~]$ tcpdump -ni et11_1 udp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:48:22.309536 28:99:3a:3b:15:39 > 44:4c:a8:a5:11:40, ethertype IPv4 (0x0800), length 66: 180.13.1.100.57657 > 180.13.1.1.bfd-control: BFDv1, Control, State Down, Flags: [none], length: 24

11:48:23.301574 28:99:3a:3b:15:39 > 44:4c:a8:a5:11:40, ethertype IPv4 (0x0800), length 66: 180.13.1.100.57657 > 180.13.1.1.bfd-control: BFDv1, Control, State Down, Flags: [none], length: 24

Same observation on R1, only receiving no sending

R1.13:53:18(config)#bash tcpdump -ni et11_36_2 udp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_36_2, link-type EN10MB (Ethernet), capture size 262144 bytes
13:55:10.568533 28:99:3a:3b:15:39 > 44:4c:a8:a5:11:40, ethertype IPv4 (0x0800), length 66: 180.13.2.100.57657 > 180.13.2.1.bfd-control: BFDv1, Control, State Down, Flags: [none], length: 24

13:55:11.445074 28:99:3a:3b:15:39 > 44:4c:a8:a5:11:40, ethertype IPv4 (0x0800), length 66: 180.13.2.100.57657 > 180.13.2.1.bfd-control: BFDv1, Control, State Down, Flags: [none], length: 24

ISIS should notify the BFD to establish session after ISIS neighbor up. So the symptom is narrowed down to ISIS neighbor issue.

Step 3: Tcpdump ISIS IIH (isis hello) on both sides

-- show ver | grep MAC to get system MAC --

R1#show ver | grep MAC
System MAC address: 444c.a8a5.1140

R2#show ver | grep MAC
System MAC address: 2899.3a3b.1539

R2: IIH 2-way and looks good, R1/R2 send IIH every 10 seconds

-- bash tcpdump -ni <intf> iih --
[admin@R2 ~]$ tcpdump -ni et11_1 iih
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:18:16.104275 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497
11:18:18.000062 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497
11:18:24.723990 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497
11:18:27.597320 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497

R1: IIH, only 1 way, from R1 to R2, no R2 to R1

[admin@bn303 ~]$ tcpdump -ni et11_36_2 iih
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_36_2, link-type EN10MB (Ethernet), capture size 262144 bytes
11:23:53.268387 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497

11:24:02.823894 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497

IIH packets are destinated to well-known LAN broadcast address (0900:2b00:0005). These packets can't reach from R2 to R1.

Step 4: Arping

Arping shows similar, R1 to R2 is good, but seeing broadcast packets (all ff) drop on the direction of R2 to R1 even with arp ping

* Arp ping is to send our brocast L2 arp request to probe hosts

[admin@R1 ~]$ arping -b -I et11_36_1 180.13.1.100
ARPING 180.13.1.100 from 180.13.1.1 et11_36_1
Unicast reply from 180.13.1.100 [28:99:3A:3B:15:39] 419.967ms
Unicast reply from 180.13.1.100 [28:99:3A:3B:15:39] 409.205ms
Unicast reply from 180.13.1.100 [28:99:3A:3B:15:39] 401.616ms
Unicast reply from 180.13.1.100 [28:99:3A:3B:15:39] 410.797ms
^CSent 4 probes (4 broadcast(s))
Received 4 response(s)

[admin@R2 ~]$ arping -b -I et11_1 180.13.1.1
ARPING 180.13.1.1 from 180.13.1.100 et11_1
^CSent 9 probes (9 broadcast(s))

Received 0 response(s)

[admin@R2 ~]$ tcpdump -ni et11_1 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:46:04.859574 28:99:3a:3b:15:39 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 180.13.1.1 (Broadcast) tell 180.13.1.100, length 28

11:46:05.859645 28:99:3a:3b:15:39 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 180.13.1.1 (Broadcast) tell 180.13.1.100, length 28

At this point, the issue is narrowed down to, broadcast packets from R2 to R1 are dropped unexpectedly, which causes R1 not seeing R2's IIH packets, then no ISIS/BFD neighbor.

Step 5: hardware issue, but who drops the packets?

Ok, now we can say, most likely the problem is hardware, but who? The best way to nail down the culprit is to mirror packets to cpu, to see if the broadcast packets leave the R2.

* The feature of mirroring interface packets to CPU is only supported on Sand (Jericho/Arad) platform.

R1.14:32:14(config)#monitor session 1 source ethernet 11/36/1 rx
R1.14:32:19(config)#monitor session 1 destination cpu

Seeing IIH from R2 to R1 on the mirror cpu. << This is the packets received on the wire of et11/36/1

R1.14:32:20(config)#bash tcpdump -ni mirror0 iih
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on mirror0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:33:04.105249 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497
14:33:13.257565 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497
14:33:21.554871 28:99:3a:3b:15:39 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0103, length 1497

But not on the physical ethernet port << This is the packets received by the kernel via et11/36/1

R1.14:33:22(config)#bash tcpdump -ni et11_36_1 iih
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on et11_36_1, link-type EN10MB (Ethernet), capture size 262144 bytes
14:33:35.056748 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497

14:33:42.990686 44:4c:a8:a5:11:40 > 09:00:2b:00:00:05, 802.3, length 1500: LLC, dsap OSI (0xfe) Individual, ssap OSI (0xfe) Command, ctrl 0x03: OSI NLPID IS-IS (0x83): p2p IIH, src-id 0000.0000.0001, length 1497

Now we know the broadcast packets are dropped on R1 between eth and kernel.

Step 6: Counters

And this is approved by the counter output on both R1 and R2

R2/Trident - No loss

R2#clear counters
R2#clear platform trident counters
R2#sh cpu counters queue | nz

--------------------------------------------------------------------------------
Linecard0/0
--------------------------------------------------------------------------------
Queue Counter/pkts* Drops/pkts
--------------- ------------------- -------------------
OSPF/ISIS 2 0
BFD 10 0
R2#sh platform trident counters interface e11/1 | egrep -i 'card|drop' | nz
R2#

R1/Jericho - Loss seen

R1#clear hardware counter drop
R1#clear counters

R1#sh cpu counters queue | grep 'Et11/36/1' | nz

CoppSystemLldp Et11/36/1 4 924 0 0

R1#sh hardware counter drop | grep o11/1
A Jericho11/1 DeqDeletePktCnt : 39148 : 2019-04-16 11:59:39 : 2019-04-16 11:59:39
P Jericho11/1 dropVoqInNullRoute : 3800339040 : 2019-04-16 11:59:39 : 2019-04-16 11:59:39

R2#sh hardware counter drop | grep o11/1
A Jericho11/1 DeqDeletePktCnt : 39153 : 2019-04-16 11:59:39 : 2019-04-16 11:59:45
P Jericho11/1 dropVoqInNullRoute : 3800605701 : 2019-04-16 11:59:39 : 2019-04-16 11:59:45

So, should goes to DeqDeletePktCnt or dropVoqInNullRoute. What does they mean?