7/31/2019

ISIS multi-topology - interfaces with different IPv4/v6 AF enabled

As we know, the ISIS requires address family configuration must be matched between the ISIS router and interface configuration. So if you have both ipv4 and ipv6 AF configuration, you must do the same under interface.

Sometime you may have some of interfaces only enabled ipv4, but dual stack globally, in this case, you will need the ISIS multi-topology.

router isis VrfDefault
   net 49.0001.0000.0000.0002.00
   router-id ipv4 100.255.255.2
   is-type level-1
   !
   address-family ipv4 unicast
   !
   address-family ipv6 unicast
      multi-topology   <<< need multi-topology enabled here
!
interface Port-Channel211
   no switchport
   ip address 100.201.1.0/31
   ipv6 address 100:201:1::/127
   isis enable VrfDefault <<< this intf has both v4 and v6
!
interface Port-Channel251
   no switchport
   ip address 100.205.1.0/31
   isis enable VrfDefault <<< but this intf only v4
   isis multi-topology address-family ipv4 unicast <<<

And both neighbors are up!!

Router1(config-if-Po251)#show isis nei
Instance  VRF      System Id     Type Interface          SNPA              State Hold time   Circuit Id
VrfDef default  RouterDualStack  L1   Port-Channel211    P2P               UP    3           48
VrfDef default  RouterV4         L1   Port-Channel251    P2P               UP    2           63


7/21/2019

MACSec

Arista EOS starts to support MACSec - 802.1AE from 4.15.4F which was released around 2016. Later related features:
Some key take-aways:
  • Purpose: to protect data traffic from various types of attacks, 
    • passive: snooping
    • active: reply, man in the middle
  • vs IPSec
    • Level: IPSec is at the IP level, vs MACSec MAC/data link level.  
    • End or Hop: IPSec provides end-to-end protection, MACsec secures Per-hop. (TLS is similar like IPSec, end-to-end)
    • HW: IPSec doesn't rely on hw but with IPSec often, MACSec is on PHY level with special hw. For example, 
      • 7500E-6CFPX-LC
      • 7280SRAM-48C6
      • 7280CR2M-30
      • 7500R2M-36CQ-LC
    • Throughput: IPSec has a ceiling, while MACsec is line-rate like 100G. 
  • Key components:
    • CAK - Conn Ass Key, master key. Either manual or from key server
    • CAN - CAK's name
    • SAK - Secu Ass Key, derived from CAK, used to data encryption. 
  • Point-to-Point vs clear 802.1Q
    • Clear 802.1Q means, move vlan tag ahead of MACSec Tag, a must for hub-spoke topology. Cisco ASR supports it. 
    • Arista EOS only supports point-to-point. 
  • Packet format:
    • Add 16-byte SecTAG after srcMAC and VLAN Tag
    • Add 16-byte ICV at end of ethernet frames
    • Ethertype = 0x88e5 (why not use 0x85ec?, hahaha)
  • MACSec proxy:
    • Provides MACSec for VxLAN traffic
    • Loopback traffic to a MACSec capable front panel port to process. 
  • MISC:
    • fallback key in case MACSec failed or during a configuration change
    • L2-protocol pass thru, skipping LLDP packets
Configuration (EOS):

mac security
license DCI1 <license>
profile toISP1
  ! key <CAN> <type, 0-clear text> <key>
  key a1 0 a123456
interface eth4/1/1
  mac security profile toISP1
!
show mac secu counters
!
show mac secu interface

7/18/2019

Arista Modular Switch - simulate LC remove/insert

!! bash cat /persist/sys/Linecard3.cfg
platform module Linecard3 remove       
platform module Linecard3 insert

7508E CPU Util% is high (1)

Saw quite slow response on one of my lab router, a 7508E with latest EOS release. 

mlagA.10:18:17(config)#show ver
Arista DCS-7508
Hardware version:    06.00
....
Software image version: 4.22.0.1F

CPU only has 64% idle cycles. Not low. 

mlagA.10:18:13(config)#show proc top once | more
%Cpu(s): 29.6 us,  3.9 sy,  0.0 ni, 64.0 id,  0.1 wa,  0.5 hi,  2.0 si,  0.0 st
  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
   23 root      20   0     0    0    0 R  89.6  0.0  26:04.70 ksoftirqd/2
   17 root      20   0     0    0    0 R  68.5  0.0  24:27.80 ksoftirqd/1

I suspect there may be some unexpected traffic hitting the CPU, so check the output of "show cpu couter queue | nz"

yo412.mlagA.10:22:38(config)#clear counters
!!! even a single command - clear counter, takes almost 10 sec to complete !!!


yo412.mlagA.10:22:47(config)#show cpu counters queue | nz | more
Arad3/0:
CoPP Class                 Queue                    Pkts             Octets           DropPkts         DropOctets
Aggregate
-----------------------------------------------------------------------------------------------------------------
CoppSystemL3LpmOverflow    Et3/6/1                  1753             473344              74945           21049856
CoppSystemL3LpmOverflow    Et3/6/2                  1112             307200              73605           20702976
CoppSystemL3LpmOverflow    Et3/6/3                   610             166912              86302           23954432
CoppSystemL3LpmOverflow    Et3/6/4                  1178             320256              77089           21414656

Looks like there is a lot of packets hitting the cpu, even the CoPP filters out most of them. But this is a full load chassis, the aggregated traffic is still too heavy to a x86 CPU. 

Try to tcpdump the incoming packets from et3/6/1 and punted to cpu. Surprisingly not many... 

mlagA.10:35:23(config)#bash tcpdump -nvvi et3_6_1
tcpdump: listening on et3_6_1, link-type EN10MB (Ethernet), capture size 262144 bytes
10:35:36.066689 00:1c:73:46:0d:b0 > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 124: LACPv1, length 110
10:35:39.530341 00:1c:73:3b:e0:22 > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 124: LACPv1, length 110
^C

2 packets captured

Try to mirror this port to cpu then tcpdump it. (This feature is only supported on 7500E/R or 7280R devices)

mlagA.10:37:46(config)#monitor session 1 source et3/6/1 rx
mlagA.10:39:12(config)#monitor session 1 destination cpu

mlagA.10:39:15(config)#bash tcpdump -nvi mirror0
tcpdump: listening on mirror0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:40:08.198512 1e:af:14:08:18:02 > 00:aa:aa:aa:bb:cc, ethertype 802.1Q (0x8100), length 252: vlan 1408, p 0, ethertype IPv4, 
    100.14.8.119.30485 > 220.200.16.1.24659: Flags [R.UW], seq 0:194, ack 0, win 61689, urg 0, length 194 
10:40:08.199078 1e:af:14:09:18:01 > 00:aa:aa:aa:bb:cc, ethertype 802.1Q (0x8100), length 252: vlan 1409, p 0, ethertype IPv4, 
    100.14.9.118.30504 > 220.200.17.1.24648: Flags [PUEW], seq 0:194, win 62028, urg 0, length 194

Do we have the route? No....

mlagA.10:40:08(config)#sh ip route 220.200.17.1
VRF: default
....
Gateway of last resort is not set

Create a null route for this prefix, response is better and "show cpu couter queue | nz" is back to normal now, no L3LPMOverflow anymore. 

mlagA.10:54:28(config)#ip route 220.200.0.0/16 null0
mlagA.10:54:59(config)#sh cpu counters queue | nz | more
Arad3/0:
CoPP Class                 Queue                    Pkts             Octets           DropPkts         DropOctets
Aggregate
-----------------------------------------------------------------------------------------------------------------
CoppSystemIgmp             Et3/1/2                   160              10240                  0                  0
CoppSystemIgmp             Et3/1/4                   160              10240                  0                  0


But cpu still high. And the busiest process is changed to SandFap instead of ksoftirqd. Hmmm.... why?

mlagA.10:57:19(config)#sh proc top once | more
%Cpu(s): 30.2 us,  4.1 sy,  0.0 ni, 62.1 id,  0.1 wa,  0.5 hi,  3.1 si,  0.0 st
...
  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
12874 root      20   0 1001m 357m 192m R 100.4  2.2 110:39.04 SandFap
13025 root      20   0 1001m 357m 192m S  69.6  2.2 110:54.53 SandFap
16765 root      20   0 1001m 359m 193m S  51.2  2.2  96:56.18 SandFap