6/28/2019

In output of "show ip bgp sum", does PfxRcd mean prefix or path?

R1 #show ip bgp summary
BGP summary information for VRF default
Router identifier 10.1.255.1, local AS number 65001
Neighbor Status Codes: m - Under maintenance
  Description              Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc
  RRv4                        10.1.255.104  4  65010        2678462     57500    0    0  216d20h Estab  928963 925526

Before addpath, for one particular neighbor, it only sends ONE best path for one particular prefix. So PfxRcd = PathRcd. But after BGP addpath, the behavior changes. Multiple paths can be sent for one prefix to speed up the convergence time. So the meaning of this number is changed to "pathRcd". 

How to run RFC2544 thruput/latency testing with Ixia

Step 1 - Connect device in a snake way like below



Step 2 - Configure IxNetworks for RFC 2544

1. Click "Add QuickTest". After configuration, it should show up in the right side



2. Finish all options. Pretty straightforward, just need to configure like
  • Ports to be used (Definitely need 2-way)
  • Packet type (Eth/IPv4/IPv6)
  • Packet size (make sure interface MTU matched)
  • Traffic option, like how long each case last
  • Stats parameters, like calculate latency (Cut through)
  • 100% line rate (if for thruput)
Step 3 - Run the test and generate the report, like


6/27/2019

Use MAC ACL to isolate the failure point

For L2 traffic, besides checking drops/discard counter, another way to isolate the failure point is to use the MAC ACL, like

mac access-list macCount
   counters per-entry
   10 permit 00:00:03:03:00:14 00:00:00:00:00:00 04:68:03:03:00:14 00:00:00:00:00:00 log
   20 permit any any log
!
interface Ethernet3/1
   switchport access vlan 3003
   mac access-group macCount in

The above MAC acl - macCount is count the number of packets with source MAC - 0000.0303.0014 and dest MAC - 0468.0303.0014. And it is applied on Eth3/1 ingress direction (egress ACL is not supported)

Router#show mac access-lists
MAC Access List macCount
        counters per-entry
        10 permit 00:00:03:03:00:14 00:00:00:00:00:00 04:68:03:03:00:14 00:00:00:00:00:00 log [match 216114288 packets, 0:00:00 ago]
        20 permit any any log

This is an Arista DCS-7280CR2A-60-F with 4.22.0F

6/19/2019

Connect Ixia Novas Qsfp28 100G via 100GBASE-CR4 cable

The arista device is Arista DCS-7280CR2-60-F loading 4.21.2F. The other side is IxNetworks's NOVUS-M100GE8Q28 card, and using Arista 100G AOC cable. 

Et60/1              connected    1        full   100G   100GBASE-CR4

Have to disable the autoneg on Ixia side to bring up the link. As shown as below

1. First stop all traffic (if the option is gray, that's the reason) then Edit L1 Properties
2. Unclick "Use IEEE Media for 100BASE CR4"
3. Uncheck "Auto Negotiate"



6/17/2019

Arista EOS Tunneling Mechanism (4) - Sw Tunnel vs Data




In the above topology, I show you that the data traffic doesn't go thru the sw tunnel. I use 2 routers to simulate hosts with default gateway pointing to R11 and R44. And we can see

  • Ping between R11 and R44 works
  • But the ping from host1 to host2 doesn't work

R11 has the correct route and ping to R44's ip address is good. 

R11#sh ip route 99.2.2.99
...
 I L1     99.2.2.0/24 [115/20] via 10.100.100.44, Tunnel100

R11.cd642.leaf18#ping 99.2.2.2
PING 99.2.2.2 (99.2.2.2) 72(100) bytes of data.
80 bytes from 99.2.2.2: icmp_seq=1 ttl=64 time=0.264 ms
80 bytes from 99.2.2.2: icmp_seq=2 ttl=64 time=0.119 ms

Host1 also has right route but ping failed, so data traffic can't pass thru. 

host1#sh ip route 99.2.2.99
....
 S        99.0.0.0/8 [1/0] via 99.1.1.1, Ethernet51/1

host1#ping 99.2.2.99
--- 99.2.2.99 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 40ms

Now, we add NHG + Decap group (will cover the details later) on both ends. 



!!!!! R11 !!!!!!
ip route 99.2.2.0/24 Nexthop-Group nhg-gre-99-net
!
nexthop-group nhg-gre-99-net type gre
   size 1
   ttl 64
   tunnel-source 11.11.11.11
   entry 0 tunnel-destination 44.44.44.44
!
ip decap-group decap-net-99
   tunnel type gre
   tunnel decap-ip 11.11.11.11

!!!!! R44 !!!!!!
ip route 99.1.1.0/24 Nexthop-Group nhg-gre-99-net
!
nexthop-group nhg-gre-99-net type gre
   size 1
   ttl 64
   tunnel-source 44.44.44.44
   entry 0 tunnel-destination 11.11.11.11
!
ip decap-group decap-net-99
   tunnel type gre
   tunnel decap-ip 44.44.44.44

Now the ping works well. 

host1#ping 99.2.2.99
PING 99.2.2.99 (99.2.2.99) 72(100) bytes of data.
80 bytes from 99.2.2.99: icmp_seq=1 ttl=62 time=0.264 ms
.....
--- 99.2.2.99 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.102/0.140/0.264/0.062 ms, ipg/ewma 0.184/0.200 ms

So basically that's how it works. 
  • On each router has static nexthop + decap group configuration to other routers
  • The application software works as a passive ISIS neighbor to establish a neighbor with one ISIS router over a GRE tunnel. 
  • So it can fetch the whole LSA DB to get a whole view of the network. 
  • By using CLI or eAPI, the software can program each router with a static route pointing to the nexthop group entry configured in step 1. 

6/14/2019

Arista EOS Tunneling Mechanism (3) - Sw Tunnel + ISIS


 

Above is a simple setup to demonstrate the feature of ISIS over GRE tunnel. It is worthy to note that:
  • This tunnel is a software tunnel without hardware programming. It ONLY works for the local packets generated by CPU and is capped by CoPP policy. So no data/traffic traffic is thru the tunnel. 
  • The only valid use case per TOI is to advertise the ISIS routes to a remote application, like below
    • The application can learn the whole view of a network topology
    • Based application or business intelligence, it can program the routers to instruct the traffic flows. So a practical SDN solution to me. 


Limitation:

  • IPv6 underlay endpoint is not supported
  • Underlay VRF is not supported
  • That says the tunnel source/destination must be ipv4 address under default VRF.

Some extras:
  • Even in the above the TOI, it says it only supports ISIS, but in the lab, the BGP session is up and running w/o any issue. But one thing to be aware is to assign the tunnel interface TTL. Otherwise, the neighbor fails to come up
R11(config-router-bgp)#sh run sec router bgp
router bgp 11
   neighbor 10.100.100.44 remote-as 44
   neighbor 10.100.100.44 maximum-routes 12000
   redistribute connected
R11(config-router-bgp)#sh ip bgp sum
BGP summary information for VRF default
Router identifier 11.11.11.11, local AS number 11
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State   PfxRcd PfxAcc
  10.100.100.44    4  44              4002      4013    0    0 01:29:18 Estab   11     11 <<<< session is up
  • Overlay VRF is supported, so tunnel interface can belong to none-default VRF
R11(config)#sh run int tunnel 101
interface Tunnel101
   description sw-tunnel-gre-vrf-v1
   vrf forwarding v1
   ip address 10.101.101.11/24
   isis enable isis.over.GRE.v1
   isis bfd
   isis network point-to-point
   tunnel mode gre
   tunnel source 11.11.11.1
   tunnel destination 44.44.44.1

R11(config)#sh int tunnel 101
Tunnel101 is up, line protocol is up (connected)
  Hardware is Tunnel, address is 0b0b.0b01.0800
  Description: sw-tunnel-gre-vrf-v1
  Internet address is 10.101.101.11/24
  Broadcast address is 255.255.255.255
  Tunnel source 11.11.11.1, destination 44.44.44.1
....

R44(config)#sh run int tu101
interface Tunnel101
   description sw-tunnel-gre-vrf-v1
   vrf forwarding v1
   ip address 10.101.101.44/24
   isis enable isis.over.GRE.v1
   isis bfd
   isis network point-to-point
   tunnel mode gre
   tunnel source 44.44.44.1
   tunnel destination 11.11.11.1

R44(config)#sh int tu101
Tunnel101 is up, line protocol is up (connected)
  Hardware is Tunnel, address is 2c2c.2c01.0800
  Description: sw-tunnel-gre-vrf-v1
  Internet address is 10.101.101.44/24
  Broadcast address is 255.255.255.255
....

R11(config)#ping vrf v1 10.101.101.44
PING 10.101.101.44 (10.101.101.44) 72(100) bytes of data.
80 bytes from 10.101.101.44: icmp_seq=1 ttl=64 time=0.234 ms
80 bytes from 10.101.101.44: icmp_seq=2 ttl=64 time=0.153 ms
...


R11(config)#sh isis neighbors vrf v1

Instance  VRF      System Id        Type Interface          SNPA              State Hold time   Circuit Id

isis.over v1       R44              L1   Tunnel101          P2P               UP    25          87

R11(config)#sh ip route vrf v1 isis
VRF: v1
....
 I L1     101.101.44.1/32 [115/11] via 10.101.101.44, Tunnel101
 I L1     101.101.44.2/32 [115/11] via 10.101.101.44, Tunnel101
 I L1     101.101.44.3/32 [115/11] via 10.101.101.44, Tunnel101
 I L1     101.101.44.4/32 [115/11] via 10.101.101.44, Tunnel101

6/12/2019

Arista EOS Tunneling Mechanism (2) - Hw/Sw Tunnel


We first talk a bit on GRE tunneling - "tunnel interface".
  • Starting from 4.17.0F, you can configure a software GRE tunnel with ISIS over it. 
    • Over the tunnel, it is ONLY for the control plane as specified in the feature limitation
    • "The only supported use case is to allow IS-IS on the switch to advertise its routes over a GRE tunnel to a remote system that hosts IS-IS topology-learning applications."  
    • So it is more like to establish isis neighbor and advertise ISIS LSDB to remote software. 
  • In 4.21.1F, the hardware GRE tunnel is supported on Jericho platforms like 7280R, 7500R and 7020R. But as of now (Jun 2019), only supported overlay routing protocol is BGP/ISIS. 
Here is the detailed comparison between 2 types of tunnel interfaces:


Some configuration tips:
  • The configuration is same, the system will automatically select hardware or software based on hardware platform and release. 
  • "show interface tunnel #" shows the tunnel type. Please note that even this show interface output indicates the interface is up, it doesn't mean it is working. Better to verify by ping the tunnel interaface ip address. 
  • Only GRE mode is supported, even there is other options like ipip or Ipsec in the CLI. 
  • Underlay endpoint address family, only the ipv4 is supported, even there is options like "tunnel source/dest <ipv6Adr>"

Arista EOS Tunneling Mechanism (1) - Tunnel vs NHG/Decap

In Arista EOS, there are 2 types of tunnel mechanisms,
  1. Regular tunnel interface - starting with "interface tunnel 10"
  2. Next-hop-group + decap-group; 
The first one is easy to understand and other vendors have similar features. The second one is very unique. Based on the EOS manual, section 35.2 and 35.3 
  • The decap group is a data structure that receives encapsulated packets and extracts the payload.
  • nexthop group is a data structure that defines a list of nexthop addresses and a tunnel type for packets routed to the specified address.
Comparison between Tunnel and next-hop-group/decap-group:
  • The tunnel is 2-way vs NHG/Decap is a 1-way
  • The tunnel is for both control plane and data plane, while NHG/Decap is ONLY for data plane, so no routing protocol over it
  • Before 4.21.1F, the tunnel is software forwarding which is capped by the CoPP policy. NHG/Decap is hardware forwarding from day-1. 
Or you can think about it in this way:
  • The tunnel interface is a traditional tunnel with a routing protocol, which is responsible for programing the forwarding information. 
  • The nhg/decap is purely a data plane tunnel, and network admin can use EAPI to program the router's forwarding. 
In the following blog posts, will cover the details of configuration and troubleshooting for some different flavor of setup. Here is a (probably uncompleted) list of EOS NHG/DecapG and tunnel features

 

6/08/2019

Trouble-shoot BGP peering issue over GRE tunnel

Starting from 4.21.1F, Arista EOS starts to support the hardware GRE tunnel interface and BGP session over the tunnel on Jericho platforms. Before the tunnel is implemented by nexthop-group and decap group. 

Here is a very simple and straightforward setup of eBGP over GRE tunnel. 


But BGP session fails to come up as shown below:

R1.gts425#sh ip bgp sum
BGP summary information for VRF default
Router identifier 1.1.1.1, local AS number 1
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc
  10.100.100.4     4  4               4766       257    0    0 00:20:40 Connect 0      0

Tunnel interface is up and works fine. 

R1.gts425#sh int tunnel 100
Tunnel100 is up, line protocol is up (connected)
  Hardware is Tunnel, address is 0101.0101.0800
  Description: tunnel-gre-sand-to-sand
  Internet address is 10.100.100.1/24
  Broadcast address is 255.255.255.255
  Tunnel source 1.1.1.1, destination 4.4.4.4
  Tunnel protocol/transport GRE/IP
   Key disabled, sequencing disabled
   Checksumming of packets disabled
  Tunnel TTL 0, Hardware forwarding not supported
  Tunnel TOS 0
  Path MTU Discovery
  Tunnel transport MTU 1476 bytes
  Up 22 minutes, 2 seconds

Ping with MTU size works totally fine

R1.gts425#ping 10.100.100.4 size 1476
...
80 bytes from 10.100.100.4: icmp_seq=5 ttl=64 time=0.118 ms

--- 10.100.100.4 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.118/0.155/0.295/0.070 ms, ipg/ewma 0.223/0.222 ms

Now let's run the tcpdump on R1 to see if hello packet out

R1.gts425(config-router-bgp)#bash tcpdump -nvvi et21 host 1.1.1.1
tcpdump: listening on et21, link-type EN10MB (Ethernet), capture size 262144 bytes
11:19:19.774224 28:99:3a:8f:91:bf > 44:4c:a8:c1:78:69, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 1, id 39325, offset 0, flags [DF], proto GRE (47), length 84)
    1.1.1.1 > 4.4.4.4: GREv0, Flags [none], proto IPv4 (0x0800), length 64
(tos 0xc0, ttl 1, id 6538, offset 0, flags [DF], proto TCP (6), length 60)
    10.100.100.1.46931 > 10.100.100.4.bgp: Flags [S], seq 3552659475, win 28720, options [mss 1436,sackOK,TS val 18764185 ecr 0,nop,wscale 7], length 0
11:19:19.774420 44:4c:a8:c1:78:69 > 28:99:3a:8f:91:bf, ethertype IPv4 (0x0800), length 126: (tos 0xc0, ttl 64, id 29177, offset 0, flags [none], proto ICMP (1), length 112)
    10.1.2.2 > 1.1.1.1: ICMP time exceeded in-transit, length 92
(tos 0x0, ttl 1, id 39325, offset 0, flags [DF], proto GRE (47), length 84)
    1.1.1.1 > 4.4.4.4: GREv0, Flags [none], proto IPv4 (0x0800), length 64
(tos 0xc0, ttl 1, id 6538, offset 0, flags [DF], proto TCP (6), length 60)
    10.100.100.1.46931 > 10.100.100.4.bgp: Flags [S], seq 3552659475, win 28720, options [mss 1436,sackOK,TS val 18764185 ecr 0,nop,wscale 7], length 0

Now we can see the reason clearly. The eBGP TCP session is default with ttl 1 and copied to outer GRE packets, so the packets get TTL expired at 10.1.2.2 which is R2. 

To fix this issue, just need to set TTL under tunnel interface

R1.gts425(config-router-bgp)#int tu 100
R1.gts425(config-if-Tu100)#tunnel ttl 10
R1.gts425(config-router-bgp)#sh ip bgp sum
BGP summary information for VRF default
Router identifier 1.1.1.1, local AS number 1
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc
  10.100.100.4     4  4                  4         4    0    0 00:00:01 Estab  0      0

5/15/2019

Switch goes into power-forever after losing all FANs

If a switch loses all fans, no matter bad hardware or removing parts, the switch should go into a "power-forever" mode. To exit this hardware protection, you have to unplug ALL power cords and reinsert them. This will reset the hardware logic. Unplug then plug power cord one by one doesn't work, neither the power flap.

5/14/2019

Shutdown the fabric link reporting CRC error

If you are seeing some SerdesCrcError in the output of "show hardware counter drop", that means you probably have some bad fabric hardware. 

R1.09:39:22#sh hardware counter drop | grep Crc
A     Fe3600-1/1   SerdesCrcErrors-18             :              82 : 2019-05-09 16:16:12 : 2019-05-14 09:31:54
A     Fe3600-4/1   SerdesCrcErrors-129            :             739 : 2019-05-09 16:32:43 : 2019-05-13 21:02:26
A     Jericho11/0   SerdesCrcErrors-43             :            2444 : 2019-05-09 16:03:03 : 2019-05-13 21:02:20
A     Jericho5/3   SerdesCrcErrors-8              :          215410 : 2019-05-09 16:02:05 : 2019-05-13 21:02:10

In this case, you can shut down the problematic fabric link to avoid them. 

R1.09:42:32(config)#platform Jericho Jericho5/3 serdes 8 shutdown
R1.09:43:11(config)#platform Jericho Jericho11/0 serdes 43 shutdown
R1.09:43:33(config)#platform Fe3600 Fe3600-1/1 serdes 18 shutdown
R1.09:43:53(config)#platform Fe3600 Fe3600-4/1 serdes 129 shutdown

4/30/2019

Arista MLAG reload-delay timers

In this blog post, I will cover the following MLAG related topics:
  • What is the MLAG reload-delay and why it is needed
  • The difference of 2 different timers
  • How to tune the timer values
After an MLAG peer boot up, all its ports are placed in err-disabled state (except the peer-link) with a reason of "mlag-issu". During the reload-delay, the MLAG agents sync all MAC and ARP information with the active peer. 

Another trigger of mlag reload delay is forwarding plane agent restart. On some platforms like TH or T3, a port speed change needs a hitful agent restart, which forces mlag interfaces to transition and starts reload-delay.

From 4.15.2F, the default reload-delay timers are different per platform:
  • All fixed systems: 300 sec
  • 7500* (Arad/Jericho): 1800 sec (due to long hw initialization time)
  • 7300* (Trident*/TH): 1200 sec
When to start the reload timer? The timers start to tick after the start of MLAG agent. In old releases, it is triggered by the sysdb agent. 

Can I lower the reload-delay timers? Yes, but be careful since if the interfaces exit err-disabled mode before sync is done, the mlag peer will blackhole the traffic. 

Which value should be used, it is highly based on system and configuration. You can look at the log messages closely. 

! this is MLAG agent up and timers are on
Apr 19 11:33:32 localhost Mlag: %AGENT-6-INITIALIZED: Agent 'Mlag' initialized; pid=3030
! LCs power on
Apr 19 11:34:08 R1 NorCalCard: %HARDWARE-6-CARD_POWERED_ON: Card Fabric3 has been powered on. model: 7512R-FM rev: 11.02 serial number: JPE16305615
....
Apr 19 11:34:24 R1 NorCalCard: %HARDWARE-6-CARD_POWERED_ON: Card Linecard11 has been powered on. model: 7500R-36Q-LC rev: 02.01 serial number: JPE16204252
! LC initialization
Apr 19 11:38:40 R1 SandFap: %SAND-6-INIT_SUCCEEDED: Initialization of Linecard12 switch asics succeeded.
....
Apr 19 11:43:12 R1 SandFap: %SAND-6-INIT_SUCCEEDED: Initialization of Linecard13 switch asics succeeded.
! interfaces up
Apr 19 11:43:01 R1 Ebra: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet12/26/3 (mlag.207_leaf_et9/3_et10/3=>et12/17/3), changed state to up
....
Apr 19 11:46:33 R1 Ebra: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet14/34/1 (peerLink=>mlagSec.et14/34/1.100g), changed state to up

This is the messages of a 7512N with 8 J/J+ LCs. So you can see, 13 mins after MLAG agent up, the last peer-link interface is up. Given 5 mins to allow IGP/iBGP session and corresponding hw programming finished, a 20-min (1200 sec) reload-delay timer should be safe with an additional 2-minute buffer. 

And there are 2 timers:
  • MLAG reload-delay timer: this is for all MLAG port-channel links. It can be changed by CLI - "reload-delay mlag <seconds>"
  • Non-MLAG reload-delay timer: most of the time, they are the timeout values for L3 uplinks. And can be modified by CLI - "reload-delay non-mlag <seconds>"
Before discussing how to tune them, let me give a good example to help your understandings:
  • Imagine there is a house which has a front door (non-MLAG/uplinks to go out) and a back door (MLAG interfaces/to reach hosts/tenants)
  • And there is a side to your neighbor (MLAG peer), which shares the same tenants/hosts. 
  • So the first thing to do is to communicate with your neighbor to have all the address information, to know who is where. 
  • With all the knowledge, open which door first?
    • First of all, the side door is opened first. (so peer-link doesn't have reload-delay and a BGP/IGP peering is required)
    • During the gap of the front and back door, the traffic will not be lost since there is a side door to exit. 
    • The opening of the back door will have half south-north traffic in, and will go thru the side door. Similar to front door, half north-south traffic. 
    • Most of the time, the MLAG interfaces are facing servers which means south-north traffic is much higher than the opposite traffic. 
  • So it is preferable to configure non-mlag reload delay <= mlag timer. 
  • The ONLY exception is, if "reload-delay mode lacp-standby" is enabled, non-mlag timer > mlag timer. This feature keeps the LACP interfaces up to speed up hardware programming. So if the north-south coming first, the router has to drop them because the MLAG port-channels are not really ready. 

Arista EOS: sand hardware-only LAG

CLI: 
platform sand lag hardware-only

目的:
  • 一般情况下,1-port LAG是software LAG;
  • 如果增加一个port,变成2-port LAG,就会变成hardware LAG,会有traffic interruption,特别是interface flap! 
  • 什么意思啊?很简单,如果是2-port LAG,断了一根线,变成1-port/software LAG,有Traffic impact. 如果这根线在Flapping,traffic就一直受影响
  • 所以索性一上来就是hw-LAG
  • 唯一的影响是,占有的资源(啥资源啊?不知道,LAG table?)会多一些

Arista EOS Memory Util%

https://eos.arista.com/introduction-to-managing-eos-devices-memory-utilisation/

经常Arista Switch会显示Mem Util% at 70%,特别是低端的Switch。其实没有用到,而是Linux Caching System把他们Reserve到Cache和Buffer里面。下面红字的都是Free Mem

------------- show processes top memory once -------------

top - 01:14:08 up 21 days,  1:18,  1 user,  load average: 0.62, 0.39, 0.41
Tasks: 310 total,   1 running, 252 sleeping,   0 stopped,  57 zombie
%Cpu(s): 10.3 us,  2.0 sy,  0.0 ni, 87.3 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st
KiB Mem:   3900776 total,  3730136 used,   170640 free,   201908 buffers
KiB Swap:        0 total,        0 used,        0 free,  2034600 cached

ckp348.jhm.ebgp.profB.WORKING.05:46:31#sh proc top mem once
top - 05:46:40 up 25 days,  7:21,  2 users,  load average: 0.42, 0.25, 0.24
Tasks: 289 total,   2 running, 287 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.4 us,  0.7 sy,  0.0 ni, 94.1 id,  0.0 wa,  0.2 hi,  0.5 si,  0.0 st
KiB Mem:   7931828 total,  4395336 used,  3536492 free,   218976 buffers
KiB Swap:        0 total,        0 used,        0 free,  2371220 cached

Arista optics with DOM support

DOM: 
  • Digital Optical Monitoring. 
  • Per industry standard - SFF-8742, and it gives end users the ability to monitor real-time parameters of SFP, such as in/output power, temperature, laser current and supply voltage
  • 2 Arista optical parts don't support full DOM:
    • 40G AOC doesn't support Tx Power
    • 10G AOC doesn't support DOM
  • EOS CLIs:
    • show int eth22/1 transceiver [detail]
  • https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html


upp106#show inv | grep SWDM

  22   Arista Networks  QSFP-100G-SWDM4  ATH173200003     20

upp106#show int eth 22/1 transceiver
If device is externally calibrated, only calibrated values are printed.
N/A: not applicable, Tx: transmit, Rx: receive.
mA: milliamperes, dBm: decibels (milliwatts).
                               Bias      Optical   Optical
          Temp       Voltage   Current   Tx Power  Rx Power
Port      (Celsius)  (Volts)   (mA)      (dBm)     (dBm)     Last Update
-----     ---------  --------  --------  --------  --------  -------------------
Et22/1     33.00      3.30      7.69     1.32      0.44      0:00:01 ago


upp106#show int eth 22/1 transceiver de
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, +  : high warning, -  : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.
                         High Alarm  High Warn   Low Alarm   Low Warn
           Temperature   Threshold   Threshold   Threshold   Threshold
Port       (Celsius)     (Celsius)   (Celsius)   (Celsius)   (Celsius)
-------    ------------  ----------  ----------  ----------  ----------
Et22/1     33.00         75.00       70.00       -5.00       0.00
                         High Alarm  High Warn   Low Alarm   Low Warn
           Voltage       Threshold   Threshold   Threshold   Threshold
Port       (Volts)       (Volts)     (Volts)     (Volts)     (Volts)
-------    ------------  ----------  ----------  ----------  ----------
Et22/1     3.30          3.60        3.47        2.97        3.10
                         High Alarm  High Warn   Low Alarm   Low Warn
           Current       Threshold   Threshold   Threshold   Threshold
Port       (mA)          (mA)        (mA)        (mA)        (mA)
-------    ------------  ----------  ----------  ----------  ----------
Et22/1     7.71          11.00       10.00       2.00        3.00
                         High Alarm  High Warn   Low Alarm   Low Warn
           Tx Power      Threshold   Threshold   Threshold   Threshold
Port       (dBm)         (dBm)       (dBm)       (dBm)       (dBm)
-------    ------------  ----------  ----------  ----------  ----------
Et22/1     1.33          6.00        5.00        -7.00       -6.00
                         High Alarm  High Warn   Low Alarm   Low Warn
           Rx Power      Threshold   Threshold   Threshold   Threshold
Port       (dBm)         (dBm)       (dBm)       (dBm)       (dBm)
-------    ------------  ----------  ----------  ----------  ----------
Et22/1     0.44          5.50        4.50        -16.00      -13.00

bn303.jhm.mlagA.profB.WORKING.17:56:54#sh int status  | grep 9/1/1
Et9/1/1        eth.mlag1601::2x25g::upp223_et1/1_et17/1<-->bn302|303_et9/1/1                 connected    in Po1601 full   25G    100GBASE-AR4

bn303.jhm.mlagA.profB.WORKING.17:56:39#sh int e9/1/1 transceiver
If device is externally calibrated, only calibrated values are printed.
N/A: not applicable, Tx: transmit, Rx: receive.
mA: milliamperes, dBm: decibels (milliwatts).
                               Bias      Optical   Optical
          Temp       Voltage   Current   Tx Power  Rx Power
Port      (Celsius)  (Volts)   (mA)      (dBm)     (dBm)     Last Update
-----     ---------  --------  --------  --------  --------  -------------------
Et9/1/1    31.11      3.23      7.08     0.36      -2.36     0:00:03 ago
bn303.jhm.mlagA.profB.WORKING.17:56:46#sh int e9/1/1 transceiver detail
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, +  : high warning, -  : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.
                         High Alarm  High Warn   Low Alarm   Low Warn
           Temperature   Threshold   Threshold   Threshold   Threshold
Port       (Celsius)     (Celsius)   (Celsius)   (Celsius)   (Celsius)
-------    ------------  ----------  ----------  ----------  ----------
Et9/1/1    31.53         80.00       75.00       -10.00      -5.00
                         High Alarm  High Warn   Low Alarm   Low Warn
           Voltage       Threshold   Threshold   Threshold   Threshold
Port       (Volts)       (Volts)     (Volts)     (Volts)     (Volts)
-------    ------------  ----------  ----------  ----------  ----------
Et9/1/1    3.23          3.63        3.46        2.97        3.14
                         High Alarm  High Warn   Low Alarm   Low Warn
           Current       Threshold   Threshold   Threshold   Threshold
Port       (mA)          (mA)        (mA)        (mA)        (mA)
-------    ------------  ----------  ----------  ----------  ----------
Et9/1/1    7.08          12.00       10.00       3.00        4.50
                         High Alarm  High Warn   Low Alarm   Low Warn
           Tx Power      Threshold   Threshold   Threshold   Threshold
Port       (dBm)         (dBm)       (dBm)       (dBm)       (dBm)
-------    ------------  ----------  ----------  ----------  ----------
Et9/1/1    0.36          5.40        2.40        -11.40      -8.40
                         High Alarm  High Warn   Low Alarm   Low Warn
           Rx Power      Threshold   Threshold   Threshold   Threshold
Port       (dBm)         (dBm)       (dBm)       (dBm)       (dBm)
-------    ------------  ----------  ----------  ----------  ----------
Et9/1/1    -2.39         5.40        2.40        -13.30      -10.30

Arista EOS ASU and ASU2

ASU = Accelerated Software Upgrade
  • EOS Docu
  • 就是在Control Plane booting的时候,Hardware继续Fwd Traffic based on old state
  • 等Control Plane is up and running, refresh hw. 所以Traffic disruption大概是30 seconds. 
  • CLI: reload fast-boot
  • 还有一个ASU+
ASU2
  • 没有Document ?
  • 就是加上Protocol GR,例如BGP, OSPF
  • Traffic disruption is about 50ms
  • CLI: reload hitless

Arista EOS PBR (1)

  • PBR = Policy Based Routing - 就是不走IP routing path,而是policy defined path
  • Arista EOS PBR configuration和Cisco有点不一样, Arista允许 Raw match
  • Cisco:
    • access-list 1 perm ip 1.1.1.0 0.0.0.255
    • route-map pbrRtmap perm 10
    • match ip address 1
    • set ip next-hop 2.2.2.2
    • Arista:
      • policy-map type pbr pbrPolicy
      • 10 match ip 1.1.1.0/24 any set nexthop recursive 2.2.2.2
      • Sand Platform实施有点特别:
        • 如果是routed/LAG上面配置, 只要一个set of TCAM,而且只在Faps that host this routed/LAG ports. 和Port ACL类似
        • 如果是VLAN/SVI上配置,每个SVI都有一个copy of TACM rules,而且是在所有Faps。
        • 所以routed/LAG scale更好
      • set nexthop recursive是为了ECMP
      • 如果no set = no action = bypass pbr and move to next stage. 
      • 如果不satisfy 如何match,就去routing
      • 如果NH不能Resolved, 直接drop
        • 知道via = vlan, eth and MAC
      • Configuration:
        • class-map type pbr match-any pbrClassMap
        • 10 match ip access pbrACL1
        • policy-map type pbr pbrPolicy
        • 10 class pbrClassMap
        • set next-hop ip-address <ip1> <ip2>....
        • interface e1
        • service-policy type pbr input pbrPolicy 
      • 如果policy-map applied to a Lag,但是Lag member port有policy-map,咋办?
        • member port's policy-map就作废了
      • ECMP support for PBR
        • set next-hop ip-addr nh1 nh2... nhN
        • N = max# of ECMP
        • No UCMP, 如果是同样的Via, 会被filter out
      • Reference:

      Arista 7160S-32CQ Error Msg: % Error: Cannot apply ip ACL test2 to Ethernet2/1 (Out of SW resources on Chip-0)

      Platform: Arista 7160S-32CQ, XP80, 32QSFP100
      2 large ip ACL with 1500 entries
      System is kind of strenched, and hitting limit with hash collision with mask-group
      IP Access List test2
              10 deny pim host 70.0.70.210 any
              20 deny ospf host 0.70.210.70 any
              30 deny pim host 140.140.70.210 any
              40 permit gre host 70.0.0.0 any
              50 permit ospf host 70.140.0.210 any
              60 deny icmp host 140.70.210.70 any
              70 permit igmp host 0.70.0.0 any
              80 permit vrrp host 0.0.0.210 any
              90 deny tcp host 70.70.210.70 any
              100 deny ospf host 0.210.210.140 any
              110 deny gre host 210.140.0.210 any
              120 deny ip host 0.0.210.210 any
              130 deny igmp host 140.70.140.70 any
              140 permit ip host 70.0.70.0 any

      dut23:44:51(config-if-Et2/1)#ip access-group test2 in
      % Error: Cannot apply ip ACL test2 to Ethernet2/1 (Out of SW resources on Chip-0)

      fu590.09:21:58#show platform algo chip 0 acl table usage
      Table ID Entries used
      -------- ------------
      0        7968
      1        5567
      2        6415
      3        0
      4        8010
      5        2520

      show platform algo chip 0 acl mask-group
      Initial revision
         Hw Acl ID 1
            Mask group ID Mask group descriptor                      Entries
            ------------- ------------------------------------------ -------
            2             ipv4PAcl,sip:8,dip:0,sport:0,dport:0       159
            1             ipv4PAcl,sip:8,dip:0,sport:0,dport:0,proto 1385
            3             ipv4PAcl,sip:9,dip:0,sport:0,dport:0,proto 19
      Final revision
         Hw Acl ID 1
            Mask group ID Mask group descriptor                      Entries
            ------------- ------------------------------------------ -------
            3             ipv4PAcl,sip:9,dip:0,sport:0,dport:0,proto 30480

      show platform algo chip 0 acl table la

      Final revision
         Hw Acl ID 1

            Table ID Mask group ID Mask group descriptor
            -------- ------------- ------------------------------------------
            0        3             ipv4PAcl,sip:9,dip:0,sport:0,dport:0,proto
            1        3             ipv4PAcl,sip:9,dip:0,sport:0,dport:0,proto
            2        3             ipv4PAcl,sip:9,dip:0,sport:0,dport:0,proto
            4        3             ipv4PAcl,sip:9,dip:0,sport:0,dport:0,proto
            5        3             ipv4PAcl,sip:9,dip:0,sport:0,dport:0,proto