Showing posts with label TCAM. Show all posts
Showing posts with label TCAM. Show all posts

12/28/2019

%SAND-6-VXLAN_ROUTE_IGNORED error message

If you see the syslog error message - %SAND-6-VXLAN_ROUTE_IGNORED like below (on arista 7280E/R series) , that's because your hardware TCAM profile is not correct. 

Dec 28 16:02:16 eVtep2 SandL3Unicast: %SAND-6-VXLAN_ROUTE_IGNORED: VXLAN routes cannot be programmed in the data path with the current TCAM profile

Just change the configuration like:

hardware tcam profile vxlan-routing

8/28/2018

Arista EOS: %QOS-3-POLICY_HW_RESOURCE_FULL

When applying Qos policy under port-channel, the system doesn't accept it. 

wa463.bug228215.16:12:21(config-if-Po20)#service-policy type qos input SPTest
% Error: Cannot apply service-policy to Port-Channel20 ()

And show logg displays an error msg:
Mar  8 16:01:25 wa463 SandAcl: %QOS-3-POLICY_HW_RESOURCE_FULL: Insufficient hardware resources to program the input policy-map SPTest.

It is because the TCAM is running out. One possible reason is PDP (per-port data policy) which uses up quite some TCAM. So try the EOS-Int image. 

wa462.bug228215.16:17:04#sh platform jericho acl tcam summary
The total number of TCAM lines per Jericho bank is 2048

========================================================
Jericho0:
========================================================
   Bank   Used           Used %          Used By
      0   2046               99         IP RACLs
      1   1554               75         IP RACLs
   2, 3   2048              100       IPv6 RACLs
   4, 5   2048              100       IPv6 RACLs
   6, 7   2048              100       IPv6 RACLs
   8, 9   2046               99       IPv6 RACLs
  10,11    210               10       IPv6 RACLs
     14     79               61 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls

Total Number of TCAM lines used is: 20479

After changing to INT image, this issue is gone. 

wa462.bug228215.16:28:09(config)#int po20
wa462.bug228215.16:28:11(config-if-Po20)#service-policy type qos input SPTest
wa462.bug228215.16:28:13(config-if-Po20)#show ver
Arista DCS-7280CR-48-F
Hardware version:    11.01
Serial number:       JPE16473148
System MAC address:  444c.a897.8c51



Software image version: 4.20.0F-INT-7767198.bloomingtonrel (engineering build)

3/27/2018

Trident Platform: route not hw programmed

If you see some v4 or v6 routes not hw programmed in the output of "show ip route", you may need to try to adjust hardware routing-table partition. 

ck494.17:37:15.(config)#platform trident routing-table partition ?
  1  16K IPv4, 6K IPv6 ( prefix len up to /64 ), 1K IPv6 ( any prefix length )
  2  16K IPv4, 4K IPv6 ( prefix len up to /64 ), 2K IPv6 ( any prefix length )
  3  16K IPv4, 2K IPv6 ( prefix len up to /64 ), 3K IPv6 ( any prefix length )
  4  16K IPv4, 0K IPv6 ( prefix len up to /64 ), 4K IPv6 ( any prefix length )

3/21/2018

How many ACLs can fit into Arista 7508E? (1)

About this post:
In this blog, I will discuss and sample number of ACLs can be programmed on an Arista Arad-based DCS-7508E system. 

System/Interface:
7508E(s1)(config-if-Et3/1/1)#show module
Module  Ports Card Type                       Model         Serial No.
------- ----- ------------------------------- ------------- -----------
1       3     DCS-7500E-SUP Supervisor Module DCS-7500E-SUP JPE13493743
2       3     DCS-7500E-SUP Supervisor Module DCS-7500E-SUP JPE15498473
3       144   36 port 40GbE QSFP+ Linecard    7500E-36Q-LC  JPE13874943
4       144   36 port 40GbE QSFP+ Linecard    7500E-36Q-LC  JPE13493196

List of contents:
1) Overview of Arista LC/Arad/TCAM banks
2) System starts with zero configuration. 
3) I will configure v4/v6 ACL on eth3/13/1 ~ 3/18/4, which are under Arad3/2
4) And check output of "show platform arad..." to monitor the TCAM usage

1. Overview of Arista LC/Arad/TCAM banks

Arista 7500E series is an Arad based switch. Each Arista 7500E-36Q-LC has 6 Arad ASICs, so each chip stands behind 6 QSFP or 24 ethernet interfaces. For example, Arad3/2 maps to eth ports, Et3/13/1 - 3/18/4

7508E(s1)#show platform arad Arad3/2 mapping
Arad3/2 (FapId: 2  BaseSystemCoreId: 2)
            Port                SysPhyPort   Voq Core FapPort OtmPort QPairs Xlge NifPort Qsgmii  Serdes
------------------------------------------------------------------------------------------------------------------
.....
            Ethernet3/13/1             140  4064    0       2       0      8    5      20      -  (20)
            Ethernet3/13/2             141  4072    0       3       8      8    -      21      -  (21)
.....
            Ethernet3/18/4             163  4248    0      25     184      8    -      11      -  (11)

And each Arad has 14 TCAM banks, 12 (#0-11) full-size 1K (160-bit entries), 2  (#12, 13) smaller (256x160-bit entries) for system-use only

7508E(s1)(config)#show platform arad Arad3/2 tcam summary
           Tcam Allocation (Arad3/2)
   Bank                   Used By    Reserved By
---------- ------------------------- -----------
      0                   dbPdpIp              -
      0               dbPdpTunnel              -
      0                dbPdpNonIp              -
      0                  dbPdpIp6              -
      0                 dbPdpMpls              -
      1         dbIp6UcastRouting              -
      2         dbIp6UcastRouting              -
      3                    dbVTT0              -
      4          dbIpMcastRouting              -
     12                 dbSystem6              -
     12              dbMplsSystem              -
     12                  dbSystem              -
     12       dbTunnelTermination              -
     13            dbEgressSystem              -

Each bank can hold 1024 TCAM lines. So per Arad can hold total ~12K ACL rule lines in theory. BUT....from above output of "tcam summary", 5 banks (0 ~ 4) are used by Pdp, ipv6 routing, VTT (vlan translation t?) and ipv4 multicast. 

In the Bank 0, there is some pdp** rules enabled, which are the default port-base CoPP policy map. And the bank usage is exclusive, so 2 types of TCAM applications can't share bank. Even the PDP is quite small, we can't squeeze other ACLs into this bank.  

7508E(s1)#show platform arad Arad3/2 acl tcam summary
The total number of TCAM lines per Arad bank is 1024
========================================================
Arad3/2:
========================================================
   Bank   Used           Used %          Used By
      0     74                7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls
Total Number of TCAM lines used is: 74

After all of these, there is slightly less than 7K lines left. But there is several questions need to be addressed before answering the question in the subject line. 
1) Are these ACL rule lines be shared by multiple ports? ethernet or port-ch?
2) How many TCAM lines are consume for v4, v6, MAC, L4 rule?
3) How about the ACL under VLAN(SVI) interfaces?

I will figure them out in later parts. 

Step 1: configure ipv4 ACL with 1 rule on 1 eth/routed port

7508E(s1)(config)#ip access-list v4-Acl-1-rule
7508E(s1)(config-acl-v4-Acl-1-rule)#permit ip 192.168.1.0/24 any
7508E(s1)(config-acl-v4-Acl-1-rule)#int et3/13/1
7508E(s1)(config-if-Et3/13/1)#ip access-group v4-Acl-1-rule in

The new ipv4 ACL is programmed into TCAM bank 5. (bank 0 - 4 are already used)

7508E(s1)#sh platform arad Arad3/2 tcam summary
           Tcam Allocation (Arad3/2)
   Bank                   Used By    Reserved By
---------- ------------------------- -----------
      0                   dbPdpIp              -
....
      3                    dbVTT0              -
      4          dbIpMcastRouting              -
      5                  dbIpPacl              -
....

And it takes 2 entries. Why 2 entries? It is because there is 1 default deny any any at the end. 

7508E(s1)#sh platform arad Arad3/2 acl tcam summary
Arad3/2:
   Bank   Used           Used %          Used By
      5      2                0         IP PACLs
      0     74                7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls
Total Number of TCAM lines used is: 76

Similar contents from the o/p of "show plat arad <id> acl tcam detail"

7508E(s1)#sh platform arad Arad3/2 acl tcam detail | more
ip access-list v4-Acl-1-rule (PACL, 1 rules, 2 entries, direction in, state success)
  Fap: Arad3/2, Shared: true, Interfaces: Et3/13/1
  Bank Offset Entries
     5      0       2

Step 2: add one more line to same ipv4 ACL 

7508E(s1)(config)#ip access-list v4-Acl-1-rule
7508E(s1)(config-acl-v4-Acl-1-rule)#perm ip 10.0.0.0/8 any
7508E(s1)(config-acl-v4-Acl-1-rule)#end
7508E(s1)#sh platform arad Arad3/2 acl tcam summary
Arad3/2:
   Bank   Used           Used %          Used By
      5      3                0         IP PACLs
      0     74                7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls
Total Number of TCAM lines used is: 77

7508E(s1)#sh ip access-lists v4-Acl-1-rule
IP Access List v4-Acl-1-rule
        10 permit ip 192.168.1.0/24 any
        20 permit ip 10.0.0.0/8 any

So, for the ipv4 ACL, it takes rule + 1 TCAM lines

Step 3: apply same ipv4 ACL to another eth port under same ASIC

7508E(s1)#sh ip access-lists v4-Acl-1-rule summary
IPV4 ACL v4-Acl-1-rule
        Total rules configured: 2
        Configured on Ingress: Et3/13/1
        Active on     Ingress: Et3/13/1
7508E(s1)(config)#int et3/13/2

7508E(s1)(config-if-Et3/13/2)#ip access-group v4-Acl-1-rule in

7508E(s1)#sh ip access-lists v4-Acl-1-rule summary
IPV4 ACL v4-Acl-1-rule
        Total rules configured: 2
        Configured on Ingress: Et3/13/1-2
        Active on     Ingress: Et3/13/1-2

7508E(s1)#sh platform arad Arad3/2 acl tcam detail | more
ip access-list v4-Acl-1-rule (PACL, 2 rules, 3 entries, direction in, state success)
  Fap: Arad3/2, Shared: true, Interfaces: Et3/13/1, Et3/13/2
  Bank Offset Entries

     5      0       3

From the above output, we can see if the same ACL applied to other eth ports under same Asic, the rule entries are shared. 

Step 4: configure a 1-line ipv6 ACL 

7508E(s1)(config)#ipv6 access-list v6-Acl-1
7508E(s1)(config-ipv6-acl-v6-Acl-1)#perm ipv6 2000::192:168:1/120 any
7508E(s1)(config-ipv6-acl-v6-Acl-1)#int e3/13/1
7508E(s1)(config-if-Et3/13/1)#ipv6 en
7508E(s1)(config-if-Et3/13/1)#ipv6 address 2000::192:168:1:1/120
7508E(s1)(config-if-Et3/13/1)#ipv6 access-group v6-Acl-1 in
7508E(s1)(config-if-Et3/13/1)#sh ipv6 access-lists v6-Acl-1 summary
IPV6 ACL v6-Acl-1
        Total rules configured: 1
        Configured on Ingress: Et3/13/1
        Active on     Ingress: Et3/13/1

7508E(s1)(config-if-Et3/13/1)#sh platform arad Arad3/2 tcam summary
           Tcam Allocation (Arad3/2)
   Bank                   Used By    Reserved By
---------- ------------------------- -----------
....
      5                  dbIpPacl              -
      6                 dbIp6Pacl              -
      7                 dbIp6Pacl              -
....

7508E(s1)(config-if-Et3/13/1)#sh platform arad Arad3/2 acl tcam summary
Arad3/2:
   Bank   Used           Used %          Used By
      5      3                0         IP PACLs
   6, 7      6                0       IPv6 PACLs
      0     74                7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls

Total Number of TCAM lines used is: 89

Now, we know v6 ACL take more TCAM resources than v4:
1) Occupies 2 banks instead of 1 since the length of address;
2) Uses 6 lines for 1-line v6 ACL, because it needs 4 lines of ICMPv6 for neighbor discovery, plus 1 line of default deny all

Step 5: add 1 more line to v6 ACL

7508E(s1)(config)#ipv6 access-list v6-Acl-1
7508E(s1)(config-ipv6-acl-v6-Acl-1)#perm ipv6 2000::10:0:0/120 any
7508E(s1)(config-ipv6-acl-v6-Acl-1)#end
7508E(s1)#sh platform arad Arad3/2 acl tcam summary
Arad3/2:
   Bank   Used           Used %          Used By
      5      3                0         IP PACLs
   6, 7      7                0       IPv6 PACLs
      0     74                7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls
Total Number of TCAM lines used is: 91

Ok, IPv6 ACLs 
1) take 2 continuous banks
2) consume 1 line per rule; plus 5 rules for default rules

Step 6: apply same ACL to another ethernet

7508E(s1)(config)#interface Ethernet3/13/2
7508E(s1)(config-if-Et3/13/2)#ipv en
7508E(s1)(config-if-Et3/13/2)#ipv6 address 2000::192:168:2:1/120
7508E(s1)(config-if-Et3/13/2)#   ipv6 access-group v6-Acl-1 in

7508E(s1)#sh ipv6 access-lists summary
....
IPV6 ACL v6-Acl-1
        Total rules configured: 2
        Configured on Ingress: Et3/13/1-2
        Active on     Ingress: Et3/13/1-2

7508E(s1)#sh platform arad Arad3/2 acl tcam detail | more
......
ipv6 access-list v6-Acl-1 (PACL, 2 rules, 7 entries, direction in, state success)
  Fap: Arad3/2, Shared: true, Interfaces: Et3/13/1, Et3/13/2
  Bank Offset Entries
  6, 7      0       7

7508E(s1)#sh platform arad Arad3/2 acl tcam sum
Arad3/2:
      5      3                0         IP PACLs
   6, 7      7                0       IPv6 PACLs
      0     74                7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls

Summary
1. ACL limitation is by Arad Chip
2. Each Arad chip is equipped with 12 TCAM banks and each holds 1K lines;
3. v4 ACL takes 1 bank and v6 uses 2
4. 1 ACL rule = 1 TCAM line
5. Same ACL applied on different ports can share TCAM lines