Showing posts with label ACL. Show all posts
Showing posts with label ACL. Show all posts

6/27/2019

Use MAC ACL to isolate the failure point

For L2 traffic, besides checking drops/discard counter, another way to isolate the failure point is to use the MAC ACL, like

mac access-list macCount
   counters per-entry
   10 permit 00:00:03:03:00:14 00:00:00:00:00:00 04:68:03:03:00:14 00:00:00:00:00:00 log
   20 permit any any log
!
interface Ethernet3/1
   switchport access vlan 3003
   mac access-group macCount in

The above MAC acl - macCount is count the number of packets with source MAC - 0000.0303.0014 and dest MAC - 0468.0303.0014. And it is applied on Eth3/1 ingress direction (egress ACL is not supported)

Router#show mac access-lists
MAC Access List macCount
        counters per-entry
        10 permit 00:00:03:03:00:14 00:00:00:00:00:00 04:68:03:03:00:14 00:00:00:00:00:00 log [match 216114288 packets, 0:00:00 ago]
        20 permit any any log

This is an Arista DCS-7280CR2A-60-F with 4.22.0F

4/30/2019

Arista 7160S-32CQ Error Msg: % Error: Cannot apply ip ACL test2 to Ethernet2/1 (Out of SW resources on Chip-0)

Platform: Arista 7160S-32CQ, XP80, 32QSFP100
2 large ip ACL with 1500 entries
System is kind of strenched, and hitting limit with hash collision with mask-group
IP Access List test2
        10 deny pim host 70.0.70.210 any
        20 deny ospf host 0.70.210.70 any
        30 deny pim host 140.140.70.210 any
        40 permit gre host 70.0.0.0 any
        50 permit ospf host 70.140.0.210 any
        60 deny icmp host 140.70.210.70 any
        70 permit igmp host 0.70.0.0 any
        80 permit vrrp host 0.0.0.210 any
        90 deny tcp host 70.70.210.70 any
        100 deny ospf host 0.210.210.140 any
        110 deny gre host 210.140.0.210 any
        120 deny ip host 0.0.210.210 any
        130 deny igmp host 140.70.140.70 any
        140 permit ip host 70.0.70.0 any

dut23:44:51(config-if-Et2/1)#ip access-group test2 in
% Error: Cannot apply ip ACL test2 to Ethernet2/1 (Out of SW resources on Chip-0)

fu590.09:21:58#show platform algo chip 0 acl table usage
Table ID Entries used
-------- ------------
0        7968
1        5567
2        6415
3        0
4        8010
5        2520

show platform algo chip 0 acl mask-group
Initial revision
   Hw Acl ID 1
      Mask group ID Mask group descriptor                      Entries
      ------------- ------------------------------------------ -------
      2             ipv4PAcl,sip:8,dip:0,sport:0,dport:0       159
      1             ipv4PAcl,sip:8,dip:0,sport:0,dport:0,proto 1385
      3             ipv4PAcl,sip:9,dip:0,sport:0,dport:0,proto 19
Final revision
   Hw Acl ID 1
      Mask group ID Mask group descriptor                      Entries
      ------------- ------------------------------------------ -------
      3             ipv4PAcl,sip:9,dip:0,sport:0,dport:0,proto 30480

show platform algo chip 0 acl table la

Final revision
   Hw Acl ID 1

      Table ID Mask group ID Mask group descriptor
      -------- ------------- ------------------------------------------
      0        3             ipv4PAcl,sip:9,dip:0,sport:0,dport:0,proto
      1        3             ipv4PAcl,sip:9,dip:0,sport:0,dport:0,proto
      2        3             ipv4PAcl,sip:9,dip:0,sport:0,dport:0,proto
      4        3             ipv4PAcl,sip:9,dip:0,sport:0,dport:0,proto
      5        3             ipv4PAcl,sip:9,dip:0,sport:0,dport:0,proto


3/21/2018

How many ACLs can fit into Arista 7508E? (1)

About this post:
In this blog, I will discuss and sample number of ACLs can be programmed on an Arista Arad-based DCS-7508E system. 

System/Interface:
7508E(s1)(config-if-Et3/1/1)#show module
Module  Ports Card Type                       Model         Serial No.
------- ----- ------------------------------- ------------- -----------
1       3     DCS-7500E-SUP Supervisor Module DCS-7500E-SUP JPE13493743
2       3     DCS-7500E-SUP Supervisor Module DCS-7500E-SUP JPE15498473
3       144   36 port 40GbE QSFP+ Linecard    7500E-36Q-LC  JPE13874943
4       144   36 port 40GbE QSFP+ Linecard    7500E-36Q-LC  JPE13493196

List of contents:
1) Overview of Arista LC/Arad/TCAM banks
2) System starts with zero configuration. 
3) I will configure v4/v6 ACL on eth3/13/1 ~ 3/18/4, which are under Arad3/2
4) And check output of "show platform arad..." to monitor the TCAM usage

1. Overview of Arista LC/Arad/TCAM banks

Arista 7500E series is an Arad based switch. Each Arista 7500E-36Q-LC has 6 Arad ASICs, so each chip stands behind 6 QSFP or 24 ethernet interfaces. For example, Arad3/2 maps to eth ports, Et3/13/1 - 3/18/4

7508E(s1)#show platform arad Arad3/2 mapping
Arad3/2 (FapId: 2  BaseSystemCoreId: 2)
            Port                SysPhyPort   Voq Core FapPort OtmPort QPairs Xlge NifPort Qsgmii  Serdes
------------------------------------------------------------------------------------------------------------------
.....
            Ethernet3/13/1             140  4064    0       2       0      8    5      20      -  (20)
            Ethernet3/13/2             141  4072    0       3       8      8    -      21      -  (21)
.....
            Ethernet3/18/4             163  4248    0      25     184      8    -      11      -  (11)

And each Arad has 14 TCAM banks, 12 (#0-11) full-size 1K (160-bit entries), 2  (#12, 13) smaller (256x160-bit entries) for system-use only

7508E(s1)(config)#show platform arad Arad3/2 tcam summary
           Tcam Allocation (Arad3/2)
   Bank                   Used By    Reserved By
---------- ------------------------- -----------
      0                   dbPdpIp              -
      0               dbPdpTunnel              -
      0                dbPdpNonIp              -
      0                  dbPdpIp6              -
      0                 dbPdpMpls              -
      1         dbIp6UcastRouting              -
      2         dbIp6UcastRouting              -
      3                    dbVTT0              -
      4          dbIpMcastRouting              -
     12                 dbSystem6              -
     12              dbMplsSystem              -
     12                  dbSystem              -
     12       dbTunnelTermination              -
     13            dbEgressSystem              -

Each bank can hold 1024 TCAM lines. So per Arad can hold total ~12K ACL rule lines in theory. BUT....from above output of "tcam summary", 5 banks (0 ~ 4) are used by Pdp, ipv6 routing, VTT (vlan translation t?) and ipv4 multicast. 

In the Bank 0, there is some pdp** rules enabled, which are the default port-base CoPP policy map. And the bank usage is exclusive, so 2 types of TCAM applications can't share bank. Even the PDP is quite small, we can't squeeze other ACLs into this bank.  

7508E(s1)#show platform arad Arad3/2 acl tcam summary
The total number of TCAM lines per Arad bank is 1024
========================================================
Arad3/2:
========================================================
   Bank   Used           Used %          Used By
      0     74                7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls
Total Number of TCAM lines used is: 74

After all of these, there is slightly less than 7K lines left. But there is several questions need to be addressed before answering the question in the subject line. 
1) Are these ACL rule lines be shared by multiple ports? ethernet or port-ch?
2) How many TCAM lines are consume for v4, v6, MAC, L4 rule?
3) How about the ACL under VLAN(SVI) interfaces?

I will figure them out in later parts. 

Step 1: configure ipv4 ACL with 1 rule on 1 eth/routed port

7508E(s1)(config)#ip access-list v4-Acl-1-rule
7508E(s1)(config-acl-v4-Acl-1-rule)#permit ip 192.168.1.0/24 any
7508E(s1)(config-acl-v4-Acl-1-rule)#int et3/13/1
7508E(s1)(config-if-Et3/13/1)#ip access-group v4-Acl-1-rule in

The new ipv4 ACL is programmed into TCAM bank 5. (bank 0 - 4 are already used)

7508E(s1)#sh platform arad Arad3/2 tcam summary
           Tcam Allocation (Arad3/2)
   Bank                   Used By    Reserved By
---------- ------------------------- -----------
      0                   dbPdpIp              -
....
      3                    dbVTT0              -
      4          dbIpMcastRouting              -
      5                  dbIpPacl              -
....

And it takes 2 entries. Why 2 entries? It is because there is 1 default deny any any at the end. 

7508E(s1)#sh platform arad Arad3/2 acl tcam summary
Arad3/2:
   Bank   Used           Used %          Used By
      5      2                0         IP PACLs
      0     74                7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls
Total Number of TCAM lines used is: 76

Similar contents from the o/p of "show plat arad <id> acl tcam detail"

7508E(s1)#sh platform arad Arad3/2 acl tcam detail | more
ip access-list v4-Acl-1-rule (PACL, 1 rules, 2 entries, direction in, state success)
  Fap: Arad3/2, Shared: true, Interfaces: Et3/13/1
  Bank Offset Entries
     5      0       2

Step 2: add one more line to same ipv4 ACL 

7508E(s1)(config)#ip access-list v4-Acl-1-rule
7508E(s1)(config-acl-v4-Acl-1-rule)#perm ip 10.0.0.0/8 any
7508E(s1)(config-acl-v4-Acl-1-rule)#end
7508E(s1)#sh platform arad Arad3/2 acl tcam summary
Arad3/2:
   Bank   Used           Used %          Used By
      5      3                0         IP PACLs
      0     74                7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls
Total Number of TCAM lines used is: 77

7508E(s1)#sh ip access-lists v4-Acl-1-rule
IP Access List v4-Acl-1-rule
        10 permit ip 192.168.1.0/24 any
        20 permit ip 10.0.0.0/8 any

So, for the ipv4 ACL, it takes rule + 1 TCAM lines

Step 3: apply same ipv4 ACL to another eth port under same ASIC

7508E(s1)#sh ip access-lists v4-Acl-1-rule summary
IPV4 ACL v4-Acl-1-rule
        Total rules configured: 2
        Configured on Ingress: Et3/13/1
        Active on     Ingress: Et3/13/1
7508E(s1)(config)#int et3/13/2

7508E(s1)(config-if-Et3/13/2)#ip access-group v4-Acl-1-rule in

7508E(s1)#sh ip access-lists v4-Acl-1-rule summary
IPV4 ACL v4-Acl-1-rule
        Total rules configured: 2
        Configured on Ingress: Et3/13/1-2
        Active on     Ingress: Et3/13/1-2

7508E(s1)#sh platform arad Arad3/2 acl tcam detail | more
ip access-list v4-Acl-1-rule (PACL, 2 rules, 3 entries, direction in, state success)
  Fap: Arad3/2, Shared: true, Interfaces: Et3/13/1, Et3/13/2
  Bank Offset Entries

     5      0       3

From the above output, we can see if the same ACL applied to other eth ports under same Asic, the rule entries are shared. 

Step 4: configure a 1-line ipv6 ACL 

7508E(s1)(config)#ipv6 access-list v6-Acl-1
7508E(s1)(config-ipv6-acl-v6-Acl-1)#perm ipv6 2000::192:168:1/120 any
7508E(s1)(config-ipv6-acl-v6-Acl-1)#int e3/13/1
7508E(s1)(config-if-Et3/13/1)#ipv6 en
7508E(s1)(config-if-Et3/13/1)#ipv6 address 2000::192:168:1:1/120
7508E(s1)(config-if-Et3/13/1)#ipv6 access-group v6-Acl-1 in
7508E(s1)(config-if-Et3/13/1)#sh ipv6 access-lists v6-Acl-1 summary
IPV6 ACL v6-Acl-1
        Total rules configured: 1
        Configured on Ingress: Et3/13/1
        Active on     Ingress: Et3/13/1

7508E(s1)(config-if-Et3/13/1)#sh platform arad Arad3/2 tcam summary
           Tcam Allocation (Arad3/2)
   Bank                   Used By    Reserved By
---------- ------------------------- -----------
....
      5                  dbIpPacl              -
      6                 dbIp6Pacl              -
      7                 dbIp6Pacl              -
....

7508E(s1)(config-if-Et3/13/1)#sh platform arad Arad3/2 acl tcam summary
Arad3/2:
   Bank   Used           Used %          Used By
      5      3                0         IP PACLs
   6, 7      6                0       IPv6 PACLs
      0     74                7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls

Total Number of TCAM lines used is: 89

Now, we know v6 ACL take more TCAM resources than v4:
1) Occupies 2 banks instead of 1 since the length of address;
2) Uses 6 lines for 1-line v6 ACL, because it needs 4 lines of ICMPv6 for neighbor discovery, plus 1 line of default deny all

Step 5: add 1 more line to v6 ACL

7508E(s1)(config)#ipv6 access-list v6-Acl-1
7508E(s1)(config-ipv6-acl-v6-Acl-1)#perm ipv6 2000::10:0:0/120 any
7508E(s1)(config-ipv6-acl-v6-Acl-1)#end
7508E(s1)#sh platform arad Arad3/2 acl tcam summary
Arad3/2:
   Bank   Used           Used %          Used By
      5      3                0         IP PACLs
   6, 7      7                0       IPv6 PACLs
      0     74                7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls
Total Number of TCAM lines used is: 91

Ok, IPv6 ACLs 
1) take 2 continuous banks
2) consume 1 line per rule; plus 5 rules for default rules

Step 6: apply same ACL to another ethernet

7508E(s1)(config)#interface Ethernet3/13/2
7508E(s1)(config-if-Et3/13/2)#ipv en
7508E(s1)(config-if-Et3/13/2)#ipv6 address 2000::192:168:2:1/120
7508E(s1)(config-if-Et3/13/2)#   ipv6 access-group v6-Acl-1 in

7508E(s1)#sh ipv6 access-lists summary
....
IPV6 ACL v6-Acl-1
        Total rules configured: 2
        Configured on Ingress: Et3/13/1-2
        Active on     Ingress: Et3/13/1-2

7508E(s1)#sh platform arad Arad3/2 acl tcam detail | more
......
ipv6 access-list v6-Acl-1 (PACL, 2 rules, 7 entries, direction in, state success)
  Fap: Arad3/2, Shared: true, Interfaces: Et3/13/1, Et3/13/2
  Bank Offset Entries
  6, 7      0       7

7508E(s1)#sh platform arad Arad3/2 acl tcam sum
Arad3/2:
      5      3                0         IP PACLs
   6, 7      7                0       IPv6 PACLs
      0     74                7 Pdp IP, Pdp Tunnel, Pdp NonIp, Pdp IPv6, Pdp Mpls

Summary
1. ACL limitation is by Arad Chip
2. Each Arad chip is equipped with 12 TCAM banks and each holds 1K lines;
3. v4 ACL takes 1 bank and v6 uses 2
4. 1 ACL rule = 1 TCAM line
5. Same ACL applied on different ports can share TCAM lines