Guest

Preview Tool

Cisco Bug: CSCuo68846 - aclqos crash on LC due to FU_MEM_hashtable_node_array_t continuous alloc

Last Modified

Jan 28, 2017

Products (1)

  • Cisco Nexus 7000 Series Switches

Known Affected Releases

6.2(6a)S17

Description (partial)

Symptom:
Following errors are logged on the switch:
%SYSMGR-SLOT10-2-SERVICE_CRASHED: Service "aclqos" (PID 12914) hasn't caught signal 6 (core will be saved).
%SYSMGR-SLOT10-5-SUBPROC_TERMINATED: "System Manager (core-client)" (PID 17560) has finished with error code SYSMGR_EXITCODE_CORE_CLIENT_ERR (11).

After the crash, monitoring memory usage on LCs, following object is showing continuous allocations:
FU_MEM_hashtable_node_array_t continuous

Module 10
4/28

module-10# sh processes memory |i aclqos
12914  247832576  0           350027776   7fe2da50/7fe2c500  aclqos

module-10# sh system internal aclqos mem-stats detail


Private Mem stats for UUID : FSM Utils(53) Max types: 82
--------------------------------------------------------------------------------
TYPE NAME                                           ALLOCS                 BYTES
                                               CURR    MAX       CURR        MAX
  13 FU_MEM_hashtable_node_array_t            71108  71108  145906852  145906852

  34 ACLQOS_MEM_parse_cntx_t                  71103  71103   18202368   18202368
  36 ACLQOS_MEM_fltrlist_t                    71103  71103   29010024   29010024
 117 ACLQOS_MEM_aclqos_fhdl_list_t            71103  71103    1990884    1990884
   
   -======================================
4/29

module-10# sh processes memory |i aclqos
12914  292724736  0           394919936   7fe2da50/7fe2c500  aclqos

module-10# sh system internal aclqos mem-stats detail 

Private Mem stats for UUID : FSM Utils(53) Max types: 82
--------------------------------------------------------------------------------
TYPE NAME                                           ALLOCS                 BYTES
                                               CURR    MAX       CURR        MAX
  13 FU_MEM_hashtable_node_array_t            86763  86763  178030912  178030912   
     
     
  34 ACLQOS_MEM_parse_cntx_t                  86758  86758   22210048   22210048
  36 ACLQOS_MEM_fltrlist_t                    86758  86758   35397264   35397264
 117 ACLQOS_MEM_aclqos_fhdl_list_t            86758  86758    2429224    2429224
     
     
   -======================================
4/29 - Evening
module-10# sh processes memory | i aclqos 
12914  302288896  0           404484096   7fe2da50/7fe2c500  aclqos


Private Mem stats for UUID : FSM Utils(53) Max types: 82
--------------------------------------------------------------------------------
TYPE NAME                                           ALLOCS                 BYTES
                                               CURR    MAX       CURR        MAX
  13 FU_MEM_hashtable_node_array_t            90114  90114  184907164  184907164     

  34 ACLQOS_MEM_parse_cntx_t                  90109  90109   23067904   23067904
  36 ACLQOS_MEM_fltrlist_t                    90109  90109   36764472   36764472
 117 ACLQOS_MEM_aclqos_fhdl_list_t            90109  90109    2523052    2523052
 
   -======================================
4/30 - Morning

module-10# sh processes memory | i aclqos 
12914  337563648  0           439758848   7fe2da50/7fe2c500  aclqos

module-10# sh system internal aclqos mem-stats detail 

Private Mem stats for UUID : FSM Utils(53) Max types: 82
--------------------------------------------------------------------------------
TYPE NAME                                           ALLOCS                 BYTES
                                               CURR    MAX       CURR        MAX

  13 FU_MEM_hashtable_node_array_t           102427 102427  210173440  210173440
  
  34 ACLQOS_MEM_parse_cntx_t                 102422 102422   26220032   26220032
  36 ACLQOS_MEM_fltrlist_t                   102422 102422   41788176   41788176  
 117 ACLQOS_MEM_aclqos_fhdl_list_t           102422 102422    2867816    2867816

Conditions:
There are 2 issue here, one is memory leaks, leaks are due to the ISSU from 5.2.x to 6.2.x this is already fixed in CSCum20932 bug . Other issue is aclqos core, the core file given for this bug is in bad state and not able to decode it. But all the conditions, logs and errors seems to be same as CSCup10049. 

The conditions for this bug is after 2-3 days of periodic stats polling there will be a aclqos core. However, this issue rarely seen on the switch.
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.

Bug Details Include

  • Full Description (including symptoms, conditions and workarounds)
  • Status
  • Severity
  • Known Fixed Releases
  • Related Community Discussions
  • Number of Related Support Cases
Bug information is viewable for customers and partners who have a service contract. Registered users can view up to 200 bugs per month without a service contract.