Guest

Preview Tool

Cisco Bug: CSCsz21900 - E5 LC crashed after RPFO with %L2-WAHOO-6-INTERRUPT_INFO : RX PSE: FFCR

Last Modified

Apr 03, 2018

Products (1)

  • Cisco 12000 Series Routers

Known Affected Releases

3.9.0.BASE

Description (partial)

Symptom:

3.9 GSR FIT Biscuit scale profile

Made a RPFO, two E5 LCs crashed with the following messages.

LC 10 and LC 11 are crashed. LC wahoo dumps are saved in the below location.

sjc-cde-027:5> pwd
/auto/tftpboot-projects/feiyang
sjc-cde-027:6> ls wahoo*
wahoo_phb_dump_slot_6   
wahoo_phb_dump_slot_7 
wahoo_phb_dump_slot_8
wahoo_phb_dump_slot_9
wahoo_phb_dump_slot_10  
wahoo_phb_dump_slot_11  
wahoo_phb_dump_slot_13 

sjc-cde-027:3> pwd
/auto/tftpboot-projects/feiyang
sjc-cde-027:4> ls
0420-rpfo-lccrash-shlog.log <- show log



RP/0/0/CPU0:BISCUIT-BR#LC/0/10/CPU0:Apr 20 12:24:44.961 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: Interrupt detected - PIPE0: 0x1401 
LC/0/10/CPU0:Apr 20 12:24:45.350 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: CPU_IF100_INT PIPE0: 0x1040 
LC/0/10/CPU0:Apr 20 12:24:45.354 : hfa_main[56]: %L2-WAHOO-6-INTERRUPT_INFO :  RX PSE: FFCRAM Invalid Address 0x114b2b8 detected on PIPE0 
LC/0/10/CPU0:Apr 20 12:24:45.746 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: FFCRAM Invalid Address detected on PIPE0: 0x114b2b8 
LC/0/10/CPU0:Apr 20 12:24:46.202 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: FFCRAM invalid Address error Agent Register value on PIPE0: 0x34 
LC/0/10/CPU0:Apr 20 12:24:46.524 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: PPC_INT PIPE0: 0x4100020 
LC/0/10/CPU0:Apr 20 12:24:46.901 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: PPC_PAK_AVL PIPE0: 0xb1f7b 
LC/0/10/CPU0:Apr 20 12:24:47.088 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: PPC_PAK_END PIPE0: 0xfffd 
LC/0/10/CPU0:Apr 20 12:24:47.180 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: PPC_PIPE_CTL PIPE0: 0x4000226 
LC/0/10/CPU0:Apr 20 12:24:47.316 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: Interrupt detected - PIPE1: 0x1 

 
LC/0/11/CPU0:Apr 20 12:24:46.567 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: CPU_IF100_INT PIPE0: 0x1040 
LC/0/11/CPU0:Apr 20 12:24:46.651 : hfa_main[56]: %L2-WAHOO-6-INTERRUPT_INFO :  RX PSE: FFCRAM Invalid Address 0x1143a80 detected on PIPE0 
LC/0/11/CPU0:Apr 20 12:24:46.853 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: FFCRAM Invalid Address detected on PIPE0: 0x1143a80 
LC/0/11/CPU0:Apr 20 12:24:46.982 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: FFCRAM invalid Address error Agent Register value on PIPE0: 0x4 
LC/0/11/CPU0:Apr 20 12:24:47.144 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: Interrupt detected - PIPE1: 0x1001 
LC/0/11/CPU0:Apr 20 12:24:47.311 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: CPU_IF100_INT PIPE1: 0x1040 
LC/0/11/CPU0:Apr 20 12:24:47.444 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: PPC_INT PIPE1: 0x4000020 
LC/0/11/CPU0:Apr 20 12:24:47.632 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: PPC_PAK_AVL PIPE1: 0xf9f7b 
LC/0/11/CPU0:Apr 20 12:24:47.828 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: PPC_PAK_END PIPE1: 0x7ffd 
LC/0/11/CPU0:Apr 20 12:24:48.019 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE: PPC_PIPE_CTL PIPE1: 0x4000226 

LC/0/10/CPU0:Apr 20 12:24:58.798 : hfa_main[56]: %L2-WAHOO-7-REG_DUMP : RX WAHOO:: pipe0_interrupt_reg 1001 pipe1_interrupt_reg 1 pipe2_interrupt_reg 1 pipe3_interrupt_reg 1 mask_of_int_enable 5fffffff cpu_latched_error_addr_reg 1c000054 cpu100_control_reg faf5ff cpu_150_control_reg 17d85 cpu_latched_error_data_hi_reg 30427b28 cpu_latched_error_data_lo_reg 1332200 cpu_latched_error_parity_reg 80403809 pf_error_reg 0 p0_fs_error_reg 0 p1_fs_error_reg 0 p2_fs_error_reg 0 p3_fs_error_reg 0 p0_imp_int_status_reg 0 p1_imp_int_status_reg 0 p2_imp_int_status_reg 0 p3_imp_int_status 0 p0_imp_pc 926 p1_imp_pc 926 p2_imp_pc 926 p3_imp_pc 926 p0_prep_int_status 0 p1_prep_int_status 0 p2_prep_int_status 0 p3_prep_int_status 0 p0_prep_pc 6f1 p1_prep_pc 6f1 p2_prep_pc 6f1 p3_prep_pc 6f1 p0_pc1_status 0 p1_pc1_status 0 p2_pc1_stats 0 p3_pc1_status 0 p0_tlu_int_status 0 p1_tlu_int_status 0 p2_tlu_int_status 0 p3_tlu_int_status 0 p0_mip_int_status 0 p1_mip_int_status 0 p2_mip_int_status 0 p3_mip_int_status 0 p0_mip_pc 11df p1_mip_pc 11df p2_mip_pc 11df p3_mip_
LC/0/10/CPU0:Apr 20 12:24:58.799 : hfa_main[56]: %L2-WAHOO-6-INTERRUPT_INFO : 2 Apr 20 12:24:45 RX PSE: FFCRAM Invalid Address 0x114b2b8 detected on PIPE0 
LC/0/10/CPU0:Apr 20 12:24:58.799 : hfa_main[56]: %L2-WAHOO-6-INTERRUPT_INFO : 1 Apr 20 00:15:16 RX PSE: FTCAM Internal Parity Error detected on PIPE0 0x4 
LC/0/10/CPU0:Apr 20 12:24:58.866 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE:  PIPE0 TRACE_PHB : 0x0 
LC/0/10/CPU0:Apr 20 12:24:58.986 : hfa_main[56]: %L2-WAHOO-7-PHB_DUMP :  0x 0: 0x42080048 0x00005d63 0x00000000 0x0000ffff  : pkg/bin/hfa_main : (PID=28698) :  -Traceback= 785c30fc 785c90fc 785c968c 8020a30 7826f31c
LC/0/10/CPU0:Apr 20 12:25:06.485 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE:  PIPE0 TRACE_PHB : 0x1 

LC/0/11/CPU0:Apr 20 12:25:02.520 : hfa_main[56]: %L2-WAHOO-7-REG_DUMP : RX WAHOO:: pipe0_interrupt_reg 1 pipe1_interrupt_reg 1001 pipe2_interrupt_reg 1 pipe3_interrupt_reg 1 mask_of_int_enable 5fffffff cpu_latched_error_addr_reg ff800000 cpu100_control_reg faf5ff cpu_150_control_reg 17d85 cpu_latched_error_data_hi_reg 29611a32 cpu_latched_error_data_lo_reg 681bca8c cpu_latched_error_parity_reg 82005100 pf_error_reg 0 p0_fs_error_reg 0 p1_fs_error_reg 0 p2_fs_error_reg 0 p3_fs_error_reg 0 p0_imp_int_status_reg 0 p1_imp_int_status_reg 0 p2_imp_int_status_reg 0 p3_imp_int_status 0 p0_imp_pc 926 p1_imp_pc 926 p2_imp_pc 926 p3_imp_pc 926 p0_prep_int_status 0 p1_prep_int_status 0 p2_prep_int_status 0 p3_prep_int_status 0 p0_prep_pc 6f1 p1_prep_pc 6f1 p2_prep_pc 6f1 p3_prep_pc 6f1 p0_pc1_status 0 p1_pc1_status 0 p2_pc1_stats 0 p3_pc1_status 0 p0_tlu_int_status 0 p1_tlu_int_status 0 p2_tlu_int_status 0 p3_tlu_int_status 0 p0_mip_int_status 0 p1_mip_int_status 0 p2_mip_int_status 0 p3_mip_int_status 0 p0_mip_pc 11df p1_mip_pc 11df p2_mip_pc 11df p3_mip

LC/0/11/CPU0:Apr 20 12:25:22.096 : hfa_main[56]: %L2-WAHOO-6-INFO_MSG :  RX PSE:  PIPE0 TRACE_PHB : 0x2 
LC/0/11/CPU0:Apr 20 12:25:22.305 : g_spa_2[157]: %L2-SPA-3-PLIM_HEARTBEAT_ERR : SPA-1XCHSTM1/OC3: bay 2 heartbeat missed, expected seq# 51935 received seq# 51934, Time since last message 38s 
LC/0/11/CPU0:Apr 20 12:25:22.879 : hfa_main[56]: %L2-WAHOO-7-PHB_DUMP :  0x 8: 0x00000000 0x0000ffff 0x022c003a 0x24710800  : pkg/bin/hfa_main : (PID=28698) :  -Traceback= 785c30fc 785c90fc 785c968c 8020a30 7826f31c
LC/0/11/CPU0:Apr 20 12:25:23.319 : g_spa_2[157]: %L2-SPA-3-PLIM_HEARTBEAT_ERR : SPA-1XCHSTM1/OC3: bay 2 heartbeat missed, expected seq# 51935 received seq# 51934, Time since last message 39s 
LC/0/11/CPU0:Apr 20 12:25:24.325 : g_spa_2[157]: %L2-SPA-3-PLIM_HEARTBEAT_ERR : SPA-1XCHSTM1/OC3: bay 2 heartbeat missed, expected seq# 51935 received seq# 51934, Time since last message 40s 
LC/0/11/CPU0:Apr 20 12:25:25.329 : g_spa_2[157]: do_nothing_plugin_f5 should be replaced by real implementation
LC/0/11/CPU0:Apr 20 12:25:25.333 : g_spa_2[157]: %L2-SPA-5-PLIM_RELOADING : Reloading SPA-1XCHSTM1/OC3 in subslot 2 after heartbeat failure ... 

 


Conditions:


3.9 FIT Biscuit Test Bed

VNC Access:

sjc-xdm-054:21/cisco123


Topology:

TGN--CE1--|
TGN--CE2--|            |--HR--TGN
          |-- BR/UUT --|
TGN--CE3--|            |--PE--TGN
TGN--CE4--|


Scale numbers:

Interface Summary 
ipv4	0 
ipv46	0 
ipv6	0 
l3vpn	4381 
l3vpn6vpe 474 
6vpe	95 
mvpn	1126 
mvpn46	4 
mvpn6	6 
Total	4950 

VRF Summary 
Total V4 VRFS	1723 (Includes Extranet and Local VRF)
Total V46 VRFS 	236 
Total V6 only VRFs 41 
Total VRFs 	2000 

MVPN Summary  
Total MVPN V4 Extranet	0 
Total V4 MVPNs + v4 Extranet 197 
Total V46 MVPNs		2 
Total V6 only MVPNS	2 
Total MVPNS		201 

Protocol Summary 
(number of interfaces between PE UUT and CEs)
static 1278 
bgp 3196 
ospf 100 
rip 200 
eigrp 120
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.

Bug Details Include

  • Full Description (including symptoms, conditions and workarounds)
  • Status
  • Severity
  • Known Fixed Releases
  • Related Community Discussions
  • Number of Related Support Cases
Bug information is viewable for customers and partners who have a service contract. Registered users can view up to 200 bugs per month without a service contract.