Cisco Bug: CSCuo41883 - PSC3 Slot 3 Offline with Reason=CARD_POWER_JUMPER_PRESENT
Dec 25, 2016
- Cisco ASR 5000 Series
Known Affected Releases
Symptom: Tuesday April 15 17:16:23 UTC 2014 Slot Card Type Oper State SPOF Attach ---------- ---------------------------------------- ------------- ---- ------ 1: PSC Packet Services Card 3 Active Yes 17 - 2: PSC Packet Services Card 3 Active Yes - 34 3: PSC Packet Services Card 3 Offline - - - 2014-Apr-15+17:21:45.544 [hat 3017 error] [8/0/4438 <hatsystem:0> atsystem_fail.c:1428] [software internal system syslog] Ignoring multiple failure on card 3 cpu 0 reason CARD_UNUSABLE 2014-Apr-15+17:21:44.955 [csp 7019 critical] [8/0/4481 <cspctrl:0> spctrl_events.c:3840] [hardware internal system diagnostic] The Packet Services Card 3 with serial number SAD153303B0 in slot 3 has failed and will be brought down and kept down. (Device=CARD, Reason=CARD_POWER_JUMPER_PRESENT, Status=[BOARD:] [CPU0 HB_cpu: 00:00] [CPU1 HB_cpu: 00:00] [CPU2 HB_cpu: 00:00] [CPU3 HB_cpu: 00:00] [GPIO_IN: 00,fe,ff,ff] [GPIO_OUT: 00,ff,00,ff]) Card 1: Counters: Successful Warm Boots : 5 (last at Tuesday April 15 19:02:00 UTC 2014) Successful Cold Boots : 50 (last at Monday November 04 06:16:39 UTC 2013) Total Boot Attempts : 0 In Service Date : Tue Oct 25 12:08:27 2011 (Estimated) Status: IDEEPROM Magic Number : Good Boot Mode : Normal Card Diagnostics : Pass Current Failure : None Last Failure : Failure: Device=PAC_VSC872, Reason=SF_TOO_MANY_MONITOR_FAIL, (0x03001431) (last at Tuesday April 15 18:59:02 UTC 2014) Card Usable : Yes Conditions: Initiate PSC migration slot 3 to 4. Issue timeline: 10 AM EST - Customer noticed high pdp activation failures. They also see OSPF down in IU context. They isolated this to line card 19 which was served by PSC slot 3. They also see SCT task with high CPU. SSD also showed port 25/1 bounce at the time of issue start. 12-1 PM EST - Customer proceeded with workaround to do psc migration from slot 3 to 4. However, PSC in slot 3 was not coming up active. Three psc cards were used to replaced the original one, but still failed. 1 PM EST - Customer opened an SR with Cisco TAC for assistance. TAC collected SSDs and syslog for analysis. Customer also started to offload subs from SGSN and changing the IU flex ratio to this SGSN. We also saw that the PSC in slot 3 showed Reason=CARD_POWER_JUMPER_PRESENT. 2 PM EST - While waiting for subs to offload, we continued to troubleshoot the psc in slot 3 with no success. We also reboot SMC card 9 (stand-by)to prep it for a possible SMC card migration. SMC card 9 also showed status as "unlocked" instead of "locked". 3 PM - We also put the original PSC card removed from slot 3 back to the chassis. Subsequently, we see several PSC cards starting to reboot itself. At this point, customer shutdowned all iu interfaces toward RNC to further isolate this SGSN. 3:40 PM EST - After collecting additional debug logs and reboot preparation, the chassis was reloaded. 4 PM EST - The node came back working again. Customer migrated the traffic back. Other than ias-timer, iar-time not having the correct values, the SGSN is operating normally. 4:30 PM - Customer continues to monitor node.
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.
Bug Details Include
- Full Description (including symptoms, conditions and workarounds)
- Known Fixed Releases
- Related Community Discussions
- Number of Related Support Cases