Guest

Preview Tool

Cisco Bug: CSCuo41883 - PSC3 Slot 3 Offline with Reason=CARD_POWER_JUMPER_PRESENT

Last Modified

Dec 25, 2016

Products (1)

  • Cisco ASR 5000 Series

Known Affected Releases

14.0(38)

Description (partial)

Symptom:
Tuesday April 15 17:16:23 UTC 2014
Slot         Card Type                                Oper State     SPOF  Attach
----------  ----------------------------------------  -------------  ----  ------
1: PSC      Packet Services Card 3                   Active         Yes    17  -
2: PSC      Packet Services Card 3                   Active         Yes     - 34
3: PSC      Packet Services Card 3                   Offline        -       -  -

2014-Apr-15+17:21:45.544 [hat 3017 error] [8/0/4438 <hatsystem:0> atsystem_fail.c:1428] [software internal system syslog] Ignoring multiple failure on card 3 cpu 0 reason CARD_UNUSABLE
2014-Apr-15+17:21:44.955 [csp 7019 critical] [8/0/4481 <cspctrl:0> spctrl_events.c:3840] [hardware internal system diagnostic] The Packet Services Card 3 with serial number SAD153303B0 in slot 3 has failed and will be brought down and kept down. (Device=CARD, Reason=CARD_POWER_JUMPER_PRESENT, Status=[BOARD:] [CPU0 HB_cpu: 00:00] [CPU1 HB_cpu: 00:00] [CPU2 HB_cpu: 00:00] [CPU3 HB_cpu: 00:00] [GPIO_IN: 00,fe,ff,ff] [GPIO_OUT: 00,ff,00,ff])

Card 1:
  Counters:
    Successful Warm Boots : 5
      (last at Tuesday April 15 19:02:00 UTC 2014)
    Successful Cold Boots : 50
      (last at Monday November 04 06:16:39 UTC 2013)
    Total Boot Attempts   : 0
    In Service Date       : Tue Oct 25 12:08:27 2011 (Estimated)
  Status:
    IDEEPROM Magic Number : Good
    Boot Mode             : Normal
    Card Diagnostics      : Pass
    Current Failure       : None
    Last Failure          : Failure: Device=PAC_VSC872, Reason=SF_TOO_MANY_MONITOR_FAIL, (0x03001431)
      (last at Tuesday April 15 18:59:02 UTC 2014)
    Card Usable           : Yes

Conditions:
Initiate PSC migration slot 3 to 4.
Issue timeline:

10 AM EST - Customer noticed high pdp activation failures.  They also see OSPF down in IU context.  They isolated this to line card 19 which was served by PSC slot 3. They also see SCT task with high CPU.  SSD also showed port 25/1 bounce at the time of issue start.

12-1 PM EST - Customer proceeded with workaround to do psc migration from slot 3 to 4.  However, PSC in slot 3 was not coming up active.  Three psc cards were used to replaced the original one, but still failed.

1 PM EST - Customer opened an SR with Cisco TAC for assistance. TAC collected SSDs and syslog for analysis.  Customer also started to offload subs from SGSN and changing the IU flex ratio to this SGSN.  We also saw that the PSC in slot 3 showed Reason=CARD_POWER_JUMPER_PRESENT.

2 PM EST - While waiting for subs to offload, we continued to troubleshoot the psc in slot 3 with no success. We also reboot SMC card 9 (stand-by)to prep it for a possible SMC card migration. SMC card 9 also showed status as "unlocked" instead of "locked".

3 PM - We also put the original PSC card removed from slot 3 back to the chassis.  Subsequently, we see several PSC cards starting to reboot itself.  At this point, customer shutdowned all iu interfaces toward RNC to further isolate this SGSN.

3:40 PM EST - After collecting additional debug logs and reboot preparation, the chassis was reloaded.

4 PM EST - The node came back working again.  Customer migrated the traffic back.  Other than ias-timer, iar-time not having the correct values, the SGSN is operating normally.

4:30 PM - Customer continues to monitor node.
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.

Bug Details Include

  • Full Description (including symptoms, conditions and workarounds)
  • Status
  • Severity
  • Known Fixed Releases
  • Related Community Discussions
  • Number of Related Support Cases
Bug information is viewable for customers and partners who have a service contract. Registered users can view up to 200 bugs per month without a service contract.