Guest

Preview Tool

Cisco Bug: CSCvt19772 - Stackwise Virtual FMAN-RP IPC channel stuck (paused)

Last Modified

Sep 10, 2020

Products (1)

  • Cisco Catalyst 9500 Series Switches

Known Affected Releases

16.12.2 16.9.3 16.9.4 17.3 17.4.1

Description (partial)

Symptom:
The config download channel from active FMAN-RP to standby FMAN-FP gets stuck in "IPC log back-pressured" state and the Recv Seq has no change forever (see the sample show command output below). As a result, no more configure can be downloaded from control-plane to the standby switch HW through FMAN. This affects all features, such arp, multicast, and etc.

The "sh pl so peer for sw ac r0" shows
...
    IPC state: IPC log back-pressured, Pending ack request  
...
    IPC Log:
        Peer name: fman-log-bay0-peer1
        Flags: Recovery-Complete
        Send Seq: 40137, Recv Seq: 40134, Msgs Sent: 0, Msgs Recovered: 0

Below is one example:

'sh pl so peer for sw ac r0'
FP Peers Information:

  Slot: 1
  Peer state: connected
    OM ID: 0, Download attempts: 3396926
      Complete: 3286000, Yields: 0, Spurious: 2757
      IPC Back-Pressure: 46051, IPC-Log Back-Pressure: 62118
    Back-Pressure asserted for IPC: 46202, IPC-Log: 66356
    Number of FP FMAN peer connection expected: 7
    Number of FP FMAN online msg received: 1
    IPC state: IPC log back-pressured, Pending ack request  <== Once the problem happens, we will see this line forever (unchanged with repeated run of this show command)
    Config IPC Context:
      State: Connected, Read-selected
      BIPC Handle: 0x3a970528, BIPC FD: 38, Peer Context: 0x3a95ba08
      Tx Packets: 11729664, Messages: 415246452, ACKs: 558383
      Rx Packets: 558417, Bytes: 31271348

      IPC Log:
        Peer name: fman-log-bay0-peer1
        Flags: Recovery-Complete
        Send Seq: 34130, Recv Seq: 34127, Msgs Sent: 0, Msgs Recovered: 0 <== the Recv Seq number stays unchanged forever (will not changed with epeated run of this show command)

'sh pl so ipc st for sw ac r0 con'
...
Server Name: fman__ripc
Type: remote IPC, Port: 2004, Channel Number: 3, FD: 25
Number of connection(s): 2 accepted, 0 closed
Accepted Connection List:

  Created: 03/19/20 16:55:08, Channel Number: 11, FD: 38
  Channel information: 10.129.2.0, Client Port: 54142, Remote Location: 1:F0
  Sent: 11729664 Buffers, 415246452 Msgs, 68988566779 Bytes, 46194 Partial Sent, 0 Err
        46203 Back-pressure, 56/8200/22 (min/max/avg) Bytes per buffer  <== this Sent Line stays unchanged forever (will not changed with epeated run of this show command)
  Received: 558417 Msgs, 35738684 Bytes, 0 Err

  Created: 03/19/20 16:55:08, Channel Number: 18, FD: 49
  Channel information: 10.130.2.0, Client Port: 56108, Remote Location: 2:F0
  Sent: 86740859 Buffers, 1805382680 Msgs, 300620491921 Bytes, 0 Partial Sent, 0 Err
        0 Back-pressure, 56/8200/49 (min/max/avg) Bytes per buffer
  Received: 4130520 Msgs, 264353276 Bytes, 0 Err

Conditions:
The issue can happen when the config download channel from active FMAN-RP to standby FMAN-FP frequently experience back-pressure for sending out messages.
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.

Bug Details Include

  • Full Description (including symptoms, conditions and workarounds)
  • Status
  • Severity
  • Known Fixed Releases
  • Related Community Discussions
  • Number of Related Support Cases
Bug information is viewable for customers and partners who have a service contract. Registered users can view up to 200 bugs per month without a service contract.