Cisco Bug: CSCvt19772 - Stackwise Virtual FMAN-RP IPC channel stuck (paused)
Sep 10, 2020
- Cisco Catalyst 9500 Series Switches
Known Affected Releases
16.12.2 16.9.3 16.9.4 17.3 17.4.1
Symptom: The config download channel from active FMAN-RP to standby FMAN-FP gets stuck in "IPC log back-pressured" state and the Recv Seq has no change forever (see the sample show command output below). As a result, no more configure can be downloaded from control-plane to the standby switch HW through FMAN. This affects all features, such arp, multicast, and etc. The "sh pl so peer for sw ac r0" shows ... IPC state: IPC log back-pressured, Pending ack request ... IPC Log: Peer name: fman-log-bay0-peer1 Flags: Recovery-Complete Send Seq: 40137, Recv Seq: 40134, Msgs Sent: 0, Msgs Recovered: 0 Below is one example: 'sh pl so peer for sw ac r0' FP Peers Information: Slot: 1 Peer state: connected OM ID: 0, Download attempts: 3396926 Complete: 3286000, Yields: 0, Spurious: 2757 IPC Back-Pressure: 46051, IPC-Log Back-Pressure: 62118 Back-Pressure asserted for IPC: 46202, IPC-Log: 66356 Number of FP FMAN peer connection expected: 7 Number of FP FMAN online msg received: 1 IPC state: IPC log back-pressured, Pending ack request <== Once the problem happens, we will see this line forever (unchanged with repeated run of this show command) Config IPC Context: State: Connected, Read-selected BIPC Handle: 0x3a970528, BIPC FD: 38, Peer Context: 0x3a95ba08 Tx Packets: 11729664, Messages: 415246452, ACKs: 558383 Rx Packets: 558417, Bytes: 31271348 IPC Log: Peer name: fman-log-bay0-peer1 Flags: Recovery-Complete Send Seq: 34130, Recv Seq: 34127, Msgs Sent: 0, Msgs Recovered: 0 <== the Recv Seq number stays unchanged forever (will not changed with epeated run of this show command) 'sh pl so ipc st for sw ac r0 con' ... Server Name: fman__ripc Type: remote IPC, Port: 2004, Channel Number: 3, FD: 25 Number of connection(s): 2 accepted, 0 closed Accepted Connection List: Created: 03/19/20 16:55:08, Channel Number: 11, FD: 38 Channel information: 10.129.2.0, Client Port: 54142, Remote Location: 1:F0 Sent: 11729664 Buffers, 415246452 Msgs, 68988566779 Bytes, 46194 Partial Sent, 0 Err 46203 Back-pressure, 56/8200/22 (min/max/avg) Bytes per buffer <== this Sent Line stays unchanged forever (will not changed with epeated run of this show command) Received: 558417 Msgs, 35738684 Bytes, 0 Err Created: 03/19/20 16:55:08, Channel Number: 18, FD: 49 Channel information: 10.130.2.0, Client Port: 56108, Remote Location: 2:F0 Sent: 86740859 Buffers, 1805382680 Msgs, 300620491921 Bytes, 0 Partial Sent, 0 Err 0 Back-pressure, 56/8200/49 (min/max/avg) Bytes per buffer Received: 4130520 Msgs, 264353276 Bytes, 0 Err Conditions: The issue can happen when the config download channel from active FMAN-RP to standby FMAN-FP frequently experience back-pressure for sending out messages.
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.
Bug Details Include
- Full Description (including symptoms, conditions and workarounds)
- Known Fixed Releases
- Related Community Discussions
- Number of Related Support Cases