Guest

Preview Tool

Cisco Bug: CSCvs53081 - Satellite disconnects might be caused by pooling ciscoNetsyncMIB with high number of sat.

Last Modified

Jul 01, 2020

Products (1)

  • Cisco ASR 9000 Series Aggregation Services Routers

Known Affected Releases

5.3.4.BASE

Description (partial)

Symptom:
[1] Satellites disconnect reason on TCP Congestion:

*Nov 26 08:57:03.557: SAT9k: [line:652] CTRL state : Listen [host_id : 0] [Host ip : 192.168.0.1] [Sat ip : 192.168.0.2] [cp_vlan : 0] [return_type : 6] [cause: SAT9k_TCP_QUEUE_CONGESTION] 
 *Nov 26 08:57:03.557: SAT9k: [line:945] CTRL state : Listen [host_id : 0] [Host ip : 192.168.0.1] [Sat ip : 192.168.0.2] [cp_vlan : 0] [return_type : 0] [cause: SAT9k_TCP_INIT] 
 *Nov 26 08:57:16.425: SAT9k: [line:239] CTRL state : Connected [host_id : 0] [Host ip : 192.168.0.1] [Sat ip : 192.168.0.2] [cp_vlan : 0] [return_type : 0] [cause: SAT9k_TCP_UP] 
 *Nov 26 08:58:15.505: SAT9k: [line:652] CTRL state : Listen [host_id : 0] [Host ip : 192.168.0.1] [Sat ip : 192.168.0.2] [cp_vlan : 0] [return_type : 6] [cause: SAT9k_TCP_QUEUE_CONGESTION] 
 *Nov 26 08:58:15.509: SAT9k: [line:945] CTRL state : Listen [host_id : 0] [Host ip : 192.168.0.1] [Sat ip : 192.168.0.2] [cp_vlan : 0] [return_type : 0] [cause: SAT9k_TCP_INIT] 
 *Nov 26 08:58:26.037: SAT9k: [line:239] CTRL state : Connected [host_id : 0] [Host ip : 192.168.0.1] [Sat ip : 192.168.0.2] [cp_vlan : 0] [return_type : 0] [cause: SAT9k_TCP_UP] 

[2] Host TCP has evidences on received buffer full < application unable to read TCP buffer quickly

177342479# t16 Failed at queuing packet to socket's rcv buffer for pcb 0x1125aeb8 (so: 0x112b7bf0), socket type 1, rcv_buffer_full ts is updated to 37763953
Dec  3 17:25:02.749 tcp/error 0/RSP0/CPU0 163899331# t13 Failed at queuing packet to socket's rcv buffer for pcb 0x111e0cc0 (so: 0x1121083c), socket type 1, rcv_buffer_full ts is updated to 37763953
Dec  3 17:25:03.926 tcp/error 0/RSP0/CPU0 163820495# t14 Failed at queuing packet to socket's rcv buffer for pcb 0x112946d8 (so: 0x1122a29c), socket type 1, rcv_buffer_full ts is updated to 37763954
Dec  3 17:25:04.065 tcp/error 0/RSP0/CPU0 146684961# t14 Failed at queuing packet to socket's rcv buffer for pcb 0x1118ba48 (so: 0x111dfc8c), socket type 1, rcv_buffer_full ts is updated to 37763954
Dec  3 17:25:04.678 tcp/error 0/RSP0/CPU0 189693297# t17 Failed at queuing packet to socket's rcv buffer for pcb 0x112a3d30 (so: 0x112b8040), socket type 1, rcv_buffer_full ts is updated to 37763955

[3] TCP unable to queue packets to application - PCB 0x1124cc74 owned by control TCP session host <> satellite

RP/0/RSP0/CPU0:rti-iptv-bb#sho tcp statistics  pcb 0x1124cc74 location 0/RSP0/CPU0
Fri Dec  6 16:49:21.166 MSK
..
Rcvd:   789 packets received from network
        609 packets queued to application
        29 packets failed queuing to application
        0 send-window shrink attempts by peer ignored
        1205 read operations by application
        34 times armed, 0 times unarmed, 0 times auto-armed
        Last read at: Fri Dec  6 16:49:18 2019
 
RP/0/RSP0/CPU0:rti-iptv-bb#sho tcp statistics  pcb 0x1124cc74 location 0/RSP0/CPU0
Fri Dec  6 16:49:33.304 MSK
... 
Rcvd:   933 packets received from network
        710 packets queued to application
        34 packets failed queuing to application
        0 send-window shrink attempts by peer ignored
        1414 read operations by application
        37 times armed, 0 times unarmed, 0 times auto-armed
        Last read at: Fri Dec  6 16:49:32 2019

[4] icpe_cpm constantly busy on gsp and mibd_infra waits for fscyncglobal

RP/0/RSP0/CPU0:rti-iptv-bb#show processes blocked location 0/rsp0/cpu0
Fri Dec  6 17:37:26.085 MSK
  Jid       Pid Tid            Name State   TimeInState    Blocked-on
65548     12300   1             ksh Reply 10562:11:20:0983   12298  devc-conaux
   95     36892   2      umass-enum Reply 10562:11:18:0219       1  kernel
   95     36892   6      umass-enum Reply 10562:11:15:0865  106546  io-usb
   95     36892   7      umass-enum Reply 10562:11:15:0865       1  kernel
   53    102430   2         attachd Reply 10562:11:21:0134   49191  eth_server
   53    102430   3         attachd Reply 10562:11:21:0133   16399  mqueue
   52    102446   2   attach_server Reply 10562:11:21:0096   16399  mqueue
65587    106547   2      devb-umass Reply    0:00:00:0112  106546  io-usb
  446    217179   1     tftp_server Reply 10562:08:59:0319   16399  mqueue
  322    217197   1      m_shelfmgr Reply    0:00:00:0000  213073  node 1/RSP0/CPU0 shelfmgr
  209    221306   8          envmon Mutex    0:00:00:0848  221306-01 #1
  239    245954   1     fsyncglobal Reply    0:00:03:0034  217182  sysdb_mc            
  317    557300   2         lpts_fm Reply    0:00:00:0052  233616  lpts_pa
1106 351871221   1        icpe_cpm Reply    0:00:00:0000  217168  gsp               <<<<
65830 430354726   1            exec Reply    0:00:00:0057       1  kernel
1193 1003643186  16       l2vpn_mgr Reply 2197:09:38:0885  581985  lspv_server
1192    581939  14        mpls_ldp Reply 4085:43:37:0656  581985  lspv_server
65890 441217378   1            more Reply    0:00:00:0012   16397  pipe
1059    581991  12             bgp Reply 10562:06:41:0973  581985  lspv_server
65898 441217386   1  show_processes Reply    0:00:00:0000       1  kernel
  329 1003655532   8      mibd_infra Reply    0:00:03:0059  245954  fsyncglobal        <<<<
1057    598393  18      te_control Reply 10562:06:22:0941  581985  lspv_server

[5] Within freq. sync. trace we can see continuous interface status requests

Dec 10 12:04:53.240 fsync/fast 0/0/CPU0 15624# t1  SAT: Successfully sent 164 messages of type 16
Dec 10 12:04:59.022 fsync/fast 0/0/CPU0 184792# t1  SAT: Successfully sent 123 messages of type 16
Dec 10 12:05:05.034 fsync/fast 0/0/CPU0 25057# t1  SAT: Successfully sent 123 messages of type 16
Dec 10 12:05:11.045 fsync/fast 0/0/CPU0 178489# t1  SAT: Successfully sent 123 messages of type 16
Dec 10 12:05:17.056 fsync/fast 0/0/CPU0 160626# t1  SAT: Successfully sent 123 messages of type 16
Dec 10 12:05:23.067 fsync/fast 0/0/CPU0 186897# t1  SAT: Successfully sent 123 messages of type 16
Dec 10 12:05:29.078 fsync/fast 0/0/CPU0 191104# t1  SAT: Successfully sent 123 messages of type 16

[6] High amount of SyncE messages visible on satellite SDAC control statistics 

------------------ show sdac control feature-channel ------------------
. . . .
SYNCE
      -----
         Channel status: Host 0: Open
         Host <-> SDACMessages sent: 240879, received: 344792793337573650  <<<<<<
         SDAC <-> PD Messages sent: 0, received: 0

Conditions:
Multiply satellites connected to the same LC and no freq. sync. configured at all in the system.
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.

Bug Details Include

  • Full Description (including symptoms, conditions and workarounds)
  • Status
  • Severity
  • Known Fixed Releases
  • Related Community Discussions
  • Number of Related Support Cases
Bug information is viewable for customers and partners who have a service contract. Registered users can view up to 200 bugs per month without a service contract.