Guest

Preview Tool

Cisco Bug: CSCun95478 - snmpTrapHandler takes too long to reestablish connection to infra node

Last Modified

Jun 22, 2020

Products (1)

  • Headend System Releases

Known Affected Releases

dncs-6.0.0.21p2

Description (partial)

Symptom:
EC traps are not reported to ECS

Conditions:
Way to reproduce:

1. Check which infra node the EC processes are connected:
[BELFAST-ec-PRI-DNCS] lsof -Pni :5223
COMMAND     PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
java        352   dncs   16u  IPv4 0xfffffe9989147e40 0t224474  TCP 172.10.61.1:33203->192.168.53.22:5223 (ESTABLISHED)
dncsInitd   894   dncs   12u  IPv4 0xfffffe998d3ada00      0t0  TCP 172.10.61.1:33215->192.168.53.22:5223 (ESTABLISHED)
java       1984 oamadm    8u  IPv4 0xfffffe998d3ad300 0t214261  TCP 172.10.61.1:33205->192.168.53.22:5223 (ESTABLISHED)
bossServe 18657   dncs   13u  IPv4 0xfffffe998f7b3200      0t0  TCP 172.10.61.1:33199->192.168.53.22:5223 (ESTABLISHED)
rpOrch    18973   dncs   10u  IPv4 0xffffffffb9c88080      0t0  TCP 172.10.61.1:33540->192.168.53.22:5223 (ESTABLISHED)
rpOrch    18973   dncs   11u  IPv4 0xfffffe99899efe00      0t0  TCP 172.10.61.1:33201->192.168.53.22:5223 (ESTABLISHED)
snmpTrapH 20002   dncs   13u  IPv4 0xfffffe99883b00c0      0t0  TCP 172.10.61.1:36285->192.168.53.22:5223 (ESTABLISHED)

2. Disconnect all interfaces connected to infra node 192.168.53.22

3. Wait 5 minutes and notice that all the process reestablish connection to the other infra node (192.168.53.21) except for snmpTrapHandler

BELFAST-ec-PRI-DNCS] lsof -Pni :5223
COMMAND     PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
java        352   dncs   16u  IPv4 0xfffffe9988c85100  0t48734  TCP 172.10.61.1:37252->192.168.53.21:5223 (ESTABLISHED)
dncsInitd   894   dncs   12u  IPv4 0xfffffe998913e880      0t0  TCP 172.10.61.1:37266->192.168.53.21:5223 (ESTABLISHED)
java       1984 oamadm    8u  IPv4 0xfffffe998d168b00  0t46528  TCP 172.10.61.1:37254->192.168.53.21:5223 (ESTABLISHED)
bossServe 18657   dncs   13u  IPv4 0xfffffe997d7f5740      0t0  TCP 172.10.61.1:37250->192.168.53.21:5223 (ESTABLISHED)
rpOrch    18973   dncs   10u  IPv4 0xffffffffb9c88080      0t0  TCP 172.10.61.1:33540->192.168.53.22:5223 (ESTABLISHED)
rpOrch    18973   dncs   11u  IPv4 0xffffffffdce80300      0t0  TCP 172.10.61.1:37248->192.168.53.21:5223 (ESTABLISHED)
snmpTrapH 20002   dncs   13u  IPv4 0xfffffe99883b00c0      0t0  TCP 172.10.61.1:36285->192.168.53.22:5223 (ESTABLISHED)

4. Force a trap. For example stop the osm process and notice that the trap doesn't make it to ECS alarm tab
5. After many minutes snmpTrapHandler gets disconnected from the broken infra node
[BELFAST-ec-PRI-DNCS] lsof -Pni :5223
COMMAND     PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
java        352   dncs   16u  IPv4 0xfffffe9988c85100  0t60041  TCP 172.10.61.1:37252->192.168.53.21:5223 (ESTABLISHED)
dncsInitd   894   dncs   12u  IPv4 0xfffffe998913e880      0t0  TCP 172.10.61.1:37266->192.168.53.21:5223 (ESTABLISHED)
java       1984 oamadm    8u  IPv4 0xfffffe998d168b00  0t57548  TCP 172.10.61.1:37254->192.168.53.21:5223 (ESTABLISHED)
bossServe 18657   dncs   13u  IPv4 0xfffffe997d7f5740      0t0  TCP 172.10.61.1:37250->192.168.53.21:5223 (ESTABLISHED)
rpOrch    18973   dncs   10u  IPv4 0xffffffffb9c88080      0t0  TCP 172.10.61.1:33540->192.168.53.22:5223 (ESTABLISHED)
rpOrch    18973   dncs   11u  IPv4 0xffffffffdce80300      0t0  TCP 172.10.61.1:37248->192.168.53.21:5223 (ESTABLISHED)
6. Force another trap. Notice that snmpTrapHandler establish a new connection to the operational infra node. However, the trap doesn't make it to the ECS alarm console
[BELFAST-ec-PRI-DNCS] lsof -Pni :5223
COMMAND     PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
java        352   dncs   16u  IPv4 0xfffffe9988c85100  0t62597  TCP 172.10.61.1:37252->192.168.53.21:5223 (ESTABLISHED)
dncsInitd   894   dncs   12u  IPv4 0xfffffe998913e880      0t0  TCP 172.10.61.1:37266->192.168.53.21:5223 (ESTABLISHED)
java       1984 oamadm    8u  IPv4 0xfffffe998d168b00  0t59984  TCP 172.10.61.1:37254->192.168.53.21:5223 (ESTABLISHED)
bossServe 18657   dncs   13u  IPv4 0xfffffe997d7f5740      0t0  TCP 172.10.61.1:37250->192.168.53.21:5223 (ESTABLISHED)
rpOrch    18973   dncs   10u  IPv4 0xffffffffb9c88080      0t0  TCP 172.10.61.1:33540->192.168.53.22:5223 (ESTABLISHED)
rpOrch    18973   dncs   11u  IPv4 0xffffffffdce80300      0t0  TCP 172.10.61.1:37248->192.168.53.21:5223 (ESTABLISHED)
snmpTrapH 20002   dncs   13u  IPv4 0xfffffe9987b33100      0t0  TCP 172.10.61.1:37952->192.168.53.21:5223 (ESTABLISHED)

7. snmpTrapHandler gets back to normal. All the traps get forwarded to ECS
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.

Bug Details Include

  • Full Description (including symptoms, conditions and workarounds)
  • Status
  • Severity
  • Known Fixed Releases
  • Related Community Discussions
  • Number of Related Support Cases
Bug information is viewable for customers and partners who have a service contract. Registered users can view up to 200 bugs per month without a service contract.