Cisco Bug: CSCun95478 - snmpTrapHandler takes too long to reestablish connection to infra node
Last Modified
Jun 22, 2020
Products (1)
- Headend System Releases
Known Affected Releases
dncs-6.0.0.21p2
Description (partial)
Symptom: EC traps are not reported to ECS Conditions: Way to reproduce: 1. Check which infra node the EC processes are connected: [BELFAST-ec-PRI-DNCS] lsof -Pni :5223 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 352 dncs 16u IPv4 0xfffffe9989147e40 0t224474 TCP 172.10.61.1:33203->192.168.53.22:5223 (ESTABLISHED) dncsInitd 894 dncs 12u IPv4 0xfffffe998d3ada00 0t0 TCP 172.10.61.1:33215->192.168.53.22:5223 (ESTABLISHED) java 1984 oamadm 8u IPv4 0xfffffe998d3ad300 0t214261 TCP 172.10.61.1:33205->192.168.53.22:5223 (ESTABLISHED) bossServe 18657 dncs 13u IPv4 0xfffffe998f7b3200 0t0 TCP 172.10.61.1:33199->192.168.53.22:5223 (ESTABLISHED) rpOrch 18973 dncs 10u IPv4 0xffffffffb9c88080 0t0 TCP 172.10.61.1:33540->192.168.53.22:5223 (ESTABLISHED) rpOrch 18973 dncs 11u IPv4 0xfffffe99899efe00 0t0 TCP 172.10.61.1:33201->192.168.53.22:5223 (ESTABLISHED) snmpTrapH 20002 dncs 13u IPv4 0xfffffe99883b00c0 0t0 TCP 172.10.61.1:36285->192.168.53.22:5223 (ESTABLISHED) 2. Disconnect all interfaces connected to infra node 192.168.53.22 3. Wait 5 minutes and notice that all the process reestablish connection to the other infra node (192.168.53.21) except for snmpTrapHandler BELFAST-ec-PRI-DNCS] lsof -Pni :5223 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 352 dncs 16u IPv4 0xfffffe9988c85100 0t48734 TCP 172.10.61.1:37252->192.168.53.21:5223 (ESTABLISHED) dncsInitd 894 dncs 12u IPv4 0xfffffe998913e880 0t0 TCP 172.10.61.1:37266->192.168.53.21:5223 (ESTABLISHED) java 1984 oamadm 8u IPv4 0xfffffe998d168b00 0t46528 TCP 172.10.61.1:37254->192.168.53.21:5223 (ESTABLISHED) bossServe 18657 dncs 13u IPv4 0xfffffe997d7f5740 0t0 TCP 172.10.61.1:37250->192.168.53.21:5223 (ESTABLISHED) rpOrch 18973 dncs 10u IPv4 0xffffffffb9c88080 0t0 TCP 172.10.61.1:33540->192.168.53.22:5223 (ESTABLISHED) rpOrch 18973 dncs 11u IPv4 0xffffffffdce80300 0t0 TCP 172.10.61.1:37248->192.168.53.21:5223 (ESTABLISHED) snmpTrapH 20002 dncs 13u IPv4 0xfffffe99883b00c0 0t0 TCP 172.10.61.1:36285->192.168.53.22:5223 (ESTABLISHED) 4. Force a trap. For example stop the osm process and notice that the trap doesn't make it to ECS alarm tab 5. After many minutes snmpTrapHandler gets disconnected from the broken infra node [BELFAST-ec-PRI-DNCS] lsof -Pni :5223 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 352 dncs 16u IPv4 0xfffffe9988c85100 0t60041 TCP 172.10.61.1:37252->192.168.53.21:5223 (ESTABLISHED) dncsInitd 894 dncs 12u IPv4 0xfffffe998913e880 0t0 TCP 172.10.61.1:37266->192.168.53.21:5223 (ESTABLISHED) java 1984 oamadm 8u IPv4 0xfffffe998d168b00 0t57548 TCP 172.10.61.1:37254->192.168.53.21:5223 (ESTABLISHED) bossServe 18657 dncs 13u IPv4 0xfffffe997d7f5740 0t0 TCP 172.10.61.1:37250->192.168.53.21:5223 (ESTABLISHED) rpOrch 18973 dncs 10u IPv4 0xffffffffb9c88080 0t0 TCP 172.10.61.1:33540->192.168.53.22:5223 (ESTABLISHED) rpOrch 18973 dncs 11u IPv4 0xffffffffdce80300 0t0 TCP 172.10.61.1:37248->192.168.53.21:5223 (ESTABLISHED) 6. Force another trap. Notice that snmpTrapHandler establish a new connection to the operational infra node. However, the trap doesn't make it to the ECS alarm console [BELFAST-ec-PRI-DNCS] lsof -Pni :5223 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 352 dncs 16u IPv4 0xfffffe9988c85100 0t62597 TCP 172.10.61.1:37252->192.168.53.21:5223 (ESTABLISHED) dncsInitd 894 dncs 12u IPv4 0xfffffe998913e880 0t0 TCP 172.10.61.1:37266->192.168.53.21:5223 (ESTABLISHED) java 1984 oamadm 8u IPv4 0xfffffe998d168b00 0t59984 TCP 172.10.61.1:37254->192.168.53.21:5223 (ESTABLISHED) bossServe 18657 dncs 13u IPv4 0xfffffe997d7f5740 0t0 TCP 172.10.61.1:37250->192.168.53.21:5223 (ESTABLISHED) rpOrch 18973 dncs 10u IPv4 0xffffffffb9c88080 0t0 TCP 172.10.61.1:33540->192.168.53.22:5223 (ESTABLISHED) rpOrch 18973 dncs 11u IPv4 0xffffffffdce80300 0t0 TCP 172.10.61.1:37248->192.168.53.21:5223 (ESTABLISHED) snmpTrapH 20002 dncs 13u IPv4 0xfffffe9987b33100 0t0 TCP 172.10.61.1:37952->192.168.53.21:5223 (ESTABLISHED) 7. snmpTrapHandler gets back to normal. All the traps get forwarded to ECS
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.
Bug Details Include
- Full Description (including symptoms, conditions and workarounds)
- Status
- Severity
- Known Fixed Releases
- Related Community Discussions
- Number of Related Support Cases