Guest

Preview Tool

Cisco Bug: CSCvu35337 - LSQ-4906: UDPD crash with "Kernel panic" at high outbound traffic load

Last Modified

Sep 13, 2020

Products (1)

  • Cisco Stealthwatch Enterprise

Known Affected Releases

7.1.1 7.2.0

Description (partial)

Symptom:
Symptoms:
1. UDPD engine drops packets, as indicated by the increasing "Total Dropped" and "Last Dropped" counter values in output.log file;
2. Network connection to the UDPD may be interrupted; the box may not be responsive;
3. The "Kernel panic" may be triggered to cause reboot of the system; (the system may also trigger auto reboot by itself without logging error and Kernel panic in kern.log). 
4. The kern.log file may contain any/all of the following errors:
4.1.    
{noformat}
May 20 12:49:34 fr-01 kernel: igb 0000:01:00.0: Detected Tx Unit Hang
{noformat}
{noformat}
May 20 21:15:41 fr-01 kernel: igb 0000:01:00.0 eth0: Reset adapter
{noformat}
4.2.
{noformat}
May 20 17:59:31 fr-01 kernel: NETDEV WATCHDOG: eth0 (igb): transmit queue 5 timed out
May 20 17:59:31 fr-01 kernel: WARNING: CPU: 8 PID: 0 at net/sched/sch_generic.c:461 dev_watchdog+0x20e/0x220
{noformat}

4.3.
{noformat}
May 20 17:59:31 fr-01 kernel: CPU: 8 PID: 0 Comm: swapper/8 Not tainted 4.19.53 #1
May 20 17:59:31 fr-01 kernel: Call Trace:
May 20 17:59:31 fr-01 kernel:  <IRQ>
May 20 17:59:31 fr-01 kernel:  ? pfifo_fast_reset+0x100/0x100
May 20 17:59:31 fr-01 kernel:  call_timer_fn+0x2b/0x130
May 20 17:59:31 fr-01 kernel:  run_timer_softirq+0x1d3/0x420
May 20 17:59:31 fr-01 kernel:  ? tick_sched_handle+0x22/0x60
May 20 17:59:31 fr-01 kernel:  ? tick_sched_timer+0x37/0x70
May 20 17:59:31 fr-01 kernel:  ? __hrtimer_run_queues+0x10c/0x270
May 20 17:59:31 fr-01 kernel:  __do_softirq+0x10d/0x2c3
May 20 17:59:31 fr-01 kernel:  irq_exit+0xc2/0xd0
May 20 17:59:31 fr-01 kernel:  smp_apic_timer_interrupt+0x74/0x130
May 20 17:59:31 fr-01 kernel:  apic_timer_interrupt+0xf/0x20
May 20 17:59:31 fr-01 kernel:  </IRQ>
{noformat}

4.4.
{noformat}
May 20 13:47:55 fr-01 kernel: clocksource: timekeeping watchdog on CPU28: Marking clocksource 'tsc' as unstable because the skew is too large:
May 20 13:47:55 fr-01 kernel: clocksource:                       'hpet' wd_now: 386ffeac wd_last: 3f9c0d82 mask: ffffffff
May 20 13:47:55 fr-01 kernel: clocksource:                       'tsc' cs_now: 118264f00c1d352 cs_last: 11825a5b30dd12d mask: ffffffffffffffff
May 20 13:47:55 fr-01 kernel: tsc: Marking TSC unstable due to clocksource watchdog
May 20 13:47:55 fr-01 kernel: TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.
May 20 13:47:55 fr-01 kernel: sched_clock: Marking unstable (1268002151500, 1706624362)<-(1270067361747, -358588255)
May 20 13:47:55 fr-01 kernel: clocksource: Switched to clocksource hpet
{noformat}

4.5.
{noformat}
May 20 17:51:31 fr-01 kernel: INFO: task kworker/1:0:19 blocked for more than 300 seconds.
May 20 17:51:31 fr-01 kernel:       Not tainted 4.19.53 #1
May 20 17:51:31 fr-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 20 17:51:31 fr-01 kernel: kworker/1:0     D    0    19      2 0x80000000
May 20 17:51:31 fr-01 kernel: Workqueue: events slab_caches_to_rcu_destroy_workfn
May 20 17:51:31 fr-01 kernel: Call Trace:
May 20 17:51:31 fr-01 kernel:  ? __schedule+0x3f5/0x880
May 20 17:51:31 fr-01 kernel:  schedule+0x32/0x80
May 20 17:51:31 fr-01 kernel:  schedule_timeout+0x1e5/0x350
May 20 17:51:31 fr-01 kernel:  ? pcpu_free_area+0x261/0x2e0
May 20 17:51:31 fr-01 kernel:  wait_for_completion+0x123/0x190
May 20 17:51:31 fr-01 kernel:  ? wake_up_q+0x70/0x70
May 20 17:51:31 fr-01 kernel:  _rcu_barrier+0x120/0x180
May 20 17:51:31 fr-01 kernel:  slab_caches_to_rcu_destroy_workfn+0x8f/0xe0
May 20 17:51:31 fr-01 kernel:  process_one_work+0x191/0x370
May 20 17:51:31 fr-01 kernel:  worker_thread+0x4f/0x3b0
ay 20 17:51:31 fr-01 kernel:  kthread+0xf8/0x130
May 20 17:51:31 fr-01 kernel:  ? rescuer_thread+0x340/0x340
May 20 17:51:31 fr-01 kernel:  ? kthread_create_worker_on_cpu+0x70/0x70
May 20 17:51:31 fr-01 kernel:  ret_from_fork+0x35/0x40
May 20 17:51:31 fr-01 kernel: NMI backtrace for cpu 5
{noformat}

4.6.
{noformat}
May 20 17:51:31 fr-01 kernel: CPU: 5 PID: 258 Comm: khungtaskd Not tainted 4.19.53 #1
May 20 17:51:31 fr-01 kernel: Call Trace:
May 20 17:51:31 fr-01 kernel:  dump_stack+0x5c/0x7b
May 20 17:51:31 fr-01 kernel:  nmi_cpu_backtrace+0x89/0x90
May 20 17:51:31 fr-01 kernel:  ? lapic_can_unplug_cpu+0xa0/0xa0
May 20 17:51:31 fr-01 kernel:  nmi_trigger_cpumask_backtrace+0xf5/0x130
May 20 17:51:31 fr-01 kernel:  watchdog+0x2ab/0x3a0
May 20 17:51:31 fr-01 kernel:  kthread+0xf8/0x130
May 20 17:51:31 fr-01 kernel:  ? hungtask_pm_notify+0x40/0x40
May 20 17:51:31 fr-01 kernel:  ? kthread_create_worker_on_cpu+0x70/0x70
May 20 17:51:31 fr-01 kernel:  ret_from_fork+0x35/0x40
May 20 17:51:31 fr-01 kernel: Sending NMI from CPU 5 to CPUs 0-4,6-47:
{noformat}

4.7.
{noformat}
May 20 17:51:31 fr-01 kernel: CPU: 24 PID: 839 Comm: systemd-journal Not tainted 4.19.53 #1
May 20 17:51:31 fr-01 kernel: Call Trace:
May 20 17:51:31 fr-01 kernel:  ? __bpf_prog_run32+0x39/0x60
May 20 17:51:31 fr-01 kernel:  ? block_page_mkwrite+0xbc/0xf0
May 20 17:51:31 fr-01 kernel:  ? seccomp_run_filters+0x58/0xb0
May 20 17:51:31 fr-01 kernel:  ? fault_dirty_shared_page.isra.79+0x37/0xb0
May 20 17:51:31 fr-01 kernel:  ? finish_mkwrite_fault+0x12c/0x140
May 20 17:51:31 fr-01 kernel:  ? do_wp_page+0x1eb/0x5f0
May 20 17:51:31 fr-01 kernel:  ? __seccomp_filter+0x43/0x4a0
May 20 17:51:31 fr-01 kernel:  ? syscall_trace_enter+0x117/0x2c0
May 20 17:51:31 fr-01 kernel:  ? do_syscall_64+0xc8/0x100
May 20 17:51:31 fr-01 kernel:  ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
{noformat}

4.8.
{noformat}
May 20 17:51:31 fr-01 kernel: CPU: 33 PID: 11933 Comm: syslog-ng Not tainted 4.19.53 #1
{noformat}

4.9.
{noformat}
May 20 17:51:31 fr-01 kernel: CPU: 43 PID: 36134 Comm: flowfan Not tainted 4.19.53 #1
May 20 17:51:31 fr-01 kernel: Call Trace:
May 20 17:51:31 fr-01 kernel:  ? __alloc_skb+0x96/0x1c0
May 20 17:51:31 fr-01 kernel:  ? udp_packet+0x73/0xc0 [nf_conntrack]
May 20 17:51:31 fr-01 kernel:  nf_hook_slow+0x43/0xc0
May 20 17:51:31 fr-01 kernel:  raw_sendmsg+0xabd/0xc80
May 20 17:51:31 fr-01 kernel:  ? ip4_datagram_release_cb+0x1e0/0x1e0
May 20 17:51:31 fr-01 kernel:  ? aa_sk_perm+0x46/0x170
May 20 17:51:31 fr-01 kernel:  ? sock_sendmsg+0x36/0x40
May 20 17:51:31 fr-01 kernel:  sock_sendmsg+0x36/0x40
May 20 17:51:31 fr-01 kernel:  __sys_sendto+0xd8/0x150
May 20 17:51:31 fr-01 kernel:  ? __schedule+0x3fd/0x880
May 20 17:51:31 fr-01 kernel:  ? syscall_trace_enter+0x1ae/0x2c0
May 20 17:51:31 fr-01 kernel:  ? __audit_syscall_exit+0x206/0x290
May 20 17:51:31 fr-01 kernel:  __x64_sys_sendto+0x24/0x30
May 20 17:51:31 fr-01 kernel:  do_syscall_64+0x55/0x100
May 20 17:51:31 fr-01 kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
{noformat}

4.10.
{noformat}
May 20 17:51:31 fr-01 kernel: Kernel panic - not syncing: hung_task: blocked tasks
{noformat}

5. When "Kernel panic" is being triggered, the system is rebooted.

Conditions:
"Kernel panic" triggers a system reboot.
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.

Bug Details Include

  • Full Description (including symptoms, conditions and workarounds)
  • Status
  • Severity
  • Known Fixed Releases
  • Related Community Discussions
  • Number of Related Support Cases
Bug information is viewable for customers and partners who have a service contract. Registered users can view up to 200 bugs per month without a service contract.