Symptom:
RACK2_CE9_Cache Engine reboot

Conditions:
<mg-1 snmpd log>

2014-07-14 15:18:41.329  [peerapp:polling] ce-9 request failed: timeout

2014-07-14 15:18:53.298  [peerapp:polling] ce-9 overtime

2014-07-14 15:18:55.641  cluster has been degraded

2014-07-14 15:18:59.362  [peerapp:polling] ce-9 request failed: timeout

2014-07-14 15:19:17.385  [peerapp:polling] ce-9 request failed: timeout

 

<ce-9 warn log>

Jul 14 15:18:14 ce-9 kernel: Supported: No

Jul 14 15:18:14 ce-9 kernel: Pid: 0, comm: swapper Tainted: G
2.6.27.19-llpf-5-default #4

Jul 14 15:18:14 ce-9 kernel: 

Jul 14 15:18:14 ce-9 kernel: Call Trace:

Jul 14 15:18:14 ce-9 kernel:  [<ffffffff8020d9f9>]
show_trace_log_lvl+0x41/0x58

Jul 14 15:18:14 ce-9 kernel:  [<ffffffff80496ad4>] dump_stack+0x69/0x6f

Jul 14 15:18:14 ce-9 kernel:  [<ffffffff8023bf49>] warn_slowpath+0xa9/0xd1

Jul 14 15:18:14 ce-9 kernel:  [<ffffffff80435ce3>] dev_watchdog+0x11c/0x1ad

Jul 14 15:18:16 ce-9 kernel:  [<ffffffff802448f1>]
run_timer_softirq+0x18d/0x204

Jul 14 15:18:16 ce-9 kernel:  [<ffffffff80240ec4>] __do_softirq+0x7d/0x107

Jul 14 15:18:16 ce-9 kernel:  [<ffffffff8020d2ac>] call_softirq+0x1c/0x28

Jul 14 15:18:16 ce-9 kernel:  [<ffffffff8020e553>] do_softirq+0x2c/0x68

Jul 14 15:18:16 ce-9 kernel:  [<ffffffff80240bf1>] irq_exit+0x3f/0x85

Jul 14 15:18:16 ce-9 kernel:  [<ffffffff8021b09c>]
smp_apic_timer_interrupt+0x95/0xae

Jul 14 15:18:16 ce-9 kernel:  [<ffffffff8020cc86>]
apic_timer_interrupt+0x66/0x70

Jul 14 15:18:16 ce-9 kernel:  [<ffffffff80212722>] mwait_idle+0x3c/0x3f

Jul 14 15:18:16 ce-9 kernel:  [<ffffffff8020b04a>] cpu_idle+0x88/0xd0

Jul 14 15:18:16 ce-9 kernel:  [<ffffffff8095bdfd>] start_kernel+0x3b1/0x3bd

Jul 14 15:18:16 ce-9 kernel:  [<ffffffff8095b38f>]
x86_64_start_kernel+0xde/0xe4

Jul 14 15:18:16 ce-9 kernel: 

Jul 14 15:18:16 ce-9 kernel: ---[ end trace d70c597f4b6d81fa ]---

Jul 14 15:18:16 ce-9 kernel: [       config  0  719]: Received notification
NETDEV_CHANGE for device eth5

Jul 14 15:27:25 ce-9 rpcbind: cannot create socket for udp6

Jul 14 15:27:25 ce-9 rpcbind: cannot create socket for tcp6

Jul 14 15:27:26 ce-9 iscsid: iSCSI logger with pid=11548 started!

Jul 14 15:27:27 ce-9 iscsid: transport class version 2.0-870. iscsi