Re: "soft lockup" in kernel

On Fri, Jul 5, 2013 at 7:00 AM, Dennis Jenkins wrpte

> Before I looked at your pastebin, I was going to ask "What kind of
>storage
> are the VMDKs on? If they are on NFS, iSCSI or FC, could the NAS/SAN be
> experiencing a problem?" But I see in the stack trace that the kernel
>thread
> hung in "vmxnet3_xmit_frame" (sending an Ethernet frame on your virtual
>NIC).

I did not spot that.

> Describe your vmware network topology.
> Do you share the same VLAN for guest traffic with NFS or iSCSI used by
>vmware
> for storing VMDKs?

No. iSCSI traffic between the VMWare hosts and the SAN uses completely
separate NICs and different switches to the "production" LAN.

> Are there any errors recorded on the Ethernet switch connected to your
> VMWare servers?

No, there is nothing in the logs on the switch.

> What path does a packet take to get from a postgresql server process in
>your
> VM to a client?

Postgres is the backend database to a web application. Only the web
application uses it. It sits on another VM in the same cluster (but not on
the same physical host, so it definitely goes through a physical switch).
Is that what you mean?

> What version of VMWare are you running?

ESXi 5.0.0.

> If you are managing it with vCenter, are there any alarms of events in
>the
> VMWare logs?

I've had a look at the task activity in VCEnter and found these two events
at almost the same time as the kernel messages. In both cases the start
time (the first time below) is 5-6 seconds after the kernel message, and
I've seen that the clock on the Postgres VM and the VCenter server, at
least, are in sync (it may not, of course, be the VCenter server's clock
that these logs get the time from).

Remove snapshot
GLIBMPDB001_replica
Completed
GLIDE\Svc_vcenter
05/07/2013 11:58:41
05/07/2013 11:58:41
05/07/2013 11:59:03

Remove snapshot
GLIBMPDB001_replica
Completed
GLIDE\Svc_vcenter
05/07/2013 10:11:10
05/07/2013 10:11:10
05/07/2013 10:11:23

For clarity, here are the kernel messages:

syslog:Jul 5 10:11:05 glibmpdb001 kernel: [1807328.506991] BUG: soft
lockup - CPU#0 stuck for 25s! [postgres:5937]
syslog:Jul 5 11:58:35 glibmpdb001 kernel: [1813775.496127] BUG: soft
lockup - CPU#3 stuck for 73s! [postgres:18723]

This is done by Veeam and not a person. There has not been a newer version
of this event since (at the time of writing this e-mail).

Regardless of this, it's not like we've only just introduced Veeam this
morning, it's been running for a few weeks, and these lockups have only
happened today.

Do you think this is worth further investigation or could there be another
explanation?

Stuart

This email and any attachments contain confidential and proprietary information of Glide Utilities Limited intended only for the use of the person to whom it is addressed. Unauthorised disclosure, copying or distribution of the email or its content may result in legal liability. If you have received the email in error, please immediately notify us by telephone on +44 333 666 5555 or email glide(at)glide(dot)uk(dot)com

The sender does not accept liability for any loss or damage from receipt or use thereof which arises as a result of internet transmission. Any views/opinions expressed within this email and any attachments are that of the individual and not necessarily that of Glide Utilities Limited.

Glide is a registered trademark of Glide Utilities Limited. Registered Office: Alpha Tower, Suffolk Street Queensway, Birmingham, B1 1TT. Registered in England & Wales. Registered Company No. 06194523.