Windows 2003 Guests intermittantly hang on shut down or reboot

[z1b903]

Folks:

This is the 2nd go around, I originally thought I had a fix witht he removal of a Windows shutdown script, but this time I found this sequence of events in the Guest VM's log file when the problem is occurring:

May 15 14:04:37.177: mks| SOCKET 21 recv error 5: Input/output error

May 15 14:04:37.177: mks| SOCKET 21 destroying VNC backend on socket error: 5

May 15 14:31:12.010: mks| SOCKET 22 recv error 5: Input/output error

May 15 14:31:12.010: mks| SOCKET 22 destroying VNC backend on socket error: 5

May 15 15:07:11.028: vmx| DISKUTIL: scsi0:2 : toolsVersion = 7300

Note the 15:07 entry began after I manually reset the Guest for VI.

Below is the same host, with a normal reboot 20 min's later

May 15 15:26:41.437: mks| SOCKET 28 recv error 5: Input/output error

May 15 15:26:41.441: mks| SOCKET 28 destroying VNC backend on socket error: 5

May 15 15:26:52.746: mks| Ignoring update request in VGA_Expose (mode change pending).

May 15 15:26:54.904: mks| SOCKET 29 recv error 5: Input/output error

May 15 15:26:54.904: mks| SOCKET 29 destroying VNC backend on socket error: 5

May 15 15:26:58.786: mks| Ignoring update request in VGA_Expose (mode change pending).

May 15 15:27:00.356: mks| SOCKET 30 recv error 5: Input/output error

May 15 15:27:00.356: mks| SOCKET 30 destroying VNC backend on socket error: 5

May 15 15:27:05.407: vcpu-3| CPU reset: soft

Pretty basic ESX3.5/VI 2.5 install, running several Guest Windows 2003 servers. From the Guest console, one see a blank "gray" screen, with the ability to move the mouse pointer around. Why is the "vcpu-3| CPU reset: soft" command not executing?

[kjb007]

Is the machine a domain member? Do you have a policy to clear pagefile on exit? If you leave it, does it eventually shut down correctly? There was another user that had this issue and it turned out to be a domain policy. If the server is a domain member, can you take the server out of the domain and reboot to see if it goes any faster?

-KjB

[kjb007]

Guess that was you. Sorry. Does the reboot not complete successfully now if the server is not in the domain like last time?

-KjB

[z1b903]

KjB007:

Yeah it's me, I'm not sure what to say anymore, this morning all is normal, I can reboot/shutdown at will, both "domain member" servers, and standalones. I'll need to test further, to flush out these symptoms. Yesterday afternoon this was not so.

May 16 08:07:33.539: vmx| TOOLS received reboot state change request; adding delayed poweron request

May 16 08:07:33.539: vmx| TOOLS sending 'OS_Reboot' (2) state change request

May 16 08:07:33.928: vcpu-1| Guest: toolbox: VMware Tools Service Shutdown.

May 16 08:07:33.929: vcpu-1| Guest: toolbox: VMware Tools Service Stopping.

May 16 08:07:33.989: vcpu-0| TOOLS autoupgrade protocol version 0

May 16 08:07:33.991: vcpu-0| TOOLS ToolsCapabilityGuestTempDirectory received 0

May 16 08:07:33.992: vcpu-0| GuestRpc: Channel 0 reinitialized.

May 16 08:07:34.075: vcpu-0| Guest: toolbox: Service: waiting for GuestInfoServer thread.

May 16 08:07:34.087: vcpu-0| Guest: toolbox: GuestInfoServer received quit event.

May 16 08:07:34.088: vcpu-0| Guest: toolbox: GuestInfoServer exiting.

May 16 08:07:34.089: vcpu-0| Guest: toolbox: Service: GuestInfoServer thread exited.

May 16 08:07:38.808: vcpu-0| VMMouse: CMD Disable

May 16 08:07:38.808: vcpu-0| VMMouse: Disabling VMMouse mode

May 16 08:07:38.808: vcpu-0| MKS switching absolute mouse on

May 16 08:07:38.904: vcpu-0| CPU reset: soft

May 16 08:07:38.906: vcpu-1| CPU reset: soft

May 16 08:07:39.007: mks| VNCENCODE 9 encoding mode change: (640x480x16depth,16bpp)

May 16 08:07:39.151: vcpu-1| CPU reset: soft

Lets just put this on hold, I have some extensive system work this weekend that will test reboots/shutdowns etc, plus I'll test today. If I can duplicate the problem and/or it becomes more consistent I'll post a new message. I also am thinking of handing this issue to VMware support, as I need to get this ESX3.5 implementation into production (our "guest" Exchange 2007 server goes live this weekend).

[z1b903]

Just an update, after testing this weekend, all is normal, Guests reboot and/or shutdown normally.

[kjb007 · Accepted solution]

If you get a chance to test a few more times, that would be great. It would be nice to see consistency. Check your SAN also, if you have multiple paths, to see if the inconsistency happens when there is a Storage processor failover, and a path switch. I'm assuming you have multiple paths to your SAN here, but in the case of only one VM, then I'm not sure if there will be a clearcut answer.

Anyway, make sure you mark the question as answered when you feel your issues were resolved.

-KjB