ESX iSCSI Storage - Error

[SAGEBCI]

Hello. Currently, while attempting to access either or the 2 LUNS that are presented from an iSCSI storage array, the error message is: "A General System Error Occured". Are there any thoughts on why, or how, this error message is occurring? There is more to this story, it may be too convoluted at this point to explain, but this is enough i think.

Thanks,

[ablej]

What errors are displayed in /var/log/vmkernel or /var/log/vmkwarning? Are there any issues with the LUNs or do you just get an error message with no fault. I have seen a admins VI client get this type of error and reinstalling the client solved the issue.

David Strebel

www.holy-vm.com

If you find this information useful, please award points for "correct" or "helpful"

[SAGEBCI]

Along with the error below, there is also the same inforamtion included in "vmkwarning". Addtioally one of my VMDK's is missing. Not sure if there is a recovery method? Thanks,

Nov 24 04:06:19 kronos vmkernel: 18:10:58:24.809 cpu0:6620)<6>megasas_service_aen[0]: aen received
Nov 24 04:06:19 kronos vmkernel: 18:10:58:24.809 cpu4:4205)<6>megasas_hotplug_work[0]: event code 0x0071
Nov 24 04:06:19 kronos vmkernel: 18:10:58:24.809 cpu4:4205)<6>megasas_hotplug_work[0]: aen registered

Nov 24 04:27:16 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 129).18:11:19:21.812 cpu7:4109)WARNING: Res3: 2187: Volume 4adb644f-9158b10e-12e2-001ec9d0ae00 ("Delphi1") may be d

amaged on disk. Resource cluster metadata corruption detected

Nov 24 04:27:16 kronos vmkernel:

Nov 24 04:27:16 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 132).18:11:19:21.812 cpu7:4109)WARNING: Res3: 2187: Volume 4adb644f-9158b10e-12e2-001ec9d0ae00 ("Delphi1") may be d

amaged on disk. Resource cluster metadata corruption detected

Nov 24 04:27:16 kronos vmkernel:

Nov 24 04:27:16 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 133).18:11:19:21.812 cpu7:4109)WARNING: Res3: 2187: Volume 4adb644f-9158b10e-12e2-001ec9d0ae00 ("Delphi1") may be d

amaged on disk. Resource cluster metadata corruption detected

[SAGEBCI]

The most recent of the error messages from "vmkwarning":

Truncated for you...

# tail -50 vmkwarning

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7122).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7123).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7124).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7125).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (198) and freeResources (200) (cluster 7128).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7129).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7130).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7131).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7133).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7134).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7135).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7137).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7138).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7139).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7140).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7142).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7143).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (198) and freeResources (200) (cluster 7147).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7149).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:38:29 kronos vmkernel: Inconsistency between bitmap (199) and freeResources (200) (cluster 7150).0:00:01:56.601 cpu7:4109)WARNING: Res3: 2187: Volume 4b0c3609-0c718b72-704b-001ec9d0ae00 ("snap-60fe2e2c-sn`p-012b2d9a-sn`p-3487e5ed-sn`p-77683c9f-sn`p-4e925fe3-Delphi0")

Nov 24 12:50:55 kronos vmkernel: 0:00:14:24.236 cpu5:4111)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:50:55 kronos vmkernel: 0:00:14:24.238 cpu4:4109)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 a244c6 1e0001b2 aed0c9 0 0 0 0 0 0 0

Nov 24 12:52:28 kronos vmkernel: 0:00:15:57.238 cpu1:4111)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:52:28 kronos vmkernel: 0:00:15:57.239 cpu1:4110)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:52:37 kronos vmkernel: 0:00:16:05.782 cpu3:4109)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:52:37 kronos vmkernel: 0:00:16:05.783 cpu3:4111)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:52:38 kronos vmkernel: 0:00:16:07.283 cpu3:4111)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:52:38 kronos vmkernel: 0:00:16:07.284 cpu3:4110)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:52:56 kronos vmkernel: 0:00:16:25.539 cpu1:4111)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:52:56 kronos vmkernel: 0:00:16:25.540 cpu1:4109)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:53:38 kronos vmkernel: 0:00:17:07.065 cpu5:4111)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:53:38 kronos vmkernel: 0:00:17:07.066 cpu7:4109)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:53:40 kronos vmkernel: 0:00:17:08.661 cpu5:4110)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:53:40 kronos vmkernel: 0:00:17:08.661 cpu5:4111)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:53:42 kronos vmkernel: 0:00:17:10.625 cpu3:4109)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:53:42 kronos vmkernel: 0:00:17:10.626 cpu5:4110)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:53:57 kronos vmkernel: 0:00:17:25.807 cpu5:4110)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:53:57 kronos vmkernel: 0:00:17:25.807 cpu5:4111)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:53:58 kronos vmkernel: 0:00:17:26.946 cpu5:4110)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:53:58 kronos vmkernel: 0:00:17:26.947 cpu5:4111)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:59:40 kronos vmkernel: 0:00:23:09.319 cpu2:4110)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:59:40 kronos vmkernel: 0:00:23:09.319 cpu5:4109)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:59:41 kronos vmkernel: 0:00:23:10.434 cpu5:4109)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 12:59:41 kronos vmkernel: 0:00:23:10.435 cpu6:4111)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 14:25:45 kronos vmkernel: 0:01:49:13.728 cpu5:4111)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 14:25:45 kronos vmkernel: 0:01:49:13.729 cpu3:4109)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 14:25:46 kronos vmkernel: 0:01:49:15.293 cpu3:4110)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 14:25:46 kronos vmkernel: 0:01:49:15.294 cpu7:4111)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 14:25:48 kronos vmkernel: 0:01:49:17.510 cpu7:4111)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

Nov 24 14:25:48 kronos vmkernel: 0:01:49:17.511 cpu2:4109)WARNING: Fil3: 1918: Failed to reserve volume f530 28 1 4b0c3609 c718b72 1e00704b aed0c9 0 0 0 0 0 0 0

#

[SAGEBCI]

Found this thread, got any clues on the commands that they used to fix it?

http://communities.vmware.com/thread/62053

[ablej]

I don’t know your environment, but is it possible to storage vMotion all VM's off the LUN? I would contact VMware support about the issue because the commands can cause data loss.

[SAGEBCI]

Thanks... Unfortunately I don't have any support options; running ESX 4 Essentials. All VM's are currently off the LUN, but one VM "hard disks" was the vmdk that we lost. It would be the bees knees if we could recover, since it was an important data store.

[Dave_Mishchenko]

I would open a support call if you can, but as a last ditch effort you might try this driver to give access to the LUN from Linux / Windows - http://code.google.com/p/vmfs/.




Dave

VMware Communities User Moderator

Now available - vSphere Quick Start Guide

Do you have a system or PCI card working with VMDirectPath? Submit your specs to the Unofficial VMDirectPath HCL.