VMware Cloud Community
martinkri
Contributor
Contributor

Problem using newly created VMFS datastore

Hi,

I have 4 servers running vpshere ESXi 4.1 connected to a IBM DS3300 iSCSI unit. I am also running vcenter 4.1

Initially I had 3 datastores hosting all my VMs, but in need of more storage space, I had to upgrade the storage unit and connected a IBM EXP3000 expanision unit to the DS3300 via SAS cables.

After I create the new LUN on the SAN unit, all the ESXi servers discover it through iSCSI and I can add the datastore.

See attached pdf-file with screenshots.

My problem is that after the LUN has been added and formatted, none of the servers can utilize it.

I can browse the empty LUN by I can't even create a new folder in it and I get "I/O error" when I try to upload files to it.

Has anyone else experience this ?

None of my existing datastores have this problem.

Thanks in advance.

See attached pdf-file with screenshots.

0 Kudos
9 Replies
admin
Immortal
Immortal

Hello

Good morning from Canada

It seems that your ESX hosts have read-only permission on that LUN that's why you cannot even create a folder

Please verify on your SAN if you granted the appropriate permissions for your hosts (read-write)

Regards

0 Kudos
MauroBonder
VMware Employee
VMware Employee

see if this help. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100067...

*Please, don't forget the awarding points for "helpful" and/or "correct" answers. *Por favor, não esqueça de atribuir os pontos se a resposta foi útil ou resolveu o problema.* Thank you/Obrigado
0 Kudos
martinkri
Contributor
Contributor

On the IBM Storage Manager, I have 1 Hostgroup that contains the 4 ESXi hosts, based on their iSCSI initiators:

9Capture.PNG

Furthermore there is a "Host-to-Logical Drive mapping" that makes it easy add a new LUN and grant access to the hosts defined in the Hostgroup.

I can't see how this could be the problem, all the ESXi hosts have access to the first three LUN's I created so why shouldn't they on the fourth?

As I wrote in the first post, the ESXi servers can see the LUN when you choose "Add datastore" and they format it in VMFS.

0 Kudos
bulletprooffool
Champion
Champion

Are you sure that you have wrote access to the new LUN?

In addition, are you sure that the LUN has been formatted and is Valid for ESX to connect to it?

Is it possibly a small block size, but a large datastore?

This is a Windows error, not a VMware error (even though it appears  when installing the VM tools)

Config maximums at:

http://www.vmware.com/pdf/vsphere4/r40/vsp_40_config_max.pdf

Block  size 8MB
File size (block size=1MB) 256GB minus 512B
File size (block size=2MB) 512GB minus 512B
File size (block size=4MB) 1TB minus 512B

File size (block size=8MB) 2TB minus 512B

One day I will virtualise myself . . .
0 Kudos
martinkri
Contributor
Contributor

Here is what what happens:

-After "Rescan HBAs", the hosts finds the newly created LUN and multiple paths to it.

-"Add Datastore"finds the LUN and creates and formats it in VMFS

-The LUN is added as a datastore:

10Capture.PNG

As you can see from this screenshot, the datastore is discovered and formatted in VMFS by one of my hosts, and the other hosts immediately discover the LUN afterwards. So the LUN is newly formatted and should be empty and ready for use.

Immediately after this, the datastore becomes inaccessible.

The only difference between the "old" LUNs and the new one, is that the new is formatted in VMFS 3.46 and the other ones were created back with vpshere4.0 and use VMFS 3.3

When I try to create a new folder in the datastore:

(vim.fault.FileNotFound) {
   dynamicType = <unset>,
   faultCause = (vmodl.MethodFault) null,
   file = "[LUN4] New Folder",
   msg = "",
}
Args:
Arg datastorePath:
"[LUN4] New Folder"
Arg searchSpec:
(vim.host.DatastoreBrowser.SearchSpec) {
   dynamicType = <unset>,
   query = (vim.host.DatastoreBrowser.Query) [
      (vim.host.DatastoreBrowser.FolderQuery) {
         dynamicType = <unset>,
      },
      (vim.host.DatastoreBrowser.FloppyImageQuery) {
         dynamicType = <unset>,
      },
      (vim.host.DatastoreBrowser.IsoImageQuery) {
         dynamicType = <unset>,
      },
      (vim.host.DatastoreBrowser.VmConfigQuery) {
         dynamicType = <unset>,
         filter = (vim.host.DatastoreBrowser.VmConfigQuery.Filter) null,
         details = (vim.host.DatastoreBrowser.VmConfigQuery.Details) null,
      },
      (vim.host.DatastoreBrowser.TemplateVmConfigQuery) {
         dynamicType = <unset>,
         filter = (vim.host.DatastoreBrowser.VmConfigQuery.Filter) null,
         details = (vim.host.DatastoreBrowser.VmConfigQuery.Details) null,
      },
      (vim.host.DatastoreBrowser.VmLogQuery) {
         dynamicType = <unset>,
      },
      (vim.host.DatastoreBrowser.VmNvramQuery) {
         dynamicType = <unset>,
      },
      (vim.host.DatastoreBrowser.VmSnapshotQuery) {
         dynamicType = <unset>,
      },
      (vim.host.DatastoreBrowser.VmDiskQuery) {
         dynamicType = <unset>,
         filter = (vim.host.DatastoreBrowser.VmDiskQuery.Filter) null,
         details = (vim.host.DatastoreBrowser.VmDiskQuery.Details) {
            dynamicType = <unset>,
            diskType = false,
            capacityKb = true,
            hardwareVersion = false,
            controllerType = <unset>,
            diskExtents = <unset>,
            thin = <unset>,
         },
      },
      (vim.host.DatastoreBrowser.Query) {
         dynamicType = <unset>,
      }
   ],
   details = (vim.host.DatastoreBrowser.FileInfo.Details) {
      dynamicType = <unset>,
      fileType = true,
      fileSize = true,
      modification = true,
      fileOwner = true,
   },
   searchCaseInsensitive = true,
   sortFoldersFirst = true,

0 Kudos
depping
Leadership
Leadership

Did you contact support already?

Duncan (VCDX)

Available now on Amazon: vSphere 4.1 HA and DRS technical deepdive

0 Kudos
martinkri
Contributor
Contributor

It is still unresolved so I have filed a support request now and hope they are able to help me with this.

0 Kudos
depping
Leadership
Leadership

Please let us know if and how it was solved.

Duncan (VCDX)

Available now on Amazon: vSphere 4.1 HA and DRS technical deepdive

0 Kudos
martinkri
Contributor
Contributor

vmware support helped me to investigate the logs further.

We checked the partition information on the LUN which is still valid and its indicating a VMFS partition.

We then checked the VMFS metadata which reported there was still VMFS data on the LUN.

From this we can see that there is corruption occurring on the LUN itself.

The corruption seems to be affecting both the data and VMFS metadata.

The corruption is only occurring on LUNs being presented from the EXP3000 expansion which was recently connected to the DS3300.

So I had to create a support request with IBM and send them some logs from the DS3300.

They informed me that I needed to upgrade the controller firmware  to version 07.35.57.00 and it's associated NVSRAM update.

Even though I had firmware version 07.35.41, the upgrade was needed in order for them to help me.

And after the firmware upgrade everything seems to be working again.

The ESXi servers can now use the new LUNs, but I haven't tested the performance on VMs that are running on these LUNs yet.

Storage Firmware upgrades are always tricky because it requires downtime, but sometimes you don't have a choice.

So now I will run some VMs for a couple of days and send IBM updated logs before we close the support case.

0 Kudos