Storage Failure Esxi 6.5 - datastore is not consumed

[GKaveH]

Hello guys

i need help with my situation i have problem with my esxi host i realized hard failure on my server and tried to browse my datastore but it's blanked

and after i rescan storage datastore become inaccessible and showed not consumed all my vm are online and i have accesses to them but i cant clone or migrate

datastore raid level is 10 with these situation

Array A (SAS,

      logicaldrive 1 (1.09 TB, RAID 1+0, Interim Recovery Mode)

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS HDD, 300 GB, OK)

      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS HDD, 300 GB, OK)

      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS HDD, 300 GB, Failed)

      physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS HDD, 300 GB, Predictive Failure)

      physicaldrive 2I:0:1 (port 2I:box 0:bay 1, SAS HDD, 300 GB, Predictive Failure)

      physicaldrive 2I:0:2 (port 2I:box 0:bay 2, SAS HDD, 300 GB, OK)

      physicaldrive 2I:0:3 (port 2I:box 0:bay 3, SAS HDD, 300 GB, OK)

      physicaldrive 2I:0:4 (port 2I:box 0:bay 4, SAS HDD, 300 GB, Failed)

when i try to cd in datastore it's okey but i cant see anything in there

[root@localhost:/vmfs/volumes/5b97c03e-9928dee6-8f37-ac162d6f9be8] ls -la

total 0

[root@localhost:/vmfs/volumes/5b97c03e-9928dee6-8f37-ac162d6f9be8]

by the way for when i was powering off one of my VMs it's become inaccessible

can anybody help me ? sorry about my English language

[daphnissov]

Your array has probably crashed if that's the case. You show multiple failed drives and two in predictive failure mode, which means failure has really already happened or is imminent. Probably best to start looking at backup restoration.

[GKaveH]

thanks for your advice already i managed to create another Vm's and copied config file of those VM's and hopefully i was able to rescue 4 of 6 Vm's but it took a lot of working and by the way I really wonder how these Vm's are working while the datastore is inaccessible.

[daphnissov]

If the datastore was inaccessible they weren't "working" they were just existing in memory like zombie processes.

[GKaveH]

i know that but they are working and i can connect to them and create file or move file from them with rsync or winscp !! because of this i could created another VM's with same config file that i copied from these VM's, but when i was turning off one them it's was become inaccessible.

[daphnissov]

Ok, well, whatever is going on, it's nothing good if the client is saying your datastore is inaccessible. So you have some storage issues to fix on your side.

[continuum]

> I really wonder how these Vm's are working while the datastore is inaccessible.

Once a VM is running ESXi just needs to know where the flat.vmdks, the delta.vmdks and the vswp-file are located on the physical hardware.

This information is cached in RAM.

That means that a VM can continue to work even in a case where the partitiontable of the disk with the datastore gets corrupted.

But after next reboot the corrupted partitiontable will obviously prevent you from even starting the VM.

Something is wrong with your disk or datastore - this can be a minor issue (bad partitiontable ) or something serious ( corrupt VMFS metadata )

If you want to know what to do next read Create a VMFS-Header-dump using an ESXi-Host in production | VM-Sickbay ​

and provide a dump like that. I maybe able to recover your VMs

Ulli