Dear all,
I've created a new cluster and enabled VSAN on it, with 3 hosts. Everything looks fine so far,
However when I select 1 SSD and 4 HDD from one of the hosts and create a disk group, the process failed with the following message:
Operation failed, diagnostics report: Disk with uuid 52b244d6-7cda-613f-46b8-eb786e6207e5 didn't appear in the vsan cluster directory within 10 seconds.
What's interesting is that, there not such a UUID existing on that host.
I've tried disabled/enabled VSAN on the cluster, and create disk group from other hosts within the cluster, with the same result.
Any idea on how to continue the troubleshooting?
Thanks!
Silvester
Please post the output of vdq -i, vdq -q and esxcfg-scsidevs -l
And what is the network speed of vmkernel nics that you used for vSAN..? Virtual SAN requires a private 1Gb network. As a best practice, use 10Gb network.
I'm using the 10Gb network for the vSAN kernrnel.
The output of vdq -i:
[
{
"SSD" : "naa.600605b005a783701acbdef240ab6fb8",
"MD" : [
"naa.600605b008d44de01b7ea39bda5e9bc3",
"naa.600605b008d44de01b7ea6d30b7c9b9d",
"naa.600605b008d44de01b7e9e25870f7118",
"naa.600605b008d44de01b7eaacb4807dcc5",
]
},
]
Output of vdq -q:
[
{
"Name" : "naa.600605b005a783701acbdef240ab6fb8",
"VSANUUID" : "52ec0da9-7e39-8749-b729-62a01411c931",
"State" : "In-use for VSAN",
"Reason" : "None",
"IsSSD" : "1",
"IsPDL" : "0",
},
{
"Name" : "naa.600605b008d44de01b7ea39bda5e9bc3",
"VSANUUID" : "52756247-baf6-c8a1-10f3-2ea0d564e972",
"State" : "In-use for VSAN",
"Reason" : "None",
"IsSSD" : "0",
"IsPDL" : "0",
},
{
"Name" : "naa.600605b008d44de01b7ead416d91ca0f",
"VSANUUID" : "",
"State" : "Eligible for use by VSAN",
"Reason" : "None",
"IsSSD" : "1",
"IsPDL" : "0",
},
{
"Name" : "naa.600605b008d44de01b7ea6d30b7c9b9d",
"VSANUUID" : "52b244d6-7cda-613f-46b8-eb786e6207e5",
"State" : "In-use for VSAN",
"Reason" : "None",
"IsSSD" : "0",
"IsPDL" : "0",
},
{
"Name" : "naa.600605b008d44de01b7e9e25870f7118",
"VSANUUID" : "52416344-3051-95a0-cbd2-884442f5d0d0",
"State" : "In-use for VSAN",
"Reason" : "None",
"IsSSD" : "0",
"IsPDL" : "0",
},
{
"Name" : "mpx.vmhba32:C0:T0:L0",
"VSANUUID" : "",
"State" : "Ineligible for use by VSAN",
"Reason" : "Has partitions",
"IsSSD" : "0",
"IsPDL" : "0",
},
{
"Name" : "naa.600605b008d44de01b7eaacb4807dcc5",
"VSANUUID" : "52098586-205a-0fe0-f6ce-8fb922daf058",
"State" : "In-use for VSAN",
"Reason" : "None",
"IsSSD" : "0",
"IsPDL" : "0",
},
{
"Name" : "naa.600605b008d44de01b7e8a995cc62833",
"VSANUUID" : "",
"State" : "Eligible for use by VSAN",
"Reason" : "None",
"IsSSD" : "0",
"IsPDL" : "0",
},
{
"Name" : "naa.600605b008d44de01b7ea64903437c1c",
"VSANUUID" : "",
"State" : "Eligible for use by VSAN",
"Reason" : "None",
"IsSSD" : "0",
"IsPDL" : "0",
},
{
"Name" : "naa.600605b008d44de01b7ea89f26e5579e",
"VSANUUID" : "",
"State" : "Eligible for use by VSAN",
"Reason" : "None",
"IsSSD" : "0",
"IsPDL" : "0",
},
{
"Name" : "naa.600605b008d44de01b7e893b47e3b64f",
"VSANUUID" : "",
"State" : "Eligible for use by VSAN",
"Reason" : "None",
"IsSSD" : "0",
"IsPDL" : "0",
},
{
"Name" : "naa.600605b008d44de01b7ea80d1e364892",
"VSANUUID" : "",
"State" : "Eligible for use by VSAN",
"Reason" : "None",
"IsSSD" : "0",
"IsPDL" : "0",
},
{
"Name" : "naa.600605b008d44de01b7eab464f5a234b",
"VSANUUID" : "",
"State" : "Eligible for use by VSAN",
"Reason" : "None",
"IsSSD" : "0",
"IsPDL" : "0",
},
]
Output of esxcfg-scsidevs
mpx.vmhba0:C0:T19:L0
Device Type: Enclosure Svc Dev
Size: 0 MB
Display Name: Local CISCO Enclosure Svc Dev (mpx.vmhba0:C0:T19:L0)
Multipath Plugin: NMP
Console Device: /vmfs/devices/genscsi/mpx.vmhba0:C0:T19:L0
Devfs Path: /vmfs/devices/genscsi/mpx.vmhba0:C0:T19:L0
Vendor: CISCO Model: UCS 240 Revis: 0809
SCSI Level: 5 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true Is SSD: false
Other Names:
vml.020d000000f40f1b1e5d2600bd554353203234
VAAI Status: unsupported
mpx.vmhba32:C0:T0:L0
Device Type: Direct-Access
Size: 14336 MB
Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)
Multipath Plugin: NMP
Console Device: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0
Devfs Path: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0
Vendor: HV Model: Hypervisor_0 Revis: 1.01
SCSI Level: 2 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: true
Is Local: true Is SSD: false
Other Names:
vml.0000000000766d68626133323a303a30
VAAI Status: unsupported
naa.600605b005a783701acbdef240ab6fb8
Device Type: Direct-Access
Size: 189781 MB
Display Name: Local LSI Disk (naa.600605b005a783701acbdef240ab6fb8)
Multipath Plugin: NMP
Console Device: /vmfs/devices/disks/naa.600605b005a783701acbdef240ab6fb8
Devfs Path: /vmfs/devices/disks/naa.600605b005a783701acbdef240ab6fb8
Vendor: LSI Model: MR9271-8i Revis: 3.40
SCSI Level: 5 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true Is SSD: true
Other Names:
vml.0200000000600605b005a783701acbdef240ab6fb84d5239323731
VAAI Status: unsupported
naa.600605b008d44de01b7e893b47e3b64f
Device Type: Direct-Access
Size: 857353 MB
Display Name: Local LSI Disk (naa.600605b008d44de01b7e893b47e3b64f)
Multipath Plugin: NMP
Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7e893b47e3b64f
Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7e893b47e3b64f
Vendor: LSI Model: MR9271-8i Revis: 3.40
SCSI Level: 5 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true Is SSD: false
Other Names:
vml.0200000000600605b008d44de01b7e893b47e3b64f4d5239323731
VAAI Status: unsupported
naa.600605b008d44de01b7e8a995cc62833
Device Type: Direct-Access
Size: 857353 MB
Display Name: Local LSI Disk (naa.600605b008d44de01b7e8a995cc62833)
Multipath Plugin: NMP
Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7e8a995cc62833
Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7e8a995cc62833
Vendor: LSI Model: MR9271-8i Revis: 3.40
SCSI Level: 5 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true Is SSD: false
Other Names:
vml.0200000000600605b008d44de01b7e8a995cc628334d5239323731
VAAI Status: unsupported
naa.600605b008d44de01b7e9e25870f7118
Device Type: Direct-Access
Size: 857353 MB
Display Name: Local LSI Disk (naa.600605b008d44de01b7e9e25870f7118)
Multipath Plugin: NMP
Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7e9e25870f7118
Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7e9e25870f7118
Vendor: LSI Model: MR9271-8i Revis: 3.40
SCSI Level: 5 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true Is SSD: false
Other Names:
vml.0200000000600605b008d44de01b7e9e25870f71184d5239323731
VAAI Status: unsupported
naa.600605b008d44de01b7ea39bda5e9bc3
Device Type: Direct-Access
Size: 857353 MB
Display Name: Local LSI Disk (naa.600605b008d44de01b7ea39bda5e9bc3)
Multipath Plugin: NMP
Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea39bda5e9bc3
Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea39bda5e9bc3
Vendor: LSI Model: MR9271-8i Revis: 3.40
SCSI Level: 5 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true Is SSD: false
Other Names:
vml.0200000000600605b008d44de01b7ea39bda5e9bc34d5239323731
VAAI Status: unsupported
naa.600605b008d44de01b7ea64903437c1c
Device Type: Direct-Access
Size: 857353 MB
Display Name: Local LSI Disk (naa.600605b008d44de01b7ea64903437c1c)
Multipath Plugin: NMP
Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea64903437c1c
Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea64903437c1c
Vendor: LSI Model: MR9271-8i Revis: 3.40
SCSI Level: 5 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true Is SSD: false
Other Names:
vml.0200000000600605b008d44de01b7ea64903437c1c4d5239323731
VAAI Status: unsupported
naa.600605b008d44de01b7ea6d30b7c9b9d
Device Type: Direct-Access
Size: 857353 MB
Display Name: Local LSI Disk (naa.600605b008d44de01b7ea6d30b7c9b9d)
Multipath Plugin: NMP
Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea6d30b7c9b9d
Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea6d30b7c9b9d
Vendor: LSI Model: MR9271-8i Revis: 3.40
SCSI Level: 5 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true Is SSD: false
Other Names:
vml.0200000000600605b008d44de01b7ea6d30b7c9b9d4d5239323731
VAAI Status: unsupported
naa.600605b008d44de01b7ea80d1e364892
Device Type: Direct-Access
Size: 857353 MB
Display Name: Local LSI Disk (naa.600605b008d44de01b7ea80d1e364892)
Multipath Plugin: NMP
Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea80d1e364892
Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea80d1e364892
Vendor: LSI Model: MR9271-8i Revis: 3.40
SCSI Level: 5 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true Is SSD: false
Other Names:
vml.0200000000600605b008d44de01b7ea80d1e3648924d5239323731
VAAI Status: unsupported
naa.600605b008d44de01b7ea89f26e5579e
Device Type: Direct-Access
Size: 857353 MB
Display Name: Local LSI Disk (naa.600605b008d44de01b7ea89f26e5579e)
Multipath Plugin: NMP
Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ea89f26e5579e
Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ea89f26e5579e
Vendor: LSI Model: MR9271-8i Revis: 3.40
SCSI Level: 5 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true Is SSD: false
Other Names:
vml.0200000000600605b008d44de01b7ea89f26e5579e4d5239323731
VAAI Status: unsupported
naa.600605b008d44de01b7eaacb4807dcc5
Device Type: Direct-Access
Size: 857353 MB
Display Name: Local LSI Disk (naa.600605b008d44de01b7eaacb4807dcc5)
Multipath Plugin: NMP
Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7eaacb4807dcc5
Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7eaacb4807dcc5
Vendor: LSI Model: MR9271-8i Revis: 3.40
SCSI Level: 5 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true Is SSD: false
Other Names:
vml.0200000000600605b008d44de01b7eaacb4807dcc54d5239323731
VAAI Status: unsupported
naa.600605b008d44de01b7eab464f5a234b
Device Type: Direct-Access
Size: 857353 MB
Display Name: Local LSI Disk (naa.600605b008d44de01b7eab464f5a234b)
Multipath Plugin: NMP
Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7eab464f5a234b
Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7eab464f5a234b
Vendor: LSI Model: MR9271-8i Revis: 3.40
SCSI Level: 5 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true Is SSD: false
Other Names:
vml.0200000000600605b008d44de01b7eab464f5a234b4d5239323731
VAAI Status: unsupported
naa.600605b008d44de01b7ead416d91ca0f
Device Type: Direct-Access
Size: 189781 MB
Display Name: Local LSI Disk (naa.600605b008d44de01b7ead416d91ca0f)
Multipath Plugin: NMP
Console Device: /vmfs/devices/disks/naa.600605b008d44de01b7ead416d91ca0f
Devfs Path: /vmfs/devices/disks/naa.600605b008d44de01b7ead416d91ca0f
Vendor: LSI Model: MR9271-8i Revis: 3.40
SCSI Level: 5 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true Is SSD: true
Other Names:
vml.0200000000600605b008d44de01b7ead416d91ca0f4d5239323731
VAAI Status: unsupported
Thanks a lot!
Silvester
If you see the output of vdq -q you can actually find that VSAN UUID 52b244d6-7cda-613f-46b8-eb786e6207e5, which you mentioned you couldn't find. That is attached to the disk naa.600605b008d44de01b7ea6d30b7c9b9d. So there could be some problem with that disk. To confirm that, do not select this data disk while disk group creation. If that works, then you can be sure there is some problem with that disk.
surprising factor is that, vdq output shows that VSAN cluster creation is successful. You can observe that some of the disk state shows that In-Use for VSAN. Can you also post the output for "esxcli vsan cluster get"
OK, I managed to delete the disk group, and tries to create another disk group, using some other disks than "600605b008d44de01b7ea6d30b7c9b9d", now this time it fails with the following message
"Operation failed, diagnostics report: Sysinfo error on operation returned status : Not found. Please see the VMkernel log for detailed error information"
And I notice this time no disk group is created (vdq -i returns nothing).
Silvester
Be nice to see the vmkernel logs..
Can you post?
a vm-support would be great
Thanks
Hi Paudieo
Here is the VMKernel/vmkwarning/vmksummary log files etracted from the ESX host. I reproduced the error twice before downloading the logs.
With my knowledge I was not able to identify anything abnormal. Any hint/suggestions will be much apprecated!
If you need other log files, please feel free to let me know.
Silvester
Hi Silvester,
the vmkernel logs have some data, but maybe out of context, it would it really be great if you could send me a full vm-support
I need to check out some more details regarding the disk naa = naa.600605b005a783701acbdef240ab6fb8
there are a few events that I need to check out
vm-support not only collects log files , but also some configuration data that would be helpful
Appreciate it !!
Also there is some vsan specific log files that would be included on a vm-support
this would narrow the problem considerably
Hi Paudeio,
If you have reviewed my vm-support log files, would you kindly suggest how I shall proceed?
Thanks!
Silvester