VC/ESXi 8.0.2 - Can't add FC Datastore

[icesatish]

Hi All,

We have VC 7.0.3, and now we installed additional VC 8.0.2.

When I'm trying to add LUNs (HPE Nimble) that are already exist in the VC 7  to the VC 8 , I can't choose anything in the "Partition Configuration", just see this : "Select partition configuration to continue" . (from the ESXi itself, I can keep going and add the disk, but I got the error : Failed to create VMFS datastore - Failed to update disk partitions for /vmfs/devices/disks/......)

OK, so, I created a NEW Lun on the storage, and try to add it to the ESXi via the VC, and now I can keep going and choose the "Partition Configuration", but getting same error : Failed to create VMFS datastore - Failed to update disk partitions for /vmfs/devices/disks/......

(on the VC 7 I can add it without any issues....)

Do you have any idea ?!

Thanks in advance

[icesatish]

Someone please ??

[navina]

Could you please provide the vmkernel.log and vobd.log

[icesatish]

Thanks for your reply !

Is it a VC or ESXi log ? 

And how can I generate it ?

thanks

[navina]

It is on the ESXi in the location /var/run/log

[icesatish]

Attaching the logs [I got the errors on 26.11.23 somewhere between 15:30 to 16:30]

[navina]

The logs are short does not have the timestamp. Propable because of log rotation.
But the logs matches an old KB https://kb.vmware.com/s/article/2149522

2023-11-26T16:02:19.600Z In(182) vmkernel: cpu72:2098467)nfnic: <1>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1922: in mismatch actual len: 16384 residual: 0
2023-11-26T16:02:19.600Z Wa(180) vmkwarning: cpu72:2098467)WARNING: nfnic: <1>: fnic_fcpio_icmnd_cmpl_handler: 1964: sc: 0x45b9c2b90f00 tag: 0x185 hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!
2023-11-26T16:02:19.600Z In(182) vmkernel: cpu72:2098467)ScsiDeviceIO: 4532: Cmd(0x45b9c2b90f00) 0x88, CmdSN 0x2e24 from world 0 to dev "eui.8fa607ca0b53f4e16c9ce900da6915d1" failed H:0x7 D:0x0 P:0x0
2023-11-26T16:02:19.605Z In(182) vmkernel: cpu72:2098467)nfnic: <1>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1922: in mismatch actual len: 16384 residual: 0
2023-11-26T16:02:19.605Z Wa(180) vmkwarning: cpu72:2098467)WARNING: nfnic: <1>: fnic_fcpio_icmnd_cmpl_handler: 1964: sc: 0x45b9c2b90f00 tag: 0x186 hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!
2023-11-26T16:02:19.609Z In(182) vmkernel: cpu72:2098467)nfnic: <1>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1922: in mismatch actual len: 16384 residual: 0
2023-11-26T16:02:19.609Z Wa(180) vmkwarning: cpu72:2098467)WARNING: nfnic: <1>: fnic_fcpio_icmnd_cmpl_handler: 1964: sc: 0x45b9c2b90f00 tag: 0x187 hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!
2023-11-26T16:02:19.609Z In(182) vmkernel: cpu0:2099837)Partition: 607: Read of GPT entries (hdrlba=21474836479) failed on "eui.8fa607ca0b53f4e16c9ce900da6915d1": I/O error
2023-11-26T16:02:19.609Z In(182) vmkernel: cpu0:2099837)Partition: 749: Read from both primary and secondary gpt table failed on "eui.8fa607ca0b53f4e16c9ce900da6915d1".
2023-11-26T16:02:19.609Z Wa(180) vmkwarning: cpu0:2099837)WARNING: Partition: 1387: Partition table read from device eui.8fa607ca0b53f4e16c9ce900da6915d1 failed: I/O error

To dig further provide the naa id or the eui id of the lun.

[icesatish]

Thanks again for your reply 

I don't think the old BUG is the issue, because I'm using a new UCS-X series 

I've tried again in order to generate new logs with current time stamp . new logs attached 

the new LUN I created

/vmfs/devices/disks/eui.417cbd41b63843b66c9ce900da6915d1.

The old LUN (that in used in the 'old' env.) 

/vmfs/devices/disks/eui.ce508a937a44cad66c9ce900da6915d1.

BTW - I saw that from the VCenter, I can see that the ESXi time is correct (10:46), but from the ESXi itself it shows at 08:46 [that's why I played little bit with the NTP now]

Can it be an issue too ? 

Thanks !

[navina]

2023-11-28T08:50:25.944Z In(182) vmkernel: cpu72:2098467)NMP: nmp_ThrottleLogForDevice:3864: Cmd 0x2a (0x45d9db109040, 3181513) to dev "eui.417cbd41b63843b66c9ce900da6915d1" on path "vmhba0:C0:T3:L33" Failed:
2023-11-28T08:50:25.944Z In(182) vmkernel: cpu72:2098467)NMP: nmp_ThrottleLogForDevice:3869: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430a12d57c80 CmdSN 0x3
2023-11-28T08:50:25.944Z Wa(180) vmkwarning: cpu72:2098467)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "eui.417cbd41b63843b66c9ce900da6915d1" state in doubt; requested fast path state update...
2023-11-28T08:50:25.944Z In(182) vmkernel: cpu72:2098467)ScsiDeviceIO: 4516: Cmd(0x45d9db109040) 0x2a, cmdId.initiator=0x430a12d57c80 CmdSN 0x3 from world 3181513 to dev "eui.417cbd41b63843b66c9ce900da6915d1" failed H:0x5 D:0x0 P:0x0 Cancelled from driver layer. Cmd

Looks like a driver issue to https://kb.vmware.com/s/article/1029039
Can you follow https://kb.vmware.com/s/article/1027206 and update the HBA Driver.

[icesatish]

Here is the output :

0000:1c:00.2 1137:0045 1137:02cf vmkernel vmhba0

0000:1c:00.3 1137:0045 1137:02cf vmkernel vmhba1

I need to take a look now on the compatibility guide as far as I understand

Will check it

[icesatish]

Please see attached screenshots

screenshot 01 -

When choosing "FC", there was no result.

screenshot 02 -

When I chose "All", I do got a result 

[Screenshot 03 - when I chose "All", I click on the VIC14425 Link, and downloaded both software. if it's the correct thing, what should I do ? install those as VIBs on the ESXi ? if so , how ?]

How to get progress here ? I don't know how to continue

Thanks so much !

[icesatish]

Navina ? Can you please help with an answer ?

Thanks again 

[DanRobinsonHP]

Vendor ID 1137 is Cisco, I would open a case with them to find out why it's not working on 8 but is on 7, as mentioned smells like a driver issue.

[icesatish · Accepted solution]

So, after open a TAC case for Cisco, the issue was that the FCoE Port-Channel between the Nexus 9k to the Fabric-Interconnect wasn't configured properly.

The mistake was because we used to work with Nexus 5k, and the config in Nexus 9k is different.

Here is what was needed to be configured on the Nexuses to make it work (of course you should change the "Po" and the "vfc" numbers according to your environment ) :

Switch# conf t

switch(config)# system qos

switch(config-config-sys-qos)# service-policy type queuing input default-fcoe-in-que-policy

switch(config-config-sys-qos)# service-policy type queuing output default-fcoe-out-policy

switch(config-config-sys-qos)# service-policy type network-qos default-fcoe-nq-policy

switch(config-config-sys-qos)# interface po120

switch(config-if)# mtu 9216

switch(config-if)# service-policy input default-fcoe-in-policy

switch(config-if)# shut

switch(config-if)# no shut

switch(config-if)# interface vfc20

switch(config-if)# shut

switch(config-if)# no shut

thanks all for the assistant !

[DanRobinsonHP]

Glad you got it fixed and thanks for the follow up.