VMware Cloud Community
bgwallace
Enthusiast
Enthusiast
‎02-28-2021 04:38 PM

USB datastore missing after reboot

Team,

Some time back I rebooted my ESXi server and for some reason the USB datastore (which contains a virtual disk that I need to recover) no longer shows up under /vmfs/volumes.  I ran the commands below, but while it seems to be mounted it still doesn't show up.  Any ideas?

[root@VM-Server:/dev/disks] esxcli storage vmfs snapshot list
59612ba3-86f44804-d327-484d7eedce7f
Volume Name: datastore3
VMFS UUID: 59612ba3-86f44804-d327-484d7eedce7f
Can mount: true
Reason for un-mountability:
Can resignature: true
Reason for non-resignaturability:
Unresolved Extent Count: 1

[root@VM-Server:/dev/disks] esxcfg-volume -l
Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).
VMFS UUID/label: 59612ba3-86f44804-d327-484d7eedce7f/datastore3
Can mount: Yes
Can resignature: Yes
Extent name: mpx.vmhba32:C0:T0:L0:1 range: 0 - 2861311 (MB)

[root@VM-Server:/dev/disks] esxcfg-volume -m "59612ba3-86f44804-d327-484d7eedce7f"
Mounting volume 59612ba3-86f44804-d327-484d7eedce7f

Many thanks,
Brad

 

0 Kudos
3 Replies
Lalegre
Virtuoso
Virtuoso
‎02-28-2021 09:49 PM

Hey @bgwallace,

Could be possible that your USB Arbitrator service got running? When this service is running the ESXi use the capability of doing passthrough and because of that it could impact the VMFS datastore.

0 Kudos
bgwallace
Enthusiast
Enthusiast
‎03-01-2021 05:31 AM

Thanks @Lalegre.  I didn’t think to mention but I did verify the USB arbitrator service is stopped.  This is very strange.  It was working fine for a year or more and then just stopped after the reboot.

0 Kudos
bgwallace
Enthusiast
Enthusiast
‎03-01-2021 08:01 PM

Attaching vmkernel.log from the mount operation.  Also, the RAID array is 3TB so I'm wondering if perhaps I'm running into some >2TB issue.  I've been pulling my hair out for a couple of days trying to figure this one out so if anyone can help me debug I would really appreciate it.  Thank you in advance.

 

2021-03-02T03:43:08.791Z cpu1:2099782)WARNING: NFS: 1226: Invalid volume UUID 59612ba2-22951df0-b678-484d7eedce7f
2021-03-02T03:43:09.215Z cpu0:2097174)ScsiDeviceIO: 3448: Cmd(0x459a40e86cc0) 0x2a, CmdSN 0x1d from world 2099782 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x0 0x0.
2021-03-02T03:43:09.215Z cpu0:2097174)NMP: nmp_ThrottleLogForDevice:3872: Cmd 0x2a (0x459a40e86cc0, 2099782) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x0 0x0. Act:NONE
2021-03-02T03:43:49.218Z cpu0:2097174)ScsiDeviceIO: 3483: Cmd(0x459a40e86cc0) 0x2a, CmdSN 0x1d from world 2099782 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2021-03-02T03:43:49.218Z cpu1:2099782)HBX: 817: Writing HB at 3346432 on vol 'datastore3' failed: Timeout
2021-03-02T03:43:49.218Z cpu1:2099782)HBX: 1035: 'datastore3': HB at offset 3346432 - Setting pulse failed: Timeout:
2021-03-02T03:43:49.218Z cpu1:2099782) [HB state abcdef02 offset 3346432 gen 263 stampUS 1131970240 uuid 603db002-b7605b34-ecf4-484d7eedce7f jrnl <FB 0> drv 24.82 lockImpl 3 ip 192.168.24.10]
2021-03-02T03:44:17.245Z cpu0:2097654)DVFilter: 6053: Checking disconnected filters for timeouts
2021-03-02T03:44:30.220Z cpu0:2099782)HBX: 817: Writing HB at 3354624 on vol 'datastore3' failed: Timeout
2021-03-02T03:44:30.220Z cpu0:2099782)HBX: 1035: 'datastore3': HB at offset 3354624 - Setting pulse failed: Timeout:
2021-03-02T03:44:30.220Z cpu0:2099782) [HB state abcdef02 offset 3354624 gen 3 stampUS 1171973178 uuid 603db002-b7605b34-ecf4-484d7eedce7f jrnl <FB 0> drv 24.82 lockImpl 3 ip 192.168.24.10]
2021-03-02T03:44:47.246Z cpu0:2097602)NMP: nmp_ResetDeviceLogThrottling:3586: last error status from device mpx.vmhba32:C0:T0:L0 repeated 4086 times
2021-03-02T03:45:11.221Z cpu0:2099782)HBX: 817: Writing HB at 3358720 on vol 'datastore3' failed: Timeout
2021-03-02T03:45:11.221Z cpu0:2099782)HBX: 1035: 'datastore3': HB at offset 3358720 - Setting pulse failed: Timeout:
2021-03-02T03:45:11.221Z cpu0:2099782) [HB state abcdef02 offset 3358720 gen 3 stampUS 1212974990 uuid 603db002-b7605b34-ecf4-484d7eedce7f jrnl <FB 0> drv 24.82 lockImpl 3 ip 192.168.24.10]
2021-03-02T03:45:11.223Z cpu0:2099782)FS3Misc: 1734: Long VMFS rsv time on 'datastore3' (held for 122008 msecs). # R: 1, # W: 1 bytesXfer: 72 sectors
2021-03-02T03:45:11.235Z cpu0:2099782)HBX: 6493: 'datastore3': HB at offset 3350528 - Marking HB:
2021-03-02T03:45:11.235Z cpu0:2099782) [HB state abcdef02 offset 3350528 gen 499 stampUS 5913864730 uuid 603d98b7-3b2cc6de-0a7d-484d7eedce7f jrnl <FB 0> drv 24.82 lockImpl 3 ip 192.168.24.10]
2021-03-02T03:45:11.247Z cpu2:2099782)HBX: 6600: 'datastore3': HB at offset 3350528 - Marked HB:
2021-03-02T03:45:11.247Z cpu2:2099782) [HB state abcdef01 offset 3350528 gen 500 stampUS 1253989899 uuid 00000000-00000000-0000-000000000000 jrnl <FB 0> drv 24.82 lockImpl 3 ip 192.168.24.10]
2021-03-02T03:45:11.301Z cpu2:2099782)HBX: 6493: 'datastore3': HB at offset 3342336 - Marking HB:
2021-03-02T03:45:11.302Z cpu2:2099782) [HB state abcdef04 offset 3342336 gen 695 stampUS 5780980265 uuid 5c256199-4556f80a-5d4f-484d7eedce7f jrnl <FB 10> drv 24.82 lockImpl 3 ip 192.168.24.10]
2021-03-02T03:45:52.305Z cpu2:2099782)HBX: 817: Writing HB at 3342336 on vol 'datastore3' failed: Timeout
2021-03-02T03:45:52.305Z cpu2:2099782)FS3Misc: 1734: Long VMFS rsv time on 'datastore3' (held for 41003 msecs). # R: 1, # W: 0 bytesXfer: 8 sectors
2021-03-02T03:45:52.305Z cpu2:2099782)HBX: 4703: 1 stale HB slot(s) owned by me have been garbage collected on vol 'datastore3'
2021-03-02T03:46:03.197Z cpu1:2097175)ScsiDeviceIO: 3483: Cmd(0x459a40efd640) 0x2a, CmdSN 0x51 from world 2099782 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2021-03-02T03:46:03.216Z cpu2:2099782)FS3Misc: 2814: 'datastore3': Aborted VMFS IO at 749924352 made it to disk, setting status to OK
2021-03-02T03:46:16.223Z cpu3:2097177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-03-02T03:46:16.223Z cpu3:2097177)ScsiDeviceIO: 3483: Cmd(0x459a40f6f4c0) 0x2a, CmdSN 0x57 from world 2099782 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x6c 0x69.
2021-03-02T03:46:17.221Z cpu2:2099782)J6: 3062: Failed to initialize journal block for volume 'datastore3' with error: Timeout
2021-03-02T03:46:28.199Z cpu2:2099782)FS3Misc: 2814: 'datastore3': Aborted VMFS IO at 749924352 made it to disk, setting status to OK
2021-03-02T03:46:28.258Z cpu1:2099782)FSS: 6092: No FS driver claimed device '59612ba2-22951df0-b678-484d7eedce7f': No filesystem on the device
2021-03-02T03:46:28.258Z cpu1:2099782)LVM: 16776: File system '' (LV 59612ba2-22951df0-b678-484d7eedce7f) mounted in 'rw' mode.

0 Kudos