VMware Cloud Community
Cyril2021
Contributor
Contributor
‎10-07-2021 07:50 AM

ESXi 7.0.3 PSOD

24 hours after installing 7.0.3 (vCenter and 1 of the 3 hosts), the upgraded host crashed with a PSOD.

[code]

2021-10-07T12:42:05.306Z cpu18:2097364)StorageDevice: 7059: End path evaluation for device naa.600508b1001c472cd365969f181995a9
2021-10-07T12:47:05.302Z cpu18:2097364)StorageDevice: 7059: End path evaluation for device mpx.vmhba32:C0:T0:L0
2021-10-07T12:47:05.302Z cpu18:2097364)StorageDevice: 7059: End path evaluation for device naa.600508b1001c472cd365969f181995a9
2021-10-07T12:52:05.297Z cpu18:2097364)StorageDevice: 7059: End path evaluation for device mpx.vmhba32:C0:T0:L0
2021-10-07T12:52:05.298Z cpu18:2097364)StorageDevice: 7059: End path evaluation for device naa.600508b1001c472cd365969f181995a9
2021-10-07T12:54:54.503Z cpu10:2097207)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x85 (0x45d8d177da48, 2099565) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed:
2021-10-07T12:54:54.503Z cpu10:2097207)NMP: nmp_ThrottleLogForDevice:3875: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x0 0x0. Act:NONE. cmdId.initiator=0x4305de73bf00 CmdSN 0x10d
2021-10-07T12:54:54.503Z cpu10:2097207)ScsiDeviceIO: 4161: Cmd(0x45d8d177da48) 0x85, CmdSN 0x10d from world 2099565 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x0 0x0
2021-10-07T12:54:54.503Z cpu10:2097207)ScsiDeviceIO: 4161: Cmd(0x45d8d177da48) 0x85, CmdSN 0x10e from world 2099565 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x0 0x0
2021-10-07T12:55:35.293Z cpu3:2097677)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device mpx.vmhba32:C0:T0:L0 repeated 1 times
2021-10-07T12:57:05.293Z cpu18:2097364)StorageDevice: 7059: End path evaluation for device mpx.vmhba32:C0:T0:L0
2021-10-07T12:57:05.294Z cpu18:2097364)StorageDevice: 7059: End path evaluation for device naa.600508b1001c472cd365969f181995a9
2021-10-07T13:02:05.288Z cpu18:2097364)StorageDevice: 7059: End path evaluation for device mpx.vmhba32:C0:T0:L0
2021-10-07T13:02:05.289Z cpu18:2097364)StorageDevice: 7059: End path evaluation for device naa.600508b1001c472cd365969f181995a9
2021-10-07T13:07:05.284Z cpu18:2097364)StorageDevice: 7059: End path evaluation for device mpx.vmhba32:C0:T0:L0
2021-10-07T13:07:05.287Z cpu18:2097364)StorageDevice: 7059: End path evaluation for device naa.600508b1001c472cd365969f181995a9
2021-10-07T13:12:05.282Z cpu18:2097364)StorageDevice: 7059: End path evaluation for device mpx.vmhba32:C0:T0:L0
2021-10-07T13:12:05.283Z cpu18:2097364)StorageDevice: 7059: End path evaluation for device naa.600508b1001c472cd365969f181995a9
2021-10-07T13:17:05.276Z cpu12:2097364)StorageDevice: 7059: End path evaluation for device mpx.vmhba32:C0:T0:L0
2021-10-07T13:17:05.276Z cpu12:2097364)StorageDevice: 7059: End path evaluation for device naa.600508b1001c472cd365969f181995a9
2021-10-07T13:21:53.335Z cpu3:2100702)FSS: 7377: Failed to open file 'hpilo-d0ccb0'; Requested flags 0x5, world: 2100702 [sfcb-smx], (Existing flags 0x5, world: 2102177 [sfcb-smx]): Busy
2021-10-07T13:21:53.436Z cpu3:2100702)FSS: 7377: Failed to open file 'hpilo-d0ccb0'; Requested flags 0x5, world: 2100702 [sfcb-smx], (Existing flags 0x5, world: 2102177 [sfcb-smx]): Busy
2021-10-07T13:22:05.273Z cpu19:2097364)StorageDevice: 7059: End path evaluation for device mpx.vmhba32:C0:T0:L0
2021-10-07T13:22:05.274Z cpu19:2097364)StorageDevice: 7059: End path evaluation for device naa.600508b1001c472cd365969f181995a9
2021-10-07T13:22:28.917Z cpu11:2103235)VSCSI: 2973: handle 9033325545005873(GID:9009)(vscsi0:1):Added handle (refCnt = 3) to vscsiResetHandleList vscsiResetHandleCount = 1
2021-10-07T13:22:28.917Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192707
2021-10-07T13:22:28.918Z cpu10:2097352)VSCSI: 3335: handle 9033325545005873(GID:9009)(vscsi0:1):Reset [Retries: 0/0] from (vmm0:MTUSRV3)
2021-10-07T13:22:29.420Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:29.922Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:30.422Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:30.922Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:31.422Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:31.924Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:32.426Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:32.566Z cpu11:2103235)WARNING: VSCSI: 3967: handle 9033325545005873(GID:9009)(vscsi0:1):WaitForCIF: Issuing reset; number of CIF:2
2021-10-07T13:22:32.566Z cpu11:2103235)WARNING: VSCSI: 2986: handle 9033325545005873(GID:9009)(vscsi0:1):Ignoring double reset
2021-10-07T13:22:32.928Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:33.430Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:33.930Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:34.430Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:34.932Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:35.271Z cpu11:2097651)WARNING: Heartbeat: 827: PCPU 16 didn't have a heartbeat for 7 seconds, timeout is 14, 1 IPIs sent; *may* be locked up.
2021-10-07T13:22:35.271Z cpu16:2342869)ALERT: NMI: 689: NMI IPI: RIPOFF(base):RBP:CS [0x10d913b(0x42001d600000):0x43128599fa30:0xf48] (Src 0x1, CPU16)
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1b7f0:[0x42001e6d913a]J6_NewOnDiskTxn@esx#nover+0x177 stack: 0x43128734ff50
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1b850:[0x42001e6d967d]J6CommitInMemTxn@esx#nover+0x176 stack: 0x1
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1b900:[0x42001e6d618a]J6_CommitMemTransaction@esx#nover+0xe3 stack: 0xec500000035
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1b950:[0x42001e6fbad4]Fil6_UnmapTxn@esx#nover+0x4fd stack: 0x0
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1ba60:[0x42001e6ff891]Fil6UpdateBlocks@esx#nover+0x4e2 stack: 0xff
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1bae0:[0x42001e6bf3fe]Fil3UpdateBlocks@esx#nover+0xeb stack: 0x8737fa00
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1bbe0:[0x42001e6c0425]Fil3_PunchFileHoleWithRetry@esx#nover+0x7e stack: 0x45389ef1bd70
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1bc90:[0x42001e6c0c0d]Fil3_FileBlockUnmap@esx#nover+0x57e stack: 0x43128585b250
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1bd40:[0x42001d63b5fb]FSSVec_FileBlockUnmap@vmkernel#nover+0x20 stack: 0xd3d7d2bcdeb8
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1bd50:[0x42001eb28f96]CBT_Ioctl@(cbt)#<None>+0xd3 stack: 0x5e298880
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1bdf0:[0x42001d65f912]DevFSFileBlockUnmap@vmkernel#nover+0x24f stack: 0x4306224157f0
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1be90:[0x42001d63b5fb]FSSVec_FileBlockUnmap@vmkernel#nover+0x20 stack: 0x430a2ac00ff0
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1bea0:[0x42001db52c03]VSCSI_ExecFSSUnmap@vmkernel#nover+0x9c stack: 0x430a2ac2ad40
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1bf10:[0x42001db50ead]VSCSIDoEmulHelperIO@vmkernel#nover+0x2a stack: 0x4300b8201220
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1bf40:[0x42001d6d9c19]HelperQueueFunc@vmkernel#nover+0x1d2 stack: 0x45389ef20b48
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1bfe0:[0x42001d9b1775]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0
2021-10-07T13:22:35.271Z cpu16:2342869)0x45389ef1c000:[0x42001d6c46ff]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0
2021-10-07T13:22:35.434Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:35.934Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:36.436Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:36.938Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:37.440Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:37.940Z cpu10:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:38.442Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:38.943Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:39.443Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:39.943Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:40.445Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:40.947Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:41.449Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:41.951Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:42.452Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:42.955Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:43.455Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:43.957Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:44.459Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:44.960Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:45.460Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:45.960Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:46.462Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:46.962Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:47.464Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:47.964Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:48.466Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706
2021-10-07T13:22:48.966Z cpu14:2097352)VSCSI: 3226: handle 9033325545005873(GID:9009)(vscsi0:1):processing reset for handle ... state 1381192706

[/code]

Any idea what this could be?
Running on a HP DL380 G9 server with 24 disks in raid 10 ADM.

Preview file
2836 KB
0 Kudos
20 Replies
maksym007
Expert
Expert
‎11-23-2022 10:03 AM

maybe it makes sense to open a SR to your Servers Vendor. 

0 Kudos