I'm currently in the process of setting up 2 ESXi hosts. I have one up and running with several VM's running on iscsi datastores.
Randomly since setup the host becomes unresponsive. I can't ping the host nor any of the VM's.
The host is accessible via BMC, but only enough to log in. After I login it doesn't do anything and I'm forced to reboot the server.
I attached the vmkernel log file in the hopes someone can see something I can't.
Edit:
Running on ESXi 6.0.0 Bild 3029758
While I don't see the exact same error occurring in your vmkernel.log, it's possible you are running into this:
The build number in your vmkernel.log shows as:
0:00:00:06.697 cpu0:32768)Init: 745: vmkernel build Number = 3029758
which is 6.0U1. The above link was a nasty bug with NICs and CPU interrupts that occurred on specific generations of processors if I remember correctly. I saw it most on servers that were equivalent in release dates as the x10 Dell series (i.e. R910). It tanked the whole host with circumstances that sound similar to yours as this is how this issue manifested itself from the outside when the problem was occurring:
"The host is accessible via BMC, but only enough to log in. After I login it doesn't do anything"
You might want to upgrade to at least 6.0U1a to take that out of the equation. If you go that route, make sure to upgrade your vCenter to the 6.0U1a version or higher before upgrading to the same version on the host or your host may not rejoin vCenter properly.
Hi,
In the logs iSCSI volumes / Paths are disconnecting from the hosts and this can have that behavior in the hosts.
So check your Storage connections and also the Storage network(ESXi host side and Storage side).
Jail
Thanks for the information. It does sound similar with the issue I am seeing. I will update both hosts as soon as I can.
Hi!
you have the following issue
2017-10-13T12:44:00.020Z cpu2:33118)NMP: nmp_ResetDeviceLogThrottling:3339: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x2 0x3a 0x1 from dev "mpx.vmhba34:C0:T0:L0" occurred 4 times(of 4 commands)
0x2 0x3a 0x1 means
0x2 - NOT READY
0x3a 0x1 - MEDIUM NOT PRESENT - TRAY CLOSED
Something is wrong with your local disks.
It's hard to say something more without full log bundle.
Also I see very long scsi reservation time on datastore1, but I can't say which is a corret path for it
2017-10-04T12:38:59.844Z cpu3:35149 opID=6d3e4b78)FS3Misc: 1759: Long VMFS rsv time on 'datastore1' (held for 528 msecs). # R: 2, # W: 1 bytesXfer: 6 sectors
2017-10-04T12:39:06.698Z cpu0:35221)FS3Misc: 1759: Long VMFS rsv time on 'datastore1' (held for 487 msecs). # R: 2, # W: 1 bytesXfer: 6 sectors
What's your HW?
Hi thanks for your response, i'm a newbie in esxi servers.
What do you mean with full log bundle?
This is my machine hardware, i began to learn in a normal computer, not a professional server
Display Name: Local ASUS CD-ROM (mpx.vmhba34:C0:T0:L0)
Vendor: ASUS Model: DRW-24F1ST c Revis: 1.00
Display Name: Local ATA Disk (t10.ATA_____ST1000DM0032D1SB10C__________________________________Z9A27DA9)
Vendor: ATA Model: ST1000DM003-1SB1 Revis: CC43
Intel(R) Core(TM) i5-4460 CPU @ 3.20GHz
Gigabyte Technology Co., Ltd.
B85M-D3H
8Gb 4CPU x 3.192GHZ
CPU
CPU Packages: 1
CPU Cores: 4
CPU Threads: 4
Hyperthreading Active: false
Hyperthreading Supported: false
Hyperthreading Enabled: true
HV Support: 3
HV Replay Capable: true
PCI
0000:00:00.0
Address: 0000:00:00.0
Segment: 0x0000
Bus: 0x00
Slot: 0x00
Function: 0x0
VMkernel Name:
Vendor Name: Intel Corporation
Device Name: Haswell DRAM Controller
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x0c00
SubVendor ID: 0x1458
SubDevice ID: 0x5000
Device Class: 0x0600
Device Class Name: Host bridge
Programming Interface: 0x00
Revision ID: 0x06
Interrupt Line: 0xff
IRQ: 255
Interrupt Vector: 0x00
PCI Pin: 0xff
Spawned Bus: 0x00
Flags: 0x0200
Module ID: -1
Module Name: None
Chassis: 0
Physical Slot: 4294967295
Slot Description:
Passthru Capable: false
Parent Device:
Dependent Device:
Reset Method: None
FPT Sharable: false
0000:00:02.0
Address: 0000:00:02.0
Segment: 0x0000
Bus: 0x00
Slot: 0x02
Function: 0x0
VMkernel Name:
Vendor Name: Intel Corporation
Device Name: Haswell Integrated Graphics Controller
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x0412
SubVendor ID: 0x1458
SubDevice ID: 0xd000
Device Class: 0x0300
Device Class Name: VGA compatible controller
Programming Interface: 0x00
Revision ID: 0x06
Interrupt Line: 0x0b
IRQ: 11
Interrupt Vector: 0x2c
PCI Pin: 0x00
Spawned Bus: 0x00
Flags: 0x0221
Module ID: -1
Module Name: None
Chassis: 0
Physical Slot: 0
Slot Description: J6B2
Passthru Capable: true
Parent Device:
Dependent Device: PCI 0:0:2:0
Reset Method: Function reset
FPT Sharable: true
0000:00:03.0
Address: 0000:00:03.0
Segment: 0x0000
Bus: 0x00
Slot: 0x03
Function: 0x0
VMkernel Name:
Vendor Name: Intel Corporation
Device Name: Haswell HD Audio Controller
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x0c0c
SubVendor ID: 0x8086
SubDevice ID: 0x2010
Device Class: 0x0403
Device Class Name: Audio device
Programming Interface: 0x00
Revision ID: 0x06
Interrupt Line: 0x0b
IRQ: 11
Interrupt Vector: 0x2c
PCI Pin: 0x00
Spawned Bus: 0x00
Flags: 0x0201
Module ID: -1
Module Name: None
Chassis: 0
Physical Slot: 3
Slot Description: J7B1
Passthru Capable: true
Parent Device:
Dependent Device: PCI 0:0:3:0
Reset Method: Function reset
FPT Sharable: true
0000:00:14.0
Address: 0000:00:14.0
Segment: 0x0000
Bus: 0x00
Slot: 0x14
Function: 0x0
VMkernel Name:
Vendor Name: Intel Corporation
Device Name: Lynx Point USB xHCI Host Controller
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x8c31
SubVendor ID: 0x1458
SubDevice ID: 0x5007
Device Class: 0x0c03
Device Class Name: USB controller
Programming Interface: 0x30
Revision ID: 0x05
Interrupt Line: 0x0b
IRQ: 11
Interrupt Vector: 0x32
PCI Pin: 0x00
Spawned Bus: 0x00
Flags: 0x0201
Module ID: 4126
Module Name: xhci
Chassis: 0
Physical Slot: 4294967295
Slot Description:
Passthru Capable: false
Parent Device:
Dependent Device:
Reset Method: None
FPT Sharable: false
0000:00:16.0
Address: 0000:00:16.0
Segment: 0x0000
Bus: 0x00
Slot: 0x16
Function: 0x0
VMkernel Name:
Vendor Name: Intel Corporation
Device Name: Lynx Point MEI Controller #1
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x8c3a
SubVendor ID: 0x1458
SubDevice ID: 0x1c3a
Device Class: 0x0780
Device Class Name: Communication controller
Programming Interface: 0x00
Revision ID: 0x04
Interrupt Line: 0x0b
IRQ: 11
Interrupt Vector: 0x2c
PCI Pin: 0x00
Spawned Bus: 0x00
Flags: 0x0201
Module ID: -1
Module Name: None
Chassis: 0
Physical Slot: 4294967295
Slot Description:
Passthru Capable: false
Parent Device:
Dependent Device:
Reset Method: None
FPT Sharable: false
0000:00:16.3
Address: 0000:00:16.3
Segment: 0x0000
Bus: 0x00
Slot: 0x16
Function: 0x3
VMkernel Name:
Vendor Name: Intel Corporation
Device Name: Lynx Point KT Controller
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x8c3d
SubVendor ID: 0x1458
SubDevice ID: 0x1c3a
Device Class: 0x0700
Device Class Name: Serial controller
Programming Interface: 0x02
Revision ID: 0x04
Interrupt Line: 0x0a
IRQ: 10
Interrupt Vector: 0x2d
PCI Pin: 0x01
Spawned Bus: 0x00
Flags: 0x0201
Module ID: -1
Module Name: None
Chassis: 0
Physical Slot: 4294967295
Slot Description:
Passthru Capable: false
Parent Device:
Dependent Device:
Reset Method: None
FPT Sharable: false
0000:00:1a.0
Address: 0000:00:1a.0
Segment: 0x0000
Bus: 0x00
Slot: 0x1a
Function: 0x0
VMkernel Name:
Vendor Name: Intel Corporation
Device Name: Lynx Point USB Enhanced Host Controller #2
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x8c2d
SubVendor ID: 0x1458
SubDevice ID: 0x5006
Device Class: 0x0c03
Device Class Name: USB controller
Programming Interface: 0x20
Revision ID: 0x05
Interrupt Line: 0x0b
IRQ: 11
Interrupt Vector: 0x2c
PCI Pin: 0x00
Spawned Bus: 0x00
Flags: 0x0201
Module ID: 4125
Module Name: ehci-hcd
Chassis: 0
Physical Slot: 4294967295
Slot Description:
Passthru Capable: true
Parent Device:
Dependent Device: PCI 0:0:26:0
Reset Method: Function reset
FPT Sharable: true
0000:00:1b.0
Address: 0000:00:1b.0
Segment: 0x0000
Bus: 0x00
Slot: 0x1b
Function: 0x0
VMkernel Name:
Vendor Name: Intel Corporation
Device Name: Lynx Point High Definition Audio Controller
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x8c20
SubVendor ID: 0x1458
SubDevice ID: 0xa002
Device Class: 0x0403
Device Class Name: Audio device
Programming Interface: 0x00
Revision ID: 0x05
Interrupt Line: 0x03
IRQ: 3
Interrupt Vector: 0x2e
PCI Pin: 0x00
Spawned Bus: 0x00
Flags: 0x0201
Module ID: -1
Module Name: None
Chassis: 0
Physical Slot: 4294967295
Slot Description:
Passthru Capable: true
Parent Device:
Dependent Device: PCI 0:0:27:0
Reset Method: Function reset
FPT Sharable: true
0000:00:1c.0
Address: 0000:00:1c.0
Segment: 0x0000
Bus: 0x00
Slot: 0x1c
Function: 0x0
VMkernel Name: PCIe RP[0000:00:1c.0]
Vendor Name: Intel Corporation
Device Name: Lynx Point PCI Express Root Port #1
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x8c10
SubVendor ID: 0x0000
SubDevice ID: 0x0000
Device Class: 0x0604
Device Class Name: PCI bridge
Programming Interface: 0x00
Revision ID: 0xd5
Interrupt Line: 0x0b
IRQ: 11
Interrupt Vector: 0x2c
PCI Pin: 0x00
Spawned Bus: 0x01
Flags: 0x0203
Module ID: 0
Module Name: vmkernel
Chassis: 0
Physical Slot: 1
Slot Description: J6B1
Passthru Capable: false
Parent Device:
Dependent Device:
Reset Method: None
FPT Sharable: false
0000:00:1c.2
Address: 0000:00:1c.2
Segment: 0x0000
Bus: 0x00
Slot: 0x1c
Function: 0x2
VMkernel Name: PCIe RP[0000:00:1c.2]
Vendor Name: Intel Corporation
Device Name: Lynx Point PCI Express Root Port #3
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x8c14
SubVendor ID: 0x0000
SubDevice ID: 0x0000
Device Class: 0x0604
Device Class Name: PCI bridge
Programming Interface: 0x00
Revision ID: 0xd5
Interrupt Line: 0x0b
IRQ: 11
Interrupt Vector: 0x2f
PCI Pin: 0x02
Spawned Bus: 0x02
Flags: 0x0203
Module ID: 0
Module Name: vmkernel
Chassis: 0
Physical Slot: 1
Slot Description: J6B1
Passthru Capable: false
Parent Device:
Dependent Device:
Reset Method: None
FPT Sharable: false
0000:00:1d.0
Address: 0000:00:1d.0
Segment: 0x0000
Bus: 0x00
Slot: 0x1d
Function: 0x0
VMkernel Name:
Vendor Name: Intel Corporation
Device Name: Lynx Point USB Enhanced Host Controller #1
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x8c26
SubVendor ID: 0x1458
SubDevice ID: 0x5006
Device Class: 0x0c03
Device Class Name: USB controller
Programming Interface: 0x20
Revision ID: 0x05
Interrupt Line: 0x0a
IRQ: 10
Interrupt Vector: 0x30
PCI Pin: 0x00
Spawned Bus: 0x00
Flags: 0x0201
Module ID: 4125
Module Name: ehci-hcd
Chassis: 0
Physical Slot: 4294967295
Slot Description:
Passthru Capable: true
Parent Device:
Dependent Device: PCI 0:0:29:0
Reset Method: Function reset
FPT Sharable: true
0000:00:1f.0
Address: 0000:00:1f.0
Segment: 0x0000
Bus: 0x00
Slot: 0x1f
Function: 0x0
VMkernel Name:
Vendor Name: Intel Corporation
Device Name: Lynx Point LPC Controller
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x8c50
SubVendor ID: 0x1458
SubDevice ID: 0x5001
Device Class: 0x0601
Device Class Name: ISA bridge
Programming Interface: 0x00
Revision ID: 0x05
Interrupt Line: 0xff
IRQ: 255
Interrupt Vector: 0x00
PCI Pin: 0xff
Spawned Bus: 0x00
Flags: 0x0200
Module ID: -1
Module Name: None
Chassis: 0
Physical Slot: 4294967295
Slot Description:
Passthru Capable: false
Parent Device:
Dependent Device:
Reset Method: None
FPT Sharable: false
0000:00:1f.2
Address: 0000:00:1f.2
Segment: 0x0000
Bus: 0x00
Slot: 0x1f
Function: 0x2
VMkernel Name: vmhba0
Vendor Name: Intel Corporation
Device Name: Lynx Point AHCI Controller
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x8c02
SubVendor ID: 0x1458
SubDevice ID: 0xb005
Device Class: 0x0106
Device Class Name: SATA controller
Programming Interface: 0x01
Revision ID: 0x05
Interrupt Line: 0x0a
IRQ: 10
Interrupt Vector: 0x2d
PCI Pin: 0x01
Spawned Bus: 0x00
Flags: 0x0201
Module ID: 4161
Module Name: ahci
Chassis: 0
Physical Slot: 4294967295
Slot Description:
Passthru Capable: false
Parent Device:
Dependent Device:
Reset Method: None
FPT Sharable: false
0000:00:1f.3
Address: 0000:00:1f.3
Segment: 0x0000
Bus: 0x00
Slot: 0x1f
Function: 0x3
VMkernel Name:
Vendor Name: Intel Corporation
Device Name: Lynx Point SMBus Controller
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x8086
Device ID: 0x8c22
SubVendor ID: 0x1458
SubDevice ID: 0x5001
Device Class: 0x0c05
Device Class Name: SMBus
Programming Interface: 0x00
Revision ID: 0x05
Interrupt Line: 0x0b
IRQ: 255
Interrupt Vector: 0x00
PCI Pin: 0x02
Spawned Bus: 0x00
Flags: 0x0201
Module ID: -1
Module Name: None
Chassis: 0
Physical Slot: 4294967295
Slot Description:
Passthru Capable: false
Parent Device:
Dependent Device:
Reset Method: None
FPT Sharable: false
0000:02:00.0
Address: 0000:02:00.0
Segment: 0x0000
Bus: 0x02
Slot: 0x00
Function: 0x0
VMkernel Name: vmnic0
Vendor Name: Realtek Semiconductor Co., Ltd.
Device Name: Motherboard
Configured Owner: Unknown
Current Owner: VMkernel
Vendor ID: 0x10ec
Device ID: 0x8168
SubVendor ID: 0x1458
SubDevice ID: 0xe000
Device Class: 0x0200
Device Class Name: Ethernet controller
Programming Interface: 0x00
Revision ID: 0x06
Interrupt Line: 0x0b
IRQ: 11
Interrupt Vector: 0x31
PCI Pin: 0x00
Spawned Bus: 0x00
Flags: 0x0201
Module ID: 4123
Module Name: r8168
Chassis: 0
Physical Slot: 4294967295
Slot Description: J6B1; relative bdf 00:00.0
Passthru Capable: true
Parent Device: PCI 0:0:28:2
Dependent Device: PCI 0:2:0:0
Reset Method: Bridge reset
FPT Sharable: true
I seem to be having a similar issue with a VM locking up on ESXi 6.5 using a NUC Skull Canyon and was hoping somebody could help me out. This host does have a lot of USB peripherals attached to it and only this Windows VM with 2 sound cards and a USB to Serial adapter seems to be locking up.
I am running 3 Windows VM's and 2 Debian VM's.