Hardware RAID 1 and ESXi - host not seeing virtual volume

[hoppah]

Please bear with me as I try to explain this problem fully.

I have a server running ESXi.  It has a number of VMs and has been working.  This server has an LSI RAID controller and had a single hard drive attached to this controller.  I am allergic to non-redundant drives, so I added a second drive and, from the RAID controller's boot BIOS (pre-boot of the server itself) I created the mirror and had the controller "rebuild" the second drive, copying an image of the first drive over it.  The controller did so and displays a single virtual volume (in its BIOS) as a RAID 1 volume comprised of the two physical disks.

At this point I would expect any server OS to see the single volume.  However, ESXi, on boot, sees both physical hard disks (seemingly ignoring the controller's supposed configuration, which is strange to me), and since they both contain identical copies of the ESXi install, it purple-screens complaining about two identical UUIDs.  When I bypass this error with the command-line boot option "overrideDuplicateImageDetection", ESXi boots from one of the two drives.  vSphere shows both drives as physical drives mounted in the system.

I am relatively certain that the RAID card is maintaining the mirror.  The second drive is not part of a datastore.  I'd prefer ESXi see the mirror as the single logical volume the RAID card is supposedly offering, but I will settle for ESXi ignoring the second drive altogether.

Any ideas?

Thanks in advance for your assistance,

H.

[a_p_]

It seems the RAID controller in your system is not a real hardware RAID controller, but a software (fake) RAID controller, which requires drivers in the OS to operate. ESXi only supports hardware RAID controllers which transparently present the physical disks as a logical volume to the OS.

André

[AKU076 · Accepted solution]

Hi,

Some RAID controllers operates in RAID or Non-RAID depending on the configuration. If no RAID are created it will operate as generic SCSI cards and when a RAID is created then it will become a proper raid controllers. When you installed the ESXi the card was probably operating in a non-RAID mode and later you converted to a RAID controller mode. Please check the drivers loaded in the ESXi kernel and if necessary please include the RAID controller module and re build the boot image.

Is it possible to let me know the model number of LSI RAID you are using and the output of kernel module listing. I think these dual mode controllers were possible in old server models, not sure they are still available. I used to see this kind of issues with Linux in olden days.

[hoppah]

Thank you - that makes a lot of sense.  The card was definitely in non-RAID mode when ESXi was installed - I only had one drive at the time.  Can you tell me how to check for the RAID module or to get an output of the kernel modules?

H.

[hoppah]

Dear a.p.,

if it were a software RAID, how could I build the RAID and create the mirror as well as rebuild the mirror drive pre-boot from the RAID card's BIOS, with no OS in RAM?

H.

[AKU076]

esxcli system module list or vmkload_mod -l will list the whole modules ,

Can you please let me know the exact hardware model ( RAID controller) you are using ?

[hoppah]

Output of esxcli system module list command:

VMware offers supported, powerful system administration tools.  Please

see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the

vSphere Security documentation for more information.

~ #

~ #

~ # esxcli system module list

Name                           Is Loaded  Is Enabled

-----------------------------  ---------  ----------

vmkernel                            true        true

chardevs                            true        true

user                                true        true

procfs                              true        true

vfat                                true        true

vmkapi_socket                       true        true

vmkapi_v2_0_0_0_vmkernel_shim       true        true

vmkplexer                           true        true

vmklinux_9                          true        true

vmklinux_9_2_0_0                    true        true

vmklinux_9_2_1_0                    true        true

random                              true        true

usb                                 true        true

ehci-hcd                            true        true

hid                                 true        true

ipmi_msghandler                     true        true

ipmi_si_drv                         true        true

ipmi_devintf                        true        true

procMisc                            true        true

iscsi_trans                         true        true

etherswitch                         true        true

netsched                            true        true

cnic_register                       true        true

e1000e                              true        true

igb                                 true        true

vmkapi_v2_0_0_0_iscsi_shim          true        true

vmci                                true        true

vmkapei                             true        true

healthchk                           true        true

teamcheck                           true        true

vlanmtucheck                        true        true

heartbeat                           true        true

shaper                              true        true

lldp                                true        true

cdp                                 true        true

ipfix                               true        true

tcpip3                              true        true

dvsdev                              true        true

vdl2                                true        true

dvfilter                            true        true

lacp                                true        true

vmkapi_v2_0_0_0_dvfilter_shim       true        true

dvfilter-generic-fastpath           true        true

vmkstatelogger                      true        true

svmmirror                           true        true

cbt                                 true        true

migrate                             true        true

esxfw                               true        true

dm                                  true        true

nmp                                 true        true

vmw_satp_local                      true        true

vmw_satp_default_aa                 true        true

vmw_psp_lib                         true        true

vmw_psp_fixed                       true        true

vmw_psp_rr                          true        true

vmw_psp_mru                         true        true

libata_92                           true        true

libata_9_2_0_0                      true        true

usb-storage                         true        true

vmkapi_v2_0_0_0_nmp_shim            true        true

hbr_filter                          true        true

libfc_92                            true        true

libfcoe_92                          true        true

libfc_9_2_0_0                       true        true

libfcoe_9_2_0_0                     true        true

rste                                true        true

lvmdriver                           true        true

deltadisk                           true        true

sunrpc                              true        true

nfsclient                           true        true

vprobe                              true        true

vmkibft                             true        true

vmfs3                               true        true

~ #

I am not certain of the RAID controller model -it's an LSI controller, but I'd have to reboot to see any info about it and I've done too much of that lately - don't want to reboot again until I absolutely have to.

H.

[hoppah]

RE: the controller... does MegaRaid 2 sound familiar?

H.

[AKU076]

Exactly , This is the card which i have seen in old dell servers which works in both modes ( SCSI and RAID). In your driver list there is no megaraid driver loaded. If i am not wrong i hope you are using two SATA drives  to create a RAID1. And VMFS is hosted on a different storage ?

As you can see your controllers are still using libata controllers which makes the system to work as normal scsi cards. Can you please make sure this server you are using is supported by VMware for the specific version.

Now to go further i need following details: The version of ESX you are using, and server model you have. Also lspci -v output from console the esxi.

We have two work arounds :

1. Re-built the boot image ( depends on ESX version).  to include the Megaraid2 controller driver - This is a tricky one and need to be very careful not to break the server which is currently booting.

2. Re-install the ESX preserving the VMFS, the new installation will detect the megaraid as it now operates in raid mode. This will be a cleaner and easier method.

Good luck

AKU

[westcoaster]

hoppah wrote:

Dear a.p.,

if it were a software RAID, how could I build the RAID and create the mirror as well as rebuild the mirror drive pre-boot from the RAID card's BIOS, with no OS in RAM?

H.

For many years now even software RAID controllers have had pre-boot BIOS support for creating and maintaning volumes. However they still require an OS, usually Windows, in order to actually use the drives in RAID mode.

[hoppah]

@Westcoaster:

Ugh, that would suck.  I sure hope that's not the case here.  Although ESX appears to be linux underneath - why couldn't one drop a driver in there for such a situation?

H.

[hoppah]

@Aku:  ESXi version is "5.1.0 #1 SMP Release build-1065491 Mar 23 2013 10:12:46 x86_64 GNU/Linux"

~ # lspci -v

00:00:00.0 Host bridge Bridge: Intel Corporation Xeon E5/Core i7 DMI2 [PCIe RP[00:00:00.0]]

  Class 0600: 8086:3c00

00:00:01.0 PCI bridge Bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a [PCIe RP[00:00:01.0]]

  Class 0604: 8086:3c02

00:00:03.0 PCI bridge Bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode [PCIe RP[00:00:03.0]]

  Class 0604: 8086:3c08

00:00:05.0 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management

  Class 0880: 8086:3c28

00:00:05.2 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors

  Class 0880: 8086:3c2a

00:00:05.4 PIC Generic system peripheral: Intel Corporation Xeon E5/Core i7 I/O APIC

  Class 0800: 8086:3c2c

00:00:1a.0 USB controller Serial bus controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2

  Class 0c03: 8086:1d2d

00:00:1c.0 PCI bridge Bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 [PCIe RP[00:00:1c.0]]

  Class 0604: 8086:1d10

00:00:1c.2 PCI bridge Bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 3 [PCIe RP[00:00:1c.2]]

  Class 0604: 8086:1d14

00:00:1c.3 PCI bridge Bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 4 [PCIe RP[00:00:1c.3]]

  Class 0604: 8086:1d16

00:00:1c.4 PCI bridge Bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 5 [PCIe RP[00:00:1c.4]]

  Class 0604: 8086:1d18

00:00:1d.0 USB controller Serial bus controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1

  Class 0c03: 8086:1d26

00:00:1f.0 ISA bridge Bridge: Intel Corporation C600/X79 series chipset LPC Controller

  Class 0601: 8086:1d41

00:00:1f.3 SMBus Serial bus controller: Intel Corporation C600/X79 series chipset SMBus Host Controller

  Class 0c05: 8086:1d22

00:00:1f.6 Signal processing controller Signal processing controller: Intel Corporation C600/X79 series chipset Thermal Management Controller

  Class 1180: 8086:1d24

00:01:00.0 PCI bridge Bridge: Intel Corporation C608/C606/X79 series chipset PCI Express Upstream Port

  Class 0604: 8086:1d74

00:02:08.0 PCI bridge Bridge: Intel Corporation C608/C606/X79 series chipset PCI Express Virtual Switch Port

  Class 0604: 8086:1d3f

00:03:00.0 Serial Attached SCSI controller Mass storage controller: Intel Corporation Patsburg Dual 4-Port SATA/SAS Storage Control Unit [vmhba0]

  Class 0107: 8086:1d68

00:03:00.3 SMBus Serial bus controller: Intel Corporation C600/X79 series chipset SMBus Controller 0

  Class 0c05: 8086:1d70

00:03:00.4 SMBus Serial bus controller: Intel Corporation C608/C606/X79 series chipset SMBus Controller 1

  Class 0c05: 8086:1d71

00:04:00.0 Ethernet controller Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller [vmnic2]

  Class 0200: 8086:105e

00:04:00.1 Ethernet controller Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller [vmnic3]

  Class 0200: 8086:105e

00:06:00.0 VGA compatible controller Display controller: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1)

  Class 0300: 102b:0522

00:08:00.0 Ethernet controller Network controller: Intel Corporation I350 Gigabit Network Connection [vmnic0]

  Class 0200: 8086:1521

00:08:00.1 Ethernet controller Network controller: Intel Corporation I350 Gigabit Network Connection [vmnic1]

  Class 0200: 8086:1521

00:ff:08.0 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0

  Class 0880: 8086:3c80

00:ff:08.3 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0

  Class 0880: 8086:3c83

00:ff:08.4 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0

  Class 0880: 8086:3c84

00:ff:09.0 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1

  Class 0880: 8086:3c90

00:ff:09.3 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1

  Class 0880: 8086:3c93

00:ff:09.4 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1

  Class 0880: 8086:3c94

00:ff:0a.0 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0

  Class 0880: 8086:3cc0

00:ff:0a.1 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1

  Class 0880: 8086:3cc1

00:ff:0a.2 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2

  Class 0880: 8086:3cc2

00:ff:0a.3 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3

  Class 0880: 8086:3cd0

00:ff:0b.0 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers

  Class 0880: 8086:3ce0

00:ff:0b.3 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers

  Class 0880: 8086:3ce3

00:ff:0c.0 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0

  Class 0880: 8086:3ce8

00:ff:0c.1 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0

  Class 0880: 8086:3ce8

00:ff:0c.6 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0

  Class 0880: 8086:3cf4

00:ff:0c.7 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder

  Class 0880: 8086:3cf6

00:ff:0d.0 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0

  Class 0880: 8086:3ce8

00:ff:0d.1 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0

  Class 0880: 8086:3ce8

00:ff:0d.6 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1

  Class 0880: 8086:3cf5

00:ff:0e.0 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent

  Class 0880: 8086:3ca0

00:ff:0e.1 Performance counters Signal processing controller: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring

  Class 1101: 8086:3c46

00:ff:0f.0 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers

  Class 0880: 8086:3ca8

00:ff:0f.1 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers

  Class 0880: 8086:3c71

00:ff:0f.2 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0

  Class 0880: 8086:3caa

00:ff:0f.3 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1

  Class 0880: 8086:3cab

00:ff:0f.4 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2

  Class 0880: 8086:3cac

00:ff:0f.5 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3

  Class 0880: 8086:3cad

00:ff:0f.6 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4

  Class 0880: 8086:3cae

00:ff:10.0 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0

  Class 0880: 8086:3cb0

00:ff:10.1 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1

  Class 0880: 8086:3cb1

00:ff:10.2 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0

  Class 0880: 8086:3cb2

00:ff:10.3 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1

  Class 0880: 8086:3cb3

00:ff:10.5 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3

  Class 0880: 8086:3cb5

00:ff:10.6 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2

  Class 0880: 8086:3cb6

00:ff:10.7 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3

  Class 0880: 8086:3cb7

00:ff:11.0 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 DDRIO

  Class 0880: 8086:3cb8

00:ff:13.0 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe

  Class 0880: 8086:3ce4

00:ff:13.1 Performance counters Signal processing controller: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor

  Class 1101: 8086:3c43

00:ff:13.4 Performance counters Signal processing controller: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers

  Class 1101: 8086:3ce6

00:ff:13.5 Performance counters Signal processing controller: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor

  Class 1101: 8086:3c44

00:ff:13.6 System peripheral Generic system peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor

  Class 0880: 8086:3c45

~ #

[westcoaster]

What model is the LSI controller? I don't see it in the device listing. Is it possible the drives are connected to the Intel controller?

[AKU076]

Hi, Apologiesfor delay. was busy and was not able to come back quickly.

Ok, For cisco servers you can try this image Cisco Custom ESXi 5.1.

These image will be custom built and tested to work with Cisco Servers. To make sure your Megaraid is loaded during re-install, Just make sure you have created the RAID 1 volume on the controller before you re-install the new ESXi image.

Hope this helps

AKU

[hoppah]

Will this use my current license key or will I have to get a new one?

H.

[AKU076]

It should be using the same license key

[hoppah]

Dear Aku,

Thanks for all the feedback - I'm in the middle of several weeks of travel and haven't had time to rebuild the server, but I'm certain you've found the root of the issue.  I appreciate the assistance and will update when I get time to reinstall.

H.