VMFS3 for MSCS shared storage, why not?

[goofygoose]

From the paper on configuring MSCS across physical hosts on ESX3.5, it was stated that RDM be used for the quorum and all other shared disk volumes.

I have been wondering why can't the quorum and shared storage be on vmfs3 partitions instead. I have a large number of MSCS clusters to be set up and using vmfs3 for shared storage will definately make the SAN administration a lot easier, not to mention the large number of LUNs that will be required to be configured on my storage controller and risk exceeding the maximum number of LUNs it can support.

Hence, I have set up a 2 node MSCS cluster across 2 different hosts, using a shared vmfs3 partition on a shared iscsi target. The shared vmdk files (of type eagerzeroedthick) for the quorum and shared storage were created at the service console using vmkfstool. The scsi controller for the quorum disk and shared disks is of type "LSI Logic" and "Physcial" mode.

The rest of the configurations were followed using the vmware MSCS configuration guide and the Microsoft MSCS configuration guides. The 2 guest OS are windows server 2003 enterprise editions.

The MSCS installation and configurations were successfully completed and my initial testings looks good, with the nodes being able to failover well in both planned and unplanned scenerios.

So my big question is... what's wrong with using vmfs3 for the quorum (and shared storage) instead of RDMs??

[aleph0]

are those VMs linux or Windows?

do you just increase vCPU without reinstalling windows? if yes, you are using thesingle cpu kernel...

if it's linux are you using the right (SMP) kernel?

what about %RDY time in esxtop: what's the value?

\aleph0

____________________________

(in italian)

###############

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

[kjb007]

The difference in this case using a physical mode RDM is that the I/O does not go through ESX or to ESX. With a regular vmdk, the I/O gets requested from ESX, and ESX issues those SCSI commands to the underlying LUN that comprises its datastore. With a virtual mode RDM, the vm makes the SCSI calls, but they go through ESX, which is why you can still use snapshots on that type of device. A physical mode RDM keeps ESX out of the picture, and ESX is passive here. The vm controls access to the disk completely.

-KjB

VMware vExpert 2009

[beagle_jbl]

What paper are you reading on configuring MSCS? You obviously have the paper for creating quorum disks as VMDK. You'll notice that using a VMDK is ONLY for MSCS on a single host. Obviously your cluster is on a single ESX host or it isn't functioning as you think it is. If one ESX server is accessing the quorum disk as a vmdk - it is LOCKED. Another host cannot assess that VMDK. That's why VCB uses snapshots to release the VMDK, and why you can't fire up a VM on more than one host. If you say you are using multihost MSCS with VMDK... I call shenanigans. If you are not clustering across multiple hosts then why are you clustering at all? Single host MSCS makes no sense to me at all.

KJB's posts were great but with one error: for multi-host MSCS clusters you can now use a physical or virtual mode RDM. I think MSExchange is the exception (physical RDM only). If you follow the documentation you should be fine.

MSCS is only supported on FC SAN, although I think I read somewhere that iSCSI support is experimental. Maybe someday. I prefer FC SAN anyway.

Why such common use of MSCS? You can't use NIC teaming which blows, and I would rather rely on VMWare HA than MSCS any day. When forced to use MSCS, I leave them as physical servers.

I share Aleph0's concern for your seemingly carefree use of vSMP. Every post of his was excellent by the way. What performance monitoring did you do? Most dB applications are memory and I/O intensive. If you keep adding vSMP servers with no thought of the consequences you'll pay once you start to tax your hosts. I'll probably get someone lecturing me on the advancements in CPU co-sheduling, but vSMP should be used sparingly and you should take steps to ensure that your vSMP boxes are spread evenly across your clusters.

Here is a VMWorld presentation on CPU sharing, resource scheduling, as well as vSMP. I'll warn you that it is very dry and an hour long:

Here is the VM Best Practices document on use of vSMP, it's for ESX2 but the concepts are essentially the same, even though CPU co-scheduling has been greatly improved:

Maybe you were correct in your assignment of vSMP - who knows?

Also, FT comes at a cost. You are limited to 1vCPU initially, you will double your memory cost per FT server, and you will not be able to snapshot an FT server which will affect using VCB as well (although what you can do is change it to HA, do your 'thang', then set it back to FT). I'm excited for FT, but not too excited.

Lastly, clustering your VC server is typically viewed as unnecessary because VC is not integral to the critical functions of your hosts and clusters.

Hopefully you don't take offense to this, but the VMWare course is excellent and provides a solid foundational knowledge for designing and building a VM environment.

[goofygoose]

I'm using windows 2003. Correct me if I am wrong, but I am under the impression that there is no need to reinstall the windows OS from increasing the number of vCPUs. Things only get a bit tricky when vCPUs from decreased from multi to uni (but not impossible).

[goofygoose]

Thanks a lot! I really appreciate your explaination! 😃

Now I know that although my configuration may run fine most of the time, there is a risk of it going into split-brain scenario in the event that there is a failure in the connectivity between the two nodes.

[goofygoose]

What paper are you reading on configuring MSCS?

I am using the paper which was downloaded from vmware's website 2 days ago (). I believe there isn't much critical difference between this and the one u posted.

You'll notice that using a VMDK is ONLY for MSCS on a single host.

I assume you meant that using vmdk for MSCS is only supported by vmware on a single host. I assure you that it is possible across different physical hosts.

If one ESX server is accessing the quorum disk as a vmdk - it is LOCKED. Another host cannot assess that VMDK. That's why VCB uses snapshots to release the VMDK, and why you can't fire up a VM on more than one host.

The SCSI Controller just have to be set to "Physical" for the "SCSI Bus Sharing" policy and multiple VMs on different hosts can be "fired up" using the same vmdk quorum disk.

I would rather rely on VMWare HA than MSCS any day. When forced to use MSCS, I leave them as physical servers.

That is a personal preference. VMWare HA is simply not the same as MSCS.

Maybe you were correct in your assignment of vSMP - who knows?

I believe I am. 😃

By the way I have already watched the vmworld presentation and read smp best practices document prior to this. Thanks for recommending anyway. 😃

I'm excited for FT, but not too excited.

Me too! I will definately not be using it for production the day it gets released. It will require some time before everyone gets confident enough to put it up for production purposes.

Lastly, clustering your VC server is typically viewed as unnecessary because VC is not integral to the critical functions of your hosts and clusters.

I really hate it when the VC goes down and having system administrators calling me when I am enjoying quality time with my loved one. 😃

Hopefully you don't take offense to this, but the VMWare course is excellent and provides a solid foundational knowledge for designing and building a VM environment.

I appreciate all replies! I have only went for the installation and configuration course so far. Hoping to go for the rest soon! 😃

Lastly, I was able to set up MSCS across different hosts using the recommended (and supported) FC RDM for quorum disks. I have also set up a seperate set using VMFS on iSCSI for the quorum disks as well. Both works fine and that prompted me to start this discussion about (1)why aren't people using VMFS for MSCS and (2)why wouldn't VMware support MSCS on VMFS.

[kjb007]

I've found the reverse to be true. I will generally create a vm with an SMP HAL, and will downgrade if needed. I've found issues when a UP HAL was used, and I've had to increase to SMP. Slowness in boot, etc.

-KjB

VMware vExpert 2009

Message was edited by: kjb007 : Changed UMP to UP

[beagle_jbl]

I assure you that it is possible across different physical hosts. The SCSI Controller just have to be set to "Physical" for the "SCSI Bus Sharing" policy and multiple VMs on different hosts can be "fired up" using the same vmdk quorum disk.

OK, just tested it, it does work - it still isn't supported or recommended. The physical mode eliminates the vmdk locking issue. I jumped the gun on that one.

You can switch Windows VM's back to unikernel, although it isn't supported by Microsoft. I've done it to every VM I've converted from physical and never had a problem. If you hardly have any VM's and you have lots of horsepower on ESX - use vSMP as liberally as you want. If you have an environment where you're going to have to actually tax your ESX hosts... you'll need to give some thought to how you deploy it.

[beagle_jbl]

So kjb, you're recommending the opposite of the VMWare best practices? I think that's really bad advice.

I've updrade many VM's from one core to 2 or 4 and never had a problem.

[kjb007]

The post was relating my experience, as you'll notice. There are documents that I've read that have stated similarly, and as I have stated, I've created the vm with an SMP HA, converted it to a template, and vm's deployed from it were modified as required. Most of my vm's are 1 vCPU, and they work very well. However, some that aren't work just as well. The vm's that I've created from scratch at 1 vCPU and that I've had to increase later, I've had issues with, which I've described. Take from that as you wish.

-KjB

VMware vExpert 2009

[jmartin819]

This might sound like a rookie question but wouldn't file locking cause issues or a snag with sharing the vmdk between two VM's?

[kjb007]

No, the vm's still have to use a shared bus, and would reserve/release correctly, once the cluster has been built.

-KjB

VMware vExpert

[Texiwill]

Hello,

Moved to Virtual Machine and Guest OS forum.

Yes you can snapshot vRDMs. Not sure I would use vRDMs for shared disk clusters as the locking could be come a bit of an issue. In a cluster the locking is controlled by the Guest OS not by ESX.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast