Which SAN and how to do VM backups?

[David_Walsh]

We are trying to choose a SAN and VM backups at the minute. We presently have 3 virtual hosts running Vmware server on Windows 2003 R2 EE and local SCSI storage. These are running 8 VMs at the minute, the most significant one being an Exchange 2003 server with about a 50gb database and 60 users, we also have a MS SQL server but this is not heavily used (4 users). We hope to add more virtual machines.

I want to upgrade all the servers to ESX 3.5 and install an iSCSI SAN. The myriad of combinations on the SAN are slightly overwhelming as we don't have a huge budget. I am also getting confused by all the backup solutions.

Question 1. We were looking at the EMC AX4-5i, the NetApp FAS2020 and the HP MSA2000i. And have now been offered an EMC NS20 at what seems to be a very good price. I don't think we can afford the NetApp by the time we've added all the extra software in. And I'm leaning towards the EMC NS20 as I like the fact that it runs CIFS/NFS natively as I need to replace our aging Unix fileserver which is presently doing this and this make it easy. So is there any reason not to buy an NS20?

My reservations about all the SANs are that I have not worked out the "best" way to do the VM backups in ESX which leads me onto question 2.

Question 2. Presently we periodically shut down the VM's at the weekend and take a complete filebackup of the VM and copy it to one of the other hosts. I hope with ESX to do this a bit more intelligently, more often and have it more automated. I am not sure of the best way to take complete VM backups say daily which quiese the Exchange and SQL databases. Should this be done by doing SAN snapshots or is it better done using VCB in ESX?

All help gratefully received - feel free to point me to threads that answer this.

[bradley4681]

Here's a recent system I put together recently. 3 HP DL385's Dual socket/Dual core with 2 HBA fiber cards each. 1 HP MSA1000 with 2TB of disk on a Raid 5 configuration. The MSA1000 had the High Availability Pack, giving it dual storage controllers and dual fiber switchs for reduncadacy. We also had another DL385 that acted as the virtual center server, and VCB proxy. We had a small 2 drive HP tape library connect to the VCB server and it was fiber connected to the MSA1000.

This entire setup costs ran about 75K and provided redundancy, VMotion, and DRS. There were 4 luns at 500GB each and balanced across the two controllers.

Cheers,

Bradley Sessions

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

[KyawH]

Here's the reason why you should not go for NS20: NFS protocol is a FILE LEVEL protocol but iSCSI and FCP (Fibre Channel Protocol) are BLOCK LEVEL protocol. So the performance will be really superior with both iSCSI and FCP. Furthermore, if your iSCSI SAN, Ethernet switches used for iSCSI and the NIC cards in ESX servers support JUMBO FRAMES, you will have much better performance with jumbo frames than normal Ethernet frame of 1500 MTU. ESX 3.5 supports JUMBO FRAMEs of up to 9000 bytes.

For the SAN question:

If you don't have a huge budget, EMC AX4-5i or DELL MD3000i or Equallogic iSCSI SANs are good choice. All those storage arrays have dual storage processors, 2 iSCSI ports per SP (4 iSCSI ports per array), built-in storage memory for read/write caches, built-in backup batteries, redundant powersupplies and fans. They are natively very high availability and they supports JUMBO FRAMES. They all have ability to replicate (array to array) to your future DR site. You can expand your storage up to 3 enclosure. SAS & SATA II drives are supported. My preference is EMC AX4-5i.

For NFS capability, you could setup a Linux or a Windows VM configured as a file server which you can use as NAS in VMware ESX 3.5. Then share the files/folders/disks over the network in VM environments.

For Backup question:

VCB with Backup Exec 12 will be a good choice for your organizations size. VCB takes a snapshot of the VM, mount it on your VCB proxy server disk and backup from there and deletes the snanpshots when done. So you do not need to turn off your VMs to do the backup. You can either install Backup Exec agents for Exchange and SQL on the VM for mailbox level and database level restore or do a full VM backup. For the sile servers, you can backup the whole VM with VCB and you will be able to restore file level if do not specify -FullVM on the pre-script.

If you do SAN snapshots and rollback, it will affect other VMs unless you assign each LUN to each VM. Careful planning is required before implementing SAN, ESX, VMs and Backups with the vendor's best practices.

Hope this helps.

-

---

If you find this information useful, please award points for "correct" or "helpful".

[David_Walsh]

As the NS20 does nfs and iSCSI and fibre, then this is not a problem. And surely doing NFS natively off the NS20 will be quicker than doing it through a VM?

[KyawH]

It really depends on where your data will be moving. If the majority of the data stay within vSwitches without leaving to physical wire, then you will have heaps of performance improvements.

Even though, NS20 has the features like CIFS functions, iSCSI connectivity and functions, Snapshot functions, Automated Volume Management and Virtual (“ thin” ) provisioning in the standard licensing, it is still a NAS, not a SAN storage. Bare in mind that if you need HA with NS20, you will have to buy dual-blade but still it cannot beat the Clariion features. With that dual-blade implementation, it can become a nightmare to manage/maintain with ESX and VMs and all that. I am not agaist with NS20 or anything. At the end of the day you will be the one who makes a business decision. BTW, did you know that AX4-5i comes with free EMC's snapview software for snapshoting at the storage level?

I am sure you will tick the boxes of features you want before entering into storage and VMware environment.

Also, do not forget backup and recovery in case of the disaster. Good Luck with your decision.

[David_Walsh]

Most of our CIFS/NFS traffic will be outside the VMs to client PCs.

I do want HA so maybe I should be re-looking at the AX4-5i. You mention the AX4-5 doing storage level snapshots but further up you say you need to assign a seperate LUN to each VM - is that a problem?

[Texiwill]

Hello,

As for which tool to use, check out http://vmprofessional.com/index.php?content=esx3backups for a comparison of various backup products available. As for hardware based backup systems, there may be one available with your SAN/NAS device already.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[KyawH]

Storage level snapshoting is intended for hosts to SAN direct attached rather than for ESX/VMware environments. Before ESX/VMware becomes so popular, people used to attach the physical machines to the SAN directly, which means the physical hosts see the LUN from the storage as a hard disk.

But in ESX/VMware environment, ESX servers see the LUNs and create Data Stores and format as VMFS. Then you create the VMs and give a portion of the VMFS to the VMs as hard disks. So in this scenario, which is very common, each LUN from the storage array is shared across many VMs. That's why in this case, if you take a snapshot of a LUN at the storage level, that LUN might get shared by many VMs and when you restore that particular LUN, all the VMs will be affected. In this case, you rather take the snapshot at the VM level.

On the otherhand, a VM can map to the LUN from the storage as a raw device mapping (RDM) where the whole LUN is owned by a single VM. In this scenario, you can take a snapshot of the LUN and restore as desired at the storage level, which will only affect one hard disk of a VM.

Hope this helps.

[Texiwill]

Hello,

Storage Level snapshots (done by your storage hardware and if the hardware supports it) is possible as KyawH states, it would make a mirror of all your VMs on the LUN at one time instead of needing a backup of each individual VM. You can use the mirror LUN attached to another ESX host as a way to restore a single VM as well. However, hardware options tend to require more hardware to make them work fully. I.e. that secondary ESX server.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[jameervin]

I would look to consolidate as many advanced features to the SAN as possible. As mentioned, snapshots. ALso local mirroring at the SAN level is also good. This can replace/augment virtual server templates. Choose your block iSCSI storage wisely. Look for one that "virtualizes" the physical storage. What I mean? One that offers iSCSI volumes on top of the physical volumes or RAID sets. This way if you have one big RAID set over lets say 12 drives, you can create 10s or 100s of volumes in the SAN. Also one of the interesting thing is that ESX server tends to create a lot of iSCSI sessions when connecting, so look at the number of simaltaneous sessions supported by the SAN.

Here is my biased section:

Take a look at StoneFly IP SANs. Our customers have been using VMware for the past couple years, and we recently joined the HCL. Here is our site. www.stonefly.com. And here are some thoughts from Virtual Strategy Magazine on our IP SANs: http://www.virtual-strategy.com/VSM-Labs/StoneFly-i4000.html.

Good luck wih your backups!

[mshulman]

Just to chime in on the SAN recommendation. I'd highly recommend a Dell Equallogic SAN. I have worked with Equallogic quite a bit now and they are simply easy to use, flexibile and have all the features built into the unit. Many other SAN's have you buy these advanced features, where Equallogic inlcudes them all, even thin provisioning. You can literally have one of these units up and running with data being accessed on it in under an hour. I also got replication configured from one unit to another in just under an hour.

The 4TB unit (usuable it 2.5-3TB depending on raid config) is about $30k. The 8TB unit is about $40k and they also have larger units, plus units with SAS disks for even greater performance. The ones I mentioned are SATA, but still give great performance.

[David_Walsh]

The Equalogic is nice, but from the quotes I am getting it is more expensive than all the units I have mentioned and out of my budget.

Also I was SAS disk as I have heard running an MS Exchange VM off SATA disks isn't a great idea.

[mshulman]

Just make sure the other units support all the features you may use. That is one place where equallogic differs from others. They give you everything under the sun in the single price.

In my experience, SATA is just fine for Exchange for smaller environments. Not knowing how many users you have, its hard to say whether or not it would suit your needs, but I have a few clients with 50-100 users in each that are running Exchange off of Equallogic units with SATA drives.

[KyawH]

Despite the speed of the disks FC/SAS/SATA, you've got to design your RAID set properly for your intended use as well to get improved performance. For example, if you are concerned about the performance of the Exchange server, use RAID1/0 in which you mirror your disks and stripe without parity. RAID1/0 give you the best read (RAID 1) and write (RAID 0). Place all your transaction logs on LUNs created out of RAID1/0. I bet you will get excellent performance on your Exchange. RAID1/0 requires either 2/4/6/8... disks pair and usable space is half of the disk size.

[jameervin]

David:

There are some great points here. One thing to note, many of the pure play iSCSI solutions offer all-inclusive pricing. The charge by feature is typically for simple storage arrays or large FC SANs. You'll find that most options give you everything you need out of the box.

SAS vs SATA

RAID levels are important to note for disk performance, but also recognize that some system cache data before they send it off to the disk . This means that unless your files are the right size you may not see the benefit. Especially in an IP SAN since the advanced ones add a layer of translation before writing to the disk. Look to max out your iSCSI cache to initiate more transactions from the cache. Another key point to consider: the speed of your network. Sometimes you network is slower than thet storage, so be sure you have dedicated resources for you IP SAN.

[Cotay]

One thing to note: From my experience, taking a SAN level snapshot of a running VMFS file system will result in a near-usless backup. In our case, on Equallogic arrays, every restored VM from a SAN level snapshot resulted in a repair install of Server 03 and a limping server at best. This was relatively light I/O 'utility' servers.

After that, we went to vRanger on a VCB backup proxy. Our VM level snapshots were redirected to a dedicated 'backup' volume on the SAN. That volume was then replicated after the vRanger job was finished to another DR Equallogic SAN. Zero data integrity issues on restores and we've been quite happy especially with the newest build. We will be moving these backups off the Equallogic arrarys to our DataDomain applainces soon.

Many SAN vendors have or will soon be addressing this issue. I know Netapp already has a Virtual Center agent that puts your VMs in snapshot mode and takes a clean SAN based snapshot. Equallogic has a version due out this fall that looks promising.

Hopefully SRM uses incorporates these SAN based tools or people will be spending big bucks spinning up garbage servers at their DR sites.

[Texiwill]

Hello,

You are correct a SAN based mirror or snapshot is a crash-consistent duplicate. Until the SANs integrate better into virtualization this is the way things are. Also, unless you ensure disk quiescing with most backup tools including vRanger you still have a crash-consistent backup. You have to write some scripts to properly implement disk quiescing. While some exist for some Windows from VMware other OS' require them to be written.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[Cotay]

This is my experience with Equallogic:

After doing all my homework before our VMware/SAN implementation, I was convinced a SAN based/crash consistent snapshot would work fine for our environment. After implementing, I found that crash consistent SAN snapshots resulted in completely unusable and corrupt Server 03 VMs. Light and heavy I/O nothing would work. This was done by bringing a snapshot online, rescanning the iSCSI target in ESX, and trying to boot a VM.

The strange part is using both VCB and vRanger, "crash consistent" backups with no sync drivers engaged resulted in a perfect restore every time. SQL, Exchange, AD, etc. We've never had a problem. We do use vRanger's VSS options on aware servers, but they worked fine even without.

My local VAC and Equallogic have validated this point. During some conversations with Equallogics Dev group, they are aware of this condition, and rushing get there VI3 snapshot agent to market. We've been beta testing and it looks promising.

My .02