Installing rsync on VDR to replicate deduplication store offsite

[mcretan]

Hi All,

As the VDR is running centos, it's possible to use yum to install rsync (see http://am3n.profusehost.net/index.php?id=70), and set it up so that a local attached or san based deduplication store's contents could be replicated offsite, or to a tape backup server.

Has anyone tried this? Can anyone think of any reasons why it wouldn't work?

Thanks in advance,

Mike Cretan

[AndreTheGiant]

Install a new package is not a problem.

But VDR backup are big file that change a lot.

Usually rsync make a new copy of all changed file.

At this point why you do not use a network share on a remote system (for example the backup server)?

Andre

[shcrgeek]

RSYNC's default is to copy changes. The only "full" copy is the very first one. All this behavior can be modified via commandline parameters, or in the rsyncd.conf file. I've used rysnc for many years to backup remote sites and it is terrific, especially when you're trying to pull a bowling ball through a stirring straw.

I've not thought of using RSYNC for this, our current thinking is to try spooling out to tape, but I think it's a worthy experiment. You could rsync to a windows server running one of a few existing rsync packages for windows, or to a native unix implementation.

Why not cfis? Maybe the OP wants to use vmdk only for robustness and speed, and then on his own terms push a copy to a secondary location... just my 2¢.

[AntonVZhbankov]

In my opinion - do not setup any additional services on VDR VM. Create new VM, setup NFS just to store backups and setup RSYNC on this VM.

---

MCSA, MCTS, VCP, VMware vExpert '2009

http://blog.vadmin.ru

[mcretan]

Thanks for your thoughts.

Here's our usecase: We have two servers, connected to a single san, and a total of 15 VM's (i.e. we're an SMB). The servers each have 500GB of available direct attached VMFS, and what we've done is create two VDR appliances (one running on each server) which use the direct attached storage as their dedupe store. The primary thing this guards against is SAN failure, but I still want to backup the VDR backup store to tape to take offsite, or prehaps replicate it to an offsite supplier.

Our backup server is a physical win2k3 server with veritas bacup exec and LTO3 tape.

We tried having a single VDR with CIFS access directly to a share on the backup server, but it was hopelessly slow, and didn't get through the first backup before giving write errors to the dedupe store. I have also read online that it is far preferable to use a local VMDK in place of a CIFS destination if at all possible.

If we run a seperate VM for the rsync, we won't be able to access the direct attached storage, plus I don't see how we could access the contents of the VDR vmdk dedupe store while it's mounted to the VDR.

So my thinking is that we add rsync to the VDR's (a five minute job), and rsync the dedupe store contents to the backup server. The big questions are

a) To what extent does the dedupe store content change;

b) Will the operation of the rsync daemon have any impact on the operation of the VDR; and

c) Will the replicated dedupe store be of any use as a recovery source?

Thanks again.

Mike

[mcretan]

I've had reasonable success with this. The steps to implement are as follows:

Login to VDR using ssh (e.g. putty) as root/vmw@re

install rsync client as follows:

yum -y install rsync

Then, setup an rsync server on your network, I used the cwrsync windows version from (note that I also had to install copssh to get it to work). Add a module on the rsync server that will act as your VDR destination.

An example rsyncd.conf file might read:

-

---

use chroot = false

strict modes = false

hosts allow = *

log file = rsyncd.log

path = /cygdrive/e/VDR/tcs-esx1-vdr

read only = false

transfer logging = yes

-

---

Then, initiate the rsync by entering the following command on the VDR:

rsync -vrti --modify-window=1 --delete /SCSI-0:1/ rsyncserver::tcs-esx1-vdr

Where /SCSI-0:1/ is your datastore mount location (use "df" to identify), and rsyncserver is the name your rsync server host, and tcs-esx1-vdr is the name of the module defined in the rsyncd.conf above.

I've also wrapped the rsync in a shell script on the VDR as follows:

#!/bin/sh

#

service datarecovery stop

rsync -vrti --modify-window=1 --delete /SCSI-0:1/ rsyncserver::tcs-esx1-vdr

service datarecovery start

This shuts down the data recovery service for the duration of the rsync to ensure that the datastore is in a static state. Next I plan to add a call to this script in my /etc/cron.d directory, plus tape backup of the rsync server content.

rsync does everything it can to prevent sending data over the wire, by generating hashes on parts of the files to sync at both sender and receiver. VDR stores most of it's data in "slab" files of ~1GB in size. Unfortunately the integrity check process touches all of the files (i.e. changes their modification time), so rsync has to do a hash check of all the slab files when it syncs to identify differences, which slows things down. However data volumes sent are only a fraction of the full slab file size - and at least we'll have an offsite copy.

Cheers,

Mike

[ceebee201110141]

Most people may know this, but just in case... in the previous post there's a typo; replace "Re: Installing rsync on VDR to replicate deduplication store offsite" with "[tcs-esx1-vdr]" - the latter is the module name.

Rsync is working well as expected, but as the previous poster mentioned, because the integrity check touches all the files, it slows down the process. In my case with hundreds of slab files, this nearly triples the time compared to the initial rsync transfer. This makes a daily offsite rsync unviable as the time taken spans more than a full day.

I can speed up the whole process by using the "--size-only" switch which ignores the timestamps and only looks at the size when making the comparison. I know I'm taking the risk that a changed slab does in fact have a different size. Not knowing exactly the mechanics of slab file creation/modifcation, my question is - can I get away with this?

[Jt3rry]

I've just recently configured VMDR with an off site component at my office as a sort of proof of concept. My approach was slightly different in that both the main site and the "DR site" have an Openfiler system serving up NFS (physical server at the office and at the DR site Openfiler running as a VM, attached to local storage). The NFS store on each end is attached to ESX and is only currently used to store VMDR and its disks (attached a second VMDK which is the backup target for the production VMs).

The main site has a full cluster, iSCSI, FC, all the bells and whistles - We've always had another local storage device as a target for VMDR, but I've always wanted to try sending the full VM offsite. The DR site, is at one of our employee's home office. This happens to be an ideal location since its cheap, he already has a site to site VPN and his own ESX whitebox with plenty of disk. The DR site ESX server is stand alone, and its not connected to vCenter in anyway - This makes recovery with VMDR a little bit odd, but it's possible.

Basically I'm using the Openfiler physical box to push via rsync, the local NFS stored VMDK which (after a backup job) now contains my de-duped production VMs over the VPN to the Openfiler running as a virtual. From the stand-alone DR/ESX server I can import the VMDR VM, re-config its network for the DR site - and launch a restore of the freshly rsync'd VM. The only hitch here that I've found is since the DR site does not contain same vSphere hierarchy (data center, cluster, host and data-store names) the restore process will warn you that you have made an "Invalid Datastore Selection" This warning can be ignored. The Virtual Machine Restore Wizard will also stop you at the Destination Selection page (since the original vSphere structure is missing) you just have to hit the drop down for the VM your trying to restore, and select the stand alone ESX server, then adjust your data-store to reflect the new local storage. Of course in a real disaster we'd need to make the network changes so the DR site becomes the main office, but that's easy.

I believe since rsync is only having to deal with the few files associated with VMDR, this may speed up the process a bit. Again, I've not been able to do much more than test at least one full rsync "seeding" between the NFS stores, as well as a second rsync job to validate that only changes are sent. Each time I was able to restore the VM at the DR site, connect to the guest and validate that changes were written. I'm still not sure how I'll schedule rsync so that it only runs after the VMDR job is fully completed, maybe a script in VMDR can kick off the rsync job once it completes its own backup...

Thanks!