VD1.2 - hot add disk not working? backup done via ...

Pitterling · ‎07-06-2010

Hi,

following the blog,

http://blogs.vmware.com/uptime/2009/07/vmware-data-recovery-taking-advantage-of-vsphere-4.html

VDR should be able to add the snapshots of the VMs to the VDR appliance and doing the backup from disk (Shared storage) to disk (Backup device).

I fullfilled the 3 requirements

a. The source virtual disks need to be on

shared storage

b. The ESX host where the VDR appliance is

running needs to have visibility to this shared storage

c. You will need a vSphere edition that

includes Hot Add as a feature

I now noticed, that the backup is running through the network. Our service consoles are limited to 100MBit connections which now becomes the

bottleneck for the backup process.

Do i need to enable anything else to make this working ?

thx, Peter

Pitterling · ‎07-08-2010

I figured out, that the user connecting to the vSphere Center did not have the appropriate rights.

It seems, that the license check failed and VDR was using the network connection to backup the Virtual machine. After changing the DisableNetworkCopy parameter to 1 - no backup was possible anymore.

Now, the user rights are set up as recommended and the logs confirm that it looks much better, BUT the backup still fail

I got vcbAPI errors 1302 and afterwards -3943

7/7/2010 17:51:49.000: 0xf43aa750: TVmSnapshot::OpenSnapshotFile: Failed to open file "[SYM39_DS] ATG Fileserver/ATG Fileserver_new.vmdk" with flags 1, error 1302 (vcbAPI: open failed)

7/7/2010 17:51:49.000: 0xf43aa750: TVmSnapshot::GetVMDiskKeys: Failed to open virtual disk file "[SYM39_DS] ATG Fileserver/ATG Fileserver_new.vmdk" error: -3943 (vcbAPI: open failed)

I checked the name resolution and the VDR appliance is able to find the hosts and i also see the following entries in the log

7/7/2010 17:51:55.000: 0xf43aa750: TVmSnapshot::OpenSnapshotFile: opened file "[SYM39_DS] ATG Fileserver/ATG Fileserver.cfg.xml" with flags 1, handle 37

7/7/2010 17:51:55.000: 0xf43aa750: Xen::xenDoReadBegin: backing up file ATG Fileserver/ATG Fileserver.cfg.xml

7/7/2010 17:51:55.000: 0xf43aa750: xopFlush: flushing any remaining data

7/7/2010 17:52:01.000: 0xebe143d8: ddupOpenFile: opened file 4E48AC73-1F79-4961-B1C9-B78BE18E3B75/vm-2382/2010-07-07 17-51-16-000/pre-backup-info.log for writing

7/7/2010 17:52:02.000: 0xebe143d8: ddupCloseFile: closed file 4E48AC73-1F79-4961-B1C9-B78BE18E3B75/vm-2382/2010-07-07 17-51-16-000/pre-backup-info.log

7/7/2010 17:52:02.000: 0xebe143d8: ddupCloseFile: 4E48AC73-1F79-4961-B1C9-B78BE18E3B75/vm-2382/2010-07-07 17-51-16-000/pre-backup-info.log: total size = 2,356, written bytes = 2,356, new bytes = 2,356

7/7/2010 17:52:02.000: 0xf43aa750: TVmSnapshot::CloseSnapshotFile: Closed file "[SYM39_DS] ATG Fileserver/ATG Fileserver.cfg.xml", handle 37

7/7/2010 17:52:02.000: 0xf43aa750: Xen::xenDoReadEnd: done backing up file ATG Fileserver/ATG Fileserver.cfg.xml, result 0

For my understanding, VDR is able to access any other file , but not the vmdk files.

Any clue what else might be wrong?

thx Peter

Pitterling · ‎07-09-2010

i changed the parameter DisableNetworkCopy back to default and my backup is running again. Something is preventing from backing up via the HotAdd feature.

I checked the vcAPI logs and found the following exception:

VcbAPI::Snapshot::Open: Exception trying to HoAdd: Failed to reconfigure proxy: No permission to perform this action.

Would kind of proxy is meant and which permission is missing?

Paul11 · ‎07-09-2010

I have the same problem with some VM's in my environment. The VDR-appliance was on a VMFS-Volume with 1 MB Blocksize, the VM's are on VMFS-Volumes with 4 MB Blocksize. After moving the VDR-aplliance to the VMFS-Volume with 4 MB, the VM's could be backuped also the Parameter DisableNetworkCopy=1. So it seems to be a problem with different block-sizes, but only for a few VM's. Can you check this please to confirm my resolution.

Paul

RParker · ‎07-09-2010

Can you check this please to confirm my resolution.

This makes perfect sense. THINK about it. LUN1 is block 1, LUN2 is Block 4. The VDR is sitting on LUN3 with Block 1.

How can you MOUNT a VM with a larger file size on a SMALLER block size, when it's a requirement to have the larger block size.

It's a VERY Logical conclusion. Since you have figured it out, just put VDR on the largest block size LUN you have, problem solved.

RParker · ‎07-09-2010

Would kind of proxy is meant and which permission is missing?

Just give VDR Administrator rights. What do you think VDR is going to do, flip out and globally delete all your VM's? Let it have FULL rights, that's much easier than trying to figure out which rights to exclude. You have to give it access to the VMs anyway, so it can corrupt the data if something goes wrong, so just let it have ALL the rights.

Pitterling · ‎07-12-2010

Admin rights made the trick.

I followed the instructions in the VDR1.2 admin guide setting the rights for the backup user. Obviously some rights are missing in the admin guide preventing the HotAdd feature.

Pitterling · ‎07-12-2010

I'm still struggeling with one of our VMs. This one has 3 vmdks. 1 is a vmdk file and the other 2 are rdm files. I know that VDR is not able to backup the RDM devices. All i want is to backup the vmdk file. VDR is still trying to Hotadd the RDM devices although they should be ignored. This obviously prevents from HotAdding the vmdkfile ?

Here are the log entries:

7/12/2010 12:42:38.000: 0xa94cb7a0: Normal backup using $[1]$[19]ATG,SolMan,DB6-CTP$[20]$[2] (Execution unit 1)

7/12/2010 12:43:55.000: 0xa94cb7a0: Performing incremental back up of disk "$[19][SYM39_DS] ATG Fileserver/ATG Fileserver_new-flat.vmdk$[20]" using "Network"

Creating snapshot

Snapshot created, ID: snapshot-7097

Disk ATG Fileserver/ATG Fileserver_1.vmdk is marked as independent. Ignoring for backup.

Disk ATG Fileserver/ATG Fileserver_2.vmdk is marked as independent. Ignoring for backup.

Establishing NFC connection to host atgesx2.wdf.sap.corp on port 902, service vpxa-nfc

Performing SearchIndex find.

Successfully obtained instance lock.

Will remove all disks belonging to VM "vm-6626", Snapshot "snapshot-7097".

VcbAPI::Snapshot::Open: Exception trying to HotAdd: Disk ATG Fileserver/ATG Fileserver_1.vmdk is bigger than the maximum file size supported on datastore SYM34_DS

Performing SearchIndex find.

Successfully obtained instance lock.

Will remove all disks belonging to VM "vm-2382", Snapshot "snapshot-7097".

Remove clone disks successful.

VcbAPI::Snapshot::GetChangeInfo: Processing 519 changes

RParker · ‎07-12-2010

This obviously prevents from HotAdding the vmdkfile ?

I don't think there is a fix for this, I think most backup have trouble distinguishing which drives can be backed up and which ones can't. So if ALL the drives are not available, NONE of them will work. That VM has to be skipped, or use conventional backup with an agent.

VDR is a pretty simple backup, I doubt there is a work around for this.

Pitterling · ‎07-13-2010

but i configured the backup job that way - include the vmdk file and exclude the RDM devices. Therefore the VDR appliance should know which device to backup and which not.

jarsenea · ‎07-13-2010

In your VM's settings, are the RDM disks set to "Independent"?

I'm backing up a VM very similar to yours (file server with 2 RDMs, 1 vmdk for the OS partition) and it works fine

Pitterling · ‎07-14-2010

Yes, they are marked as indipendent. The backup itself is running without any issues, but the Network option is used instead of the HodAdd feature.

VDR still tries to HotAdd the RDM devices ...

Disk ATG Fileserver/ATG Fileserver_1.vmdk is marked as independent. Ignoring for backup.

Disk ATG Fileserver/ATG Fileserver_2.vmdk is marked as independent. Ignoring for backup.

VcbAPI::Snapshot::Open: Exception trying to HotAdd: Disk ATG Fileserver/ATG Fileserver_1.vmdk is bigger than the maximum file size supported on datastore SYM34_DS

Are you sure that your backup is done by HotAdd ?

jhanekom · ‎11-09-2010

Just adding this for reference purposes in case someone stumbles on the "no permissions" issue again (doesn't help much with the secondary problem of disk sizes.)

If you want to confirm that you have the "no permissions" issue, log onto the VDR appliance's console and do the following:

grep HotAdd /var/vmware/datarecovery/vcb*.log
if you have the permissions issue, you will see lots of entries that state "No permission to perform this action."

It seems the VDR documentation is wrong (http://www.vmware.com/pdf/vdr_12_admin.pdf, page 15-16), when it states that the account used for the VDR appliance needs the following permissions on the VC object for the appliance:

Datastore->Allocate space
VirtualMachine->Configuration->Add new disk <-- appears to be incorrect
VirtualMachine->Configuration->Change resource
VirtualMachine->Configuration->Remove disk
VirtualMachine->Configuration->Settings

I changed "Add new disk" permission above to "Add existing disk", and things started to work. (Suspect this is perhaps a typo in the documentation.)

caldwelr · ‎01-03-2011

We tried the added permissions as suggest and still weren't getting hot-add backups. Our logs contained the following error on failed attempts even with the modified permissions.

[2010-12-21 00:41:17.339 EBBBDB90 info 'vcbAPI'] VcbAPI::Snapshot::Open: Exception trying to HotAdd: Failed to reconfigure proxy: No permission to perform this action.

We confirmed that it was a permissions issue by successfully running hot-add backups with Admin access. After some trial and error we were able to accomplish hot-add backups by adding "Datastore->Browse Datastore" permission.

It's pretty evident that this doesn't work with the permissions as documented. Seems like either most people are just using admin accounts or they're just not noticing that they're not getting hot-add backups.

Sharantyr3 · ‎05-16-2011

Hi,

Sorry for coming back to this thread but a small tip that could help people out there :

I had the error with VDR :

File "server-flat.vmdk?[20]": can't read, error -3943 ( open failed)

Trouble reading files, error -3943 ( open failed)

In fact it was a P2V of a server that kept its IDE disk controller. And IDE does not support hot-add !

Changing disk to SCSI controller solved the problem.

All

VD1.2 - hot add disk not working? backup done via network