ghettoVCB.sh - Free alternative for backing up VM's for ESX(i) 3.5, 4.x, 5.x, 6.x & 7.x

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

lamw · ‎10-04-2009

If things have been working for a few months, then I would start looking at the changes that could have occurred in your environment, if the script has changed, then it's probably something else. Make sure no changes were done or if any of the virtual disks have been removed from the VM, etc,etc. You will need to do some troubleshooting

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

someone908 · ‎10-05-2009

Hi,

first thanks for this great script, seems to be the only alternative for using VCB on Windows.

One small question / feature request:

Would it not minize the downtime if the script shuts down a VM, take the snapshot, Power the VM back on again, clone the disk and then remove the snapshot ? So the downtime would be time requied for shutting down, take Sanpshot and start again. No matter how big the disks are.

Thanks in advantage.

hburroughs · ‎10-06-2009

I second the request to shut down the VM, take a snapshot, power the VM back up, while cloning the disk and remove the snapshot afterwards. I have modified the scripts in the past to get close to this functionality, but of course I have to re-edit them anytime the upstream GhettoVCB is modified. I feel more confident when I can do pristine copies of the VMs, but it is inconvenient for the downtime required. Maybe a new toggle in the script file? The one problem with this behavior would be if you provided a list of VMs..... do you cycle each off, snapshot, power back on, backup, remove snapshot, THEN do the next for the next VM or do you take snapshots of ALL the VMs you listed first, THEN backup each one and delete the snapshot before moving to the next vm, backup then delete snapshot, etc..... I understand there would be performance impacts due to more than one snapshot in play... but that would be the user's responsibility. They still could pass a single VM to the script (my favorite is echo "vmname" | sh ghettoVCB.sh - ) which reads the VM from the command line vs a config file.

hburroughs · ‎10-06-2009

I've made the changes with about 10 lines of code. (one if statement needs a condition added as well) The evolution of ghetoVCB has actually made it easier to implement part of this. Right now the behavior works best with one VM at a time. If you are backing up multiple VMs, it will wait till the last backup is done, then power that VM off, snapshot, power back on (waiting x seconds), then backup/clone, then delete snapshot. It would take a good deal of rewriting the ghettoVCB flow to handle snapshots first for all VMs. I'll post my changes as soon as I run a few more tests.

banthorpe · ‎10-06-2009

Guys – got a problem with the ghetto backup script. It runs OK for a few backups then complains that it cannot create the directory /vmfs/volumes/BACKUP (see below).

In the script I define:

VM_BACKUP_VOLUME=/vmfs/volumes/BACKUP/ESXIDMZ01

BACKUP is an NFS share mounted as a datastore on the ESXi server. This is actually drive F: on the backup server and on that drive I have pre-created the ESXIDMZ01 folder. I have tried also pre-creating the VM folder below that “OCSEDGE” but it fails with or without that directory existing. There seems to be no consistency.

I also see that “ls” error at the end of the backing up of EXCHEDGE01 (below)

Any ideas?

/vmfs/volumes/49303d2e-240b718c-15f5-00226409b2b0 # ./ghettoVCB.sh ./vmbackups

Taking backup snapshot for C2C ... ################

Start time: Tue Oct 6 14:02:20 UTC 2009

Destination disk format: VMFS thin-provisioned

Cloning disk '/vmfs/volumes/ESXIDMZ01_1/C2C/C2C.vmdk'...

Clone: 100% done.

Removing snapshot from C2C ...

End time: Tue Oct 6 14:17:33 UTC 2009

Completed backup for C2C! ####################

Taking backup snapshot for EXCHEDGE01 ... ################

Start time: Tue Oct 6 14:17:35 UTC 2009

Destination disk format: VMFS thin-provisioned

Cloning disk '/vmfs/volumes/ESXIDMZ01_1/EXCHEDGE01/EXCHEDGE01.vmdk'...

Clone: 100% done.

Removing snapshot from EXCHEDGE01 ...

ls: /vmfs/volumes/BACKUP/ESXIDMZ01/EXCHEDGE01: Input/output error

End time: Tue Oct 6 14:56:00 UTC 2009

Completed backup for EXCHEDGE01! ####################

mkdir: Cannot create directory `/vmfs/volumes/BACKUP/': Input/output error

Unable to create "/vmfs/volumes/BACKUP/ESXIDMZ01/OCSEDGE"! - Ensure VM_BACKUP_VOLUME was defined correctly

hburroughs · ‎10-06-2009

Follow these steps to modify the current ghettoVCB.sh to support what was discussed above: (I just fixed a minor issue if you run the script with this enabled and the VM off... it wouldn't be able to startup since the disk would be locked)

1. Add the following code at the top of the program before the # DO NOT MODIFY LINE

\# This will power off the each VM as they are processed in the list, snapshot them,

\# power them back up, run the backup, and delete the snapshot. Then go to the next VM and repeat.

\# Would be nice to make it power off and snapshot them all first,

\# then power them all back on and move forward.

\# 1=on, 0 = off

POWER_OFF_SNAPSHOT_ON=1

\# Time to wait after powering on the VM before starting the backup/clone process

POWER_ON_WAIT=120

2. Next, replace the if statement directly following "#section that will power down a VM prior to taking a snapshot and backup and power it back on" with this:

if [\[ $ -eq 1 ]] || [\[ $ -eq 1 ]]; then 3. Add this code directly after the section for #If we need to power the VM back on before backup (POWER_OFF_SNAPSHOT_ON = 1) if [\[$ -eq 1 ]] && [\[ "$" != "Powered off" ]]; then

echo "Powering back on $ before backing up (waiting $ seconds before copying)" $ vmsvc/power.on $ > /dev/null sleep $

fi

That is it! I've tested this on a only a non-critical system, and since you are modifying your existing ghettoVCB.sh script, please be careful and test completely before using in production.

lamw · ‎10-06-2009

I'm not sure which use case you're trying to accomplish with the suggested solution.

I see basically 2:

A) Backup while the VM is online

B) Backup while the VM is offline

The solutions to the following are:

A) Take a snapshot while the VM is running, backup the disk and commit the snapshot (no downtime)

B) Power down the VM, no snapshot is needed and backup the disk and power VM

I'm assuming you're referring to case B, you may minimize the actual downtime of the VM by doing an offline snapshot but unfortunately you introduce a snapshot which needs to be committed which turns out to be a similar use case as A, so why not just backup the VM while it's running?

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

fqu89 · ‎10-07-2009

I've had the same request than someone98 a few weeks ago...

The reason of this request is :

-It's safer to make a backup of a VM while it's off because databases are closed, files are closed, nothing is in the memory of the VM and in fact you are sure that you won't have any surprise (database inconsistency, files truncated, and so on) when you will use this backup.

-In this case, if you power down a VM and wait until the backup is done before powering it on, it can take HOURS to complete (lots of gigs to transfer)

-so it would be nice to have the ability to shutdown a VM, take a snapshot, power it on, make the backup and delete the snapshot.

banthorpe · ‎10-07-2009

problem I have does not seem to be with taking the online snapshot etc. The problem seems to be with the target datastore where the files get moved to. For some reason I fails after backing up 1 or 2 VMs OK saying that it can't create a specific directory for the next VM. However, it managed to do it fine for the first 2 so why fail now?

aliog · ‎10-07-2009

Hi,

for reference when setting up a persistent cron job in ESXi 3.5 update 4 i had to use the following commands to stop and restart crond in my /etc/rc.local

kill $(pidof crond)

crond

not

busybox crond

as mentioned in the help. Other than that all working great.

lamw · ‎10-07-2009

Definitely understandable with databases, but that's why VMware has the option of quiescing the filesystem to have all it's data flushed to disk prior to taking a snapshot. The only requirement is to have the latest version of VMware Tools running and when a snapshot is taken, there is a few options to include memory and also to quiesce the fileystem.

[root@himalaya ~]# vmware-vim-cmd vmsvc/snapshot.create
Insufficient arguments.
Usage: snapshot.create vmid [snapshotName] [snapshotDescription] [includeMemory] [quiesced]

Creates a snapshot for the vm.

I do have to admit, in the current version, this is not being utilized and I don't have options to enable this but this is just a boolean of 0 or 1 which can be modified in the script. The reason we have not provided this additional functionality is we don't backup any databases for our dev environment nor any systems that are sensitive that require their filesystem to be flushed to disk.

I don't think suggested solution really saves much more time and you do incur more time to power down and power on the system and you're still working with a snapshot. I do agree, that you do need to take into consideration about backing up databases and even with VMware's product and others, there are things like VSS and quiescing VM/etc.

You can definitely test with the params and see if it helps, else the code that has been provided by others can be used but will not be part of ghettoVCB.

Thanks again for your comments

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

lamw · ‎10-07-2009

What are the differences between the first 2 VMs and the third? I've seen this issue once or twice in the forums and it all goes back to the NFS server as you've mentioned with the timeout. I would say that either the size of the VM differs significantly or the snapshot is much larger than the first two and hence it's failing. Also is the link to your NFS server being saturated, are there other data going through it? 1gb link to NFS server vs 10gb link vs 4/8gb FC link will all act very different and also the storage on the backend and how many spindles/disk are being accessed.

This is more of an environmental issue than with the script, we've backed up VMs in the hundreds of gig without any issue against an NFS.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

lamw · ‎10-07-2009

Thanks for the information, I'll update the document tonight with the correct params.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

mcunningham44 · ‎10-08-2009

Has anybody encountered the following error when running the script?

mkdir: cannot create directory `/vmfs/volumes/nfs_storage_backup': Operation not permitted

DSTAVERT · ‎10-08-2009

what do you have for the VM_BACKUP_VOLUME=

The backup volume must point to a mounted datastore (unless you are using non persistent storage) and the path must exist.

mcunningham44 · ‎10-08-2009

Well, the script was working fine then I changed the NFS share name. I am using non persistent storage. This is going to a VMWare Cluster and I noticed that on 3 of the hosts now, it was mounting to nfs_storage_backup and now its going to nfs_storage_backup (1). Because of the (1) the path wasn't working for the compression. So I removed the nfs_storage_backup (1)...now when I go to run the script i get the above error. Any suggestions? Thanks!

DSTAVERT · ‎10-08-2009

NO spaces in the path. or special caharacters.

DSTAVERT · ‎10-08-2009

Can you mount that datastore?

mcunningham44 · ‎10-08-2009

The system created the nfs_storage_backup (1) on its own... i don't know why it used that when its defined as NFS_LOCAL_NAME=nfs_storage_backup in the script.

DSTAVERT · ‎10-08-2009

Can you mount that share on your ESXi host as you have it listed in the script under persistent.

mcunningham44 · ‎10-08-2009

I'm not sure. All I know was the script was working fine on each host in the cluster and for some reason it stopped with the Operation Not Permitted Error. I would like to know what caused this error to occur. Is it plausible that only 1 HOST in the cluster can be mounted to the NFS share at a time? Because currently there is 1 of the 4 HOSTS in the Cluster that work fine and mount to this location without a problem and STILL works fine.

DSTAVERT · ‎10-08-2009

As long as there are fewer than 8 NFS shares on a host (the default but can have been changed) an NFS share can be be add to as many ESXi hosts as you have. When you say that the system made the change what do you mean "what system?". Is the NFS share already present on the host? Again I ask can you mount the share as is is under the non persistent settings of the script. If it can't then the script won't either. You did say that it was working and then you changed something. Check your editing.

lamw · ‎10-08-2009

I believe someone here or in the general forums ran into something similar when using vimsh to mount an NFS datastore, it seems to remember the original label, hence when you try to re-mount as something else it'll think it already exists. I think the user had to manually re-add with the same name, delete the NFS datastore and then manually re-mount using the new name.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

mcunningham44 · ‎10-09-2009

Would this also eliminate the mkdir: cannot create directory `/vmfs/volumes/nfs_storage_backup': Operation not permitted, that I've been starting to get? I never got this error before, and for some reason I started getting it today and I'm not sure whats causing it?

Also,

When I goto mount it manually via "/usr/bin/vmware-vim-cmd hostsvc/datastore/nas_create nfs_storage_backup x.x.x.x /NFS_SHARE 0" I get a message saying this already exists below:

(vim.fault.AlreadyExists) {
dynamicType = <unset>,
name = "",
msg = "The specified key, name, or identifier already exists."
}

mpaytonnj · ‎10-09-2009

First, thanks for this awesome script and for updating/answering questions. We have Enterprise licensing, including the "official" VCB, but were looking for something simpler than a VCB proxy setup for our ESXi servers. This fits the bill perfectly.

Second, rather than using a manually maintained list of machines to include in a backup we do something a little different and I figured I'd offer to share it in case anyone finds it interesting...

Due to various factors ( vmotion, VMs added/removed, VMs moved from dev to production, new hosts added/removed etc. ) our inventory is constantly changing on all hosts and manually updating a list of VMs to backup in multiple locations got to be a bit much.

Therefore we found it easier to maintain a single file of machines to exclude from backup, rather than multiple files of VMs to include.

Basically, we assume the VM should be backed up unless it's specifically excluded.

To accomplish this on ESX using vcbMounter, I get the VM names from /etc/vmware/hostd/vmInventory.xml ignoring any that are listed in a specified "exclude" file and passing the rest to vcbMounter to be backed up.

I modified this for ESXi/ghettoVCB.sh to write the VM name(s) to the file used as the argument to ghettoVCB.sh.

It still requires maintaining the exclude list, so it's possible to miss something. But in our environment this has proven to be easier/more effective, and all ESX(i) machines in our environment look at the same exclude file. Or if you don't want to exclude any VMs, leaving the exclude file blank means you no longer have to manually maintain any files.

So, if anyone is interested, I can post what passes for my "code".

Disclaimer...I am not a shell programmer. I know there are cleaner/more graceful ways of doing this than the hack I threw together. real programmers will probably laugh or throw rocks at me. But it works for us, it's only a few lines and it is probably easy to clean up/condense by someone with more experience.

mcunningham44 · ‎10-12-2009

Thanks for your help and the script!!

lamw · ‎10-14-2009

Hi everyone,

I'm currently in development (as time permits) on some new features for the ghettoVCB script and I'm looking for some individuals that may be interested in taking part in a BETA. The requirement is pretty simple, willing to help test and provide feedback on the next version of the script (remember to test on non-production VMs)

If you're interested, please PM on VTMN with your email address and the script you would like to take part in and I'll be selecting a group users and send out email notification in the coming weeks.

Thanks

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

DSTAVERT · ‎10-14-2009

I would dearly love to participate but fundamentally refuse to use twitter or expose an email address to it.

voro · ‎10-14-2009

I can help if you want. I have a spare R200 with latest ESXi 3.5 and a V4 Essential license for it.

Oh, I can't DM you on twitter, probably cause you're not following me

lamw · ‎10-15-2009

Drop me your email via PM on VMTN.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

lamw · ‎10-15-2009

Drop me your email via PM on VMTN.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

goppi · ‎10-16-2009

This is a really great extension to ESXi.

The only thing missing IMHO is the ability to exclude certain vmdks from backup.

You very often have setups where you only want to backup the system volumes of machines via image and not the datavolumes which are sometimes 100GB+ in size. These are most often beeing backed up by different systems with a more granular backup technology.

lamw · ‎10-16-2009

There is a BETA in the coming weeks, if you're interested drop me your email via VMTN PM

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

S3BASTI3N · ‎10-18-2009

for pb with 100GB+ (for data volume) I use an independant disk (wich is not affected by snapshot) and I use a modified GhettoVCB script : I add a BACKUP_INDEPENDENT_DISK option and if set to 0 , I exclude "independent" disk when the script "browse" all vmdk

It works fine until a new version of GhettoVCB comes with this option !

lamw, i'm interested by your beta

Sebastien

lamw · ‎10-18-2009

If you're interested in the BETA, please send me a PM on VMTN with your email.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

BasicVirtu · ‎10-20-2009

First off thank you, and great work on your scripts.

I recently upgraded to ESX4 and ESXi4 and would like to try using one of the scripts either ghettoVCB or ghettoVCB2 to copy vms from our ESX4 cluster to an ESXi server that has plenty of local storage. Would you recommend the newer script or the older script? I was hoping to use compression but did not see that feature in the new script. Also I would be willing to test any beta scripts since I am starting out with our test server. Thanks

lamw · ‎10-20-2009

ESX(i) 4.0 is supported by both ghettoVCB and ghettoVCBg2, please go over the documentation carefully and you'll see the use cases for each depending on the environment you're in. Compression is only available with ghettoVCB and it's not available for a certain reason in ghettoVCBg2.

Regarding the BETA, if you're interested please follow the directions in an earlier comment that specifies on how to apply.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

jberne74 · ‎10-21-2009

Thanks a mill for this cost saving solution William. I have everything up on my ESXi host, tied my NFS datastore to it, edited the VM_BACKUP_VOLUME variable appropriately, created the vm list file and changed permissions on the ghettoVCB.sh all using Putty on my windows machine to ssh into the ESXi host. But when i try to run the script, i still get the:

-ash: ./ghettoVCB.sh: not found

error.Can i get any help here. Thnx so much.

lamw · ‎10-21-2009

Make sure you're downloading the latest version which is attached to this VMTN document. Make sure you've edited the script directly on ESX(i) host (not with Windows system such as notepad). These are generally the issues people run into if they're hitting this error. How did you upload the script to your ESX(i) host?

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

jberne74 · ‎10-21-2009

I edited the script on the ESXi host using Putty.I uploaded the script to the ESXi host using WinSCP transfer(in text mode).I probably will go ahead and download the latest version attached to this VMTN document, edit and make the changes in the morning.I will let you know how this goes.Thanks a million again.

jberne74 · ‎10-22-2009

Thanks William.Everything's on track now. Apparently the original download was corrupt.Thanks again.

lamw · ‎10-22-2009

BETA email invitations will be going out this weekend ... keep an eye out

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

DSTAVERT · ‎10-23-2009

Since we are getting close to Halloween walking around with an eye out won't be that noticeable. Good timing.

stighenning · ‎10-23-2009

---

I found the ghettoVCB.sh to work great!

But how can I automaticlly generate the vm_backup_list.. so that the ghettoVCB.sh through cron will backup every vmware that is running on the VMWare ESXI 4.0 server ?

I tried using "cat /tmp/vms_list | grep | sed ....something " > new_file_backup_list

(but I cant get this to work..)

---

DSTAVERT · ‎10-23-2009

This code will generate a list of all VMs on the host.

vim-cmd vmsvc/getallvms | awk -F " " '{print $2}' | sed '1,1d' > /tmp/currentvms

You can create a script that includes that and add

ghettoVCB.sh /tmp/currentvms

You would need to add your paths etc and add your script to cron

stighenning · ‎10-24-2009

Hi,

Your command works perfect with the images-names that do not have spaces in the name.

But the images with spaces, i.e. "WIN 2K3 PRODUCTION" will not be in the list properly..

Is there a command that could cut out the "WIN 2K3 PRODUCTION" line in this /tmp/vms_list :

"";"";""

"Windows user/pass: administrator/";" ";""

"576";"WIN 2K3 PRODUCTION";"[vmware]";"WIN 2K3 PRODUCTION/Win2k3 32 bit.vmx"

"";"";""

lamw · ‎10-24-2009

The command provided by @DSTAVERT is not correct ... it's way more complicated than that when you have spaces/etc. that needs to be taken into consideration. The actual command is actually within the ghettoVCB script if anyone has taken a look at the source.

vmware-vim-cmd vmsvc/getallvms | sed 's/[[:blank:]]\{3,\}/   /g' | awk -F'   ' '{print "\""$1"\";\""$2"\";\""$3"\""}' |  sed 's/\] /\]\";\"/g' | sed '1,1d'

This will handle VM names that have spaces in their display name (a big no no in my books!)

Just substitute wither vmware-vim-cmd for classic ESX or vim-cmd for ESXi

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators