ghettoVCB.sh - Free alternative for backing up VM's for ESX(i) 3.5, 4.x, 5.x, 6.x & 7.x

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

tohe · ‎06-01-2010

Hi William,

maybe it was a misunderstanding, I did not mean that the backup failed. I wanted to mention that the deletion, following the rotation count, fails with my biggest VM (TITAN - you see in log)

Today I formated the NFS store with XFS, maybe with this filesystem the problem is gone.

lamw · ‎06-01-2010

That's correct, generally the NFS issue is hit on large deletions as mentioned earlier. If the backup was successful and no orphaned snapshots were left behind, then the fix that was implemented in the most recent release of the script worked!

Moving off of ext3 which is what I'm assuming your NFS volume was configured originally to XFS is the other fix which is described in the documentation above.

Glad to hear you got it working

=========================================================================

William Lam

VMware vExpert 2009

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

tohe · ‎06-01-2010

I noticed now that I have a really bad performance with XFS. The backup of my first sever lasts 3 times longer than with ext3. Do you have some recommendations for parameters to tune up the XFS performance?

The other option would be to go back to ext3 and try the

data=writeback mode

, is it correct?

What about an other filesystems like ext4? Do they have the same problem?

lamw · ‎06-01-2010

In terms of performnce tuning, I would recommend doing some research on the web, those will probably be your best resources. For those that reported back from switching from EXT3 to XFS, did not mention any performance issues, again this is not an issue we hit so it's hard for us to really say.

That's correct, you can try to change it to "writeback mode" and see if you're able to get around the NFS timeout issue.

Also other things that may affect your testing is the # of spindles behind your NFS volume, you may also want to research that with the various filesystems you'll be performing the test on and ensure you're using the most optimal configuration.

I'll be interesting to see what comes out of your testing and perhaps there is a better solution.

Thanks

=========================================================================

William Lam

VMware vExpert 2009

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

Whtiy · ‎06-10-2010

While setting up the script to backup our server i found two whings:

1. Round brackets

VMs with brackets in the name (like "machine xy (Test)") do not work. The brackets have to be escaped for the grep command.

2. Exclusion List

The VM name should only match an entry in the exclusion list if it is exactly the same. For example, i wanted to exclude an excluded machine for one run and added a "#" in front of the name in the exclusion list. But the name still matched because the grep command in the script also matches, if the machine name is a part of an line in the exclude file. So i changed the grep command a bit (line 475) from:

grep -E "$" "$" > /dev/null 2>&1 to: grep -E "^$\$" "$" > /dev/null 2>&1

Urs

diplan · ‎06-17-2010

Hello,

We have a problem with my backup and we can´t find the problem.

We use the ESXi 4.0.0 VMWare Server and ghettoVCB (last modified 05/12/2010)

We try to backup a VM with two disks. The first disk is located under "/vmfs/volumes/RAID-SAS/VM_1" (35GB). The backup works on this disk.

The second Disk is located under "/vmfs/volumes/RAID-SATA/VM_1"(800GB). Means the disks are in diffrent datastores. If i try to backup this disk the backup fail.

Below please find the logfile of the failed backup.

I already try to set the "SNAPSHOT_TIMEOUT" to "115", didn´t solve the problem.

Any ideas whats going wrong? Or need more information about configuration and environment?

Here is the log from the faild backup:

/ghettoVCB # ./ghettoVCB.sh -f ./vms_to_backup -g ./ghettoVCB.conf -d debug

2010-06-17 09:13:03 -- info: ============================== ghettoVCB LOG START ==============================

2010-06-17 09:13:03 -- debug: Succesfully acquired lock directory - /tmp/ghettoVCB.lock

2010-06-17 09:13:03 -- debug: HOST BUILD: VMware ESXi 4.0.0 build-244038

2010-06-17 09:13:03 -- debug: HOSTNAME: esx1

2010-06-17 09:13:03 -- info: CONFIG - USING GLOBAL GHETTOVCB CONFIGURATION FILE = ./ghettoVCB.conf

2010-06-17 09:13:03 -- info: CONFIG - VM_BACKUP_VOLUME = /vmfs/volumes/RAID-SATA/backup/

2010-06-17 09:13:03 -- info: CONFIG - VM_BACKUP_ROTATION_COUNT = 2

2010-06-17 09:13:03 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2010-06-17_09-13-03

2010-06-17 09:13:03 -- info: CONFIG - DISK_BACKUP_FORMAT = thin

2010-06-17 09:13:03 -- info: CONFIG - ADAPTER_FORMAT = buslogic

2010-06-17 09:13:03 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = 0

2010-06-17 09:13:03 -- info: CONFIG - ENABLE_HARD_POWER_OFF = 0

2010-06-17 09:13:03 -- info: CONFIG - ITER_TO_WAIT_SHUTDOWN = 3

2010-06-17 09:13:03 -- info: CONFIG - POWER_DOWN_TIMEOUT = 5

2010-06-17 09:13:03 -- info: CONFIG - SNAPSHOT_TIMEOUT = 15

2010-06-17 09:13:03 -- info: CONFIG - LOG_LEVEL = debug

2010-06-17 09:13:03 -- info: CONFIG - BACKUP_LOG_OUTPUT = stdout

2010-06-17 09:13:03 -- info: CONFIG - VM_SNAPSHOT_MEMORY = 0

2010-06-17 09:13:03 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = 0

2010-06-17 09:13:03 -- info: CONFIG - VMDK_FILES_TO_BACKUP = all

2010-06-17 09:13:06 -- info: Initiate backup for VM_1

2010-06-17 09:13:06 -- info: Creating Snapshot "ghettoVCB-snapshot-2010-06-17" for VM_1

2010-06-17 09:13:07 -- debug: Waiting for snapshot "ghettoVCB-snapshot-2010-06-17" to be created

2010-06-17 09:13:07 -- debug: Snapshot timeout set to: 900 seconds

2010-06-17 09:13:08 -- debug: Waiting for snapshot creation to be completed - Iteration: 0 - sleeping for 60secs (Duration: 0 seconds)

2010-06-17 09:14:09 -- debug: Waiting for snapshot creation to be completed - Iteration: 1 - sleeping for 60secs (Duration: 30 seconds)

2010-06-17 09:15:10 -- debug: Waiting for snapshot creation to be completed - Iteration: 2 - sleeping for 60secs (Duration: 60 seconds)

2010-06-17 09:16:11 -- debug: Waiting for snapshot creation to be completed - Iteration: 3 - sleeping for 60secs (Duration: 90 seconds)

2010-06-17 09:17:12 -- debug: Waiting for snapshot creation to be completed - Iteration: 4 - sleeping for 60secs (Duration: 120 seconds)

2010-06-17 09:18:13 -- debug: Waiting for snapshot creation to be completed - Iteration: 5 - sleeping for 60secs (Duration: 150 seconds)

2010-06-17 09:19:13 -- debug: Waiting for snapshot creation to be completed - Iteration: 6 - sleeping for 60secs (Duration: 180 seconds)

2010-06-17 09:20:14 -- debug: Waiting for snapshot creation to be completed - Iteration: 7 - sleeping for 60secs (Duration: 210 seconds)

2010-06-17 09:21:15 -- debug: Waiting for snapshot creation to be completed - Iteration: 8 - sleeping for 60secs (Duration: 240 seconds)

2010-06-17 09:22:16 -- debug: Waiting for snapshot creation to be completed - Iteration: 9 - sleeping for 60secs (Duration: 270 seconds)

2010-06-17 09:23:17 -- debug: Waiting for snapshot creation to be completed - Iteration: 10 - sleeping for 60secs (Duration: 300 seconds)

2010-06-17 09:24:18 -- debug: Waiting for snapshot creation to be completed - Iteration: 11 - sleeping for 60secs (Duration: 330 seconds)

2010-06-17 09:25:19 -- debug: Waiting for snapshot creation to be completed - Iteration: 12 - sleeping for 60secs (Duration: 360 seconds)

2010-06-17 09:26:20 -- debug: Waiting for snapshot creation to be completed - Iteration: 13 - sleeping for 60secs (Duration: 390 seconds)

2010-06-17 09:27:21 -- debug: Waiting for snapshot creation to be completed - Iteration: 14 - sleeping for 60secs (Duration: 420 seconds)

2010-06-17 09:28:22 -- info: Snapshot timed out, failed to create snapshot: "ghettoVCB-snapshot-2010-06-17" for VM_1

2010-06-17 09:28:22 -- debug: Removing /vmfs/volumes/4bf393c5-edde2dfb-7cb5-001517d7e5d5/backup/VM_1/VM_1-2010-06-16_11-08-26

2010-06-17 09:28:22 -- info: Backup Duration: 15.27 Minutes

2010-06-17 09:28:22 -- info: Error: Unable to backup VM_1 due to snapshot creation!

2010-06-17 09:28:22 -- info: ###### Final status: ERROR: All VMs failed! ######

2010-06-17 09:28:22 -- debug: Succesfully removed lock directory - /tmp/ghettoVCB.lock

2010-06-17 09:28:22 -- info: ============================== ghettoVCB LOG END ================================

jcbosacoma · ‎06-19-2010

Hello,

Where are you getting the latest version of ghettoVCB? http://sourceforge.net/ is not producing a download but an html file.

Thanks

jcbosacoma · ‎06-19-2010

Hello,

I am getting an error when running the script with no compression.

Failed to clone disk : The file already exists (39).

but creates a gz file on the target. Any thoughts?

Also if i run the script with no compression i get the following errors:

Failed to clone disk : The file already exists (39).
mv: unable to rename `/vmfs/volumes/WinL/AABBES.COM-2010-06-19': Input/output error

any thoughts?

Thank you...

lamw · ‎06-20-2010

Sorry for the late reply, I've been studying for an exam these last few weeks and just been swamped with other things.

I think I mentioned this somewhere in the documentation, but I'm not a fan of using spaces or special chars in a naming convention for your VMs or hosts. Yes, Windows and VMwares allows you to do this, but I prefer to use underscores or dashes to separate words.

Regarding #2, I'll go ahead and mark this down and make sure it's resolved in a future release of the script. Thanks for your findings.

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

lamw · ‎06-20-2010

Sorry for the late reply, I've been studying for an exam these last few weeks and just been swamped with other things.

I'm a little confused from your comments. You're able to backup this VM which has 2 VMDKs and one is successful and the other is not? Based on the logs, it's not able to create a snapshot which means if the VM was powered on a backup would have not happened.

Could you provide some more details in your exact execution? Are you trying to backup each VMDK and found that only the first one works but not the second?

FYI - Please refer FAQ #1 for posting logs.

Thanks

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

lamw · ‎06-20-2010

Sorry for the late reply, I've been studying for an exam these last few weeks and just been swamped with other things.

You may want to refer to FAQ #18

Thanks

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

sembsl · ‎06-21-2010

Hi, great little tool. I have used this to successfully backup a number of machines.

I noticed in the VMA client GUI that when it is doing the copy (the slow bit) there is a little percentage bar at the bottom. I like this feature and would like to incorporate this into the script.

The command that actually does the copy is; /usr/bin/vmkfstools --server "SSS" --username "UUU" --password "funkytmppass" -i "ds file" -a buslogic -d thin "ds file"

Is there any way to get the progress of the command? I've tried looking at the file size but it seems to allocate it all upfront... maybe a separate VMA interface call?

Any ideas appreciated.

diplan · ‎06-21-2010

Hello,

In the first step we backed up only disk number one of the VM. The backup worked well. The second disk was independend (no snapshot).

Then we tried to backup the vmdk file with the option "VMDK_FILES_TO_BACKUP=" didnt work.

Here are the 3 versions we try(ghettoVCB.conf):

VMDK_FILES_TO_BACKUP="VM_1_1_1.vmdk"

VMDK_FILES_TO_BACKUP="/vmfs/volumes/RAID5-SATA/VM_1/VM_1_1_1.vmdk"

VMDK_FILES_TO_BACKUP="/vmfs/volumes/4bf393c5-edde2dfb-7cb5-001517d7e5d5/VM_1/VM_1_1_1.vmdk"

This didnt work.

after that we configured the second disk to be "dependant" in order to be included in the snapshots.

Here the link to the logfile: http://pastebin.com/brtVtgLm

Summary:

1. When both disks are configured to be "dependant" the backup fails.

2. When only the first disk is configured to be "dependant" then the backup succeeds!

lamw · ‎06-21-2010

If you ran the vmkfstools interactively, you'll get a % completed but there is no bar that exists. This tool is meant to run in the background, if you need to see the current progress you can run it interactively OR tail the logs

If you're referring to the progress bar found in the vSphere Client, that is only available when using the vSphere API, when you manually call vmkfstools from the Service Console or the unsupported Busybox console, this is not reflected on the UI. ghettoVCBg2 uses the vSphere API, hence when logging directly into an ESX or ESXi host while the backup is being performed, you'll notice the progress for a given VMDK copy.

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

lamw · ‎06-21-2010

So the format of VMDK_FILES_TO_BACKUP variable is just the VMDK name, you don't need to specify the full path and if you do, it probably will not pass the checks as the script is looking for a specific name (e.g. mydisk.vmdk) This is clearly documented, so the 1st configuration is correct and should be used.

Secondly, make sure you disks are can be snapshotted.

I'll need the following pieces of information:

1) Can you please provide screen shots of your VM configurations, specifically for each of the VMDK. Click on the VM settings, highlight each VMDK and provide a snapshot of the configuration

2) You mentioned this VM resides on 2 separate datastores? Are they both VMFS or NFS? If its VMFS, can you provide the VMFS blocksize configured for each datastore that this VM in question resides in

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

sembsl · ‎06-21-2010

I ran the command interactively - it took about 5minutes - and completed successfully. But I did not see any percentage completed output.

I also tried extracting the logs (hostd, message) from the ESX server using vilogger - but these logs also did not contain a % output. Which logs were you referring to?

lamw · ‎06-21-2010

If you ran the copy command manually with vmkfstools, you should have seen a counter displaying the progress which should be changing on your screen.

The logs I'm referring to is the ghettoVCB logs that are being generated, they also capture the status. Also note that timestamps are captured in the log along with the completion time for your records

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

jjj0923 · ‎06-21-2010

ok - here's my problem.

I put ghettovcb.sh and the control file that contains the names of the file to backup in the root directory of my esxi3.5 box and after I reboot the server - the files are GONE!

why is this?

thanks in advance.

Jeff

lamw · ‎06-21-2010

Changes in the ESXi unsupported Busybox console are not persisted after reboots other than for a few configuration file. You should store your script in either the local VMFS volume or on shared storage else they'll be wiped each time the host reboots

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

sembsl · ‎06-21-2010

I must have screwed something up, because It didn't output anything.. I did find tho, if i put "--verbose 3" on the command line it spits out a bunch of XML, where one of the fields is a percentage . I will strip out that field.

mkennetha · ‎06-22-2010

Hi there,

My users need to retain multiple snapshots of their virtual machines. Is there any way to backup machines with snapshots? Would it work to:

a) create a snapshot

b) "vmkfstools -i 2nd_last_snap.vndk ...."

c) remove the last snapshot

Has anyone figured out the addressing scheme for snapshots using "vim-cmd vmcvs/snapshot.remove". The docs specify the arguments snapshotLevel and snapshotIndex, but I haven't figured out how they work with branched trees yet..

Thanks!

jjj0923 · ‎06-22-2010

multiple snapshots are accomodated for right in the script file

VM_BACKUP_ROTATION_COUNT=5

or whatever number you want to save and you can vary to backup schedule in the cron entry.

mkennetha · ‎06-22-2010

Thanks, but doesn't VM_BACKUP_ROTATION_COUNT just determine the number of backups to save for each virtual machine?

I need to be able backup virtual machines that have virtual machine snapshots. I don't really need to retain the snapshots in the backup, but I can't delete the snapshots on the virtual machines. From the "Features" section at the top of this page:

"VM(s) that intially contain snapshots will not be backed up and will be ignored"

lamw · ‎06-22-2010

@mkennetha you are correct, the variable VM_BACKUP_ROTATION_COUNT is used to determine the number of backups for each VM which is a complete backup of a given VM w/o snapshots.

As mentioned in the documentation and in FAQ #11, VMs w/prior snapshots is NOT supported. It's not a best practice to keep snapshots for long durations and more commonly they're miss-used and miss-understood. Here is another article on why snapshots are NOT meant to be used for backups - http://blogs.vmware.com/kb/2010/06/vmware-snapshots.html

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

mkennetha · ‎06-22-2010

Thanks for your reply.

I certainly agree that snapshot retention is neither resource-friendly nor suitable as a backup strategy.

The situation that I am confronted with is that the developers I support have a workflow in place which makes copious use of snapshots. Poorly conceived, perhaps, but nevertheless I do have to provide some kind of backup strategy.

Up to now, I've been more or less manually using the workflow described in my first post. I haven't been able to reliably script it yet, because I haven't been able to address snapshots after a snapshot tree has been forked. I suppose the "snapshotIndex" is relevant, but I haven't been able to find documentation and my own experimental poking has not yet yielded results.

giunat · ‎06-23-2010

Hi,

I'm using this script to backup my vm in vsphere environment (i send a batch from Windows).

The script work right with ESX4 ,but I have some problem with ESXi 4: the backup is OK (also the restore) but the log file is empty.

Kindly, can you help me?

Thanks for all

vmbru · ‎06-23-2010

Pretty nifty code.

Wondering...

1. Are you phasing off this script and telling users to switch to the ghettoVCBg2? If so skip #4.

2. To truly create a backup window, don't see a way to terminate the backup process if we want the backups to stop at... lets say 7am then resume at 6pm. Is this possible?

3. any dedupe features coming... like VDR has? or would that hose the esxi host too much? Backups are fast, the compressing of VM's to *.gz is the slowest. The backup/compress every night will take awhile for 30 or more VM's.

4. email feature? we have script that can email log but would be nice there were an email feature.

We're beta testing VDR, but I need something for production until Azmir fixes the slow Integrity Checks. : ) waiting patiently.

- VMware truly needs to focus efforts on its backup stragety of VM's for its Advanced or higher license customers, the core software and other features are fine, VDR just needs more attention.

lamw · ‎06-23-2010

Have you taken a look at some of the resources above in the documentation with regards to executing the script from a Windows environment? That should give you some tips on setting it up, if you're sending the correct command, it should log to the destination datastore

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

giunat · ‎06-23-2010

Hi William,

thanks for reply.

this is the batch file that i have scheduled in Windows:

C:\backup_vm\plink.exe root@10.1.0.13 -pw password "nohup /vmfs/volumes/4bbc4a19-fe5178cd-1a04-d8d3855fa43e/copia_vm/ghettoVCB.sh -f /vmfs/volumes/4bbc4a19-fe5178cd-1a04-d8d3855fa43e/copia_vm/serverweb > /vmfs/volumes/ce5226e0-34047eed/Backup/backuplogserverweb.txt &"

The backup is correct but the log is empty(ESX 4i).

The same script works correctly in ESX4

I have read some tips but I'm unable to identify the solution.

Thanks

lamw · ‎06-23-2010

Hello,

1. I'm not phasing ghettoVCB, on the contrary, I have more users on this script than ghettoVCBg2. The primary reason I can think of, is ghettoVCB supports free ESXi which is used for many small shops, labs and even some mid size shops. I continue to get feedback from the community and provide enhancements/fixes as appropriate and as time permits. Please refer to FAQ #15 for more questions on enhancement requests

2. Not possible and I don't plan on supporting this type of complicated backup process. Remember this solution is not doing any type of "incremental" backup, this is using vmkfstools which I expect users of this script to understand how that works and how the backups are actually taking place. If you understand that, then you'll see that implementing a "pause" option would not work

3. Please refer to comment #2 from above and FAQ #25. If you're looking for de-dupe capabilities, this will have to be on the actual storage subsystem whether that be an NFS/ZFS server or an array that has de-depe capabilities. Things like change block tracking is not possible using vmkfstools, this is provided via the vSphere API and is implemented in various commercial backup solutions such as VMware vDR, Veeam Backup/Replication, etc. if you're looking for those features, you need to look at a commercial product. As much as I would love to implement a ghettoCBT, it just does not exists or available via the methods this script

4. Yes it would be nice, there have been some posts on the internet on getting email sent from classic ESX, the problem arises when you're dealing with ESXi. If you need email capabilities, have a remote management system grab the logs after the backup or after n-period of time and email it to you, this allow for custom configuration/etc. if a user wanted the email to come from a centralized system. No plans to support this in the future

I've heard horror stories with vDR and from what I "hear", it's still a 1.0 product. If you're looking for a mature and reliable commercial product, take a look at Veeam Backup and Replication.

Hopefully this answered your questions

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

http://pastebin.com/tY1Ypby3

If you find this information useful, please award points for "correct" or "helpful".

XavierE · ‎06-23-2010

Giunat,

I personally use the vMA appliance -you could use any linux flavor- to remotely via SSH trigger the ghettovcb script on ESX/ESXi hosts. The backup repository is a NFS server attached to all hosts which also centralizes the configuration/scripts. It has been working really good and have found no problems. I wrote a small script to handle the logging and email notifications that include the backup log.

If you are interested on implementing in this -similar- way let me know and I can email you more details.

traugust · ‎06-23-2010

Hi Wiliam,

Thanks for this great tool - it really helps all of us 'part-time-admins' in small projects a lot!!

I would like to mention 3 things:

1. I ran into a problem when the NFS mount ran out of space during disk clone. This was reported by vmkfstools on the screen and in the debug log as follows:

--- snip ---
2010-06-23 11:19:08 -- debug: Waiting for snapshot "ghettoVCB-snapshot-2010-06-23" to be created
2010-06-23 11:19:08 -- debug: Snapshot timeout set to: 900 seconds
Destination disk format: VMFS thin-provisioned
Cloning disk '/vmfs/volumes/vmsOnC2edev13/vmc2eocs03/vmc2eocs03.vmdk'...

Clone: 0% done.
Clone: 1% done.
Clone: 2% done.
Clone: 3% done.
Clone: 4% done.
Clone: 5% done.
Clone: 6% done.
Clone: 7% done.
Clone: 8% done.
Clone: 9% done.
Clone: 10% done.
Clone: 11% done.
Clone: 12% done.
Clone: 13% done.
Clone: 14% done.
Clone: 15% done.
Clone: 16% done.
Clone: 17% done.
Clone: 18% done.
Clone: 19% done.
Clone: 20% done.
Clone: 21% done.
Clone: 22% done.
Clone: 23% done.
Clone: 24% done.
Clone: 25% done.
Clone: 26% done.
Clone: 27% done.
Clone: 28% done.
Clone: 29% done.
Clone: 30% done.
Clone: 31% done.
Clone: 32% done.
Clone: 33% done.
Clone: 34% done.
Clone: 35% done.
Clone: 36% done.
Clone: 37% done.
Clone: 38% done.
Clone: 39% done.
Clone: 40% done.
Clone: 41% done.
Clone: 42% done.
Clone: 43% done.
Clone: 44% done.
Clone: 45% done.
Clone: 46% done.
Clone: 47% done.
Clone: 48% done.
Clone: 49% done.
Clone: 50% done.
Clone: 51% done.
Clone: 52% done.
Clone: 53% done.
Clone: 54% done.
Clone: 55% done.
Clone: 56% done.
Clone: 57% done.
Clone: 58% done.
Clone: 59% done.
Clone: 60% done.
Clone: 61% done.
Clone: 62% done.
Clone: 63% done.
Clone: 64% done.
Clone: 65% done.
Clone: 66% done.
Clone: 67% done.
Clone: 68% done.
Clone: 69% done.
Clone: 70% done.
Clone: 71% done.
Clone: 72% done.
Clone: 73% done.
Clone: 74% done.
Clone: 75% done.
Clone: 76% done.Failed to clone disk : No space left on device (1835017).
2010-06-23 12:05:29 -- info: Removing snapshot from vmc2eocs03 ...
2010-06-23 12:06:04 -- info: Backup Duration: 46.95 Minutes
2010-06-23 12:06:04 -- info: Successfully completed backup for vmc2eocs03!

2010-06-23 12:06:05 -- info: Initiate backup for vSphere Management Assistant
--- snip ---

As you can see ghettoVCB did not find this situation to be an error and continued the backup and reported backup success. Ist there any chance to change this? Perhaps checking the vmkfstools call's return value?

2. I built a small customization for logwatch (see http://www.logwatch.org/) that parses a directory containing the ghettoVCB log files of several hosts (i.e. on the central NFS backup datastore) and outputs a report on the executions and statuses of the backup operations that were performed as cron jobs on several ESXi hosts. And the logwatch tool runs as cron job on the Server that provides the NFS share to the ESXI hosts. The E-Mail report looks as follows \[due to testing the error-detection there are more error messages that success messages in the report, but that errors were produced intentionally ]:

 ################### Logwatch 7.3 (03/24/06) #################### 
        Processing Initiated: Thu Jun 24 08:17:00 2010
        Date Range Processed: all
      Detail Level of Output: 0
              Type of Output: unformatted
           Logfiles for Host: xxxxxxxxxxxxxxxxxx
  ################################################################## 
 
 --------------------- GhettoVCB VM backups Begin ------------------------ 

 !!!WARNING!!! You may have backup errors
 localhost.localdomain:vmc2eocs03 @ 2010-06-23 11:19:08:
 		Clone: 76% done.Failed to clone disk : No space left on device (1835017).: 1 time(s)
 localhost.localdomain:vmc2eocs03 @ 2010-06-23 13:03:42:
 		Clone: 76% done.Failed to clone disk : No space left on device (1835017).: 1 time(s)
 localhost.localdomain: @ 2010-06-22 11:07:25:
 		 -- info: ###### Final status: ERROR: All VMs failed! ######: 1 time(s)
 localhost.localdomain: @ 2010-06-22 11:07:01:
 		 -- info: ###### Final status: ERROR: All VMs failed! ######: 1 time(s)
 localhost.localdomain: @ 2010-06-23 07:02:02:
 		 -- info: ###### Final status: ERROR: All VMs failed! ######: 1 time(s)
 localhost.localdomain: @ 2010-06-23 12:11:28:
 		 -- info: ###### Final status: ERROR: Only some of the VMs backed up! ######: 1 time(s)
 
 
 ------------------------------------------------------------------
 ------------------------       Backup summary      ---------------
 ------------------------------------------------------------------
 Host:			Build:
 	VM:
 		Backup     			Success:   	duration:
 ------------------------------------------------------------------
 localhost.localdomain	VMware ESXi 4.0.0 build-208167
 	vSphere Management Assistant
 		2010-06-23 12:06:05	success	5.38 Minutes
 ------------------------------------------------------------------
 	vmc2eappsvct-esxi
 		2010-06-22 06:33:03	success	9.15 Minutes
 		2010-06-22 07:00:03	success	9.15 Minutes
 		2010-06-22 11:07:01	!!omitted!!	- unknown -  
 		2010-06-22 11:07:25	!!omitted!!	- unknown -  
 		2010-06-23 07:02:02	!!omitted!!	- unknown -  
 		2010-06-23 07:31:48	success	9.15 Minutes
 		2010-06-23 11:14:26	- unknown -	- unknown -  
 		2010-06-23 11:19:06	!!omitted!!	- unknown -  
 		2010-06-23 12:52:41	success	10.98 Minutes
 ------------------------------------------------------------------
 	vmc2eocs03
 		2010-06-23 11:19:07	!!Error!!	- unknown -  
 		2010-06-23 13:03:41	!!Error!!	- unknown -  
 ------------------------------------------------------------------
 ------------------------------------------------------------------
 esxihost2.x.com	VMware ESXi 4.0.0 build-208167
 	FW-Builder
 		2010-06-24 05:33:32	success	12.43 Minutes
 ------------------------------------------------------------------
 	VAS win2k
 		2010-06-24 05:56:59	success	17.02 Minutes
 ------------------------------------------------------------------
 	vasa xp
 		2010-06-24 05:46:00	success	10.97 Minutes
 ------------------------------------------------------------------
 	vmccccontrol
 		2010-06-24 06:14:02	- unknown -	- unknown -  
 ------------------------------------------------------------------
 ------------------------------------------------------------------
 
 ---------------------- GhettoVCB VM backups End ------------------------- 

 
 ###################### Logwatch End #########################

I assume this to be a good starting point for a community tool - but I have not the time to maintain this script for the community and provide support and a comprehensive documentation for the tool - So I would leave a *.tar.gz somewhere with some short usage notes and provide this script as is. Where and how would I leave such a script?

3. One information I woluld like to have parseable in the log file (for my logwatch report ) would be the actual disk size of the backup performed (and perhaps the free space on the backup datastore before and after backup operation). This would be a useful information for my logwatch report, so in case you are doing work on your script anyway this would be a nice addition.

Again, thanks for your great work

Greetings

Thomas

gautelund · ‎06-24-2010

1. Good catch! I presume all we need to do is add something like this to the script immediately after each vmkfstools clone operation:

(untested yet)

2. Up to you IMHO.

3. Interesting. Maybe you would propose a patch?

gautelund · ‎06-24-2010

William,

after your updates in april-may it has been a few weeks before I got around to try it out on our servers (instead of my own cooked version). It appears to be a "drop in replacement", and of course I'd prefer to use "upstream" instead of a separate version.

Nevertheless I still have a few questions. They might be a bit lengthy for a comment here, so they're here: http://pastebin.com/EkfWfG2K

Take a look and let me know.

lamw · ‎06-24-2010

Hi traugust,

Thanks for the commends, I'm aware of this scenario and will be properly handled in a future release. I'll make a note of your comments but depending on my free cycles, some of these may or may not make it into future release.

Very cool report btw.

Thanks

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

lamw · ‎06-24-2010

gautelund,

Let me take a look at get back to you, got lots on my plate Remember this is not the only script I manage and have to keep up to date.

Thanks for the feedback

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

http://yfrog.com/mtvmdisk1jx

If you find this information useful, please award points for "correct" or "helpful".

diplan · ‎06-29-2010

Hi,

Here are the two screenshots.

Both disks are VMFS typed. The First disk has a maximum disk size 256GB. The second disk 1024GB maximum size.

The first disk is stored in RAID5-SAS with blocksize 1MB.

The second disk is stored in RAID5-SATA with blocksize 4MB

lamw · ‎06-29-2010

This can be an issue if your main VM configuration file is stored on a VMFS volume with a smaller blocksize configured than it's other VMDK with larger block size. We personally hit a snag with a similar configuration that caused our VM to actually crash as documented here - http://www.virtuallyghetto.com/2010/05/vsphere-esx-40-crash-vm-bug.html

It's probably best to either keep the same blocksize or if you're going to have a mix, ensure that the VM's configuration file is actually stored on the VMFS volume with the larger blocksize else you will be restricted by the VMFS volume with the small blocksize.

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

bklynjae · ‎07-01-2010

If we are running this script locally on a shared lun and configuring it to run from cron. Would the cron job be configured on the VMA?

Do you have a vma/ghettovcb2 for dummies.. Sorry..

lamw · ‎07-01-2010

It does not have to be on vMA, any remote manage host will do just fine. You can also schedule it via Windows using the task scheduler. You just need to ensure the scheduled task/cron is outside of the ESX or ESXi host

FYI - You're posting in ghettoVCB and not ghettoVCBg2

In either case, the documentation has information on setting up a cronjob

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

http://pastebin.com/Gr2dcZvq

If you find this information useful, please award points for "correct" or "helpful".

diplan · ‎07-07-2010

Hello,

Thank you for your help. After i moved all date on the SAS store to the SATA store i cant delete the SAS store in the GUI.

I solved the problem with the vmkfstools:

vmkfstools --createfs vmfs3 --blocksize 4M -S RAID5-SAS.new /vmfs/devices/disks/mpx.vmhba2\:C0\:T0\:L0\:3

Now i have an other problem:

If i try to backup the VM i get this message:

2010-07-07 13:45:48 -- info: Snapshot found for VM_1, backup will not take place

I already tryed to delete the snapshot in the vClient. I used the "delete all" button in the snapshot magager. After i delete all snapshots i get the same errormessage again.

Here is a list with existing files after deleting snapshots in the manager:

After deleting the snapshot the ghetto script tells me that snapshots are found anyway. How can i delete the snapshot that the gehtto script is working?

lamw · ‎07-07-2010

The script searches for 2 types of files: *-delta.vmdk (snapshot descriptor) and *.vmsn (snapshot state files). You will need to get rid of these else you'll continue to hit the error. I would recommend manually creating a snapshot and then deleting it, hopefully this will cleanly remove any stale files leftover. If not, then you can manually delete these .vmsn files only

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators