ghettoVCB.sh - Free alternative for backing up VM's for ESX(i) 3.5, 4.x, 5.x, 6.x & 7.x

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

Whtiy · ‎08-20-2010

Hi

Wanted to exclude one single VMDK from a machine, but did not found how to do it. It's listed in the features list:

- Ability to include/exclude specific VMDK(s) per VM (requires individual VM backup policy setup)

Don't want to do an include list, because, if i add a new disk to the VM and forget to add it to the include list, it is not backed up. And that could end ins some troubles...

Thanks

Urs

tohe · ‎08-20-2010

Hi William,

do you have plams to implement an OVF option, I mean backup the VMs in OVF?

lamw · ‎08-20-2010

Yes this is a feature that is supported, please take a look at the documentation for an example. Specifically search for "Backup policy for "VCAP" (backup only 2 specific VMDKs)" and you'll see an example of the VM backup policy file that is required to backup individual VMDKs

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

lamw · ‎08-20-2010

No plans, remember that we're still running the script in either the classic ESX Service Console or ESXi's Busybox console. There is no support for the ovftool and it may install on classic ESX, though it definitely won't install on ESXi.

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

ralfoertner · ‎08-22-2010

Hello!

Having mails of the3 logs would be a very interesting feature for me.

But how do I set it up?

I filled all the fields mail server and so on.

But where must netcat be placed on the esxi?

If yes in which directory?

And what version must be used?

A fedora or can it also be a debian?

Is it true I have to upgrade to esxi 4.1 (I'm using 4.0 now)?

Thank you for your help!

Ralf Oertner

Whtiy · ‎08-22-2010

Hello William

Specifically search for "Backup policy for "VCAP" (backup only 2 specific VMDKs)" and you'll see an example of the VM backup policy file that is required to backup individual VMDKs

Thats exactly what i don't want. The example is about making a list of files to include (Only backup the VMDK's which are in the list). I need (would like to have) a list of VMDK's to exclude[/b] from being backed up. Backup all VMDK's except the ones in the list.

Thanks

Urs

lamw · ‎08-23-2010

Sorry, to do what you're asking for you'll need to specify VMDKs you wish to backup, any that are not listed will not be backed up.

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

http://pastebin.com/g4WfvUXd

If you find this information useful, please award points for "correct" or "helpful".

RSW-Orga · ‎08-24-2010

Hello William,

i experimented with your backup script(07/27/2010). i worked very well. but i found a little bug and made the email a little prettier.

Hope this is the right way to send you this bugreport

Here is the patch to fix it:

Thanks,

Dieter

lamw · ‎08-24-2010

@RSW-Orga Much appreciated. I'll take a look after VMworld, definitely planning a minor release with few features + fixes.

Thanks

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

http://pastebin.com/zBUkKhzp

If you find this information useful, please award points for "correct" or "helpful".

chrb · ‎08-25-2010

I noticed the latest version of the script has a mail option. You may be interested in this mail script I wrote:

I call it from a nightly buildbot (you could call it from cron). It produces a concise email of which hosts were backed up and which couldn't be (snaphots). You will need to modify it for your local setup.

nsrugis · ‎08-25-2010

Does anyone know how to stagger the backups of a ghettovcb script? I looked through the comments and googled it with no luck. I have 5 guests on 2 hosts and want to back them all up using ghettoVCB and cron. Thanks!

Nate

spitzmann2 · ‎08-26-2010

I'd like to store VM backups in another ESXi host directly. the GhettoVCB script has an option to store backups in another computer by NFS. How can I use that for an ESXi host destination ?

What's the best way to deal with VM backups when there's no "entherprise class" solution around ? I only have two servers running ESXi 4.1 and a USB mass storage with 500GB. What can I do to make a good backup solution with that ?

AdiNugraha · ‎08-30-2010

Hi, can anyone explain the DISK_BACKUP_FORMAT parameter ? I'm trying to create thin disk backups but all the backup I did using the script comes out as thick disks..

HMuster · ‎09-01-2010

Hi jayclen,

I've experienced the same problem with ESXi 3.5 U5. I almost became desperate, but then I saw your hint and it works fine. Thank you!

Regards

eqj · ‎09-01-2010

In MKBackup you can specify the name of the VM according to the alias in VSphere. How does this work? And how do you do this by executing the ghettoVCB.sh script directly? There is VMDK_FILES_TO_BACKUP but that is the VMDK name and not the alias.

ccrome · ‎09-01-2010

For anybody having the following problem:

info: ERROR: failed to locate and extract VM_ID for

I found that on my setup, my line endings in my vms-to-backup file got translated to DOS style CRLF, so there were extra ^M at the end of the line. This caused the script grep of /tmp/vms_list to fail. The line in the script was:

VM_ID=`grep -E "\"$\"" /tmp/vms_list | awk -F ";" '{print $1}' | sed 's/"//g'`

but since VM_NAME had a ^M in it, the grep failed.

The simple solution was to translate the line endings back to unix style, and everything worked just fine.

BinarySolo · ‎09-03-2010

Hi William,

This has worked nicely for me - many thanks. One question though:

I notice that the EVENTS tab in the esxi client does not display information regarding the backup being performed. Is this what you would expect or does the absence of events indicate a problem with my backup?

Thanks,

Peter

JEJensen · ‎09-08-2010

GhettoVBC on ESX(i) 4.1.0 give the same messages

You're not running ESX(i) 3.5+ or 4.0+!

Any solution for this ?

chrb · ‎09-08-2010

You have an old version of ghettovcb. The new versions work with ESXi 4.1.

yulena · ‎09-08-2010

Hi, thanks for the Script is nice, well ..... , i have a issue when i try to backup 2 VM.

I have a file with the names of the VM and we use the cron for execute the script, but only backup the first VM of the file, if we change the order of the name then backup only the first, and in the log at the end say" delete the snap of VMNAME ......", but dont show the intention to try to do the other snap.

I will like to know if there is something to do to fix my problem.

Thank for any help.

See u.

JEJensen · ‎09-09-2010

You're not running ESX(i) 3.5+ or 4.0+!

And ghettovcb-restore.sh do you have a solution for this script ?

GarFin1965 · ‎09-12-2010

Hi, i had the same problem as you , forwarding to an IIS server, which also expected "$" echo "QUIT" >> "$"

sed "s/$/`echo -e
\r`/" "$" > "$".DOS

"$" "$" "$" < "$".DOS > /dev/null 2>&1 if ; then logger "info" "ERROR: Failed to email log output to $:$ to $\n"

fi

rm -rf "$" rm -rf "$".DOS

fi

afragola · ‎09-13-2010

Hi. Thank you for your great work

is there a way to schedule and rotate backup on 2 different directory?

i mean, day 1 it backups on a directory, day 2 on a second directory, day 3 backup on directory 1 BUT only if in directory 2 is present previous backup, so that backups can be divided on different datastores for higher security...

is it possible to take care of this situation? monitor backup and rotation this way?

thanks for any kind answer

lamw · ‎09-15-2010

Hi,

Rotation is only supported on on directory per VM, this is not a feature that I plan on supporting.

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

lamw · ‎09-15-2010

The solution is actually pretty simple, I've just not had the time to update. Now that VMworld is over, I should be able to get a new release of the script. To be honest, the fix is just adding 4.1 build number into the script, everything else should stay the same. You can take a look at the code that does the version check from ghettoVCB.sh and add that to ghettoVCB-restore.sh.

I don't have the time to verify the restore, but you're more than welcome to let me know if you hit any issues and please post anything released to ghettoVCB-restore here -

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

http://www.waldrondigital.com/2010/05/11/ghettovcb-e-mail-rotate-logs-batch-file-for-vmware/

If you find this information useful, please award points for "correct" or "helpful".

waldrondigital · ‎09-16-2010

gvlog2em update - Solution for downloading/parsing ghettoVCB backup logs in Windows without RCLI:

I could not get the experimental email feature to work, and I sold the NAS/NFS/SMB box from my environment where I used to store the backup logs and point my gvlog2em batch file to (SMB). I added some instructions on my blog for using WGET.exe to pull your log files from the web-based datastore without needing RCLI tools. You simply pull the log using WGET, place it on your local Windows disk, and then call my gvlog2em.bat log parser.

Scroll to the bottom of the post where the text is in RED:

RSW-Orga · ‎09-29-2010

Hi

I saw you released a new version 2010-09-28. there is still this little bug in line 257:

should be

awk '{print $1}'

lamw · ‎09-29-2010

Hi RSW-Orga,

Thanks for making a note of this, this has been fixed and you can download the latest version now.

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

philled99 · ‎09-29-2010

Hi. This is a great script, but I have a couple of questions.

1) I am using the experimental email support and have noticed that the email settings in VM specific config files in a separate directory get overriden by the email settings in ghettoVCB.sh. Surely the config files specified in the directory by -c should take precedence, otherwise anyone using the -c option is going to end up spamming you with emails (as it's your email details in ghettoVCB.sh by default)?

2) I can't get the functionality to backup a subset of a VM's VMDK files to work. With the setting VMDK_FILES_TO_BACKUP="Ash.vmdk" I get these VMDK files backed up:

/vmfs/volumes/4c2b6a95-f03e05a5-9f13-0019bb5303ea/Ash-Samba-Backups/Ash_1_1.vmdk

/vmfs/volumes/4c2b6a90-ce10ce23-db24-0019bb5303ea/Ash-Homes/Ash_1.vmdk

/vmfs/volumes/4c2b6a90-ce10ce23-db24-0019bb5303ea/Ash/Ash.vmdk

But with the setting VMDK_FILES_TO_BACKUP="Ash.vmdk" I don't get any backed up. What am I doing wrong?

lamw · ‎09-29-2010

Hi Philled99,

1) Remember when using the -c option, you are only specifying specific configuration at a VM level, this has nothing to do with global configurations which is contained in either ghettoVCB.conf. Email settings is one of the various ghettoVCB global configurations which can be controlled directly in the script or specifying a configuration file using -g option. Please carefully look over the documentation for more details and examples

2) Please take a look at FAQ #0 and #1

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

http://pastebin.com/k5wRJU2S

If you find this information useful, please award points for "correct" or "helpful".

kamal2222ahmed · ‎10-06-2010

Hi Lam,

I have read through the whole FAQ and comment section but still have the nfs issue even though i created a new NFS share and formatted it as XFS.

Here is the configuration:

Number of VM's = 9

Size of VM to be backed up = 60 GB

VMware ESXi 4.1.0 build-260247

For execution and dry run, see stdout screen capture at :

I would appreciate if anyone can help and resolve this nfs issue once and for all.

Thanks,

-Kamal.

NickelNetworks · ‎10-06-2010

Hello all, using ghettovbc on ESX 4.1 and a Nexenta NFS storage via a cron with good success.

Question is can the script check the target storage space before trying to run a backup for a VM? Last night ran out of space and the error reported is:

2010-10-06 00:23:15 -- info: Initiate backup for TRB1

2010-10-06 00:23:16 -- info: ERROR: error in backing up of "/vmfs/volumes/nas-store01/TRB1/TRB1.vmdk" for TRB1

2010-10-06 00:23:16 -- info: Backup Duration: 1 Seconds

2010-10-06 00:23:16 -- info: ERROR: Unable to backup TRB1 due to error in VMDK backup!

There is no error on the VMDK, there is just no space on the target NFS only.

Besides this minor issue it works great.

Also, is the new VM Paravirtual SCSI driver supported?

Regards,

Paul Aviles

www.nickelnetworks.com

RSW-Orga · ‎10-06-2010

i don´t think this kind of check should be part of the script. There might be systems that do not behave good when you try to determine the free space left. ( Had a problem a few years ago, when a df-command took about 30min to finish).

I think it would be better if you could create some pre-run, post-run scripts that get called before or after each backup.

With this you should be able to analyse your free disk space and work some other magic to make sure the backup will succeed.

kamal2222ahmed · ‎10-06-2010

Hi Paul,

Is there any notes or documentation, on how to setup Nexenta NFS storage. Did you setup Nexenta as a Virtual Machine ? I am using openfiler http://www.openfiler.com/

Thanks,

-Kamal.

NickelNetworks · ‎10-06-2010

Perhaps there is a way in the API to check for the total and free space. Based on the fact that VMW is moving away from the linux console in future releases it probably is a bad idea on relaying on DF or linux commands if there is a way via some API to get the same.

Regards,

Paul Aviles

www.nickelnetworks.com

NickelNetworks · ‎10-06-2010

Using Nexenta on a regular physical server. This was an HP X1400 Storage Server which is nothing else than an HP DL 320 G6. I got tired of wasting my time with Windows and HP and found Nexenta one day and been very impressed with the performance.

What I did is created a 40 GB partition and installed Nextenta on it.

I also created a larger raid 5 pool for my VM's and a log pool of 6 GB. (it is used as a caching thing I guess)

Then I created the volume using the definitnion I did on the raid 5 pool. All this is done on the Data Management, Data Sets menus.

Then on Data Management / Shares I selected Create a New Folder and used the following fields.

Name: nas01-vol1/nas-store01 (nas01-vol1 is the volume name created on which the pools exist. A volume has the pools and logs)

Folder Name: whatever name you want

Description: whatever description you want

Record Size: 128K

Log BIAS: Latency

Deduplication; off

Compression: off

Number of Copies:1

Case sensitive: mixed

Unicode Only: off

Sync: standard

Once is created you go back to the Data Management / Shares / Show Summary and there you have checkboxes on the right for CIFS, NFS, FTP, RSYNC, etc. Check on NFS and then Edit. On Auth Type I used the first option on the Pull Down.

Anonymous is checked (must be checked, but security is disabled or controlled by a field called Extra Options later)

Read-Write: *

Root: @10.5.36.0/24 (this is my local subnet)

Extra Options: anon=0 (this disables anonymous access)

Then in ESX you add the storage as follows:

Server: IP of Nexenta

Folder: /volumes/nas01-vol1/nas-store01

Datastore Name: nas-store01

That is pretty much it.

Regards,

Paul Aviles

www.nickelnetworks.com

lamw · ‎10-06-2010

Hi NickelNetworks,

I'll have to agree with RSW-Orga, this check should not be part of the script. The expectation is that the destination datastore should have sufficient space for your backups, after all, these are for your backups. Yes, we can perform clever tricks to check for space, but there are some down sides as mentioned plus the other issue is how do you check for "real" free space? What if systems are thin provisioned? What if the storage volume is thin provisioned on the array? What is free space at that point? I think this is something that should be monitored from an organization and alerted on if you're low on free space and not the job of the script. Sounds like there needs to be some monitoring setup for your backup datastores if you're running out of space

Regarding any type of API calls .... remember all operations are happening in the Service Console or Busybox Console, we're not using any APIs,etc.

Thanks

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators

http://www.waldrondigital.com/2010/05/11/ghettovcb-e-mail-rotate-logs-batch-file-for-vmware/

If you find this information useful, please award points for "correct" or "helpful".

waldrondigital · ‎10-09-2010

Just updated my gvlog2email Windows batch to version 0.99. The new version has built-in functionality to pull the ghettoVCB logfile out a datastore using WGET (no need for CLI tools, making it very portable). All you need is a Windows machine to run/schedule the task.

A great way to get notifications of your backup status if the built-in functionality of gvcb isn't working for you or is not sufficient for your needs. I use this script personally and at clients - IT WORKS!

serez · ‎10-15-2010

hello TS

i`ve been going through your guide (until comments) since 4days ago to understand how to use this thing. and i think that my problem isnt here yet.

i`m currently testing this script onto esxi 4.1.0 and after configured everything i found that no VMs are backed up.

Log :

/ghettoVCB # ./ghettoVCB.sh -a

2010-10-15 15:59:13 -- info: ============================== ghettoVCB LOG START ==============================

2010-10-15 15:59:13 -- info: CONFIG - GHETTOVCB_PID = 70106

2010-10-15 15:59:13 -- info: CONFIG - VM_BACKUP_VOLUME = /espinBackup

2010-10-15 15:59:13 -- info: CONFIG - VM_BACKUP_ROTATION_COUNT = 3

2010-10-15 15:59:13 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2010-10-15_15-59-13

2010-10-15 15:59:13 -- info: CONFIG - DISK_BACKUP_FORMAT = zeroedthick

2010-10-15 15:59:13 -- info: CONFIG - ADAPTER_FORMAT = buslogic

2010-10-15 15:59:13 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = 0

2010-10-15 15:59:13 -- info: CONFIG - ENABLE_HARD_POWER_OFF = 0

2010-10-15 15:59:13 -- info: CONFIG - ITER_TO_WAIT_SHUTDOWN = 3

2010-10-15 15:59:13 -- info: CONFIG - POWER_DOWN_TIMEOUT = 5

2010-10-15 15:59:13 -- info: CONFIG - SNAPSHOT_TIMEOUT = 15

2010-10-15 15:59:13 -- info: CONFIG - LOG_LEVEL = info

2010-10-15 15:59:13 -- info: CONFIG - BACKUP_LOG_OUTPUT = stdout

2010-10-15 15:59:13 -- info: CONFIG - VM_SNAPSHOT_MEMORY = 0

2010-10-15 15:59:13 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = 0

2010-10-15 15:59:13 -- info: CONFIG - VMDK_FILES_TO_BACKUP = all

2010-10-15 15:59:13 -- info: CONFIG - EMAIL_LOG = 0

2010-10-15 15:59:13 -- info:

2010-10-15 15:59:14 -- info: Initiate backup for WinXP Test

2010-10-15 15:59:14 -- info: Creating Snapshot "ghettoVCB-snapshot-2010-10-15" for WinXP Test

2010-10-15 15:59:17 -- info: ERROR: error in backing up of "/vmfs/volumes/datastore1/WinXP Test/WinXP Test.vmdk" for WinXP Test

2010-10-15 15:59:20 -- info: Removing snapshot from WinXP Test ...

2010-10-15 15:59:20 -- info: Backup Duration: 6 Seconds

2010-10-15 15:59:20 -- info: ERROR: Unable to backup WinXP Test due to error in VMDK backup!

2010-10-15 15:59:20 -- info: Initiate backup for OpenSUse Test

2010-10-15 15:59:20 -- info: Creating Snapshot "ghettoVCB-snapshot-2010-10-15" for OpenSUse Test

2010-10-15 15:59:23 -- info: ERROR: error in backing up of "/vmfs/volumes/datastore1/OpenSUse Te

2010-10-15 15:59:27 -- info: Removing snapshot from OpenSUse Test ...

2010-10-15 15:59:27 -- info: Backup Duration: 7 Seconds

2010-10-15 15:59:27 -- info: ERROR: Unable to backup OpenSUse Test due to error in VMDK backup!

2010-10-15 15:59:27 -- info: ###### Final status: ERROR: No VMs backed up! ######

2010-10-15 15:59:27 -- info: ============================== ghettoVCB LOG END ==================

can you help me to clarify what is the main problem of this?

thank you

lamw · ‎10-15-2010

One issue I see is that VM_BACKUP_VOLUME variable needs to be the full VMFS path (e.g. /vmfs/volumes/datastore/mybackup_folder, you do not have it setup up correctly which is probably why it's failing. You probably would get more details by changing the debug level. Try updating that and see if you have better success and make sure you go through the documentation and look at the example configs to help

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware VCP3,4

VMware VCAP-DCA4

VMware scripts and resources at:

VMware Code Central - Scripts/Sample code for Developers and Administrators