VMware Cloud Community
ChrisInTexas
Contributor
Contributor

Free ESXi Backup Solution for Windows

I have spent the last few days trying to find a free backup solution to the newly free ESXi for windows only enviroments (in particular Windows XP). The solution for me was the following:

1. Installing Windows Services for UNIX (WSFU)

2. Copying the ESXi Server password and group files to Windows

3. Configuring WSFU for accepting ESX Server connections

4. Sharing the Windows folder for NFS compatibility

5. Configuring the ESXi Server to mount the Window NFS Share as Datastore.

6. Setup Backup Script

Attached is the complete steps.

I take NO credit for any of this. This is just a complation of others work formated to suit my needs and felt others could benift from it as I have.

by Jason Mattox from Vizioncore (direct copy of his work, I just added more information to make it work in Windows XP)

(NFS Server port information)

by robink (The backup script)

(ssh on ESXi)

Tags (3)
0 Kudos
522 Replies
Intrepidity
Contributor
Contributor

Thanks for the speedy response. I now got a good understanding of how I can set up a solution and the scheduling using a nas device, a windows server or a linux server.

However, I have yet another lame newbie question. Well, sorry, but here it comes.

Setup: 2 HP servers running embedded ESXi and hosting the VMs locally. Both servers have enough capacity on the datastore to host the VMs from the other server.

Question: Would it be possible to add the datastore of the other server in both servers and kind of cross-backup them? VMs from server 1 -> server 2 and VMs from server 2 -> server 1. This would have been done daily scheduled by crontab if possible.

Pros: If a server go down I can power on the VM straight away on the other physical server without having to copy it across the network first (slow due to big VMs). Easy testing of backups since they´re allready on a datastore where they can be booted and tested directly (without network connection for the tested VM).

This would be sufficient backup of the VMs since the most important data form within the VMs are backuped to a tape device anyhow. I am pretty much not allowed ANY investments right now due to the economic state of the company, so network storage or a backup server is kind of hard to get funding for right now. My second option is using an old server with server 2003 OS and USB attached drives. The scheduling through windows seems like the easiest solution for me anyhow. Might use both solutions with a backup weekly to the old server just in case. Transfer to/from the old server would probably be too slow to use it for daily backups.

Unless I get a negative response I will test this setup. I suppose it depends if it´s possible to add the datastore of the opposite server or not. I´ll at least give it a shot.

Thanks for taking the time to write such a money-saving script for the rest of us Smiley Happy

0 Kudos
lamw
Community Manager
Community Manager

You might want to check out Xtravirt Virtual SAN Appliance, it does exactly what you've stated, unused local storage.

http://engineering.xtravirt.com/products/phd-technologies/xtravirt-virtual-san.html

They're basically presenting the local storage out as an iSCSI Target to other ESX Server(s). I have not personally used their free product but I've had a quick glance when they had released this about a month or two ago. Good luck with that and post back if it works out for you.

Glad the script is working great for your organization.

0 Kudos
kpc
Contributor
Contributor

Hi Bolgard.

The name is the thread is slightly misleading as the script has evolved over the past few months to one that runs directly on the ESXi server itself. A new thread should read Free Linux backup script If you still want to use Windows then the info is there (somewhere). Personally I'd recommend the LInux/ESXi route as it's far more more contained and versatile. You need to first enable SSH on the ESXi server (plenty of guides around), then SCP iawm's script to it and setup your NFS share. It all depends on what technologies you are using, e.g. Windows, Linux and how good/bad your linux is.

I use only Linux where I work, so have a 1TB NFS share that I use for my VM backups, I mount the NFS through the VI client, run the script - nice and simple... Hope that helped a little.....

I think I can manage using Linux. But you answered my questions spot on, what I didn't know was where to run the script and how. Thanks!

EDIT: If I have a virtual disk of size 80 GB, but using only 6 GB, will the backup be 80 GB or 6 GB?

If you've only used 6GB of an 80GB VM and if you've set your backup to export the VM as 'thin' disks (e.g. -2 gb sparse) it will only copy that 6GB to your backup location, much quicker backups, but they do need to be converted back to 'fat' disks if you need to restore etc. Iams script has a setting that you can set for this. Hope this clears this up Smiley Happy

0 Kudos
blueivy
Contributor
Contributor

I eventually completed a backup using OpenFiler - 22 hours for approximately 500GB of VM's.

My preferred distro is CentOS so I've just wiped OpenFiler and used that. My first problem appeared to be that I couldn't mount the XFS partition created by OpenFiler. I eventually formatted the partition using XFS again and it worked fine.

I also used the async option (thanks kpc) and I'm now getting a huge throughout (maximum 13603kbps according to the VI clientbut it's still pretty low. I was hoping for a lot more in a Gigabit network!

A question about async - aync is obviously faster because of the reduced overhead and can cause corruption. Just a thought but would an MD5 hash check at the end (although it's slow) not confirm the file was fine?

0 Kudos
kpc
Contributor
Contributor

I eventually completed a backup using OpenFiler - 22 hours for approximately 500GB of VM's.

My preferred distro is CentOS so I've just wiped OpenFiler and used that. My first problem appeared to be that I couldn't mount the XFS partition created by OpenFiler. I eventually formatted the partition using XFS again and it worked fine.

I also used the async option (thanks kpc) and I'm now getting a huge throughout (maximum 13603kbps according to the VI clientbut it's still pretty low. I was hoping for a lot more in a Gigabit network!

A question about async - aync is obviously faster because of the reduced overhead and can cause corruption. Just a thought but would an MD5 hash check at the end (although it's slow) not confirm the file was fine?

Hi blueivy

I'm guessing sync does a few extra things like resend packets, manage failures etc - just a guess. I've been wanted to read up on it for a while just never enough time Smiley Happy All I know is that with async set my backups are about 5 times quicker. If you need some timings to compare post some of yours and i'll post some back.

Cheers

0 Kudos
polysulfide
Expert
Expert

Doesn't VCB in LAN mode work with ESXi local storage?

If it was useful, give me credit

Jason White - VCP

0 Kudos
alusrc
Contributor
Contributor

lamw,

Does ESXi have enough capability to send out an e-mail report of the backup job through some modification of your script? Do you think it would be a worthwhile change? I was thinking that if I could just have the stuff that's normally echoed out to the console also go to an e-mail I could check at a glance to see if anything didn't backup correctly. However, I don't have the bash script skills to even begin to tackle it.

0 Kudos
lamw
Community Manager
Community Manager

I highly doubt ESXi would include some sort of mail client, you would need to investigate if it does, I would assume it won't to keep it thin.

In terms of some email report, you could easily add an extra echo line and append to a file and then relying on SSH keys, you could retrieve that file and send it out from your remote host. That might be a better approach instead of trying to do everything within the ESXi console.

Update:

The current busybox distro has "ssmtp is an extremely simple Mail Transfer Agent" but that's not part of "BusyBox v1.2.1 (2008.06.26-17:58+0000)" which is what ESXi is using.

0 Kudos
greenhorn1
Contributor
Contributor

I am trying to get my backups running on the new ESXI systems. I have setup a test environment I have a snap server for storage. I have enabled SSH on the ESXI box. I have installed Putty and WenSCP. I have copied the passwd file and group file from the ESXI server to my backup server. This is where I have WSFU installed and running. I am stuck on step 4 of this PDF. I get two different errors. One is Mount V3 is not supported via TCP and Unable to mount file system.

Once I have my test bed setup I have to bring this into production where I will be running Windows Storage server 2003 R2 with 5.4 TB usable. I have at this time 4 ESXI servers which I plan on using some of this 5.4 TB to do either snapshots or I will bring the VM down and copy to the NAS then to tape. Please help.

0 Kudos
mylesw
Contributor
Contributor

I've been following this thread with a lot of interest. We have a test installation of VMWare ESXi 3.5 on a server in my home office. I am installing a single VMWare ESXi 3.5 server on a colocated box at a data center this weekend. I have a VM that has been created on my home server with all the base settings that I want to use for future VMs on production, but I don't have the budget to afford the full VMWare VMotion solution for this. My hope is that I can backup my VM at home and transport it to my colocated server.

I'm using VMExplorer to do the backup which has done a good job. However the problem, which seems to be in common with others here, is the time that it takes to backup a 20GB VM over my network. I need something that is much faster.

I'd like to put an external USB hard drive on my home server and backup directly to it. But I can't seem to see any device in Linux on my VMWare server box that corresponds to the external hard drive (ie /dev/sda etc.).

How have others connected an external USB drive directly to the VMWare server so that backups can be made to it and avoid the network altogether?

Myles

0 Kudos
neffernin
Contributor
Contributor

The script works great! I setup a LUN and tested it out. The issue I'm having is that if you remove the storage from ESXi it kills the file structure on the LUN; therfore I can't unmap it then remap it on another server. When you remove the datastore from VMWare it says "Are you sure you want to remove the selected datastores? This operation will permanently delete all of the files on the datastore". Is there a way to remove the iSCSI mapping without this happening from ESXi?

0 Kudos
neffernin
Contributor
Contributor

The script works great! I setup a LUN and tested it out. The issue I'm having is that if you remove the storage from ESXi it kills the file structure on the LUN; therfore I can't unmap it then remap it on another server. When you remove the datastore from VMWare it says "Are you sure you want to remove the selected datastores? This operation will permanently delete all of the files on the datastore". Is there a way to remove the iSCSI mapping without this happening from ESXi?

0 Kudos
neffernin
Contributor
Contributor

Is there a way to remove a datastore from one ESXi server and add it to another? When I try to remove the datastore it deletes all of the data on it. Doing this w/ a LUN via iSCSI

0 Kudos
ppcfox
Contributor
Contributor

Hi lamw,

thank you for your script.

I use the script with nonpersitent nfs storage and i think there is a little bug in your script.

if ; then

$ hostsvc/summary/fsvolume | awk '{print $1'} | grep "\"$\"" > /dev/null 2>&1 if ; then #1 = readonly #0 = readwrite $ hostsvc/datastore/nas_create "$" "$" "$" 0

fi

fi

the storage was never mounted until i disable the second if-statment.

Sorry for my worse english

ppcfox

0 Kudos
lamw
Community Manager
Community Manager

ppcfox,

It should actually be checking to make sure the exit status "$?" is not equal to 0 which means it's already mounted. I'll update the code shortly, but yes if you just change that from "1" to "0" then, that should solve it.

Let me know if you have any other questions

Update: I've updated the script and re-uploaded to the DOC's page.

0 Kudos
bigs77
Contributor
Contributor

Just thinking alound with regards to the emailing of reports....

I'm by no means a scripting guru and have no idea of how to do this... but

Firstly so long as the report is a text file, I suggest reading this into a variable in a script. From there perhaps in the config.pl script have the user specify the IP address of an internal mail server, then script a telnetting to the internal smtp server (ensuring relaing is allowed for the IP address that your running the script & backup from....) then in this script then send the variable that read the report in the body of the message... I have dabbled with smtp commands but don't know of any way to add an attachment...

Anyway I may be dribbling somthing out of the corner of my mouth as this may not be at all possible but hopefully it has given someone some ideas as to how to do this...

0 Kudos
kpc
Contributor
Contributor

Just thinking alound with regards to the emailing of reports....

I'm by no means a scripting guru and have no idea of how to do this... but

Firstly so long as the report is a text file, I suggest reading this into a variable in a script. From there perhaps in the config.pl script have the user specify the IP address of an internal mail server, then script a telnetting to the internal smtp server (ensuring relaing is allowed for the IP address that your running the script & backup from....) then in this script then send the variable that read the report in the body of the message... I have dabbled with smtp commands but don't know of any way to add an attachment...

Anyway I may be dribbling somthing out of the corner of my mouth as this may not be at all possible but hopefully it has given someone some ideas as to how to do this...

Not sure I quite understand your logic, but one way to do this, which I think lamw touched upon, was to kick off all ESXi backups from a central backup server that would SSH into each ESXi server and run the backup. You would output anything you want to a log file and either SSH the file back to your central backup server or if your using NFS just mount the same share and CP the file. If you want an idea of how mail can work check out one of my ealier scripts on the forums.

An added advantage of running from a central server and from one script is that your backups would run one after another, thereby reducing disk contention on the NFS share. Very handy if you have multiple ESXi servers.

0 Kudos
DSTAVERT
Immortal
Immortal

I am reluctant to open ssh on a production server even though I have the management interface / NFS shares protected by a physical firewall and certainly understand any organizations decision to disallow it. I am however, running the script from a central server in a test environment using ssh so I may change my stripes.

Assuming the script copies off to an NFS share, it is fairly easy to echo the output of the script or parts of the script to a file in a "drop" directory. Script a scan of the directory to forward the file as email. Even if the script copies off to some other storage device it is pretty easy to create and add an NFS share specifically to hold the echoed output.

-- David -- VMware Communities Moderator
0 Kudos
polysulfide
Expert
Expert

SSH is pretty secure just the way it is,especially on a secure network.

If you want to go for just a little extra, you can:

Configure SSH to listen on an obscure port

Configure key-based authentication

Configure the ESX firewall to only allow incoming SSH form your "management station"






If it was useful, give me credit

Jason White - VCP

0 Kudos
alusrc
Contributor
Contributor

lamw,

How do we modify your script to just backup every VM on the host?

0 Kudos