ghettoVCB.sh - Free alternative for backing up VM's for ESX(i) 3.5, 4.x, 5.x, 6.x & 7.x

VMware Code Central - Scripts/Sample code for Developers and Administrators

( http://communities.vmware.com/message/1122064 / https://bugs.maemo.org/show_bug.cgi?id=3033 )

If you find this information useful, please award points for "correct" or "helpful".

devzero · ‎02-18-2010

>22.09.2009 05:59 Antwort chayawatv sagt:

>I try to use this scipt to do backup with compression enabled.

>Howerver, I cannot uncompress "filename".gz as it show "tar: Invalid tar magic".

>Currently, I use ESX 3.5i U4 free edition and backup it to local drive of ESX server itself.

>Any thought about the issue?

>Thank you

busybox v1.2.1 has a bug and this is the whole problem. so ghettovcb on esx3i with compression=on is BROKEN !!!

solution:

add busybox from esx4i to esx3i box and change ghettovcb appropriately to use that one for tar and gzip.

alternative: don`t use compression

i wonder that this thread has more than 700 comments but this is found that late - nobody using compression or test restore of their (probably corrupted) backups ?

vlho · ‎02-18-2010

Hi lamw,

I have problem with backup of vmdk stored across multiple datastores when I use individual configuration.

My virtual machine akord-pro have this files:

- akord-pro.vmx on datastore raid1 (main configuration)

- akord-pro.vmdk on datastore raid1

- akord-pro_1.vmdk on datastore raid1

- akord-pro_2.vmdk on datastore raid5

- akord-pro_3.vmdk on datastore next1

- akord-pro_4.vmdk on datastore next1

I need backup only first three vmdk, i.e.: akord-pro, akord-pro_1 and akord-pro_2

However script backup only vmdks from datastore raid1, no vmdk from next datastore (akord-pro_2)

I think that it is because of script searches akord-pro_2.vmdk including full path but in configuration file is akord-pro_2.vmdk defined without full path.

Am I right?

Output from running ghettoVCB script:

...

2010-02-07 12:06:14 -- debug: HOST BUILD: VMware ESX 4.0.0 build-208167

2010-02-07 12:06:14 -- debug: HOSTNAME: vma.domain.cz

2010-02-07 12:06:15 -- info: CONFIG - USING CONFIGURATION FILE = ./akord-pro

2010-02-07 12:06:15 -- info: CONFIG - VM_BACKUP_VOLUME = /vmfs/volumes/nfs172v1/_backup

2010-02-07 12:06:15 -- info: CONFIG - VM_BACKUP_ROTATION_COUNT = 3

2010-02-07 12:06:15 -- info: CONFIG - DISK_BACKUP_FORMAT = thin

2010-02-07 12:06:15 -- info: CONFIG - ADAPTER_FORMAT = lsilogic

2010-02-07 12:06:15 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = 0

2010-02-07 12:06:15 -- info: CONFIG - ENABLE_HARD_POWER_OFF = 0

2010-02-07 12:06:15 -- info: CONFIG - ITER_TO_WAIT_SHUTDOWN = 4

2010-02-07 12:06:15 -- info: CONFIG - POWER_DOWN_TIMEOUT = 5

2010-02-07 12:06:15 -- info: CONFIG - SNAPSHOT_TIMEOUT = 15

2010-02-07 12:06:15 -- info: CONFIG - LOG_LEVEL = debug

2010-02-07 12:06:15 -- info: CONFIG - BACKUP_LOG_OUTPUT = stdout

2010-02-07 12:06:15 -- info: CONFIG - VM_SNAPSHOT_MEMORY = 0

2010-02-07 12:06:15 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = 0

2010-02-07 12:06:15 -- info: CONFIG - VMDK_FILES_TO_BACKUP = akord-pro.vmdk,akord-pro_1.vmdk,akord-pro_2.vmdk

2010-02-07 12:06:16 -- info: Initiate backup for akord-pro

2010-02-07 12:06:16 -- info: Creating Snapshot "ghettoVCB-snapshot-2010-02-07" for akord-pro

2010-02-07 12:06:16 -- debug: Waiting for snapshot "ghettoVCB-snapshot-2010-02-07" to be created

2010-02-07 12:06:16 -- debug: Snapshot timeout set to: 900 seconds

2010-02-07 12:06:17 -- debug: Waiting for snapshot creation to be completed - Iteration: 0 - sleeping for 60secs (Duration: 0 seconds)

2010-02-07 12:07:18 -- debug: findVMDK() - Searching for VMDK: "/vmfs/volumes/4a574526-5eb4fd94-375a-001517b9527d/akord-pro/akord-pro_3.vmdk" to backup

2010-02-07 12:07:18 -- debug: findVMDK() - Searching for VMDK: "/vmfs/volumes/4a574526-5eb4fd94-375a-001517b9527d/akord-pro/akord-pro_4.vmdk" to backup

2010-02-07 12:07:18 -- debug: findVMDK() - Searching for VMDK: "/vmfs/volumes/4a2fa88f-cc11effe-5c89-001517b9527d/akord-pro/akord-pro_2.vmdk" to backup

2010-02-07 12:07:18 -- debug: findVMDK() - Searching for VMDK: "akord-pro_1.vmdk" to backup

2010-02-07 12:07:18 -- debug: findVMDK() - Found VMDK! - "akord-pro_1.vmdk" to backup

Destination disk format: VMFS thin-provisioned

Cloning disk '/vmfs/volumes/raid1/akord-pro/akord-pro_1.vmdk'...

Clone: 100% done.

2010-02-07 12:27:03 -- debug: findVMDK() - Searching for VMDK: "akord-pro.vmdk" to backup

2010-02-07 12:27:03 -- debug: findVMDK() - Found VMDK! - "akord-pro.vmdk" to backup

Destination disk format: VMFS thin-provisioned

Cloning disk '/vmfs/volumes/raid1/akord-pro/akord-pro.vmdk'...

Clone: 100% done.

2010-02-07 12:35:53 -- info: Removing snapshot from akord-pro ...

2010-02-07 12:36:05 -- info: Backup Duration: 29.82 Minutes

2010-02-07 12:36:05 -- info: Successfully completed backup for akord-pro!

...

Thank you for reaction or tip how to do that right.

lamw · ‎02-18-2010

Hi,

Your assumptions are correct, the way the script figures out whether a VMDK is stored locally with the .vmx configuration file OR if it's hosted on another datastore is by looking at the path defined in the .vmx file. By default, if it's a relative path aka "myvmdk.vmdk", it means the VMDK is being stored locally in the datastore along with the .vmx configuration file. If it has an absolute path, then the VMDK is being stored in another datastore other than the default. You may want to double check that in fact your VM has all VMDKs currently mounted and that the .vmx file reflects these changes appropriately, in case you manually edited the file.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

soleblazer · ‎02-18-2010

Urgent question! I just noticed that on virtual machines that have more than one disk the following happens:

The script correctly backs all the vmdks up. When I register the machine directly from the backup folder the first disk is correctly on the backup device, but the second disk is being pointed to the original datastore and not the new location...

So as an example. I have a machine called fdn003. After the backup, I go to the folder fdn003-2010-02-17--1. In that folder I see the vmx file, the first vmdk file. I then see a directory composed of characters, these characters seem to spell out the code for the datastore when I run a vdf command. Why is it doing this, shouldnt all the disks be in one directory and the vmx updated? Please help.

lamw · ‎02-18-2010

soleblazer,

Ah, yes for the case of VMs with VMDKs that were dispersed across multiple datastores, due to the way VMware defaults the name, the only way to keep track of these was to create a directory structure that maps the original datastore UUID, that is the characters you're referring to. To accommodate this if you want to register and run the VM from the backup datastore, you just need to go into the VM's settings and basically locate the disks and re-point them to backup datastore since the .vmx file will contain the original configurations.

If you use the restore script, this should be handled for you automatically, by default the recommendation is to restore the backup to a different datastore and the script will handle the registration of the disks.

Hopefully this makes sense

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

soleblazer · ‎02-18-2010

William,

That makes sense. So on the vm's that have multiple drives I would need to do this:

Add the vm to the inventory

Remove the disk that is pointing to the original datastore

Add a new disk using the path that your script uses

Is that it?

lamw · ‎02-18-2010

That is correct, just do a 'remove' which is just a pointer but don't delete from disk, especially if you have the VM still running else it potentially can delete the actual disks from under the VM.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

soleblazer · ‎02-18-2010

Thanks alot. I tested it out and it works fine. I am doing san maintenance this weekend and I'm simply trying to be able to survive if the SAN should somehow get erased. I really appreciate your help and script!

lacikaaa · ‎02-19-2010

I tried removing them. If i run manually the script from shell, then it works fine, but when it is started from cron with the same arguments, configs, everything, then only one backup left / vm, and it is the second after 2 running. I havent modified the script and the parameters are the same, this is the only difference. Any idea?

In the logs it looks like everything is ok, there isnt any extra info compared to other logs which are from manual runnings.

Thank you!

buekkes · ‎02-22-2010

Considering your remarks on encryption and (unnecessary) load, I tried to figure out an alternative approach, but I didn't work out completely, yet.

On the remote box (IP replaced by 1.2.3.4) I exported

/home/export/eq6/gotthard 1.2.3.4(rw,async,all_squash,sync,insecure)

with "insecure" required on Debian Lenny. The ports of rpc.statd, lockd and rpc.mountd are statically bound to 2231, 2232 and 2233. "rpcinfo -p" confirms their ports and "exportfs -a" confirms the existance of the exported directory.

On the ESXi I establish tunneling of two ports:

dbclient -i .ssh/key_for_nfs -L 2049:1.2.3.4:2049 sleeper@1.2.3.4 sleep 10m

dbclient -i .ssh/key_for_nfs -L 2233:1.2.3.4:2233 sleeper@1.2.3.4 sleep 10m

My assumption was, that I should be able to mount the nfs-directory by means of vSphere Client now. Nonetheless, any attempt to do this (host: 127.0.0.1 or external IP of ESXi, directory: /home/export/eq6/gotthard) is in vain. After refreshing the list, the gui tells me, that the NFS server can't be found. Unfortunately, all documentation on the way of ESXi's nfs-mapping doesn't tell something on the required ports. Has anybody an idea?

lamw · ‎02-22-2010

Take a look at the vmkernel logs on why it's failing to mount the NFS server. You can also use vmkping to ensure you can ping your NFS server. The requirements for ESXi to mount an NFS datastore is pretty trivial, so long as you have a valid VMkernel interface that communicate to your NFS server, you should be able to mount it, assuming permissions on the NFS server is correct.

Can you mount this volume using another hosts that's not ESXi?

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

buekkes · ‎02-22-2010

Yes, mounting the NFS datastore directly from ESXi (without any tunnel) works fine.

The logfiles (ESXi's IP replaced by 5.6.7.8) contain the following lines:

Feb 22 19:19:18 vmkernel: 83:01:05:03.352 cpu5:9855)WARNING: NFS: 898: RPC error 13 (RPC was aborted due to timeout) trying to get port for Mount Program (100005) Version (3) Protocol (TCP) on Server (5.6.7.8)

Feb 22 19:19:18 Hostd: VmkCtl mount or ProcessNas failed: NFS Error: Unable to Mount filesystem: Unable to connect to NFS server

Feb 22 19:19:18 Hostd: AdapterServer caught exception: vim.fault.PlatformConfigFault

Feb 22 19:19:18 Hostd: Task Completed : haTask-ha-host-vim.host.DatastoreSystem.createNasDatastore-11884 Status error

Feb 22 19:19:18 Hostd: Activation : Invoke done on

Feb 22 19:19:18 Hostd: Arg spec:

Feb 22 19:19:18 Hostd: (vim.host.NasVolume.Specification) { dynamicType = , }

Feb 22 19:19:18 Hostd:

Feb 22 19:19:18 Hostd: Throw vim.fault.PlatformConfigFault

Feb 22 19:19:18 Hostd: Result:

Feb 22 19:19:18 Hostd: (vim.fault.PlatformConfigFault) { dynamicType = , faultCause = (vmodl.MethodFault) null, text = "NFS Error: Unable to Mount filesystem: Unable to connect to NFS server", msg = "", }

Right now, I haven't found any clue on "RPC error 13 (RPC was aborted due to timeout) trying to get port for Mount Program (100005) Version (3) Protocol (TCP) on Server (5.6.7.8)"

lamw · ‎02-22-2010

Yea, the tunnel probably is doing something odd to the way ESXi tries to mount up the NFS datastore using it's defaults. Remember, this is a very limited environment, you may or may not be able to get this working. You could try having a poke around /etc/services and perhaps tweak some entries on which ports NFS can be mounted on. Try at your own risk and definitely on a test system.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

buekkes · ‎02-23-2010

It works: Beside ports 2049 (nfs) and 2233 (mountd) also portmapper's 111 has to be tunneled to the ESXi. On the ESXi the host 127.0.0.1 has to be used, while on the machine offering nfs-services its external IP is required (as written above).

And now let's integrate the stuff into your script

wobbe98 · ‎02-23-2010

I’m experiencing some strange behavior running the script from cron on an ESXi4 U1 server backing up to an NFS store (CentOS 4.8).

I’m starting the script from another script file that I have scheduled in crontab.

This extra script does nothing special; it just starts the ghetto script and creates a new file at the end. This new file I use to trigger an email message from the CentOS NFS server.

Last night’s run stopped after two VM’s were backuped. On the second VM there was still a “Consolidate Helper” snapshot active. This snapshot was not shown trough the vSphere client. To remove the residual snapshot I had to create a new snapshot and after that I could delete all snapshots again. At this point the script continued its work backing up the next VM’s. After a while I got an email with the log file but looking at the log file the script had not jet finished. The last line on the backup log was Clone: 71% done.

But somehow the script had finished because the email trigger file got created.

A few minutes later I got another email message with another part of the backup log.

So the script finished again! Looks like some how two instances of the script were running at the same time. But how can that be and how would that be possible? Checking with “ps aux | grep crond” I see only on cron deamon running. And earlier manual runs of the script finished just fine.

I’ll will restart the server today and see how things workout tonight.

lamw · ‎02-23-2010

Interesting, well if you get it all working, I wouldn't mind posting a set of instructions for your configuration for others that may be interested.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

lamw · ‎02-23-2010

I would double check your kickoff script or your cron entry, the script will always have 1 instance running unless another is kicked off. You can probably easily view this by looking in your cron logs under /var/log/cron and see when ghettoVCB was kicked off and see if you some how got another instance running.

I'm considering in a future release of the script to output a lock file, but I'm still thinking about that, since there may be users wanting to have more than once instance running.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

wobbe98 · ‎02-24-2010

Another night another failure, this time if failed on the first VM and again on the second. (didn't get to restart the sever)

My kick off script is no more then two line. Not much can go worng there. I ran it tree times yesterday without any problems.

/vmfs/volumes/NLJRSR63/vcb/srv32/ghettoVCB.sh -f /vmfs/volumes/NLJRSR63/vcb/srv32/vmserver.list -l /vmfs/volumes/NLJRSR63/vcb/srv32/backup.log

touch /vmfs/volumes/NLJRSR63/vcb/srv32/backupdone

My crontab entry,

00 23 * * 0-6 /vmfs/volumes/NLJRSR63/vcb/srv32/backup.sh

btw. I don't see any cron logs under /var/log

also strange: I know the script is running but cant find it with ps.

lamw · ‎02-24-2010

Can you manually run the backup without using the cron job? Are you able to successfully and repeatably backup VMs through a manual execution? If you do run a manual or automated, can you also enable logging to debug level and save the output file to : http://pastebin.com/ and just reply back with the link. Without any logs, I can't tell exactly what's happening if the backups are even attempted or if there is something that's causing these VMs to not backup.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

sfang · ‎02-24-2010

Hi everybody,

My needed : I would like to backup all the vms one time by week.

Just one question about this script please. I've modified it and excute on my esxi 4.0 host. It working fine but it has just backup the vmx file (which is the configuration file). It's not complete for me to restaure. I have thinked that we can backup all the global vm file. It's possible please, may be to change some setting in the script but I have not found.

thanks for your reply,

/Frédéric

lamw · ‎02-24-2010

For setting up backup schedule, please take a look at the Cron FAQ section.

The script by default will backup your VM configuration file (.vmx) and all relevant and valid VMDKs (*.vmdk and *-flat.vmdk). If it's only backing up the .vmx file, then there is something wrong and you need to take a look at the logs to see what might be going on. I would also recommend enabling debug mode to get more verbosity in the logs, please take a look at FAQ #1 for any issues with the script.

Thanks

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

http://pastebin.com/rzMcfuNE

If you find this information useful, please award points for "correct" or "helpful".

wobbe98 · ‎02-25-2010

Manually the backup runs fine.

But running fron cron it look like snapshot won't be removed.

Today it failed again. I manually removed the snapshot and the script continued.

But it's now haning again on the second VM.

(I also noticed that the backup files of the first VM were stored under a previous backup folder. Is this done by design when there are more that one backups in a day?)

lacikaaa · ‎02-25-2010

Hi!

The solution was that I turned on loggin with "-l". So instead of running like this:

0 0 * * 1-5 ghettoVCB.sh -f backuplist -c config > ghettoVCB-backup-$(date +\%s).log

i did this:

0 0 * * 1-5 ghettoVCB.sh -f backuplist -c config -l ghettoVCB-backup-$(date +\%s).log > /dev/null 2>&1

wobbe98 · ‎02-25-2010

Thanks for the tip. That seems to help a lot. I've just completed a backup run from cron without a problem.

gautelund · ‎02-25-2010

I feel almost lame staking this claim, but the attached and presumably authoritative version of ghettoVCB.sh apparently has DOS line-endings. No matter where and with what http client I download. E.g. using GNU wget 1.10.2 on a Linux box:

% wget http://communities.vmware.com/servlet/JiveServlet/download/8760-49-32036/ghettoVCB.sh
--16:49:02--  http://communities.vmware.com/servlet/JiveServlet/download/8760-49-32036/ghettoVCB.sh
           => `ghettoVCB.sh'
Resolving communities.vmware.com... 195.18.221.160, 195.18.221.178
Connecting to communities.vmware.com|195.18.221.160|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 29 511 (29K) [application/x-sh]

100%[========================================================>] 29 511      102.40K/s

16:49:03 (102.18 KB/s) - `ghettoVCB.sh' saved [29511/29511]

% file ghettoVCB.sh
ghettoVCB.sh: ASCII English text, with CRLF line terminators

Should the attached file be "unix text"?

voro · ‎02-25-2010

I guess it should be "unix text"

I tried downloading with firefox and opening it with Notepad++ and it says it's ANSI with Windows line endings.

You could convert it with that text editor or in linux with the "flip" tool (apt-get install flip if debian/ubuntu)

lacikaaa · ‎02-25-2010

sed 's/.$//g' input > output

run this, and it will remove ^M from end of line so it will work.

lamw · ‎02-25-2010

There's nothing wrong with the script per-se, but there is an issue when using wget and downloading from VMTN. Take a look at FAQ #17

Basically either deal with removing the ^M characters using solutions OR use scp or WinSCP to transfer the script.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

lamw · ‎02-25-2010

Is this on NFS datastore or local VMFS volume?

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

wobbe98 · ‎02-25-2010

From local to NFS.

lamw · ‎02-25-2010

Yea, I'm betting you're seeing the same NFS issue that others are seeing as well. If you just take a look at the last 20 or so threads, you'll see some of my replies. For more details, or as much as I've been able to gather take a look at FAQ #18 and "Our NFS Configuration" in the document.

Unfortunately, at this point in time, we've not been able to identify what commonalities between all the NFS servers being used that is causing a small portion of users to have this 'connection' problem during the removal process of a large files. There is nothing wrong with the script, but it has something to do with either the NFS server or ESXi host configurations/etc.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

wobbe98 · ‎02-25-2010

Adding > /dev/null 2>&1 to the cron job seems for have fixed the problem. Thanks again lacikaaa.

gautelund · ‎02-25-2010

William, thanks for the reply. (And also thanks to others, I know how to convert "DOS format" to Unix.)

I said "no matter where and with what http client" - I tried multiple browsers (IE, FF, Opera) on a couple of OSs and the result is the same no matter: I get a file with DOS line endings, and whether I scp or WinSCP to the ESX box I still have a file with DOS line endings. No http client should affect this. Therefore I can't think see any other explanation than that the file on the server has DOS line endings. Please enlighten me if I'm wrong.

lamw · ‎02-25-2010

Not sure why it's not working for you. I can right click on the script from this VMTN document, save as and then use WinSCP to copy it to ESXi host and ensure it has executable permission and execute a backup. If this was an issue with the source of the script, there would be many many more users with this problem.

Again, if you continue to run into this problem, you can easily strip out the ^M characters and there's been a few different solutions on accomplishing this from using sed or replace in vi.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

pebr · ‎02-25-2010

I'm having the same issue with the ^M - regardless which client or OS. But I've to test it at home without using Squid Proxy (here at work) - maybe he's adding the ^M

lamw · ‎02-26-2010

Odd ... again, I've done this in many many locations and never once had an issue.

If you guys like, I can do screenshot of my exact steps and show you that it does in fact work and that no "hidden" ^M characters are embedded with the version that's hosted by VMTN documents.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

DSTAVERT · ‎02-26-2010

I don't have any issues ether. Remember you can't do ANY editing of the script in Windows. You must do the editing on the vMA appliance or on the ESXi host.

Wolfi3 · ‎02-26-2010

Wont mind a few screen shots.

Will maybe show me what i am doing wrong.

gautelund · ‎02-26-2010

No need to go to such extremes as to document your steps in detail, William. At this point this is just puzzling and funny. To convert forth and back is no problem for me, and if this issue doesn't affect (m)any others, it doesn't matter. Though it seems like some people can reproduce it, voluntarily or not.

I have tried so many browsers, and so many platforms and locations, passing through proxies or not passing through proxies. The result is consistently the same: I can e.g. right-click the attachment above in IE and save. The file size is always 29511 bytes, and if I immediately inspect it with a hex viewer, without transfering it anywhere, line breaks are 0d 0a ("DOS format"). No matter what OS, client/browser, proxy/no proxy, or ISP(!).

Hell, at this point I am willing to believe that all backbone pipes into Norway automatically converts this script. Maybe I should have myself filmed while doing it, and then send you the film and a zipped sample of the resulting script.

lamw · ‎02-26-2010

Tell you guys what, I'll contact Corey from the VMTN team who currently manages the forum and see if he can look into this and perhaps he can shed some light. I'm sure there are multiple nodes serving out this file, perhaps certain nodes is holding a corrupted version of the script or some how including the ^M character. Again for me within the US and specifically in California, I've not had any issues and I've had several people I know try the download and it works perfectly fine. No guarantee I'll get an answer from Corey, but I'll see if I can get more info on this.

In the meantime, I can put up the latest version on our site as an alternative download location and see if you guys can use wget,curl,etc to retrieve the file.

Sorry for the inconvenience, this one is definitely a funny and odd case

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

conexcol · ‎02-28-2010

I'm having exactly the same problem with tar (tar cannot store file)

VMware ESXi 4.0.0 build-219382. The datastore is a clean 1TB hard drive with VMFS 3.33, 8MB Block Size, and 930.75GB free space.

This is what happens if I do it manually:

 
ls -al vm1-2010-02-28/
drwxr-xr-x    1 root     root                980 Feb 28 10:48 .
drwxr-xr-x    1 root     root                700 Feb 28 10:56 ..
-rw-------    1 root     root        21474836480 Feb 28 10:50 vm1-flat.vmdk
-rw-------    1 root     root                543 Feb 28 10:50 vm1.vmdk
-rwxr-xr-x    1 root     root               3591 Feb 28 10:40 vm1.vmx
-rw-------    1 root     root       182536110080 Feb 28 10:48 vm1_1-flat.vmdk
-rw-------    1 root     root                547 Feb 28 10:48 vm1_1.vmdk

/vmfs/volumes/backup/vm1 # tar -cvf compressed_vm1.tgz vm1-2010-02-28/ 
vm1-2010-02-28/
vm1-2010-02-28/vm1.vmx
tar: cannot store file 'vm1-2010-02-28/vm1_1-flat.vmdk' of size 182536110080, aborting

Thanks

ippup · ‎02-28-2010

hi

did you find any way of resolving the issue.

i'm downloading the file directly from esx and i'm still getting the same error

: not found.sh: line 6:

: not found.sh: line 8:

: not found.sh: line 11:

: not found.sh: line 18:

: not found.sh: line 21:

....

any pointers ?

buekkes · ‎02-28-2010

Remove the wrong line feeds, I also suffered from them, even by directly downloading via wget. Open vi, execute the command

:%s/^VM//g

which has to be entered as

:%s/\]ctrlkeyv and ctrl-keyM\]//g

gautelund · ‎02-28-2010

See buekkes reply. And many more tips in the FAQ as well as previous comments. There are many ways to fix this, mentioned above. Or just google http://lmgtfy.com/?q=convertdosformattounix.

(Some of us are trying to pry into WHY this happens to some of us, but I sense from your post that fixing it for yourself is more within your reach...

lamw · ‎02-28-2010

@ippup @buekkes

Folks, please take a look at the last 5-6 posts regarding this question and this is also documented on the FAQ, specifically #17. I do ask that users go through the documentation before posting as much of the questions have been answered.

You can NOT download the script using wget from the ESXi host or using datastore browser to upload the script, these will indefinitely cause you to include the ^M character which will cause the script to have issues when you try to execute.

Regarding remediation of this issue, most of the suggestions provided by users work in removing this ^M character on classic ESX BUT not on ESXi due to the version of sed and vi. (e.g. removing ^M with <ctrv><ctrlm> within vi will just fail) Depending on the OS you have available, you can use tools like dos2unix or others to remove any of thee ^M characters before uploading to your ESX(i) host.

As mentioned in my comment on Feb 26, 2010 9:55 AM, I've contacted one of the guys who runs the VMTN community to see if he can shed any light on why only a small handful of users are having this problem, I've not been able to reproduce this problem across different OSes + browsers.

In the meantime for ALL users having this problem, I've uploaded the current version of the script onto our site and you CAN use wget on ESXi 4.0 (only system I've tested) to pull down the script:

wget http://engineering.ucsb.edu/~duonglt/vmware/ghettoVCB.sh

Remember to set executable permission on the script. Let me know if you guys run into any issues

Thanks for you patience

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators

If you find this information useful, please award points for "correct" or "helpful".

lamw · ‎02-28-2010

Note, if you're trying to replicate the compression functionality from the script, you need to also specify "z" option which filters the archive into gzip.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

VMware Code Central - Scripts/Sample code for Developers and Administrators