P2V RedHat Linux failed

[thesimpsons]

Hello Folks,

I recently installed VMWare Converter Standalone version 4. I've tried several times to convert RedHat Linux (version 4) physical box to VM but did not get any luck. The conversion keeps failing when it tries to clone /var directory. Note that I can convert Linux machine with Suse Enterprise version 10 without any issue. Do you guy have any ideas about this?

Toan

[vmweathers]

Try this: http://kb.vmware.com/kb/1008303

I would suspect /var/log/lastlog is the culprit.

(If your question has been resolved please mark the answers as "Helpful" or "Correct".)

[thesimpsons]

Hello,

I've tried that option before I posted the question on this forum. Even with that option sets to true, still the same problem. For some reasons, it cannot clone the /var directory. As I stated in the previous thread, I can P2V Suse Enterprise Linux without anny issue. I don't know why it doesn't work with RedHat. Note that I've tried several different boxes, the result is the same. Always failed when try to clone the /var directory. Thanks for your input though.

Toan

[vmweathers]

Can you please post the logs? Select the task in the main window and click the "Export logs..." link for that task.

Also, did you kill the agent process? The setting will not be respected unless the agent initializes again, which requires killing it.

(If your question has been resolved please mark the answers as "Helpful" or "Correct".)

[thesimpsons]

Server log or agen log?

[vmweathers]

Neither is individually sufficient. I need the log bundle that is create by clicking that link. Please see the attached picture.

(If your question has been resolved please mark the answers as "Helpful" or "Correct".)

[thesimpsons]

Sorry, I can't post the whole log since it contains some information on the servers...I can copy some text if that helps...

[vmweathers]

No problem, I'll "private mail" you with an email address that you can send the logs to.

(If your question has been resolved please mark the answers as "Helpful" or "Correct".)

[vmweathers]

Ok, this is odd, there is no interesting output at all about why cloning of /var fails. For completeness, below are the relevant logs (with IPs/names masked out).

If you have a support contract with VMware, please create an SR for this issue so we can track this issue and let me know the SR number.

2009-03-13 18:47:26.568 'task-1' 4028627856 info Copying files from <IP>:/var to /mnt/p2v-src-root/var

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose LD_LIBRARY_PATH=/usr/lib/vmware-converter

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose PATH=/bin:/sbin:/usr/bin:/usr/sbin

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose Invoking /usr/lib/vmware-converter/copyFileSystem.sh with the following arguments:

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args[0]: --sshClient

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args[1]: /usr/lib/vmware-converter/ssh

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args[2]: --user

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args[3]: <username>

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args[4]: --host

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args[5]: <IP>

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args[6]: --port

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args[7]: 22

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args[8]: --sourceMountPoint

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args[9]: /var

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args\[10]: --targetMountPoint

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args\[11]: /mnt/p2v-src-root/var

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args\[12]: --sshConfigFile

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args\[13]: /usr/lib/vmware-converter/ssh.conf

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args\[14]: --sourceTarOption

2009-03-13 18:47:26.569 'task-1' 4028627856 verbose args\[15]: --sparse

2009-03-13 18:48:12.581 'Helper' 4037016464 verbose Converter Agent Helper ServiceInstance RetrieveContent

2009-03-13 18:53:12.722 'Helper' 4062182288 verbose Converter Agent Helper ServiceInstance RetrieveContent

2009-03-13 18:58:12.808 'Helper' 4078959504 verbose Converter Agent Helper ServiceInstance RetrieveContent

2009-03-13 19:03:12.857 'Helper' 4053793680 verbose Converter Agent Helper ServiceInstance RetrieveContent

2009-03-13 19:08:12.863 'Helper' 4078959504 verbose Converter Agent Helper ServiceInstance RetrieveContent

2009-03-13 19:13:12.916 'Helper' 4099462848 verbose Converter Agent Helper ServiceInstance RetrieveContent

2009-03-13 19:18:12.994 'Helper' 4045405072 verbose Converter Agent Helper ServiceInstance RetrieveContent

2009-03-13 19:23:12.975 'Helper' 4090932112 verbose Converter Agent Helper ServiceInstance RetrieveContent

2009-03-13 19:28:13.012 'Helper' 4037016464 verbose Converter Agent Helper ServiceInstance RetrieveContent

2009-03-13 19:33:13.062 'Helper' 4078959504 verbose Converter Agent Helper ServiceInstance RetrieveContent

2009-03-13 19:38:13.082 'Helper' 4062182288 verbose Converter Agent Helper ServiceInstance RetrieveContent

2009-03-13 19:43:13.089 'Helper' 4090932112 verbose Converter Agent Helper ServiceInstance RetrieveContent

2009-03-13 19:47:27.915 'task-1' 4028627856 error /usr/lib/vmware-converter/copyFileSystem.sh did not finish producing output after 3600 seconds

2009-03-13 19:47:27.916 'task-1' 4028627856 info Exception caught during volume copying: /usr/lib/vmware-converter/copyFileSystem.sh did not finish producing output after 3600 seconds

2009-03-13 19:47:27.916 'task-1' 4028627856 info Unmounting partition /dev/sdb7

2009-03-13 19:47:27.916 'task-1' 4028627856 verbose LD_LIBRARY_PATH=/usr/lib/vmware-converter

2009-03-13 19:47:27.916 'task-1' 4028627856 verbose PATH=/bin:/sbin:/usr/bin:/usr/sbin

2009-03-13 19:47:27.916 'task-1' 4028627856 verbose Invoking /bin/umount with the following arguments:

2009-03-13 19:47:27.916 'task-1' 4028627856 verbose args[0]: /dev/sdb7

2009-03-13 19:47:29.930 'task-1' 4028627856 verbose Command return code: 1; result string: umount: cannot umount /mnt/p2v-src-root/var: Device or resource busy

2009-03-13 19:47:29.930 'task-1' 4028627856 error '/bin/umount /dev/sdb7' failed. Return code: 1; message:

umount: cannot umount /mnt/p2v-src-root/var: Device or resource busy

(If your question has been resolved please mark the answers as "Helpful" or "Correct".)

[vmweathers]

Ok, I'm going to give you some debugging steps to perform to try to isolate this problem, and it's going to be involved, but please bear with me:

First let's make sure there's nothing obvious being noted in /var/log/ on the source. Please look at these items on the source and see if there is any indication why the tar might have failed:

• /var/log/messages

• /var/log/warn

• the output of the dmesg command

Next I'd like you to do a test of tarring /var to another system just like we do in the helper VM. From this other system that is simulating the actions of the helper VM do this:

• mkdir -p /tmp/p2v-src-root/var (assuming you have enough space in /tmp for the contents of /var from the source)

• ssh root@<IP> "tar --one-file-system --sparse -C /var -cf - ." | tar -v -C /tmp/p2v-src-root/var -xf -

If this fails then please try to determine which files did not get cloned by doing a "binary search" of the /var tree on the source and /tmp/p2v-src-root on the destination. "du -sh /var/foo" will tell you the size of each directory, so you can compare directory sizes until you find directories that differ in size. That's what I mean by "binary search". Hopefully the verbose output of the tar command will assist with this, since it should tell you the names of the last files that were copied. The goal of this exercise is to determine if the tar copy fails due to some specific file on the source.

If it instead succeeds, then we'll you to login to the helperVM to continue debugging. This requires you to follow this kb article: http://kb.vmware.com/kb/1008209

Once you have proceeded to this step please update the forum with your status. Also if you need to send me any private data then please use the email address I already provided you with.

(If your question has been resolved please mark the answers as "Helpful" or "Correct".)

[thesimpsons]

We looked at the physical box that we want to convert during the time it tries to clone /var, the process shows that the tar command is running with 100% CPU...is this the reason why it casued the failure of the conversion process?

[vmweathers]

That's very interesting. Given that the whole /var is less than 1GB in size, it really shouldn't take greater than 30 minutes to clone it. Also our code that times the clone out after 30 minutes only kicks in when there are 0 blocks written into the destination filesystem for a full 30 minutes. That's what implies it is an issue with the source. And this is what makes me suspect it is some sort of magical file(s) that is causing this.

Anything interesting in /var/log as I was asking about?

(If your question has been resolved please mark the answers as "Helpful" or "Correct".)

[faugusto]

Hi All,

I'm having the same problem, when I execute the tar command it stops trying to copy the /var/log directory. Anyone know what is happening ?

Thanks

[vmweathers]

1. Have you enabled the sparse file flag in converter-agent.xml, per ?

If so, did you kill the agent process?

2. Have you tried following the debugging steps I outlined in http://communities.vmware.com/message/1198729#1198729

3. Finally if you don't think you can follow those instructions to try to isolate the culpable file(s), then you might consider just backing up the /var/log directory somewhere, and then deleting everything in it, since they are afterall just logs.

(If your question has been resolved please mark the answers as "Helpful" or "Correct".)

[Brucky]

I don't know if this helps but I have had exactly the same issue with a RedHat AS 4 server hanging on the /var directory when being P2Ved.

It appears that there is a known issue with the version of tar on this release of AS4 which does not recognise or deal with sparse files. The version is 1.14 and there is an update at:

I have yet to install the rpm but on debugging using the commands you outlined above it appears that the lastlog sparse file is the issue even though the KeepSparse option in the Converter Agent XML file is set to true.

Since the P2V Helper VM issues a remote call to the tar command on the source server it is reliant on that version handling sparse files correctly. Until it does the P2V will exhibit the same issues.

I will hopefully let you know if the upgrade of tar helps but I have to be careful that its rpm does not alter any other c libraries which my production environment relies upon.

I hope this helps?

- Brucky

[vmweathers]

Oh, very interesting, thanks for posting this update. The lastlog file really isn't that interesting, could you try removing it then redoing the P2V? (It will be recreated... it only tracks the logins different UIDs have made to the system).

(If your question has been resolved please mark the answers as "Helpful" or "Correct".)

[Brucky]

Unfortunately I have not got the bottle to delete this file as it is the file which logs user logins and as such should be held open by the login process. Plus recreating a sparse file is not the easiest of things.

If anyone else would like to try on their production server and let me know I would be very grateful

[faugusto]

Hi people,

After many days we found the reason of this error. What is happening was two things:

1- Error on partition (File System - LVM) on /var

2- The file lastlog was showing with size of 1.2 TB

After recover the partition and remove this file we can convert the machine.

[]'s

[shanedawg]

I had the same issue with the /var/log/lastlog being labeled as 1.2TB when I ran "ls -lh /var/log"

I removed the file using rm and ran touch to recreate it. I was able to see new logins going into the new file so all is working well and the vmware conversion completed successfully. Thanks for posting the info on this one!!

edit:

This was using 4.0.1 linux converter standalone and the source was a fedora 4 box with tar version 1.15.1

[Gilbert2797]

I know this is an old post but I'm running into the same trouble trying to convert a Red Hat Linux 4 ES.

When you say "recover" partition and remove this file.  What does this exactly mean?

What commands did you use?