Unable to upgrade embedded ESXi 3 to U2 on IBM BladeCenter HS21 XM

[SunF1re]

Hi all,

we just purchased an IBM BladeCenter and for a start loaded with 3x HS21 XM Blades with the embedded ESX3i. In general everything seems to work fine but we are unable to upgrade to the latest U2 patch. The Blades are shipped with embedded ESXi 3.5 build 71173.

When we run the VMware Infrastructure Update client does show the updates and when we choose to install the firmware update progresses to about 98% and then displays and error message "Failed to install Firmware update.: Unable to Install". I attached a sceenshot of what is showing in the ESX server event log after the failed upgrade. We also tried the vihostupdate from the remote CLI package with the very same result.

Did anybody run into similiar issues? This for all 3 Blades so I guess this may be a more general problem with the embedded ESX3i running off a USB Flash drive on the IBM's?

Any help is appreciated.

Thanks, Thorsten

[Dave_Mishchenko]

I had a few hiccups with early patches (on the installable version) and found it helpful to use remoteinstall.exe. You can find this if you extract the patch file for Update 2 and it's easy to run - http://www.vm-help.com/esx/esx3i/Patch_3i_without_VC.html.

If you search the forum for HP and patch/ update, you'll find some others that have had problems patching the embedded version. I'd definitely log a support call with VMware / IBM as in some cases with the HP embedded update it was necesarry to use the recovery CD after a patch hosed the USB drive.

[SunF1re]

I already tried all that and the image we are running is the one comming from the recovery CD. No luck no matter what we do. And as for opening up a support call: IBM tells us that we need to go to VMware and VMware sends up back to IBM... its like a bad joke! So far the support user experience could hardly be any worse.

[alex555550]

Hy,

have you tried to download the new ISO without OEM Custom.? And Use this Version? Maybee IBM changed the partitioen Layout and so the USB ESXI get an problem.

[SunF1re]

Where can I download an ISO for the embedded version? I only see a download for the installable version and if I try to install this it won't find a hard drive to install to because the server is diskless - only the integrated USB/Flash drive is there to boot the embedded ESXi server. The ESXi server once booted will fetch the VM's off a SAN via iSCSI.

[alex555550 · Accepted solution]

Hy,

you`re right. To do this, you must boot from an Linux cd and flash the dd image to the USB Stick.The other way is to open the blade, unplug the usb key, and reload the usb key with a clean Image. PDF

Have you tried to log in to the ESXI to the shell, and clean the /tmp directory.

Edit1:

Can you post the esxupdate.log , please.

updatedb

find / |grep esxupdate.log

Edit2:

Have you already updated the Bios, bmc, ... to the latest level?

Message was edited by: alex555550

[Dave_Mishchenko]

IBM should be able to provide you with a recovery CD. As you've noted the Installable version will not install to CD and you would have to use another method to extract the image to the USB drive.

Using the regular image will likely mean that you won't have the same CIM providers so what you see for the health status or in a 3rd party management tool may be reduced.

[dilpreet]

I don't know of any specific issue with the IBM image per se. I will have the right folks have a look.

[SunF1re]

Ok, so here is what I did. At first I booted into some linux livecd and run fsck.vfat on all FAT partitions that the recovery CD created on the USB disk and there where errors on one of them (mind that the recovery CD's we used are already updated versions we received from IBM support because the original ones that came with the blades where even more broken!). After rebooting into ESX again the update worked BUT after another reboot it would not show the new build number of U2.

So after so many hours of fighting with the IBM image I got tired of it and decided to try alex's suggestion of dd'ing the extracted ESXi U2 installable image onto the USB drive and this seems to work perfectly. And since the HS21 is on the supported hardware list I am certain that all required drivers are present.

Thanks everybody for helping!

[dilpreet]

The issue with your upgrade now is that your missing the IBM

customizations.

Sent from my iPhone

On Aug 3, 2008, at 1:36 AM, "SunF1re" <communities-emailer@vmware.com

[alex555550]

How much custom. is done from IBM? I didn´t know theire Image. At March this year I tested ESXI on an x3650 (no embeded) an he displayed everything. I was supressed that he can display the power used by the psus for example, without on driver.

[Dave_Mishchenko]

The customization provided by hardware venders will involve CIM providers. Driver wise you'll have the same package so while installable will run on the HS21, if you look at healt status for the server you may notice that you only have the basics for CPU and memory. If that's not an issue for you then what you have will be fine. In terms of embedded vs installable, there's bits of difference between the two, so a vanilla copy of embedded from vmware will be the same as getting installable from vmware.

[dilpreet]

Currently, it isn't much (for IBM) but there are important

differences between the OEM image and the VMware image on the iso.

You will also probably run into support issues with IBM if you

continue to use the incorrect image. I would recommend tracking back

to IBM support and getting the corrected recovery CD so that you can

have an image which is supported and is upgradable in the future.

Thank you for your patience. Please track back to me regarding

support issues if you run into any.

[shormell]

I am having the exact same issue. After a day with IBM, they found this:

http://www-304.ibm.com/systems/support/supportsite.wss/docdisplay?brandind=5000008&lndocid=MIGR-5077...

They sent me an updated cd (no download option), and I reinstalled the blade (losing all my settings of course) and I am still unable to patch the blade.

The user experience is TERRIBLE for this. VMWare sent me to IBM, IBM tried to send me back to VMWare. I have two new and fast blades I cannot use.

I'm hoping that someone has figured out a way to get support from someone for ESXi on IBM blades.

[dilpreet]

Sorry for the run around but the recovery CD needs to be given to you from IBM since they have the "customizations" embedded in their image. Here is another issue from IBM which has the same sort of resolution:

https://www-304.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5077383&brandind=500...

Track back with them if you are still having issues with the image.

[joncellini]

What build did your embedded ESXi origionally ship with? We have a bunch of HP OEM images that shipped with U1 - after patching them to U2 we have run into several issues (inability to reconnect to VC, partition errors, a protection mode loop making the host unmanagable) that caused them to be very unstable. HP provided us a recovery CD that had a U2 base image and after re-flashing they have been much more stable.

[rpm1]

I am also having the exact same issue. have have 2 new IBM Blade HS21 XM machines shipped with ESXi 3.5.0 71173

we ordered the new recovery CD as per IBM article about recovery CD defective.

still no difference... my ESXUPDATE.log has the following

-

---

Verifying files...

Installing packages...

/altbootbank/cim.tgz failed on MD5 digest check;

Expected :961b7c6b1882d49df494da6bb4e62e50; got 10073f8af4cba5d7a8fa04c111399c62

Install script returned with status 1

Install helper script returned with error 256

-

---

[benny_hauk]

Just obeserved the same issue. First we observed all HS21-XM blades came without the hypervisor on the flash. Then once we used the Recovery Tools CD to put it on there we observed that when it came time to install the update (in our case it was Update 2 we were installing) that it failed every time to update.

A call to IBM and a replacement CD is on its way. I just told them we are observing the issue in Support Document MIGR-5077218 and need the replace CD the document refers to. From what I'm reading here I'm not encouraged that this is going to fix anything, but I suppose I'll have to try this before either IBM or VMware will take any further steps. I'll post an update once things get better one way or another.

As far as this finger-pointing thing goes, I can't imagine anything killing this embedded business faster than VMware and IBM support folding up their arms and refusing to take responsibility. If I call VMware (which is who I'll probably call next if their VMware Infrastructure Update utility continues to fail to update their code after using the new Recovery CD) and they try to tell me to call IBM, I won't be amused. IBM is their partner (as is Dell and any other ESXi embedded players) and as such I expect them to partner together and fix the problem ("call me when you two figure it out" sort of thing). I'm not paying support to both companies in order to spend time being the reconcilitory agent between their 2 support teams. If VMware can't properly cooperate with their partners then the whole idea of embedded hypervisors should be flushed.

I would encourage anyone who contacts VMware support to not allow them to close the case by saying it's IBM's issue, not theirs. I appreciate your interaction here, dilpreet. I totally agree the fixed CD needs to come from IBM if the "vanilla" install from VMware works. However, when I call VMware I need them to walk me through how to build that vanilla install (posts above expose one method we can use) and if that vanilla install successfully updates the version to Update 2 then I think it's valid to say the issue is with IBM and their CD.

Here's my final question: IBM is shipping me a replacement CD. I'm thinking this is simply the version of the Recovery Tools CD that has ESXi 3.5 Update 2 on it. If that's the case then the first thing I'll want to do is run the Update utility on it right? (to resolve the August 12 bug)

[Dave_Mishchenko]

Here's my final question: IBM is shipping me a replacement CD. I'm thinking this is simply the version of the Recovery Tools CD that has ESXi 3.5 Update 2 on it. If that's the case then the first thing I'll want to do is run the Update utility on it right? (to resolve the August 12 bug)

It may not even be Update 2, but if it is then you'll definitely want to install the patch.

[dilpreet]

All I can say as of now is that we are working on figuring this issue

and others like it out. Please stay tuned.