Hello,
I'm having problems with ESXi (3.5 U2 latest, both embedded and installable) on three different hosts. Hardware is HP DL380 G5. Both NICs on every server are connected to 1000FDX ports without any duplex issues. ESXi network configuration is the default: both vmnic0 and vmnic1 are used for VM Network and Management Network. Switches show no errors on the ports.
VM Network is not showing any performance problems. I'm getting steady 30-40MB/s to and from guest machines.
Accessing the management network (copying to datastore, converter access, downloading VI client etc.) is painfully slow. Ranging between 100kB/s to 3MB/s - usually around 1MB/s.. needless to say that this is very frustrating when for example converting existing virtual machines to the ESXi hosts.
Any idea where to start looking for a solution?
Well - I guess he "left us hanging" with that post!
Kevin
I've done some tests on this issue too.
My setup is simple:
One diskless ESXi 4 (but 3.5 U4 gives the same results too)
Storage on a iSCSI target (using software initiator)
Storage on a NFS server
No traffic shaping enabled
All gigabit network
I have a VM with one disk on each storage.
The VM can transfer data from one disk to the other with >70MB/s, which is expected.
BUT, a file copy from one datastore to the other via datastore browser (or cp/dd (bs=4096) from the console) is always capped very neatly at 10-12MB/s.
I'm really going crazy on this!!
Agreed. The is strong evidence that supports the theory that the mgt I/F has been capped.
I believe the mgt I/F should always be available or reserved for mgt functions - however, VMWare should allow us to create another interface path that can be 'saturated' for file copy/backup operations.
I can't begin to understand why there is such an bend in their thought process, unless this is to help support the promotion of other products that can function 'better'. Except for SAN based file operations, I'm not aware of any product that increases the effective throughput on the mgt interface.
Kevin Buchanan
Chief Information Officer
Lexington Memorial Hospital
336-238-4286
kbuchanan@lexmem.org
In my experience what I originally thought was a cap turned out to be a hardware limitation of our aging hardware. I was able to increase the speed by enabling write-back caching on the storage.
In my experience what I originally thought was a cap turned out to be a hardware limitation of our aging hardware. I was able to increase the speed by enabling write-back caching on the storage.
And this made you achieve considerably higher rate than 10MB/s?
In my experience what I originally thought was a cap turned out to be a hardware limitation of our aging hardware. I was able to increase the speed by enabling write-back caching on the storage.
And this made you achieve considerably higher rate than 10MB/s?
Agreed. The is strong evidence that supports the theory that the mgt I/F has been capped.
I believe the mgt I/F should always be available or reserved for mgt functions - however, VMWare should allow us to create another interface path that can be 'saturated' for file copy/backup operations.
I can't begin to understand why there is such an bend in their thought process, unless this is to help support the promotion of other products that can function 'better'. Except for SAN based file operations, I'm not aware of any product that increases the effective throughput on the mgt interface.
If VMWare would just join in and confirm if it is capped or not I would accept the answer either way. It's the "not knowing" that keeps me from sleeping at night.
I was fiddling around with the network settings in ESXi 4 a couple of days ago and I'm pretty sure that I once was able to copy a file at about 60MB/s, but havn't been able to repeat it so I'm starting to doubt my memory. Maybe I was just wishing very hard. Now my system has gone productive so I can't go berserk in the configs anymore
Write back cache did NOT make any significant improvement with us. I even used a HP P800 with 2Mb cache...no difference.
Kevin Buchanan
Chief Information Officer
Lexington Memorial Hospital
336-238-4286
kbuchanan@lexmem.org
Did they change the console to be read only in 4.0?? In 3.5 u3, they have been consistently saying it was a mistake/oversight and they was going to change it back to a read only console unless you purchased the license.
Kevin Buchanan
Chief Information Officer
Lexington Memorial Hospital
336-238-4286
kbuchanan@lexmem.org
Did they change the console to be read only in 4.0?? In 3.5 u3, they have been consistently saying it was a mistake/oversight and they was going to change it back to a read only console unless you purchased the license.
Hmz, read only how? I just enabled ssh and I'm allowed to do anything once I'm logged in. Anything except doing quick copies that is
It did indeed increase the speed by more than double.
Maybe they changed their mind...they put it in the release notes.
Kevin Buchanan
Chief Information Officer
Lexington Memorial Hospital
336-238-4286
kbuchanan@lexmem.org
I don't know what controller you are using but make sure you have a functional battery backed cache or you risk data loss or corruption.
I was fiddling around with the network settings in ESXi 4 a couple of days ago and I'm pretty sure that I once was able to copy a file at about 60MB/s, but havn't been able to repeat it so I'm starting to doubt my memory. Maybe I was just wishing very hard. Now my system has gone productive so I can't go berserk in the configs anymore
I managed to repeat it!
I can get >50MB/s in a copy operation, but unfortunately that doesn't make me understand more of the issue.
It depends on what file I copy. I have one vmdk that looks like this on the disk:
-rw------- 1 nobody nogroup 757596160 2009-06-18 16:05 SuseEnt-s001.vmdk
-rw------- 1 nobody nogroup 391184384 2009-06-18 16:05 SuseEnt-s002.vmdk
-rw------- 1 nobody nogroup 290127872 2009-06-18 16:05 SuseEnt-s003.vmdk
-rw------- 1 nobody nogroup 443678720 2009-06-18 16:05 SuseEnt-s004.vmdk
-rw------- 1 nobody nogroup 335413248 2009-06-18 16:05 SuseEnt-s005.vmdk
-rw------- 1 nobody nogroup 161480704 2009-06-18 16:05 SuseEnt-s006.vmdk
-rw------- 1 nobody nogroup 334692352 2009-06-18 16:05 SuseEnt-s007.vmdk
-rw------- 1 nobody nogroup 82640896 2009-06-18 16:05 SuseEnt-s008.vmdk
-rw------- 1 nobody nogroup 871 2009-06-18 16:05 SuseEnt.vmdk
In datastore browser it shows as just SuseEnt.vmdk 2.7GB.
If I copy this file I can see that data fly over the network at >50MB/s. If I copy any other file not made up of parts I get only ~10MB/s.
This apply only in datastore browser. If I copy from console with cp I still get 10MB/s. Is datastore browser opening up multiple simultaneous streams for the separate parts?
What's up with this?!?
So, are you only moving 2.7G at 50MB/sec? What command are you using to copy the files?
Kevin Buchanan
Chief Information Officer
Lexington Memorial Hospital
336-238-4286
kbuchanan@lexmem.org
It isn't the unsupported or ssh that is read only it is the RCL that is read only.
Thanks for the clarification.
Kevin Buchanan
Chief Information Officer
Lexington Memorial Hospital
336-238-4286
kbuchanan@lexmem.org
Sehr geehrter Absender,
ich bin bis 03.07.2009 nicht im Haus.
In dringenden Fällen wenden Sie sich bitte an
Herrn Münkle (ulrich.muenkle@albwerk.de Tel.: 07331/209-504) oder
Herrn Pfisterer (norbert.pfisterer@albwerk.de Tel.: 07331/209-508).
Ab 06.07.2009 stehe ich Ihnen gerne wieder zur Verfügung.