Hello everybody,
the old thread seems to be sooooo looooong - therefore I decided (after a discussion with our moderator oreeh - thanks Oliver -) to start a new thread here.
Oliver will make a few links between the old and the new one and then he will close the old thread.
Thanks for joining in.
Reg
Christian
These number seem off to me. Any input?
ngerasim wrote:
SERVER TYPE: Windows Server 2008 R2 CPU TYPE / NUMBER: 4 vCPU (16xE7440 CPUs) HOST TYPE: BL680c G5 STORAGE TYPE / DISK NUMBER / RAID LEVEL: EMC CX3-40F (RAID 5 MetaLUN 32 spindles) Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 5.57 10517 328 14% RealLife-60%Rand-65%Read 118.15 494 3 14% Max Throughput-50%Read 57.20 974 30 17% Random-8k-70%Read 114.58 502 3 20%
These number seem ad to me. Any input?
Something must be wrong with your setup. Your numbers are very low. ..or did you already have heavy load on the system when you ran the test?
Lars
There was no load on the system at all when I ran the tests.
Only 1 VM (the test VM) on a single ESX 4.1 Host on its own dedicated MetaLUN.
Host total resource capacity. 16x2.4 Ghz CPU, 131068 MB RAM.
Same test on local datastore.
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 6.53 | 9125 | 285 | 20% |
RealLife-60%Rand-65%Read | 115.68 | 499 | 3 | 13% |
Max Throughput-50%Read | 17.80 | 3061 | 95 | 28% |
Random-8k-70%Read | 68.48 | 634 | 4 | 25% |
It seems insane to me that my local datastore is outperforming the SAN!!!
ngerasim wrote:
Same test on local datastore.
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 6.53 | 9125 | 285 | 20% |
RealLife-60%Rand-65%Read | 115.68 | 499 | 3 | 13% |
Max Throughput-50%Read | 17.80 | 3061 | 95 | 28% |
Random-8k-70%Read | 68.48 | 634 | 4 | 25% |
ngerasim wrote:
It seems insane to me that my local datastore is outperforming the SAN!!!
SAN is not only about performance, but surely your SAN should give better number than these.
I don't know how your SAN is connected to your VMware environment, but please see this posting:
The NFS numbers during the first test rounds were also way off.
Lars
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 5.40 | 11004 | 343 | 11% |
RealLife-60%Rand-65%Read | 7.11 | 6135 | 47 | 13% |
Max Throughput-50%Read | 4.18 | 11468 | 358 | 15% |
Random-8k-70%Read | 7.98 | 5472 | 42 | 12% |
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 2.35 | 21317 | 666 | 23% |
RealLife-60%Rand-65%Read | 5.06 | 6210 | 48 | 22% |
Max Throughput-50%Read | 2.34 | 21782 | 680 | 23% |
Random-8k-70%Read | 3.16 | 7934 | 61 | 25% |
Test system is as follows.
HP BL680c G5 Blade. 1xQLogic QMH2462 4GB card using Virtual Connect. EMC CX3-40 SAN (1xMetaLUN comprised of 3 smaller LUNs with 5 spindles in each smaller LUN and the MetaLUN and all LUNs are attached to the same SP)
Storage is dedicated to this VM only. All spindles are FC.
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 5.48 | 10616 | 331 | 20% |
RealLife-60%Rand-65%Read | 117.75 | 499 | 3 | 18% |
Max Throughput-50%Read | 33.20 | 1694 | 52 | 28% |
Random-8k-70%Read | 116.08 | 501 | 3 | 18% |
brainslice wrote:
SERVER TYPE: VMwareVM Win2008R2SP1 4vCPU 8GB Ram CPU TYPE / NUMBER: X5470 HOST TYPE: DL380G5, ESX 4.1 STORAGE TYPE / DISK NUMBER / RAID LEVEL: HDS USP-VM / AMS2500, 4xR5(7+1) 600GB 15k SAS, HDP pool, 8Gb FC. Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 5.40 11004 343 11% RealLife-60%Rand-65%Read 7.11 6135 47 13% Max Throughput-50%Read 4.18 11468 358 15% Random-8k-70%Read 7.98 5472 42 12%
SERVER TYPE: HVM LPAR Win2008R2SP1 4vCPU 8GB Ram CPU TYPE / NUMBER: E5540 HOST TYPE: Hitachi BladeSymphony2000 E55A2 STORAGE TYPE / DISK NUMBER / RAID LEVEL: HDS USP-VM / AMS2500, 4xR5(7+1) 600GB 15k SAS, HDP pool, 8Gb FC. Test name Latency Avg iops Avg MBps cpu load Max Throughput-100%Read 2.35 21317 666 23% RealLife-60%Rand-65%Read 5.06 6210 48 22% Max Throughput-50%Read 2.34 21782 680 23% Random-8k-70%Read 3.16 7934 61 25% Can you please advise how your test VM is setup? How many disks, SCSI controllers, SCSI types, etc.
Host:
HP DL380 G5
2x Xeon X5470
32GB Ram
2x Qlogic QLE2560
Dual fabric connected to 2x Brocade 5140
Each fabric connected to one cluster of the USP-VM
Several 1TB datastore LUNs.
FC RR multipath for each datastore LUN, 2 paths active active using one SP on each cluster.
Datastore LUNs are carved out of an HDP pool which provides thin provisioning and I/O distribution.
HDP pool is composed of 4 raid groups of R5(7+1) for a total of 32 spindles of 600GB 15K SAS.
HDP pool is shared with about 30 other production VMs and a few SQL servers.
VM:
Win2008R2
4 vCPU
8GB RAM
1 virtual lsi SAS controller
1 100GB virtual disk
Can someone confirm my test is setup correctly?
Version 2006.07.27
'TEST SETUP ====================================================================
'Test Description
'Run Time
' hours minutes seconds
0 5 0
'Ramp Up Time (s)
0
'Default Disk Workers to Spawn
NUMBER_OF_CPUS
'Default Network Workers to Spawn
0
'Record Results
ALL
'Worker Cycling
' start step step type
1 5 LINEAR
'Disk Cycling
' start step step type
1 1 LINEAR
'Queue Depth Cycling
' start end step step type
8 128 2 EXPONENTIAL
'Test Type
NORMAL
'END test setup
'RESULTS DISPLAY ===============================================================
'Update Frequency,Update Type
4,WHOLE_TEST
'Bar chart 1 statistic
Total I/Os per Second
'Bar chart 2 statistic
Total MBs per Second
'Bar chart 3 statistic
Average I/O Response Time (ms)
'Bar chart 4 statistic
Maximum I/O Response Time (ms)
'Bar chart 5 statistic
% CPU Utilization (total)
'Bar chart 6 statistic
Total Error Count
'END results display
'ACCESS SPECIFICATIONS =========================================================
'Access specification name,default assignment
Max Throughput-100%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
32768,100,100,0,0,1,0,0
'Access specification name,default assignment
RealLife-60%Rand-65%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
8192,100,65,60,0,1,0,0
'Access specification name,default assignment
Max Throughput-50%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
32768,100,50,0,0,1,0,0
'Access specification name,default assignment
Random-8k-70%Read,ALL
'size,% of size,% reads,% random,delay,burst,align,reply
8192,100,70,100,0,1,0,0
'END access specifications
'MANAGER LIST ==================================================================
'Manager ID, manager name
1,WPDMA392
'Manager network address
10.66.66.250
'Worker
Worker 1
'Worker type
DISK
'Default target settings for worker
'Number of outstanding IOs,test connection rate,transactions per connection
64,ENABLED,500
'Disk maximum size,starting sector
8000000,0
'End default target settings for worker
'Assigned access specs
'End assigned access specs
'Target assignments
'Target
C:
'Target type
DISK
'End target
'End target assignments
'End worker
'End manager
'END manager list
Version 2004.07.30
Question. I set the queue depth on the ESX hosts, however didnt change the queue depth in Windows Server. Is there a specific setting I need to "set" to get more of a queue depth or better performance? This is for the ESXi Server 2003 VMs running on ESXi 4.1 u1
Hey All,
I need some help with my IOmeter Test Setup. The servers I am using are in a special corporate network, so there is no easy way for me to upload the config file.
Basically I am getting TOO MANY IOPS back from IOmeter based on my setup. IOmeter is reporting 1650 IOPS during a 3 minute test on this hardware.
Thank you for any feedback, I am calculating theoretical IOPS at 700-900, 1650 seems way too high. MB/s seems to be around 5-7.
Drew
HP P2000
Blade
VM
IOmeter Settings:
I think I get strange random performance because I use thin provisioned disk in this test? 18.49 latency on local server disks on "random-8k" test with only 2658 iops seems very wrong?
No other load or any other VM on the server
iometer version: 2006.07.27
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 2.45 | 23690 | 740 | 36% |
RealLife-60%Rand-65%Read | 16.95 | 2896 | 22 | 2% |
Max Throughput-50%Read | 1.36 | 39750 | 1242 | 53% |
Random-8k-70%Read | 18.49 | 2658 | 20 | 3% |
Max Throughput-100%Read | 1.33 | 39542 | 1235 | 55% |
RealLife-60%Rand-65%Read | 16.84 | 2921 | 22 | 6% |
Max Throughput-50%Read | 1.35 | 40095 | 1252 | 53% |
Random-8k-70%Read | 18.48 | 2663 | 20 | 7% |
Old style VMTN communities table:
SERVER TYPE:Dell PowerEdge R710 CPU TYPE / NUMBER: 2 x 6 core Intel HOST TYPE: VM w2k3 r2 enterprise x64 thin disk, paravirt scsi STORAGE TYPE / DISK NUMBER / RAID LEVEL: local 6 x sas 15k on perc raid controller |*TEST NAME*|*Avg Resp. Time ms*|*Avg IOs/sec*|*Avg MB/sec*|*% cpu load*| |*Max Throughput-100%Read*|2.45|23690|740|36%| |*RealLife-60%Rand-65%Read*|16.95|2896|22|2%| |*Max Throughput-50%Read*|1.36|39750|1242|53%| |*Random-8k-70%Read*|18.49|2658|20|3%| |*Max Throughput-100%Read*|1.33|39542|1235|55%| |*RealLife-60%Rand-65%Read*|16.84|2921|22|6%| |*Max Throughput-50%Read*|1.35|40095|1252|53%| |*Random-8k-70%Read*|18.48|2663|20|7%| |
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 0.00 | 13010 | 406 | 0% |
RealLife-60%Rand-65%Read | 6.35 | 2215 | 17 | 0% |
Max Throughput-50%Read | 315.48 | 19267 | 602 | 1% |
Random-8k-70%Read | 5.77 | 2344 | 18 | 0% |
I'm slightly confused by results. Probably it's because testfile was as big as RAM (32 000 000 sectors ~= 15,2Gbytes).
SERVER TYPE: ESX 4.1 / VM:Server 2008 R2 - 4 GB mem CPU TYPE / NUMBER: 1 HOST TYPE: Fujitsu Siemens RX300S4 20 GB mem With brocade 815 HBA STORAGE TYPE / DISK NUMBER / RAID LEVEL: HP MSA P2000G3 8GB FC / 12 Disks SAS 10K 300 GB / Raid5
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 3.06 | 19160 | 598 | 65% |
RealLife-60%Rand-65%Read | 153.71 | 253 | 1 | 54% |
Max Throughput-50%Read | 4.36 | 13370 | 417 | 47% |
Random-8k-70%Read | 97.74 | 266 | 2 | 75% |
Old style VMTN communities table:
SERVER TYPE: ESX 4.1 / VM:Server 2008 R2 - 4 GB mem CPU TYPE / NUMBER: 1 HOST TYPE: Fujitsu Siemens RX300S4 20 GB mem With brocade 815 HBA STORAGE TYPE / DISK NUMBER / RAID LEVEL: HP MSA P2000G3 8GB FC / 12 Disks SAS 10K 300 GB / Raid5 |*TEST NAME*|*Avg Resp. Time ms*|*Avg IOs/sec*|*Avg MB/sec*|*% cpu load*| |*Max Throughput-100%Read*|3.06|19160|598|65%| |*RealLife-60%Rand-65%Read*|153.71|253|1|54%| |*Max Throughput-50%Read*|4.36|13370|417|47%| |*Random-8k-70%Read*|97.74|266|2|75%| |
I've made few test to investigate performance of iSCSI-exported storage. To do it, I've tried next configurations: Standalone (Physical) installed Windows 2008 Server with iSCSI initiator in MPIO mode. VmWare ESXi 4.1 with iSCSI-exported storage in MPIO mode (round-robin policy iops=3), with previous standalone Win2008 Server, converted to virtual, so I can test storage in two modes: As virtual disk from esxi's iSCSI-connected datastore As "direct" connection to SAN vSwitch, so I could use native Windows iSCSI initiator from virtual machine. So results are below, and I hope it will be interesting for all:
SERVER: Supermicro 6016T-NTRF, X8DTU-F, Xeon E5620, 16Gb RAM, Intel E1G44HT I340-T4 Quad Gigabit NIC (82580) STORAGE: Supermicro CSE-836E16-R1200B, X8DTH-iF, 2x Xeon E5506, 16Gb RAM, 8x2Tb Hitachi 7k2000 SATA, 80Gb Intel X25-M SSD, Intel E1G44HT I340-T4 Quad Gigabit NIC (82580) STORAGE: TrinityNAS (based on NexentaStor), RAIDZ2 pool, SSD cache Jumbo frames is on everythere. Tested on Iometer 1.1.0 with OpenPerformance32.icf pattern.
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 0.00 | 13010 | 406 | 0% |
RealLife-60%Rand-65%Read | 6.35 | 2215 | 17 | 0% |
Max Throughput-50%Read | 315.48 | 19267 | 602 | 1% |
Random-8k-70%Read | 5.77 | 2344 | 18 | 0% |
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 0.00 | 6085 | 190 | 0% |
RealLife-60%Rand-65%Read | 2.22 | 771 | 6 | 0% |
Max Throughput-50%Read | 159.68 | 9751 | 304 | 0% |
Random-8k-70%Read | 1.53 | 619 | 4 | 0% |
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 0.00 | 12626 | 394 | 0% |
RealLife-60%Rand-65%Read | 3.09 | 1073 | 8 | 0% |
Max Throughput-50%Read | 327.01 | 19979 | 624 | 0% |
Random-8k-70%Read | 2.15 | 875 | 6 | 0% |
Any comments?
Table over results:
Physical parameters | Virtual Parameters | Max Throughput-100% Read | RealLife-60%-Rand-65% Read | |||||||||
Storage | Raid | Phys Host | vscsi type | vmdk | Latency | Avg iops | Avg MBps | cpu load | Latency | Avg iops | Avg MBps | cpu load |
Local disks | Perc H700 with 6 x 600gb SAS 15k 3.5" - Raid 10 | Dell PowerEdge R710 | LSI Logic SAS | 40 gb thick | 2,66 | 22429 | 700 | 88 | 16,18 | 2957 | 23 | 80 |
Local disks | Perc H700 with 6 x 600gb SAS 15k 3.5" - Raid 10 | Dell PowerEdge R710 | LSI Parallel | 40 gb thick | 3 | 19621 | 613 | 0 | 16,04 | 3018 | 23 | 27 |
Local disks | Perc H700 with 6 x 600gb SAS 15k 3.5" - Raid 10 | Dell PowerEdge R710 | Vmware Paravirtual | 40 gb thick | 2,15 | 27769 | 867 | 32 | 15,9 | 3011 | 23 | 7 |
iSCSI SAN | Md3000i 4 disk dg/vd - Raid 5 | Dell PowerEdge R710 | LSI Logic SAS | 40 gb thick | 15,38 | 3907 | 122 | 16 | 36,71 | 1108 | 8 | 45 |
iSCSI SAN | Md3000i 4 disk dg/vd - Raid 5 | Dell PowerEdge R710 | LSI Parallel | 40 gb thick | 15,32 | 3904 | 122 | 0 | 35,71 | 1131 | 8 | 25 |
iSCSI SAN | Md3000i 4 disk dg/vd - Raid 5 | Dell PowerEdge R710 | Vmware Paravirtual | 40 gb thick | 15,14 | 3967 | 123 | 1 | 35,07 | 1119 | 8 | 18 |
iSCSI SAN | Md3000i 2 disk dg/vd - Raid 1 | Dell PowerEdge R710 | LSI Logic SAS | 40 gb thick | 15,17 | 3958 | 123 | 17 | 52,25 | 902 | 7 | 34 |
iSCSI SAN | Md3000i 14 disk dg/vd - Raid 10 | Dell PowerEdge R710 | LSI Logic SAS | 40 gb thick | 17,14 | 3520 | 110 | 16 | 15,45 | 3696 | 28 | 18 |
iSCSI SAN | Md3000i 14 disk dg/vd - Raid 5 | Dell PowerEdge R710 | LSI Logic SAS | 40 gb thick | 17,06 | 3535 | 110 | 16 | 19,49 | 2542 | 19 | 29 |
Local SSD | no raid - ESB2 intel - Crucial RealSSD C300 2,5" 128gb | Dell Precision T5400 | n/a | n/a | 7,15 | 8243 | 257 | 11 | 6,68 | 8629 | 67 | 9 |
Local SSD | no raid - ICH9 intel - Intel 80gb G2 M | Dell Latitude E6400 | n/a | n/a | 9,31 | 6402 | 200 | 35 | 16,26 | 3305 | 25 | 56 |
Local disks | Perc 5/i - 4 disks 300gb sas 15k raid 5 | Dell PowerEdge 2950 | n/a | n/a | 3,64 | 17175 | 536 | 5 | 37,42 | 1197 | 9 | 3 |
Hosts used in test:
Host | Model | Cpu | Memory | i/o controller | Local disk(s) | OS | NIC |
vSphere server | Dell PowerEdge R710 | 2 x Intel Xeon X5680 3.33 Ghz 6 core, 12M cache, 6.40 GT/s QPI 130W TDP, Turbo, HT | 96 GB memory for 2 cpu (12 x 8 GB Dual Rank RDIMMs) 1333 MHz | Perc H700 Intergrated, 1 GB NV Cache, x6 Backplane | 1 x SDcard 6 x 600 GB SAS 6 Gbps 15K 3.5” , raid 10 | VMware ESXi 4.1.0 build 348381 on SDcard | Embedded broadcom Gbe LOM with TOE and iSCSI offload (4 port) & Intel Gigabit ET Quad port server adapter PCIe x4 & Intel X520 DA 10GbE Dual Port PCIe x8 |
Workstation | Dell Precision T5400 | 1 x Xeon E5440 2,83 Ghz quad core | 16 GB Memory fully buffered dimm | Intel 5400 chipset (intel ESB 2 Sata raid controller) | 1 x Crucial RealSSD C300 2.5” 128GB Sata 6 gb/s | Windows 7 Enterprise x64 | Broadcom 57xx & Intel Pro 1000 PT dual SA |
Laptop | Dell Latitude E6400 | 1 x 2.53 Ghz Intel Core 2 Duo | 4 GB Memory | Intel ICH9 | 1 x Intel 80gb SSD gen2 M | Windows 7 Enterprise x64 | Intel 82567 |
Physical server | Dell PowerEdge 2950 | 2 x Intel Xeon 5150 - 2.66 ghz dual core 4mb L2 cache | 16 gb memory 533 mhz | Perc 5/i | 4 x 300 gb 15k sas, raid 5 | Windows 2008 R2 Enterprise | Broadcom BCM5708C NetExtreme II & Intel Pro/1000 PT Dual Port SA |
iSCSI SAN used in test:
Dell PowerVault MD3000i – 15 x 600gb sas 15k (one global hotspare)
2 x PC5424 Dell PowerConnect switches (2 isolated iscsi subnets as recommended for MD3000i)
LAN switches:
Cisco 2960 series and Nexus 5010
Virtual Machines used:
VM | OS | vcpu | scsi | vmdk | Memory | NIC | VM HW vers |
Iometer01 | Windows 2008 R2 SP1 (x64) | 1 (default) | LSI Logic SAS (default) | 40 gb thick | 4 gb | Vmxnet 3 | 7 |
Iometer02 | Windows 2003 R2 Sp2 x64 | 2 | LSI Logic Parallel | 40 gb thick | 8 gb | Vmxnet 3 | 7 |
Iometer03 | Windows 2008 R2 SP1 (x64) | 2 | Paravirtual | 40 gb thick | 8 gb | E1000 (default) | 7 |
Comparison between virtual scsi types and guest OS iometer performance
Comparison of VM Iometer01,02,03 running on local server disks to see if there is a noticeable difference on different guest OS and virtual scsi adapter type:
SERVER TYPE: VM iometer01 - W2K8 R2 SP1 x64 - LSI Logic SAS CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Perc H700 with 6 x 600GB SAS 6gbps 15k 3.5" - Raid10 | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 2.66 | 22429 | 700 | 88% |
RealLife-60%Rand-65%Read | 16.18 | 2957 | 23 | 80% |
Max Throughput-50%Read | 1.38 | 42340 | 1323 | 63% |
Random-8k-70%Read | 17.52 | 2745 | 21 | 38% |
SERVER TYPE: VM iometer02 - W2K3 R2 SP2 x64 - LSI Parallel CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Perc H700 with 6 x 600GB SAS 6gbps 15k 3.5" - Raid10 | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 3.00 | 19621 | 613 | 0% |
RealLife-60%Rand-65%Read | 16.04 | 3018 | 23 | 27% |
Max Throughput-50%Read | 1.34 | 39659 | 1239 | 0% |
Random-8k-70%Read | 17.56 | 2751 | 21 | 26% |
SERVER TYPE: VM iometer03 - W2K8 R2 SP1 x64 - VMware Paravirtual CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Perc H700 with 6 x 600GB SAS 6gbps 15k 3.5" - Raid10 | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 2.15 | 27769 | 867 | 32% |
RealLife-60%Rand-65%Read | 15.90 | 3011 | 23 | 7% |
Max Throughput-50%Read | 1.22 | 48797 | 1524 | 48% |
Random-8k-70%Read | 17.50 | 2738 | 21 | 7% |
Logs: iometer01-local-01, iometer02-local-01, iometer03-local-01
Comparison of same VMs on MD3000i SAN:
SERVER TYPE: VM iometer01 - W2K8 R2 SP1 x64 - LSI Logic SAS CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PowerVault MD3000i iscsi SAN, diskgroup & virtual disk 0 with 4 disks using Raid 5 (database i/o type 128k segment) | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 15.38 | 3907 | 122 | 16% |
RealLife-60%Rand-65%Read | 36.71 | 1108 | 8 | 45% |
Max Throughput-50%Read | 12.40 | 4816 | 150 | 17% |
Random-8k-70%Read | 40.56 | 1103 | 8 | 41% |
SERVER TYPE: VM iometer02 - W2K3 R2 SP2 x64 - LSI Parallel CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PowerVault MD3000i iscsi SAN, diskgroup & virtual disk 0 with 4 disks using Raid 5 (database i/o type 128k segment) | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 15.32 | 3904 | 122 | 0% |
RealLife-60%Rand-65%Read | 35.71 | 1131 | 8 | 25% |
Max Throughput-50%Read | 16.82 | 3644 | 113 | 0% |
Random-8k-70%Read | 40.50 | 1107 | 8 | 27% |
SERVER TYPE: VM iometer03 - W2K8 R2 SP1 x64 - VMware Paravirtual CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PowerVault MD3000i iscsi SAN, diskgroup & virtual disk 0 with 4 disks using Raid 5 (database i/o type 128k segment) | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 15.14 | 3967 | 123 | 1% |
RealLife-60%Rand-65%Read | 35.07 | 1119 | 8 | 18% |
Max Throughput-50%Read | 12.44 | 4791 | 149 | 1% |
Random-8k-70%Read | 41.34 | 1105 | 8 | 12% |
Logs: iometer01-san-01, iometer02-san-01, iometer03-san-01
Comment:
Different windows server OS and virtual scsi adapter types does not change the performance in a dramatic way (adding disks or changing raid systems has a much larger impact)
However it looks like LSI Logic SAS uses a lot more cpu than the other types of virtual scsi adapters, I only did each test once so maybe more tests are needed to confirm that.
Local server disks are much faster than the cheap MD3000i iscsi SAN for single VM performance
Note: SAN diskgroup and virtual disk was only using 4 disks on the SAN box so the results cannot be directly compared to server disks
Dell PowerVault Md3000i iSCSI SAN iometer performance with different configurations
Comparison to see the effect with various raid and diskgroups:
First tested with a small diskgroup & virtual disk/lun of 4 x drives using raid 5. Then I also tested on a 2 drive raid 1 lun on the iscsi SAN.
After that I tested with a 14 drive disk group (the virtual disk does not fill all space then because of the VMware 2 tb limit). The 14 drive disk group I tested with raid 5 and raid 10.
1mb block on 4 spindles r5 iscsi san:
SERVER TYPE: VM iometer01 - W2K8 R2 SP1 x64 - LSI Logic SAS CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PowerVault MD3000i iscsi SAN, diskgroup & virtual disk 0 with 4 disks using Raid 5 (database i/o type 128k segment) | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 15.18 | 3954 | 123 | 17% |
RealLife-60%Rand-65%Read | 34.52 | 1141 | 8 | 47% |
Max Throughput-50%Read | 12.45 | 4798 | 149 | 17% |
Random-8k-70%Read | 41.05 | 1114 | 8 | 38% |
Raid 1
SERVER TYPE: VM iometer01 - W2K8 R2 SP1 x64 - LSI Logic SAS CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PowerVault MD3000i iscsi SAN, diskgroup & virtual disk 0 with 2 disks using Raid 1 (database i/o type 128k segment) | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 15.17 | 3958 | 123 | 17% |
RealLife-60%Rand-65%Read | 52.25 | 902 | 7 | 34% |
Max Throughput-50%Read | 12.40 | 4803 | 150 | 17% |
Random-8k-70%Read | 59.63 | 919 | 7 | 23% |
Raid 10 – 14 disks
SERVER TYPE: VM iometer01 - W2K8 R2 SP1 x64 - LSI Logic SAS CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PowerVault MD3000i iscsi SAN, diskgroup & virtual disk 0 with 14 disks using Raid 10 (database i/o type 128k segment) | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 17.14 | 3520 | 110 | 16% |
RealLife-60%Rand-65%Read | 15.45 | 3696 | 28 | 18% |
Max Throughput-50%Read | 14.29 | 4144 | 129 | 17% |
Random-8k-70%Read | 13.66 | 3936 | 30 | 24% |
Raid 5 – 14 disks
SERVER TYPE: VM iometer01 - W2K8 R2 SP1 x64 - LSI Logic SAS CPU TYPE / NUMBER: 2 x Intel X5680 3.33GHz HOST TYPE: Dell PowerEdge R710 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PowerVault MD3000i iscsi SAN, diskgroup & virtual disk 0 with 14 disks using Raid 5 (database i/o type 128k segment) | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 17.06 | 3535 | 110 | 16% |
RealLife-60%Rand-65%Read | 19.49 | 2542 | 19 | 29% |
Max Throughput-50%Read | 14.38 | 4112 | 128 | 17% |
Random-8k-70%Read | 16.16 | 2754 | 21 | 38% |
Comment:
As expected raid 10 gives better performance at the cost of less space. Random i/o sees a big improvement with more disks added to the raid.
Note: I also tested with different block sizes and with and without storage i/o control, but these settings did not seem to have much of an impact on performance. Storage I/O control should only kick when doing multiple VM I/O loads so that is as expected I think. I will use a default block size of 8MB on all my datastores.
Note:
All tests are done without using jumbo frames on the iscsi traffic
Physical test results for comparison
SERVER TYPE: Physical Dell Prec T5400 - Win 7 Enterprise x64 CPU TYPE / NUMBER: 1 x Intel Xeon E5440 HOST TYPE: Phyiscal Dell Prec T5400 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Crucial RealSSD C300 2.5" 128gb sata | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 7.15 | 8243 | 257 | 11% |
RealLife-60%Rand-65%Read | 6.68 | 8629 | 67 | 9% |
Max Throughput-50%Read | 11.13 | 5067 | 158 | 11% |
Random-8k-70%Read | 5.45 | 10427 | 81 | 30% |
SERVER TYPE: Physical Dell Latitude E6400 - Windows 7 Enterprise x64 CPU TYPE / NUMBER: 1 x 2.53 Ghz intel core duo 2 HOST TYPE: Physical Dell Latitude E6400 - Windows 7 Enterprise x64 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Intel 80gb SSD gen2 M | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 9.31 | 6402 | 200 | 35% |
RealLife-60%Rand-65%Read | 16.26 | 3305 | 25 | 56% |
Max Throughput-50%Read | 62.80 | 903 | 28 | 21% |
Random-8k-70%Read | 10.58 | 4996 | 39 | 37% |
SERVER TYPE: Physical Dell PowerEdge 2950 - W2K8 R2 x64 CPU TYPE / NUMBER: 1 x 2.53 Ghz intel core duo 2 HOST TYPE: Physical Dell PowerEdge 2950 - W2K8 R2 x64 STORAGE TYPE / DISK NUMBER / RAID LEVEL: Perc 5/i - 4 x 300 gb sas 15k - raid 5 | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 3.64 | 17175 | 536 | 5% |
RealLife-60%Rand-65%Read | 37.42 | 1197 | 9 | 3% |
Max Throughput-50%Read | 4.91 | 12721 | 397 | 3% |
Random-8k-70%Read | 40.15 | 1161 | 9 | 1% |
Comment:
As expected a server with raid of many disks is faster than single SSD on sequential throughput but slower on random I/O.
Note: These are physical tests and since virtualization has some overhead they are usually faster than the virtualized servers (iometer01-03) in similar configurations. Also note that I have tested some mainstream SSD disks here which is not usually used in servers (server ssd costs a lot more), however it is still an interesting comparison when for example a developer has to choose between running a virtual machine in vmware workstation on an SSD laptop/workstation or use a shared VMware LabManager server with SAN storage. The PE2950 server tested is a generation 9 Dell server and is much older than the generation 11 R710 servers, but that is one of the advantages of virtualization that you can buy new servers each year and move virtual servers to new hosts to upgrade the speed (then over time virtualization might actually be faster than the old model of buying a dedicated server for a solution and running it for 4 years).
Sono fuori ufficio e con limitato accesso alle email.
Rientrerò il 21/03/2011.
Per urgenze contattare l'assistenza tecnica allo 045 8738738.
Francesco Bonetti
RTC SpA
I'm calling serious BS on your DAS results. Its physically impossible to get 22k IOPS out of a 6 disk RAID 10 config. Also, some of your results have 0% processor utilization, exactly how does that work?