Hello everybody,
the old thread seems to be sooooo looooong - therefore I decided (after a discussion with our moderator oreeh - thanks Oliver -) to start a new thread here.
Oliver will make a few links between the old and the new one and then he will close the old thread.
Thanks for joining in.
Reg
Christian
Hi Gabriel,
I don't use DAS anywhere I have tested local server disks and SAN boxes. I guess with DAS you are refering to the internal raid with 6 disks on the R710 server? Is 22k iops not normal for internal raid 10 disks with 6 spindles? I don't have enough experience with this test to know that, I would love to know if I have done something wrong in the test
I have followed what I think is the normal procedure here, created a virtual machine with different versions of windows server and installed iometer 2006 then run the test. The numbers are from the logfiles parsed by the web page http://vmktree.org/iometer/
I am not posting here to have the "best numbers" I want to find what is normal performance with a test that is repeated by other people so I can use that test as a reference for finding issues with i/o at other sites.
This is the procedure I have done:
If there are some other tests you would like me to do to verify the numbers I'm happy to do so.
Why some tests say 0% cpu load I have no idea I'm not an iometer expert, I have attatched the logfile from one test "iometer02-san01.csv" which is the test that seems to have the lowest cpu load so maybe you can see if something is not right there?
Thanks for your input, interresting to know if the test numbers are sane.
My last day with Ellett Brothers will be March 25th, please call 803-345-3751 to be directed to the appropriate person.
Thanks!
Steve,
When using iometer inside a VM there are several things that may affect your results. One is disk and disk controller cache that may affect the results and give higher numbers. Using a larger iometer test file will solve this problem.
Another thing that may affect the results is if the clock inside the guest is not correct. This happens especially with high cpu load. In 3.5 the solution to this issue was the descheduled time service. This service is however not available anymore on newer ESX versions.
You're also running your tests on the local C: drive which typically is the drive where the OS is installed. It would be better to use a dedicated disk drive.
Regarding the cpu load, by studying the csv file we can see that the load is indeed higher than 0%, but since you're using multiple workers (2 cpus), only the load of the second cpu is reported by that page.
Lars
My last day with Ellett Brothers will be March 25th, please call 803-345-3751 to be directed to the appropriate person in the IT department.
Thanks!
Hi Lars,
Yes I agree with all you are saying but I think this is the most efficient way to have a simple iometer reference test (a "standard" test that it is easy to teach others to do). I would follow your suggestions if my goal were to have a competition between SAN products in performance, but my goal is simply to see systems that have major issues with either random or sequential i/o performance at other sites. My company frequently install heavy database solutions on old SANs that often perform very badly which is a waste of time and money, nice to have some numbers to show whats wrong in an efficent way.
I guess one option would be to have a virtual machines and add extra VMDK disks on other LUNs to test them, that would also be efficient. But what do most people do when they post results here I wonder? I did my first tests here on a server with no other VMs on a LUN with no other VMs, my goal was to get the best result on a non busy environment so I can see the effects later when the environment is busy.
I think the test file is large enough in this test, the VM has 8 gb of ram and the PERC controller and MD3000i controllers does not have more than 1 gb cache. However I'm rather new at using iometer before I used simpler tools like HDtune, HDtach. Those tools are not so good at showing random i/o performance as I think iometer is, and it is very nice to have some reference numbers and comments from this community. It might also have an effect on my own storage purchases in the future I guess
Do you think 22k iops is impossible with 6 drives like Gabriel says here? For comparison I did a test on a physical PE2900 host with 8 drives and it got these numbers:
W2k8 r2 enterprise
Raid 10
8 x SAS Seagate 300 gb sas 15k
Perc 6/i
SEP11 AV installed
SERVER TYPE: Dell PowerEdge 2900 III CPU TYPE / NUMBER: HOST TYPE: Dell PowerEdge 2900 III STORAGE TYPE / DISK NUMBER / RAID LEVEL: R10 8xSAS 15K Seagate 3.5" Perc 6/i | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 2.98 | 18170 | 567 | 2% |
RealLife-60%Rand-65%Read | 15.79 | 3079 | 24 | 0% |
Max Throughput-50%Read | 3.06 | 19046 | 595 | 3% |
Random-8k-70%Read | 17.55 | 2813 | 21 | 0% |
However on this host the test file might have been too small I guess as the server has 32gb of ram and w2k8r2 does a lot of strange i/o caching.
Just for fun I also tested a PE2900 server with mainstream intel 80gb g2 ssd drives and then it had an awesome performance in the random I/O as expected:
W2k8 r2 ent
Raid 5
8 x intel ssd 80gb gen2
Perc 6/i
no AV
SERVER TYPE: Dell PowerEdge 2900 III CPU TYPE / NUMBER: HOST TYPE: Dell PowerEdge 2900 III STORAGE TYPE / DISK NUMBER / RAID LEVEL: R5 8xIntel 80gb SSD gen2 Perc 6/i | ||||
Test name | Latency | Avg iops | Avg MBps | cpu load |
Max Throughput-100%Read | 3.04 | 19950 | 623 | 2% |
RealLife-60%Rand-65%Read | 4.59 | 12480 | 97 | 1% |
Max Throughput-50%Read | 3.12 | 18678 | 583 | 3% |
Random-8k-70%Read | 5.21 | 10396 | 81 | 0% |
I know its impossible to make the perfect test but at least I hope to avoid doing some really bad mistakes that makes the numbers meaningless. And I'm hoping these numbers can be compared to others in this forum.
Didn't know about the results only showing one cpu thats interresting.
Thanks for your input,
S
Steve,
When I setup a new system I usually do the same thing as you and run a few tests before it gets any load just to get a baseline of that system. If I later get performance issues I can compare with my initial results in the troubleshooting process.
The sequential tests are often highly affected by the cache and that's why you get such high nimbers on those tests. The disk controllers can predict sequential reads and will often read larger blocks than the ones requested to get maximum performance on such operations. The disks also have their own cache that helps a bit. These things help are especially helpful for operations such as backup and large file copy operations.
In a VMware environment with many VMs sharing the same storage, these numbers are not as important as the random workload tests. Random and RealLife show a workload more similar to the one you can expect during heavy load and are also the ones worth paying most attention to. As your tests show, the SSD tests show really good performance.
Lars
Hmm, I am getting pretty bad result from my StarWind iSCSI setup.
I have my SAN setup as Raid 60, it seem like when it is doing RealLife random access, I suffered the most. Only averageing 3Mbps..
This is a Dell R710 server connect to a Chenbro 72TB box running raid 60 thru gig nic.
Maybe raid 60 is not that great for random access, but 3Mbps..
Anyone have any insight ? thanks so much !
Max Throughput-100% Read
IOps - 44145.3
MBps- 1379.5
RealLife-60%Rand-65%Read
IOps - 3664.4
MBps- 28.62
Through iSCSI .
Max Throughput-100% Read
IOps - 2967.95
MBps- 92.74
RealLife-60%Rand-65%Read
IOps - 400.4
MBps- 3.12
I am no longer employed with Ellett Brothers, please call 803-345-3751 to be directed to the appropriate person in the IT department.
Thanks!
Jaseliu,
For random tests you should not pay as close attention to the MBps as the IOps. Having 3664 IOps on rge Random test is quite good, but your number over iSCSI (400) is quite bad. How many disks do you use in your setup? What type of disks?
Lars
Hi, thanks for the reply.
I am running 24 7200rpm SATA HDs Raid 60 setup.
I think I sort of figure out why I am getting the bad numbers.
Here is why, not sure what's the difference, but it's kinda weird to me.
I ran the first test without a filesystem in Iometer, and that's the crappy result I got back.
Then, I formatted the drive to be NTFS, and let IOMeter created a temp file there, and here is my result.
Max Throughtput 100% Read
IOPs: 2995.9
MBps 93.6
RealLife 60% Rand
IOps: 2271.2 (Read: 1474 Write 796)
MBps 17.7
Max Throughput 50%
IOps: 1323.5
MBps: 41.3
Random 8k 70% Read
IOps: 2670.7
Mbps: 20.8
That's a good amount of increase of the raw test that I did, but why is that ? and why is MBps not as important in random test ?
Thanks !
Test name | Avg. Resp Time /ms | Avg IOps | Avg MBps |
---|---|---|---|
Max Throughput-100%Read | 5.022 | 12206 | 381 |
RealLife-60%Rand-65%Read | 235.3 | 125 | 0.97 |
Max Throughput-50%Read | 190.2 | 155 | 4.8 |
Random-8k-70%Read | 234.5 | 125 | 0.97 |
Kontakt drift@eogs.dk eller +45 3330 7587
SERVER TYPE: Windows 2008 R2 Standard, 8Gb RAM | |||
Test name | Avg. Resp Time /ms | Avg IOps | Avg MBps |
Max Throughput-100%Read | 5.696 | 10673 | 333 |
RealLife-60%Rand-65%Read | 14.06 | 2063 | 16 |
Max Throughput-50%Read | 19.50 | 1491 | 46 |
Random-8k-70%Read | 78.60 | 377 | 3 |
Its informative one provides a lots of information on topic also provide more data on topic.
here are my insane specs
it's a 2 server configuration.
1 file server, 1x 10GBps infiniband interconnect, 1 vm server
file server config:
8gb ram
quad core 3ghz@1333fsb
adaptec 5805 SAS controller running 16 sata drives WD 1TB black edition with BBU
Mellanox Connect-X infiniband card
6 Arrays to of 2tb raid 6 ( 1 array is built from 7x 1tb disks raid 6, 3 per drives)
ESXI Server: ESXI 4.1 U1
16GB ram
8gb compact flash disk
CPU: 2x quad core 3ghz @1333fsb
network card 10GB infiniband to connect to my file server
Infiniband switch: Topspin 90
-------------
Test | average io response(ms) | Maximum IO response(ms) | cpu utilization | IOPS | MBPS |
---|---|---|---|---|---|
4K 100% read 0% random, 1 queue | 0.1235 | 73.1755 | 22.74% | 8113.81 | 31.51 |
16K 100% read, 0% random, 1 queue | 0.1559 | 12.5543 | 21.27% | 6386.21 | 99.91 |
4K 75% read, 0% random, 1 queue | 0.1316 | 25.0660 | 20.85% | 7565.66 | 29.63 |
16K 75% read, 0% random, 1 queue | 0.2561 | 109.4688 | 13.62% | 3903.28 | 61.95 |
4K, 100% read, 100% random 1 queue | 6.3223 | 63.3976 | 0.65% | 158.97 | 0.62 |
16K, 100% read, 100% random 1 queue | 6.6691 | 77.2706 | 0.53% | 151.92 | 2.35 |
4K, 75% read, 100% random, 1 queue | 6.5592 | 106.5505 | 0.60% | 152.16 | 0.56 |
16K 75% read, 100% random, 1 queue | 7.3537 | 62.5770 | 0.44% | 135.34 | 2.12 |
same tests with 64 queue length | |||||
4K 100% read 0% random, 64 queue | 1.5147 | 86.9491 | 96.93% | 42083.38 | 164.65 |
16K 100% read, 0% random, 64 queue | 2.9452 | 79.2967 | 54.66% | 20837.64 | 328.29 |
4K 75% read, 0% random, 64 queue | 3.1938 | 132.7558 | 44.08% | 20049.75 | 78.04 |
16K 75% read, 0% random, 64 queue | 18.0842 | 464.8004 | 9.62% | 3497.97 | 54.66 |
4K, 100% read, 100% random 64 queue | 122.8217 | 594.2059 | 1.54% | 583.22 | 2.27 |
16K 100% read, 100% random, 64 queue | 106.5224 | 591.4899 | 1.72% | 532.28 | 8.31 |
4K, 75% read, 100% random, 64 queue | 218.6931 | 660.9712 | 0.90% | 283.34 | 1.11 |
16K 75% read, 100% random, 64 queue | 233.4813 | 621.7400 | 1.04% | 271.87 | 4.24 |
anyway, i ran some tests as i just finished configuring it a few days ago after pain and pain of configuration
All, We just purchased a NetApp SAN and I am looking for feedback on whether or not the performance is where it should be. Help is appreciated. I need to go into production shortly.
SERVER TYPE: 2008 64bit R2 VM ON ESXi 4.1.0
CPU TYPE / NUMBER: 4 VCPU - 8GB Ram
HOST TYPE: IBM x3690 X5 - Xecon x7560 2.27Ghz
STORAGE TYPE / DISK NUMBER / RAID LEVEL: Netapp FAS2040 / two disk aggregates - see below for details / RAID DP
SAN TYPE / HBAs: 1GB ISCSI - Server has two 1GB ports set to go round robin to the Netapp SAN. Cisco 2960 swiches. Netapp 2040 has 2 trunked ports comprised of two 1GB ports. So it has two 2GB trunks.
Note: Jumbo Frames are in use
Here is the results from an array made up of 12 300GB - 15k SAS disks. This is on controller A within the FAS2040.
Here is the results from the second array on a seperate DS4243 which attached to the 2040 via a SAS channel. It is comprised of 24 - 1TB SATA Disk. I believe they are 7200 RPM.
Lastly here are the results from some local disk. This is two 300GB 15k SAS 2.5" SAS disks that are local in the IBM3690.
Any feedback is appreciated! Again I am going live and need to know if these results are acceptable for this type of configuration. I have dug thru the three threads on this site and just want to say they are awesome. I have found a ton of good info to compare but still would like feedback if anyone has any.
Thanks in advance,
SERVER TYPE: Intel Modular Server
CPU TYPE / NUMBER: Xeon 2.4GHz/2
HOST TYPE: ESX 4.1
STORAGE TYPE / DISK NUMBER / RAID LEVEL:
Intel Modular Server / 6 / 50
Test name Latency Avg iops Avg MBps cpu load
Max Throughput-100%Read
0.00
7617
238
31%
RealLife-60%Rand-65%Read
5.33
1855
14
47%
Max Throughput-50%Read
38.90
2374
74
68%
Random-8k-70%Read
4.06
1656
12
45%
Any thoughts? does this look about right for this type of server (blade) with 6 discs in a RAID 50 array?
Hello All,
I've found this a useful thread!
Please note that when the latency numbers report 0 for Max Throughput Latency it probably means you've used the latest version of IOMeter and parsed it via - http://vmktree.org/iometer/
I have discovered that the first data point have moved from column O to column R and the 4th data point has gone from AT to AW.
This means:
The first field in the table reports write MBps (Decimal) instead of Average Response Time
The fourth field is not CPU but I'm not certain which one it is, doesn't seem to be the data in AT which I'd expect.
I expect to be posting some numbers for our environment later, I'll be interested in feedback.
One other thing, we believe that our Jumbo Frames via ESX 4.1 is not working. We are able to send an unfragmented ping of 8972 bytes yet when doing a wireshark capture within an image using our iSCSI interface the largest packets are 1490 - we also see no Jumbo frames being counted at the SAN or our switches. (Cisco 3750's) Does anyone know if the count of giants using the show interface command will increase when Jumbo frames work?
Thanks,
John
hi guys
this is mine
using iometer-2008-06-22-rc2.win.32 and http://www.mez.co.uk/OpenPerformanceTest.icf
formated at http://vmktree.org/iometer/
too late I realize I did theses test on a Windows 7 box
I executed 2. Only difference is the amount of Hard Drives first is 3 disk RAID 5 and the second is 9 Disk RAID 5. Based on that what do you think is much better for my Storage and Vmware Infraestructure Less disk or more Disk.
your inputs good or bad? is there a web page that says that value is low because .....
BTW why RealLife and Random Are so low 6 MBps?
thanks a lot
--------------- Update -----------------
I asked to one of our pre-sale storage guy about this since I really don't know what those test are or mean.
Max Throughput-100%Read = All the bandwith only for READ. It uses all the Bandwith available. This is secuential.
RealLife-60%Rand-65%Read = A common or RealLife metric which says Reading 65 percent but in a random way it means that blocks that are being read are not secuencial.
Max Throughput-50%Read = This means 50% read 50 % write secuencial block.
Random-8K-70%Read = Using 8K blocks and doing 70% read on random blocks.
is that above is correct much disks better performance or at least less latency
SERVER TYPE: Windows 2008 R2 on ESXi 4.1
CPU TYPE / NUMBER: 2 vCPU
HOST TYPE: HP DL380 G7
STORAGE TYPE / DISK NUMBER / RAID LEVEL: HP P2000i 1Gb G3 MSA / 16x146gb 10k / RAID10
ESXi software iSCSI MPIO
Test name | Latency | Avg iops | Avg MBps | cpu load |
---|---|---|---|---|
Max Throughput-100%Read | 5.86 | 10151 | 317 | 0% |
RealLife-60%Rand-65%Read | 22.44 | 2095 | 16 | 14% |
Max Throughput-50%Read | 5.65 | 10475 | 327 | 0% |
Random-8k-70%Read | 18.04 | 2732 | 21 | 10% |
Is the random IO performance a bit low?