$10K iSCSI SAN for production environment (dedicated device or Software SAN)?

[pacmantravis]

We're looking into getting a new iSCSI SAN for our production website which will be hosted on esxi 4. Currently the setup consists of 2 physical servers (a dedicated IIS server and a dedicated SQL server for the website). The site is a CRM site that we provide to our clients and it has approximately 150-300 users on it at any given time. As of now the two servers are set up in basic RAID 1 arrays (146GB 15k x 2).

We're virtualising them to increase availability, scalability and hopefully performance. We're pretty familiar with esxi as that is what we are running in our "in-house" environment. We currently have 12 VM's running on 3 hosts (DRS, vMotion and HA enabled) which are stored on a Thecus NS8800. Surpisingly enough, the Thecus is actually doing very well and the iscsi performance has been very nice. However, we know that the Thecus is not a production system, so we would like to add another SAN to the setup for our more mission critical stuff.

Our biggest priorities in the new SAN are: Performance (for SQL), redundancy, and reliability.

We've been looking at the Dell MD3000i, IBM DS3300, Sun 2510 and others. It seems like when these are fully configured they are just right at or just above our budget. Also, I know that Dell and IBM firmware lock their devices to only only their hard drives, does anyone know if SUN or other mfr's do the same? Also, does anyone have any suggestions on what other SAN's I should be looking at? How do the iStor iS512 and the Jetstor 516iS compare?

Since we don't see ourselves needing TB's and TB's of data in the near future, another route we've been thinking about is getting a used PowerEdge 2900, filling it up with disks and nics and using a software like Falconstor, Datacore Sanmelody or starwind 5.0. However, a BIG issue with that is that unless I am mistaken, all of those programs sit on TOP of Windows which means that we have to deal with update reboots, etc.

Any advice is greatly appreciated! Thank you!

[Mindflux]

If you are open to a different platform you might give a SAN with NexentaStor a try (Solaris kernel, debian userland). You can run everything off a ZFS filesystem, which will allow snapshots and all sorts of stuff without extra licenses.

One of their partners is pogolinux, if you get in touch with Paul Bibaud over ther and he can give you some quotes based on your criteria. But just to give you an example I'm eyeing 2 Storage Director Z350's with 8TB of space (16x250GB) and 2 Intel X25-E SSD's in a mirrored config for the Nexenta OS, 24GB of RAM and a single Xeon 5504. This with redundant PSU's, redundant SAS controllers and all is something like 11k (each) WITH a CDP Plugin ($1,900), So if you don't want AutoCDP (block level duplication to another SAN)that'll put you under 10K with the config I'm eyeing.

You mentioned not needing wads of space... so you could even cut that back further or go with the Storage Director Z245/250. He can definitely get something within your budget.

[pacmantravis]

I've actually heard of NexentaStor and am very familiar with ZFS. However, is Nexentastor supported with ESXi 4?

Thanks for letting me know about pogolinux as well.

Can I ask why you decided to use 24GB of RAM though? I know that ZFS likes RAM, but isn't 24GB a bit of overkill for just storage?

[Mindflux]

It was what was recommended to me by Paul Bibaud. I had it configured with 12GB and he said to up the RAM. This is what he told me:

"You will need to boost the cpu and memory a little bit. These systems use the main compute subsystem for management of the array and it's data. In order to get efficient scrubbing of your data, as well as, good read performance, it is best to have as much memory as your budget will allow."

If I had to guess it's because ZFS doesn't use RAID controllers with huge cache sizes. I don't know enough about it either, but with RAM as cheap as it is 24GB is like $911 list at pogolinux on the box I'm looking at. That's just a couple hundred more than a decent RAID controller.

From what I understand the iSCSI target/initiator with Solaris works with VMWare. Nexenta also has a VMWare plugin ($1,900) to manage your servers from Nexenta, but there is some verbage about the iSCSI Datastore not working in ESX >= 4.0..seen at the bottom here:

Here's a document on VMWare/NexentaStor:

http://www.nexenta.com/corp/component/option,com_remository/Itemid,77/func,startdown/id,30/

[kopper27]

what if you buy server with a good controller(could be RAID 1 for openfiler and RAID 1 or 5 for Data) and get Openfiler as a Software SAN and configure it as iSCSI

I've heard it works pretty well

and BTW you install OpenFiler installs like a normal linux OS on the server it's based on linux so you will avoid the overhead using Software SAN that runs over Windows and Openfiler is free.

A friend of mine had a very bad experience using StarWinds (you have to paid and it runs on windows)

www.openfiler.com

[pacmantravis]

I've messed wih OpenFiler before but wasn't too impressed with it. Plus, I don't believe it's really production ready for an ESXi environment.

Also, while ZFS is pretty nifty, we are running a pretty high transaction SQL Database and I dont feel that RaidZ would be the best for that type of situation. Ideally we are going to want to have 3 different partitions for just out SQL server. 2 Raid 1 partitions (1 for OS/TempDB and 1 for Logs) and then a RAID 10 partition just for the DB itself.

The iStor iS512 looks pretty interesting to me as it has virtual raids which means we wont have to waste space when configuring our raid volumes.

So many decisions!

EDIT: Just remembered that it's possible to create a mirrored storage pool (basically RAID 10) within ZFS

[dilidolo]

This is why you need SLC SSD as write cache (ZIL) for your zpool. As for RAM, it's used for read cache, but again, you can use MLC SSD as read cache (L2ARC). RAIDZ is not best for performance but you can use RAID-10.

BTW. ZFS will get deduplication in about 2 weeks so you can use higher rpm but smaller disks for your VM environment. However, with SSD as ZIL, disk rotation speed is not as important as before. Also ZFS supports IO compression.

I use Solaris as my storage server for my VMware lab, using NFS, iSCSI and FC protocols, works very well. I have 4 old 10K SCSI disks in Raidz, with Intel x25-e as ZIL.Through 4Gb FC, I get 330MB/s read and 280MB/s write average inside VMs, IOPS is also very very high. For iSCSI with multipathing (2 gb nic), 250MB/s. I have vCenter and SQL 2005 VMs, it actually feels much faster than my old physical P4 machine.

Then it really depends on how much you know about Solaris ZFS/Comstar, I'd say use whatever you know best.

[pacmantravis]

Hmm, the thought of SSD does intrigue me.

However, I see that you said you are using it in your lab environment...would you use your current set up in a production environment?

[dilidolo]

If I have proper hardware, then yes. But it depends on how much you know about the system for troubleshooting.

It's still relatively new and there are bugs, but it's a single purpose system, once configured, it's very hard to mess it up.If you want, you can buy support from SUN.

[pacmantravis]

It looks like a Pogolinux/Nexentastor system is definitely going to be an option for us.

Also, I noticed that Sun has a "startup" program that is giving 40%+ discounts on their new 7000 series units. I like that 7110 and if we can get it at discount, it may be the way to go for us. Only thing missing from it that I can get with Pogolinux are the SSD's.

[dilidolo]

The company I worked for bought 2 Nexentastor, they told me their tech support was very poor.

If you can get SUN 7000 under budget, I'd definitely go with it. The write optimized SSD in SUN 7000 is basically Intel X25-E. You can use X25-E for write cache and X25-M for read cache.

[Mindflux]

I saw that Sun StartUp program last night while googling for something. I'm guessing your company would have to BE a NEW StartUP though? We've been in business over 15 years so I don't think we'd qualify. (We're shopping for SANs too)

I just stumbled across this: It's the $15,995 7110 (4.2TB) sold elsewhere:

http://pc.pcconnection.com/1/1/660235-sun-microsystems-storage-4-2tb-7110-unified-storage-system-pro...

Not a bad discount for digging through google.

[Mindflux]

I've never heard positive nor negative about NexentaStor's support. In fact it's hard to find anyone really mention much about them. Pogolinux's support for me on my servers has been great, so I'd hope at the very least they'd be a decent fallback. They are NexentaStor's largest reseller so I'd hope they'd be able to kickstart somebody's brain over there to get you appropriate support.

[pacmantravis]

Unfortunately the entry level 7110 does not come with any SSD's. We'd have to step up to the higher 7210 Model with SSD's for that and that is in the $30K+ range.

The 7110 from Sun is $10,995 for all 146GB Drives or $15,995 for all 300GB drives.

The only "negative" I see is that it does not have redundant controllers...the Sun 2510 does have storage controllers though, so I dont know.

[Mindflux]

Yeah the lack of controllers is kinda boggling.

It sure seems like a nice box though.

[pacmantravis]

Anyone with experience with any of the units that have been mentioned?

I got a pretty fair quote for an MD3000i full of disks and was wondering how that compated to say a Sun 7110 or a Pogolinux Z250 with Nexentastor and 2 SSD's

[Mindflux]

The quote I got on an MD3000i was near 12k (with tax).

There are plenty of folks using MD3000i's on here if you search. They must be doing something right.

If you qualify you can get 0% for 36 months on the MD3000i's through Dell as well.

Please keep in mind if you ever want to add another SAN for block level replication, the MD3000i cannot do it.

Would you mind listing your MD3000i specs and price?

I have a reseller quote from someone else for a couple grand less than Dell's direct quote. Pretty shocking.

[pacmantravis]

I dont want to go into too much detail on pricing, but it was basically dual controllers, 8 x 146GB SAS and 4 750GB SATA for roughly $8500

[Mindflux]

Hrmm.. that's not TOO bad.

Their page is acting up right now, but you should check out this guy:

http://serverconsultants.net/

He has 'ready to go' hardware from Dell with 3 year warranties FROM Dell. I know he's got at least 3 MD3ki's. His price was about 2 grand less than what Dell had quoted me for the same box.

[current99]

The MD3000I is quite dumb, i.e no replication options or value added options of any use.

Nexenta now do gold support they started this about three months past was not an option before.

Nexenta and SUN have unlimited snaps and replication built in, with zfs dedup due soon.

Falconstor is Linux baised and is Enterprise class S/W with every option availiable you could think of

with regards to Application aware agents, but it may be out of your budget buy the time you add it all up

with Hardware.

Alan