Good RAID Controllers with SSD caching and ESX support

Here’s the original title of this article:
“RAID controllers with >2TB drive support, 6Gb/sec (SATA3) support, and various levels of SSD caching:  results of my taking both the LSI 9260-8I and the Adaptec 2405Q out for a spin”

Kind of a mouthful, eh?  Hopefully my new title is both more concise and intelligible. The title is the easy part. This is going to be a rough/deep ride through the quagmire of new RAID technology intricacies.  Hang in there, this epic journey is not for the faint of heart!  Making a reasonable cost, large storage array that also performs well is a challenge.  Ideally, I’d like to serve up some speedy NAS style workload (NFS, iSCSI, CIFS, etc.) to all my lab machines.

So, with this post, I’m finally beginning to write up some detailed information about my recent tests of the Adaptec 2405Q with maxCache, and the LSI 9260-8i RAID controller with CacheCade.  Both are SSD read caching technologies.  I guess you could say I caught the bug when I was using a Seagate Momentus hybrid drive this past winter, thinking, wouldn’t it be nice if my RAID arrays performed more like my affordable and large hybrid drive?  And was operating system agnostic, that is, didn’t require Windows-only software to run?

My posts about this RAID shopping began way back in May 2011, over at this spot on the extremely helpful homeservershow.com/forums.  Since ESX 4.1 Update 1 doesn’t have Intel Z68 motherboard SRT (Smart Response Technology) support with SSD caching, I’m seeking alternatives to speed up the performance of all my VMs.

I’ve always found testing RAID performance to be complex and time consuming process, especially when you sprinkle in my desire to try out the latest and greatest techniques of making a RAID array perform more like a hybrid hard drive.  Think of a cross between SSD and a traditional spinning platter RAID arrays, taking the best of both worlds and mushing them together.  In a perfect world, this would get you speed, capacity, and resistance to data loss from inevitable drive failures.

Notice, I stated I want my choice to be VMware ESX compatible, which really limits choices even further, as seen over at the VMware Hardware Compatibility Guide (which doesn’t yet have ESX  5.0 info).  I don’t wish to dedicate this storage chunk to just one system or virtual machine, and with ESX 5.0′s support for >2TB volumes, I’d like to manage this big lump of roughly 10TB of storage with ease. I know I’ll change my mind many times on how to carve things up over the next 4 years of expected life of this system.

And just to make the picture even muddier, I’m also very interested in the upcoming “Host Cache Configuration” setting in ESXi 5.0, but that appears to be no substitute for true caching of a VM’s “hotspots.”  Playing with that software cache will have to wait, and there’s very little info out there yet anywhere Googling , and of course, vSphere 5.0 (with ESXi 5.0) isn’t even shipping yet.

So, back to the here and now.

August 9th 2011, a bunch of new firmware/drivers/MegaRAID software releases hit the LSI site, requiring a new set of RAID5 speed testing for me.  I’ve been on the April firmware, and am  not getting consistent results in Windows or ESX, so I’ll hold off until I retest before I publish more.  Meanwhile, here’s the most interesting pieces of that recent news from LSI I just spotted today:

LSI is claiming read/write SSD caching of RAID arrays, finally!   Even better, I own the card that is eligible for this free upgrade for CacheCade 1.1 owners like me (I paid $226 for 1.1 in April, and already placed my order for 2.0).  Here’s a little diagram to give the gist of what I’m talking about here, and a whole bunch of background links to enjoy/ponder:

LSI MegaRAID CacheCade gets write caching capability
LSI MegaRAID® CacheCade™ Software 2.0 Video Overview
MegaRAID CacheCade Pro 2.0 Software
LSI MegaRAID CacheCade Pro 2.0 Caching Software Technical Brief
MegaRAID® CacheCade Pro 2.0 Software FAQ

The SSD Review:
LSI MegaRAID CacheCade Pro 2.0 Review – Enter Write Caching
LSI MegaRAID CacheCade Pro 2.0 Review – Basic Concepts and Application
LSI MegaRAID CacheCade Pro 2.0 Review – Real World Results and Conclusion

At first, I was exited at the idea of CacheCade 2.0, since I already have a 9260-8i.  But LSI claims to be the first vendor to offer this, hmm, I’m used to being on the bleeding edge, but not necessarily with my treasured storage, at least until I have 2 or 3 copies of everything critical anyway.

So I dove in deeper to all those links, and noticed that for the new read/write caching that this CacheCade Pro 2.0 offers, there’s this BIG gotcha here:

– Cached write data protected by non-volatile CacheCade cache pools (RAID 0, 1, 10) and data availability protected by RAID data redundancy

– RAID 5 in upcoming release (9265 Series only)

So that means you may need a lot of SSDs, so I may need to re-visit exactly how LSI’s flavor of RAID10 caching works.  But I’m finding no new MegaRAID instruction manual yet, that’d be too easy.  The issue with multiple SSDs is cost of course, my RAID5 config in my vZilla build has 7 (2TB) drives and 1 (96GB) SSD drive.  Time for even more research.

I will make updates right back here in this post, when I learn something significant, or have some speed tests of the 9260-8i with the new LSI August firmware/drivers.

Footnote:  I no longer own the Adaptec 2405Q, 4 ports was simply too restrictive, and port multipliers were not compatible with it.  Yes, I realize there are other higher end Adaptec controllers out there too, always eager to hear what others with similar goals have found, please use Disqus comments below…

Update Sep 09 2011 :  Nice new article spotted here:
LSI MegaRAID CacheCade Pro 2.0 Review – Test Bench and Protocol


written by

The author didn‘t add any Information to his profile yet.
  • http://tinkertry.com tinkererguy

    Much better results with the LSI 9260-8i using the August 2011 drivers under Windows 7 x64, with the August firmware.  This is looking promising

    RAID50 with CacheCade 1.1:  ATTO Disk Benchmark average reads 250MB/sec, write 400MB/sec

    RAID5:  still testing

  • http://tinkertry.com tinkererguy

    RAID5:  worse performance with August firmware/drivers, this is strange.

  • http://tinkertry.com tinkererguy
  • http://tinkertry.com tinkererguy

    Moving LSI to a different PCI slot seems to have made a big difference in speed, stay tuned…

  • Gregwelch

    i went with the 9260-4i, hoping the ssd cache can be attached anywhere :)

  • Greg Welch aka welchwerks

    also i already agree with your last comment that 4 ports are limiting :(

  • http://tinkertry.com tinkererguy

    Here’s somebody who got the 9265-8i controller working with ESXi 5.0, nice video!
    http://www.youtube.com/watch?v=mEBwt6Q_diU

  • http://usb3gvn.com/ USB 3G

    Oh nice, thanks for your
    information!

  • tinkererguy

    1Q2012 is when CacheCade 2.0 is due (read and write caching of RAID5), but only on the 9265-8i (not the 9260, unfortunately).

  • 49thnorth

    Have you looked at Nexentastor? I have had great results with it as it uses a ZFS file system cabable of utilizing SSD as cache. Accessed via VMWare NFS. Take  a look at the community edition.

  • 49thnorth

    Have you looked at Nexentastor? I have had great results with it as it uses a ZFS file system cabable of utilizing SSD as cache. Accessed via VMWare NFS. Take  a look at the community edition.

  • http://twitter.com/welchwerks Greg Welch
    • tinkererguy

      Awesome, can’t wait to hear how it goes!

  • Audiblackrs4

    cachecade firware kills the power management for any configured drive

  • Nick K

     I am looking at options build an ESXi host, either one along the lines of your vZilla or possibly going the LGA2011 socket route a la 
    http://www.ivobeerens.nl/2012/03/13/intel-x79-whitebox-for-vsphere-5-and-hyper-v-3/

    My main concern is storage, I am having trouble getting my head round what I will need for my build!

    I don’t need masses of storage, at the moment I have a file server (2003) which has 4 x 500gb drives giving me 1TB mirrored (I’m not sure what RAID setup it is!) which is accessible over my domain. I guess I’m looking for at least 2TB storage that has the redundancy RAID provides and a VM to serve this data exclusively to the domain (pass-through?)

    The RAID card you used is a little expensive for my budget (at the moment), some Googling and I found someone using an Adaptec 2405, could you give me an idea of the con’s of going with a cheap RAID like this?

    Any thoughts on what I need to think about would be appreciated!

    Many thanks,

    Nick

    • tinkererguy

      For me, I may want Thunderbolt someday, passed through to a VM, and this ASUS motherboard link you provided has somebody commenting that VMDirectPath doesn’t work.  For most folks, that probably doesn’t matter at all.

      Another consideration is that the free VMware Hypervisor only supports 32GB maximum, so my 4 DIMM slots don’t hold me back either:
      http://www.tinkertry.com/32gb-memory-on-asrock-fatal1ty-z68-professional-gen3-motherboard/

      Finally, I have built in graphics, so one less PCI slot to take up, which I’ll need for Thunderbolt or FC or other adapters down the road.

      So everybody has different needs, of course, new systems may work fine with ESXi for your needs, hard to say until somebody tries that exact (unsupported by VMware) hardware.

      As far as RAID, yeah, had the Adapter 2405 or other similar cards may work fine, but you should check VMware’s HCL to ensure compatibility, if you do think you may just want to have VMware VMFS filesystem on the RAID, free to share with all your VMs, rather than just assigned to one (passthrough, which requires VMDirectPath motherboard/CPU combo).  My needs are admittedly different, as I’m seeking SSD caching of reads and writes, outlined here:
      http://www.tinkertry.com/whylsi9265-8i/

      I hope this helps you somewhat, without confusing you.  Any information is good information, as you try to make informed decisions for your own needs.

    • http://TinkerTry.com/about Paul Braren

      For me, I may want Thunderbolt someday, passed through to a VM, and this ASUS motherboard link you provided has somebody commenting that VMDirectPath doesn’t work.  For most folks, that probably doesn’t matter at all.
      Another consideration is that the free VMware Hypervisor only supports 32GB maximum, so my 4 DIMM slots don’t hold me back either:http://www.tinkertry.com/32gb-memory-on-asrock-fatal1ty-z68-professional-gen3-motherboard/
      Finally, I have built in graphics, so one less PCI slot to take up, which I’ll need for Thunderbolt or FC or other adapters down the road.
      So everybody has different needs, of course, new systems may work fine with ESXi for your needs, hard to say until somebody tries that exact (unsupported by VMware) hardware.
      As far as RAID, yeah, had the Adapter 2405 or other similar cards may work fine, but you should check VMware’s HCL to ensure compatibility, if you do think you may just want to have VMware VMFS filesystem on the RAID, free to share with all your VMs, rather than just assigned to one (passthrough, which requires VMDirectPath motherboard/CPU combo).  My needs are admittedly different, as I’m seeking SSD caching of reads and writes, outlined here:http://www.tinkertry.com/whylsi9265-8i/
      I hope this helps you somewhat, without confusing you.  Any information is good information, as you try to make informed decisions for your own needs.

  • http://www.eoptionsonline.com/ Albert Gray

    Quite Impressive Configuration.You will also get a nice and well configuration of Hp controller products.