The reasoning behind vZilla’s use of the LSI 9265-8i RAID Controller (hint: SSD for RAID5 read-and-write caching)

You may recall my article The reasoning behind vZilla’s storage configuration, which dove into why I settled on my particular storage configuration given the number of drives I had on hand. Watt burn consideration played a large part of that decision, since this system is left running 24×7. I’m also spoiled forever with SSD speeds, and I’ve seen what RST SSD caching can do, so I also wanted my RAID5 array to behave more like an SSD. But full ESXi 5.0 support requirement was critical, so this admittedly narrowed down the field of options.

You may also recall my decision to move away from the ESX whitebox favorite, the affordable Dell PERC 5, which is based on an older LSI controller. Going with a several year old solution on a brand new system, and covering up PCI pins with electrical tape just to allow any Z68 mobo I tried to boot, really bothered me.

It was time to build for the long-haul, with an eye toward to a much more modern RAID controller, with:

  • Dynamic volume expansion (expand RAID volume size while keeping all the data)
  • SATA3 support (good for latest wave of SSDs that need it, as demonstrated here)
  • overall speed, stability, and support (LSI has a long history of VMware support)
  • CacheCade 2.0 read/write caching of RAID5 (unique in the industry), a promised 1Q2012 feature
  • Expandability, just in case I ever wind up with more than 8 drives, I can upgrade for $278 USD with the compatible Intel RAID Twenty-four port Expander Card RES2SV240

So I briefly tested an Adaptec 6805Q with maxCache read caching, long before this 6805 versus 9265 article came out. I also tested the LSI 9260-8i and the 9265-8i, and settled on the LSI 9265-8i RAID adapter. Very painful and time consuming to test, fraught with missteps and learning. One example of a learning experience was my use ATTO Disk Bench at defaults:  I forgot to crank up the dataset to 2TB, to avoid falsely-fast cache-only results (the 9265 has 1GB of cache).

I also found that publishing any kind of benchmarks tends to bring on endless, heated flame wars. So, given I don’t have a lot of enterprise level 7200rpm drives, or SSDs, on hand, nor the time to carefully document every single combination of cache settings, this all made the proposition of benchmark testing less appealing. So I’m purposefully chosen to lay quiet on the final numbers I’m getting, not until I get CacheCade 2.0 anyway.

I had a bit of worry, when I went to install the LSI 9265-8i on the initial August 2011 ESXi 5.0 release, and found drive support wasn’t built in, despite being listed on the VMware HCL. I was quite relieved when I figured it out, and wrote up an article on the somewhat complex process.  That article quickly became one of my top 5 read stories on this blog, ever, since this info was non-existent elsewhere:
How to make ESXi 5.0 recognize an LSI 9265-8i RAID controller

Then, with the release of ESXi 5.0 Update Rollup 1, with LSI 9265-8i driver support baked right in, installation became easy, as seen in the video I created and shared this week, seen at Build your own vSphere 5 datacenter , using a Z68 motherboard.

To be fair, it’d be nice if LSI actually supported MSM (MegaRAID Storage Manager) in a VM someday soon, especially since MSM used to sort-of-work in ESX/ESXi 4.1. But those aren’t showstoppers, I’m able to get to the WebBIOS to configure the RAID initially and add SSDs as needed, and that suffices. And I can always dual-boot to native Windows and run MSM from there, in a pinch.

This saga all leads to today, when I read this wonderful, challenging comment/question that came in overnight, from Jay Oliphant:
http://tinkertry.com/vzillacompleted/#comment-419357216

First of all, I have been enjoying following your blog about vZilla, Regarding the LSI RAID controller and SSD caching – I’m assuming this is all handled by the RAID controller itself. Does it do this all at the block level, and move more frequently accessed blocked to the SSD? If the SSD fails, do you lose the whole raid? Or merely just the extra “cached” performance? As far as ESXi is concerned, I assume it can only “see” a single volume presented to it and is unaware of the SSD caching happening in the background. I use ESXi 5 @ home as well, and am looking into potentially speeding up the performance of my home server, as it is currently only a Dell PERC controller with a regular RAID 5 array. 

There’s a lot in that excellent commentary, let me try to tackle this, one or two sentences at a time:

Regarding the LSI RAID controller and SSD caching – I’m assuming this is all handled by the RAID controller itself.
Yes, ESXi 5.0 doesn’t need anything beyond built-in drivers in the hypervisor to see the volume on the array, it doesn’t know or care if it’s cached.
FYI, I did buy the Battery Backup Option, as listed on tinkertry.com/vzilla and seen wire-tied externally (where it’s kept cool) at tinkertry.com/vzillacompleted.

Does it do this all at the block level, and move more frequently accessed blocked to the SSD?
I don’t know for sure, I’m hoping to interview somebody from LSI to find out! I would guess that you’re right, I’ve read nothing to indicate that it “knows” filesystems, so it seems it must be block level.

If the SSD fails, do you lose the whole raid? Or merely just the extra “cached” performance?
Just the cache goes away, I tested removal of the SSD (physical and logical), using my older 96GB SATA2 Kingston SSD, and the data on the RAID0, RAID50, and RAID5 arrays that I tested remained intact.

As far as ESXi is concerned, I assume it can only “see” a single volume presented to it and is unaware of the SSD caching happening in the background.
Yes, it’s totally unaware, and that’s the beauty of it, no special software-based support is needed, in contrast to the much-more-affordable but software-heavy Intel RST technology (that does work well for Windows, as I demonstrated here). This means this LSI RAID should also work nicely for native Hyper-V installs too, although that’s not my priority.

So, to sum things up, I’ve pinned all my hopes on the CacheCade 2.0′s read/write caching that the 9265-8i will offer 1Q2012, and will not offer for the less pricey 9260 series. I’m sure hoping LSI makes good on that promise, and that the speed is good. I briefly tried the CacheCade 1.0 hardware dongle, but with older SATA2 SSDs, the speed boost wasn’t worth the price, so I returned it, saving up for the CacheCade 2.0 FastPath bundle instead. I’m hoping the cost will be under $300 or so, based on similar products already listed.

FYI, I discussed CacheCade 2.0 in detail, way back in September 2011 here:
tinkertry.com/goodraidcontrollerswithssdcachingandesxsupport

If the CacheCade 2.0 never comes out for the 9265-8i, I’ll certainly look like quite the chump, and will have overspent considerably. But my speed with current my no-SSD-cache-yet RAID array is sufficient.

You could say that I’m already putting my eggs into this LSI RAID basket, trusting some of my VMs already to this array. But I always keep backups. And I’ll certainly have full backups, before I finally get to try CacheCade 2.0.

I’m intentionally holding off on buying the SSD, until CacheCade 2.0 is actually available, since prices continue to fall, and SSD controller chips and firmware levels rapidly mature.  My hope is that I’ll be able to get a best-in-class SATA3 drive, with a budget of roughly $200. Ideally, I want an SSD with automatic TRIM-like BCG housekeeping, since RAID attached SSDs don’t support TRIM commands, and I want the cache to maintain performance over time.

A RAID adapter with both proper UEFI support (instead of a slow BIOS routine), and a battery-less (supercapacitor/ultracapacitor) design would have been preferred, but I still don’t see such adapters on the market, and it’ll probably be quite a while before those RAID adapters are mature, with full ESXi support, time I didn’t have for this project.

At month 7 and counting, I feel reasonably confident I’m finally nearing the end-zone. A perfect touchdown would be for vZilla to soon have a best-in-class RAID5 array, with SSD-like speeds, or at least speeds that are far greater than any similarly-priced NAS storage vendor.

A little more time, and testing, will help me finish this tale. Stay tuned!


Share Button

written by

The author didn‘t add any Information to his profile yet.

7 Pingbacks/Trackbacks

  • Jay Oliphant

    Wow, this is an excellent write up! Thank you for your response. I will definitely start looking into moving from the PERC controller and going to an SSD caching unit. Keep up the great work, and I look forward to continue following your site!

  • tinkererguy
  • tinkererguy
  • http://twitter.com/welchwerks Greg Welch

    So im not alone :)
     
    [quote] One example of a learning experience was my use ATTO Disk Bench at defaults: I forgot up the dataset to 2TB, to avoid falsely-fast cache-only results (the 9265 has 1GB of cache).[quote/]

    I did the exact thing, even posted the finding in HSS forums.
    my 9260 4i has the 512 cache

  • http://twitter.com/welchwerks Greg Welch

    what do you think about this for a raid expander , has an lsi controller

    http://www.astekcorp.com/serial-attached-scsi-expanders/a33606-pci

  • Jay Oliphant

    Is CacheCade 2.0 out? I found this on Newegg - 
    http://www.newegg.com/Product/Product.aspx?Item=N82E16816118164  

    • tinkererguy

      Jay, CacheCade 2.0 is out, but only for the 9260/9280 series, with the
      LSISAS2108 RAID-on-Chip (ROC)
      which can support read/write caching of RAID0, but not RAID5.

      The 9265-8i has the
      LSISAS2208 Dual-Core RAID on Chip (ROC)
      and will be able to do read/write caching of RAID5.

      But after waiting for many months since announce of CacheCade 2.0, the version I’m waiting for is still not even on the LSI store site yet:
      http://store.lsi.com/store.cfm/Advanced_Software_Options/

  • tinkererguy

    Hmm, I’m not sure, I haven’t tried that model, or found a list that shows it’ll work.
    Sorry I missed spotting this comment Greg, hope all is well with you!

  • Tom Daugherty

    I just called LSI support to ask about the release timeline for CacheCade 2.0 with write cache support.  He said, unofficially, they’re looking at a March release.

    I have been using the 9265-8i controller with hardware dongle for 6 months or so, works pretty well. The SSD cache is made of 2x80GB SSDs RAID0.  I haven’t found (or looked) for a way to check cache hits for the read cache. But subjectively I feel it makes a big difference.

    • tinkererguy

      Thank you for the great news, I really appreciate your taking the time to share this helpful info.  I suppose I won’t totally believe CC Pro 2.0 until I get to try it myself.

      Are your 3.5″ drives in a RAID5, and how many drives do you have?Have you tried ATTO DiskBench on your configuration?  If so, what kind of results are you getting (with “Total Length” set to 2TB)?  Did you go with LSI FastPath as well?

      • Tom Daugherty

         I’m using 6 3TB 3.5″ (sata3) drives in raid6. Two 80GB SSDs (sata3) in raid0 as the cachecade cache. I have a 7th 3tb drive as a cold spare. Drives are from 4 different lots, to spread failure rates.

        I’m not using the fastpath. I didn’t think it would do anything since the SSD aren’t direct access, but are only acting as cache.  Maybe I should follow up on that.

        I haven’t run DiskBench. I might as well try that now and see how it looks. Will post results when it’s done.

      • Tom Daugherty
      • Tom Daugherty

         ATTO Diskbench results.

  • Jean

    Got my own vZilla this week and found out the drive encryption with the LSI 9365-8i isn`t included in the price of the card. It seems to require that one buys a $90 `SafeStore` key from LSI plus significantly more expensive self encrypting drives (SED). I was sure you mentioned somewhere on your site – although I can`t seem to find it now – something about liking that the LSI controller protected your data if your system was stolen. Also the vZilla parts list mentions LSI card `with hardware encryption` while the drives were ones you `already had`. Are the drives you have SEDs? Did I / am I misunderstanding the requirements for or level of data protection from the LSI 9265-8i?

    Any info would be appreciated.

    • tinkererguy

      Excellent point, I forgot to point out that only the 9260-8i with the hardware CacheCade key seemed to allow drive password protection (had to key in password into 9260 at boot time to unlock RAID array). But I no longer have that device.

      With the 9265-8i I’m now using, it’s unclear if I’ll be getting the password capability back, once I do finally get CacheCade 2.0:
      http://store.lsi.com/store.cfm/Advanced_Software_Options/

      Thank you for pointing out my missing this crucial info!

    • tinkererguy

      Excellent point, I forgot to point out that only the 9260-8i with the hardware CacheCade key seemed to allow drive password protection (had to key in password into 9260 at boot time to unlock RAID array). But I no longer have that device.

      With the 9265-8i I’m now using, it’s unclear if I’ll be getting the password capability back, once I do finally get CacheCade 2.0:
      http://store.lsi.com/store.cfm/Advanced_Software_Options/

      Thank you for pointing out my missing this crucial info!

  • Tom Daugherty

    FYI on CacheCade 2.0

    The
    release for CacheCade PRO 2.0 for the LSI MegaRAID 9265-8i and 9285-8i
    has been pushed back to the end of March, 2012. However, this release
    date is still tentative and subject to change. You can get the latest
    updates about product releases and support from LSI.com or follow us on twitter at @LSIStorage and @LSISupport. Thank you for contacting LSI technical support! Please, don’t hesitate to contact us for further assistance. Kindest Regards, Scott DykesTechnical Support EngineerRAID Storage Division

  • Tom Daugherty

    FYI on the CachCade 2.0 release. Thought I’d pass it along.The
    release for CacheCade PRO 2.0 for the LSI MegaRAID 9265-8i and 9285-8i
    has been pushed back to the end of March, 2012. However, this release
    date is still tentative and subject to change. You can get the latest
    updates about product releases and support from LSI.com or follow us on twitter at @LSIStorage and @LSISupport. Thank you for contacting LSI technical support! Please, don’t hesitate to contact us for further assistance. Kindest Regards, Scott DykesTechnical Support EngineerRAID Storage Division

  • Pingback: Paul Braren’s “Build Your Own VMware ESXi and Microsoft Hyper-V lab at Home, using Commodity Hardware” presentation replay from Saturday March 10th | Tinkertry()

  • ascoolasice79

     I’m also interested in building my own ESXi. My searching about this whole topic brings me to your blog and it is so usefull to me. Thanks for your work!

    I want to use local storage and therefor I’m looking for a good RAID-Controller. I think the brand new LSI 9266-4i is a good choice for six Drives (WD Green) in RAID 5 + one SSD. Perhaps you can help me in explaining where the difference between the Fastpath, the Cachevault and the Cachecade Option is?!

  • Pingback: Sandybridge Z68 versus Ivy Bridge Z77 motherboards, LSI 9265-8i versus 9266-8i RAID adapters: would I change how I’d build vZilla this year? | Tinkertry()

  • dresner

    Even with the new version coming out, I’m still considering the LSI 9265-8i because of its baked in support in ESXI 5u1.  I’m looking at building 2 RAID 10 arrays on the card, 1 made up of SSDs and one of Mechanical WD RE4 drives.  Interestingly, when you talk about the cache cade 2, and really hoping it is released for you!  I’m looking at a feature in ESXi 5u1 that allows you to select host cache location on an SSD.  So can’t you just plug in an SSD on your card or mobo and use that for your cache?  SEems like you can also put your VM swapfile on a separate drive as well. 

    • tinkererguy

      Thank you for letting me know about the new firmware now out for the 9265/9266!
      http://www.lsi.com/support/products/Pages/MegaRAIDSAS9265-8i.aspxRead me for 23.4.1-0028 says:This is a major product refresh with numerous bug fixes and enhancements, including adding CacheCade Pro 2.0 and CacheVault support (if enabled).By the way, Host Cache Configuration is more a way to reduce memory pressurehttp://www.tinkertry.com/vsphere5hostcacheconfiguration/ when over-committing memory (such as running a lot of VMs that are actively using 32GB, I’d start hitting swap on a spinning disk, so this is a way to boost physical ram with SSDs used as ram).  Not the same as caching the RAID5 array for all reads and writes that I’m really seeking.  Testing will see how it goes! 

      • Tom Daugherty

         Have you heard anything about CacheCade upgrades to Pro 2.0?  I have a physical key for CacheCade, I’m thinking it’s just for the old version. I’ve upgraded the firmware etc, and still don’t have CacheCade 2.0 or write-back.

        • tinkererguy

          No, haven’t heard any hints they’ll be an upgrade path at all, I’m afraid, and while the firmware will support CacheCade Pro 2.0, I still need to buy it (hardware key or software key, I’m not sure yet).

          • Tom Daugherty

             Just heard from LSI support. There is no upgrade path for CacheCade 1 to CacheCade 2.  I just bought the 1.0 in October with the understanding, from LSI themselves, that we would be upgraded to 2.0 when it came out.  So those of us who bought this setup have to spend another $280 to upgrade.  Good times.

            I’ll let you know when I have the write cache working with the cachecade 2.

  • Pingback: LSI 9265/9266 CacheCade Pro 2.0 information | Tinkertry()

  • Pingback: LSI FAQ about CacheCade Pro 2.0 / FastPath may be coming soon, meanwhile, here’s the 30 day trial keys, and the open questions… | Tinkertry()

  • Vin

    Excellent post. What was your feeling on the adaptec 2405 for ssd caching and raid performance. I’m building at the mo and looking for an adapter to suit 2 x 2tb sata3 disks plus 1 x 120gb 6gbs ssd.

    • tinkererguy

      Well, the 2405Q would be more of an apples to apples comparison, and I did kick the tires on that card, but at the time, heard nothing about RAID5 read and write caching coming to that piece of gear  http://www.adaptec.com/en-us/products/controllers/hardware/ssd-caching/entry/sas-2405q/

      Your few spindles would work fine, unclear, are you trying to use the SSD as cache for a RAID0 perhaps?

  • Pingback: Tinkertry’s First Birthday, June 1, 2012 | Tinkertry()

  • Pingback: TinkerTry IT @ home | Presented at Connecticut VMUG User Conference on May 14 2013, “Virtualization at home can be practical, affordable, and available, 24×7.”()

  • Pingback: TinkerTry IT @ home | Presented at Security BSides Boston on May 18 2013, “Build Your Own VMWare ESXi and Microsoft Hyper-V lab at Home, Using Affordable and Efficient Hardware”()