Superguide: Home virtualization server enthusiasts' colorful variety of ESXi whiteboxes

Posted by Paul Braren on Jan 5 2014 (updated on Jan 14 2016) in

(I've not been updating this article since May 2015, because I've been focusing my efforts on a big step up from typical of Intel NUC / Shuttle PCs, moving to 128GB of RAM with the Xeon D-1540, details here)

Let's start with the shopping tools. If you're after which mobo works with which CPU, then pcpartspicker.com and the virtualized computing forum are excellent resources. But if you're after solutions that work well with VMware ESXi 5.5, the going gets a little bit tougher. This guide will help you as you begin to shop around for options. While Dell, HP, and IBM all make plenty of server class gear, the initial cost of the hardware, and the ongoing cost of the electricity and cooling load often means fully redundant gear is overkill. Many home lab builders would rather find an (unsupported) consumer class system they can leave running 24x7, staying under ~125 watts total consumption. Lately, Supermicro and Tyan motherboards with Xeon processors are coming down in price as well, even when loaded up with ECC DIMMs, such as the gear over at ServeTheHome.com.

There are companies out there that make pre-assembled solutions that claim to run quietly and efficiently, such as UK's Brontastor. But what if you really want to dive in and assemble your own system, learning a lot along the way? That's what I did back in 2011, with Z68 Sandy Bridge vZilla build.

Over the past 2+ years, many TinkerTry visitors asked me about newer ESXi home builds out there that folks have blogged about in detail. Of course, I keep an eye on things. I've come across quite a few articles, collected below for your convenience.

What's my overall impression of how Haswell has been going for ESXi builds lately? Folks seem happy overall, with fairly low watt burn, even with the capable GPU that goes largely unused in most home labs.

But honestly, things haven't changed that much since Sandy Bridge, with most of the focus on mobile CPU development. The biggest limitation of Haswell is the 32GB maximum chipset support. It's not that we're waiting for a higher density DIMM for those 4 slots, it's worse than that. The 32GB limit is in Intel's Haswell specs. And some folks report early adopter pains, such as pcdoc's series of Haswell stories, but those sorts of issues are true . See also VMware ESXi 5.5 Free Hypervisor breaks past 32GB RAM restriction and Should I try to upgrade my Intel Core i7 2600 (Sandybridge) past 32GB of RAM for juggling a lot of VMs, or plan to rebuild on Core i7 4770 (Haswell)?

I typically have about 6-8 VMs running 24x7, and those are using up about 90% of my 32GB available. I realize the X79 chipset offered 64GB compatibility. But it's getting too long in the tooth for me to bother with this upgrade to my Sandy Bridge system. Let's hope that the wait isn't too long before there's an affordable option for home virtualization enthusiasts to get past 32GB of RAM, whether that be a Core i7, Xeon, or even AMD processor. So far, it's not looking like the next chipsets, Haswell-E and Broadwell, will have support for >32GB of RAM either, but at least M.2 support is likely (for faster than SATA SSDs). Time will tell. See also related articles about M.2 and Timeline of USB, SATA, and Thunderbolt speeds.

This article is really about collecting just some of the apparent successes already out there, intended to help you be more informed if you're researching your own white-box build in early 2014.

Please remember the fairly narrow focus for this particular post:

Haswell or recent Xeons that work well with ESXi 5.5
well suited for 24x7 use, defined loosely as quiet and economical, using less than <125 watts for the system + internal storage
less than roughly $2000 USD
internal fast storage suitable for day-to-day use (assuming you have the appropriate licensing)
alternatively, for an approach that more closely approximates a datacenter's LAN topology in your home, see Mark Gabryjelski's build in the Ivy Bridge section below
my own focus happens to be more on simplicity of initial set up, which for my lab means:
- no separate DNS
- no managed network switch
- no SQL Server installation

such as my vZilla build outlined in this simple-as-possible guide with video that is generic enough to work for most ESXi compatible hardware. There will be a refresh of those instructions soon, based on the latest 5.5.0b version of the vCenter appliance, things are even simpler now, with no more hosts file tweaks needed on the vCenter appliance, since SSO configuration now works without those tweaks.

If you also have a successful home lab build you'd like me to consider adding to this list, please drop a comment below!

by +Mark Gabryjelski @MarkGabbs on Jan 13 2014
YouTube.com/channel/UCNwtoV_iLvZy1fTcVl5UnUg
With Mark's consent, I put together his build under a single Amazon (TinkerTry Affiliate) Wishlist, just click the Amazon logo at right, with a screenshot below it showing exactly how the shopping cart looks configured with 32GB of RAM and 4 1TB drives.

Presented at Security BSides Boston on May 18 2013, “Build Your Own VMWare ESXi and Microsoft Hyper-V lab at Home, Using Affordable and Efficient Hardware”

by +Paul Braren @tinkererguy on May 18 2013
TinkerTry.com/security-bsides-boston-presentation-may-18-2013

Intel® Atom™ Processor C2000 product Family (codenamed Avoton and Rangeley) is the next generation System-On-Chip (SoC) built on Intel’s 22-nanometer process technology.

Atom (aka Avoton/Rangely) server SoC (System on a Chip) supports VT-d

ESXi and Atom Avoton/Rangeley new SoCs
damiendebin.net, Mar 03 2014

Here's a motherboard/CPU combos with the C2558, the Supermicro A1SRi-2558F, see also Amazon listing:
Supermicro Mini ITX DDR3 1600 NA Motherboards MBD-A1SRi-2558F-O

Amazon commenter Rads confirms this Supermicro Atom C2750 works with ESXi 5.5 just fine. Note that Supermicro hasn't yet stated any of their Atom based servers support ESXi 5.5 yet, at Supermicro compatibility chart.

See also ASRock C2750D4I. Note that nobody seems to be saying they've successfully tested (and afforded) 64GB yet, examples here and here.

Nov 05 2014 Update

William Lam is on a roll with Apple stuff lately, all three of these are an awesome read:
A killer custom Apple Mac Mini setup running vSAN
How to install ESXi 5.5 Patch03 on the new Mac Pro 6,1?
Want to run ESXi on an Apple MacBook Pro, MacBook Air & iMac? #YesYouCan!

Mar 12 2015 Update

Big week in the home server world, with the announcement of the very promising Broadwell-DE, aka, Xeon D. Read ALL about it at this linkfest:
Broadwell-DE announced – Intel Xeon D-1520/1540 crams incredible virtualization lab specs into tiny mITX size

Things are changing!

Superguide: Home virtualization server enthusiasts' colorful variety of ESXi whiteboxes

)

Haswell ASRock Z87 Extreme6 LGA 1150 build

Haswell ESXi Build – Low Power & Low Noise

Haswell ESXi Whitebox

Haswell low power whitebox for ESXi and Hyper-V

Lab Designs

Shuttle PC - vSphere 5.5 White Box Gotchas

The New Haswell Fueled ESXi 5.5 Home Lab Build

Three Example Home Lab Storage Designs using SSDs and Spinning Disk

)

ASRock Z77 Extreme4-M Motherboard works nicely with ESXi 5.0

How to Build A Home Lab for $2,000 - The Presentation

Running ESXi 5.0 & 5.1 on 2012 Mac Mini 6.2

The Shuttle XH61V Ultra Mini vSphere Lab

The Updated Home Lab 2.0

The vTesseract Lab

VMware ESXI Low Power Home Server Lab Build

)

Drezilla – Allan Dresner’s Core i7 X79 Home Storage and Virtualization Server, your thoughts?

Intel X79 whitebox for vSphere 5 and Hyper-V 3

)

Baby Dragon Home Lab

)

Building ESXi 5 Whitebox Home Lab Servers

Haswell ESXi Whitebox

Presented at Security BSides Boston on May 18 2013, “Build Your Own VMWare ESXi and Microsoft Hyper-V lab at Home, Using Affordable and Efficient Hardware”

Superguide: “vZilla” Z68/Core i7/LSI 9265-8i – built for virtualization, storage, and speed

)

8-Core, 32GB RAM, 360GB Flash, 2TB, Dual-NIC Home Lab Part List

ESXi Home Lab Specs

ESXi 5 on MicroServer?

Home Lab (HP MicroServer N40L)

Quad-Core, 32GB RAM, 240GB Flash, 2TB, Dual-NIC Home Lab Part List

Jul 07 2014 Update

Atom (aka Avoton/Rangely) server SoC (System on a Chip) supports VT-d

Nov 05 2014 Update

Mar 12 2015 Update

Support TinkerTry