How to use a CyberPower UPS solution for proper automated shutdown of your VMware ESXi lab during power outages, including all VMs

Posted by Paul Braren on Nov 27 2012 (updated on Apr 22 2019) in

This article has been popular for many years right through 2019, and it still works right up through vSphere 6.7 Update 2 and PowerPanel Business Edition Agent for Virtual Machine version 4.0! The latest info has been appended via detailed updates below.

Introduction

CyberPower-from-Amazon — CyberPower PFCLCD Series available in 4 capacities at Amazon.

Shutting down my old Windows Server v1 system was pretty simple, done by attaching the USB 2.0 signal cable from the UPS to the server, installing software, then attaching the UPS. Worked great, for years. Power goes out, software triggers graceful shutdown. Data protected, safe.

Shutting down my new precious Windows Server 2012 Essentials VM is pretty simple too, done by attaching the USB 2.0 signal cable from the UPS to the server, installing software in the VM, then attaching the recognized CyberPower USB device to the VM. Also works.

But what about a better way? A method that'd be able to handle VMs running under ESXi 5.1 (through 6.0), automatically. No need to install anything in each of the VMs, other than VMware Tools. No need to turn on SSH access to ESXi. Using an affordable CyberPower UPS (PFCLCD series go for as little as ~$118 USD), with user replaceable batteries, and a free VMware-ready software bundle.

Backstory

I began my yearning for an elegant, easy-to-set-up, robust, and automatic way to shutdown ESXi during power outages way back in the spring of 2011. I had done some research before buying my UPS, covered here, choosing a UPS that was compatible with my efficient vZilla PFC power supply, without any disturbing/worrisome buzzing power supply issues (during power outages).

I had used a variety of APC and Tripp Lite products for decades, but wasn't as familiar with CyberPower UPSs. I had discovered that my older/cheaper APC UPSs couldn't stay running with any generator brands I tried, clicking and turning off (read related story here), so I was especially open minded about giving CyberPower a shot. Especially since poking around their site seemed to indicate some sort of VMware support.

CyberPower's various sizes of UPSs in their series of affordable, consumer line of products can be seen on Amazon and NewEgg. I chose the CyberPower CP1350PFCLCD PFC Compatible 1350VA 810W Pure Sine Wave Tower UPS at $169, with free shipping (Amazon Prime). And I quickly became fond of the easy to use, always available (even if the PC is not) LCD display, with the simple way to mute alarms, described and partially demonstrated here. So I went and bought 2 more similar units, going with the same exact model for my home entertainment center, and a smaller CP850PFCLCD for my home office (pictured in my office here) at $112.92, more than adequate for my laptop, 3 monitors, and an Ooma VOIP device, mostly just to gracefully handle fairly frequent power blips. And the UPSs batteries are replaceable

My efforts at ESXi shutdown have mostly been around failed tests of a variety of scripting techniques that I didn't particularly enjoy trying to get working, under a wide variety of OSs. Lots of scripting prerequisites to install as well. Surprise gotchas like "doesn't work with free Hypervisor, because it relies on vCLI." Lots of dusting off my my limited VI editor / Linux skills.

Basically, I had stumbled over numerous barriers that I really felt weren't worth sharing, as they were so complex and problematic, that I knew the likelihood of folks actually doing this sort of thing for themselves was slim. There was also the very real potential price issues, given VMware compatibility of UPSs tends to be priced as an "enterprise" feature.

All I really wanted was something far simpler. Ideally easier than existing vCLI techniques (that won't work with the free Hypervisor), described here: www.jschweg.com/2011/03/ups-shutdown-with-esxi

So I'm very happy to have found a free way to do this that doesn't take long to implement. I used a fresh copy of ESXi 5.1 Hypervisor, for my ultimately successful configuration/demonstration below, using the 60 day trial period. Retesting with the free Hypervisor license mode will be next, since I don't yet know anything about how this CyberPower solution is "talking" to the ESXi host.

I may also eventually try to get UPS monitoring working within the vMA that I plan to use as a syslog capture appliance. But some tests so far with vMA have revealed some difficulties there, particularly surrounding firewall config for email alerting.

So for now, I'm going with the simplest and perhaps most robust approach. I may wind up just sticking with it, since it apparently seems to work so well. You may too, once you see it's not difficult to set up and test.

What's the solution? Well, it's a pre-configured appliance (VM) based on CentOS, where the hard parts were already handled by CyberPower. No Linux experience necessary. At all.

The piece-of-mind in having an unattended, automated, and simple way to know that your home/lab environment will protect itself in a power outage situation is wonderful. This sort of configuration will reduce your chances of data loss, or running into potential RAID caching/corruption issues. You'll be confident you can bring your servers back up after the power outage, without worry.

So read the steps below. And note that I seem to have some issues with reliability of USB communications on some of my USB ports and not others, which feels more like a hardware problem, not sure yet. So the advice would be to remove the USB device from the VM configuration, move the cable to another USB port, then try again, should you also run into communications reliability issues.

If you're still not quite convinced it's this easy, go ahead and check out the simple YouTube video seen below, where I demonstrate, recorded live, uncut (only one blank section removed), done with only one rehearsal.

You may also wish to browse through CyberPower's documentation here.

Note that CyberPower has excellent support staff, who already know about the lack of documentation for the appliance, and a weak description on the download site that doesn't use the word "appliance". I'm confident those minor shortcomings will be addressed.

Here's the step-by-step instructions for download/configure/test

1) Download CyberPower PowerPanel Business Edition.
Updated Dec 10 2015 - all of CyberPower's related downloads are now here, with the direct file download URL you want listed here:

PowerPanel Business Edition Agent Virtual Appliance 3.0 (ESXi, ESX)

PowerPanel Business Edition shutdown software works with serial or USB connections with UPS. Supports VMWARE vSphere ESXi 4.1 and later. VMWARE vSphere ESXi requires VM: CPU's:1, Memory: 2G, Disk: 20G. Virtual Appliance OS: CentOS 6.2, Username/Password: admin/admin. PowerPanel® Software Username/Password: admin/admin

~~Grabbing that 672MB zip file that's apparently coming from Europe (seen in full URL above) takes a full hour, to me here in North America, with an apparently 1Mbit/sec max speed.~~

Updated Dec 10 2015 - It seems CyberPower has moved to Amazon S3 storage in the US West region, for faster downloads for most folks. For me, I'm now at about 20 minutes.

2) Unzip the download you will find it unzips another zip file, extract that one as well

3) Import the VM use vSphere Client's File, "Deploy OVF Template..." menu navigate to the OVF file extracted in step 2 above

4) Configure the VM's settings turn on VMware Tools Time sync assuming you've attached your CyberPower UPS to the ESXi 5.1 host, attach the CyberPower USB device to the VM

5) Power up the VM

6) Point your browser to the VM's IP address login as admin/admin

7) Configure shutdown of ESXi 5.1 on the Event Action, Settings (probably works fine for ESXi 5.0 as well):

8) Configure Event Action, Events be sure "Utility power failure" column marked "Shutdown Initiated" is set, I chose 30 seconds:

You may also wish to configure automatic shutdown and startup settings of your most valuable VMs, including this CyberPower Appliance VM you just created, described here by VMware.

Finally, for more details, catch the complete walk-through video below, which includes VM download, deployment, configuration, and shutdown test procedures. The actual section where I cover the install and configuration of the appliance, followed by web configuration of shutdown, is only about about 15 minutes:

Jul 02 2013 Update

Special thanks to site visitor Brian, who left this comment alerting me that 2.3.2 is now out, downloaded from the 4th/last link listed here, or the direct URL from here (which is still very slow from EU to NA, taking over an hour): eu.cyberpowersystems.com/support/download/ppbe232_rev2_centos.zip Brian apparently had an issue with an older version of the appliance that I did not experience, looking forward to hearing how that turns out for Brian. I'll also be testing 2.3.2 out myself, of course. Stay tuned!

Jan 18 2014 Update

These same instructions have been confirmed to work fine with ESXi 5.5, and the newer CyberPower PPBE 2.3.3 appliance, downloaded from here. Download links in above article now updates. There is no need to re-record video and screenshots, since the only difference I noticed is that I can now see the watts being used in realtime (they used to always falsely show 0%). See also visitor comments and screenshots below, as well as the closely related article, TinkerTry.com/superguide-cyberpower-pfclcd-ups-mini-towers-protect-your-homes-computers-and-entertainment.

Feb 22 2014 Update

PowerPanel Business Edition Agent Virtual Appliance 2.4 (ESXi, ESX) is now out, see release notes here. I haven't bothered to test it, not seeing anything significant enough in the release notes to warrant re-testing everything out, I'm pretty happy with 2.3.3 as it is.

Apr 04 2015 Update

From site visitor comment below, we now have confirmation that one of the last builds that allows USB support is 2.6.1, downloaded from CyberPower here:
eu.cyberpowersystems.com/download/download.aspx?file=ppbe261_agent_centos.zip

Aug 28 2015 Update

New release!
PowerPanel Business Edition Agent Virtual Appliance 2.9 (ESXi, ESX)
Testing it today, sure was easy to download and install. At last, AWS (Amazon Web Services) S3 is used for the download server, no more of those very slow downloads from EU to US, nice! Over an hour to download before became 2.5 minutes now. Here's the direct download link:
s3-us-west-2.amazonaws.com/cyber-power/software/ppbe290_agent_centos.zip

2.9 works with ESXi 6.0.0b just fine — So far, testing going well!

2.9 works with ESXi 6.0.0b just fine here's status

Dec 14 2015 Update

New release!

PowerPanel Business Edition Agent Virtual Appliance 3.0 (ESXi, ESX)
AWS (Amazon Web Services) S3 is still used for the download server, no more of those very slow downloads from EU to US, nice! Still took me about half an hour though, so it seems some sort of server-side throttling has been activated since 2.9. Here's the direct download link:
s3-us-west-2.amazonaws.com/cyber-power/software/ppbe300_agent_centos.zip

There's a report of some rather quirky behavior with 3.0, see the fantastically detailed comments by Georges Zwingelstein in the open disqussion below. I haven't yet had the chance to fully test this version out yet, but it would seem you might want to use a different version for now.

Jan 22 2016 Update

New release! I have not tested 3.1.1 yet.

Pick whichever is faster, which is typically closer.

From AU (cpsww.com.au):

PowerPanel Business Edition 3.1.1 Agent Virtual Appliance (supports VMWARE vSphere ESXi 4.1 and later)

From EU (cyberpower-eu.com):

PowerPanel Business Edition 3.1.1 Agent Virtual Appliance, supports VMWARE vSphere ESXi 4.1 and later.

Jan 28 2016 Update

I've had multiple people contact me, including Anthony Capone below, telling me that the issues with 3.0 seem to have been fixed in 3.1.1, this is great news!

Mar 24 2016 Update

TinkerTry visitor Steve Lowe has kindly provided us with some free code to get your PPBE working well with your RHEL6 (Red Hat Enterprise Linux 6), as seen in the discussion below.

These scripts are not applicable and not needed for the pre-configured VMware appliance.

Specifically, these scripts are applicable to PowerPanel Business Edition users. At that download site, you'll find the particular download that these scripts are intended for:

PowerPanel® Business Edition 3.1.2 (for 64 bits x86 of most Linux builds, VMWARE vSphere ESXi 4.1/5), PowerPanel® Business Edition software provide graceful shutdown service for UPS and PDU. Connects UPS with serial or USB connections and network. Download
Size: 157,638,656 bytes, MD5:6e6d1f9058f7adb46b191d0f6002146c

But before considering using these, read Steve's clarifications:

Found a little quirk with ppbe on RHEL6, it does not like the usb port going to sleep. This was a real bear to track down. Also SMTP ports were being blocked by the RHEL firewall

Here is the code, you will have to remove the iptables entries that CP creates and then run my add code. I am still waiting for CP to get back to me about the last UDP port that they are using.

To remove the entries that CP creates list the firewall entries with iptables-save, then use the delete option (-D) for iptables and then save it.

I fixed the add script so that it will check for CP’s entries and remove them and then add the new entries. The main difference between what CP is doing and what I am is that I am allowing ESTABLISHED connections.

RHEL6 scripts for CyberPower owners using RHEL6, by Steve Lowe

For CyberPower PFCLCD owners using the PPBE RPMs for power status via USB.

Disclaimer:
I have not personally tested these scripts, they are provided on an as-is basis, with no assumption of support implied or given. The responsibility to protect your data is completely yours.

# remove firewall entries
declare -a ppbe_fw

ppbe_fw[0]=OUTPUT:161:udp
ppbe_fw[1]=INPUT:162:udp
ppbe_fw[2]=OUTPUT:3052:tcp
ppbe_fw[3]=INPUT:3052:tcp
ppbe_fw[4]=OUTPUT:3052:udp
ppbe_fw[5]=INPUT:3052:udp
ppbe_fw[6]=OUTPUT:53568:tcp
ppbe_fw[7]=INPUT:53568:tcp
ppbe_fw[8]=OUTPUT:53566:udp
ppbe_fw[9]=INPUT:53566:udp

for i in "${ppbe_fw[@]}"
do
  INOUT=`echo ${i} | awk -F: '{print $1}'`
  PORT=`echo ${i} | awk -F: '{print $2}'`
  PROTO=`echo ${i} | awk -F: '{print $3}'`

  if [ `command -v iptables` ] ; then
    if [ "`iptables-save | grep ${INOUT} | grep ${PORT} | grep ${PROTO} | grep ESTABLISHED`" ] ; then
#       echo "iptables -D ${INOUT} -p ${PROTO} -m state --state NEW,ESTABLISHED --dport ${PORT} -j ACCEPT && DELETE=1"
      iptables -D ${INOUT} -p ${PROTO} -m state --state NEW,ESTABLISHED --dport ${PORT} -j ACCEPT && DELETE=1
    else
      iptables -D ${INOUT} -p ${PROTO} -m state --state NEW --dport ${PORT} -j ACCEPT && DELETE=1
    fi;
  fi;
done;

service iptables save
service iptables restart

# add firewall entries
declare -a ppbe_fw

ppbe_fw[0]=OUTPUT:161:udp
ppbe_fw[1]=INPUT:162:udp
ppbe_fw[2]=OUTPUT:3052:tcp
ppbe_fw[3]=INPUT:3052:tcp
ppbe_fw[4]=OUTPUT:3052:udp
ppbe_fw[5]=INPUT:3052:udp
ppbe_fw[6]=OUTPUT:53568:tcp
ppbe_fw[7]=INPUT:53568:tcp
ppbe_fw[8]=OUTPUT:53566:udp
ppbe_fw[9]=INPUT:53566:udp

for i in "${ppbe_fw[@]}"
do
  INOUT=`echo ${i} | awk -F: '{print $1}'`
  PORT=`echo ${i} | awk -F: '{print $2}'`
  PROTO=`echo ${i} | awk -F: '{print $3}'`

  if [ `command -v iptables` ] ; then
    if [ "`iptables-save | grep ${INOUT} | grep ${PROTO} | grep ${PORT} | grep -i ESTABLISHED`" ] ; then
      iptables -D ${INOUT} -p ${PROTO} -m ${PROTO} --dport ${PORT} -j ACCEPT && DELETE=1
    fi;
    if [ ! "`iptables-save | grep ${INOUT} | grep ${PORT} | grep ${PROTO}`" ] ; then
#       echo "iptables -I ${INOUT} -p ${PROTO} -m state --state NEW,ESTABLISHED -m ${PROTO} --dport ${PORT} -j ACCEPT"
      iptables -I ${INOUT} -p ${PROTO} -m state --state NEW,ESTABLISHED -m ${PROTO} --dport ${PORT} -j ACCEPT
    fi;
  fi;
done;

service iptables save
service iptables restart

Jun 12 2016 Update

TinkerTry visitor Steve Lowe has an update, some more scripts to help you get your PPBE working well with your RHEL6 (Red Hat Enterprise Linux 6), as seen in this discussion.

if [ -f /root/ppbed_old_port.lst ] ; then
OLD_PORT=`cat /root/ppbe_old_port.lst`
USB_LIST=`lsusb | grep 0501 | cut -d' ' -f4`
CURR_PORT=`echo ${USB_LIST} | awk -F: {'print $1'}`

if [ ${OLD_PORT} != ${CURR_PORT} ] ; then
PATH=/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/root/bin
echo `date +"%c"` >> duh.ppbed
service ppbed restart >> duh.ppbed
echo ${CURR_PORT} > /root/ppbe_old_port.lst
fi;
fi;

Aug 19 2016 Update

The place you download the appliance has been relocated:

PowerPanel Business Edition For Virtual Machines | Software
cyberpowersystems.com/products/software/ppbe-vm-software
with a shiny new UI, and far better download speeds courtesy of AWS.

There are other connection types detailed in the :

PowerPanel Business VMware
VMware Installation Guide
Download 3.7 MB

The direct download for the appliance:

PowerPanel Business | Virtual Appliance | Agent
Shutdown software supports VMWARE vSphere, ESXi 4.1 and later
Download 487 MB

Sep 09 2017 Update

The place you download the latest appliance is now under the "Downloads" tab here:

PowerPanel Business Edition For Virtual Machines | Software
cyberpowersystems.com/product/software/powerpanel-business-edition-for-virtual-machines

Jul 23 2018 Update

9aa08105eaf78910d6ebf54c482b8221e136c725a4b6ae4ce21062b96f34460d — Image courtesy of Chad Delatte, who asked - Anyone get this working with ESXI 6.7? I keep getting an error "VM Host settings are not set properly" when I try to verify the credentials. SSH is running.

Issues with VMware vSphere 6.7 support were reported here at TinkerTry 3 months ago.

At this time in late July, PowerPanel Business Edition For Virtual Machines 3.30 ppbe330_agent_centos.zip is still the latest version of the appliance, and using it under vSphere 6.7 is unsupported, as discussed here.

As far as what happens next, I received word on vSphere 6.7 support:

From: [redacted]@cpsww.com
Sent: Tuesday, July 3, 2018 1:23 PM
To: Paul Braren
Subject: RE: PowerPanel Business Edition

Software doesn’t support 6.7 yet, they’re working on it but we don’t have an ETA.

Meanwhile, we also have some very interesting tips and revelations from TinkerTry visitor user891 in the comments below.

Feb 04 2019 Update

We have an addendum for folks who are on ESXi 6.0, thanks to the detailed comment dropped below by Zenon Kazienko to Boyan:

Followup to Boyan, Goda and others. (Paul you may want to add this note as a addendum update to your article, I'm happy to provide any additional detail you would like) In short, PPBE3.3.0 does not shutdown esxi 6.0.0. It will say that communication with host is sucessful, and a login to the host will appear in the esxi logs, but no shutdown takes place. I went backwards and tried suing PPBE Agent 3.2.0 and I was able to sucessfully power down all my running VM's then the host. So 3.3.0 and esxi 6.0.0 is not compatible.

Near as I can figure it has something todo with the login it using to shutdown the esxi host:
When running the PPBE Agent 3.3.0 and a shutdown event happens I see the following in the esxi logs "User root@192.168.11.110 logged in as Java/1.6.0_35" As previously stated no actual shutdown occurs.

But when running PPBE Agent 3.2.0 and a shutdown event happens I see the following in the esxi logs "User root@192.168.11.110 logged in as VMware VI Client/4.0.0" After which a graceful shutdown executes.

Boyan, connect with me on linkedIn to if you would like me to help locate the older 3.2.0 installer

Apr 05 2019 Update

Special thanks to TinkerTry commenters Robby and Micke. Robby cooked up an interesting script for your Linux. Micke alerted us to CyberPower PowerPanel® Business Edition Agent for Virtual Machine Version, unknown whether ESXi 6.7.x is happy with it yet.

Apr 07 2019 Update

Thanks to Chad Delatte for his comment below about 4.0 now being available, it's further down on that same CyberPower PowerPanel® Business Edition Agent for Virtual Machine page, here's the 4.0 direct download link. The screenshot he shared seems to indicate that he got it working just fine, but actual shut down testing isn't done yet.

a155f07d0dbd498ffefad2d9b3b43289208cb7c89dfdb235d79b12659b8dc9ea

Also, note that this appliance is now said to not work for the free hypervisor ESXi 6.7. I would encourage folks to consider subscribing annually to VMUG Advantage EVALExperience, using TINKERTRY to get 10% off, featuring no more expiring license keys for your lab. That's right, no more rebuilding, and pretty much every major piece of software for vSphere clusters that VMware makes is included, as well as VMware Workstation and VMware Fusion!

Apr 18 2019 Update

I've opened a problem ticket with CyberPower Systems, by opening a ticket here, using these words in the Submit Ticket "Description: field:

I'm looking for the Agent version of the new "Version 4.0" VMware appliances that allow graceful shutdown of VMware ESXi 6.7 (not 6.5) and it seems that the only download available on the downloads page:
https://www.cyberpower.com/eu/en/software/sku3
is this one, the direct download link
https://www.cyberpower.com/ww/en/File/GetImageByDocId/SU-19010001-07
that's described like this:

File Name PowerPanel® Business Local for Virtual Machine
Operating SystemVMware ESXi 6.7/VMware ESXi 6.5/VMware ESXi 6.0
File Type Software & Utility
Version 4.0
Language English
MD569102C5C6DEED585853033E19C974FC2
File Size 1.10 GB
Description Local module.
VMware vSphere ESXi VM requirements: CPU: 1; Memory: 2G; Disk: 20G; Virtual Appliance OS: CentOS; OS Username/Password: admin/admin; PowerPanel® Software Username/Password: admin/admin.

but that downloaded file is called:
PPB Local_Virtual Machine_v4.0.zip
what I really need is a file named something like:
PPB Agent_Virtual Machine_v4.0.zip

Can you point me to the exact download I need to gracefully shut VMware ESXi 6.7 down (that allows me to specify login credentials to ESXi), paired with my USB signal cable to my PFCLCD UPS?

Apr 19 2019 Update

All set, it works! Special thanks to Chad Delatte's comment here, and Alessandro Colomba's comment here:

Append /local to the Local PPB URL, or similarly /remote for the remote PPB.

The full URL becomes something like: http://ppb-local.example.com:3052/local/

Login with the default credentials. Under Settings -> Shutdown Events, expand the "Local Host Shutdown Requirements" section, and you'll be able to enter your ESXi credentials, and verify them.

I just tested this with ESXi 6.7 Update 2, deploying from the 4.0 direct downloaded appliance using ESXi Host Client, mapping the Host USB Device to it, and tada, it worked just fine, this is great news!

CyberPower-PowerPanel-Business-Appliance-4-0-UI-for-setting-Local-Host-Shutdown-Requirements-ESXi-Account-login--TinkerTry — When you enter ESXi credentials then click on VERIFY, you are greeted with a friendly "Verify settings successfully".

On another system, I still need to actually test that graceful shutdown of all VMs and ESXi itself works as intended, when removing wall power from the UPS.

Apr 22 2019 Update

I had a chance to test this 4.0 version out by pulling my PFCLCD UPS's power cord from the wall. It works! ESXi shut down all VMs and itself down, alerting me via email along the way. This is good. Nice to finally have this squared away, even if we don't seem to officially have US support for this capability. Not a big deal to pull the code from the EU, just takes a bit longer.