The Datastores That Would Not Die

As part of a recent cleanup of our vSphere infrastructure, I was tasked with removing disused datastores from our Nimble CS500.  The CS500 had been replaced with a newer generation all-flash Nimble, and the VMs had been moved a couple of months ago.  Now that the new array was up and had accumulated some snapshots, I was cleaning up the old volumes to repurpose the array.  I noticed, however, that even though all of the files were removed from the datastores, there were still a lot of VMs that “resided” on the old volumes.

VMware addresses this in a KB (2105343) titled “Virtual machines show two datastores in the Summary tab when all virtual machine files are located one datastore.” It suggests that the VM is pointing to an ISO that no longer exists on the old datastore.

After looking at the configs, I realized that, sure enough, some of the VMs were still pointing to an ISO file that was no longer on that datastore.  Easy, right, except that on one of the test VMs, I set the optical drive setting back to “Client Device.”  It was still pointing at the old datastore.

Looking through the config again, I noticed that the Floppy Drive setting is missing from the HTML5 client.  I fired up the Flex client and set the floppy drive to “Client Device” as well.  Still no go.  For the few VMs that were pointing at a nonexistent ISO, setting the optical drive back to “Client Device” worked, but for VMs that were pointing at a nonexistent floppy, changing the floppy to “Client Device” wasn’t working.  A bug in the floppy handling?  Perhaps.

I created a blank floppy image on one of my new datastores and pointed the VM’s floppy to that new image.  Success!  The VM was no longer listing the old datastore, and I could then set the floppy to “Client Device.”  After checking out other VMs, I realized that I had over 100 VMs that had some combination of optical drive or floppy drive pointing at a non-existent file on the old datastores.  PowerCLI to the rescue!

$vm = $args[0]

$cd = get-cddrive -vm $vm
floppy = get-floppydrive -vm $vm

set-floppydrive -Floppy $floppy -FloppyImagePath "[datastorename] empty-floppy.flp" -StartConnected:$false -Confirm:$false

set-floppydrive -Floppy $floppy -NoMedia -Confirm:$false
set-cddrive -cd $cd -NoMedia -Confirm:$false

Simply save this as a .ps1 file and pass it the name of the VM (in quotes if it contains spaces.) It will get the current floppy and CD objects from the VM, set the floppy to the blank floppy image previously created, and then set both the CD and floppy to use “NoMedia.” This was a quick and dirty script, so you will have to install PowerCLI and do your own Connect-VIServer first. Once connected, however, you can either manually specify VMs one at a time or modify the script to get VM names from a file or from vCenter itself.  All of these settings can be changed while the VM is running, so there is no need to schedule downtime to run this script.

After all of this work, I found that there were still a few VMs that were showing up on the old datastores.  Another quick Google search revealed that any VMs that had snapshots that were taken with the CD or floppy mounted would still show up on that datastore.  Drat!  After clearing out the snapshots, I finally freed up the datastores and was able to delete them using the Nimble Connection Manager.

So now a little root cause analysis: why were there so many machines with a nonexistent CD and floppy mounted?  After seeing that they were all Windows 2016 VMs, I went back to our templates and realized that the tech who built the 2016 template left the Windows ISO (long since deleted) and floppy image (mounted so he could F6-load the paravirtualized SCSI driver during OS installation) mounted when he created the template.  I converted the template to a VM, removed the two mounts (using the same two-step method for the floppy) and converted it back to a template.

With that job done, I’m continuing to plug away at converting our other vCenters from 6.5 to 6.7U1.  Have a great day, everyone!

The Dell PowerEdge FX2: A Dead End?

A small crowd was gathered by the server enclosure at the edge of the Dell EMC VMworld booth, gawking at the blinkenlights on the newly announced MX7000 blade chassis.  I dodged the eye candy and the suited salesdroids surrounding it and instead searched the booth for their PowerEdge FX2 display.  Alas, the entire booth was dedicated to the 7 rack unit MX7000 and its array of server, storage and networking building blocks.

When the FX2 was announced in 2014, it held a lot of promise.  A 2U enclosure with horizontally-mounted blades, the FX2 could hold up to eight dual-processor Xeon servers, consolidating power supplies, cooling and management.  The FC430 blade crammed two Xeons and 8 DIMM slots into a quarter-width sled.  The FC630 blade offered three times as much RAM capacity, access to another PCIe chassis slot, and a “real” PERC controller in a half-width sled, and the FC830 was a rack-width quad-Xeon sled with more DIMM slots and extra drive bays.

With the introduction of the 14th generation Dell PowerEdge lineup, the FC630 half-width blade was replaced by the FC640, but the FC830 and FC430 were never updated.

Seeing the writing on the wall, I grabbed one of the Dell EMC guys that was floating around the booth and asked him about the future of the FX2.  He told me that the FC430 and FC830 wouldn’t be updated because the “processors were too big”, and that he couldn’t comment on the future of the FX2 platform further.

Now, I’ve been in IT for a while, but that explanation just seemed strange.  What did he mean by the “processors are too big?” They need a bigger heatsink because they run hotter?  They’re a physically bigger die?  After digging a bit, the answer appears to be “both of the above.”

The Broadwell-based Xeons that underpin the 13th generation PowerEdge use the 37.5mm by 37.5mm LGA1151-3 socket–a socket size that has been carried over virtually unchanged since the 35mm by 35mm Socket 478 of nearly two decades ago.  However, the Skylake-based “Purley” Xeons that power the 14th generation servers use a much larger socket called LGA3647, so-called because it has nearly 2,500 more pins than the old 1151-3.  Those extra pins are used for additional memory channels among other features.

Those extra pins mean that the actual socket had to grow–in this case, to 76mm by 56.5mm.  That’s about three times the real estate of the previous generation CPU.  The FC430 blade was already so tight that it gave up DIMM slots and the option of having a decent RAID controller to fit in the space allotted.  There is simply no room to fit a Purley Xeon without radical rework like breaking off a lot of support hardware onto a daughterboard, which complicates manufacturing, repair and cooling.

fc430

A PowerEdge FC430 blade.

So, I now understand the demise of the FC430, but what about the future of the FX2 as a platform?

The folks over at Wccf Tech have posted Intel documents about the Xeon roadmap.  The Cooper Lake and Ice Lake chips, expected in 2019 and 2020 respectively, are expected to use a socket called LGA4189.  With 15% more pins than the LGA3647, I would expect the socket to be 15-20% larger as well.  While that doesn’t sound like much, it may be just too much to fit in a dense enough blade for the FX2 to continue to make sense.

Heat will also be a factor.  The same source above shows that Ice Lake will have a TDP of “up to 230W.”  That’s 75 watts higher than the hottest processor Dell offers in the FC640.  To handle the more powerful Ice Lake processors will probably take larger heatsinks and better cooling than is possible with any of the current FX2 form factors.

So how will these larger and hotter processors affect the blade landscape?  Dell knows the answer, and the answer lies in the MX7000.  Dell’s newest blade architecture, just released to the public in the past couple of months, is your traditional big box with vertical blades in the same vein as the old M1000e.  However unlike the M1000e, which offered full-height and half-height blades, the MX7000 only offers full-height single-width (2 CPU) and double-width (4 CPU) options.  Again, the helpful salesguy at the Dell EMC booth said that they were “unlikely” to be able to offer half-height blades because the days of 125 watt 37.5mm by 37.5mm CPUs are over.

So, are blades still worth it?  The M1000e could fit 16 servers into 10U and the FX2 can still fit 20 current generation servers into 10U.  The MX7000 can only fit 8 servers into 7U, and Lenovo can fit 14 servers into 10U.  The FX2 offers very high density in a form factor that is more flexible because it can be purchased and racked in smaller increments than the larger blade systems, but it appears that the days may be numbered for that form factor.