Homelab Again

“We’re putting the band back together.”

Five o’clock. Time to shut down the ol’ laptop and go watch some TV. Or not. I feel the urge to create again. I have a bunch of hardware laying around that’s gathering dust, and it’s time to assemble it into something I can use to learn and share.

I’m not looking for anything flashy here. Money is tight and the future is uncertain for me and millions of others, so while I’d love a few old Dell servers or some shiny new NUCs, I’m sure I can cobble together something useful out of my closet for free. Initially I was going to put together a single machine and set up nested virtualization, but my best available equipment only has four cores and frankly, I feel like it’s cheating anyway. I’m a hardware junkie, so I cobble.

With a day’s work, I have scraped together a cluster of machines. One is from my original homelab build: a Dell Optiplex 9010 motherboard and i5 processor bought off eBay and transplanted into a slim desktop case. It has four DIMM slots, so it’s going to be one of my ESXi hosts. The other VM host is a Dell Optiplex 3020 SFF i5 desktop. Alas, it only has two DIMM slots, so it maxes out at 16GB with the sticks I have. I also manage scrape together a third system with eight gigs of RAM and a lowly Core 2 Duo. Transplanted into a Dell XPS tower case formerly occupied by a Lynnfield i7, it has enough drive bays that I can drop in a one terabyte SATA SSD and three 800GB SAS SSDs in RAID-0 on an LSI HBA. This system takes the role of an NFS-based NAS running on CentOS 8 and has no problems saturating a gigabit link.

It’s not all roses–my networking is all gigabit–but it totals eight cores, 48GB of usable RAM and over 3TB of flash storage. A few hours after that, I have a two-node ESXi 6.7 cluster configured and vCenter showing all green. I toyed briefly with installing 7.0, but I want to give that a little more time to bake, and besides, if what I’m planning doesn’t directly translate then I’ve done something wrong.

What are my goals here? Well, I want to work on a “let’s automate VMware” series. Not satisfied with the kind of janky scripts I’ve cobbled together to automate some of my uglier processes, I want to learn how to do this the right way. That certainly means Terraform, some kind of configuration management (Ansible, Salt, Puppet, Chef or a combination), building deployment images with Packer, and figuring out how to safely do all of this with code that I’m storing in Github. Possible future projects might involve integrating with ServiceNow, Splunk, monitoring (Zabbix, Solarwinds) or other business tools, and using the resulting tooling to deploy resources into AWS, Azure, DigitalOcean and/or VMware Cloud.

Why do this on my own? While I do have the resources at work, I feel more comfortable doing this on my own time and at my own pace. Plus, by using personal time and resources, I feel safer sharing my findings publicly with others. While my employer has never given me any indication that it would be an issue, I’d just rather be able to do and refine my skills on my own time.

So stay tuned; there’s more content to come from the homelab!

Linux Stories: The Radio Ripper

It was early 2005.  My Southern California commute had grown to 65 miles one way, averaging three hours a day.  I was growing tired of top 40 music and news radio.  After a particularly late night at work, I was listening to my local NPR station when the broadcast for the local minor league baseball team came on.  While I had never been a sports nut, I found that the broadcaster (Mike Saeger, now of the San Antonio Missions) interwove the narrative of the game with fascinating stories about the team, the players and baseball’s past.  Unfortunately, it typically started at 7:00 or 7:30 in the evening, meaning that I would typically catch the first 30 minutes at best before arriving home.

My goal became to record the games to listen to on my commute.  At first I considered piping a radio into a sound card, but I couldn’t get the station in the mountains where I lived.  I eventually figured out that the NPR station broadcasting the games had a website with a live feed.  Upon further digging, I found that the station was using a RealPlayer streaming audio format for its live feed.

I looked at options for automated recording of the feed and was unimpressed with my options on the Windows platform.  After some searching, I found that mplayer on Linux could play a RealPlayer feed, and more importantly, it could record it to a PCM-coded WAV file.  Thus began my quest to build a recording rig.  My goal was to record the game to a WAV file and convert it into a low quality MP3 that I could play from my chewing-gum-pack sized Creative MuVo.

I chose Fedora Core as my distro, mainly because most of the relevant forum posts and blogs I found on the topic had a Fedora or Red Hat focus.  I downloaded Fedora Core 3 to a CD and looked through the scrap pile at work to find a suitable machine.  I wanted something with low power consumption, since California’s electricity rates were astronomical and this machine would be running 24/7.  I settled on a Dell Latitude C600 laptop.  It had a 750MHz mobile Pentium III and 192MB of RAM.  It had been relegated to the scrap heap because it had a weak battery, some missing keys and a broken screen, none of which would be an issue for my purposes.  More importantly, with the lid closed, it drew only 15 watts at full tilt and 12 watts at idle with the 60GB drive spun down.  The weak battery still lasted 30 minutes, which was enough to ride out most power interruptions.

I’m a command line guy–I had to be dragged kicking and screaming into Windows–so I deselected the GUI and went with a minimal install.  All told, the running OS took less than a gigabyte of disk space and 48 megs of the 192 megs of available RAM, leaving plenty for my tools.  I installed mplayer, sox and lame to handle the audio processing.  My shell script would spawn mplayer in the background, wait four hours, and then signal it to terminate.  Sox would then be launched to re-encode the stream into low bit-rate mono, which was then piped into lame to create the final MP3.  The encoding process would take several hours but still complete in time for me to download the file to my MP3 player the next morning.  Scheduling was done by looking at the team’s schedule at the beginning of the week and using a series of ‘at’ commands to run the script at the appropriate days and times.

As baseball season drew to a close, I looked for alternatives.  I looked at downloading NPR weekend shows like Car Talk and Wait Wait–shows that were not yet available as podcasts.  Those shows were available at NPR’s website, again in RealPlayer format.  However, the filenames changed each week, and the shows were broken into segments that had to be downloaded individually.  I was able to use wget to get the HTML for the page and grep to parse out the filenames into a text file.  I then fed the text file into mplayer, one line at a time, to download the segments.  Finally, I used sox to encode and concatenate the individual files into a single file, which was converted to MP3 by lame.

After about a year, I recognized a mistake I had made.  I chose Fedora Core not knowing about their support policy.  Fedora Core 3 only received security patches for about 16 months from its release, or less than a year after I installed it.  There was an unofficial process for upgrading to FC4, but it broke a lot of stuff and took a lot of cleanup.  After that upgrade, I left the system at FC4 until it was decommissioned.

This was my first real-world experience with Linux, and it really helped me to feel comfortable with the operating system.  While I have used Debian, Ubuntu, SuSE and other Linuxes, this experience with Fedora drove me to choose CentOS (or Red Hat if OS support is required) as my go-to Linux server OS.

Homelab 2018 – The hardware

Like many, I’ve dreamed of having a miniature datacenter in my basement.  My inspiration came from a coworker who had converted his garage into a datacenter, complete with multiple window A/C units, tile floor, racks and cabinets full of equipment and his own public Class C back when individuals could request and receive their own Class C.

I lived in a house in the mountains that had a fairly high crawlspace underneath, and I dreamed of some day scooping it out level, pouring cement and putting in a short server cabinet and two-post. To that end, I had a lot of equipment that I had saved from the landfill over the previous couple of years: old second generation ProLiants, a tape changer, a QNAP that had been discarded after a very expensive data recovery operation and an AlphaServer 4100.

However, there were two important lessons that I learned that changed my plans:  first, that home ownership can be very expensive, and something simple like pouring a floor was beyond both my abilities and my budget, and second, that electricity in California is ruinously expensive for individuals.  While commercial customers generally get good prices regardless of their usage, it doesn’t take many spinning hard drives before the higher residential tiers kick in at 35+ cents per kWh.  I think my coworker got away with it because he was doing web hosting on the side out of his garage at a time when web hosting still actually paid something.

Once I realized the sheer cost of having the equipment turned on, to say nothing of the cost of air conditioning the basement at the time when my poorly insulated house was already costing over $300 a month to cool during the summer, I gave up on my dreams and donated most of the equipment to that coworker.  My homelab ended up being a single core Dell laptop “server” with a broken screen that ran Fedora and consumed 12 watts.

Fast forward to 2016, and I realized that I needed to make a change.  Working as the sole IT admin in a SMB means that I always had zero to near-zero budget and had to assemble solutions from whatever discarded hardware and free software I could cobble together.  While that does provide some valuable experience, modern companies are looking for experience with VMware or Hyper-V on SAN or virtual SAN storage, not open source NAS software cobbled together and running QEMU on an old desktop.

I looked at building that homelab I always wanted, but again, electricity cost had only gone up in the intervening 15 years, and I was now in a transitional state of four people living in a two bedroom apartment.  I wasn’t willing to sacrifice what little tranquility we had at home by having a couple of R710s screaming in the living room.  Thus, I decided to build my own “servers.”

I already had built my “primary” desktop specifically to move some workloads off of my laptop.  It was an i5-4570S desktop with 6GB of RAM and a discarded video card that I used for light gaming and running one or two small VMs in Virtualbox.  My goal was to build two or three compact desktops I could run trialware on to familiarize myself with vCenter, SCCM and other technologies I was interested in.  By keeping the machines compact, I could fit them under my desk, and by using consumer-grade hardware, I could keep the cost, noise and power usage down.

To save space, I chose the Antec VSK2000-U3 mini/micro-ATX case.  Looking back, this was a huge mistake.  These are about the size of a modern SFF Dell case, but it is a pain finding motherboards and heatsinks that fit.  However, they did the job as far as fitting into the available space under the desk.  These cases use a TFX form factor power supply–common in a lot of desktop and SFF form factor Dells, so used power supplies are cheap and plentiful on eBay.

When choosing motherboards, my goal was to find a mini or micro-ATX board that had four RAM slots so I could eventually upgrade them to 32GB using 8GB sticks–not as easy a task as one might think.  The first motherboard I found on Craigslist.  It was a Gigabyte Mini-ATX motherboard with an i5-3470S CPU.  Due to the location of the CPU on the motherboard, I couldn’t fit it in the case without the heatsink hitting the drive tray, so I ended up swapping the mobo with the board in my home machine, as my nice Gigabyte motherboard and i5-4570S fit the Antec case.

Thinking I was clever, I chose an eBay take-out from an Optiplex 9010 SFF as my second motherboard.  It had four RAM slots and was cheaper than any of the other options.  However, I soon found out that Dell engineered that sucker to fit their case and no others.  Their proprietary motherboard wouldn’t fit a standard heatsink.  I ended up getting the correct Dell heatsink/fan combination from eBay, which fit the case perfectly and used a heat pipe setup to eject the heat out of the back of the case.  I also had to get the Dell rear fan and front panel/power button assembly to get the system to power on without complaining.  Fortunately, the Dell rear fan fit the front of the Antec case where Antec had provided their own fan, so no hacking was needed.  Finally, the I/O shield in the Optiplex is riveted into the Dell case and can’t be purchased separately, so I’m running it without a shield.  This system runs with an i5-3570K, which I rescued out of another dead machine I rescued from the trash.


Optiplex 9010 SFF heatsink, bottom view.  The end of the fan shroud on the left stops just millimeters from the rear grille of the Antec case, like they were made for each other.

Once the homelab was up and running, I upgraded RAM when I could afford it.  The two homelab machines and my desktop started out with 8GB each and now have 24GB each.  To further save electricity, I have them powered down when not in use.  (My primary desktop stays on, as it runs Plex and other household services.)  Right now, each system has a hard drive and a 2.5″ SSD.  (They have an empty optical drive bay, so additional drives can fit with a little work.)  I picked up some four-port gigabit Intel 1000 NICs (HP NC364T) since the onboard Realtek NICs on the Gigabyte boards aren’t picked up by ESXi.

So the big question: will these work with ESXi 6.7?  In short, no.  They run 6.5 Update 2 just fine, but the 3570K crashes when booting 6.7, possibly because it doesn’t have VT-d support.  However, they work great running 6.0 and 6.5, which will get me through my VCP studies.  For the price, power consumption, heat and noise, they do the job just fine for now.

The Littlest Datacenter Part 5: Monitoring

Part 1 of this saga can be found here.

Good monitoring doesn’t come ready to use out of the box.  It takes a lot of work: deciding what to monitor, deciding on thresholds and when and how to alert.  Alert Fatigue is a thing, and the last thing you want is to miss an important alert because you silenced your phone to actually get a full night’s sleep.  But monitoring and alerting, if configured in a sensible way, can save hours of downtime, user complaints and the IT practitioner’s sanity.

I had been searching for monitoring utopia for more than a decade.  I tried Zenoss, Nagios and Zabbix on the FOSS side, and briefly toyed with PRTG and others before being turned down by management due to cost.  In every case, it took a lot of work to set up.  Between agent installation and updates, SNMP credential management, SNMPWALKing to find the right OIDs to monitor, and figuring out what and when to alert, monitoring in a mixed environment takes a lot of TLC to get right.

In my mini datacenter build, I had the opportunity to build my monitoring system the way I wanted.  I could have made a business case for commercial software, but this build and move already had taken a significant chunk of money–remember that the entire warehousing and shipping/receiving operation had also been moved and expanded.  And besides, having tested several of the commercial products, I knew that the initial setup probably would have been faster, but the ongoing work–tweaking thresholds, adding and removing sensors, and configuring monitoring–was going to take a lot of work regardless of the system I chose.

With that in mind, I chose Zabbix.  I had used Nagios in the past, but I preferred a GUI-based configuration in this case.  In an environment where I was deploying a lot of similar VMs or containers, a config file-based product like Nagios would probably make more sense.  Having chosen Zabbix, the question was where to install it.  At the time, Zabbix didn’t have a cloud-based product, and installing it within the environment didn’t make sense, as various combinations of equipment and Internet failure could make it impossible to receive alerts.

After looking at AWS and Azure, I went with simple and cheap: DigitalOcean.  Their standard $5/month CentOS droplet had 512MB of RAM, one vCPU, 25GB of disk space, a static IP and a terabyte of transfer a month.  I opted for the extra $1/month for snapshotting/backups, which would make patching less risky.

The first step was to set up and lock down communications.  I went with a CentOS VM at each of the company’s two locations and installed the Zabbix proxy.  The Zabbix proxies were configured for active mode, meaning that they communicated outbound to the Zabbix server at DigitalOcean.  pfSense was configured to allow the outbound traffic, and DigitalOcean’s web firewall restricted the Zabbix server from receiving unsolicited traffic from any IPs other than the two office locations.  Along with the built-in CentOS firewall, it was guaranteed to keep the bad guys out.  I also secured the Zabbix web interface with Let’s Encrypt, because why wouldn’t I?

Next up was configuring monitoring.  Zabbix comes with a Windows monitoring template.  After installing the agent on all of the Windows hosts and VMs, I configured monitoring by template.  I found it best to duplicate the base template and use the duplicate for each specific function.  For instance, one copy of the Windows template was used to monitor the DNS/DHCP/AD servers.  In addition to the disk space, CPU usage and other normal monitoring, it would monitor whether the DHCP and DNS servers were running.  Another copy of the template would be tweaked for the VM hosts, with, for instance, more sane disk space checks.  Linux monitoring, including of the pfSense boxes, was configured similarly.  Ping checks of the external IPs was done from the Zabbix host itself.

Environmental monitoring was also important due to the small closet size and lack of generator support for the building.  SNMP came to the rescue here.  Fortunately, I had two Tripp Lite and one APC UPS in that little server closet.  Using templates I found online, I was able to monitor battery charge level and temperature, remaining battery time, power status, humidity and battery/self test failures.  The Tripp Lite units had an option for an external temperature/humidity sensor, and I was able to find the remote sensors on eBay for less than $30 each.  One I mounted in front of the equipment in the rack to measure intake air temperature, and the other I mounted dangling in the airflow of the A/C to measure its output temperature.  That way, I would be alerted if the A/C unit failed, as well as monitor its cycle time and see how cold the output air was compared to historical data.

The primary infrastructure to monitor was the Hyper-V cluster.  Fortunately, I found a Hyper-V and CSV template online that I was able to tweak to work.  Not only did it monitor the CSVs for free disk space and connectivity, it could alert if either of the hosts had zero VMs in active status–an indication that the host had rebooted or otherwise lost its VMs.  A Dell template monitored server systems and could report a disk or fan failure.

No monitoring system is complete without graphs and alerts, so I built a custom “screen”, which is Zabbix terminology for a status dashboard.  I created a 1080×1920 portrait screen with battery and temperature graphs, a regularly updated picture of the interior of the room provided by an old Axis camera, and a list of active alerts and warnings.  I mounted a 1080p monitor in portrait orientation in my office and used an old PC to display the screen.

Finally, I tackled the issue of Alert Fatigue.  At a previous employer that had a 24-hour phone staff, I would receive eight to ten phone calls in the middle of the night, and of those calls, maybe only one a month would actually need my attention that night.  I vowed to tweak all of the unnecessary alerts out of my new monitoring system.

I used a generic IT Google Apps email account on the Zabbix server to send the alerts to my email.  I then set my GMail to parse the alert and, if the alert status was “Disaster”, forward it as a text message to my phone to wake me up.  I then went through all of my alerts and determined whether they were critical enough to get a “disaster” rating.  A VM that had just rebooted wasn’t critical.  The Windows service that recorded surveillance on the NVR not running was critical.  The power going out wasn’t critical.  The remaining battery dropping below four hours, indicating that the power had been out for over an hour, was critical.  By setting up my monitoring this way, I would still be awakened for actual issues that needed immediate correction, but less critical issues could be handled during waking hours.  I also tiered my issues to indicate when a problem was progressing.  For instance, I would get an email alert if the room temperature reached 75 degrees, a second at 80 degrees, and a phone alert at 85 degrees.  That would give me plenty of time to drive in or remote in and power down equipment.

Many alerts I disabled completely.  The Windows default template alerts if CPU usage reaches something like 70%.  Do I care if my VMs reach 70% CPU?  If I don’t care at all, I can turn the alert off completely.  If I’m really concerned about some runaway process eating CPU for no reason, I can tweak that setting so that I’m not alerted until the CPU exceeds 95% for 15 minutes continuously.  At any rate, that’s not going to be flagged as a “disaster.”

The Zabbix droplet worked great.  There was no noticeable downtime in the two years I ran it.  I was able to run about 2,000 sensors on that droplet with overhead to spare even with one vCPU and 512MB of RAM.  (DigitalOcean has since increased that base droplet to 1GB of RAM.)  I probably would have replaced my GMail-to-text kludge with PagerDuty if I had known better, as it can follow up alerts with automated phone calls.  at any rate, I slept much better knowing my environment was well-monitored.

Next time:  Lessons learned, or “What would I do differently today?”



The Littlest Datacenter Part 4: Environmental and Security

You can find part 1 of this saga, including the backstory, here.

Strip malls and low-rent office/retail buildings present a number of challenges to the fledgling IT-focused company.  From poor electrical infrastructure to lack of security to lack of options for broadband, the small business IT guy has his work cut out for him.  Add to that the fact that management typically chooses space based on price, and you’re lucky if you even get the address before moving day, much less a voice in the selection process.

And so I found myself cramming 100TB of disk into a closet large enough for a 42U cabinet and a 2-post.  And when I say large enough, I mean JUST large enough.  I was able to convince them to put a wide enough door that I could maneuver the cabinet out of its wheels so I could get behind it.  To add to the fun, this closet was in the office space, so noise was a factor.

Fortunately, I had a say in how the room was built.  My goal in building the room was to maximize the use of space, keep the noise and cold air in, and provide a modicum of security; after all, sensitive equipment would be in this room.  To that end, the builder put fiberglass insulation into the walls, and then doubled the drywall on the outside facing the office. The inside walls were drywalled and then covered in plywood lagged to the studs, providing a strong base for mounting wall-based equipment and further reducing sound.  The roof was a lid consisting of solid steel sandwiched between plywood above and drywall below.  A steel door with a push button combination lock and steel frame completed the room.

As a warehousing operation, shipping was the most important function of the business.  Delayed shipments could result in financial penalties.  Since shipping was a SaaS function, my goal was to provide the business with the power and Internet so they complete the bulk of the day’s shipping even in the event of a power outage.  Installation of a generator was impossible due to the location, so I had to settle for batteries.  I ended up with five UPSes in total.  One Tripp Lite 3kVA and one APC 3kVA split the duties in the server cabinet, and one Tripp Lite 3kVA UPS kept the 2-post (with PoE switch for the cameras) and wall-mounted equipment alive.  I also had a 1,500VA unit at each pair of shipping tables to power the shipping stations (Dell all-in-ones) and label printers.  Additional battery packs were added to each unit so that a total uptime of about five hours could be achieved.  That gave plenty of time to either finish the shipping day or make a decision about renting a portable generator for longer outages.  So far, there has been only one significant outage during a shipping day, but the production line was able to work through it without a hitch.

Cooling for the server room was provided by a Tripp Lite SRCOOL12K portable air conditioner.  The exhaust was piped into the area above the drop ceiling.  While this did the job, I would have preferred a variable inverter drive unit with dual hoses for more efficiency.  We investigated a mini-split, but due to property management requirements, it would have taken months and cost many thousands of dollars.  The server equipment could go for well over an hour before heat buildup began to become an issue, which was enough time to open the door and use a fan.  Equipment could also be shut down remotely, further reducing heat production.

In addition to the physical security, infrastructure security had to be considered as well.  To that end, I deployed a physically separate network for the surveillance, access control and physical security systems.  Endpoints ran with antivirus and GPO-enforced firewalls and auto-patching.  Ninite Pro took care of keeping ancillary software up to date.  As all of the company equipment was wired, the wireless network was physically segmented from the rest of the network for BYOD and customer use.  pfBlocker was deployed on the pfSense firewalls to block incoming and outgoing traffic to countries where we did not do business, and outbound traffic was limited initially to ports 80 and 443, with additional ports added on an as-needed basis.  Finally, I deployed Snort on the firewalls themselves and in various VMs to catch any intrusions if they happened.

Coming up: Monitoring and lessons learned.


The Littlest Datacenter Part 3: Backup

Part 1 of this saga can be found here.

I had an interesting backup conundrum.  I had 26TB of utterly incompressible and deduplication-proof surveillance video data that needed to be backed up.  As this video closely recorded the fulfillment operation and was used to combat fraud by “customers”, it needed to be accessible for 90 days after recording.  Other workloads that needed to be backed up included the infrastructure (AD/DNS/DHCP) servers, a local legacy file server, the PBX, and a few other miscellaneous VMs.  Those, however, totaled less than 2TB and were easily backed up using a multitude of options.

The video data was difficult to cloudify.  Just the initial 26TB was massive, but it also changed at a rate of about 50GB per hour during shipping hours.  And in case of actual hardware failure, getting that 26TB back and into production was equally difficult.

This combined with a limited budget led me to a conclusion I didn’t want to reach: tape.  I needed to get a copy of the data out of the server room, and having a 52TB array sitting under somebody’s desk wasn’t appealing.  One thing I did know was that management told me that an offsite copy wasn’t necessary; in the event of a full-site disaster (fire, earthquake, civil unrest), the least of our worries would be defending shipments that had already been made.  With that in mind, I decided to go with a disk-to-disk-to-tape option.

When it comes to buying server hardware with massive amounts of disk, I generally look at Super Micro.  I had been quoted a few Dell servers with 4TB drives, but the cost was generally breathtaking.  I wanted a relatively lightly powered box with a ton of big drives at a reasonable price, and I got it.  I picked up a new 16-bay box with a single 8-core CPU, an LSI RAID controller and 64GB of RAM for cheaper than used Dell gear.  For drives, I went with 6TB enterprise-class SAS drives: four from WD, four from HGST, four from Seagate and four from Toshiba.  I configured these drives in a RAID-60 with two of each model of drive in each half of the RAID-60.  That way, if I had a bad batch of Seagate drives (which NEVER happens, of course), I could lose all four and still have a running, if degraded, array.  This RAID arrangement gave me about 72TB usable–enough for two full backups and a number of incrementals.

For tape duties, I picked up a new-old stock Dell PowerVault PV124.  This LTO-5 SAS changer was chosen at a time when the initial build called for only 16TB for video.  Holding 16 tapes, its raw uncompressed capacity was about 24TB, and it eventually was using all 16 tapes for a single backup.

Veeam was chosen to handle the backup duties, because at the time, it was the only solution I could find that could do VM-level backups AND handle SAS tape libraries.  Backups were made nightly to the local proxy, with full copies to tape occurring weekly.  The tapes were then stored in a fireproof safe in a steel-and-concrete vault at the other end of the building.

Coming soon: Environmental, monitoring and security.

The Littlest Datacenter Part 2: Internet and Firewalls

Part 1 of this saga can be found here.

As mentioned before, this was a SaaS-focused business.  Most of the vital business functions, including ordering, shipping and receiving, pricing, accounting and customer service, were SaaS.  That meant that a rock-solid Internet connection was required.  But again, a small business runs on a small budget.  Combined with the fact that the business was in a strip mall, and we were lucky to get Internet at all.

Fortunately, we were able to get Fios for a reasonable cost and installed reasonably quickly.  Previously the business had been running IPCop on a tiny fanless Jetway PC, but I felt we had outgrown IPCop, and the Jetway box, though still working, was a bit underpowered for what I needed.  I settled on pfSense as my firewall of choice, but I didn’t want to run it on desktop hardware.

Fortunately, Lenovo had a nearly perfect solution for my budget: the RS140 server.  It was a 1U rackmount server with a four-core Xeon E3 processor with AES-NI for fast crypto, and it came with 4GB of RAM for a hair over $400.  The price was so good I bought two.  Each I fitted out with an additional 4GB of RAM and two SSDs, a 240GB from SanDisk and a 240GB from Intel.  There was a bit of consternation when I found that the server came with no drive trays, but I found that I could mount the SSDs in 3.5″ adapters and mount them directly into the chassis with no drilling.

The SanDisk and Intel SSDs in each server were configured in software RAID-1 using the onboard motherboard RAID, and the integrated IPMI was finicky but good enough that I could remotely KVM into the boxes if need be.  The servers were then configured into an active/passive pair using the pfSense software, and I used a new HPe 8-port switch to connect them to the Fios modem.

The firewalls worked so well I bought a matching pair for the other location and connected them with an IPSec tunnel so they could share files more securely.

You may ask why I used hardware for the firewalls instead of virtualizing them.  The answer is, I initially did virtualize them in Hyper-V.  However, I just wasn’t comfortable with the idea of running my firewalls on the same hardware as my workloads.  There have been rumors of ways to escape a VM and compromise the host, and indeed recent revelations about hypervisor compromise through bad floppy drivers and side channel data leakage a la Spectre and Meltdown have confirmed my suspicions about virtualized firewalls.

Coming soon: Backup, environmental, monitoring and security.

The Littlest Datacenter Part 1: Compute and Storage

I was tasked with building a datacenter.  Okay, not really.  The company was expanding into a low-cost strip mall, which meant limited connectivity options, no power redundancy and strict rules regarding modifications.  It also meant that I was limited to two racks in a tiny closet in the middle of an office space.  Finally, as always, there was minimal budget.

The Requirements

The COO was very SaaS-focused for business applications.  As the sole IT person (with additional ancillary duties), I was happy to oblige.  File storage, office applications, email, CRM, shipping and accounting functions were duly shipped off to folks who do that kind of thing for a living, leaving me with a relatively small build: AD/DNS/DHCP, phone system and surveillance.  While the systems I was replacing used independent servers that replicated VMs across, it was a decidedly more… manual failover process than I wanted.  As a shipping-focused business that was penalized for missing shipping deadlines, systems needed to be redundant and self-healing to the extent possible within the thin budget.  Finally, I knew that I would eventually be handing off the environment to either a managed service provider or a junior admin, so everything needed to be as simple and self-explanatory as possible.

The infrastructure VM (AD, DNS, etc.) and ancillary VMs were pretty straightforward.  The elephant in the room was the surveillance system.  Attached to 27 high-resolution surveillance cameras, it would have to store video for 90 days for most of the cameras for insurance reasons.  Once loaded with 90 days of video, it would consume 26TB of disk space and average about 50GB/hour of disk churn during business hours.

The Software

Because of costs, I settled on Hyper-V as my VM solution.  As it’s included with the Windows licenses I was already buying, it was cost-effective, and it had live migration, storage migration, backup APIs, remote replication and failover capabilities.  Standard licensing allows two Windows VMs to run on one license, further reducing costs.

Next to consider was the storage solution.  As I mentioned, the existing server pair consisted of two independent Hyper-V systems, with one active and one passive.  Hyper-V replication kept the passive host up to date, but in the event of a failure or maintenance, failing over and failing back was a long and arduous process.  I opted for shared storage to allow HA.  Rather than roll my own shared storage, I decided to buy.

After talking with several vendors, I settled on Starwind vSAN.  I had used their trialware with good results, and it had good reviews from people who had chosen it.  As it ran on two independent servers with independent copies of the data, it protected both from disk failure and host, backplane, operating system, RAID controller and motherboard failure.  Starwind sold a virtual appliance which was an OEM-branded but very familiar Dell T630 tower server, so I ordered two, which was substantially cheaper than sourcing the servers and vSAN software separately, and about a sixth of the cost of an equivalent pair of Dell servers and separate SAN.

The Hardware

I settled on a pair of midrange Xeons with 12 cores each–24 cores or 48 threads per host.  This was enough to process video on all of the cameras while leaving plenty of overhead for other tasks.  The T630 is an 18-bay unit with a rack option.  Dual gigabit connections went to the dedicated camera switch, while another pair went to the core switches.  For Starwind, a dual-port 10 gigabit card was installed in each host.  One port on each was used for Starwind iSCSI traffic, and the other for Starwind sync traffic.  Both were redundant in software, and they were direct-connected between the hosts with TwinAx.  Storage for each host consisted of sixteen 4TB Dell drives and two 200GB solid state drives for Starwind’s caching.

In an effort to reduce complexity, I went with a flat network.  Two HPe switches provided redundant gigabit links to the teamed server NICs and the other equipment in the rack.  Stacked and dual-uplinked HPe switches connected the workstations and ancillary equipment to the core.

The Software

Windows Server 2012R2 standard provided the backbone, with Starwind vSAN running on top.  Two Windows VMs powered the AD infrastructure server and the surveillance recording server.  I later purchased an additional Windows license and built a second DC/DNS/DHCP VM running on the second host.

Coming Soon:  Firewalls, backup, environmental, monitoring and security


The Datastores That Would Not Die

As part of a recent cleanup of our vSphere infrastructure, I was tasked with removing disused datastores from our Nimble CS500.  The CS500 had been replaced with a newer generation all-flash Nimble, and the VMs had been moved a couple of months ago.  Now that the new array was up and had accumulated some snapshots, I was cleaning up the old volumes to repurpose the array.  I noticed, however, that even though all of the files were removed from the datastores, there were still a lot of VMs that “resided” on the old volumes.

VMware addresses this in a KB (2105343) titled “Virtual machines show two datastores in the Summary tab when all virtual machine files are located one datastore.” It suggests that the VM is pointing to an ISO that no longer exists on the old datastore.

After looking at the configs, I realized that, sure enough, some of the VMs were still pointing to an ISO file that was no longer on that datastore.  Easy, right, except that on one of the test VMs, I set the optical drive setting back to “Client Device.”  It was still pointing at the old datastore.

Looking through the config again, I noticed that the Floppy Drive setting is missing from the HTML5 client.  I fired up the Flex client and set the floppy drive to “Client Device” as well.  Still no go.  For the few VMs that were pointing at a nonexistent ISO, setting the optical drive back to “Client Device” worked, but for VMs that were pointing at a nonexistent floppy, changing the floppy to “Client Device” wasn’t working.  A bug in the floppy handling?  Perhaps.

I created a blank floppy image on one of my new datastores and pointed the VM’s floppy to that new image.  Success!  The VM was no longer listing the old datastore, and I could then set the floppy to “Client Device.”  After checking out other VMs, I realized that I had over 100 VMs that had some combination of optical drive or floppy drive pointing at a non-existent file on the old datastores.  PowerCLI to the rescue!

$vm = $args[0]

$cd = get-cddrive -vm $vm
floppy = get-floppydrive -vm $vm

set-floppydrive -Floppy $floppy -FloppyImagePath "[datastorename] empty-floppy.flp" -StartConnected:$false -Confirm:$false

set-floppydrive -Floppy $floppy -NoMedia -Confirm:$false
set-cddrive -cd $cd -NoMedia -Confirm:$false

Simply save this as a .ps1 file and pass it the name of the VM (in quotes if it contains spaces.) It will get the current floppy and CD objects from the VM, set the floppy to the blank floppy image previously created, and then set both the CD and floppy to use “NoMedia.” This was a quick and dirty script, so you will have to install PowerCLI and do your own Connect-VIServer first. Once connected, however, you can either manually specify VMs one at a time or modify the script to get VM names from a file or from vCenter itself.  All of these settings can be changed while the VM is running, so there is no need to schedule downtime to run this script.

After all of this work, I found that there were still a few VMs that were showing up on the old datastores.  Another quick Google search revealed that any VMs that had snapshots that were taken with the CD or floppy mounted would still show up on that datastore.  Drat!  After clearing out the snapshots, I finally freed up the datastores and was able to delete them using the Nimble Connection Manager.

So now a little root cause analysis: why were there so many machines with a nonexistent CD and floppy mounted?  After seeing that they were all Windows 2016 VMs, I went back to our templates and realized that the tech who built the 2016 template left the Windows ISO (long since deleted) and floppy image (mounted so he could F6-load the paravirtualized SCSI driver during OS installation) mounted when he created the template.  I converted the template to a VM, removed the two mounts (using the same two-step method for the floppy) and converted it back to a template.

With that job done, I’m continuing to plug away at converting our other vCenters from 6.5 to 6.7U1.  Have a great day, everyone!

The Dell PowerEdge FX2: A Dead End?

A small crowd was gathered by the server enclosure at the edge of the Dell EMC VMworld booth, gawking at the blinkenlights on the newly announced MX7000 blade chassis.  I dodged the eye candy and the suited salesdroids surrounding it and instead searched the booth for their PowerEdge FX2 display.  Alas, the entire booth was dedicated to the 7 rack unit MX7000 and its array of server, storage and networking building blocks.

When the FX2 was announced in 2014, it held a lot of promise.  A 2U enclosure with horizontally-mounted blades, the FX2 could hold up to eight dual-processor Xeon servers, consolidating power supplies, cooling and management.  The FC430 blade crammed two Xeons and 8 DIMM slots into a quarter-width sled.  The FC630 blade offered three times as much RAM capacity, access to another PCIe chassis slot, and a “real” PERC controller in a half-width sled, and the FC830 was a rack-width quad-Xeon sled with more DIMM slots and extra drive bays.

With the introduction of the 14th generation Dell PowerEdge lineup, the FC630 half-width blade was replaced by the FC640, but the FC830 and FC430 were never updated.

Seeing the writing on the wall, I grabbed one of the Dell EMC guys that was floating around the booth and asked him about the future of the FX2.  He told me that the FC430 and FC830 wouldn’t be updated because the “processors were too big”, and that he couldn’t comment on the future of the FX2 platform further.

Now, I’ve been in IT for a while, but that explanation just seemed strange.  What did he mean by the “processors are too big?” They need a bigger heatsink because they run hotter?  They’re a physically bigger die?  After digging a bit, the answer appears to be “both of the above.”

The Broadwell-based Xeons that underpin the 13th generation PowerEdge use the 37.5mm by 37.5mm LGA1151-3 socket–a socket size that has been carried over virtually unchanged since the 35mm by 35mm Socket 478 of nearly two decades ago.  However, the Skylake-based “Purley” Xeons that power the 14th generation servers use a much larger socket called LGA3647, so-called because it has nearly 2,500 more pins than the old 1151-3.  Those extra pins are used for additional memory channels among other features.

Those extra pins mean that the actual socket had to grow–in this case, to 76mm by 56.5mm.  That’s about three times the real estate of the previous generation CPU.  The FC430 blade was already so tight that it gave up DIMM slots and the option of having a decent RAID controller to fit in the space allotted.  There is simply no room to fit a Purley Xeon without radical rework like breaking off a lot of support hardware onto a daughterboard, which complicates manufacturing, repair and cooling.


A PowerEdge FC430 blade.

So, I now understand the demise of the FC430, but what about the future of the FX2 as a platform?

The folks over at Wccf Tech have posted Intel documents about the Xeon roadmap.  The Cooper Lake and Ice Lake chips, expected in 2019 and 2020 respectively, are expected to use a socket called LGA4189.  With 15% more pins than the LGA3647, I would expect the socket to be 15-20% larger as well.  While that doesn’t sound like much, it may be just too much to fit in a dense enough blade for the FX2 to continue to make sense.

Heat will also be a factor.  The same source above shows that Ice Lake will have a TDP of “up to 230W.”  That’s 75 watts higher than the hottest processor Dell offers in the FC640.  To handle the more powerful Ice Lake processors will probably take larger heatsinks and better cooling than is possible with any of the current FX2 form factors.

So how will these larger and hotter processors affect the blade landscape?  Dell knows the answer, and the answer lies in the MX7000.  Dell’s newest blade architecture, just released to the public in the past couple of months, is your traditional big box with vertical blades in the same vein as the old M1000e.  However unlike the M1000e, which offered full-height and half-height blades, the MX7000 only offers full-height single-width (2 CPU) and double-width (4 CPU) options.  Again, the helpful salesguy at the Dell EMC booth said that they were “unlikely” to be able to offer half-height blades because the days of 125 watt 37.5mm by 37.5mm CPUs are over.

So, are blades still worth it?  The M1000e could fit 16 servers into 10U and the FX2 can still fit 20 current generation servers into 10U.  The MX7000 can only fit 8 servers into 7U, and Lenovo can fit 14 servers into 10U.  The FX2 offers very high density in a form factor that is more flexible because it can be purchased and racked in smaller increments than the larger blade systems, but it appears that the days may be numbered for that form factor.