It seemed like a simple request. All I wanted was for my Linux virtual machine to stay at the 1920×1080 resolution I specified. VMware Tools, however, had different ideas and resized my VM’s screen resolution every time I resized the window in the remote console. Normally this is no biggie, but users were connecting into this box with VNC and the VNC server was freaking out when the resolution changed. Google has dozens of blog posts and forum questions about resolving this issue in Windows VMs and in VMware Fusion/Workstation, but I couldn’t find anything on the fixing this in ESXi 6.7 even after an hour of searching.

Finally I dug into my local filesystem and found the culprit: a dynamic library called libresolutionKMS.so. On my CentOS 7 VM, it was located in /usr/lib64/open-vm-tools/plugins/vmsvc. Like the Windows equivalent, moving or deleting this file and rebooting was sufficient to stop this issue in its tracks. In my case, I moved it back to my home directory to make sure nothing broke, but it appears that this stopped the auto-resizing without affecting any other operations, and VMware still reported the VMtools as running.

As this is Linux, open-vm-tools is updated by my package manager, so this file will need to be moved or deleted each time after open-vm-tools is updated. Given the lack of any other way to disable this behavior, this seems like a workable kludge for my situation.

Part 1 of this saga can be found here.

As mentioned in the previous post, there several ways to skin the airflow separation cat. Each has its advantages and disadvantages.

Initially we considered raised floor cooling. Dating back to the dawn of computing, raised floor datacenters consist of chillers that blow cold air into the space under the raised floor. Floor tiles are replaced with perforated tiles in front of or underneath the enclosures to allow cold air into the equipment. Hot air from the equipment is then returned to the A/C units above the floor. The raised floor also provides a hidden space to route cabling, although discipline is more difficult since the shame of poor cabling is hidden from view. While we liked the clean look of raised floors with the cables hidden away, the cost is high and the necessary entry ramp at the door would have taken up too much floor space in our smaller datacenter.

We also looked at a hot aisle design that uses the plenum space above a drop ceiling as the return. Separation is achieved with plastic panels above the enclosures, and the CRAC units are typically placed at one or both ends of the room. Because this was a two-row layout in a relatively tight space, it was difficult to find a location for the CRACs that would avoid creating hot spots.

The decision became a lot easier when we found out that one of the spaces we were vacating had APC in-row chillers that we could pick up at a steep discount. The in-row units are contained within a single standard rack enclosure, so they are ideal for a hot aisle/cold aisle configuration. They solved the hot spot issues, as they could be placed into both rows. They also use temperature probes integrated into the nearby cabinets to keep temperatures comfortable efficiently.

With the cooling situation sorted, we turned our attention to containment. We opted for the Schneider/APC EcoAisle system, which provided a roof and end-doors to our existing two-row enclosure layout to create a hot aisle and a cold aisle. The equipment fans pull in cooler air from the cold aisle and exhaust hot air into the hot aisle, while the in-row chillers pull hot air from the hot aisle and return chilled air back into the cold aisle.

There are two options for this configuration.  A central cold aisle can be used, with the rest of the exterior space used as a hot aisle.  This possibly could reduce energy consumption, as only the central aisle is cooled and therefore the room doesn’t need to be sealed as well from air leaks.

The second option, which we ended up choosing, was a central hot aisle.  In our case, the exterior cold aisles gave us more room to rack equipment, and using the entire room as the cold aisle gives us a much larger volume of cool air, meaning that in the case of cooling system failure, we have more time to shut down before temperatures become dangerous to the equipment.

The central hot aisle is covered by a roof consisting of lightweight translucent plastic insulating panels, which reduce heat loss while allowing light in. (The system includes integrated LED lighting as well.) The roof system is connected to the fire system, so in the case of activation of the fire alarm, electromagnets that hold the roof panels in place will release, causing the roof panels to fall in so the sprinklers can do their job.  We can also easily remove one or more panels to work in the cable channel above.

Our final design consists of eleven equipment racks, three UPS racks and four chiller racks.  This leaves plenty of room for growth, and in the case of unexpected growth, an adjacent room, accessible by knocking out a wall, doubles the capacity.

We decided on 48U racks, both to increase the amount of equipment we can carry and to raise the EcoAisle’s roof. To accommodate networking, one enclosure in each aisle is a 750mm “network width” enclosure to provide extra room for cable management. As the in-row CRACs were only 42U high, Schneider provides a kit that bolts to the top of the units to add 6U to their height.

Next time:  Power and cooling

I’m back in my house, in my chair, with my family. I’m so thankful for them and happy and contented to be back home.

But the whirlwind of the past few days hasn’t faded yet from my mind. There are so many notes, so much information, and I so want to fire up the homelab before it all disappears, but I just can’t. My brain needs to unclench first.

This was my first time at VMworld in San Francisco. Unlike the Vegas show last year, Moscone Center is broken up into three distinct buildings, all separated by busy streets. What this means is that it takes 10 to 15 minutes to get from, say the third floor of Moscone West to Moscone South. Because most events and sessions are scheduled for 30 or 60 minutes, it means leaving an event early or arriving late to the next one. While there was some of this in Vegas, usually events were only separated by a floor and a short walk. It’s something I have to take into account when scheduling sessions next year, and it meant that I had to duck out early or skip some sessions altogether.

The sessions I attended were all excellent. I tried to keep my sessions to topics that would help in my current situation, i.e. wrangling thousands of random virtual machines that are in various stages of use or abandonment, managing and deploying VMs with as few touches as possible, and trying to automate the hell out of all of it. To that end, I focused on the code-related sessions as much as possible, and I was not disappointed. VMware and the community are hard at work solving problems like mine with code, and it’s great to have such a variety of tools ready for incorporation into my own workflows.

Additionally, I attended great sessions on vSphere topics such as VMware Tools, ESXi performance, certificate management and vMotion. These not only gave me insight into how these functions work under the hood, they hinted at new technologies being planned for ESXi and vCenter to make these products work better. This was a great relief, as I’ve been concerned for a while that vSphere would go into a more slow maintenance-only product cycle as the push toward cloud increases. I’m happy that VMware continues to invest heavily in its on-prem products.

If there was one word that summed up the overarching theme of this VMworld, it’s Kubernetes. From the moment Pat Gelsinger stepped onto the stage, Kubernetes was the topic at hand. Kubernetes integration will involve a complete rearchitecting of ESXi, and as someone who sees my customers experimenting with using containers for their build processes, I’m happy that VMware is going to make this easier (and faster) to do and manage in the future.

Let’s face it though. Most of the sessions were recorded and will be made available after the show. This is true of most major software trade shows, and if sessions were the only reason to attend, one could reasonably just stay home and watch videos in their pajamas.

It’s the interactions that matter. Being able to ask questions and get clarifications is very important, and I found that valuable for certain topics and sessions. However, the most important thing that you don’t get sitting at home is the interaction with the community.

Last year was my first VMworld. I didn’t know anybody, and I didn’t really know what to do to get the most out of the show. I scheduled sessions to fill every time slot, even if the product or topic wasn’t interesting or relevant. The time that I wasn’t in a session was spent roaming the conference floor collecting swag from vendors. By the time I was done, I had fifteen t-shirts, a jacket, a drone, a game console and more pens and hats than I would ever use. I did attend a couple of parties and met a couple of great people (like Scott Lowe, who I only realized later was the proprietor of the Full Stack Journey podcast I had been listening to on the plane.)

I didn’t have the real community experience until the Dallas VMUG UserCon a few months later. There I met great people like Ariel Sanchez Mora and the leaders of both the Dallas and Austin VMUGs. But it was a talk Tim Davis gave on community that really made me realize how important my participation would be to me. I dropped a few bucks on WordPress.com and threw out and replaced my old Twitter account and started participating. I’ve seen been attending both the Cedar Park and Austin VMUG meetings as well as the Dallas UserCon, and it’s been great having a group of peers to talk with and bounce ideas off of.

Contrast last year’s VMworld to this year’s. This VMworld I collected two shirts (both from VMUG events) and no vendor swag. I visited a couple of vendors that I specifically wanted to talk to but otherwise just did a little browsing. Instead of filling my calendar and stressing out about catching them all, I strategically picked topics that would be relevant for me in the next year.

Most of my downtime between sessions was spent at the blogger tables at the VMware Social Media and Community booth outside the Solutions Exchange. It was a little darker than the VMUG area, and with a lot less foot traffic. There, I could recharge my batteries (literally), blog about the event and catch up on some of the other topics I’ve been working on and chat with the other community members who rotated in and out as they went to their sessions and events. I also got to drop in on vBrownbag sessions being given by community members, which provided some good background learning while I was working.

So what do I plan to have accomplished by VMworld 2020?

1. Build out those automation workflows, VMware integrations and tools that I need to better manage the clusters under my purview.
2. Blog about it.
3. Work with my customers to get their automated ephemeral build and test envrionments working so they don’t have to rely on sharing and snapshotting their current environments.
4. Blog about it.
5. Earn my VCP.
6. Blog about it.
7. Have a presentation ready for vBrownbag.
8. Blog about it.

Thanks to everyone I met this year; it was great.

First things first: some logistics. I picked a less than desirable hotel to save some money. The hotel… okay, let’s call it what it is. The MOTEL in question is on Harrison Street in SoMa. It’s an area where a 1 bedroom loft still sells for $885,000 (I looked one up), but every other parking meter has automotive glass shards all around it. This was the cheapest accommodation within walking distance of the venue where I wouldn’t have to share a bathroom with strangers, but it’s still more than twice what I paid in Las Vegas last year.

I got up at about 6:00 this morning. This was not by choice; it was a couple of gunshots a few blocks away that woke me, and I wasn’t going back to sleep. That was after the people in the room next door chased away a guy who was trying to burgle a truck in the parking lot, not once, but twice. There was also a shouted discussion at around 5:00am between two men about their upbringing and manliness. Needless to say, I didn’t get involved, and I don’t know if the gunshots an hour later were related.

So I packed up my belongings and took a Lyft to the Intercontinental, which is next door to the venue and has reverted back to regular non-gouging pricing for my last night in San Francisco.

There’s a different vibe at VMworld today. The halls have thinned out substantially, and goodbyes are being exchanged. While there are some familiar faces among the die-hards remaining, it’s very clear that VMworld is winding down. That doesn’t stop me from making the most of my time though. I have two back-to-back sessions to attend late in the morning.

The first session was presented by William Lam and Michael Gasch. The VMware team has collaborated on an event engine called VEBA, an appliance that brokers events from vCenter to OpenFAAS to allow event-driven functions. They demonstrated using VEBA and functions to trigger actions from events, such as posting a message in Slack when a critical VM is powered off.

The second session dove into automating vSphere through the PowerShell CLI. This demo was put on by Kyle Ruddy, who focuses heavily on using code to automate VMware.

I then made my way back to the VMware community booth, where I could comfortably catch up on my various feeds, plug in to charge, and type. I actually fly out in the morning since trying to leave for the airport at 4:00pm on a Thursday to fly out tonight is the perfect way to end up stuck overnight at an airport. Besides, I need the sleep.

With the general sessions behind me, today would be all about breakout sessions and networking. My first session was at the VMware {CODE} alcove. Knowing that there are issues being able to see the screen from the back, I got in line early to get a front row seat. The session did not disappoint. David Stamen has written a comprehensive suite of Powershell scripts to automate the management of VCSA patches, including querying for available patches, downloading, staging and deploying. Tasks can be run on multiple vCenters simultaneously, so vast swaths of vCenters can be updated simultaneously with a couple of commands.

In the second session, Kyle Ruddy and JJ Asghar demonstrated how to automate template builds and modifications using Packer, Jenkins, Ansible and InSpec. This was another one of those whirlwind sessions that covers a lot of ground in a very short time. They have made the scripts they use available, but realistically, it was more of a proof of concept demo than something one could take home and immediately implement. However, they do provide the scripts they use, and some blog searching could probably help cobble together such a setup.

The third session covered the mechanics of SSL certificates in vSphere, including best practices regarding maintaining certificates. Finally, a tech preview hinted at some utilities to make certificate management easier in a future release. Finally, a great session about VMware Tools: what they are, what they consist of and how to manage updates and maintenance. Again, the tech preview hinted at future technologies to make updating and using VMware Tools easier.

The evening ended with beer and Billy Idol at the Bill Graham Civic Auditorium. What could be better?

This time I was smart. Rather than being crammed into a room with thousands of other people to watch the Tuesday keynote on a screen, I instead made my way to the Social Media and VMware Community area in the lounge and grabbed a seat at the blogging table. Not only could I watch the keynote while blogging, chatting with others and keeping my devices topped off.

The second general session expanded upon the previous day’s session and presented examples of fictional companies using VMware’s Tanzu offerings to produce its fictional products. The presentation explained how Tanzu could cover all of the theoretical technological needs of business, from IoT to edge to datacenter to networking.

My first session of the day was a deep dice into the mechanics of vMotion on vSphere. Detailed hints for tuning vSphere to optimize the vMotion process were also provided, including NIC and vmkernel settings. The presenters also hinted at future functionality that would allow vSphere to be more intelligent about handling both local, remote and storage vMotions to eliminate the need to tweak it.

Back at the Social Media and VMware Community booth, I watched Ariel Sanchez Mora give a presentation on how to start and run mini local hackathons. As someone who is working to get more involved with the Austin-area community as well as improve my coding skills and solve some real issues in the enterprise, I am looking into doing this soon. Right after, Gabriel Maentz presented a talk about using vRealize and Terraform to improve automation in the enterprise.

Finally, I attended a briefing by HPe about some new initiatives, including Greenlake, which uses a consumption-based model to provide a fully-managed platform that extends from on-prem to cloud providers like Google Cloud. More directly applicable to me, though, it improved Nimble VVol support coming in the near future, and Nimble Cloud Volumes, which allow replicating data between cloud providers and Nimble Arrays with no egress charges. Also discussed was something I had heard about previously from my current Nimble rep: Nimble dHCI, which allows automatic provisioning of Nimble storage arrays and HPe Proliant servers into the dHCI “stack”, reducing operator work, while providing a single support point for issues anywhere in the stack. And unlike some other solutions, standard HPe arrays and servers can be ordered to grow the stack, rather than having custom hardware.

After a show and drinks, I hit the hay, preparing for a busy Wednesday.

I was awakened right on time by a fire truck screaming past my window. After taking care of my morning tasks, I packed up my tech and headed to Moscone South. We were herded into a massive theater for the General Session, where VMware CEO Pat Gelsinger lays out the roadmap for the next year of VMware. I had heard about VMware’s intended acquisition of Pivotal and Carbon Black, so I was interested in how those products were being positioned in VMware’s portfolio.

Gelsinger announced Tanzu, a new product from VMware that combines the technologies from multiple acquisitions, including Heptio, Pivotal and Bitnami, into a complete container workload management, management and security ecosystem.

The management layer will be provided by Tanzu Mission Control, which is a management platform that will manage Kubernetes, whether deployed on prem, in the cloud, or in some combination of both.

I was stunned by the announcement of Project Pacific, VMware’s headlong dive into the Kubernetes pool and the backbone of Tanzu. The fact that VMware would be involved in Kubernetes was not the shocking part; any platform company not getting involved will be left behind. What stunned me is that Kubernetes will be deeply integrated into ESXi itself. It’s the first major rearchitecting of ESXi in some time, with containerization and Kubernetes at its core. According to Gelsinger, the new architecture of ESXi will allow containerized workloads to run up to 8% faster than running on Linux on bare metal. This is an incredible result, and speaking as an IT practitioner who will soon be deploying containers on our vSphere cluster, this news is very exciting. (Of course, the best part is that I can roll out Kubernetes today and then re-deploy workloads onto Tanzu when it becomes available.)

After the General Session, I proceeded to the VMware {CODE} classroom for a quick how-to and demo about using ‘kubeadm’ to deploy Kubernetes. In the VMware Community booth, community member Marc Huppert gave a talk about the evolution of homelabs over the years and Jaap Brasser gave a talk about the basics of using APIs to automate vSphere. In the Performance Best Practices session, there were many tips about vSphere performance, including how server power management settings and guest page sizes affect vSphere performance and ability to manage memory usage as the hardware approaches its limits.

Finally, I attended a session about certificate management in vSphere. This was a technology preview and subject to change of course. Discussed were new changes designed to make certificate management in vCenter easier, starting with a reduction in the number of certificates required and the ability to deploy certificates using an API. Also discussed were the support of external identity providers using OAuth (including Ping and Okta) or ADFS, with SAML coming later. Interested persons can sign up for the beta on VMware’s site and it should soon be available.

After dinner, I wound down for the night. Rubrik was having a blowout at a nearby night club, but those aren’t really my scene. I headed back to the room and turned in.

The first day of VMworld is largely a day of logistics. After stashing my laptop behind the microwave (my motel room was not particularly secure) and double-checking Google Maps, I began my half mile trek to Moscone Center. After several days of blistering temperatures in Austin, the cool breeze of San Francisco was a welcome relief. The check-in process went much more quickly than last year in Vegas, and I was soon back on the street heading for Moscone West to get my VMworld backpack and water bottle.

First on the agenda was Opening Acts, put on by VMunderground at a nearby venue. The event brings together technologists to meet and talk. Opening Acts also put on several panel talks with industry experts and veterans, who discussed topics pertinent to IT careers, including pay, equity, benefits, certifications and education, and work-life balance.

After a quick lunch, I proceeded to the Hands on Labs area, where I was able to access VMware’s virtual labs to work on virtualized labs to learn and experiment. While not as permanent as a homelab, it’s a great way to get guided help on specific topics, and since these labs are not locked down to the topic at hand, they allow users to experiment or try out the installed software and virtualized hardware without permanently breaking anything.

As evening drew near, I headed to the VMUG Member Party at SPiN!, a local taphouse. Snacks and drinks were provided, and we were treated to not only VMware COO Sanjay Poonen making an appearance, but Michael Dell himself showed up to thank the VMUG membership for our support. After everything settled down, I headed back to Tabletop Taphouse for the VMunderground 2019 event. There I played tabletop games with some new friends.

The cool breeze had turned sharp and cold by the time we left the venue, and I grabbed a Lyft back to the hotel, double-checked the next day’s schedule, plugged in all of my technology to recharge and drifted off to sleep.

I don’t have to do a lot of business travel, so when I do, I try to make the most of it by finding something local to do. Whether it’s touring the city, going to a museum or just sampling the local cuisine, I do my best to to get in some “me time.”

And so I found myself booking a flight to my second VMworld in San Francisco. I wasn’t interested in seeing the usual sights–I lived here for over two decades and have spent plenty of time in the Bay Area at the handlebars of a Honda Helix.

One thing I hadn’t seen, however, was the Computer History Museum in Mountain View. I typically arrive at these events the day before so I can beat the crowds, check in to my hotel and get a good night’s sleep and a shower before the big event. In this case, I flew in to San Jose, caught a quick Lyft and bought my ticket. The CHM offers a free bag check, so I dropped off my big duffel and began wandering.

This museum is massive and I got there just as they were opening. The walking path winds through the museum exhibits in historical chronological order, starting with abacuses and slide rules.  Following the path leads to mechanical and electromechanical calculators typically used in finance and engineering, as well as mechanical  cash tills. 

Next up are mechanical tabulators and calculators that use punched cards. On display are a multitude of machines that sort, count and duplicate punched cards. Several early analog computers are on display as well. Finally, the first electronic computers come into view. While most artifacts are of pieces of early computers or peripherals (including a massive Univac control console), the jewel in this part of the exhibit is the JOHNNIAC, a very early vacuum tube computer built in the early 1950s by the RAND Corporation for its own internal use. Besides being one of the longest-lived early computers (with over 50,000 hours on its hour meter), it was also upgraded several times during its lifetime, including core memory and additional instructions.

In the Memory and Storage exhibit, the entire history of storage technology is on display, from vacuum tubes and drum storage to hard drives and flash. Of interest is the progression of hard drives from washing-machine sized peripherals weighing hundreds of pounds to CompactFlash-sized microdrives. Other artifacts on exhibit here include game cartridges, Zip disks, optical disks and floppies.

The Supercomputers exhibit contains something I didn’t think I would ever get to see: a Cray 1 supercomputer. The familiar ‘sixties piece of furniture’ look is there, but the unit was a lot smaller than I thought it would be. Several other supercomputers, including a piece of a Cray 2, are on display here.

Past the supercomputers, a multitude of minicomputers are on display. Among them are the iconic PDP-8 and PDP-11 systems from DEC, as well as the DEC VAX and other systems.

I was running a bit late at this point, so I sped a bit through the rest of the exhibits. The personal computer era was more my era, so other than repeating waves of nostalgia, I had seen most of that exhibit before. The usual suspects were present (Commodore PET, VIC-20 and 64, Atari 800, early Apples and Macintoshes, etc.), as well as an original PC clone made by PCs Unlimited, a little company founded by Michael Dell and later renamed after himself. There were also dozens of artifacts I remember from my childhood, such as the Speak and Spell, the Heathkit HERO robot, the Tomy robot, Colecovision, etc. It’s a must-see for any Gen X’er.

The exhibit comes to a conclusion with the dot-com bust that happened in about 2000, and perhaps the poster child for the dot-com bubble, the Pets.com sock puppet. Many though of Pets.com as the prime example of hubris of dot-com executives who thought that they could profitably ship cat sand and 50-pound bags of dog food to customers anywhere in the US for free.

Now to the best bit. CHM has two fully functioning IBM 1401 computers, and every Wednesday and Saturday, docents, all of whom worked on these systems back in the day, fire them up and do demonstrations. On this visit, one of the docents was Paul Laughlan, who also was known as the author of Apple DOS, Atari DOS and Atari BASIC. (Paul’s wife wrote the Assembler/Editor for Atari home computers.) As a lifelong Atari fan, I was a little tongue-tied, but we talked quite a bit about the past. He then allowed visitors to type their names onto punch cards at a punching station and fed them into the 1401 to get a souvenir printout. I did get a picture of their PDP-1, though there were no demos that day. (The PDP-1 is turned on one Saturday a month and visitors can play Spacewar! against each other.)

There’s an entire additional wing of the museum dedicated to programming languages and software culture, but I bade my farewell and grabbed a Lyft into San Francisco to get ready for Day 1 of VMworld. It was an item checked off my bucket list and a lot of fun.

I was recently tasked with decommissioning an old Bugzilla instance.  The company had long since moved to Jira’s SaaS offering and the old Bugzilla VM had been left to languish in read-only mode.  Now two full versions behind, the decision was made to attempt to export the Bugzilla data and import it into Jira rather than continue to update and support the aging software.

Unfortunately, there are not a lot of great options out there for moving data to an active Jira Cloud instance.  It’s trivial to connect a local Jira instance to Bugzilla and import all of its data.  However, that option is not available in Jira Cloud.  There are three options for getting Bugzilla data into Jira Cloud:

1. Export the Bugzilla data as CSV and import it into Jira Cloud.  Attachments cannot be imported; they must be put on a separate web server and the URLs inserted into the CSV before import into Jira.  In our experience, notes were also not coming in, but due to the unpalatability of running a separate webserver for attachments, we didn’t pursue this further.
2. Export the existing Jira Cloud instance and dump it into a local Jira instance.  Use the Jira import tool to merge in the Bugzilla data, and then export the local Jira instance and re-import it back into a Jira Cloud instance.  This would involve a substantial amount of work and downtime on a very busy Jira Cloud instance, and would involve a bit more risk than we were willing to take on.
3. Export Bugzilla into a clean local Jira instance, and optionally then export it to a separate Cloud instance.  This would involve paying for an additional Jira Cloud instance, and it would be an extra system to manage and secure.

Because this is legacy data (regular use of Bugzilla ended several years ago), I was given the green light to export the roughly 30,500 bugs to searchable PDF files and store them on a shared local folder.  We would then be able to decommission the Bugzilla VM entirely.

Pressing CTRL-P 30,000 times

Bugzilla has a print preview mode for tickets.  It includes all of the data in the ticket including the header fields and date-stamped notes.  Fortunately, it turns out that Bugzilla, being an older “Web 2.0” product, will pull up any ticket by number in print preview mode straight from the URL.  The format is:

https://bugzilla.example.com/show_bug.cgi?format=multiple&id=2000

where ‘2000’ is the bug number. With that in mind, it was time to generate some PDFs.

After experimenting with Google Chrome’s ‘headless’ mode, I was able to write a quick batch script to iterate through the tickets and ‘print’ them to PDF. Here’s the code:

for /l %%i in (1001,1,2000) do (
  mkdir c:\bugs\bug%%i
  "c:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --headless --disable-gpu --user-data-dir="c:\users\tim\appdata\local\google\chrome\user data" --ignore-certificate-errors --disable-print-preview --print-to-pdf="c:\bugs\bug%%i\bug%%i.pdf" "https://bugzilla.example.com/show_bug.cgi?format=multiple&id=%%i"
    )

This code is a bit messy since the verbosity of the third line makes it very long. The ‘for’ in the first line specifies the range of bug numbers to loop through. In this case, I’m printing bugs 1001 through 2000. The second line creates a folder for the bug. I put each bug in its own folder so that the attachments can be put in the same folder as their corresponding PDF.

The third line calls Google Chrome in ‘headless’ mode. I used the ‘–disable-gpu’ and ‘–disable-print-preview’ options to fix issues I was having running this in a Server 2016 VM. The ‘%%i’ is the bug number and is passed in the URL to bring up the bug in Bugzilla. Note that I used ‘–ignore-certificate-errors’ because the cert had expired on the Bugzilla instance (ignore if not using TLS.) Possibly due to the version of Bugzilla we were running, headless Chrome would lose its cookie after a few connections. I resolved this by temporarily turning on anonymous access in Bugzilla so the bugs could be viewed without having to log in. (This was an internal system, so there was no risk having this open for a couple of hours.)

While I could easily just plug in ‘1’ and ‘30500’ into the ‘for’ loop and let it run for a few days, this batch file was taxing only one core on my local system and barely registering at all on the Bugzilla host. Since I had eight cores available, I duplicated the batch file and ran ten copies simultaneously, each pulling down 3,000 bugs. This allowed the local system to run at full load and convert more than 30,000 bugs to PDF overnight.

Attach, mate.

Our version of Bugzilla stores its attachments in a table in the database and would have to be extracted. Fortunately, Stack Overflow came to the rescue. The below SQL script parses the attachments table and generates another SQL file that extracts the attachments one at a time.

use bugs;
select concat('SELECT ad.thedata into DUMPFILE  \'/bugs/bug'
, a.bug_id
, '/bug'
, a.bug_id 
, '___'
, ad.id
, '___'
, replace(a.filename,'\'','')
, '\'  FROM bugs.attachments a, bugs.attach_data ad where ad.id = a.attach_id'
, ' and ad.id = '
, ad.id
,';') into outfile '/bugs/attachments.sql'
from bugs.attachments a, bugs.attach_data ad where ad.id = a.attach_id;

I created a temporary directory at /bugs and made sure MariaDB had permissions to write to the directory. I saved this as ‘parse.sql’ in that directory and then ran it with:

mysql -uroot -ppassword < /bugs/parse.sql

After looking at the freshly created /bugs/attachments.sql to verify that paths and filenames looked good, I edited it to insert a ‘use bugs;’ line at the top of the file. Then I created a quick shell script to create my 30,500 directories to match the directories created by my PDF script above:

#!/bin/bash
for i in {1..30500}
 do
   mkdir bug$i
 done

After running that script, I verified that all my directories were created and gave write permission to them to the ‘mysql’ user. It was then time to run my attachments.sql file:

mysql -uroot -ppassword < /bugs/attachments.sql

A quick ‘du –si’ in the /bugs folder verified that there were indeed files in some of the folders. After confirming that the attachments were named correctly and corresponded to the numbered folder they were in, I used ‘find’ to prune any empty directories. This isn’t strictly necessary, but it means fewer folders to parse later.

cd /bugs
find . -type d -empty -delete

Putting it all together

The final step in this process was to merge the attachments with their corresponding PDF files. Because I used the same directory structure for both, I could now use FileZilla to transfer and merge the folders. I connected it to the VM over SFTP, highlighted the 30,500 folders containing the attachments, and dragged them over to the folder containing the 30,500 folders created during the PDF processing. FileZilla dutifully merged all of the attachments into the PDF folders. Once completed, I spot-checked to verify that everything looked good.

I wouldn’t have used this technique for recent or active data, but for old legacy bugs that are viewed a couple of times a year, it worked. The files are now stored on a limited-access read-only file share and can be searched easily for keywords. It also allowed us to remove one more legacy system from the books.