Firmware Updates

I don't know about the rest of you, but firmware updates are one of the most scary things for me in terms of I.T. administration tasks. It probably goes back to the days when firmware images needed to be placed on a floppy disk and the system booted to that disk. I'd hear the disk chunking away and, as the message on the screen warned, "DO NOT POWER OFF YOUR MACHINE DURING THIS PROCESS," the drive would start making that unmistakable sound that only a drive unable to read from its 3.5 inch (8.9 cm) media can make. I just knew that I was looking at a very expensive doorstop at that point.

More times than not however, the update would proceed just fine and I'd wipe the sweat from my brow, let out a sigh of relief thinking, "Whew... Dodged another bullet."

I may have been scarred for life by the few failures I've encountered though. One that sticks out in my mind was adding a second processor (Pentium Pro!) to a Compaq Proliant 5000 series. Compaq required updating the system with its own HAL for Windows. Of course, at the time, this was only available on floppy disk. Let's just say it didn't go too well and it was one of those all-nighter fixes.

Recently, I've been upgrading firmware on some IBM storage solutions as well as Dell blade servers and their associated enclosures. I still get the sweats, my heart beats a little harder, and I don't poop right for days before the upgrade is going to take place. It's been years since I've had a firmware upgrade cause any major problems but I'm still conditioned to be frightened to death of them.

The IBM update was the most scary because things had to be done in a certain order. First were the actual drives themselves. We've got seven shelves of 16 drives comprised of disks from three different manufacturers and a total of four different models. All I/O needed to be stopped on these disks meaning the 15 servers all using that storage needed to be shut down. Once that happened, the firmware on each drive could "safely" be updated.

The thoughts racing through my mind included the sheer amount of data I was talking about here if these updates trashed the drives, how long would it take to recover from tape once the SAN was in usable condition again, and would the troubles somehow be a result of something I did wrong?

Oh, did I mention that I'm doing this all from my basement on a system that's 50 (80.5 km) miles from my house? Here's hoping the remote access cards with the power-on capability work!

After the drives were updated, each of the two controllers in each of the enclosures needed to be updated. After that was firmware on the actual SAN controllers as well as the NVSRAM.

It took only about 2 hours and went off without a hitch. Everything powered back on correctly and we were up-and-running again shortly after the updates completed.

How many years will it take before I forget about the handful of times I got burned by a firmware update?