Who, Me? The right tool for the job is a motto to live by. But in this week’s Who, Me? a Register reader recalls what happens when the wrong tool is used by a right tool.
Today’s story, from “Keith”, takes us back to the heady days of the 1990s and the expensive funnelling of market data to customers keen to maintain their edge.
The markets were about to close on the day in question when all the alarms in the world seemed to go off at once on Keith’s console. Important customers had suddenly lost connectivity and that vital data had stopped flowing. He glanced at the time – maintenance windows never opened earlier than half an hour after the markets had closed – so something bad must have happened.
He sprinted to the communications room, flung open the door … and found the head of comms standing sheepishly beside a rack of suddenly dead modems and multiplexers and a screeching UPS. A screwdriver was in his hand. Oh dear.
The targeted job? Changing a modem. Simple enough, but thanks to the evolutionary manner in which the cabling had been built up over the years, not as straightforward as one might have hoped. All the power leads had been cable-tied into a bundle.
No matter, the comms boss had decided to get ahead of the game and remove the cable tie. Sure, the markets were still open, but it wasn’t as if he was going to unplug anything, right? Just a bit of prep work.
Of course, the cutters were on the other side of the room. But he had a screwdriver, so to save time he had opted to break the tie by just inserting the screwdriver, twisting, and … oops.
“The tip of the driver had slipped,” said Keith, “and cut through a random power cord and sent a spike through that whole rack, and the UPS was complaining about an over/under voltage spike, shrieking to wake the dead.”
Usually, the circuit breaker in the power strip would have prevented too much damage, but so severe was the shock that even that had been broken. Keith’s team broke into two groups – one to find a spare strip and reconnect it, the other to recover the hardware.
“Many of them were single power,” he said ruefully. “Not only that, but almost every device had a blown fuse in it.”
The newer devices had their own circuit breakers, but much of the kit was of the “replacement deferred due to budget” variety; technically still within their lifespan, but very long in the tooth.
“Not a problem, right?” thought Keith. “We had a box full of fuses lying around somewhere?”
No, they did not. What they did find – after scouring the stores, closets, and basements – were fuses of entirely the wrong type and rating. In the end, decommissioned devices were torn open and their fuses scavenged.
“I think we finally got that rack fully back up at about 8pm, some four hours after the market close,” recalled Keith, “and that had included upgrading two devices from new install hardware that had been slated for other use.
“Several devices required a reconfiguration by hand since of course we had no backups of the configs, except a few random printouts.”
The following morning a meeting was convened that featured the very biggest of cheeses, including the CEO and president. As head of operations, Keith was present. As was Sir Screwdriver himself, the head of comms.
Unsurprisingly, the action items came thick and fast. “Comms and ops had to check each other’s wiring for compliance to the standards,” remembered Keith, “which I got the job of writing down from the word-of-mouth lore that we had been using.”
The work included checking out the devices and power supplies. Not a problem for the 100 or so servers in the ops room and their tidy cabling. It was an altogether different challenge for Keith in the comms room and its less-than-structured approach to cable management, however. He turned up a dozen miswired devices and racks running from two power strips on the same UPS – all thankfully fixed and documented before Sir Screwdriver could return to his time-saving ways.
Some things really weren’t better in the old days.
Have you ever stuck a screwdriver where you shouldn’t? Or tried to recover a configuration located only in the memory of the longest serving member of staff? Tell us your story with an email to Who, Me? ®
A time when cabling was not so much ‘structured’ than ‘survival of the fittest’
Source: Expert Gwapo Pinoys
0 Comments