Config cockup leaves Reg reader reaching for the phone

Who, Me? Facebook went down and Twitch flashed its privates last week thanks to alleged config cockups. However, who among us has not suffered the stomach-dropping fear that follows the ill-advised submission of a seemingly innocuous command?

Welcome back to Who, Me?, The Reg's weekly column that recounts personal tales of catastrophe.

Memories of today's story, courtesy of "Kildare", was triggered, in his words, by "the recent period of world peace brought about by Facebook's absence."

It was the early 2000s, and Kildare was the network manager of a large London hospital. He and his team were working through a network upgrade: "Frankly," he said, "it was so bad that almost anything that was done would count as an upgrade!"

The routers were being changed out, and a Cabletron box had replaced one of the big boys. An SSR 8600 to be precise.

This was before Cabletron was hacked to pieces. Of the kit, Kildare observed: "These things were a bit of a brute, but were a major step forward from the old Retix device (Max 6 X 10Mb interfaces and it used RIP2 – ah those were the days!)."

The SSR differed from Cisco hardware in a number of ways. Perhaps the most important was how configuration worked. "When changes were applied to the command line they were applied directly," explained Kildare, "rather than being held in a buffer and applied with a second command."

"Now," he added, "I expect frequent readers of this column will at this point think 'What could possibly go wrong?'"

It was not so much "what", though. More "when".

And the "when" arrived with the inevitability of a Lego brick to the soft foot of a careless parent.

"All the changes had resulted in a section of config no longer being required," said Kildare. "Line numbers were created when the config was displayed, so a command along the lines of delete 225 – 235 would do the job."

Easy, right?

Not quite.

"Unfortunately – (big word that) that's not quite what I typed: what I actually typed was delete 225 – 135."

• Things that are not PogChamp: Amazon's Twitch has its source code, streamer payout data leaked
• Facebook rendered spineless by buggy audit code that missed catastrophic network config error
• Cloud load balancer snafu leads to 3D printer user printing on a stranger's kit
• See that last line in the access list? Yeah, that means you don't have an access list

Kildare stared aghast at the screen, fervently hoping the router would spot the mistake.

It did not.

What it did was wipe all the lines between the numbers he'd typed. Numbers that, alas, included the config to permit connection to his PC.

Did we mention that the suddenly very, very lonely router was located a mile away?

Fortunately for Kildare, this was in the era before IP telephones ruled the roost and he was able to place a call to a colleague local to the stricken device.

He hurriedly dispensed instructions on how to recover the borked device and the day was saved. "But I did," he told us, "learn to re-read everything I ever typed VERY carefully before hitting the enter key."

This hack's moment of arse-swooping horror involved carelessly highlighting an UPDATE SQL statement and missing the WHERE clause before hitting Ctrl-e. How about you? Share your own bit of Facebook-esque silliness with an email to Who, Me? ®