Early in my career I was asked to help rearchitect a global law enforcement network. Six continents, tens of thousands of switches, thousands of routers, and a design that seemed to have been assembled by a committee, then corrected by a second committee that hated the first. With prejudice.
My job was to make it simple. I was young, which in network architecture is known as having fresh eyes, and I was extremely good.
I’d like to share my methods so you can get the same results.
Troubleshooting is for other people
There are two kinds of work on a network. Design happens at a whiteboard, with good smelling markers, all while people nod at you. Troubleshooting happens in a terminal window, and nobody nods at all.
A serious architect should spend as little time as possible on the second kind. When a problem stands between you and the whiteboard, the professional approach is to get through it fast. Reading each line carefully is a fine habit for people who have are slow.
The add keyword is optional
One day I needed to find out whether a data path was available. All I had to do was add a VLAN to a trunk port. A junior engineer would type this:
switchport trunk allowed vlan add 412
Notice the word add, sitting in the middle of the line, slowing everything down. I prefer the streamlined version:
switchport trunk allowed vlan 412
Some will tell you the first command adds a VLAN to the list, while the second replaces the entire list with the one VLAN you typed. I’ve looked into this since, and I still find it pedantic.
‘reload in’ is for the weak
Before touching a remote device, some engineers type reload in 10. If they lock themselves out, the box reboots after ten minutes, the unsaved change disappears, and the old configuration comes back. If everything goes fine, they cancel it.
This is the equivalent of leaving yourself a note saying you might be wrong. I did not leave myself a note. I had real work waiting.
Results
I pressed Enter, and my session stopped responding.
The data path, I can now report, was not available. Neither was Europe.
The trunk had been carrying everything. It was now carrying VLAN 412, which was not the VLAN I was logged in through. With one line I had removed more complexity than the previous decade of engineering combined. An entire continent no longer had a single traffic problem.
I’d been asked to simplify the network. I want that on the record.
Stakeholder outreach
With the network in this improved state, I began a proactive stakeholder engagement campaign. I called everyone I had a number for.
It was between 2 AM and 4 AM in Europe, which I’ve since learned is when Europeans are at their least engaged. Many stakeholders were unavailable. Several were unavailable repeatedly.
My demeanor throughout was one of calm urgency. Reports that I was shitting myself are accurate only in the metaphorical sense.
The executive briefing
After some hours, someone answered, and I delivered a crisp executive briefing. It covered the trunk, the VLAN, the elapsed time, the continent and the end of my career, all in one breath. Saving literal seconds.
This person had been asleep. They knew nothing about any of it, and I attribute the poor reception to a lack of preparation on their part.
They asked me to slow down and start again. I agreed, to humor them. I told it again at about a third of the speed, and for reasons I still can’t explain, it worked much better.
They said they understood and would drive in. They did not seem upset. I assumed they had missed the severity, because to them this was apparently a drive to work.
Resolution
They arrived and rebooted the router. I had not saved my change, having lost the ability to do so at the exact moment I made it. I consider this a built-in safety feature of my approach. The router came back with its old configuration, and Europe came back with it.
Some readers may note this is exactly what reload in 10 would have done by itself, in ten minutes, with nobody’s phone ringing. I don’t see what that proves.
Post-incident review
Leadership took a keen interest in my work. There were a number of very large conversations. The findings:
- Root cause: A VLAN was insufficiently added.
- Contributing factor: The word
add, which is easily missed by anyone not reading it. - Contributing factor: Europe.
- Corrective action: Establish a Configuration Control Board to review all future changes.
I arrived to take complexity out of that network. I would like it known that I did eventually add something. It has a charter and meets regularly.
I also left that assignment with a rule I’ve repeated ever since: if you can’t troubleshoot it at 3am, it’s too complicated. I have no idea why it stuck with me.
Editor’s note
The outage was real. I took down Europe for hours by leaving one word off one command, because I was in a hurry to get back to the work I thought mattered.
Add reload in before you touch anything you can’t drive to.
The part I remember most is the person who picked up. They were woken in the middle of the night by someone making no sense, and they calmed me down, listened, and got in the car. I thought my career was over. To them it was a drive to work. I had rushed the command, then I rushed the call, and they fixed the second one by getting me to slow down. I’ve been trying to do that ever since.
If you can’t troubleshoot it at 3am, it’s too complicated. I learned that from the person I woke up at 3am.
Slow is smooth. Smooth is fast.
Leave a Reply