Lessons learned should include principles not just tactics

 / 

We recently had a major production incident which took 48 hours to fully resolve. Our technical experts did well under trying circumstances and worked till 3am to ensure the impact on our clients was minimal.

We had multiple sessions afterwards to reflect on what happened and how we responded. The executive summary alone was two pages long.

We identified 17 action items that will help us improve our systems, infrastructure and general approach. Its interesting how clearly one could put the items into two distinct buckets.

One bucket for things that would prevent or mitigate a recurrence of this exact problem. These are more tactical.

The second bucket for things that would actually be helpful in a broader array of circumstances. These are more strategic.

I value the improvements that are strategic much more. One of my values in life is that principles are more important than tactics.

Daniel Kahneman articulates one of those principles well: “The correct lesson to learn from surprises is that the world is surprising." Thus 20 years of IT experience has taught me to prefer big cushions and margin for error.

When people ask, "What are you preparing for?" I say, "For unforeseen problems, because anything that can go wrong will eventually go wrong."