Balancing agility with confidence in your ability to figure out actually known problems

While reading an article about design docs at google I came across the following bit:

Most software projects do have a set of actually known problems. Subscribing to agile methodologies is not an excuse for not taking the time to get solutions to actually known problems right. Additionally, prototyping itself may be part of the design doc creation. “I tried it out and it works” is one of the best arguments for choosing a design.

And I think the author here is onto something I also observed: Identifying and working on actually known problems is one of the things that the "agile way" is in tension with. It's right there in the agile manifesto: "Responding to changeover following a plan". If you continuously emphasize responding to change over following a plan, eventually you will overshoot and be at a point where making a plan and following it would be the better idea.

Stressing to work more agile back when the manifesto was smart for the vast majority of teams because software is fundamentally different from physical things. Just shipping something and seeing how the thing actually performs in the real world is much more feasible for software then it is for physical things. But software teams did not have their own methodologies to take advantage of this and often this gap was filled by adopting heavy-weight processes that were good at ensuring low high-cost faults for physical products. So back then, the advise made a lot of sense for most teams. Nowadays developers and product managers have less experience in planing complex systems, because they grew up being taught that responding to change trumps following a plan. And naturally, this means they - on average - trust themselves less and therefore try less hard to foresee problems than the folks who were surrounded by people making big plans all the time. And while responding to change - dancing with the system - certainly makes tons of sense, we should strive to identify actually known problems before we go and build towards a solution that doesn't solve those.