Selection bias is the selection of individuals, groups, or data for analysis in such a way that proper randomization is not achieved, thereby failing to ensure that the sample obtained is representative of the population intended to be analyzed. That’s the definition pulled right off the page on Wikipedia. Survivorship bias is a form of the selection bias defined above.
It’s a logical error that can occur where any group is divided by people or things that made it past a selection process. Some will make it past the test and others will not. Those that failed to get past the test may be overlooked as a consequence. Because survivorship bias focuses on the ‘winners’ in this way, or those that pass the selection process, the failures can become dangerously ignored.
Adopting this style of thinking can be challenging. It forces us to encounter failure with a more analytical lens than we may be used to. It’s easy to cast aside the things that aren’t working.
The data is in the damage
During WWII, Abraham Wald was a mathematician working with Columbia University on statistical problems toward the war effort. His work with the University would later be considered breakthrough research for Operational Research.
One of his most famous conclusions came from his interpretation of aircraft damage who had returned from flying missions. He was tasked to provide advice on how to minimize losses of American bombers to enemy fire.
Wald figured a way to estimate damage for outbound aircraft based on the data from damaged aircraft that had returned from their missions. The image on the left shows hypothetical data for damage points on returning aircraft. Some viewed these pockmarked areas as the obvious places to fix additional armor. Wald argued that the damaged areas were represented on planes that had returned. As a result, he inferred that additional armor should be placed in the areas left clear on the diagram. The planes that were shot down would have likely taken damage at these locations and hence, not returned to be included in the data set.
Survivorship bias is an important facet of operational assessment for leaders to incorporate into their understanding of problem-solving. It lends well to issues we face in that we are required to really analyze the consequence of both the ‘wins’ and ‘losses’ in groups that pass through any mode of selection. The nuanced leader would do well to regard the assets that do not pass through a selection process just as carefully as those that do.
