Missing Bullet Holes & Finding Your Gaps

Written by Tom Stanley, Investment Adviser, June 2024

The Case Of The Missing Bullet Holes & Finding Your Gaps


  • Abraham Wald, a Jew persecuted by the Nazis in World War II was a math genius and helped the Americans develop their air superiority by analysing bullet holes in planes and determining how to optimally armour planes.
  • Wald’s approach was to challenge assumptions and paved the way for the development of ‘survivorship bias’ – a concept where we often see success, but do not observe failures – think “history is written by the victors”.
  • We often make assumptions without checking their validity – testing and being aware of the assumptions you are making as an investor is at the core of being ‘prudent and thoughtful’.
  • Visualising your investments and assets split by asset class (e.g. global shares, residential property, nz shares etc.) is an easy way to tease out the assumptions you are making and biases you may have.
  • The biases in your investments can often create ‘gaps’. These ‘gaps’ may be intentional, but in many cases they are not.
  • I’ve developed this template so you can identify your gaps – if you want to book a call to discuss either reply to this note or book a call with your adviser via the links at the bottom of this note.

For those who want the fully story – lets dig in.

P.S. Apologies, this note is a little late, the Amicus team and Stanley household have recently been hit with gastro, COVID and Influenza… hopefully you’re faring better than we have been this winter.

Abraham Wald & The Case of the Missing Bullet Holes:

To start this month’s note I want to share a story from a book that opened my eyes to thinking in probabilities.

Stay with me here.

Even though I studied engineering at university and worked as a consulting engineer for a number of years, I was never a fan of statistics or probabilities in school (most aren’t) – ‘stats’ seemed boring. Not once did I have a teacher put ‘stats’ in context.

Around 15 years ago I read Jordan Ellenberg’s “How Not To Be Wrong” – it’s not the easiest read, but it applied statistics to real life in a way that I had never considered.

The below is an adaption from How Not To Be Wrong by Jordan Ellenberg.

This story, like many World War II stories, starts with the Nazis hounding a Jew out of Europe and ends with the Nazis regretting it. Abraham Wald was born in 1902 in what was then the city of Klausenburg in what was then the Austro-Hungarian Empire. By the time Wald was a teenager, World War I was in the books and his hometown had become Cluj, Romania. He was the grandson of a rabbi and the son of a kosher baker, but the younger Wald was a mathematician almost from the start. His talent for the subject was quickly recognized, and he was admitted to study mathematics at the University of Vienna…

But when Wald’s studies were completed, it was the mid-1930s, Austria was deep in economic distress, and there was no possibility that a foreigner could be hired as a professor in Vienna.

Wald took odd jobs and when the Nazis conquered Austria, a move to America was on the cards where he was eventually offered a professorship of statistics at Columbia in New York. And that was where he fought the war.

The Statistical Research Group (SRG), where Wald spent much of World War II, was a classified program that yoked the assembled might of American statisticians to the war effort — something like the Manhattan Project, except the weapons being developed were equations, not explosives.

The atmosphere (of the SRG) combined the intellectual openness and intensity of an academic department with the shared sense of purpose that comes only with high stakes. “When we made recommendations,” W. Allen Wallis, the director, wrote, “frequently things happened. Fighter planes entered combat with their machine guns loaded according to Jack Wolfowitz’s recommendations about mixing 5 types of ammunition, and maybe the pilots came back or maybe they didn’t. Navy planes launched rockets whose propellants had been accepted by Abe Girshick’s sampling-inspection plans, and maybe the rockets exploded and destroyed our own planes and pilots or maybe they destroyed the target.”

The mathematical talent at hand was equal to the gravity of the task. In Wallis’s words, the SRG was “the most extraordinary group of statisticians ever organized, taking into account both number and quality.”

The smartest person in the room was usually Abraham Wald.

Still an “enemy alien,” he was not technically allowed to see the classified reports he was producing; the joke around SRG was that the secretaries were required to pull each sheet of notepaper out of his hands as soon as he was finished writing on it.

Wald was, in some ways, an unlikely participant. His inclination, as it always had been, was toward abstraction, and away from direct applications. But his motivation to use his talents against the Axis was obvious. And when you needed to turn a vague idea into solid mathematics, Wald was the person you wanted at your side.

So here’s the question. You don’t want your planes to get shot down by enemy fighters, so you armor them. But armor makes the plane heavier, and heavier planes are less maneuverable and use more fuel. Armoring the planes too much is a problem; armoring the planes too little is a problem. Somewhere in between there’s an optimum. The reason you have a team of mathematicians socked away in an apartment in New York City is to figure out where that optimum is.

The military came to the SRG with some data they thought might be useful. When American planes came back from engagements over Europe, they were covered in bullet holes. But the damage wasn’t uniformly distributed across the aircraft. There were more bullet holes in the fuselage, not so many in
the engines.

The officers saw an opportunity for efficiency; you can get the same protection with less armor if you concentrate the armor on the places with the greatest need, where the planes are getting hit the most. But exactly how much more armor belonged on those parts of the plane? That was the answer they came to Wald for. It wasn’t the answer they got.

The armor, said Wald, doesn’t go where the bullet holes are. It goes where the bullet holes aren’t: on the engines.

Wald’s insight was simply to ask:

Where are the missing holes? The ones that would have been all over the engine casing, if the damage had been spread equally all over the plane?
Wald was pretty sure he knew. The missing bullet holes were on the missing planes. The reason planes were coming back with fewer hits to the engine is that planes that got hit in the engine weren’t coming back. Whereas the large number of planes returning to base with a thoroughly Swiss-cheesed fuselage is pretty strong evidence that hits to the fuselage can (and therefore should) be tolerated.

If you go to the recovery room at the hospital, you’ll see a lot more people with bullet holes in their legs than people with bullet holes in their chests. But that’s not because people don’t get shot in the chest; it’s because the people who get shot in the chest don’t recover.

Wald’s recommendations were quickly put into effect, and were still being used by the navy and the air force through the wars in Korea and Vietnam. I can’t tell you exactly how many American planes they saved, though the data-slinging descendants of the SRG inside today’s military no doubt have a pretty good idea.

Why did Wald see what the officers, who had vastly more knowledge and understanding of aerial combat, couldn’t?

It comes back to his math-trained habits of thought.

A mathematician is always asking, “What assumptions are you making? And are they justified?” This can be annoying. But it can also be very productive. In this case, the officers were making an assumption unwittingly: that the planes that came back were a random sample of all the planes. If that were true, you could draw conclusions about the distribution of bullet holes on all the planes by examining the distribution of bullet holes on only the surviving planes. Once you recognize that you’ve been making that hypothesis, it takes only a moment to realize it’s dead wrong; there’s no reason at all to expect the planes to have an equal likelihood of survival no matter where they get hit.

To a mathematician, the structure underlying the bullet hole problem is a phenomenon called survivorship bias. It arises again and again, in all kinds of contexts. And once you’re familiar with it, as Wald was, you’re primed to notice it wherever it’s hiding.

This ‘Case of the Missing Bullet Holes’, ‘survivor bias’, and the revelation of where to place armour on fighter planes stuck with me thereafter.

If we don’t think critically, it is very easy to get caught up in survivorship bias and make assumptions that are not well considered.

Learning from Abraham Wald:

How does this story relate to you, your retirement, or your wealth building journey?

It comes down to assumptions, and more importantly:

“What assumptions are you making? And are they justified?”

Like the army officers, it can be difficult to step back and even consider what assumptions you may be making (intentionally or otherwise). To help address this, I developed a framework and template that teases out assumptions – I call it “What Are Your Gaps? & Are They Intentional?”  
Or “WAYGATI?” – click here to download excel template.

The inspiration for this framework was conversations with clients.
As I reviewed investment strategies, and retirement plans of clients, I quickly came to realise the natural bias toward residential property.

In many cases people had 70% to 90% of their net worth in property and had never really considered the implications of that exposure.

I get it, for some time the rules of the game in New Zealand have been skewed to support residential property investment and immense wealth has been made in this asset class in the past 40 years.

But, I quickly realised that investors, young and old, were often making one critical assumption:

“Past returns were going to be predictive of future returns.”

But, is that reasonable assumption?

My view – it’s not.

I’m not saying property is not a good investment, or predicting a crash in property values – what I am saying is I don’t believe it is reasonable to expect property prices to double every 10 years forever.

As I have mentioned in previous notes, the lack of price data, the ‘ability to add value’, and the ‘forced saving’ associated with paying down a mortgage, often lead to property investors achieving great long term outcomes.
“Carl Richards once made the point that a house might be the best investment most people ever make. It’s not that housing provides great returns – it does not. It’s not even the leverage. It’s that people are more likely to buy a house and sit on it without interruption for years or decades than any other asset. It’s the one asset people give compounding a fighting chance to work.” – On Time: Warren Buffet & The Rolling Stones.
The typical mum & dad property investment formula is simple: buy a rental, fix it up, use the rents to pay off the mortgage (likely plus some top up now days)… and the key ingredient: keep on keeping on for 20 to 30 years.

Property has been great and can be great way to build wealth, the rules, especially when it comes to leverage are still skewed in its favour.

What I am trying to highlight is that unless you are a developer, or property investor with a particular edge, having such a significant amount of your wealth in one asset class, often in one region/location, and in one or two properties (link) is very high risk, especially considering potential returns.

Testing the above assumption is not what I want to focus on in this note, let’s just look at a real-life example that I recently ran through with a couple nearing retirement.

Case Study: Jack & Jill – What Are Your Gaps?

Jack & Jill (not clients real names) are small business owners who have done a great job building their wealth over time – they have just turned 60 and have built their businesses over the past 20 years.

Over the years Jack and Jill carved off funds investing into rental properties that have nearly all paid themselves off.

As they look to step back from their businesses, they are looking for two things, income, and stability.

We took stock of their position using the above template – the outcome?

  • 63% of their wealth was in rental properties in Christchurch – we talked of their experiences of the earthquake and risks to rents that they would be relying on. 
  • 86% of their wealth was in ‘illiquid’ assets that weren’t generating sufficient income for them to live the way they wanted in retirement* 
    *Despite having a significant net worth (>$3.2m total not including owner occupied property), and property portfolio ($2.0m net of debt), they were only clearing around $30,000 of income per annum (after tax, rates, insurance and all costs). Of course, their businesses were generating additional cashflow, but they needed be working in the businesses to sustain/maintain that cashflow. 
  • While their KiwiSaver and other investments did offer diversification and offered daily liquidity – their KiwiSaver was not accessible for another 5 years so couldn’t be relied on in the short term. 

One area we focused on was the income from their rental properties. Rents totalled around 3.8% of the property values, but after tax, rates, insurance, interest, maintenance, and property management. The take home income was closer to 1.2% of the value of the properties – a number that surprised Jack and Jill.

While they wanted to have the option to retire ASAP, they couldn’t chip a brick off a rental once a month and sell it to supplement their income. They also couldn’t sell a couple shares in their private businesses as they were still seeking buyers.

Simply looking at returns on cash (currently 4.5% after tax or more if held via a PIE), prompted Jack and Jill to consider alternatives to what they perceived to be the only way to generate income in retirement (rents from fully paid-up rentals).

While Jack and Jill’s investment strategy had served them well in the past. Their assumptions needed to be challenged on a forward looking basis.
Visualing their current positioning and investments in graphical format created an experience that can be likened Abraham Wald observing the image of bullet holes in planes.

When visualised the concentration of their wealth was clear – their gap was liquidity – they could have the retirement they wanted, they just needed to tweak how they were investing to allow for higher income generating assets and the flexibility to draw on capital when they wanted to.

Questions to Ponder:
What assumptions are you making in how you invest?

How are your goals and plans changing? and do those changes influence the assumptions you make?

Next steps – What are your gaps?

I hope this article, and the case of the missing bullet holes helps you test your assumptions and plans.

I also hope the “What Are Your Gaps & Are They Intentional?” template is useful.

If you do want to discuss or run through this exercise in person, please feel free to reach out.

Links to book a call with our advisers are below – you can also just reply to this email.

We are here to help

Share this post

We’re locals, protecting locals

Scroll to Top