Last Updated on December 3, 2021 by Oddmund Groette
Survivorship bias in backtesting
Survivorship bias is about the things we don’t see or to a certain degree ignore. We tend to see the winners and not the losers. Unfortunately, this is very typical in trading and backtesting. To avoid this, you need to understand what survivorship bias in trading is.
In this article, we describe what survivorship is and how survivorship bias influences backtesting, trading, and investing. We show examples of survivorship bias, how survivorship bias overestimates your backtests, and how you can avoid and minimize survivorship bias in trading and investing.
Unfortunately, almost all traders ignore survivorship bias, either on purpose or lack of knowledge.
Why is it important to understand survivorship bias? It’s important because it overstates backtested results and leads to many disappointments when you start live trading. Additionally, understanding survivorship bias is also an asset in everyday life.
Survivorship bias is invisible in backtesting, trading, and investing:
Survivorship bias is about the things we don’t see or to a certain degree ignore.
We like success stories about super-traders/investors or entrepreneurs, and we “forget” to look at the probabilities for success and look at all those that fail. Decision-making is not only about tangible results, but just as much about what Nassim Taleb calls alternative histories. The same goes for economics: it’s just as much about a study of what we don’t see, opportunity costs, and unintended consequences.
Why it is important to understand survivorship bias in backtesting
In all aspects of life, success stories are more prevalent than fiascos. We hear rock stars on the radio, we see business tycoons in the news, and we read about miracle cures in medicine.
But we rarely hear about the losers and the fiascos. Thus, we succumb to illusions and overestimate the probabilities of success. For every rock star, there are probably 1 000 “failed” musicians. For every successful author, there are probably 100 who fail to make ends meet.
It’s the same in the stock market. Most listed companies fail to beat short-term Treasury bills, according to Hendrik Bessembinder:
Hendrik Bessembinder reveals most stocks fail to deliver meaningful returns
Most stock market indices are an average adjusted for the market cap. However, averages are in most cases grossly simplifications and generalizations that are not very relevant.
It’s because a “normal” company is unlikely to resemble the averages. This is especially true in the stock market.
The reason is simple: averages are generalizations that sometimes are deceptive. Why is that? Because averages are weighted according to market capitalizations they rarely reflect the performance of the “average stock”, which is better reflected in the median.
This sequence of numbers 1, 5, 7, 9, and 99 have an average of 24.2 but the median number is 7. Just one number “distorts” the average immensely.
The same is true in the stock market. We have several times mentioned Hendrik Bessembinder in our articles. His findings in Do Stocks Outperform Treasury Bills? from 2017 describe why survivorship bias is likely to distort your backtests:
- From 1926 to 2015 only 42.1% of common stocks returned more than short-term Treasuries during their lifetime as a public company.
- 50% of the stocks delivered negative returns.
- A random common stock had a median life of only seven years (it was delisted, bought up, or merged – whatever reason).
- Monte Carlo simulations show 96% chance of underperforming a value-weighted index, 99% underperformed an equal-weighted index and 72% underperformed short-term Treasuries.
Clearly, all the averages are skewed to just a few star performers, just like in everyday life. 86 out of 26 000 stocks made half the value creation, and 1 000 stocks made all the alpha above short-term Treasuries. The few outliers made the difference, not the typical median stock. The biggest companies had much more staying power, which makes a lot of sense: most of the new and small companies have an unproven business model.
Bessembinder’s findings show why most retail investors and traders fail, but it clearly shows the problems of picking stocks today and using them for backtests. We fall prone to hindsight bias and we ignore the losers along the way.
Examples of survivorship bias in backtesting, trading and investing:
Survivorship bias exists in parts of life. Below we list some examples bort from our daily lives and trading/investing:
Survivorship bias example in backtesting:
In March 2021, Seeking Alpha published a strategy that showed evidence of strong outperformance compared to the S&P 500 (the article is behind a paywall). The strategy held at all times 40 stocks based on holdings of the best hedge funds.
How did the author come to this conclusion?
It turns out he started in the wrong end. He started by picking stocks among the best hedge funds from 2008 until 2021. He then used the quarterly holdings of these funds going back to 2008.
Needless to say, the result is fantastic. However, the observant reader spots that the author picked the winners during this period after the fact. This is, of course, unlikely to be repeated.
The problem is that if you used the same criteria back in 2010 to pick the 2010 stocks, the strategy would have chosen other hedge funds and holdings.
The article has 23 comments, but just one mentioned the flaw of survivorship bias. Pretty amazing. Almost all traders neglect survivorship bias.
Survivorship bias in trend-following
Micheal Harris published an article on the 19th of October 2019 where he tested a trend-following strategy on two different samples of quotes (priceactionlab.com). One test was done on quotes with delisted stocks, and the other ignored survivorship bias. The strategy was as follows:
Buy at the open of next month if monthly close > 12-month moving average
Sell at the open of next month if monthly close < 12-month moving average
In other words, a pretty simple strategy. The test period was from the year 2000 until October 2019.
How did the strategy perform?
When Harris tested on Dow 30 the “survivorship bias” portfolio returned 5.6% annually and the “no survivorship bias portfolio” returned 7.7%. On the S&P 100, the difference was similar: 6.5% vs 8.4%. Ignoring delisted stocks clearly improves the result, but unfortunately, this is not what happens in real trading and investing.
Survivorship bias in momentum investing
Micheal Harris also tested the effect of survivorship in momentum strategies in an article called Examples of Survivorship Bias in Cross-Sectional Momentum published on the 11th of June 2020.
The strategy was like this:
Buy the 10 stocks with the highest 6-month rate of change. Rebalance monthly. Position size is 10%. Buy next open.
The backtest range is from 1993 until 2020 and performed on both S&P 100 and Nasdaq 100.
The results in the S&P 100 are initially impressive: The CAGR is 26% versus only 9.6% for SPY.
Unfortunately, almost all the excess alpha is the result of survivorship bias. When Harris included the failed and delisted companies, the CAGR drops to 12.2% (not including commissions and slippage).
Is it the same difference in Nasdaq 100? It turns out the difference is even bigger. The initial test yielded an incredible CAGR of 46% and an equally impressive low max drawdown of only 41%.
However, when including the delistings the CAGR drops to 16.4% and has a max drawdown of 83%. The drawdown happened in 2000-2002 when the dot-com bubble burst and many companies went to zero. We certainly would be pretty devastated with such a drawdown.
Survivorship bias example from WW2:
A brilliant way to use logic happened in world war two. A well-known example, the US military looked for solutions on how to reduce fatalities among its planes and pilots. Many of the planes returning back home safely showed bullet holes in patterns like this:
How should the military go about reinforcing the airplanes heavily damaged by Nazi artillery?
The conclusion seemed simple: to reinforce the areas of the plane hit by bullets (marked by red dots on the picture). That seemed like a logical reason.
Problem is, those areas marked with red dots made it safely back home.
A brilliant mind, Abraham Wald, came to the opposite conclusion: the military should reinforce the areas of the fuselage where it had no red dots. Why? Because these are the places the planes would not survive if they were hit.
This is a perfect example of survivorship bias!
The success of dropping out of school
Both Mark Zuckerberg and Bill Gates didn’t finish university. Is college for suckers? If they can be successful without finishing university, why can’t you? Again, we see the success stories and not all the failed drop-outs. The fact is that drop-outs are more likely to be unemployed and have lower salaries.
Both Gates and Zuckerberg chose scalable businesses. Nassim Nicholas Taleb writes about scalable and non-scalable careers and concludes that scalable professions are only good if you are successful, and results are mostly random. Read more here:
Who doesn’t click on articles like “the personal traits that made Jeff Bezos successful”? But Bezos’ success could just as well be a result of luck and randomness.
How to avoid and minimize survivorship bias in backtesting, trading, and investing:
It’s probably impossible to completely eliminate survivorship bias. It turns up its ugly head in all aspects of trading. But is it possible to at least reduce or minimize it?
Below we list some possible solutions to minimize or eliminate survivorship bias:
Norgate Data provides databases including delistings
Using a database that includes delistings is the best way to deal with survivorship bias. Does such a database exist?
Norgate Data is a provider of historical and end-of-day quotes. Their specialty is survivorship bias-free data for US and Australian stock markets. They don’t provide live quotes or intra-day or tick data. We have used their data in the past and we highly recommend their products (we don’t have any referral with Norgate).
This is a hassle-free service and they even provide code for Amibroker to include delistings in backtests (at least some years ago, we believe Norgate still offers it).
ETF’s and mutual funds are less influenced by survivorship bias
Does it help to backtest on ETFs and mutual funds?
The churn and delistings among funds are not as frequent as among stocks, but it still exists. In a study by Morningstar 41% of funds that existed in 1999 were not in existence in 2009, 49% was out of business in 2014, and 58% didn’t exist in 2019.
Moreover, many funds speculate in closing down funds. They simply start x number of funds and close those that underperform. The “good” funds, mostly a result of chance, are “proven” star performers and the managers can charge fat management fees.
Are ETFs any better? Most likely not. ETFs are not even tested in a tough bear market. During the GFC in 2008/09 only a handful of ETFs existed. Howard Marks, the famous investor, is pretty reluctant to ETFs because of these reasons.
Indices provide survivorship free data
By sticking to the main indices, like Nasdaq and the S&P 500, you are not prone to survivorship bias.
Day trading might reduce the impact of survivorship bias if you’re using shorter time spans for backtesting. For example, the delistings in S&P 100 have been very few over the last years.
Conclusion: Survivorship bias in backtesting
You should always be vigilant toward survivorship bias in backtesting. Unfortunately, survivorship bias is prevalent in backtesting, trading, and investing.
If you get a good result in your backtests, always question your result before you start using your hard-earned money. Luck, randomness, curve-fitting, and survivorship bias are important factors that overrate most backtests. Having an understanding of survivorship bias in backtesting is crucial for success when you switch to live trading.