Pitfalls Of Backtesting And Quantifying Strategies

Last Updated on June 11, 2021 by Oddmund Groette


Why backtest and quantify strategies:

It’s good advice to do backtesting in order to gain some insight into how a trading idea might work. If you don’t do that you are basically just guessing how a certain strategy will perform. It’s a big leap in the right direction. You’ll get a hint on how the strategy performs, at least in the past, although obviously there is no guarantee that it will make money in the future.

Results will never be as good as theoretical backtesting

However, some words of caution are needed. Real results will never be as good as theoretical backtesting. It’s 100% certain that the real-time results will be worse than the actual backtesting. Why? Underneath you’ll find some personal thoughts of why the results differ so much.

The backtest might be liable to favorable conditions

The backtest is performed in a certain period and the markets may have been favorable to that strategy in that period. Most traders neglect this aspect. Testing over a longer time frame might minimize this.

Take for example so-called trend following strategies. They perform quite badly in certain periods covering many years. If testing a moving average breakout, this might yield mediocre results over a 5 year period. This is a typical strategy that needs to be tested over a period of at least 10 years. Obviously, you don’t want to over-haul a strategy in response to one year just because something didn’t work. That’s when you’re almost guaranteed that it would have worked the next year had you kept it as it was.

The ever-changing market cycles make trading a difficult task to follow. You have to accept drawdowns to make money and every strategy has drawdowns.

You open positions after the fact

You can only trade the strategy after the fact. For example, you are using entry on the close. Problem is, you only know if it’s a trade after the close. In order to trade on the close, you might have to guess/estimate there wil be a trade at the close. So when backtesting, it’s crucial you take this factor into consideration. One way to do this is to trade at the open the day after the signal.

Curve fitting

Another reason is the curve fitting aspect. This certainly applies if you’re having a lot of parameters or variables. It’s easy to come up with a system that has performed remarkably well. You just need to put in a lot of parameters. That will explain the past, but most likely not the future.

The more simple the system, the more likely it’s to stand the test of time.

Curtis Faith explains in The Way of The Turtle some trend-following strategies. They are incredibly simple. Over several decades they have worked well in currencies and commodities (not on stocks). However, over a period of 1-3 years, they sometimes experience quite huge drawdowns. Still, these systems are so simple that they are less prone to be curve fitted.

Survivorship bias

Another problem is survivorship bias. Put simply, this relates to the use of stocks/tickers that have “survived” the testing period. For example in 2008/2009 a lot of stocks went bust (Lehman being an example). This means that companies that have gone bust are excluded from the analysis. In day trading this might be less of an issue, but not when testing over a much longer time frame. If you download quotes for REITs back to 2005, this will exclude several stocks that went bust during the financial crisis.

Chance, luck and randomness should not be ignored

If you’re testing a lot of strategies, some will show good returns simply by chance. Unless there might be some logical reason for a strategy, you are guaranteed to find many good strategies the more you test. Hence, there must always be some reasoning behind the parameters.

Garbage in, garbage out

The quotes are usually not correct. If using high and low in the test it’s probably a huge gap from theory to live trading. There are a lot of wrong quotes on high and low. In real trading, this will have a huge impact, probably most of all the factors mentioned in this article.

Transaction costs

Slippage and commissions. Commissions are easy to calculate, but slippage not. This is a huge unknown, especially if you base your strategies on chasing the stock. If you wait to get hit, this is of course less of an issue.

Markets change

Changes in the market. Obviously, the future is unpredictable and you can bet there will be totally random and dramatic changes in the marketplace. No one expected terrorists to hijack planes and send them into a skyscraper. Such totally unpredictable disasters will happen sooner or later. Correlations among different asset classes also increase during such happenings. You can never backtest such things.

Behavioral mistakes

Last, but not least, is the psychological aspect. Can you handle drawdowns and continue trading? Can you actually follow the strategy? Based on my personal experience, this is something you have to consider thoroughly before you implement a strategy. It’s a lot easier said than done to follow a strategy 100%.

Prepare for the worst

How can you prepare for this? As a rule of thumb, it might be wise to expect a maximum of 50% of the profits from backtesting. Exaggerate slippage and commissions. Exaggerate maximum drawdown. Hope for the best, but prepare for the worst. Never be too optimistic when seeing a very nice equity curve, the downfall will be bigger. There is only real trading that matters.

Does backtesting really work?

It depends. First, you need to understand and test valid and logical ideas. Second, you need to have an understanding of the pitfalls mentioned in this article.

Over the years I have probably tested close to 1000 ideas. So far no strategy has performed better in real life than backtesting. So consider this as a fact of life. When executing a backtested strategy live you discover a lot of factors you didn’t think of when backtesting. Hence, always start with the minimum amount of money and always papertrade a strategy before you go live.

Last: a piece of advice

I’ll finish the article with a piece of very good advice (at least it works for me): It might be very boring and tedious to test strategies manually by hand but believe me, you can learn a lot more. It’s easy to use a program and to scan thousands of stocks. However, by doing testing by hand you can extract info that otherwise would get lost. It might be a little boring doing it the old fashioned way, but consider it as an investment.

For example by testing one stock at a time by pasting quotes into Excel, instead of just scanning 1000 stocks in one minute using a fancy program, you can one by one look at each different stock. Look at its equity curve, look at max drawdown, etc. If you use scanning, spend some more time doing it manually stock by stock.