The Importance of Backtesting
|September 3, 2012||Posted by Oddmund Grotte under trading topics|
It’s a good advice to do backtesting in order to gain some insight into how a trading idea might work. If you don’t do that you are basically just guessing on how a certain strategy will perform. It’s a big leap in the right direction. You’ll get a hint on how the strategy performs, at least in the past, although obviously there is no guarantee that it will make money in the future. However, some words of caution are needed. Real results will never be as good as theoretical backtesting. It’s 100% certain that the real time results will be worse than the actual backtesting. Why? Underneath you’ll find some personal thoughts of why the results differ so much.
- The backtest is performed in a certain period and the markets may have been favourable to that strategy in that period. Most traders neglect this fact. Testing over a longer time frame might minimize this. Take for example so called trend following strategies. They perform quite badly in certain periods covering many years. If testing a moving average breakout, this might yield mediocre results over a 5 year period. This is a typical strategy that needs to be tested over a period of at least 10 years. Obviously you don’t want to over-haul a strategy in response to one year just because something didn’t work. That’s when you’re almost guaranteed that it would have worked the next year had you kept it as it was. The ever changing market cycles makes trading a difficult task to follow. You have to accept drawdowns to make money and every strategy have drawdowns.
- You can only trade the strategy after the fact. For example: You are using entry on the close. Problem is, you only know if it’s a trade after the close. In order to trade on the close you might have to guess/estimate there wil be a trade at the close. So when backtesting, it’s crucial you take this factor into consideration. One way to do this is to trade at the open the day after the signal.
- Another reason is the curve fitting aspect. This certainly applies if you’re having a lot of parameters. It’s easy to come up with a system that has performed remarkably well. You just need to put in a lot of parameters. That will explain the past, but certainly not the future. The more simple the system, the more likely it’s to stand the test of time. Curtis Faith explains in The Way of The Turtle some trend following strategies. They are incredible simple. Over several decades they have worked well in currencies and commodities (not on stocks). However, over a period of 1-3 years they sometimes experience quite huge drawdowns. Still, these systems are so simple that they are less prone to be curve fitted.
- Another problem is the survivorship bias. Simply put, this relates to the use of stocks/tickers that has “survived” the testing period. For example in 2008/2009 a lot of stocks went bust (Lehman being an example). This means that companies that have gone bust are excluded from the analysis. When daytrading this might be less of an issue, but not when testing over a much longer time frame. If you download quotes for REITs today back to 2005, this will exclude several stocks that went bust during the financial crisis.
- If you’re testing a lot of strategies, some will show good returns simply by chance. Unless there might be some logical reason for a strategy, you are guaranteed to find many good strategies the more you test. Hence, there must always be some reasoning behind the parametres.
- The quotes are usually not correct. If using high and low in the test it’s probably a huge gap in theory and real trading. There are a lot of wrong quotes on the high and low. In real trading this will have a huge impact, probably the most of all the factors mentioned in this article.
- Slippage and commissions. Commissions are easy to calculate, but slippage not. This is a huge unknown, especially if you base your strategies on chasing the stock. If you wait to get hit, this is of course less of an issue.
- Changes in the market. Obviously the future is unpredictable and you can bet there will be totally random and dramtic changes in the marketplace. Noone expected terrorists to hijack planes and send them into a skyscraper. Such totally unpredictable disasters will happen sooner or later. Correllations among different asset classes also increase during such happenings. You can never backtest such things.
- The last, but not least, is the psychological aspects. Can you handle drawdowns and continue trading? Can you actually follow the strategy? Based on my personal experience, this is something you have to consider throughly before you implement a strategy. It’s a lot easier said than done to follow a strategy 100%. I’ll get back to this in a later article.
How can you prepare for this? As a rule of thumb it might be wize to expect maximum 50% of the profits from backtesting. Exaggerate slippage and commissions. Exaggerate maximum drawdown. Hope for the best, but prepare for the worst. Never be to high when seeing a very nice equity curve, the downfall will be bigger. There is only real trading that matters. This certainly applies if someone wants to sell you a system based on backtesting. Simply don’t buy it. Only tested and verified strategies will do you good. Why would someone sell a good system for 1000 USD? If the system is so good there is more money in trading it than selling it.
Over the years I have probably tested close to 1000 ideas. So far no strategy has performed better in real life than backtesting. So consider this as a fact of life. When executing a backtested strategy live you discover a lot factors you didn’t think of when backtesting. Hence, always start with the minimum amount of money.
I’ll finish the article by a very good advice (at least it works for me): It might be very boring and tedious to test strategies manually by hand, but believe me, you can learn a lot more. It’s so easy to use a program and to scan thousands of stocks. However, by doing some by hand you can extract info that otherwise would get lost. It might be a little boring doing it the old fashioned way, but consider it as an investment. For example by testing one stock at a time by pasting quotes in to Excel, instead of just scanning 1000 stocks in one minute using a fancy program, you can one by one look at each different stock. Look at it’s equity curve, look at max drawdown etc. Doing it the “slow” way increases learning. First you can scan, and if the strategy shows promise, spend some more time doing it manually stock by stock.