Last Updated on June 11, 2021 by Oddmund Groette
If you have spent considerable time developing a strategy, it’s frustrating if it performs poorly in live trading. What is happening? Why does it perform so poorly when it worked so well in backtests?
In this article, we’ll briefly mention some arguments for why your strategies mostly perform poorer than in your backtests. They do so for several reasons: behavioral mistakes, curve fitting, survivorship bias, and altering the trading size are the most common reasons.
As a rule of thumb, you should always expect your strategies to perform worse than on paper. All backtesting, no matter how cautious you are, involves an element of curve fitting.
Backtests compress time:
You can test a strategy over many decades in just one second, thus compressing time. In live trading, you spend years doing the daily grind of buying and selling, while a backtest is done in seconds or minutes.
You lose a lot of information when just crunching numbers. Moreover, what looks easy on paper, is not as easy in live trading.
A trading backtest never replicates live trading. We fool ourselves constantly by our cognitive errors and behavioral mistakes. If your strategy has shown a 20% drawdown, how do you deal with it in live trading? In a backtest, you know the strategy performed well after the drawdown. But in actual trading, where the future is uncertain, you don’t know that. Do you keep on trading, do you start tweaking or adding variables, or do you stop trading?
Our biases make it very difficult to do what the strategy tells us to do. Markets have an uncanny habit for shaking out the faint at heart – at the exact wrong time.
How do you deal with behavioral mistakes?
Victor Niederhoffer says that a bad system is better than no system at all. Stick to your systems until you either abandon them or put them back to paper trading.
The worst drawdown is yet to come:
The biggest drawdowns are yet to come. One of the reasons you chose a particular trading strategy is most likely because it has small drawdowns. This is kind of curve-fitting. Thus, you can expect any strategy to have a bigger drawdown than in your backtest.
What is curve fitting?
Curve fitting is when you use variables and parameters that fit the past but is unlikely to predict future prices. The future is never like the past. Despite this, we change variables and parameters until we get the results we want.
To elaborate, when we curve fit, we don´t fit our models to market behavior. We fit them to market data. That is a huge difference since market data consists of market behavior and random market noise. For that model to be profitable in the future, historical data’s random patterns must repeat themselves. However, the one primary trait of random patterns is that they do not hold any predictive value, since they are random.
Too many variables:
The more you put into your strategy, the more likely you are to curve-fit your strategy. The simpler you make it, the better. A system might be so complex that it has no predictive value. Just a slight change in the market might turn the strategy into a loser. Moreover, be on the lookout for trades that might explain a significant part of the profitability. Such trades could be due to chance and noise and unlikely to repeat.
The world changes:
Such an obvious fact is easy to forget. No strategy lasts forever.
The markets are mostly random:
Because markets are mostly random, many of your strategies and edges are the result of noise. It’s genuinely not an edge, but just something that happened to be profitable.
Correlation is not the same as causation:
Because markets are predominantly random and evolving, most correlations are spurious. Most relationships are indirect, not direct. Whatever you do and conclude, the result might come from chance and is not proof of causation. Any strategy that seems statistically significant might be so due to noise or hidden factors.
There are many false positives in the markets.
Survivorship bias is more prevalent and important than you think. For example, in March 2021, Seeking Alpha published a strategy that outperforms the S&P 500 by holding 40 stocks based on holdings of the best hedge funds (the article is behind a paywall).
How did the author conclude this?
He started in 2021 by picking 40 large hedge funds that outperformed the S&P 500 from 2008 until 2021. He then used the quarterly holdings of these funds going back to 2008.
Needless to say, the result is fantastic, obviously because he has picked the winners during this period. This is, of course, unlikely to be repeated. The problem is that if you used the same criteria back in 2010 to pick the 2010 stocks, the strategy would have chosen other hedge funds and holdings. Among the 23 comments about the results, just one mentioned the flaw of survivorship bias.
Almost all traders neglect survivorship bias.
Underestimating trading costs – slippage:
Backtests require realistic entry and exits. However, these are often based on “after the fact”. Thus, a strategy that enters on the close needs to buy seconds before the close (or in after hours). This, of course, might change the results significantly.
There is no perfect strategy:
Many are looking for the perfect strategy with minimal drawdowns. It doesn’t exist. The price you pay for making money in the markets is pain from drawdown and temporary setbacks. You can’t expect to make money without risk. As we have written numerous times: It’s better to have many “imperfect” strategies and let diversification take care of the drawdowns:
News is a distraction:
The world is bombarding you with news from all angles. It’s challenging to ignore the news. An abundance of information and free commissions make a recipe for poor trading results. We suggest you keep both news channels and social media at a distance.
Improper size and money management:
Make a rational plan before you start on how to allocate your capital. It’s how you deal with losses that are paramount for your survival as a trader. How do your strategies perform together as a portfolio? How much capital should be allocated to each trading style?
The pendulum between pessimism and optimism makes you change the size. After a good run, you increase the size, and after a bad run, you decrease the size – only to find out the market turned around. Then you return to your original size. It doesn’t matter how good your strategy is if you can’t execute it properly.
The best advice we can give is always to trade smaller than you like.
How to reduce the risk of poor live trading
Below we briefly mention some methods to minimize disappointment when you go live with your strategies:
By changing your variables’ values, you get to measure how the results change from even small modifications. For example, if one of the variables is ADX(5)>40 try changing it to ADX(5)>45 and so on. Does it improve because of just a few winners?
Out of sample:
Out of sample testing involves testing your strategy on data not included in the backtest. This can be done by splitting your data in two: one part for developing your strategy, for example, from the year 2000 until 2017, and then testing out of sample from 2018 until today.
An even better method than out of sample is to use an incubation period.
How do you do an incubation period?
You open a demo account with live or delayed quotes and run your strategies as if it was live. This is the best out of sample test you can get. You get to see how it performs and get to see the drawdowns live.
We suggest you do an incubation period for several months, preferably at least six months.
Monte Carlo simulation:
Monte Carlo simulations are used to model different outcomes of the variables and the parameters in your strategy. It makes random sequences to evaluate your trading system’s robustness to find how the element of risk and randomness might influence the forecasting abilities.
It works by reshuffling the order of the trades in a backtest and can expose weaknesses that otherwise had would be hidden in the backtest. The simulation then gives you a list of potential outcomes – CAGR, drawdowns, risk of ruin, etc with probabilities.
Trade smaller than you like:
Most traders are too optimistic about how much pain they tolerate. Everything looks easy in a backtest, but when real money is at stake, we tend to make many behavioral mistakes. Greed makes you count the chips before they are won and makes you trade sizes that are too big for your bankroll.
But success is about building up small and frequent profits over time. Prepare for a daily grind. To manage that, you need to trade a smaller size than you would like. This is the only way to detach you from money.
Don’t despair – some ideas work better than others:
We have been making quantified strategies for 20 years. Some of the strategies we made prior to 2018 have been recently published as a paid service:
Disclosure: We are not financial advisors. Please do your own due diligence and investment research or consult a financial professional. All articles are our opinions – they are not suggestions to buy or sell any securities.