What is backtesting, and how do you backtest trading strategies? And does backtesting work? In your day-to-day life, before you buy anything — a mobile phone, a house, or a car — you would want to check its features and history to know if it is worth your money. That also applies to trading, but here, what you check are your trading strategies. With a wide range of markets to trade, you need to backtest your strategies to be sure they work.
The steps involved in how to backtest a trading strategy include defining the criteria of the strategy, selecting the market and timeframe to test it on, loading up the historical data, writing the code and implementing the backtesting, and finally, evaluating the results.
A backtest is a tool that not only small retail traders use but also big institutions. The world’s most successful hedge fund, Jim Simons’ Medallion Fund, uses backtesting continuously to develop new strategies. Why? Because backtesting a trading strategy works!
Backtesting a trading strategy works because you can falsify or confirm a trading idea, you can automate all your trading based on the backtests, exploit the law of large numbers, limit behavioral mistakes, and lastly, you can save a lot of time in executions. Backtesting is not a waste of time! We have done backtesting daily for over 20 years, and this article summarizes the main reasons why you should backtest and why it works.
Backtesting course
But first, a humble reminder that we have made a backtest course for beginners. We have been trading (successfully) and backtesting since the end of the 1990s, and thus have developed a pretty extensive knowledge about backtesting and historical analysis of trading strategies. We have compiled that knowledge into an inexpensive backtesting course. You can read more about the course and order by clicking on the banner below:
What is a backtest? What is backtesting?
This website is all about backtesting. But what exactly is a backtest?
A backtest is a way of testing a trading strategy on historical data. It is a way to simulate the historical performance of a trading strategy using historical data before committing real funds to the strategy on live trading. Backtesting is based on the assumption that if the strategy performed well in a particular market previously, it has a good chance of working again. And on the flip side, if the strategy did not perform well in the past for that market, it may not work well in the future.
So, let’s say you have a swing trading strategy that says that if the S&P 500 Index has a positive return in the past month, it will give a positive return over the next week. Now, you want to test that theory to know if true or not. While you can forward-test it by using a demo account and waiting for the positive months to come (you check how the market performs in the month that follows each of them), that would be very time-consuming.
A faster way to test your theory is to test it on historical price data. You go into the past to find all the times the market has a positive monthly return and then check how the market performed in the week or month that followed.
If the performance on historical data is good, you might assume that the strategy would perform well in the current market, but if the performance is poor, you may discard the strategy or tweak and re-test it. You can, of course, backtest any time period you like, the point is to measure how your predictions on past data work on future unknown data.
Thus, backtesting is a very important step in creating a trading system. It can help a trader test, optimize, and improve their strategies, thereby giving them the confidence to apply the strategy in live trading.
A backtest has strict rules for when to buy and when to exit. In other words, you can code the strategy and find out with 100% certainty how the strategy has performed in the past. Thus, this is a backtest on historical data and strict trading rules. That is why it’s called a “back test” (history). We can argue it’s a kind of quantified technical analysis – technical analysis backtesting.
Of course, this doesn’t give any certainties about the future, but you know if the strategy has performed well or poorly in the past. If something has performed poorly in the past, it’s unlikely that it will perform well in the future. However, the ever-changing market cycles make strategies perform well in certain markets, and poorly in others. This is the reality of trading.
Opposite, if a backtest proves that your idea has worked well in the past, it most likely will perform better than any idea that has performed poorly. But of course, a positive backtest is no guarantee that it will work in the future. But we believe it’s the best indication you can get.
A backtest follows this procedure:
- Find an idea you want to test.
- Define clear and concise entry and exit parameters – they must be quantifiable.
- Specify the market you want to test on.
- Specify the time frame you want to test.
- Code the strategy.
- Run the strategy on the in-sample period.
- Test the out of sample backtest.
This is all there is to it.
If the backtest returns a positive and promising result, we recommend that you paper trade the strategy for several months before you commit real money. This can save you a lot of money! We have written more about our procedures in our trading lessons based on 20 years of full-time trading and investing.
Why do you backtest?
There are many reasons to backtest a strategy, and these are some of them:
- To know if you have a trading edge in the market (what is a statistical trading edge?): Many traders, even experienced traders, lose money, and they don’t lose money because they don’t know how the market works. But simply because their trading strategies don’t have any statistical edge in the market they are trading. No matter how you master trading discipline, control your emotions, and manage risk, you cannot make money from a market unless your strategy has an edge. If you want, risk only 0.1% of your account per trade, without an edge, the account would slowly bleed out. The only way to know if your strategy has an edge in a market is by backtesting it. The likelihood of trading profitably in the market is boosted if you backtest your strategy before trading. Emotional control and risk management only help a profitable strategy to make money — they can’t do anything if the strategy has no positive expectancy.
- To improve a strategy: Another reason to backtest a strategy is to see how you can tweak it to improve its performance. This involves altering some variables or adding new variables to see how they improve performance. The process is sometimes called optimization. Too much of it can lead to curve-fitting and dimmish the strategy’s robustness.
- Check for correlation in trading: for those more advanced, you might want to backtest a portfolio of trading strategies. You see, even if you have 20 very good backtested trading strategies, they might not work well together if they have many overlapping trades.
How to backtest trading strategies
These are the steps to follow when doing a backtest:
- Define the logic of the strategy: The first step is to define the strategy you want to backtest and clearly state the criteria for each action with specific trading rules. The logic you want to backtest should be clear. For example, if your strategy is a moving average crossover system, you clearly state the type of moving average, the period of the two moving averages, and the entry and exit criteria. The entry and exit logic could be (as an example):
- When the 20-day SMA crosses above the 100-day SMA, exit any short position and enter a long position
- When the 20-day SMA crosses below the 100-day SMA, exit any long position and enter a short position
- Select the market(s): At this stage, you specify the market you want to test the strategy on, as well as the time frame. The market you backtest must be the one you want to trade the strategy. You don’t use a strategy backtested on the silver market to trade the gold market, even though both are precious metals. Note that you can backtest the strategy on different markets and on different time frames and choose the ones with the best performance, but beware of curve fitting.
- Get the data: You will need historical data of the market you want to backtest. While some trading platforms offer some data, they may be limited. You may have to pay to get enough historical data for your backtesting. You have to divide your data into two — in-sample data for backtesting and out-of-sample data for optimization. However, be careful about the data you backtest on. Your backtest is only as good as the quality of your data! There are plenty of bad data out there.
- Implement your backtesting: You may have to learn a programming language like Python, but some trading platforms like TradeStation and Amibroker use languages that are easy to use.
- Evaluate your results: After the backtesting, you need to evaluate your result to know how the strategy performed. Here are some of the performance metric statistics used in trading:
- The total number of trades: This tells you how many trades were taken during the period under study. The higher the number of trades, the more reliable the result.
- Holding time: This is the length of time that the strategy has open positions in the market. The lower the holding time, the better.
- Profit factor: This is the total profit divided by the total loss for the entire trading period. It indicates the amount of profit per unit of risk. A profit factor of more than 1 indicates a profitable strategy, but the higher the better. Note that on some platforms, this is presented as “Average size of winner/loser”.
- Maximum drawdown: This is the difference between the higher amount and lowest amount the trading capital got to during the backtesting. The lower the maximum drawdown, the better the strategy.
- The final part of a backtest is to backtest out-of sample or incubate the strategy (more about this later in the article). This involves testing the data on unknown data, ie finding out how well the backtest predicts the future unknown data.
Where can you backtest a trading strategy?
You backtest a trading strategy on a trading platform. Most trading platforms have a strategy tester section where you can backtest your strategy, but not all of them are free to use.
Alternatively, you can use Excel or a spreadsheet.
How to backtest a trading strategy for free
As a retail trader, you may want a free platform for backtesting, and there is some free software available in the market for backtesting a trading strategy. Some of the free backtesting software are Microsoft Excel (Libre Office), TradingView, NinjaTrader, Trade Brains, etc.
How to backtest a trading strategy on paid platforms
But the paid platforms may offer more features. One of the most commonly used paid platforms is Amibroker. The analysis window of the Amibroker platform allows you to back-test your trading strategy on historical data. At about 450 USD for a lifetime license, the platform is pretty cheap, and it offers full customization features for backtesting and some lightning-fast optimization features. It is easy to test strategies on a portfolio level with the platform.
Practically all of the backtest done on this website are done by Amibroker. We are extremely pleased, and we also use it for live trading by adding just a little extra code. Considering the cheap and its features, we believe Amibroker is a good option for most retail traders. Based on our experiences, we have created an online Amibroker course that consists of several modules (basic, live trading, and trading strategies):
Other interesting and relevant articles about backtesting platforms are found in our trading platform guide.
How long should a trading strategy be backtested?
It depends on a lot of factors. It is easy to say backtest for a period of one year or two, but statistically, it is not only a question of duration but also of sample size. That said, we recommend including several types of markets, like bear and bull markets. They behave differently.
When backtesting, the key thing is that you want to have enough sample size (the number of trades) that can be representative of the population (in this case, the market).
So, in terms of duration, you would need as much time as can give you enough trades to have a statistically reliable result. While a sample of 250 trades may be sufficient, the bigger the sample size is, the smaller the margin of error (in most cases), and the more reliable the result. If your trading system generates enough trades, a sample of 500 – 750 trades is good. The best is to have both a large sample size and a long test period.
The number of trades you get would depend on the strategy and how many trades it generates per day, week, or month. For a day trading strategy that generates a trade per day, it means you get about 250 trades per year, so backtesting over a 2-3 year duration should be able to give you enough data to make reliable assumptions, yes?
Not so fast. The problem is that 2-3 years is a short period and normally only included just one business cycle. We like to backtest using about 20 years, so we are sure we have included bear markets in our dataset.
On the other hand, a swing trading strategy that generates only four trades per month would get you only about 48 trades per year. So, to get up to 500 trades sample that would ensure reliable assumption, you will have to backtest over a period of more than 10 years.
Thus, having fewer trades over a longer time span might be better than having many trades over a shorter time span.
How do you backtest a trading strategy in Excel?
When backtesting a trading strategy in Excel, here are the steps to take:
- Get your data into Excel: There are two basic ways to do this: The first option is to manually download historical data from Yahoo Finance directly as CSV and then load it into Excel. This works, but you’ll need to re-download that historical data and then copy and paste either the entire dataset or a subset to update your strategy. The other option is using code to automatically grab data from Yahoo Finance. You can search on Google for some VBA codes you can use. Some third-party tools, like AnalyzerXL, can make the job simple.
- Create your indicator: Now that you have got your data into Excel, you use them to construct an indicator or indicators for your strategy. You can use different columns to create different indicators or different aspects of an indicator, with each one taking part in the calculation. You can use other add-ons or third-party tools for more functionality.
- Formulate your trading rules: After creating your indicator, you need to formulate your trading rules. You create the calculation in a new column you name the “Signal” column. Your trading rule can be as simple as going long if the calculation is above a certain level and short if below it. But you could also have more complex rules with criteria for when to vary the position size or when to stay out of the market.
- Get the equity curve: You can get the equity curve with the Excel function, but if you need other trading stats like CAGR and the Sharpe ratio to compare the strategy with other strategies, you will have to do a bit of work or use a third-party tool like AnalyzerXL.
We used Excel as our main backtesting tool all the way up to 2017. Excel is a lot better than most traders imagine! You don’t need any fancy tools to backtest, the main asset is, after all, you, who put in the trading rules. As a matter of fact, Excel can be a very useful tool because you, in most cases, need to test a strategy on one instrument at a time. This way, you see small details you otherwise wouldn’t.
We even used Excel for live trading. We had a VBA script that we used for both sending orders, but also when we closed our positions. It worked really well, but it’s not as dynamic as dedicated software platforms (like Amibroker, Tradestation, etc.).
Do professional traders backtest?
Yes, professional traders always test their strategies before deploying them. They know how important backtesting is to the profitability of a trading strategy and cannot afford to make the mistake of trading a strategy that is not backtested. Some even go ahead to forward-test their strategies — if they are intraday trading strategies that can generate enough trades within a short period — before committing real money to them.
Even professional traders who use discretionary trading methods still backtest their strategies. They do so manually by either going back in time to check the occasions where their trade setups occurred and how the market reacted. All serious traders keep a trading journal. Alternatively, they can use strategy tester software that prints historical data as though they are in real-time and then trade their setups as they occur.
Can you backtest for free?
Yes, there are some trading platforms that allow you to test your strategy for free, but you may have to look for historical data from market data vendors to get enough data. Some of the free backtesting platforms include TradingView, NinjaTrader, and so on.
Alternatively, you can use free software like Microsoft Excel, which enables you to backtest with the Excel function. However, you have to source your data elsewhere. A common source of data is Yahoo Finance, where you can either download the data manually or write a code that can do that. However, if using free data please be careful about bad data.
Backtesting software
This refers to a platform you need to test a strategy. It can be a stand-alone platform or a trading platform. Most trading platforms have a strategy tester or analysis window where you can backtest your strategy, but not all of them are free to use. Some of the free backtesting software are Microsoft Excel, TradingView, NinjaTrader, Trade Station, Trade Brains, etc.
How to backtest a strategy in Python
Python is a free open-source and cross-platform language with a rich library for almost every imaginable task, including automated trading. Thus, Python is an excellent choice for backtesting a trading strategy.
If you want to backtest a strategy with Python, here are the steps to follow.
- Import the necessary libraries for backtesting
- Download the needed market data
- Calculate daily returns
- Create strategy-based data columns
- Create strategy indicators
- Create signals and positions
- Implement the backtesting
- Analyze results
We have written about Python trading strategy in a previous article.
How do you backtest trading strategies without coding?
To backtest without coding, you have to use a code-free trading software. Such a platform allows you to create codes with a simple drag-and-drop interface. Examples of code-free platforms include TradeStation, Amibroker, MetaTrader 5, TradingView, QuantShare, and Forex Tester.
Out-of-sample backtesting
One of the most important things in backtesting is to use your trading rules on unknown data. Why? Because one of the most common traps when backtesting is to curve fit. The more variables you use to get a trading strategy, the more likely you are fitting the rules to the data.
To avoid this, you need to backtest on unknown or future data. For example, if you have data from the year 2000 until today, you can make rules based on the data until 2017 and then test the trading rules on the data from 2017. If the strategy performs well on the unknown data, you might have something going.
We have explained out-of-sample backtesting in a separate article.
Walk-forward testing
Another way of testing your backtested strategy on unknown data is a method called walk-forward. It’s a kind of optimization where you “walk” forward in your dataset, hence the name. We have explained Walk-forward here.
Curve fitting
The enemy of a backtest is curve fitting. In this lesson, we explain what this is and how you can avoid it.
Let’s start by defining what curve fitting really is:
Curve fitting is when a strategy or edge is not fit to market behavior, but market noise, leading to failure in live trading. Curve fitting is overoptimization that is unlikely to fit into future unknown data.
We have in a previous lesson explained how optimization helps you in trading, but you need to be careful so as not to fall into the curve-fitting trap.
Statistics and performance metrics
When you have a backtest ready, you’d like to know if it’s good or bad. How robust is the strategy? Is it likely to be a result of chance?
There are many ways to evaluate a backtest. Just looking at the end result, the CAGR or the annual returns, might be very misleading.
If you’re a short-term trader, we are pretty confident in saying that most traders would abandon a strategy if the drawdowns are too big no matter how high the returns. This is why you need to look at strategy and system performance metrics.
Survivorship bias – don’t fool yourself
Survivorship bias in trading and backtesting is about the things we don’t see or to a certain degree ignore. We tend to see the winners and not the losers. Unfortunately, this is very typical in trading and backtesting. To avoid this, you need to understand what survivorship bias in trading is.
Survivorship bias is about the things we don’t see or to a certain degree ignore.
We like success stories about super-traders/investors or entrepreneurs, and we “forget” to look at the probabilities for success and look at all those that fail. Decision-making is not only about tangible results, but just as much about what Nassim Taleb calls alternative histories. The same goes for economics: it’s just as much about a study of what we don’t see, opportunity costs, and unintended consequences.
The importance of good datasets (garbage in equals garbage out)
Now, let’s turn to a very important aspect of backtesting: good data sets. Good data is important in trading. After over twenty years of day trading, we have experienced the expensive way the importance of good data. The famous saying “garbage in, garbage out” is indeed true.
Your backtest is only as good as the data you are testing on. Make sure you are backtesting on reliable and “clean” data. In the long run, it pays off to spend money on a good data source for backtesting.
We have probably lost tens of thousands of dollars on trading strategies that are based on “garbage”. Sad but true.
Yahoo!finance is often wrong
Problem is, it is not much you can do about it. Or is it? By writing this blog we’ve been contacted by several people. Yesterday one sent me his own data on SPY which he has downloaded from Interactive Brokers (IB) himself. We’ll do some testing on this dataset to see the differences between that and EOD data from Yahoo!finance. No doubt the dataset downloaded from IB is better than what you get from many providers.
First, we’ll show you two errors in SPY that we still can remember to this very day (in Yahoo!Finance):
This example is from the 30th of November 2011.
It’s correct that it was a big gap-up opening, but the low is completely wrong. Even in a paid data feed such as IQFeed.net this low price is included (on EOD data, not intraday data). In many strategies, if you rely on the low of the day to set profit targets, this will turn out to be a huge winner. But in reality, this low trade never happened. The fact is that this day had a low that was only some 20 cents lower than the open! Not 2 dollars as shown here.
Here is the second example:
This one is from the 9th of April 2012. It shows the gap is filled, but it’s fake. The high of the day was 75 cents higher than the open, not close to 2 dollars as shown in the chart! Fading the gap this turns out to be a fake huge winner.
Worth noting is that the CLOSE is basically 100% right. OPEN is also reasonably correct. It is the HIGH and LOW prices of the day which are sometimes (very) wrong.
A comparison between two data providers: Yahoo!finance and Interactive Brokers
Below is a comparison of the quotes comparing the manually downloaded quotes from IB and the EOD quotes from Yahoo!finance. It shows the percentage difference between the OPEN to HIGH and OPEN to LOW (the OPEN to HIGH from IB is deducted from the OPEN to HIGH from Yahoo!finance).
The first bar shows that Yahoo!finance has a lot of high quotes that are a lot higher than IB’s. The second chart shows the same attributes: The low in Yahoo!finance is a lot lower than IB’s.
The question is: are these differences so brutal that it makes a theoretically good strategy useless?
We have previously written about opening gaps in SPY. And yes, the results are a lot worse. We tested on all three options: EOD from Yahoo/Finance, intraday data collected from IB, and intraday data from IQFeed.net. All three yields significantly different numbers! When using EOD data from IQFeed, we get the same result as in Yahoo!finance.
So the conclusion must be: if you’re testing on only the CLOSE and OPEN data, you’re (mostly) on solid ground no matter your data provider.
If you’re using the HIGH and LOW on EOD, you must be careful. Always test the strategies by paper trading: Just on the quotes you actually see, or trade as small as you can for a period.
Should a trading strategy work on all assets?
Why should, for example, a 3-day RSI work on both SPY and XLE?
We frequently see discussions on for example Twitter, where one trader trades different strategies for different instruments/stocks, while another trader believes this is data mining. For example: To buy when RSI(3) is oversold might work on different stocks and not on others.
If we trade this strategy on the stocks it works on, and not on the stocks it does not work on, is this data mining and curve fitting? Or should we only trade strategies that work more or less on all S&P stocks?
We believe the first is correct. Why should one strategy work on both SPY and XLE? These are completely different stocks from different sectors. There is no logic to implement the same strategies in different sectors. We have tested thousands of strategies and all sectors behave differently. Even within sectors, you have to regroup the stocks.
How to pick stocks for each strategy
We pick stocks/assets for our strategies based on how they perform using this strategy. If a stock does not show an upward-sloping equity curve, we simply reject it from that asset.
Is this data mining or curve fitting? We don’t think so. This is the way we have done it over the last 20 years, and it works well for us.
Have we just been lucky? We don’t think so. Our trading is “out of sample” testing, and our group of stocks performs better than any random group. We have tested this a lot of times. The key to successful trading is to look for patterns/attributes in stocks and trade what you find.
Of course, it’s only logical that stocks have different patterns (at least to us). Sometimes you just happen to find a random pattern, so there must be some kind of logic behind why this pattern should exist.
Trade many uncorrelated strategies
We believe many will reject our methodology. That’s fine. We read a lot of blogs and see a whole lot of different theories. Usually, the blog post ends like “this is not recommended as a stand-alone strategy”.
We have picked up so many ideas from blogs and traded them, and still, the author thinks his strategy is not worthwhile as a tradeable strategy. The best thing is to trade a lot and trade with different strategies. This is the holy grail of trading! Even some erratic equity curves contribute when having about 20 uncorrelated strategies. A backtest might not be tradeable on its own, but together with other strategies it might be complementary.
Trading and backtesting should be a little boring
At the end some words on backtesting, boredom and introverted traders:
We believe backtesting and trading should be a little boring. Perhaps contradictory, but the more boring, the better you can do.
Why? Trading only suits people with the following attitudes: patience, analytical, and somewhat risk-averse.
Yes, you have to take risks in trading, but it needs to be balanced. All the traders we know that have lasted more than ten years have been somewhat introverted and very rational/analytical and not risk-takers. They backtest and base their decisions on that and include a margin of safety.
They don’t trade to get a rush, but to make money. They are happy to backtest strategies, but not keen on execution. When there is little prey in the markets, they simply wait out for better times. Of course, they have made money before and have the financial means to wait for many months.
How to backtest a trading strategy – an example
Let us now show you a couple of examples of backtesting:
Let’s say you have a swing trading strategy that says that if the S&P 500 Index has a positive return in the past month, it will give a positive return over the next week. You can backtest it easily to find out if it makes money. In this case, your market is the S&P 500 Index — you can test it on e-mini S&P 500 futures or the SPY ETF (SPY ETF trading).
Your strategy rules are:
- Go long on the next day’s open if the monthly bar closes positively
- Exit at the close of the trading week
You can get the market data of more than 60 years since it is a monthly signal. After testing, you evaluate the performance using stats like profit factor, Sharpe ratio, maximum drawdown, or any other statistic that measure the performance of a trading strategy. If the performance on historical data is good, you can assume that the strategy would perform well in the current market, but if the performance is poor, you may discard the strategy or tweak and retest it.
Let’s end this section of the article and make another specific backtest with trading rules and settings. The strategy is called the Turnaround Tuesday strategy and is one of the most well-known strategies there is, yet it’s still working pretty well.
We make the following trading rules:
- Today is Monday.
- The close must be at least 1% lower than Friday’s close.
- If one and two are true, then enter at the close.
- Exit at the close on Tuesday.
This is an overnight trading strategy that owns S&P 500 (SPY) for 24 hours.
This is what the equity curve looks like:
The 171 trades have an average gain of 0.67%, something we consider excellent considering you only hold the position for 24 hours! Even during bear markets, it has worked pretty well.
We did this backtest in less than two minutes by using Amibroker. This shows the power of validating or scrapping your trading idea by backtesting. This is why backtesting is so powerful!
Conclusion
Because emotions run deep in any human being and we tend to jump to conclusions, we recommend backtesting. You can validate your ideas quickly (or falsify them). Backtesting is powerful, and actually pretty easy if you practice.
You don’t need any fancy tools to backtest. Start with a spreadsheet, and if you like it, go on to something more sophisticated like Amibroker, Tradestation, etc. From there you need to formulate trading ideas, backtest, test on known and unknown data, validate and falsify, and make conclusions. It’s as simple as that – that is how to backtest a trading strategy.
Does backtesting work?
Let’s discuss why we belive backtesting works and is the best way to approach trading and investing for the great majority of people:
Backtesting works because you can confirm or falsify a trading idea
Backtesting works because you can easily check if something has worked in the past or not.
- Is, for example, the Turnaround Tuesday strategy in stocks true or just a myth? Just define the rules and start the backtest. You’ll find out in five minutes.
- Do you believe you see an interesting pattern in the chart? Then quantify it with strict buy and sell rules and test it.
Did the strategy work in the past? If something has not worked in the past, you can easily falsify your hypothesis and go on to test another idea.
Because most ideas don’t work, you should not spend much time testing a hypothesis. Many traders waste months, even years, in both programming software and tweaking their strategies only to find out it was a waste of time. You don’t need “perfect” strategies to make money in the markets. You need many strategies that complement each other.
We have written multiple times on this website that one of the main reasons for your success (or no success) depends on your ability to test and generate trading ideas. One of our main trading lessons is that we spend about 80% of our time testing ideas back and forth between ourselves.
Backtesting works because it lets you automate
If you have successfully backtested a strategy, you can easily go “live” with the strategy. In Tradestation, you just check a box and you are good to go.
In Amibroker, you need to add code to automate and let Amibroker keep track of your positions and strategies. We have even made a course that lets you automate your Amibroker strategies to Interactive Brokers:
Obviously, this saves you a lot of time. There is no need to check quotes, prices, or follow the markets. The computer does all the work!
Backtesting works – you can exploit the law of large numbers
Your computer can easily trade and supervise hundreds of strategies. This lets you exploit the law of large numbers and you can diversify into time frames, asset classes, directions, and types of strategies.
The main reason for the success of the Medallion Fund is twofold: they use enormous amounts of data to generate hundreds of uncorrelated strategies. Because of the low internal correlation among the strategies they can use leverage to boost returns.
Low correlation among your trading strategies is one of the most important factors in trading. Why? Because if you lose money in one strategy, you might gain in another uncorrelated strategy. This is a topic we have covered in many articles:
- What does correlation mean in trading? (Trading strategies and correlations)
- Uncorrelated assets and strategies – benefits and advantages (examples and backtests)
- Does your trading strategy complement your portfolio of strategies?
However, leverage is dangerous and certainly not something we recommend. Only use leverage if you have many years of experience.
Backtesting a trading strategy works because it removes emotions
Ample evidence points out that individual investors underperform the averages, and women are better investors than men. The main reason for this is behavioral mistakes:
Investors tend to sell into a panic and buy after a big rise. Most of the time you need to do the complete opposite. A backtest can’t capture such mistakes and that’s why you need to stick to the trading plan.
In order to stick to the trading plan, you need to trade smaller than you’d like or prefer. This is the best way to keep detachment to money.
Likewise, women do better because they save, invest, and forget about it. They are not trying to be geniuses! They don’t have any ego.
The closer you follow the markets, the more likely you are to overrule your systems when your “intuition” tells you to sell or buy. But most of the time the intuition is plain wrong, unfortunately.
Overruling your systems and strategies is unlikely to work. You have not backtested overruling, so how do you know if it works? That’s why you backtest trading strategy.
Backtesting and trading software
You can use Excel when you are backtesting, but we recommend using some software that has most of the tools built in. Many coders and programmers like to use python (to better model strategies), but there is a reason why programmers and coders are bad traders. Software costs money, but you save a lot of time. Trading is much about making it simple – not complex. When you are backtesting trading strategies, it’s a lot about quantity.
Backtesting works because it saves time
Today we tested about 15 hypotheses in Silvers Miners (ticker code SIL) in about 1.5 hours. One of those ideas seems promising, the rest are probably just a waste of time. You can generate and test hundreds of strategies in just a single day. Even better, you can falsify or confirm ideas quickly.
Trading is mainly about trial and error. And luckily, backtesting a trading strategy is a great tool for that and at the same time, it saves you a lot of time.
Disadvantages of backtesting (why it doesn’t work)
Of course, there are also drawbacks and negatives with backtesting. You rarely manage to find trading strategies that perform better in live trading than in tests. Why is it so? This is because of the disadvantages of backtesting. You need experience in testing to avoid the many pitfalls along the way.
Let’s look at the disadvantages of backtesting. There are many reasons why backtesting doesn’t work, like curve fitting, market cycles, chance, luck, and randomness. But the good thing is that you can avoid or at least minimize many of the disadvantages of testing. We are proponents of quantifying trading strategies, but you need to understand how to avoid the many pitfalls in testing. We explain why:
Live trading will never be as good as theoretical backtesting
However, some words of caution are needed. Real results will never be as good as theoretical backtesting. It’s 100% certain that the real-time results will be worse than the actual backtesting.
Why? below you’ll find some personal thoughts of why the results differ so much.
The backtest might be liable to favorable conditions
The backtest is performed in a certain period and the markets may have been favorable to that trading strategy in that period. Most traders neglect this aspect. Testing over a longer time frame might minimize this.
Take for example so-called trend following strategies. They perform quite badly in certain periods covering many years, and much better in others.
If you are testing a moving average breakout, this might yield mediocre results over a five-year period. This is a typical strategy that needs to be tested over a period of at least 10 years.
Obviously, you don’t want to change a strategy in response to one year just because something didn’t work. That’s when you’re almost guaranteed that it would have worked the next year had you kept it as it was.
The ever-changing market cycles make trading a difficult task to follow. You have to accept drawdowns to make money and every strategy has drawdowns.
Backtests work because you know the after the fact
You can only trade the strategy after the fact.
For example, you are using entry at the close. Problem is, you only know if it’s a trade after the close. In order to trade on the close, you might have to guess/estimate there wil be a trade at the close.
So when backtesting, it’s crucial you take this factor into consideration. One way to do this is to trade at the open the day after the signal.
Backtesting involves elements of curve fitting
Another reason is the curve fitting aspect. This certainly applies if you’re having a lot of parameters or variables.
It’s easy to come up with a system that has performed remarkably well. You just need to put in a lot of parameters. That will explain the past, but most likely not the future.
The more simple the system, the more likely it’s to stand the test of time.
Curtis Faith explains in The Way of The Turtle some trend-following strategies that are incredibly simple. Over several decades they have worked well in currencies and commodities (not on stocks).
However, over a period of 1-3 years, they sometimes experience quite huge drawdowns. Still, these systems are so simple that they are less prone to be curve fitted.
Survivorship bias
Another problem is survivorship bias.
Put simply, this relates to the use of stocks/tickers that have “survived” the testing period.
For example in 2008/2009 a lot of stocks went bust (Lehman being an example). This means that companies that have gone bust are excluded from the analysis at later dates.
In day trading this might be less of an issue, but not when testing over a much longer time frame. If you download quotes for REITs back to 2005, this will exclude several stocks that went bust during the financial crisis.
Also, if you backtest indices you are unlikely to be a victim of survivorship bias.
Chance, luck, and randomness should not be ignored
If you’re testing a lot of strategies, some will show good returns simply by chance.
Unless there might be some logical reason for a strategy, you are guaranteed to find many good strategies the more you test.
Hence, there must always be some reasoning behind the parameters.
Backtesting involves garbage in, garbage out
The quotes you buy or download are usually not correct. If using high and low in the test, you can be sure there are errors compares to live trading. There are a lot of wrong quotes on high and low!
In real trading, this will have a huge impact, probably most of all the factors mentioned in this article.
Transaction costs are unknown
Slippage and commissions are easy to underestimate. Commissions are easy to calculate, but slippage is not.
This is a huge unknown, especially if you base your strategies on chasing the stock. If you wait to get hit, this is of course less of an issue.
Markets change – no backtest can accommodate change
The market changes all the time.
Obviously, the future is unpredictable and you can bet there will be totally random and dramatic changes in the marketplace. No one expected terrorists to hijack planes and send them into a skyscraper.
Such totally unpredictable disasters will happen sooner or later. Correlations among different asset classes also increase during such happenings. You can never backtest such things.
All traders do behavioral mistakes
You need to understand the most obvious trading biases. The psychological aspect is just as important as the strategy. Can you handle drawdowns and continue trading? Can you actually follow the strategy?
Based on my personal experience, this is something you have to consider thoroughly before you implement a strategy. It’s a lot easier said than done to follow a strategy 100%.
Always prepare for the worst
How can you prepare for the worst?
As a rule of thumb, it might be wise to expect a maximum of 50% of the profits from testing. You can exaggerate slippage and commissions, and expect a much higher drawdown than in the backtest.
Hope for the best, but prepare for the worst.
Never be too optimistic when seeing a very nice equity curve, the downfall will be bigger. There is only real trading that matters.
Disadvantages of backtesting – conclusion
To get good results from testing, you need to understand the potentially many pitfalls of testing. A backtest can’t capture that markets change, are random, that you tend to curve fit, and you make behavioral mistakes. However, if you are street smart, you can even offset some of the disadvantages of backtesting.
A piece of advice on backtesting
We’ll finish the article with a piece of very good advice (at least it works for us):
It might be very boring and tedious to test strategies manually by hand but believe me, you can learn a lot more. It’s easy to use a program and to scan thousands of stocks.
However, by doing testing by hand you can extract info that otherwise would get lost. It might be a little boring doing it the old-fashioned way, but consider it as an investment.
For example, by testing one stock at a time by pasting quotes into Excel, instead of just scanning 1000 stocks in one minute using a fancy program, you can one by one look at each different stock. Look at its equity curve, look at max drawdown, etc.
If you use scanning, spend some more time doing it manually stock by stock.
Does backtesting really work? Conclusion
Mostly, yes. First, you need to understand and test valid and logical ideas. Second, you need to have an understanding of the disadvantages mentioned in this article.
We have probably tested close to 2000 ideas. So far no strategy has performed better in real life than in backtesting. So consider this as a fact of life.
When executing a tested strategy live you discover a lot of factors you didn’t think of when testing. Hence, always start with the minimum amount of money and always papertrade a strategy before you go live.
Facts and Frequently Asked Questions about backtesting
Let’s end this long article with some FAQ and facts about backtesting:
Why is backtesting important?
Backtesting is important because it helps you determine facts – the opposite of anecdotal evidence (which financial markets are full of). By analyzing trading statistics and historical performance you get to falsify or confirm your trading idea. You get to know the historical probability of a trade being a success or not.
Why would you trade your money if you really have no idea if the strategy has worked in the past? Our experience is that most traders don’t have any positive statistical edge in the first place, thus making most of the focus on psychology and money management a wasteful exercise.
Trading is very much a game of statistics and probabilities, and thus you need to know the odds of your strategy. When you have a tradable strategy, you just need to push buttons and execute.
Furthermore, backtesting helps you to get other and perhaps better trading ideas. Trading is about trial and error!
Any strategy that has not worked in the past, is unlikely to yield many positive results in the future. Opposite, a strategy that has shown promise in the past, might be an inefficiency that will work in the future, and you might even be able to optimize and improve the strategy without being liable to curve fitting.
How long should you backtest a trading system?
We use the following rules of thumb:
- At least 100 trades (with a few exceptions)
- Must include different business cycles (at least a bear market)
These two are the only two rules we use. However, we like to test as far back in time as possible.
Any end-of-day trading strategy is normally backtested decades back, while we use about ten years for intraday data (for day trading).
Backtesting day trading strategies (how to)
In theory, there is no difference between backtesting on daily bars or intraday bars (5 min bars, for example).
However, backtesting on intraday historical data is more demanding than using end-of-day data. There are two reasons for that:
- It’s technically more different (both code and rules). Some rules are actually very difficult to quantify.
- Intraday data are less likely to work well in the future. There is more randomness involved in the time series data.
Is backtesting a waste of time?
Of course not!
Most tests end up worthless, but at least you are improving your backtesting, and you get to see what is not working, which is also a great asset to have. Trading is about compounding knowledge!
Best backtesting software
There is no answer that fits all traders. It depends on many factors. What works for you, is not necessarily the best choice for other traders.
For example, we use Amibroker. It’s a fantastic program for backtesting (fast and cheap), but it’s a bit cumbersome if you want to trade your tested strategies live. You need to write a few extra lines of code to both connect to Interactive Brokers (or the broker of your choice) and to keep track of your strategies.
Other quantitative traders might stick to Tradestation, Multicharts, Ninjatrader, etc.
The fact is that trading platforms are kind of a commodity product that is hard to differentiate from.
How to backtest option strategies
You test option strategies just the same way you do with stocks or futures. That said, it’s a bit more complicated due to the many different strikes and expirations. You certainly would need experience from end-of-day backtesting before you venture into testing options.
Backtest forex – how do you do it?
Again, the process and principles are the same for any backtest – no matter the market. The main advantage of forex is that data is abundant.
But sadly, we believe forex is mostly a waste of time. It’s a low bar to trade it, but most quantitative forex traders lose and stop trading. We believe it’s a bad idea to start forex trading. Please read our reasons why:
Is backtesting accurate?
No! Nothing is accurate in trading or predictions about the future. The future is unknown – also with a test.
But in our opinion, it’s the best estimation of the future you’ll ever have.
Backtesting in Excel
Excel and spreadsheets are very underrated testing software. We both did backtesting and live trading (VBA script) for our very successful day trading until 2018.
That said, spreadsheets have their limitations, especially if you want to test portfolios.
Additionally, Excel has limitations when it comes to trading history analysis and trading performance metrics. With a software you get all this with the click of a button, while in Excel you need to “code” it.
Strategy backtesting software
If you are serious about trading, we don’t recommend Excel but that you purchase some specific trading software, like Amibroker, for example. It’s cheap, reliable, and lightning fast.
Backtest trading strategy Python
Python is extremely popular among quant traders, something that we find a bit odd, but we are not coders, either. We suspect it’s mainly programmers and coders that like using Python.
But why start from “scratch” when you can use Amibroker, Tradestation, etc.? To us, it doesn’t make sense, even though Python has a ton of free resources. You should make an analysis of trades, not spend time coding. Any strategy test should be made as fast as possible.
How many trades should you backtest?
We believe it’s more important to include different business cycles than to specifically focus on the number of trades, even though the latter should be at least 100 – preferably more.
Technical analysis backtesting
We have a rule: if it can’t be tested, don’t trade it. This applies to technical analysis as well. We mention it because it’s very tricky to test technical chart patterns. Most of the patterns are not backtested, and thus you should avoid them.
In our list of trading strategies we quantify some technical patterns:
What is the the best platform to backtest trading strategies?
We use Amibroker because we believe it’s the best platform to backtest trading strategies. But as you might guess, other traders might disagree.
The most used platform among professional and independent traders is probably Tradestation because of the simplicity to use it for BOTH backtesting and live trading.
Thus, the best platform boils down to what you’ll use it for.
Cab you backtes crypto?
Crypto and Bitcoin can easily be backtested. But all cryptos have so far limitations because of their short and erratic history. It’s unlikely the past predicts the future very well because of this.
An example of two crypto backtests can be found here:
- Trend following and momentum strategies on bitcoin (crypto)
- Does RSI work on crypto or Bitcoin trading? Is RSI good for crypto?
What is the best website about backtesting?
We are biased, but we believe the blog and website you are reading now is one of the best. Apart from this one, we can recommend the quantocracy website which contains a nice summary of quant articles.
Backtesting vs forward testing
Backtesting is when you test known data while forward testing is when you test the trading rules made on the known data on future unknown data.
Backtesting – pros and cons
Let’s briefly list the pros (and advantages):
- You can confirm or falsify a trading idea
- It lets you automate
- You can exploit the law of large numbers
- Backtesting reduces emotions
- It saves time
These are the cons (disadvantages):
- It might be liable to favorable conditions
- It work because you know the after the fact
- Backtesting involves elements of curve fitting
- Survivorship bias (survivorship bias in trading)
- Chance, luck, and randomness should not be ignored
- It involves garbage in, garbage out
- Transaction costs are unknown
- Markets change – no backtest can accommodate change
- All traders do behavioral mistakes
Despite having more bullet points containing disadvantages than advantages, we still believe it is the far superior way of making money in the markets. If you have no idea if your strategy has made money in the past, how can you be sure it’s profitable?
Automated backtesting
Some programs allow you to look for a trading strategy that is based on a zillion trading rules.
For example, you can program the software to look for the best settings for any technical trading indicator. But we believe such software requires a TON of experience due to all the curve fitting that takes place.
Backtesting trading strategies free
You can find web-based software on the web that lets you backtest for free. That is an option if you’d like to find out if it is for you, but it’s not a viable long-term solution. The best option is to spend some time on finding the right software and then purchasing it.
What is Walk Forward Backtesting?
A walk-forward optimization is when you use both in-sample and out-of-sample tests frequently.
It’s done like this:
If you have 20 years of data, you might divide the data into ten equal parts (in length). You then have two years of data for each set. The first year is the in-sample backtest, and the second year is the out-of-sample test (where you test the trading rules made in year one).
This is repeated ten times and the final results are evaluated to make the final parameters for the strategy.
It’s a bit cumbersome, but most software platforms offer this ability. We don’t use it ourselves because we have not found it very useful.
What are anchored and non-anchored backtesting?
Anchored and unanchored (non-anchored) are two forms of walk-forward backtesting.
An anchored walk-forward test uses a fixed start date in each simulation, while a non-anchored uses a start date that is moved gradually forward by an amount equal to the length of each out-of-sample period.
Backtesting trading strategy – Tradingviews
Tradingview is a very popular platform that has gained many users with the rise of crypto. It lets you connect to a wide range of brokers if you want to place trades.
What is the best data provider for backtesting?
You need data to backtest. We recommend Norgate for end-of-day data because they have “clean” and error-free data. Any quantified and data-driven strategy needs good data, otherwise, the backtest is useless.
You can also download free data from Yahoo and Google, but beware of incorrect High and Low readings. The open and close are mostly correct, though, at least on the most liquid tickers.
FAQ
Q: What is backtesting?
A: Backtesting is the process of testing a trading strategy using historical data to evaluate its performance.
Q: How does backtesting work?
A: Backtesting works by using historical data to simulate trades that the strategy would have made in the past. By analyzing the performance results, traders can assess the effectiveness and profitability of their strategy.
Q: What is the purpose of backtesting?
A: The purpose of backtesting is to assess the viability and profitability of a trading strategy. It helps traders make informed decisions based on historical performance data.
Q: How can I perform backtesting?
A: To perform backtesting, you can use specialized software such as backtesting.py or write your own code in a programming language like Python. These tools allow you to input your trading strategy and historical data to simulate trades and analyze the results.
Q: What are the benefits of backtesting?
A: Backtesting allows traders to evaluate the performance of their strategies without the risk of losing real money. It helps in analyzing different trading strategies, assessing risk levels, and optimizing the expected returns.
Q: What is the difference between backtesting and scenario analysis?
A: Backtesting and scenario analysis are both methods used in evaluating trading strategies. However, backtesting relies on historical data analysis and simulation trading results, while scenario analysis involves examining the effects of specific events or scenarios on the strategy.
Q: How can backtesting with Python be helpful?
A: Backtesting with Python offers traders the ability to implement and test their trading strategies using a widely-used and versatile programming language. Python has libraries such as backtesting.py that provide the necessary tools for historical data analysis and strategy evaluation.
Q: What factors should be considered when analyzing the performance of a strategy through backtesting?
A: When analyzing the performance of a strategy through backtesting, factors such as the strategy’s win rate, standard deviation, trade duration, and risk assessment should be considered. These indicators help assess the profitability and risk associated with the strategy.
Q: What are the risks of backtesting?
A: One of the risks of backtesting is overfitting, which occurs when a strategy is tailored too closely to historical data and may not perform well in real market conditions. Traders should be cautious not to rely solely on backtesting results and consider the limitations of historical market analysis.
Q: How can backtesting aid in financial analysis?
A: Backtesting aids in financial analysis by providing a historical perspective on the performance of investment strategies. It allows traders to test their trading strategies, evaluate the performance, and make informed decisions based on the results.
Final Words
Effective trading and investment decisions rely heavily on thorough analysis and reliable data. Backtest accuracy is crucial in assessing the viability of trading strategies. Through rigorous trading system testing and strategy development and testing, investors can ascertain the potential risks and rewards of their proposed approaches.
Trading model evaluation plays a pivotal role in identifying the strengths and weaknesses of different models, while historical market research provides valuable insights into past trends and patterns. Quantitative trading backtests help in understanding the performance of trading strategies under various market conditions.
Conducting historical investment analysis allows investors to gauge the performance of their past investments and make informed decisions for the future. Performance metrics in trading provide a clear picture of the effectiveness of various trading strategies and enable investors to make data-driven choices.
Algorithmic trading validation ensures the reliability and effectiveness of automated trading systems, while historical stock performance data aids in understanding the behavior of specific stocks over time. Backtesting for investment helps investors to assess the potential outcomes of their investment decisions before implementation.
A comprehensive strategy backtest report provides a detailed analysis of the performance of a trading strategy under different market scenarios. Trading system evaluation helps investors to assess the effectiveness of their trading systems and make necessary adjustments for optimal performance.
By analyzing strategy simulation results, investors can anticipate potential outcomes and adjust their strategies accordingly. Historical trading data is instrumental in understanding market trends and patterns over time, while backtesting tools enable investors to test their strategies efficiently and accurately.
Evaluating strategy performance metrics is essential in determining the success of a trading strategy, while quantitative analysis of trading aids in understanding the quantitative aspects of market movements. Historical trading strategy analysis allows investors to learn from past experiences and refine their future strategies, ultimately leading to more informed investment decisions. Algorithmic trading backtest reports provide a comprehensive evaluation of the performance of automated trading strategies, facilitating informed decision-making in the realm of algorithmic trading.
