You have probably read about curve fitting in trading books, trading magazines, and social media. The danger of Curve fitting is ubiquitous when designing trading strategies. It has the potential to ruin your trading career if not dealt with correctly and can be hard to notice, even for traders with decades of experience. In this article, we will learn what curve fitting is, and why you should try to avoid it.
Curve fitting is when a strategy or edge is not fit to market behavior, but market noise, leading to failure in live trading. Curve fitting is overoptimization that is unlikely to fit into future unknown data.
(Before we go on we’d like to mention that we have a backtesting course that covers all aspects of how to backtest.)
Part I: What is curve fitting?
Backtesting in search of edges
When we perform backtests, we analyze data in search of recurrent patterns that have predictive potential. In other words, we want to know if the tested pattern can tell us when the market is prone to going up or down so that we can be in the market only when it is favorable to us. If we succeed to find patterns that we think mirror market behavior, we have a trading edge.
Since our trading business is, or at least should be entirely reliant on edges, their quality and robustness dictate how well we cope with carving out profits in the market. Therefore, it is critical that our edges continue to work well also into the future if we want to make any money.
A thought experiment
Insinuate that your observation of what you presume is an edge, is flawed and holds no merit. Insinuate that you cannot know if your edge will continue to deliver going forward and that an overwhelming majority of patterns you call edges, will not work at all.
Quite scary, isn’t it, when you are about to risk your own money on those very edges?
Well, this is not a thought experiment. It is reality.
The harsh truth about observations of market behavior
When we search for edges in the markets, most of what we assume is an edge, will be outright garbage! True edges are hard to find, and in your search, you will sometimes be completely certain that you have an edge ready to trade, only to see it fall apart completely once exposed to new market data. This is one of the aspects that makes trading so hard for beginners to succeed in, and that needs to be overcome before risking real money!
Asking a question
Now that we know about the tendencies of the markets to deceive us into believing in false edges, it is time for us to ask a question to understand why this is, and what it has to do with curve fitting.
The question reads as follows:
Of all patterns we observe, only a few are true edges. How come that some of the observations we make are edges and others are not?
Or the same question veiled in other words:
How come some observations are true edges and others are curve fit?
To answer this, let us begin by learning a few lessons:
Lesson 1: Markets are mostly random
The first thing every trader needs to grasp to be able to understand the concept of curve fitting, is that a majority of market action is random noise. Most market activity simply cannot be derived from any form of analysis and needs to be accepted as nothing else than random market noise.
Lesson 2: Most people want explanations, even to the inexplicable
We as humans have an urge to explain everything we see and experience. By doing so we bring order to a chaotic world, at the cost of quite often lying to ourselves. This tendency among humans can often be observed when financial news media covers recent market activity. The expert may ascribe soaring markets to some recent event, which seems perfectly reasonable. However, once the market turns around, so do often the experts explaining the downturn with the very same arguments.
In such cases, it is apparent that humans like to fit explanations to reality and not the other way around since both of our expert´s comments cannot be true at the same time.
The severe fallback of this inclination of the human mind is that reality is not very inclined to conform to our description of it. Curve fit edges will not hold, regardless of what reason we ascribe to its logic.
Lesson 3: Correlation does not equal causality – market behavior and market data are not the same
The third and last lesson we must learn before we can grasp the concept of curve fitting is that market data and market behavior are not the same.
Market behavior is non-random price action that holds predictive value, while market data consists of market behavior AND market noise combined. The consequence of this is that what seems to work in the backtest carried out on market data, cannot be taken for true market behavior before being put under scrutiny. It may very well be a result of randomness, thus holding no value going forward.
So, what is curve fitting?
Using what we now know from the three lessons in this article, we may define what we mean by curve fitting. Our definition reads as follows:
Curve fitting is when random market noise forms haphazard patterns in price data, that are later viewed and considered an edge, despite being a product of sheer randomness.
To elaborate, when curve fitting, we don´t fit our models to market behavior. We fit them to market data. That is a huge difference since market data consists not only of market behavior but also of random market noise. For that model to be profitable going forward, the random patterns observed in historical data must repeat themselves. However, the one main trait of random patterns is that they do not hold any predictive value, since they are random.
Therefore, curve fit models nearly always fall apart in live trading.
An Illustration of the Concept of Curve Fitting
For it all to become a little clearer, let us illustrate the concept of curve fitting with an example:
Alan is building a house on the outskirts of the desert to investigate local wildlife. Since his arrival, it has not rained once, so he finds it unnecessary to build a roof that can handle large rain masses. After all, this is a desert, and who would expect rain to be abundant?
What he does not know, is that the period since his arrival has been the driest period for over 100 years. Quite soon after finishing his construction, he understands that his observation of the weather and the following conclusion that rain does not fall here was flawed. It was based on some random weather phenomenon that was not representative of normal weather behavior in the area. Soon, his house is flooded by the rain masses. His construction was curve fit to his observation of a random drought period.
Going back to trading
If we were to translate this story to fit with trading and curve fitting, we would get the following translations
Drought period = Market noise
Alan´s observation of the drought period = Our presumed edge in the market
The immediate failure of the construction = The failure of a curve fit edge
Alan curve fit his construction to a type of weather that was not typical of that region, despite believing it was. In the same way, traders curve fit their models to market data instead of market behavior.
Another thing Alan did, that we have not gone into yet, was to find an argument that supported his conclusion that rain is scarce in the area.
“…After all, this is desert, and who would expect rain to be abundant?…”
Most likely he had found supporting arguments for the very opposite claim if that was what he chose to believe in. It is always important to remain vigilant as soon as one´s mind wanders away attempting to explain the observed patterns. Those explanations may very well be flawed in themselves and should not be taken as evidence enough to trade any edge! As we know, we are inclined towards explaining even the inexplicable; or in other words; lying to ourselves. If such a lie would persuade us to trade curve fit edges, that could end less well.
Of course, many edges can be explained, which in such cases adds to their credibility. However, such explanations should only be taken into consideration after extensive robustness testing.
Curve fitting of robust edges
Many times when designing strategies you will find yourself in the grey zone. You might consider adding one more filter to improve the results only a little or choose a parameter value that has performed considerably better than the surrounding ones. For example, if your edge consists of a moving average, you might choose 14 for the average length, because 13 and 15 have performed much worse.
Given that your edge is robust, this type of optimization will most often lead to your edge being partly curve fit. This means that while it will most likely continue to perform going forward, it will do so with much poorer results than in the backtest. This could be devastating once you are about to determine your capitalization, meaning how much money is needed to trade. You could easily risk too much and face drawdowns that wipe you out completely!
Part II: What can be done to avoid curve fitting?
Let’s look at some ways to avoid or even eliminate curve fitting from your backtests:
Changing perspective
Until now we have only covered what curve fitting is, and why it is a pitfall that needs to be avoided. In this part of the article, we will briefly touch upon different methods that could be employed to validate an edge and discern true edges from curve fitted ones. These methods are often referred to as “robustness testing” since what we do is to validate the robustness of our edges.
With most things in trading, there are no universal truths that apply under all circumstances. The same applies to what methods to use to mitigate curve fitting. Each trader will, as his experience grows, find his own way of applying some of the methods available. Therefore, in this section of the article, only the most common methods to avoid curve fitting will be touched upon.
We will cover the following topics, in the order presented below:
- Less is more
- Parameter Stability
- In sample and out-of-sample testing
- Incubation
- Walk forward testing
- Monte Carlo simulation
Each topic is only presented briefly. Feel free to deep dive into our articles on every method! They are all linked below!
Let us begin!
1. Less is more in trading
In trading, you should always strive to make your edge as simple as possible. Overcomplicated edges with numerous conditions and rules tend to be curve fit.
- 2 Reasons Why Less Is More In Trading
- Simple Vs Complex Trading Strategies
- Street Smarts Beat Book Smarts In Trading?
2. Parameter stability
Parameter stability is a good indication of whether a strategy is curve fitted or not. In general, we want as many parameter combinations to produce desirable results as possible.
3. In-sample and out-of-sample testing
To avoid curve fitting, out-of-sample and in-sample testing is a crucial part of every trader’s methodology, in one way or another.
By dividing our data into a training set and a validation set, we can test our idea on the training set, and later verify it on the validation set.
4. Incubation
To put it outright, incubation is out of sample testing, but with one major difference:
With typical out-of-sample testing, all data is historical. With incubation, we use future data that is not available yet.
It is easier than it seems! Read our article on incubation, and you will soon get a grasp of it! It is one of the best methods available for those wanting to discover curve fitting before it is too late!
5. Walk forward testing
Walk forward testing is a concept that takes out of sample testing to the next new level. As all other methods discussed in this article, it is a great method of determining the robustness of a strategy and preventing curve fitting.
6. Monte Carlo simulation
Monte Carlo simulation in trading is a method that can be helpful in some cases. It works by reshuffling the order of the trades in a backtest and has the potential to expose weaknesses that otherwise had been hidden in the backtest.
Curve fitting – conclusion
Curve fitting is when our models are fit to random market noise and not true market behavior, which means that your trading strategy stops working. However, edges could also be partially curve fit, resulting in a massively degraded performance in live trading.
All in all, curve fitting is a major concern that every trader, beginner and experienced, actively need to battle to be able to trade profitably. The difference between profitable and failing traders is not that some do not curve fit. It is that some have found methods like those presented in this article that they actively use to decide which edges are worth trading and not.
