Curve Fitting In Trading: What is it? | Backtest Strategies - Overfitting

Curve fitting is when a strategy or edge is not fit to market behavior, but market noise, leading to failure in live trading. Curve fitting (overfitting) is overoptimization that is unlikely to fit into future unknown data.

You have probably read about curve fitting in trading books, trading magazines, and social media. The danger of Curve fitting is ubiquitous when designing trading strategies. It has the potential to ruin your trading career if not dealt with correctly and can be hard to notice, even for traders with decades of experience. In this article, we will learn what curve fitting is, and why you should try to avoid it. Or should you always avoid it?

(Before we go on we’d like to mention that we have a backtesting course that covers all aspects of how to backtest.)

Table of contents:

Key Takeaways

Curve fitting is the mistaken modeling of a trading strategy based on market noise rather than market behavior, leading to poor performance in live trading
True trading edges are rare and require distinguishing genuine market behavior patterns from random noise.
Avoiding curve fitting involves simplicity in strategy, parameter stability, in-sample/out-of-sample testing, incubation, walk forward testing, and Monte Carlo simulation.
Curve fitting (optimization) can be useful if you know what you are doing.

Curve Fitting in Trading Strategies The Enemy of Profit

Part I: What is curve fitting? (Overfitting)

Backtesting in search of edges

When we perform backtests, we analyze data in search of recurrent patterns that have predictive potential. In other words, we want to know if the tested pattern can tell us when the market is prone to going up or down so that we can be in the market only when it is favorable to us. If we succeed to find patterns that we think mirror market behavior, we have a trading edge.

Since our trading business is, or at least should be entirely reliant on edges, their quality and robustness dictate how well we cope with carving out profits in the market. Therefore, it is critical that our edges continue to work well also into the future if we want to make any money.

A thought experiment

Insinuate that your observation of what you presume is an edge, is flawed and holds no merit. Insinuate that you cannot know if your edge will continue to deliver going forward and that an overwhelming majority of patterns you call edges, will not work at all.

Quite scary, isn’t it, when you are about to risk your own money on those very edges?

Well, this is not a thought experiment. It is reality.

The harsh truth about observations of market behavior

When we search for edges in the markets, most of what we assume is an edge, will be outright garbage! True edges are hard to find, and in your search, you will sometimes be completely certain that you have an edge ready to trade, only to see it fall apart completely once exposed to new market data. This is one of the aspects that makes trading so hard for beginners to succeed in, and that needs to be overcome before risking real money!

Asking a question

Now that we know about the tendencies of the markets to deceive us into believing in false edges, it is time for us to ask a question to understand why this is, and what it has to do with curve fitting.

The question reads as follows:

Of all patterns we observe, only a few are true edges. How come that some of the observations we make are edges and others are not?

Or the same question veiled in other words:

How come some observations are true edges and others are curve fit?

To answer this, let us begin by learning a few lessons:

Lesson 1: Markets are mostly random

The first thing every trader needs to grasp to be able to understand the concept of overfitting, is that a majority of market action is random noise. Most market activity simply cannot be derived from any form of analysis and needs to be accepted as nothing else than random market noise.

Lesson 2: Most people want explanations, even to the inexplicable

We as humans have an urge to explain everything we see and experience. By doing so we bring order to a chaotic world, at the cost of quite often lying to ourselves. This tendency among humans can often be observed when financial news media covers recent market activity. The expert may ascribe soaring markets to some recent event, which seems perfectly reasonable. However, once the market turns around, so do often the experts explaining the downturn with the very same arguments.

In such cases, it is apparent that humans like to fit explanations to reality and not the other way around since both of our expert´s comments cannot be true at the same time.

The severe fallback of this inclination of the human mind is that reality is not very inclined to conform to our description of it. Curve fit edges will not hold, regardless of what reason we ascribe to its logic.

Lesson 3: Correlation does not equal causality – market behavior and market data are not the same

The third and last lesson we must learn before we can grasp the concept of curve fitting is that market data and market behavior are not the same.

Market behavior is non-random price action that holds predictive value, while market data consists of market behavior AND market noise combined. The consequence of this is that what seems to work in the backtest carried out on market data, cannot be taken for true market behavior before being put under scrutiny. It may very well be a result of randomness, thus holding no value going forward.

So, what is curve fitting?

Using what we now know from the three lessons in this article, we may define what we mean by curve fitting. Our definition reads as follows:

Overfitting is when random market noise forms haphazard patterns in price data, that are later viewed and considered an edge, despite being a product of sheer randomness.

To elaborate, when curve fitting, we don´t fit our models to market behavior. We fit them to market data. That is a huge difference since market data consists not only of market behavior but also of random market noise. For that model to be profitable going forward, the random patterns observed in historical data must repeat themselves. However, the one main trait of random patterns is that they do not hold any predictive value, since they are random.

Therefore, curve fit models nearly always fall apart in live trading.

An Illustration of the Concept of Overfitting

For it all to become a little clearer, let us illustrate the concept of curve fitting with an example:

Alan is building a house on the outskirts of the desert to investigate local wildlife. Since his arrival, it has not rained once, so he finds it unnecessary to build a roof that can handle large rain masses. After all, this is a desert, and who would expect rain to be abundant?

What he does not know, is that the period since his arrival has been the driest period for over 100 years. Quite soon after finishing his construction, he understands that his observation of the weather and the following conclusion that rain does not fall here was flawed. It was based on some random weather phenomenon that was not representative of normal weather behavior in the area. Soon, his house is flooded by the rain masses. His construction was curve fit to his observation of a random drought period.

Going back to trading

If we were to translate this story to fit with trading and curve fitting, we would get the following translations

Drought period = Market noise

Alan´s observation of the drought period = Our presumed edge in the market

The immediate failure of the construction = The failure of a curve fit edge

Alan curve fit his construction to a type of weather that was not typical of that region, despite believing it was. In the same way, traders curve fit their models to market data instead of market behavior.

Another thing Alan did, that we have not gone into yet, was to find an argument that supported his conclusion that rain is scarce in the area.

“…After all, this is desert, and who would expect rain to be abundant?…”

Most likely he had found supporting arguments for the very opposite claim if that was what he chose to believe in. It is always important to remain vigilant as soon as one´s mind wanders away attempting to explain the observed patterns. Those explanations may very well be flawed in themselves and should not be taken as evidence enough to trade any edge! As we know, we are inclined towards explaining even the inexplicable; or in other words; lying to ourselves. If such a lie would persuade us to trade curve fit edges, that could end less well.

Of course, many edges can be explained, which in such cases adds to their credibility. However, such explanations should only be taken into consideration after extensive robustness testing.

Curve fitting of robust edges

Many times when designing strategies you will find yourself in the grey zone. You might consider adding one more filter to improve the results only a little or choose a parameter value that has performed considerably better than the surrounding ones. For example, if your edge consists of a moving average, you might choose 14 for the average length, because 13 and 15 have performed much worse.

Given that your edge is robust, this type of optimization will most often lead to your edge being partly curve fit. This means that while it will most likely continue to perform going forward, it will do so with much poorer results than in the backtest. This could be devastating once you are about to determine your capitalization, meaning how much money is needed to trade. You could easily risk too much and face drawdowns that wipe you out completely!

Part II: What can be done to avoid overfitting?

Let’s look at some ways to avoid or even eliminate overfitting from your backtests:

Changing perspective

Until now we have only covered what curve fitting is, and why it is a pitfall that needs to be avoided. In this part of the article, we will briefly touch upon different methods that could be employed to validate an edge and discern true edges from curve fitted ones. These methods are often referred to as “robustness testing” since what we do is to validate the robustness of our edges.

With most things in trading, there are no universal truths that apply under all circumstances. The same applies to what methods to use to mitigate curve fitting. Each trader will, as his experience grows, find his own way of applying some of the methods available. Therefore, in this section of the article, only the most common methods to avoid curve fitting will be touched upon.

We will cover the following topics, in the order presented below:

Less is more
Parameter Stability
In sample and out-of-sample testing
Incubation
Walk forward testing
Monte Carlo simulation

Each topic is only presented briefly. Feel free to deep dive into our articles on every method! They are all linked below!

Let us begin!

1. Less is more in trading

In trading, you should always strive to make your edge as simple as possible. Overcomplicated edges with numerous conditions and rules tend to be curve fit.

2. Parameter stability

Parameter stability is a good indication of whether a strategy is curve fitted or not. In general, we want as many parameter combinations to produce desirable results as possible.

3. In-sample and out-of-sample testing

To avoid curve fitting, out-of-sample and in-sample testing is a crucial part of every trader’s methodology, in one way or another.

By dividing our data into a training set and a validation set, we can test our idea on the training set, and later verify it on the validation set.

4. Incubation

To put it outright, incubation is out of sample testing, but with one major difference:

With typical out-of-sample testing, all data is historical. With incubation, we use future data that is not available yet.

It is easier than it seems! Read our article on incubation, and you will soon get a grasp of it! It is one of the best methods available for those wanting to discover curve fitting before it is too late!

5. Walk forward testing

Walk forward testing is a concept that takes out of sample testing to the next new level. As all other methods discussed in this article, it is a great method of determining the robustness of a strategy and preventing curve fitting.

6. Monte Carlo simulation

Monte Carlo simulation in trading is a method that can be helpful in some cases. It works by reshuffling the order of the trades in a backtest and has the potential to expose weaknesses that otherwise had been hidden in the backtest.

Overfitting Is Good In Trading: How To Curve Fit Successfully

Curve fitting is always mentioned negatively in trading. This section takes the contrarian view and explains why curve fitting is good. Curve fitting is good in the sense that not all stocks can fit into a certain trading strategy or style.

If you are new to systematic trading, we recommend you look at our complete backtesting guide. This article assumes you have at least a basic knowledge of backtesting.

First, let’s start with explaining…..

Curve fitting is the process of optimizing the parameters of a trading strategy to fit the historical data. This can be done by manually adjusting the parameters or by using an optimization algorithm.

The problem with curve fitting is that it can lead to overfitting. Overfitting is when a strategy performs well on historical data but poorly on new data. This is because the strategy has been optimized to fit the specific noise in the historical data, rather than the true underlying market dynamics.

Curve fitting is likely to happen with more parameters you put into the strategy. The more parameters, the more likely the strategy won’t adapt to future data. Generally, a trading strategy is more likely to survive the simpler it is. Complex strategies are very unlikely to last for long.

Unfortunately, all backtesting is partially curve-fitting, because history rarely repeats itself.

If you are worried about curve fitting, you might want to either run out-of-sample tests, walk-forward analyses, or, better yet, run your strategies live in a demo account for many months as an incubation period. We recommend reading our article on out of sample backtesting.

When curve fitting is good – an example

However, we have been using curve-fitted trading strategies with success for a long time. In this section, we’ll give you an example of how we did curve fitting when we were systematic and automated day traders from 2001 until 2018.

This is what we did for a stock trading strategy (trading multiple stock tickers, for example, the whole universe on NYSE but using filters for size, volatility, volume, etc.):

Define a trading strategy with specific trading rules.
Backtest the strategy on a wide range of listed stocks.
Pick the stocks that have worked well over the last N years, for example, 5 years. Test those stocks going forward.
Is there a pattern? Are there groups of stocks that have performed badly?
Exclude sectors/groups performing poorly.
We excluded all stocks that performed poorly.
Rinse and repeat quarterly, semi-annually, or annually (we did quarterly).

This might not be a recipe for success for all traders, but it worked for us. And in our opinion, it makes sense. Please continue reading to better understand why.

Is Jim Simons curve fitting?

We are looking for patterns, and we might argue that if you reject 10 patterns but trade one, you are curve-fitting.

Can we argue that curve fitting is the edge? Look at what Jim Simons and his team have done in the Medallion fund. Jim Simons has multiple times said that they don’t ask why a strategy might work, they just trade it as long as it’s statistically significant.

We might argue that this is curve fitting. After all, they test a vast number of hypotheses, scrapping those that don’t work and keeping those that do. If you backtest thousands of strategies, you’ll be sure to find something by chance simply because you backtest a lot.

Is curve fitting a trading edge?

What could be the deciding factors for including or excluding a stock? We list the two most obvious factors:

Sectors or industries. Some sectors don’t work. Oil stocks and commodities are notoriously bad, in our opinion. But it might change over time. For example, REITS were good in the early 2000, but it all changed in 2007 when the financial crisis started coming in.
Volatility: volatile stocks are not good. Of course, it depends on the type of strategy. We have always regarded the most boring stocks as the best one to trade. Volatile smallcap stocks have not made money since at least 1963!

Perhaps curve fitting is a trading edge in itself?

There are other factors we looked at, especially when we were day trading stocks, but we believe you’ll make some interesting discoveries yourself if you try this approach.

This is not a recommendation to make complex strategies. But you might consider giving the above approach a try. There are many reasons why a particular strategy might not work for certain stocks and vice versa, other stocks might behave differently.

To summarize, don’t expect a strategy to work on all stocks or all markets. Markets differ. But make sure you know what you are doing! This is a part of the backtesting process that is highly dependent on experience.

Curve fitting – conclusion

Curve fitting occurs when our models are fitted to random market noise rather than true market behavior, which means that your trading strategy stops working. However, edges could also be partially curve fit, resulting in massively degraded performance in live trading.

All in all, curve fitting is a major concern that every trader, beginner and experienced, actively needs to battle to trade profitably. The difference between profitable and failing traders is not that some do not curve fit. Some have found methods like those presented in this article that they actively use to decide which edges are worth trading and which are not.

FAQ:

What are the challenges traders face when searching for true edges in the markets?

Explore the difficulties traders encounter in identifying true edges amidst a sea of potential patterns, highlighting the unpredictable nature of market behavior.

How do lessons about market randomness, human tendencies, and correlation vs. causality contribute to understanding curve fitting?

Gain insights into lessons about market randomness, human biases in explaining market phenomena, and the distinction between market behavior and market data.

How can traders avoid or eliminate curve fitting from their backtests?

Explore various methods, including less complexity in trading strategies, parameter stability, in-sample and out-of-sample testing, incubation, walk forward testing, and Monte Carlo simulation.