Data Mining And Boredom (The Same Strategy Can’t Be Used On All Stocks)
Last Updated on July 7, 2022
Why should, for example, a 3-day RSI work on both PG and HAL? Read below for why it makes sense to differ strategies among stocks.
I saw a discussion on Twitter the other day about data mining and backtesting.
Put short, one trader trades different strategies for different instruments/stocks. The other trader believes this is data mining. For example: To buy when RSI(3) is oversold might work on different stocks and not on others.
If we trade this strategy on the stocks it works on, and not on the stocks it does not work on, is this data mining? Or should we only trade strategies that work more or less on all S&P stocks?
I believe the first is correct. Why should one strategy work on both APA and PG? These are completely different stocks from different sectors. To me, there is no logic to implement the same strategies in different sectors. I have tested thousands of strategies and all sectors behave differently. Even within sectors, you have to regroup the stocks.
How to pick stocks for each strategy
I pick stocks for my strategies based on how they perform for this strategy over the last 3 years. If a stock does not show an upward sloping equity curve, I simply reject it from my strategy.
Is this data mining? Perhaps it is. But this is the way I have done it over the last 12 years and it works really well for me.
Have I just been lucky? I don’t think so. My trading is “out of sample” testing and my group of stocks performs better than any random group. I have tested this a lot of times. The key to successful trading is to look for patterns/attributes in stocks and trade what you find.
Of course, it’s only logical that stocks have different patterns (at least to me). Sometimes you just happen to find a random pattern so there must be some kind of logic behind why this pattern should exist. Even better if you have some logic search BEFORE you test patterns.
Trade many uncorrelated strategies
I believe many will reject my methodology. That’s fine. I read a lot of blogs and see a whole lot of different theories. Usually, the blog post ends like “this is not recommended as a stand-alone strategy”. I have picked up so many ideas from blogs and traded them, and still, the author thinks his strategy is not worthwhile as a tradeable strategy. The best thing is to trade a lot and trade with different strategies. This is the holy grail of trading! Even some erratic equity curves contribute when having about 20 uncorrelated strategies. (I believe a lot of the blog authors don’t trade).
Trading should be boring
At the end some words on boredom and introverted traders:
I believe trading should be boring. Perhaps contradictory, but the more boring the better you can do.
Why? Trading only suits people with the following attitudes: patience, analytical, and somewhat risk-averse.
Yes, you have to take risks, but it needs to be balanced. All the traders I know that have lasted more than ten years have been somewhat introverted and very rational/analytical and not risk-takers. They don’t trade to get a rush, but to make money. They are happy to test strategies, but not keen on execution. When there is little prey in the markets, they simply wait out for better times. Of course, they have made money before and have the financial means to wait for many months.
Well, I (mostly) don’t trade stocks, but indices. I would still like to offer a defense of “Why should one strategy work on both APA and PG”. There’s probably a bit of difference between saying that a strategy should work on EFA, EEM, and SPY vs saying it should work on 3 random S&P 500 stocks. There’s more room for idiosyncrasies when it comes to individual stocks, and far more noise (which makes proving that a model is “good enough” far more difficult).
What I look to model are not the movements of any one index, but the underlying behaviors of traders and investors that subsequently produce similar patterns in ALL indices. If something works on SPY but not on QQQ, does that mean it must be meaningless? Of course not.
But there are some issues with that approach: there is a greater chance that the effect you’ve found is spurious, and since you’re limited to a single instrument there is far less data to work with, again leading to greater uncertainty about whether the model you’re testing is a true reflection of the process behind the price movements. Not to mention that you’re left with the question: why do QQQ traders behave differently from SPY traders? I’d rather not have to answer that at all!
Of course you have good results with your approach, so it can work. Diversification counters the uncertainty, as long as a reasonably large portion of your strategies are truly good. But from my perspective, and in connection with your assessment of trading personalities, I think that requiring your models to work across a wide array of instruments is the more risk-averse approach.
Thanks for comment. Yes, I agree with you, but I think the “multiple strategy effect” diminishes the effect of my once in a while “spurious” strategies. Some strategies I have traded for years, some I trade for less than 6 months.
Hi Oddmund, and thanks again for an informative post.
I have a couple of side questions, if you care to comment. You mention above that you pick stocks for your strategies based on the last 3 years of performance. Does this mean you test your strategies doing backtesting of the last 3 years? Do you leave some time period for ‘out of sample’ testing, or you ‘curve-fit’ and jump straight to real trading?
Do you have a minimum number of backtest trades, so you will feel comfortable that the equity curve does not look good by mere luck?
I have to admit, I am a fan of your ideas, because you seem to have a conscious ‘keep it simple’ approach (that seems to work well, based on your writings), versus most other writings out there.
Hi, thanks for your nice words. I backtest my strategies roughly (I mainly trade stocks) to get a grasp of the strategy. I might test manually 500 stocks, and trade just 100 of them based on a visual look of the equity curve for each stock. Then I start trading with minimum size possible (this is my out of sample test). I might do this for one month. If my results are somewhat near testing, I increase the size (gievn i’m profitable). Experience tells me I will be glad to get 50% of the backtesting results, mainly due to overoptimization and slippage. I try to get as much fills as possible. Over a 3 year period I will see at least 150 trades in each stock. Still, my approach is overoptimized so it’s impossible to get nowwhere near the testing.
Thanks Oddmund, it’s almost unbelievable how simple and practical you keep things. A good lesson for all of us.
What I have learned is that you test a lot of things, but you have to trade to get ideas. Most of my ideas come from real trading. And I try to trade a lot to smoth things out.
Hi Oddmund, good post.
I have also found that what allot of people consider to be dangerous “data mining” actually improves performance in real trading. I think the key is two things – Being able to adjust to your own real-time results without panicking, and also continually looking for new strategies, and improving old ones. I think some people want to find one strategy and parameter set that will work on a large basket of markets for 20 years without updates (like some trend followers), which is a totally different thing, in my opinion. The other group are people who think everything needs to be theoretical or “intelligent” which I think blinds many to opportunities.
One other thing. I sometimes say ““this is not recommended as a stand alone strategy” at the end of my posts, but it is not necessarily for the reason you suggest. I think I say that for a few reasons. First, one strategy on one market (at least based on my approaches) is usually not going to generate enough trades to be super worthwhile and deliver great results. Second, considering each idea I have as part of a team helps me to stay relaxed when trying something new Lastly on the markets I trade most often I have a group of signals (what I call trading factors), that I rank every day to give a net signal, and in this case when I say, “not stand alone” it is really describing how I use it – not that the idea is not worthwhile, etc.
Really thought-provoking post. Takeaways for me: (1) it doesn’t matter how you win, if you win. Optimize or not, it’s really secondary after a consistently profitable track record and if you keep your bets reasonable.
(2) Secondly, I liked your point re not everyone on blog-sphere actually do real trading, which is also my hunch, and I wonder why. I ‘d be happy with a system that generates upward of 12% non-compounded and CAR/MDD close to 1.
and (3) I loved your point: not working for the man. I’m longing for the day where my capital reaches significant size to be able to ditch my day time job for good.
Insightful blog, Oddmund,
I also trade stocks mostly on a mean reversion basis. I was interested in your idea so I decided to run a simple test which basically runs one of my mean reversion systems across a universe of 3000 US stocks over a 3 year period. I then filtered this list of stocks based on a minimum number of trades for each stock (5) and also its annual return (must be positive). I then tested the same system against this new watch list of stocks over the next 3 years (out of sample). The results indicated a significant drop in performance compared to just trading against the entire portfolio of 3000 stocks. Any thoughts on this? Is this how you approach filtering your universe of stocks to trade for a system?
To some extent I use the same method. However, I have a lot more trades, usually much more than 100. And I use this method mainly for daytrading, not EOD.
I believe the relevant questions here is, should we optimize a systems parameters or should we optimize the stocks it trades? The answer I believe is to optimize the stocks to trade for reasons you have stated. Now consider this scenario, suppose we have a trading system for the S&P 500. If we decide to cherry pick 250 stocks which we believe the system will perform best on, then assuming that the benefits of diversification are already achieved with 250 stocks, in actual trading the worst case scenario must be that the performance of the 250 stocks is the same as of the 500 stocks (this is the case where we have curve fitted in stock selection), however the best case scenario is that these 250 cherry picked stocks perform better than the 500 (the case where the 250 stocks are truly better). So with this logic there is nothing to loose from this approach.
I have traded over 3000 tickers in my career. Usually what I do is to look at my best stocks overall, those I actually make money in, and test on those. I’m a mean reversion trader (most of the time), and those stocks tend to be better than others for a reason. So that is an obvious method to get started with “optmisation”.
One way to address this is through hypothesis testing. Backtest the strategy on all instruments and then look for statistically significant (negative) deviations from the mean. If the results are not extreme enough it’s probably just random chance.