Profitable systems vs curve fitting
- This topic has 18 replies, 9 voices, and was last updated 7 years ago by
.
-
Here are my tips:
- OOS-testing (out-of-sample testing) = optimize system on for example 50 % of the data, then test performance on the untouched data.
- Walk-forward optimization (built in tool for iterative optimization)
- Test the system on a live demo account (a form of live OOS testing)
- Reduce number of optimized variables to increase the robustness/reliability of the system. Or optimize each variable with for example 4 increments/steps instead of 30,
- Increase the number of trades the backtest makes on the available data, either by buying more data or making the system trade more often. 500 trades is much more statistically valuable than 50 trades.
- When performing your backtest, I have a rule of thumb. If the optimization finds the best variables out of 100 possible combinations, look at the whole list of results. The worst combination of variables, is that still a profitable strategy? Does that strategy lose less money than the best one wins? What is the ratio between losing/winning combinations? The result you find in the middle of your optimized results (number 50 on the list), will PROBABLY be the closest your real outcome in the end, even if you are using the best set of variables.
- Take into account the different CFD spreads throughout different times of day, possible overnight fees for open positions, and slippage if you are trading in low-liquidity market. Always apply this to the result of the backtest.
- If you are backtesting on 3 months of data and start a system based on that backtest, why would that work 5 months from now? You did not take into account how markets can change in 5 months, only 3. I would call this a form of “lindy effect”. The shorter the timeframe/data is, the quicker market conditions will change and systems will stop working, and the opposite for higher timeframes. For this reason you may want to periodically recalibrate your systems, or just decide to scrap them.
If backtests could foretell the future and market behaviour didn’t change, we would all be rich, but in the process of becoming rich, we would change the markets, so that is a paradox. One truth about system trading is that you are not trading the underlying security, but really the system’s equity curve itself, which is affected by the underlyign behaviour.
10 users thanked author for this post.
Just another note about curve fitting. This is actually fine if your are optimizing your strategy every day for say, the last 3 months. You’ll still have a bit of the curve left to play with.
A good test of this is the good old golden cross strategy. Pick a small time frame – like M5 (or 100 ticks) and two moving averages and see how the golden cross goes in-and-out of favour as the price action dynamics change over time. You’ll see periods of winning streaks and loosing streaks in a cyclical fashion, from one epoch to the next. This will normaly coincide with business cycles, seasonality and macroeconomic events. If your strategy is one that dies out slowly, curve fitting is not so bad as long as you re-optimize every night in order to stay with the curve. *calls for an auto-optimize function*
Best,
M
ps Anything with a moving average of any kind is subject to curve fitting and needs regular re-optimization!
2 users thanked author for this post.
In my opinion what should be done in analysing the results of an optimization is the following:
- Analyse the shape of the N dimensional surface obtained by changing the variable value so, let’s suppose you are optimizing N variables, you take the N dimensional surface in the N+1 dimensional space and you investigate the stability of the variable in consideration by studying all the partial derivative of your surface with respect to the variables. If the surface is “smooth” around the optimal value for the ith variable then the optimization is promising
- for every variable that you opimize one should be able to draw a time series representing the optimal value of the variable i_th as a function of time. So for example you have 5 years of data, you calculate the optimal value of a variable every 3 months and you will have 3*4*5 data representing the optimized variable as a function of time. By looking on how smooth is this curve you will have a good idea of how stable is your system.
I am a physiscist by background and would love to be able to translate abstracts concepts like the one above into practice but I lack proper computer science education. Pls shout if you want to collaborate in a more systematic way.
Regards
Francesco
The first thing I do before getting into the more time-consuming backtest metods is to just change one of my variables at a time (stress test), using small steps up and down.
Write down the new result from each time you change one variable one small step.
If you get a dramatic change in your equity curve then this variable is probably not giving a realistic return for future market behaviour. By doing this you can see for what values the equity curve is going up/down. And if you get one value that gives a significant better return than the values surrounding, then this value is most likely an out-lier and not suitable for future market behaviour.
If your strategy’s equity curve is still intact after this “stress test” then you can continue with more proper optimisation.
I also make sure that the strategy works on several markets before spending any more time on it.1 user thanked author for this post.
@Francesco: I appreciate your mathematical way looking at the problem (i’m from the physics department myself) but i doubt that such analysis can be done with PRT. I think one would need matlab or similar to make such studies.
1 user thanked author for this post.
Suppose you are running a 1hr strategy.
3yr Backtest.
WF 70/30 test passed
5 variables optimized.
first approach: reoptimizing over the last 6 monhts every 3 month.
second approach: reoptimizing over the max period available every month.
Which one would you choose ideally?
Many thanks.
Francesco
1 user thanked author for this post.
@victormork – Yep, you’re right.
Also on the subject of optimization time… lots of different algorithms available for iterations of this kind. I actually made a suggestion in another thread that PRT should role out more intelligent iterating options for their optimizer. You’re basically approaching this like a sorting problem. The goal is to follow the profit trail (or hit rate, or whatever you’re sorting by) so that it won’t be nesecary to try every single combination in a brute force manner.
Here’s an example. Say you have 4 variables to iterate through.
- Set A to iterate and fix B, C an D somewhere sensible, middle of the range perhaps.
- Find best option for A
- Fix A, and iterate through B; find best of B
- Fix A and B, iterate through C.. and so on
Once done, repeat from the beginning again. Keep repeating until the system can’t get more profitable. Order your variables logically by most significant to least significant.
That’s just one method. Many are available.
3 users thanked author for this post.
Thank you to all of you, one thing I am a bit surprised not to find on PRT is a sharp ratio optimizer, after all who really cares about maximizing the profits? We all want a steady curve, will that something the developer are going to look at?
Thanks.
Francesco
1 user thanked author for this post.
Steady equity curve of multiple strategies /portfolio is what we are all looking for for sure.
There are many different ratios or ratings scores for backtests, I know that the backtest report of the platform will be updated soon, it’s not too late to add more significant things to it, any ideas or suggestions for that? Thanks.
1 user thanked author for this post.
Find exclusive trading pro-tools on 
