Technique help to reduce over-fitting
- This topic has 63 replies, 12 voices, and was last updated 3 years ago by
.
-
Hi everyone,
Today, there is a few tools/technique to avoid over-fitting, namely probacktest walk forward analysis with OOS, avoid too much variables (subjective, right?), monte carlo, robustness tester by Vonasi, avoid trade in low time frame (subjective also, right?) and etc…
I like to see if there is other techniques to help avoid over-fitting, here I’m sharing 2 more that I use occasionally:
1. Optimize variables together with possibility to trade on different bar, example as below
1IF timeok AND Not OnMarket AND close > open AND C1 AND (oddtest = 1 or barindex mod oddtest > 0) THENC1 can be for example C1 = close > Average[Period], then you optimize oddtest and Period.
oddtest with range 2 to 10, step 2, but can be increased according to the need
Period range and step depending which way to analyze the optimization:
a) No. of combination exceeding 100: the top result will be selected and the more occurrence of a Period value at the top can means higher chance when trade opened at different combination of barindex, it is more resilient. PS: Con is PRT always arrange result to highest gain%, but we know highest gain% not always means a better strategy
b) No. of combination not exceeding 100: exact the optimization table to excel, sort the average gain and win rate, then rate them accordingly (depends which that you like)
Note: PRT optimization windows is limited up to 100 combinations. Con of this technique is limited no. of variables can be studied at a time, basically use it when you have dilemma to choose which value for a certain variable
2. Design variables to conflict each other when optimizing it too overfit, example as below
123456789SLpoint = SL * 0.01 * closestartBreakeven = MIN(atrH1, 75)startBreakeven = MAX(startBreakeven, 50)trailingATR = MIN(atrH1, SLpoint)trailingATR = MAX(trailingATR, startBreakeven)SET STOP %LOSS SLAs you know, higher stop loss can contribute to high win rate. Above code is having,
- SL (stop loss) as % of asset value, higher can contribute higher win rate
- trailingATR = points gained for trailing start, lower can contribute to higher win rate
In this case, if you optimize SL to be as high as possible to gain higher win rate (indirectly total gain too), then it can impact trailingATR which also contribute to win rate. So the optimization has to balance by itself for highest gain.
Maybe you have other technique that you used during optimization to reduce over-fitting to share?
1 user thanked author for this post.
are we supposed to trust a scenario where our strategy take 3 times less trades than ordinary ? is it really a feasible scenario ? i don’t think so. A difference of 20 or 30% on the number of trades on a same period would be still possible, but not more.
I agree with you totally, if the strategy normally take 300 trades/year, then not acceptable to have too little trades. Actually when using Vonasi tool, I need to adjust the setting, not following the default 10-10, 20-5, 21-5, but something like adjust the 10, 20 and 21 to something lower, these values are used to modulas the barindex thus, the higher value, it will gives less trades, because it means it allows trade on every 10 bars, 20 bars and 21 bars. For me, it depends on strategy too, I think Vonasi is fan of high time frame strategy, thus 10, 20, 21 makes sense for high time frame as probably not aimed for high frequency trades. As I’m fan of high frequency trades, generally the strategy should have 2-4 trades/day, thus I need to adjust it lower, to something between 3 to 10, e.g. 5-21, 6-21. So I guess overall, need to apply accordingly to the goal of the strategy
Same concept for the technique 1a), you will adjust the oddtest value accordingly to suit the need, but in this usage, it may allow lower trades, because your analysis is not based on single oddtest, but oddtest can give you the “top” parameters in all combination of oddtest, thus help to narrow down the range
Btw, I have a mistake on giving my example above, oddtest should start with 1 to 10, step should be in odd number, e.g. 1 or 3, when oddtest = 1, it means no bar will be skipped, so you will see the original result. When oddtest increase in the step of 3, the backtest will then skip barindex in modulas of 4, 7, 10
oddtest with range 2 to 10, step 2, but can be increased according to the need
But I find Monte Carlo assessment sometimes helpful for me, for example recently, I analyze one of my strategy that looks good on profit curve to see that the strategy has been focus too much on low profit trade (see attachment), and not good enough to cut stop loss earlier, so I go back to update it. I think what important in these assessment is also to be able to see if any room of improvement to the strategy, instead of just to have a nice graph (on top of nice equity curve).
Honestly, I cannot take any credit here, rather I should thank to all the fantastic developers in this forum, that enable me to learn more and more 🙂 (I’m really new here and continuous learning)5m timeframe and the number of trades of the results is pretty similar
Yeah, for 5m like MoD seems ok too, but I find these sets give too little trades when I’m playing on 1m time frame
how would you know what is the better number to change?
Not sure I can say better number, but I test with several sets actually, changing Qty lower and increase the Random to maximize the combination up to 100. I maintain Qty no less than 4 and ensure no less than 25% of trades vs the trade no. from the strategy, e.g. if strategy has 300 trades, I want it to have no less than 75 trades
1 user thanked author for this post.
Hi, I personally do not use anything but when I build a trading system I ALWAYS stop at 31/12/2017 with any time frame …. after checking that from 01/01/2018 to today the system is similar both in performance and in drawdown …. if the result is not good I do not change anything and delete immediately without changing the variables that would fix it for the past but the future would probably be a disaster ….. if it is ok instead I put it in demo for a period minimum of 4 months up to one year based on the number of operations ….. after this second verification I can put it in real ….. as long as it continues to be completely similar to the performance of the Backtest.
as long as it continues to be completely similar to the performance of the Backtest.
Yes but without a robustness test how do you know that your back test didn’t just get lucky and hit all the winners and your forward test didn’t just get lucky and hit all the winners? It’s the trades that you don’t see in your back and forward test that are the ones you really need to see to know that you didn’t just get lucky.
1 user thanked author for this post.
Hi Dow Jones ….. Yes, of course the WFA must be used is a great tool … but I feel at ease with the system described above it gives me security and also results … of course out of 50 trading systems only one it’s working …. then you have to be careful to put it in demo for some time and not everyone has this patience …. the important thing is NEVER reoptimize it if it’s not good, otherwise you deceive yourself.
1 user thanked author for this post.
Find exclusive trading pro-tools on 
