how to avoid overfitted strategy

• 05/15/2023 at 9:24 AM #214705
  Aragorna

  Hello to all,

  I’m writing this post after many tries to make a strategy, and many doubts are still there. The problem of avoiding  a overfitting strategy is the first. I’ve read many posts about it, and the solutions that have been showed to avoid it, like for example Montecarlo, walk forward or Robustness test; but when I’m doing one of this test, I’ve already used a lot of energy to make a strategy. So I’d like to understand with the community, what is the process of making a strategy that can avoid since the very beginning a overfitted strategy. Just some free thoughts.

  1. when I make a strategy, what is the number of indicators, above which I’m pretty sure I’m overfitting my strategy?
  2. when I use, let’s say, 2 indicators for a strategy  on a market, and I choose some parameters, in that moment I’m fitting the indicators to the market. If, for example, I barely know one of the 2 indicator but I see some potential, in that case I don’t know how to set parameters for it. How bad is to optimize it at the beginning, just to find the area of correct values ? and if the values that give good result for a strategy are spread and not well oriented, does it mean I’m on the wrong way?
  3. another question is, if my strategy with these 2 indicators, set with standard parameters, gives in the media not a extraordinary result and then choosing the parameters it gives great results, I’m overfitting?
  4. I’ve read many strategies in the forum, the majority of them, when I Changed slightly a parameter, gave bad result, but no one here said anything about a overfitted. Is that a way to determine if a strategy is overfitted?

  all these questions come before the classic tests to determine overfitting, thank’s to who can answer me.

  regards

  Alessio

  

  Aragorna

  Participant

  Topics: 36

  Replies: 130

  Been thanked: 12 times
  05/15/2023 at 11:27 AM #214710
  PeterSt

  Dear Alessio,

  To my firm belief, one can graduate on mastering Backtesting.
  I have often thought to write out the rules I apply (to) myself, but the context is too complicated to describe, and what would be applicable to me, won’t be applicable to you.

  Let’s say that it may takes years to understand your own one single strategy, with its parameters, all thought of (designed) yourself. Of course all depends on the methods used and the parameters to control all.

  I don’t have real answers for you, but I do have one crucial measure (I recently told about it already) :
  Backtest over 50% of the available time period – when done, do the 100%. You will see immediately *if* you over-optimized. So at least you have a start now.

  ---

  Attempt after all for short answers :

  when I make a strategy, what is the number of indicators, above which I’m pretty sure I’m overfitting my strategy?

  None that I know of. But wait : if you mean that you throw several parameters to iterate at the same time, then my answer is : definitely 2. But usually only 1 is allowed.
  The 2 I would use to find “direction”. You speak of that yourself too :

  when I use, let’s say, 2 indicators for a strategy  on a market, and I choose some parameters, in that moment I’m fitting the indicators to the market. If, for example, I barely know one of the 2 indicator but I see some potential, in that case I don’t know how to set parameters for it. How bad is to optimize it at the beginning, just to find the area of correct values ?

  This is good. Call it empirical finding. A crucial necessity. But :

  and if the values that give good result for a strategy are spread and not well oriented, does it mean I’m on the wrong way?

  Yes. And it is one of the more easy things to see through; never ever pick a value which doesn’t belong in the sequence, no matter it brings you 10 fold the profit.
  Regarding this, examine the list of closed positions. 10:1 you trapped a News event and caught accidentally the right direction.

  if my strategy with these 2 indicators, set with standard parameters, gives in the media not a extraordinary result and then choosing the parameters it gives great results, I’m overfitting?

  Don’t listen to the media anyway ? An example would be an RSI. 30-70 ? sure. But start with thinking whether you go short at 70 or whether you go Long at 70 ! Both work, but it is the further context (of your own strategy) which will be for the better.
  In the very end you could end up with not using the RSI at all, because it’s vague. It’s ambiguous.

  I’ve read many strategies in the forum, the majority of them, when I Changed slightly a parameter, gave bad result, but no one here said anything about a overfitted. Is that a way to determine if a strategy is overfitted?

  This is difficult to answer. The general consensus could be that nothing you run into is any good to begin with, but this is not the message  – people may have worked very hard on it and are enthusiastic about their produce and like to share it (often asking for improvement). But let me give another kind of (indirect) answer) :
  Usually if *I* change one of the parameters randomly – and let’s say 20 participate in the good-working of the strategy – then barely anything happens. This is because they all work together and all pull on each other. Example : if you set up a TP/SL ratio, then you can change one of them, the other will be pulled with it, and the result is relatively small. This example is visible.
  Now if I do not have this ratio and only change the SL to a deeper level, my gain may still remain in balance, although the risk/reward changes drastically. How come ? well, my win ratio will rise (say from 60% to 90%) BUT (!!) I stay much longer in the market and net I make the same – or even less profit. This is not visible, this is knowledge (experience, if you want). But anyway, in this case too you can drastically change a parameter (a SL of 100 is really different from a SL of 2000), the net result my be quite the same. And this is because of so many other parameters “pulling” on the result. What this actually does (in my experience) is making the strategy more robust. Thus not more profitable – more robust. More consistent. Less dangerous that it may be losing in the future.

  ---

  Your starting point could be to recognize what actually causes which. Take the longer in the market example. Don’t think about the SL now, but the means to take the most profit. What is better ? a large trailing distance so it won’t go out with smaller retracements, or a short distance so it can happily hop to the next trade and make profit again. Small prerequisite : the strategy must be profitable first.  This means a fair number of winning trades (for me this is around 62%). This does NOT mean the most profit. Not at all. So first the winning trades, and next the trailing to expand the profit on those trades. And :

  The more trades, the more possibilities in everything.

  Now think back of your parameters. What are they actually about ? (no public answers needed – it is only for yourself to think over)
  Peter

   

  

  PeterSt

  Participant

  Topics: 73

  Replies: 2544

  Been thanked: 829 times
  05/15/2023 at 1:00 PM #214736
  Aragorna

  Hi Peter, thank you for your precise answer.

  Attempt after all for short answers :

  1)when I make a strategy, what is the number of indicators, above which I’m pretty sure I’m overfitting my strategy?

  None that I know of. But wait : if you mean that you throw several parameters to iterate at the same time, then my answer is : definitely 2. But usually only 1 is allowed.
  The 2 I would use to find “direction”. You speak of that yourself too :

  so 2 indicators are the maximum allowed but one has to give the direction. There is possibility of overlapping and this can cause random results, for my little experience.

   

   

  when I use, let’s say, 2 indicators for a strategy  on a market, and I choose some parameters, in that moment I’m fitting the indicators to the market. If, for example, I barely know one of the 2 indicator but I see some potential, in that case I don’t know how to set parameters for it. How bad is to optimize it at the beginning, just to find the area of correct values ?

  This is good. Call it empirical finding. A crucial necessity. But :

  and if the values that give good result for a strategy are spread and not well oriented, does it mean I’m on the wrong way?

  Yes. And it is one of the more easy things to see through; never ever pick a value which doesn’t belong in the sequence, no matter it brings you 10 fold the profit.
  Regarding this, examine the list of closed positions. 10:1 you trapped a News event and caught accidentally the right direction.

  So a sequence is necessary to have a defined boundary of the strategy.  then it’s possible to see random good result outside the sequence but sequence is priority. 

   

   

   

  if my strategy with these 2 indicators, set with standard parameters, gives in the media not a extraordinary result and then choosing the parameters it gives great results, I’m overfitting?

  Don’t listen to the media anyway ? An example would be an RSI. 30-70 ? sure. But start with thinking whether you go short at 70 or whether you go Long at 70 ! Both work, but it is the further context (of your own strategy) which will be for the better.
  In the very end you could end up with not using the RSI at all, because it’s vague. It’s ambiguous.

  my doubt is: if I start a strategy (that has an idea behind) with 2 indicators and my equity goes up and down, but after the choose of the parameters the equity is good, I’m overfitting? strategy should work the same with every parameters and then it has only to be optimized? because if it’s so, it’s very difficult to find some thing that works without fitting some how.

   

   

   

  I’ve read many strategies in the forum, the majority of them, when I Changed slightly a parameter, gave bad result, but no one here said anything about a overfitted. Is that a way to determine if a strategy is overfitted?

  This is difficult to answer. The general consensus could be that nothing you run into is any good to begin with, but this is not the message  – people may have worked very hard on it and are enthusiastic about their produce and like to share it (often asking for improvement). But let me give another kind of (indirect) answer) :
  Usually if *I* change one of the parameters randomly – and let’s say 20 participate in the good-working of the strategy – then barely anything happens. This is because they all work together and all pull on each other. Example : if you set up a TP/SL ratio, then you can change one of them, the other will be pulled with it, and the result is relatively small. This example is visible.
  Now if I do not have this ratio and only change the SL to a deeper level, my gain may still remain in balance, although the risk/reward changes drastically. How come ? well, my win ratio will rise (say from 60% to 90%) BUT (!!) I stay much longer in the market and net I make the same – or even less profit. This is not visible, this is knowledge (experience, if you want). But anyway, in this case too you can drastically change a parameter (a SL of 100 is really different from a SL of 2000), the net result my be quite the same. And this is because of so many other parameters “pulling” on the result. What this actually does (in my experience) is making the strategy more robust. Thus not more profitable – more robust. More consistent. Less dangerous that it may be losing in the future.

  <hr />

  Your starting point could be to recognize what actually causes which. Take the longer in the market example. Don’t think about the SL now, but the means to take the most profit. What is better ? a large trailing distance so it won’t go out with smaller retracements, or a short distance so it can happily hop to the next trade and make profit again. Small prerequisite : the strategy must be profitable first.  This means a fair number of winning trades (for me this is around 62%). This does NOT mean the most profit. Not at all. So first the winning trades, and next the trailing to expand the profit on those trades. And :

  The more trades, the more possibilities in everything.

  Now think back of your parameters. What are they actually about ? (no public answers needed – it is only for yourself to think over)
  Peter

  I usually don’t use stop loss but, for what I learnt, strategy has to work with its entries and exit conditions. SL and TP is a plus for the result.

   

   

   

  Last doubt is about the time for backtest. I’ve seen many strategies fitted for the past, also 6-7 years ago, and a little for the last 1-2 years. Trying to fit a strategy for a market that has passed many different context and situations, with a great difference in volatility during the years, in particular indexes,  make me think that is better the fit a strategy for the last 3-4 years. I see that many automatic traders continue to modify parameters of a strategy many times during a year and this makes me puzzled.

   

  

  Aragorna

  Participant

  Topics: 36

  Replies: 130

  Been thanked: 12 times
  05/15/2023 at 7:08 PM #214764
  phoentzs

  @PeterSt

  I would hang in here for a moment, as this has been on my mind for a long time. Which do you think is better for SL/TP/Trail…% or pips? I’ve noticed the effect on my M1 systems that I have a higher profit on the same system with percentages than with pips. All settings the same, drawdown and number of trades also very similar. Whether pips or percent. I’m only 1% higher in total profit. Surely because the Nasdaq was 1500 points lower than today at the beginning of the year. But shouldn’t I also see an effect on the drawdown? Have you made similar observations? And what do you think is “better”… more robust?

  

  phoentzs

  Participant

  Topics: 86

  Replies: 960

  Been thanked: 198 times
  05/16/2023 at 7:00 AM #214774
  PeterSt

  I have done “everything” to cover for changes over time because this indeed  implies massive differences. But this is a very tough job and it requires many extra lines of code.

  but

  “Everything” is not all yet and this is because I am undecided on a couple of things. Reasoning them out is perhaps not useful. Give some considerations maybe is better ?

   

  I think in terms of money. This is already personal because for technicalities it may not be the best (scalping and then setting up a system in $ would not be much under control – and my focus is not on scalping). Thus, if I don’t see $ then I don’t know what I am doing. So I converted all to $. SL is in $. Trailing start is in $. Baling out (in loss) is in $. Buying position is in $. All is in $.
  But wait. Something like Range is in points/pips. Undoable for comparison with a base system. What to do with Nasdaq points vs Fx pips vs Russell2000 points. They don’t compare at all (especially dive into the Russell2000 and see what I mean). So I started to convert that to $ too. Quite crazy. But it tells you right away about those differences. What the actual forces *are* when Nasdaq is compared to Russell2000. So, all good ?

  No, not good, because indeed when the price of the Nasdaq is 20% lower, I should be able to earn 20% less. Or better put : I will earn 20% less.
  OK, too bad ?
  No, just bad.

  That I earn 20% less is accepted, but my system won’t work any more. “Start trailing at $100 profit”. That should be turned into $80 now. Don’t do that and all stops working (well).

  Obviously, working with % should solve that. Start trailing at … well …
  See ? I can’t even put it to words.

  Together with working with $, all my systems have a base investment in the code. This is 140K and origins from whatever time the EUR/USD was 1,40 and the EUR investment was 100K. Ah, so the USD investment was 140K. Yea, right. So with the portfolio in USD that would be correct. Even when obtaining the STOXX50.
  (hang on, because it starts to be undoable)

  So when I now do 2 pieces of Nasdaq Micro, all should be automatically converted from that base of 140K. Thus, I like to set the trailing start at 100. 100-what was that ? I don’t even know any more. Probably USD 100. Yea, because the investment-base was USD 140K. OK, so USD50K (2 Nasdaq micro) = 50/140 = 0,357. Wait, let me show you (attachment). Wow, I am right – See LotSizeFactor. But what I did was pulling a EUR/USD chart and feed it with 50K investment. Aha, so I can do that with every instrument. How nice.

  So I can talk to a same base system in terms of money and what I earn (in backtest) from it, always has references everywhere. And oh, if I change the 50K into 100K, everything must be 100% double everywhere. If not, the system is not linear and I can’t know what I am doing.
  And Yes, if the EUR/USD changes over time, I still work with the same $ and that is thus covered too when backtesting (you can imagine how much this influences). And No, it is of no use that I can see my earnings in USD if the EUR/USD was 1,60 back in 2008 while a few months back it was 0,97. Thus my reference is euros.

  ---

  By now you can envision why a bit of a system is 1000 lines of code in its base to cover for this. Crazy but very worth while.

  My next task – and when working on the Russel2000 you will know it is necessary – is to internally start working with percentages. It will be a huge endeavor because all must remain “readable” as well. In the 2nd attachment you may observe what this could all be about; you see money and you may also spot the rangeamount I was talking  about. Such things are required for checking out whether all works as intended. Percentage stuff needs to be added. I myself still want to say that trailing starts at $100, for that bas system of ~$140K. The system should make percentages of that. That looks to be 0,0714% to me at this moment. This counts for today with EUR/USD at 1,0872 and this also counts for when EUR/USD is 0,97. In other words, backtesting should be linear when today is compared with Dec 2022. The system’s strangled parameters should remain consistent. I could also say : the markets perhaps did not change at all when Dec 2022 is compared to May 2023, but my system(s) should cope with those technicalities. People may not realize how important this is.

  But you did, phoentzs.
  Maybe I should learn that 0,00714% is the same as $100. And that 0,143% is the same as $200. And that a SL of 1% is not too bad. And that 1% of 50K really is the same 1% at 500K.
  Ehh …

  1 user thanked author for this post.

  GraHal

  

  PeterSt

  Participant

  Topics: 73

  Replies: 2544

  Been thanked: 829 times
  05/29/2023 at 1:08 PM #215338
  coincatcha

  I’ve spent a lot of time learning and researching this topic.

  I agree with PeterSt. If you do your preliminary structural work on half available data or less (depending on time frame), then you save your precious data for the moments of truth! But be careful here, because if you stumble on something you can’t unsee it and bias will kick in and come through your optimisations. The fewer conditions, the less you can optimise anyway so perhaps a good place to start. Say you adjust ma’s or ema’s, well your just changing their speed of signal, if you only do this on a small sample then move to a larger sample you will see very quickly if the trigger/entry or other feature has been curve fitted to first data set. Mind you, this is all only relevant if you have statistical significance in your first data set or the whole set. If you use 1 min data and your bot trades 32 times in 5 months then you will never have a full picture (statistical significance) so your decisions must be made without this. This is a dangerous place to be. Doesn’t mean it can’t work, because it can but one needs to know you are playing without all the information and bots like this will need to monitored closely.

  For me, each piece of the bot has to produce some edge on its own. This way when I do inevitably optimise something thorough WF or otherwise I can easily see and understand the impact on the back test results.

  Recently, I’m running through some ideas from Kevin Davey’s book Building Winning Systems. This can answer your process questions in detail. Over time, it is no surprise that one finds the process truths through experience of trial and error.

  It was also no surprise then that the bots I have that survived their first year and grew my account actually pass all of his tests once broken down and tested by component.

  Whilst I’d love to be able to code every idea I ever have and all the bells and whistles to manage my portfolio I’m some years off that, it is amazing what can be achieved with 20 lines or less of code if you can already understand in depth price action because you’ll hone straight in on what you are trying to exploit.

   

  CC

   

   

   

   

  1 user thanked author for this post.

  GraHal

  

  coincatcha

  Participant

  Topics: 3

  Replies: 33

  Been thanked: 13 times
• Author
  Posts