Technique help to reduce over-fitting

Forums › ProRealTime English forum › ProOrder support › Technique help to reduce over-fitting

  • This topic has 63 replies, 12 voices, and was last updated 3 years ago by avatarPaul.

Tagged: , ,

Viewing 15 posts - 16 through 30 (of 64 total)
1 2 3 4 5
  • 08/27/2020 at 6:46 PM #142745
    Dow Jones
    April O’Neil wrote:

    like Dow Jones said, our tests must me run on a same number of trades

    I agree cannot be too low, but I kind of misleading you, I guess the usage depends on each people. I was accepting it when it produces 25% of trades from the total trades of the strategy (I forget to mention, AND also to have at least 100 trades), generally I set this rule for myself because I trade low time frame and high frequency (2-4 trades/days). But this tool will be hardly to have no. of trades too close to the total trades, because the mechanism it uses is restricting certain bar that is allowed to trade, so it provides opportunity to test those signals that were blocked due to on-going trade.

    I take it if strategy pass the VST with flying color, then it means the strategy can be robust to all kind of market structure, when market change, it can handle it well, so when it open trades, each trade is consistent in terms of average gain and win rate which can matter less even if the trades are little, that’s why the tool is not assessing the total gain. Of course if it doesn’t pass with flying color, it doesn’t mean it is a bad strategy, like you said, the strategy has different mean, like certain period not performing but overall still acceptable. So the VST tool maybe just not suitable or the Qty should be adjusted…or to rely on other robust test methodology.

    I hope it make sense.

    08/27/2020 at 7:01 PM #142746
    April O’Neil

    Thanks Dow Jones, i finaly understand ! All my bad

    Gonna work on what you sent and VST to try to get a protocol and ratios we can use with low time frame strategies.

    See you!

    08/27/2020 at 7:28 PM #142748
    Francesco

    Moreover, i would like to make an assumption: we are in a lucky historical period.
    We have the possibility to backtest a system that has gone through “standard” and “chaotic” phases such as covid in our previous 200k bars (> 5m tf).
    Being able to make robust a system that has successfully gone through these phases (and therefore a good variability of phases for what can be a general vision) can ensure us some good stability for the future?

    08/27/2020 at 7:36 PM #142749
    Vonasi
    Francesco wrote:

    can ensure us some good stability for the future?

    If only before the 2008 crash we had had a similar crash before to test on! Any back/forward /robustness test results are pretty much made pointless when something extreme that has never happened before happens. All we can do is see how our strategy would have coped and hope that the next extreme event isn’t really, really extreme. A big stop loss and quit is the only thing that can save us in extreme events.

    1 user thanked author for this post.
    avatar Francesco
    08/27/2020 at 7:44 PM #142756
    Paul
    Vonasi wrote:

    The point of my ‘Vonasi Robustness Tester’ and also my ‘Strategy Every Bar’ tests is to see trades that we might not see in a normal back test.

    There is also another way. Taking cumulative positions with defparam cumulateorders = true and make sure each position handles it’s own stoploss & profit target (as it was only 1 position on market).

    Partial closing works in backtesting and it could be usefull this way.

    It would be interesting to see the effects on optimising, which in part avoids curvefitting because it tests every signal.

    The problem is to to have a compact code to handle an unlimited cumulative positions, each position independently.

    And second, how to adapt a trailing stop to suit this test, besides sl&pt?

    After optimising parameters on every signal, then with a switch turn it off to see the results of a normal backtest which has less trades because it skips signals when in position.

    It may result into a more reliable strategy.

    08/27/2020 at 7:54 PM #142758
    Vonasi

    Good point regarding CUMULATEORDERS = TRUE Paul though as you point out it does not work for all strategy exit styles. Still it is another weapon in our armoury if the strategy suits.

    08/28/2020 at 12:03 AM #142772
    Paul
    Vonasi wrote:

    If you want to control different positions with different levels then you will need different stop loss and take profit variables for each position. I haven’t ever bothered to code something like that because we can’t use it in real life because IG consider two positions of 1 to be one position of 2 and won’t let us close half of it using pending orders

    you wrote this in another topic and this gave me hope it might be possible! Only for sl & pt with stop/limit orders or is that a stretch to far in the limits of prt because of the unknown number of cumulative positions?

    It would be a great base to test/optimise all signals with a simple r/r ratio, settle on some values and extend the code with ts etc further.

    08/28/2020 at 6:18 AM #142776
    Dow Jones
    Paul wrote:

    After optimising parameters on every signal, then with a switch turn it off to see the results of a normal backtest which has less trades because it skips signals when in position. It may result into a more reliable strategy

    Thanks, Paul. I had thought about that before, but too complicated for me to come up with any workable solution. So exactly why I started this article, maybe someone explore it and can share.

    Personally I have doubt whether it is possible because we don’t have command to know the tradeprice, set SL and set TP for individual trade, but just the average price (positionprice) and the SET STOP LOSS is for the last trade and close all trades when the SL is hit (same for TP), please correct me if I’m wrong. Today the most genius workaround is the VRT tool only.

    Like @vonasi said, easier if PRT introduce such mechanism in probacktest. I imagine with a checkbox to select possibility to run “multiple instances” of back test that will consolidate all trades, i.e. when signal trigger, open it as individual trade until it close, if another signal came up before first trade close, open it as another individual trade instance. At the end, consolidate all the trades.

    I think this alternative also make back test more reliable without have to rely on huge back test history.

    08/28/2020 at 1:25 PM #142824
    Paul
    Dow Jones wrote:

    to know the tradeprice,

    Another angle, what if you don’t need the tradeprice, instead remember the next barnumber and it’s openingprice after a signal? It doesn’t take into account spread for the entry/exit, but it could be added manually later.

    Are arrays needed for this & v11? Maybe it can be coded up to a fixed small number of cumulative positions, but then it will be much better if there’s no limit.

    08/28/2020 at 4:00 PM #142847
    Vonasi

    Whenever a SET STOP pPROFIT or SET TARGET pPROFIT instruction is encountered in a strategy code it adjusts the stop and target distance for all open positions. So positions will close out individually at different levels. The downside is that we cannot control each position individually – if you send a new stop or target distance then it is applied to all open positions – they cannot have different distances.

    09/02/2020 at 2:03 PM #143210
    zilliq

    Hi Guys,

    For me the best way to avoid overfitting (and so have the best correlation Backtest/OOS) is

    1/ Not more than 3-4 variables (I see on the forum some strategies with 10-12 variables…The more variables we have in the strategy, the more linear will be the equity curve, but the worst OOS we will have…Kind of polynomial regression)

    2/ Keep it simple ! The more simple is the strategy, the best OOS results we will have

    3/ A long long length of the backtest. The problem for me is that if we have 100 000 units it will take hours to have a backtest. I test on 1 million unit with 3 variable but I stop because of the time it takes. Ideally we need more than 100 trades in the backtest. The more you have, the best it is

    Exemple of results on OOS we should have if you have no (or minor) overfitting (Demo Strategy “1000 dollars / day” on forex EUR/USD since mid august after a 200 000 units backtest )

    Zilliq

    2020-09-02_14h58_36

     

     

    09/02/2020 at 4:20 PM #143219
    GraHal

    @zilliq how come some Long entries are a red down arrow and some Short entries are also a red down arrow??

    I guess you are teasing us and you are not going to share your excellent looking Strategy?? 🙂

    09/02/2020 at 5:41 PM #143224
    zilliq

    Hi @Grahal

    No it’s normal green arrow for long and red for short

    Simple strategy based on Bollinger Boll and ATR (Keep It Simple As Always).

    There are some months/Years I had codes with hundreds lines of code and I did not a lof of backtest IS/OOS. Now codes of my “incubator” have no more than 50 lines may be, but I do more and more backtests and robust tests

    Have a nice evening

    2020-09-02_18h35_15

     

     

    09/02/2020 at 6:08 PM #143226
    GraHal
    Z

    zilliq wrote:

    No it’s normal green arrow for long and red for short

    Am I losing it 🙂 … see attached ?

    09/02/2020 at 6:49 PM #143231
    zilliq

    I see @Grahal it’s a bug on graphs of PRT

    On small view there is a bug, but on a big view it’s correct

    2020-09-02_19h47_23

    2020-09-02_19h47_09

    1 user thanked author for this post.
    avatar GraHal
  • Author
    Posts
Viewing 15 posts - 16 through 30 (of 64 total)
1 2 3 4 5

Find exclusive trading pro-tools on ProRealCode Marketplace

Similar topics:
multi tf filters: how to reduce the time for each backtest

Create your free account now and post your request to benefit from the help of the community
Register or Login