Strategy Robustness Tester

Forums ProRealTime English forum ProOrder support Strategy Robustness Tester

  • This topic has 161 replies, 20 voices, and was last updated 3 years ago by avatarRoger.
Viewing 15 posts - 46 through 60 (of 162 total)
  • #107395

    I can access it now – great job – thank you!

    #107418

    I found myself copying and pasting the VRT snippet from one System to another and it got on my nerves, so …

    Here’s the VRT snippet as an .itf … if you make it available in your System backtesting list then it’s real easy to copy into the various strategy codes.

    All thanks goes to Vonasi of course!

     

    #107505

    Vonasi (or anybody) the attached .itf will not randomise … might you be able to check it out PLEASE!

    I’ve tried everything I can think of (I won’t bore with details, unless you want to save you time?) but I still get the same results for every combo.

    It is not that Vectorials won’t work with the VRT as I have done a full analysis on another Vectorial.

    I’ll be mad if it is something simple, but if I don’t find out I’ll never sleep tonight! 🙂

    Many thanks in anticipation

     

     

     

    #107512

    Change line 214 to:

     

    #107516

    Nice one!

    Thank you, so only one entry for tradeon needed in line 214?

    The other Vectorial I analysed worked fine and I made 2 entries directly before Buy & Sellshort. Maybe the preceding code was different than the .itf above, I’ll check it all out anyway.

    The Randomiser is working fine now!

    Thank You

    #107517

    Another good use of the Robustness Tester (perhaps I should start using GraHal’s name of VRT as it is less typing!) is to use it to check robustness either side of chosen variable values.

    For example your strategy might only trade if the low is above the 200 period moving average and your strategy results look good but perhaps that value of 200 just means that you miss out on the  one trade that opens just before the biggest all time historical drop and you curve fitted it out. So just carry out robustness tests using VRT with values either side of 200 and you will soon see how curve fitted your 200 period average is. Start with 10 period steps either side then drop down to 5 period and then 1 period and check every equity curve from every test and analyse the scores by plotting charts and you will soon see if you just got lucky with your 200 period average value.

    From my testing so far it seems that getting a high % of winning trades score is not too difficult but getting an average gain per trade high score is more difficult. Perhaps we should put more weight in our analysis to this score?

    #107518

    Thank you, so only one entry for tradeon needed in line 214?

    That condition in line 214 had to be true before any orders to buy or sellshort could be sent to market so one TRADEON condition in that line controlled all possible orders.

    1 user thanked author for this post.
    #107553

    A very important thing when carrying out any robustness testing is to ensure that any money management or increasing and decreasing of position sizing based on profit or loss is turned off. If the position sizing is based on some sort of probability or based on the value returned by an indicator then it is OK but if it is totally an equity or strategy profit/loss based position sizing then it should be turned off and only a fixed value of position size used for the robustness testing.

    Adding a line just before any BUY or SELLSHORT that fixes the size is the easiest way to do this.

     

    1 user thanked author for this post.
    #107642

    Another good way to use the robustness tester especially on fast time frames with limited data samples it to run it on our current data sample and record the results – then wait some months and run it on our new OOS data. If the robustness results are similar and especially if market structure has changed considerably (trending market to side ways market or to mean reversal market or any such change) then we can consider our robustness quite good.

    1 user thanked author for this post.
    #108212

    Hello!

    Thank you Vonasi for an excellent tester! I have tested this on some of my strategies and it is very interesting to see the results. I still have to find a way to know what will be my own limit of what is good and whats not. Sometimes its very clear and somtimes its a bit harder of course, when the results are not good but not so bad either 🙂

    However I have a question, I believe there could be a problem with strategies that sometimes uses order triggered in the opposite direction as an alternative stoploss. Some strategies that have larger SL are sometimes stopped out by an order in the other direction. If the tester do not allow orders to be triggered during some periods, then the postion that is on market will continue in the wrong direction. That would actually not happened in real time. I hope you understand what I am trying to explain.

    I tested with adding the trigger for Buy as an exit for shortpositions and vice versa, and that seems to solve the problem.

    I have attached the different results adding the exit and not.

    I believe this is not a very common problem, but I had two strategies where it made a qiute big difference.  Do you think my assumption is correct, and if soo what do you think of the way of solving it?

     

    #108261

    Yes I am still working out my own personal acceptance levels of results. The win rate % seems easier to get good robustness results on than the average gain per trade and sometimes just looking at every equity curve produced can tell us an awful lot as well as looking for a high value score.

    Yes adapting the robustness tester to some strategies is more complicated than others. My strategies tend to be very simple and usually trade only in one direction and so these are easier to add the tester to. If you reverse directions then you will really need to add extra conditions to allow the strategy to reverse if it is already on the market but not open any new positions if TRADEON is not positive. Strategies that are on the market 100% of the time and continually reversing are difficult to randomise entries on and really all we can do is randomise the first starting position which will tell us something about the robustness even if it is not a lot!

    Something like the following should work for strategies that close a position by reversing trade direction (not tested).

     

    2 users thanked author for this post.
    #108722

    I think it would be interesting to see what sort of robustness test results everyone is getting. Here is what I consider to be my most robust strategy test results so far.

     

    The images attached are for three robustness tests on an SP500 weekly strategy with settings for QTY and RANDOM of 21/5 and 20/5 and 10/10.

    21/5  test = 60.1  avg gain score and  86.9 %winners score = 73.5 average score

    20/5  test = 49.2 avg gain score and  78.5 %winners score = 63.8 average score

    10/10 test = 64.4 avg gain score and 87.4 %winners score = 75.9 average score

     

    Average of all tests:

    57.9 avg gain score average

    84.3 % winners score average

    71.1 overall average score

     

    The final image is the strategy performance without the robustness testing.

    I think that the above differing results confirm the importance of carrying out multiple robustness tests with different settings on every strategy.

    #108728

    with settings for QTY and RANDOM of 21/5 and 20/5 and 10/10.

    I just want to be clear in my mind … do you mean using settings  …

    21/5

    Qty                 2    21 1
    Random         1   5   1

    20/5

    Qty                 2  20 1
    Random        1  5  1

    10/10

    Qty                 2  10  1
    Random        1  10  1

     

     

     

    #108729

    Attached is probably my best / most robust System … Av Score – 81.9 (on settings of 10/5)

    #108733

    Yes – exactly those settings.

    That robustness test looks very good GraHal – lots of trades too. Some details of which instrument, time frame and trading style might be interesting to others along with perhaps some other robustness tests using different settings for comparison. The more random trades we can test the more we can persuade ourselves that what we have created is robust.

Viewing 15 posts - 46 through 60 (of 162 total)

Create your free account now and post your request to benefit from the help of the community
Register or Login