Strategy Robustness Tester
Forums › ProRealTime English forum › ProOrder support › Strategy Robustness Tester
- This topic has 161 replies, 20 voices, and was last updated 3 years ago by Roger.
Tagged: casino, robustness, walk forward
-
-
09/12/2019 at 10:51 PM #10739509/13/2019 at 9:08 AM #107418
I found myself copying and pasting the VRT snippet from one System to another and it got on my nerves, so …
Here’s the VRT snippet as an .itf … if you make it available in your System backtesting list then it’s real easy to copy into the various strategy codes.
All thanks goes to Vonasi of course!
09/13/2019 at 8:09 PM #107505Vonasi (or anybody) the attached .itf will not randomise … might you be able to check it out PLEASE!
I’ve tried everything I can think of (I won’t bore with details, unless you want to save you time?) but I still get the same results for every combo.
It is not that Vectorials won’t work with the VRT as I have done a full analysis on another Vectorial.
I’ll be mad if it is something simple, but if I don’t find out I’ll never sleep tonight! 🙂
Many thanks in anticipation
09/13/2019 at 9:19 PM #10751209/13/2019 at 10:01 PM #107516Nice one!
Thank you, so only one entry for tradeon needed in line 214?
The other Vectorial I analysed worked fine and I made 2 entries directly before Buy & Sellshort. Maybe the preceding code was different than the .itf above, I’ll check it all out anyway.
The Randomiser is working fine now!
Thank You
09/13/2019 at 10:05 PM #107517Another good use of the Robustness Tester (perhaps I should start using GraHal’s name of VRT as it is less typing!) is to use it to check robustness either side of chosen variable values.
For example your strategy might only trade if the low is above the 200 period moving average and your strategy results look good but perhaps that value of 200 just means that you miss out on the one trade that opens just before the biggest all time historical drop and you curve fitted it out. So just carry out robustness tests using VRT with values either side of 200 and you will soon see how curve fitted your 200 period average is. Start with 10 period steps either side then drop down to 5 period and then 1 period and check every equity curve from every test and analyse the scores by plotting charts and you will soon see if you just got lucky with your 200 period average value.
From my testing so far it seems that getting a high % of winning trades score is not too difficult but getting an average gain per trade high score is more difficult. Perhaps we should put more weight in our analysis to this score?
09/13/2019 at 10:08 PM #107518Thank you, so only one entry for tradeon needed in line 214?
That condition in line 214 had to be true before any orders to buy or sellshort could be sent to market so one TRADEON condition in that line controlled all possible orders.
1 user thanked author for this post.
09/14/2019 at 9:45 PM #107553A very important thing when carrying out any robustness testing is to ensure that any money management or increasing and decreasing of position sizing based on profit or loss is turned off. If the position sizing is based on some sort of probability or based on the value returned by an indicator then it is OK but if it is totally an equity or strategy profit/loss based position sizing then it should be turned off and only a fixed value of position size used for the robustness testing.
Adding a line just before any BUY or SELLSHORT that fixes the size is the easiest way to do this.
123456789//positionsize calculationsif (lots of position sizing calculations based on equity or profit/loss) thenpositionsize = (whatever you calculated)endifif TRADEON and (long entry conditions) thenpositionsize = 1// just add this linebuy positionsize contracts at marketendif1 user thanked author for this post.
09/16/2019 at 12:25 AM #107642Another good way to use the robustness tester especially on fast time frames with limited data samples it to run it on our current data sample and record the results – then wait some months and run it on our new OOS data. If the robustness results are similar and especially if market structure has changed considerably (trending market to side ways market or to mean reversal market or any such change) then we can consider our robustness quite good.
1 user thanked author for this post.
09/23/2019 at 10:04 AM #108212Hello!
Thank you Vonasi for an excellent tester! I have tested this on some of my strategies and it is very interesting to see the results. I still have to find a way to know what will be my own limit of what is good and whats not. Sometimes its very clear and somtimes its a bit harder of course, when the results are not good but not so bad either 🙂
However I have a question, I believe there could be a problem with strategies that sometimes uses order triggered in the opposite direction as an alternative stoploss. Some strategies that have larger SL are sometimes stopped out by an order in the other direction. If the tester do not allow orders to be triggered during some periods, then the postion that is on market will continue in the wrong direction. That would actually not happened in real time. I hope you understand what I am trying to explain.
I tested with adding the trigger for Buy as an exit for shortpositions and vice versa, and that seems to solve the problem.
I have attached the different results adding the exit and not.
I believe this is not a very common problem, but I had two strategies where it made a qiute big difference. Do you think my assumption is correct, and if soo what do you think of the way of solving it?
09/23/2019 at 12:59 PM #108261Yes I am still working out my own personal acceptance levels of results. The win rate % seems easier to get good robustness results on than the average gain per trade and sometimes just looking at every equity curve produced can tell us an awful lot as well as looking for a high value score.
Yes adapting the robustness tester to some strategies is more complicated than others. My strategies tend to be very simple and usually trade only in one direction and so these are easier to add the tester to. If you reverse directions then you will really need to add extra conditions to allow the strategy to reverse if it is already on the market but not open any new positions if TRADEON is not positive. Strategies that are on the market 100% of the time and continually reversing are difficult to randomise entries on and really all we can do is randomise the first starting position which will tell us something about the robustness even if it is not a lot!
Something like the following should work for strategies that close a position by reversing trade direction (not tested).
1234567if ((not onmarket and tradeon) or (shortonmarket)) and (your long entry conditions) thenbuy 1 contract at marketendifif ((not onmarket and tradeon) or (longonmarket)) and (your short entry conditions) thensellshort 1 contract at marketendif2 users thanked author for this post.
09/28/2019 at 3:01 PM #108722I think it would be interesting to see what sort of robustness test results everyone is getting. Here is what I consider to be my most robust strategy test results so far.
The images attached are for three robustness tests on an SP500 weekly strategy with settings for QTY and RANDOM of 21/5 and 20/5 and 10/10.
21/5 test = 60.1 avg gain score and 86.9 %winners score = 73.5 average score
20/5 test = 49.2 avg gain score and 78.5 %winners score = 63.8 average score
10/10 test = 64.4 avg gain score and 87.4 %winners score = 75.9 average score
Average of all tests:
57.9 avg gain score average
84.3 % winners score average
71.1 overall average score
The final image is the strategy performance without the robustness testing.
I think that the above differing results confirm the importance of carrying out multiple robustness tests with different settings on every strategy.
09/28/2019 at 7:27 PM #108728with settings for QTY and RANDOM of 21/5 and 20/5 and 10/10.
I just want to be clear in my mind … do you mean using settings …
21/5
Qty 2 21 1
Random 1 5 120/5
Qty 2 20 1
Random 1 5 110/10
Qty 2 10 1
Random 1 10 109/28/2019 at 7:59 PM #10872909/28/2019 at 9:26 PM #108733Yes – exactly those settings.
That robustness test looks very good GraHal – lots of trades too. Some details of which instrument, time frame and trading style might be interesting to others along with perhaps some other robustness tests using different settings for comparison. The more random trades we can test the more we can persuade ourselves that what we have created is robust.
-
AuthorPosts
Find exclusive trading pro-tools on