Strategy Robustness Tester

Vonasi · Sun, 01 Sep 2019 12:34:14 +0000

Following on from a discussion elsewhere regarding walk forward testing I had the idea to create a bit of code that you can add to any strategy and see how it would have performed if you had traded on only odd days or only on even days, or odd or even months or odd and even years or any combination of the three. By comparing the output of all 23 tests we can get an idea of how consistently a strategy makes profit. The gain figure in the optimisation results should be ignored as the number of trades for each test will be very different. The average gain per trade, win rate and equity curve for each test can tell us a lot about the strategies robustness though. You just add the code to the start of your strategy code and then insert the extra condition ‘and tradeon’ in the conditions whenever you send buy or sellshort orders to the market. Then just optimise with a variable ‘choice’ with values 0 to 22 step 1. Then just click on each result in the optimisation report to see the equity curve for each test. Here is the code to add to the start of your strategy in case it is of use to anyone: //Day Month and Year Robustness Tester //By Vonasi //20190901 //0 = all days //1 = odd days only //2 = even days only //3 = odd months only //4 = even months only //5 = even years only //6 = odd years only //7 = odd days in odd months only //8 = even days in even months only //9 = even days in odd months only //10 = odd days in even months only //11 = even months in even years only //12 = odd months in odd years only //13 = even months in odd years only //14 = odd months in even years only //15 = odd days in odd months in odd years only //16 = even days in even months in even years only //17 = even days in odd months in odd years only //18 = odd days in even months in even years only //19 = odd days in even months in odd years only //20 = even days in odd months in even years only //21 = even days in even months in odd years only //22 = odd days in odd months in even years only oddday = opendayofweek mod 2 = 1 evenday = opendayofweek mod 2 = 0 oddmonth = openmonth mod 2 = 1 evenmonth = openmonth mod 2 = 0 oddyear = openyear mod 2 = 1 evenyear = openyear mod 2 = 0 tradeon = 0 if choice = 0 then tradeon = 1 endif //odd days only if choice = 1 and oddday then tradeon = 1 endif //even days only if choice = 2 and evenday then tradeon = 1 endif //odd months only if choice = 3 and oddmonth then tradeon = 1 endif //even months only if choice = 4 and evenmonth then tradeon = 1 endif //even years only if choice = 5 and evenyear then tradeon = 1 endif //odd years only if choice = 6 and oddyear then tradeon = 1 endif //odd days in odd months only if choice = 7 and oddday and oddmonth then tradeon = 1 endif //even days in even months only if choice = 8 and evenday and evenmonth then tradeon = 1 endif //even days in odd months only if choice = 9 and evenday and oddmonth then tradeon = 1 endif //odd days in even months only if choice = 10 and oddday and evenmonth then tradeon = 1 endif //even months in even years only if choice = 11 and evenmonth and evenyear then tradeon = 1 endif //odd months in odd years only if choice = 12 and oddmonth and oddyear then tradeon = 1 endif //even months in odd years only if choice = 13 and evenmonth and oddyear then tradeon = 1 endif //odd months in even years only if choice = 14 and oddmonth and evenyear then tradeon = 1 endif //odd days in odd months in odd years only if choice = 15 and oddday and oddmonth and oddyear then tradeon = 1 endif //even days in even months in even years only if choice = 16 and evenday and evenmonth and evenyear then tradeon = 1 endif //even days in odd months in odd years only if choice = 17 and evenday and oddmonth and oddyear then tradeon = 1 endif //odd days in even months in even years only if choice = 18 and oddday and evenmonth and evenyear then tradeon = 1 endif //odd days in even months in odd years only if choice = 19 and oddday and evenmonth and oddyear then tradeon = 1 endif //even days in odd months in even years only if choice = 20 and evenday and oddmonth and evenyear then tradeon = 1 endif //even days in even months in odd years only if choice = 21 and evenday and evenmonth and oddyear then tradeon = 1 endif //odd days in odd months in even years only if choice = 22 and oddday and oddmonth and evenyear then tradeon = 1 endif Then alter your strategy code to use the TRADEON variable so that trades are only opened if the day, month and year meet the test criteria. if (your long entry conditions) and tradeon then buy 1 contract at market endif if (your short entry conditions) and tradeon then sellshort 1 contract at market endif

This topic has 161 replies, 20 voices, and was last updated 3 years ago by Roger.

Tagged: casino, robustness, walk forward

Viewing 15 posts - 46 through 60 (of 162 total)

← 1 2 3 4 5 6 7 8 9 10 11 →

09/12/2019 at 10:51 PM #107395

Vonasi

I can access it now – great job – thank you!

Vonasi

Moderator

Topics: 192

Replies: 5088

Been thanked: 1811 times

09/13/2019 at 9:08 AM #107418

GraHal

I found myself copying and pasting the VRT snippet from one System to another and it got on my nerves, so …

Here’s the VRT snippet as an .itf … if you make it available in your System backtesting list then it’s real easy to copy into the various strategy codes.

All thanks goes to Vonasi of course!

GraHal

Participant

Topics: 212

Replies: 8785

Been thanked: 2185 times

09/13/2019 at 8:09 PM #107505

GraHal

Vonasi (or anybody) the attached .itf will not randomise … might you be able to check it out PLEASE!

I’ve tried everything I can think of (I won’t bore with details, unless you want to save you time?) but I still get the same results for every combo.

It is not that Vectorials won’t work with the VRT as I have done a full analysis on another Vectorial.

I’ll be mad if it is something simple, but if I don’t find out I’ll never sleep tonight! 🙂

Many thanks in anticipation

GraHal

Participant

Topics: 212

Replies: 8785

Been thanked: 2185 times

09/13/2019 at 9:19 PM #107512

Vonasi

Change line 214 to:

if tradetime and tradeon then

1	if tradetime and tradeon then

Vonasi

Moderator

Topics: 192

Replies: 5088

Been thanked: 1811 times

09/13/2019 at 10:01 PM #107516

GraHal

Nice one!

Thank you, so only one entry for tradeon needed in line 214?

The other Vectorial I analysed worked fine and I made 2 entries directly before Buy & Sellshort. Maybe the preceding code was different than the .itf above, I’ll check it all out anyway.

The Randomiser is working fine now!

Thank You

GraHal

Participant

Topics: 212

Replies: 8785

Been thanked: 2185 times

09/13/2019 at 10:05 PM #107517

Vonasi

Another good use of the Robustness Tester (perhaps I should start using GraHal’s name of VRT as it is less typing!) is to use it to check robustness either side of chosen variable values.

For example your strategy might only trade if the low is above the 200 period moving average and your strategy results look good but perhaps that value of 200 just means that you miss out on the one trade that opens just before the biggest all time historical drop and you curve fitted it out. So just carry out robustness tests using VRT with values either side of 200 and you will soon see how curve fitted your 200 period average is. Start with 10 period steps either side then drop down to 5 period and then 1 period and check every equity curve from every test and analyse the scores by plotting charts and you will soon see if you just got lucky with your 200 period average value.

From my testing so far it seems that getting a high % of winning trades score is not too difficult but getting an average gain per trade high score is more difficult. Perhaps we should put more weight in our analysis to this score?

Vonasi

Moderator

Topics: 192

Replies: 5088

Been thanked: 1811 times

09/13/2019 at 10:08 PM #107518

Vonasi

GraHal wrote:

Thank you, so only one entry for tradeon needed in line 214?

That condition in line 214 had to be true before any orders to buy or sellshort could be sent to market so one TRADEON condition in that line controlled all possible orders.

1 user thanked author for this post.

GraHal

Vonasi

Moderator

Topics: 192

Replies: 5088

Been thanked: 1811 times

09/14/2019 at 9:45 PM #107553

Vonasi

A very important thing when carrying out any robustness testing is to ensure that any money management or increasing and decreasing of position sizing based on profit or loss is turned off. If the position sizing is based on some sort of probability or based on the value returned by an indicator then it is OK but if it is totally an equity or strategy profit/loss based position sizing then it should be turned off and only a fixed value of position size used for the robustness testing.

Adding a line just before any BUY or SELLSHORT that fixes the size is the easiest way to do this.

//positionsize calculations 
if (lots of position sizing calculations based on equity or profit/loss) then
positionsize = (whatever you calculated)
endif 

if TRADEON and (long entry conditions) then
positionsize = 1// just add this line
buy positionsize contracts at market
endif

//positionsize calculations

if (lots of position sizing calculations based on equity or profit/loss) then

positionsize = (whatever you calculated)

endif

if TRADEON and (long entry conditions) then

positionsize = 1// just add this line

buy positionsize contracts at market

endif

1 user thanked author for this post.

GraHal

Vonasi

Moderator

Topics: 192

Replies: 5088

Been thanked: 1811 times

09/16/2019 at 12:25 AM #107642

Vonasi

Another good way to use the robustness tester especially on fast time frames with limited data samples it to run it on our current data sample and record the results – then wait some months and run it on our new OOS data. If the robustness results are similar and especially if market structure has changed considerably (trending market to side ways market or to mean reversal market or any such change) then we can consider our robustness quite good.

1 user thanked author for this post.

GraHal

Vonasi

Moderator

Topics: 192

Replies: 5088

Been thanked: 1811 times

09/23/2019 at 10:04 AM #108212

PastaPesto

Hello!

Thank you Vonasi for an excellent tester! I have tested this on some of my strategies and it is very interesting to see the results. I still have to find a way to know what will be my own limit of what is good and whats not. Sometimes its very clear and somtimes its a bit harder of course, when the results are not good but not so bad either 🙂

However I have a question, I believe there could be a problem with strategies that sometimes uses order triggered in the opposite direction as an alternative stoploss. Some strategies that have larger SL are sometimes stopped out by an order in the other direction. If the tester do not allow orders to be triggered during some periods, then the postion that is on market will continue in the wrong direction. That would actually not happened in real time. I hope you understand what I am trying to explain.

I tested with adding the trigger for Buy as an exit for shortpositions and vice versa, and that seems to solve the problem.

I have attached the different results adding the exit and not.

I believe this is not a very common problem, but I had two strategies where it made a qiute big difference. Do you think my assumption is correct, and if soo what do you think of the way of solving it?

PastaPesto

Participant

Topics: 13

Replies: 29

Been thanked: 6 times

09/23/2019 at 12:59 PM #108261

Vonasi

Yes I am still working out my own personal acceptance levels of results. The win rate % seems easier to get good robustness results on than the average gain per trade and sometimes just looking at every equity curve produced can tell us an awful lot as well as looking for a high value score.

Yes adapting the robustness tester to some strategies is more complicated than others. My strategies tend to be very simple and usually trade only in one direction and so these are easier to add the tester to. If you reverse directions then you will really need to add extra conditions to allow the strategy to reverse if it is already on the market but not open any new positions if TRADEON is not positive. Strategies that are on the market 100% of the time and continually reversing are difficult to randomise entries on and really all we can do is randomise the first starting position which will tell us something about the robustness even if it is not a lot!

Something like the following should work for strategies that close a position by reversing trade direction (not tested).

if ((not onmarket and tradeon) or (shortonmarket)) and (your long entry conditions) then
buy 1 contract at market
endif

if ((not onmarket and tradeon) or (longonmarket)) and (your short entry conditions) then
sellshort 1 contract at market
endif

if ((not onmarket and tradeon) or (shortonmarket)) and (your long entry conditions) then

buy 1 contract at market

endif

if ((not onmarket and tradeon) or (longonmarket)) and (your short entry conditions) then

sellshort 1 contract at market

endif

2 users thanked author for this post.

PastaPesto,

GraHal

Vonasi

Moderator

Topics: 192

Replies: 5088

Been thanked: 1811 times

09/28/2019 at 3:01 PM #108722

Vonasi

I think it would be interesting to see what sort of robustness test results everyone is getting. Here is what I consider to be my most robust strategy test results so far.

The images attached are for three robustness tests on an SP500 weekly strategy with settings for QTY and RANDOM of 21/5 and 20/5 and 10/10.

21/5 test = 60.1 avg gain score and 86.9 %winners score = 73.5 average score

20/5 test = 49.2 avg gain score and 78.5 %winners score = 63.8 average score

10/10 test = 64.4 avg gain score and 87.4 %winners score = 75.9 average score

Average of all tests:

57.9 avg gain score average

84.3 % winners score average

71.1 overall average score

The final image is the strategy performance without the robustness testing.

I think that the above differing results confirm the importance of carrying out multiple robustness tests with different settings on every strategy.

Vonasi

Moderator

Topics: 192

Replies: 5088

Been thanked: 1811 times

09/28/2019 at 7:27 PM #108728

GraHal

Vonasi wrote:

with settings for QTY and RANDOM of 21/5 and 20/5 and 10/10.

I just want to be clear in my mind … do you mean using settings …

21/5

Qty 2 21 1
Random 1 5 1

20/5

Qty 2 20 1
Random 1 5 1

10/10

Qty 2 10 1
Random 1 10 1

GraHal

Participant

Topics: 212

Replies: 8785

Been thanked: 2185 times

09/28/2019 at 7:59 PM #108729

GraHal

Attached is probably my best / most robust System … Av Score – 81.9 (on settings of 10/5)

GraHal

Participant

Topics: 212

Replies: 8785

Been thanked: 2185 times

09/28/2019 at 9:26 PM #108733

Vonasi

Yes – exactly those settings.

That robustness test looks very good GraHal – lots of trades too. Some details of which instrument, time frame and trading style might be interesting to others along with perhaps some other robustness tests using different settings for comparison. The more random trades we can test the more we can persuade ourselves that what we have created is robust.

Vonasi

Moderator

Topics: 192

Replies: 5088

Been thanked: 1811 times

Author

Posts

Viewing 15 posts - 46 through 60 (of 162 total)

← 1 2 3 4 5 6 7 8 9 10 11 →

Find exclusive trading pro-tools on