Help with WF

• 09/01/2019 at 10:47 AM #106060
  GraHal

  Vonasi wrote:

  a WF efficiency of 150% indicates that your strategy performed a lot worse in the 70% IS periods than it did in the 30% OOS periods which might be of concern.

  The higher WFE % in OS (above 100%) could be due to entry triggers occurring more often and / or exit triggers occurring less often … during OS than during the optimised IS period.

  Isn’t above what we would be ecstatic about during Live trading!? 🙂

  As Vonasi says, it might be of concern if due to one / a few exceptional spikes … so the equity curve and associated trades during OS needs to be analysed.

   

  So can anybody think of other reasons why WFE % would be higher during OS than IS?

  

  GraHal

  Participant

  Topics: 212

  Replies: 8785

  Been thanked: 2185 times
  09/01/2019 at 10:55 AM #106063
  GraHal

  Stefanb wrote:

  First, I optimize my algo in a regular back test 70% of the time.

  Surely we would only need to optimise over 70% IS of the time if we were then going to use WF on 1 repetition so we need the other 30% as OS.

  I optimise over 100% of the time and then do a WF over 5 repetitions to see the variations of variable values over the 5 x OS periods and also how these compare with the variable value I chose over the 100% of time.

  If variable values over WF 5 x OS and 100% non WF are similar / within a narrow range then I get excited!!! 🙂

  

  GraHal

  Participant

  Topics: 212

  Replies: 8785

  Been thanked: 2185 times
  09/01/2019 at 11:14 AM #106065
  Vonasi

  What you mention in your first reply GraHal is one of the issues with WF testing. I’ve mentioned it somewhere before I believe. If you have a strategy that uses a filter that keeps it out of the market during downturns then it might be that one of your IS or OOS tests has absolutely no trades in it at all which messes up your WF results. Also if you use something like average range in your strategy to decide on TP and SL levels then on something like the DJI we can see that average range has steadily increased as the years go by and so you will see higher returns in later OOS periods compared to their earlier IS periods and also an increasing in returns as each repetition is tested which will look like the WF is saying that your strategy is getting better and better!

  WF is not the perfect robustness test for all strategies. In my mind it is great if you have a strategy that trades very regularly and has consistent TP and SL levels but then in my mind this is the hardest strategy to create to work on all of history due to changing market characteristics and varying average range!

  I’m considering coding something to add to strategies that allows us to run an optimisation test but only allowing the strategy to open trades at certain times such as only on odd days or even days of the week or odd numbered or even numbered days of the year. Maybe even just odd days in odd months, even days in even months, even days in odd months and odd days in even months! This way we could see a whole bunch of almost random OOS testing results compared side by side.

  1 user thanked author for this post.

  GraHal

  

  Vonasi

  Moderator

  Topics: 192

  Replies: 5088

  Been thanked: 1811 times
  09/01/2019 at 11:24 AM #106066
  GraHal

  GraHal wrote:

  Surely we would only need to optimise over 70% IS of the time if we were then going to use WF on 1 repetition so we need the other 30% as OS.

  And only then if we are using the WF 1 x Dummy variable (value = 1) procedure?

  When we WF and use the same range and step for variables during WF as we did during non-WF optimisation then there is no need to have limited non-WF optimisation to less than 100%.

  Its not as of the variables are saying … aha we know you have optimised us already over 100% of this period so we are going to screw you during a WF Optimisation! :)

  No … the WF Optimisation will find an optimum values for an IS period and then see what results are like during the following OS period.

  

  GraHal

  Participant

  Topics: 212

  Replies: 8785

  Been thanked: 2185 times
  09/01/2019 at 4:32 PM #106087
  Stefanb

  Tested to optimize an algo in the usual “backtest” of 100% data.

  Then I did a WF test with fixed values ​​of 5 periods 70% in sample.

  Got the following results.

  What do you think?

  Is it a worthless algo?

  

  Stefanb

  Participant

  Topics: 15

  Replies: 74

  Been thanked: 22 times
  09/01/2019 at 7:26 PM #106126
  GraHal

  NO not at all worthless, but your wallet may suffer if there was an extended downturn  downtrend?

  The equity curve during donwtrends does not show as good as during uptrends.

  Maybe some work on the SellShort side might benefit?

  Maybe a filter so that Longs are not taken if Close < Average[xxx] (Close) ??

   

  

  GraHal

  Participant

  Topics: 212

  Replies: 8785

  Been thanked: 2185 times
  09/02/2019 at 3:11 PM #106236
  Nicolas

  WF testing is everything but not a “click and forget” stage for an algo development. In fact, there is no easy way to decide if a strategy is robust or not. What we can conclude from your analysis is that the algo was performing ‘correctly’ (I did not analyze all the numbers, just looked at the equity curve) by being re-optimized each time it has ran 30% of the time. Since it has been correct in the past, 5 times, it should be considered as “robust”.

  Don’t forget about fees, and that the optimisation backtest are not made with tick-by-tick mode which can lead to different results when checked in a normal backtest (without optimiser).

  

  Nicolas

  Keymaster

  Topics: 81

  Replies: 20863

  Been thanked: 4773 times
  09/02/2019 at 6:19 PM #106260
  Stefanb

  Thanks for your input Nicolas.

  There is a lot to think about clearly.

  You have to do as much as you can to make sure that the algo is not over-optimized.

  If you optimize a code and then do a back test 70% of the time and then WF on 5 periods with 70% IS then you have done what you can do.

  Tick ​​by tick must not be forgotten either!

  After that, you can let it go on demo for a while for extra security.

  It sounds like a good plan or……

  

  Stefanb

  Participant

  Topics: 15

  Replies: 74

  Been thanked: 22 times
  09/03/2019 at 12:23 PM #106365
  Nicolas

  5 periods or whatever is your preferred number (joking).

  

  Nicolas

  Keymaster

  Topics: 81

  Replies: 20863

  Been thanked: 4773 times
• Author
  Posts