Why is backtesting so unreliable?

• 10/22/2019 at 2:57 PM #110841
  nonetheless

  I’ve developed a few codes that appear to be successful in backtesting, but whenever I run them in demo nothing comes close to that kind of profit.

  I adapted the following from the Pure Renko Strategy posted by Doctrading.

  This version is designed for the DJI 5min chart on UK time and shows incredible results – 75% wins over 17 months with only 3 losing days. Well, you know what they say about things that sound too good to be true. Run it live and it loses money every day. Same story with other seemingly successful codes I’ve developed. I’d be very curious to know why there should be such a huge discrepancy.

  Thanks in advance for any enlightenment.

  // Definition of code parameters DEFPARAM CumulateOrders = False // Cumulating positions deactivated // The system will cancel all pending orders and close all positions at 0:00. No new ones will be allowed until after the "FLATBEFORE" time. DEFPARAM FLATBEFORE = 143000 // Cancel all pending orders and close all positions at the "FLATAFTER" time DEFPARAM FLATAFTER = 210000 boxSize = 13 once renkoMax = ROUND(close / boxSize) * boxSize once renkoMin = renkoMax - boxSize IF high > renkoMax + boxSize THEN WHILE high > renkoMax + boxSize renkoMax = renkoMax + boxSize renkoMin = renkoMin + boxSize WEND ELSIF low < renkoMin - boxSize THEN WHILE low < renkoMin - boxSize renkoMax = renkoMax - boxSize renkoMin = renkoMin - boxSize WEND ENDIF buy 1 CONTRACT at renkoMax + boxSize stop sellshort 1 CONTRACT at renkoMin - boxSize stop // Stops and targets SET STOP $TRAILING 12

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

  // Definition of code parameters DEFPARAM CumulateOrders = False // Cumulating positions deactivated // The system will cancel all pending orders and close all positions at 0:00. No new ones will be allowed until after the "FLATBEFORE" time. DEFPARAM FLATBEFORE = 143000 // Cancel all pending orders and close all positions at the "FLATAFTER" time DEFPARAM FLATAFTER = 210000 boxSize = 13 once renkoMax = ROUND(close / boxSize) * boxSize once renkoMin = renkoMax - boxSize IF high > renkoMax + boxSize THEN WHILE high > renkoMax + boxSize renkoMax = renkoMax + boxSize renkoMin = renkoMin + boxSize WEND ELSIF low < renkoMin - boxSize THEN WHILE low < renkoMin - boxSize renkoMax = renkoMax - boxSize renkoMin = renkoMin - boxSize WEND ENDIF buy 1 CONTRACT at renkoMax + boxSize stop sellshort 1 CONTRACT at renkoMin - boxSize stop // Stops and targets SET STOP $TRAILING 12

   

  

  nonetheless

  Participant

  Topics: 83

  Replies: 1237

  Been thanked: 978 times
  10/22/2019 at 3:20 PM #110847
  Vonasi

  nonetheless – Welcome to the forums. When posting code you are expected to use the ‘Insert PRT Code’ button to make the code more readable to others. I have tidied up your post. Please use the button in future posts. 🙂

  1 user thanked author for this post.

  nonetheless

  

  Vonasi

  Moderator

  Topics: 192

  Replies: 5088

  Been thanked: 1811 times
  10/22/2019 at 4:09 PM #110863
  Nicolas

  Same story over and over again 🙂

  These are the things you should be aware of:

  1. try to not over optimize your settings, it will fails if you don’t validate an OOS period (overfitting)
  2. spread in backtests, did you set it?
  3. broker slippage: no real way to simulate that in backtests
  4. renko bricks construction: only dependent to price moves, so results will be completely different if you start a backtest with a different time period, because price will start moving at first bar of history
  5. set stop trailing instruction: doesn’t respect the IG trailing stop step, don’t use it
  6. always make backtests with tick by tick box checked

   

  1 user thanked author for this post.

  nonetheless

  

  Nicolas

  Keymaster

  Topics: 81

  Replies: 20863

  Been thanked: 4774 times
  10/22/2019 at 4:59 PM #110876
  Mauro T. “Algorithm System”

  You should never do tests with 5 minute time frames, 17 months are very few.
  Normally you only have profits on paper but then you lose money.

  

  Mauro T. “Algorithm System”

  Participant

  Topics: 18

  Replies: 313

  Been thanked: 76 times
  10/22/2019 at 5:18 PM #110883
  nonetheless

  Thanks for the advice – I think I’ll stick to manual trading, at least I know how that works.

  Best of luck to all!

  

  nonetheless

  Participant

  Topics: 83

  Replies: 1237

  Been thanked: 978 times
  10/22/2019 at 6:04 PM #110887
  GraHal

  Just confirm that you had the tick by tick Mode  box selected in your backtest @nonetheless  … as without using the tick by tick mode then bt results look like  the holy grail!!

  Just tying to save you getting too disillusioned with Auto-Trading 🙂

   

  

  GraHal

  Participant

  Topics: 214

  Replies: 8830

  Been thanked: 2201 times
  10/22/2019 at 6:16 PM #110889
  GraHal

  Is attached something similar to what you got @nonetheless … because attached is without tick by tick as that is all I could get due to too many instances of … well, there’s a lot to it and you be better to read up elsewhere on this website.

  Suffice it to say that Real Live  is guaranteed to be nothing like attached due to TP and SL being met in the same bar  … in short your Trailing Stop is way to low / close to Trade Price.

   

  

  GraHal

  Participant

  Topics: 214

  Replies: 8830

  Been thanked: 2201 times
  10/22/2019 at 7:20 PM #110893
  Mauro T. “Algorithm System”

  The problem is not the tick by tick, but the reduced time of the test ….. And even as it is done, with the computer it is easy to get back exceptional results.

  1 user thanked author for this post.

  nonetheless

  

  Mauro T. “Algorithm System”

  Participant

  Topics: 18

  Replies: 313

  Been thanked: 76 times
  10/22/2019 at 7:37 PM #110897
  GraHal

  Below looks like it might be more realistic … see how it is Short during a down period and vice versa?

  I’m putting below on Forward Test on Demo for a while … never had a Renko System before! 🙂

  Nonetheless Thank You! 🙂 🙂

  EDIT / PS

  I’m sure below could be improved more, e.g. by adding one of the many coded Trailing Stops available on here.

   

  //https://www.prorealcode.com/topic/why-is-backtesting-so-unreliable/#post-110889 // Definition of code parameters DEFPARAM CumulateOrders = False // Cumulating positions deactivated // The system will cancel all pending orders and close all positions at 0:00. No new ones will be allowed until after the "FLATBEFORE" time. //DEFPARAM FLATBEFORE = 143000 // Cancel all pending orders and close all positions at the "FLATAFTER" time //DEFPARAM FLATAFTER = 210000 boxSize = 110 //50 once renkoMax = ROUND(close / boxSize) * boxSize once renkoMin = renkoMax - boxSize IF high > renkoMax + boxSize THEN WHILE high > renkoMax + boxSize renkoMax = renkoMax + boxSize renkoMin = renkoMin + boxSize WEND ELSIF low < renkoMin - boxSize THEN WHILE low < renkoMin - boxSize renkoMax = renkoMax - boxSize renkoMin = renkoMin - boxSize WEND ENDIF buy 1 CONTRACT at renkoMax + boxSize stop sellshort 1 CONTRACT at renkoMin - boxSize stop // Stops and targets SET STOP PLOSS 75 SET TARGET PPROFIT 150

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

  //https://www.prorealcode.com/topic/why-is-backtesting-so-unreliable/#post-110889 // Definition of code parameters DEFPARAM CumulateOrders = False // Cumulating positions deactivated // The system will cancel all pending orders and close all positions at 0:00. No new ones will be allowed until after the "FLATBEFORE" time. //DEFPARAM FLATBEFORE = 143000 // Cancel all pending orders and close all positions at the "FLATAFTER" time //DEFPARAM FLATAFTER = 210000 boxSize = 110 //50 once renkoMax = ROUND(close / boxSize) * boxSize once renkoMin = renkoMax - boxSize IF high > renkoMax + boxSize THEN WHILE high > renkoMax + boxSize renkoMax = renkoMax + boxSize renkoMin = renkoMin + boxSize WEND ELSIF low < renkoMin - boxSize THEN WHILE low < renkoMin - boxSize renkoMax = renkoMax - boxSize renkoMin = renkoMin - boxSize WEND ENDIF buy 1 CONTRACT at renkoMax + boxSize stop sellshort 1 CONTRACT at renkoMin - boxSize stop // Stops and targets SET STOP PLOSS 75 SET TARGET PPROFIT 150

   

  1 user thanked author for this post.

  nonetheless

  

  GraHal

  Participant

  Topics: 214

  Replies: 8830

  Been thanked: 2201 times
  02/24/2020 at 11:25 PM #120355
  GraHal

  The above code (or a fork of) was doing very well for me on Demo Forward Test but for the last several days it gets Rejected / Stopped every time the code is read.

  I get the Rejected with the error message … System Error.

  Can anybody see anything in the code that ProOrder or IG may not like??

  Is While, Wend approved terms etc ??  Thee must be some reason surely??

  I know IG have had big Issues today, but lots of my other Systems taking trades just fine today so why is this System getting Rejected for System Error??

  

  GraHal

  Participant

  Topics: 214

  Replies: 8830

  Been thanked: 2201 times
  02/24/2020 at 11:42 PM #120356
  Paul

  perhaps it’s not the code, I got trouble too in recent days on several systems, mainly system error

   

  1 user thanked author for this post.

  GraHal

  

  Paul

  Participant

  Topics: 40

  Replies: 812

  Been thanked: 660 times
  02/24/2020 at 11:51 PM #120357
  GraHal

  Yeah thanks Paul … I just checked and lots of others are now being rejected for System Error.

  

  GraHal

  Participant

  Topics: 214

  Replies: 8830

  Been thanked: 2201 times
  02/24/2020 at 11:59 PM #120359
  GraHal

  I’ll correct myself … the above code is the only System that is getting Rejected on my SB Account.

  Lots of Systems are repeatedly getting Rejected on my CFD Account for System Error.

  

  GraHal

  Participant

  Topics: 214

  Replies: 8830

  Been thanked: 2201 times
  02/25/2020 at 12:32 AM #120361
  Paul

  not only system error on my cfd demo account.  It’s messy a.t.m.!

   

   

   

   

   

  

  Paul

  Participant

  Topics: 40

  Replies: 812

  Been thanked: 660 times
  02/25/2020 at 12:47 AM #120365
  nonetheless

  I’ve had stuff getting rejected all day, esp on the DOW and DAX, NAS systems working fine.

  Rang IG about it but got a recorded message acknowledging platform problems and to stay on the line only if I wanted to close a position. Presumably they’re inundated with complaints.

  BTW @Grahal glad to hear that Renko thing was actually working for you! Maybe I should take another look at it…

  

  nonetheless

  Participant

  Topics: 83

  Replies: 1237

  Been thanked: 978 times
• Author
  Posts