Hi Oli,

Personally I don’t see anything strange so far. Indirectly this relates to you not showing any, say, proof of oddness. In other words, would you have shown where the Demo system went off vs the Live system, then I wouldn’t have a case.

It is a bit strange to tell this in aftermath, but what you still can do is this :
Provide your code with Graph and GraphOnPrice all over the place for key variables, and run it in backtest from the 100% exact time as you did with your Live system. This means, to the second (which you already can’t really because it is per the minute, but alas). You will soon see that your backtest does different things than your Live system (and Demo for that matter), but the point is : it will make you wonder why. And you will find out. Please notice that you can run this same backtest in Demo and in your live account.
You will run into a dozen of questions (which you may put over here in this thread) and you will learn what’s all going on.

You said that you have been testing thoroughly for the past 3 years and of course I believe that. But what I tried to say in between the lines in my text above is : if you had done that per the means I now suggest, you would have come up with the proof of oddity long gone. And thus : you did not test your systems the past 3 years like this ?
None of my systems is ever doing something strange suddenly. I could add : PRT is decent enough to not let this happen. However, *if* this happens, I immediately run my available backtest against it in retrospect, and I will immediately find out the culprit. Always. And with that said : none of my systems is ever doing something strange suddenly. Ah, I said this already. But the point is : it is always me who did something wrong or strange or whatever it is, causing the Live system do something unexpected. Anyway, it can always be explained.

Now what if you already did this all ? …
Look for oddness in the data itself. Obviously you will have looked whether the data in both Live and Demo was different (of course for Live you’d need to look back in “history” but that is no problem). Assumed it is not different : look for spikes. A spike in this case could be a huge one – one which is not realistic, hence did not happen for real on the Exchange, but still is there – on both Demo and Live. Now think of different treatment of oddities in Demo vs Live …
Data providers and especially beyond (like the broker – like PRT) can filter out unrealistic spikes. But did that happen in Demo ? No, that most probably did not happen in Demo. This means that your code in Demo will have seen the spike and act upon it, while the code in Live never saw it. Still you would be able to look at it today (from Monday morning I suppose). Set your timeframe to 1 second or even ticks perhaps.

I am not making this up, and when I dig very hard I can find a ticket for this from 6 years ago or so, where my Demo system indeed was molested forever, because it lost 100K in one trade. The price went up for a full USDollar (on the 1,20 or so for the EUR/USD Fx), it won a million or whatever it was, and next it went short after that (Reversed IIRC) and lost 1.1 million. Thus net a 100K loss.
But the spike was wrongly provided in the data and this can happen. … And as you can imagine, all systems MUST be able to filter such oddity and they thus do. But the Demo in PRT does not. Btw this was with PRT-IB, and while this is a decent environment, I can imagine that with IG and their CFD environment this can be worse.

Regards and please let us know your findings !
Peter

Hi again Oli.

Please notice that in Live (not Demo) these spikes will never be an issue (and if it is, you can claim money back when you lose because of it). Thus, this is only about the realism of Backtesting.

I must say, my personal largest effort goes into realism of Backtesting indeed. It is almost more important than the good strategy itself (nothing new to you 🙂 ). But this one with the spikes ? I don’t think anything can be found to make that more real during backtest. One thing to mention perhaps : other platforms may show this as “outliers” up to even don’t count them in the backtest result. PRT does not contain this “statistic” but they could make it for our better insight. Thus, my own example with a profit of 1 million and a loss of 1.1 million upon normal profits and losses of say 1000, are outliers. They should not be counted because they are “not normal”. Still, in normal life and Live, outliers exist all over, like with (Fed) news. That too urges for huge swings your backtest in the past and Live for today will capture. Still, your backtesting – optimising actually, will always be in favor of that outlier. Thus, it will always “form” the trades such that when normally profits of 100 occur, that outlier of 1000+ will form your trades (in Live no “forming” can happen of course).
This is more difficult to explain, so I hope you get the gist of this ?

Both feet on the ground, anomalies because of these spikes are rare. I have seen them on the charts a couple of times, but really only a few times (which does not tell all because I can not see the charts of all instruments). However :
What I do know for sure is how historical data fails very often on minor unrealistic spikes. It is, for example, no exception at all to find data with the last trade price beyond bid and ask, which can not be in real life. This happens many times per day with regular data providers, and your backtesting will rely on it. Thus, bid was 6000.25, ask was 6000.75, but price at the same time is 5995.75. This simply can not exist. This is the problem of historical data providers. PRT is actually also one (of its own) and I don’t know at all whether PRT does a good job here; we just can not tell without the bid/ask quotes data (which PRT does not provide).

So is the latter a real problem with backtesting ? … not that I have seen, as long as you don’t want to make profit over 0.25 or 0.50 points.

Stick with the advised proof of what happened with the Graphing tools (ran afterwards through backtest) and I take it you will be sufficiently good.
Regards,
Peter

Here you have it at the tick level;
The more experienced eye can see that the oblique red downwards like just shows like that because of the graphics rendering. It is all the same ne “tick”, reaching the bottom at 00:01, staying there until the 11th second.

It is hard to tell for me what happens after that (say until in the 12th second) because I am not experienced with looking at the Russel2000 at the re-opening, while this looks like normal trades to me. The first larger jump-up after the “stall” is 5 points. This seems too much to be realistic.