Difference in outcome backtest and automated trading
- This topic has 48 replies, 6 voices, and was last updated 11 months ago by
.
-
can I see a value in the bid-ask band?
If you have the cross hairs set up on your cursor then you can read across to see the value.
‘Cursor Details’ shows bid and ask values (I think?) if you show the Bid Ask Band.How can you ever rely on automatic trading in this environment?
If Live and Backtest show profit > 95% then surely that is good enough?
outcome didn’t match either with the live trading.
What % difference of better still profit value difference are you experiencing between backtest and Live and over how many bars / over what period?
1 user thanked author for this post.
I think the first thing I would do is turn off the trailing stop and try to develop a strategy based only on entry and exit or SL and TP. Then you have a basic structure, or possibly a functioning strategy. Such things are also more robust against various tick-by-tick errors.
1 user thanked author for this post.
But I keep asking myself… Who released V12?!!! MT4 works even better. 😉
1 user thanked author for this post.
Dear Martin,It seems to me that everyone is struggling with the same issue, having a reliable outcome in the backtest versus live trading
You may be right, but if that is so then all these everybody’s don’t know how to make backtesting equal to what happens in Live. And True, I see most people asserting that this is not possible. Still me myself and I can do it.
I show pictures like these more often, and they are always from about “today” so nothing made up or fit to a distant past where things coincidentally worked out. This system started on January 8 and it is not the most liquid instrument (Mini DAX). I could show Micro Dax as well, but with less trades so less speaking (but Micro Dax is almost “dead” and it still works well for reliable backtesting). Notice that this is real Live (and not Demo like you showed, which latter would cause issues to begin with for IB).
Open these pictures in a new tab or window, or else the forum software shrinks it to “unreadable”.The first pic shows the overview as far as possible with some sense. The 2nd is a bit more zoomed and it would testify of 100% exact same trades per the 1 minute bars. With some eye pinching you can already see the triangles of the trade prices. The 3rd shows those triangles very well, and where you see one triangle, the overlay was exact. Where you see two triangles, the price in backtesting was slightly different. And yes, it could happen that because of a price difference the trade shifts to the next (or previous) bar. In the very end this could lead to very different results, especially in this case where trades can last long (as in several days).
Because I backtest against Live history (check in retrospect), I deem my backtesting for development 99% reliable. And 99% really is sufficient. And oh, notice that this is something different than being able to backtest against future markets. That is a whole different story. 😉
Later I can try to pull up an example of EUR/USD, which is way more difficult in my current case because that runs on 3-second systems. But you will see how I will be able to explain the differences. I will promise you in advance that these differences are nothing to write home about, hence they won’t cause the systems to fail against Live as such (compared to the backtesting I did in advance).
It would really be a good thing if people could take it for granted that backtesting against the very same results in Live *is* possible. If people would do that, it would be far more easy to “know” and take it for granted that any significant differences are their own fault. Regarding this, I may also show what happened to myself on the 15th of January, and how I changed the code to never let that (difference) happen again. And that is after a full year of fighting with PRT-IB AutoTrading which is really full of pitfalls. So am I done now ? probably still not.
Peter
2 users thanked author for this post.
EUR/USD …
Please notice that this is already hard to show because of the 3 seconds timeframe used. There’s a lot of bar data and the orders disappear in the bars. Also I can’t show more than back to Jan 8 because there’s just not more data. And the history of 1M bars won’t show the Live trades. So here we go :
Remember to open the pictures in e new tab/window in order to not let shrink the picture by the forum software.
Shown is a small part of Jan 11 (2024). I could show all the trades in 6 or so of these screenshots, but hopefully this is sufficient. There is no trade different to be seen from of Jan 8 (the content of the chart) up till today. You will trust me on that 🙂 …
You can see the triangles as well and all are accurate up to in-bar. You can see that Limit and Stop orders are used and that (thus) the triangles are not at the opening (or close) of bars.
Let’s keep in mind that each bar is 3 seconds.Like GraHal rightfully told, this is about flashing spread as well. But is it that important ? If it were, then nothing would match for Live and Backtest (in retrospect like here). Btw notice that the spread for IB is very different than for IG and that something like I show here would be impossible at IG’s (way too many fake spikes in order to be able to work with such ultra high frequency timeframes).
I provided the 2nd picture because you can see almost everywhere the ever s slightly different prices at the triangles. But notice that this is 20 times more zoomed (vertically) than what I showed in my previous post. This actually allows to see the differences between the Live and Backtest triangles.
Yes, it shows losses here (not the first trade shown) meaning that the code is very well capable of technically doing this correctly, which is quite different from Entries which must be the same (also shown) which is again different from Exits with profit. The 3rd screenshot shows those in isolation.I am running out of attachments and must hop to the next post …
2 users thanked author for this post.
The exit of the last picture shown in the previous post, looks for the gain in backtest like you see here in the first pic. The 2nd pic shows what happened Live.
Now, don’t look at the gain (profit) only, but also observe the difference in amount/contracts. This is – or could be – typical difference in Backtest vs Live #1. In other words, we must take very good care of systems to be fully linear from all angles, and this is one angle. Explaining in detail what is going on would be out of the scope of the subject of this thread, if you only can see that because of a difference in number of contracts the result will obviously also be different. Point here is :Although with Forex also slightly in order (rounding to 1000) with something like Nasdaq you may run into huge differences because it is not possible to have smaller steps than ~16000 (on the Micro). I deliberately say “~” 16000 because this varies per day. Why ? well, because the price changes each day. And backtest up to a year or more ago, and there you go (broke).
So this too is an angle of linearity – something which is hardly known for IG because there the minimum is 5000 with above that any decimal step we like. Not so for IB. There it is 16000 or 32000 or 48000 etc. … TODAY.Of course all depends on how you determine the order size. But but but … if you do that too simple, your profit of 1000 in your backtest maybe 600 in (future)) realty only, because the price changed meanwhile. And because you can also take profit on $ vs points, you may wonder which of the two will lead in what situation to which differences (unlinearity).
And then I did not talk about the differences which will occur with a currency pair, if you coincidentally (haha) don’t have your portfolio in the currency of the 2nd of the pair (like USD in this case).
So the pitfalls are everywhere and if you don’t calculate them in, your backtests will be worthless for technical reasons.Summarized and back to the subject of this post : because I can not start the backtest (for this showcase) when the real trading started for this system – which was Jan 3 – the contract amount is initially different. Is that an issue ? of course it is, if my code would take profit on $. The Exits would not be in the same place and the whole trading sequence would be different.
But as you can see not in my case, because I take care of that.About KISS … No, it can’t. My codes require 400-500 lines only to arrange for these (un)linearities. And you know what ? when you think you caught them all, in LIVE you will run into another one not thought of.
But Martin, this is all not (yet) your situation IMHO. Your situation is more of the code being technically “off”. That’s why my happening on Jan 15 is coming up. I do things wrong just the same. So, next post …2 users thanked author for this post.
there are big differences between live trading and (forward) backtesting
I might be wrong, but nowhere in this Topic have you quantified what these big differences are?
I’m not getting at you, I am interested in getting to the bottom of this. I did ask for profit differences or a % difference … did I miss the answer somewhere? 😉
1 user thanked author for this post.
This post shows the foremost reason of backtesting being different from Live. Actually this is about the numerous situations a broker can’t handle, or maybe put better : which the broker in combination with ProRealTime can’t handle sufficiently well. Almost without exception they cause losses which in backtest provided profits. It is almost always about your system being kicked out, that causing loss of income (the trade could not occur) or plain loss (the system is kicked out when in a loss state and you better stop the trade).
Maybe it depends on the number of autotrades one implies, but I can promise you that for IG I can easily cough up 20+ anomalies in the communication between IG as a broker and PRT needing to deal with it. For IB ? same thing. 20+ easily. But sadly different issues, implying that I need to cover for 40+ of these situations. Here is in this post I show my last one, and I show it so you can see that such a thing is actually your own fault. But you must first run into it – sometimes several times because you also need to recognize the situation. So here goes :
First pic shows a tick chart of an Exit of a trade which comprises 9 contracts of MES (Micro S&P 500). This is (thus) IB. But we see 17 occurrences ??
Hmm …
Let’s look at the 2nd pic. That shows what happened in the bar which triggered the situation. All kinds of anomalies show, which is actually PRT-IB doing bad here. Thus, the 0->0 you see several times is not correct (it makes no sense) which already causes the analysis of what happened, difficult.Analysis deemed necessary while I already started an email towards PRT that I want my money back. But then I did not see yet what happened by means of the 1st pic. So hmm … that shows a going Short while my system doesn’t even allow to go short. The third pic seems to show the same as the 2nd, but there is a difference : the color of the top triangle; so, that is from the backtest (in retrospect), with the mouse pointer pointing at that backtest Exit. It shows that a limit order above the price (this is on purpose) causes “an” exit.
The graphing tells us that it is most likely that both the Limit and Stop order were triggered here. So what ?
Well So what is this : We recently talked about the orders at IB not being OCO (One Cancels the Other) orders per se. This, while over at IG they are. I mean, they are by the very same means of putting orders, while in IB they are not. So what I reasoned to have happened is this :First off, one of the orders is triggered. Let’s say it is the top one (the Limit). Because of the roughness of the point step, this is not even visible in the tick data, which probably is exactly the problem (both the Limit and the Stop can be triggered at “the same time” which actually is within the 1 minute of the TF used here). Now figure this :
Both pending orders contain 9 contracts (the amount of the running position). But 9 + 9 = 18 and something can’t be. The broker – IB in this case – chokes on this, and tells PRT that something is not completely right. PRT tells herself, ok, then I will throw out this System.
And so it went.Meanwhile the position has vanished, but the Limit order is still there. It will now go Short …
OK, but wait, why is that Limit order still there ? hahaha, one of the numerous issues with the PRT code for IB. Told about it a dozen times before I gave up because nothing is solved anyway (really not in a year’s time) and what happens in combination is that my System is not handing the new pending order each minute (it is not there any more) while IB long gone told PRT to take action (in the API interface) because IB can’t deal with this situation of 18 contracts for a position of 9 and the order stays in. It as become a fixed Market order waiting for the price to be there (again) which happens 5 or so minutes later.Wait, what was the subject of this post again ?
That because I myself allow for something “wrong” to happen, my backtest will not/never show this anomaly and goes out with profit. You can’t see the backtest profit here (it was something like $50) but reality was that I had to manually close the trade at a loss of $108 (the “system” went Short while it clearly had to be Long).I know, the subject is not literally “backtesting produces a different result against live”, but still the effect is similar and as severe. Maybe you will believe me that I ran into such things seriously over a 100 times. With each of them I lose money that my backtests did not show. Demo/Papertrading also would not show any of it.
But hey, exactly all these happenings cause that my backtesting becomes more and more reality. Remember, technically speaking. Conquer the markets with it, is still a very different subject.Peter
2 users thanked author for this post.
Hi Grahal,
maybe you missed it, but I wrote in the beginning of this post that I have sometimes 70 to 90 pips difference in USD/JPY between live and live forward back testing on a daily base. That was my reason to start this post, asking whether I was doing something wrong.
1 user thanked author for this post.
70 to 90 pips difference over a 24 hour period and out of how many pips movement (loss and profit) contained in your total trades for the 24 hour period is this 70 to 90 pips?
Number of trades for the 24 hour period would also be very useful.
I’m just trying to get a feel for … am I too accepting of differences? It could be that you have not analysed / quantified as I am asking … I know I don’t! 😀
1 user thanked author for this post.
Hi Grahal, I start when Tokyo opens and deal to London close.
That is 17 hours in liquid markets.
It means 17 times a day entering a position a the top of the hour.
Nothing sophisticated just buy or sell depending on trend with the trailing stop. That’s all1 user thanked author for this post.
Find exclusive trading pro-tools on 
