Pattern matching price movements in PRT – machine learning
Forums › ProRealTime English forum › ProBuilder support › Pattern matching price movements in PRT – machine learning
- This topic has 18 replies, 1 voice, and was last updated 2 years ago by Khaled.
-
-
02/03/2017 at 5:45 PM #23818
I am wondering if anyone out there has experimented with pattern matching, in PRT, or using another platform/language.
By pattern matching, I mean for example: comparing the last 20 bars, trying to find a sequence of 20 bars in previous history that best match it. From this you can make predictions on how such a similar event will fall out. You compare two price sequences by finding a common low and/or high point as your base(s), and then compare each close. The difference between each close is then squared and a total score is produced for the sequence. The sequence with lowest score is the best match. Advanced algorithms would compare highs and lows as well.
Now, the problem is that PRT has a limitation on the number of iterations (for loops/if statements) that you can include in a system. This is probably to reduce server load. It also counts all indicators used in a system, so the max iterations are applied to the whole system. I believe this limit is 150 iterations per bar or something. It is possible to check for example this day’s price movement from 9;00 to 11:00 as compared to the same time period for each of the previous 100 days or so. To me, this is not enough. I have tried this out, but I am still unsure whether pattern matching is a viable strategy. In what markets/timeframes do you know/believe it performs well, or at all?
To battle this limit on PRT iterations, I have a few ideas in mind:
- Create multiple systems/indicators, that each check a certain time period every day.
- Build a smarter pattern matching algorithm that relies on high/low points instead of brute-forcing and checking every bar in every sequence. Does anyone have any ideas on this?
- If I can only compare a 15-bar sequence to 100 other 15-bar sequences, then create a filter that only compares it to good sequences. For example: sequence has a minimum range or is in a down/up trend.
- Reducing the number of iterations by storing/checking information in variables in a better way. Why, oh why, are there no arrays (except indicator arrays) in probuilder/proorder?
- If the above fails (which may be likely), build this pattern matching algorithm in another language.
Would appreciate any ideas on how to accomplish this. Ready to explain my ideas further if there are questions. I might post some code soon of my experiments so far.
I will also add that a bar sequence can be compared in two ways: with or without relativity. To compare two price movements that look similar, but where one has a range of 100 points and the other 50 points, the large one will be compressed to more easily be compared with the smaller one.
02/08/2017 at 7:32 AM #24281Since I have no replies I will share some code and promising results I have. This code is a proof-of-concept indicator that uses pattern matching to analyze for example the daily open of DAX30 on a 15m or 30m timeframe. I have included an image also that shows that this indicator can be used to create an edge. Since neither the indicator or system is finished, I don’t want to publish it in the library yet, but feel free to experiment with the indicator below. If people find this prototype of use it can be published in the library.
Indicator name: Wing’s Prophecy Boot version 0.1
Indicator description: Every day at a set time (time of open for example) indicator returns a score of positive or negative. This is the indicators prediction of how price will move. Bigger score means more probability.
How it works: At set time (example: 090000) check the 10 preceding bars. Compare these 10 bars to the exact same sequence of bars, in each of the approx. 100 previous days. Find the three best matches. Look how these best matches did, in terms of price change n bars later. Based on these previous movements, and their build-up’s similarities to this day’s price movements (accuracy of the pattern matching), form a ‘prophecy’ for how this day’s price will move.
Redommended timeframe: 15m or 30m.
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116//////////////////////////////////////////////////////// Wing's Prophecy Boot version 0.1 - (Pattern Matching indicator)//// Made by user "Wing" of ProRealCode.com// https://www.prorealcode.com/user/wing///////////////////////////////////////////////////////// Parameters:sep=92 // Number of bars between each pattern match. Should be equivalent to the amount of bars per day. Roughly 47 for 30m TF, 92 for 15m etcstarttime=090000 // What time each day to compare current pattern to previous daysfut= 16 // How many bars from best pattern match to look forward and see if pattern results in price increase/decrease. Must be less than 'sep'.hindsight=95 // Number of days to look back. DO NOT CHANGE THIS PARAMETER unless you receive an error message. Generally, it can't look back more than 100 days.//////////////////////////////////////////////////best=0waw=barindextbest=100000tbest2=100000tbest3=100000lo=lowest[10](close)cl=closeif time=starttime thensep2=sepif time[(sep-4)]=starttime thensep2=sep-4elsif time[(sep-3)]=starttime thensep2=sep-3elsif time[(sep-2)]=starttime thensep2=sep-2elsif time[(sep-1)]=starttime thensep2=sep-1elsif time[(sep+1)]=starttime thensep2=sep+1elsif time[(sep+2)]=starttime thensep2=sep+2elsif time[(sep+3)]=starttime thensep2=sep+3elsif time[(sep+4)]=starttime thensep2=sep+4endifwaw=waw-sep2FOR i = hindsight DOWNTO 0 DObest=(0.01*(square((cl-lo)-(cl[barindex-waw]-lo[barindex-waw]))+square((cl[1]-lo)-(cl[barindex-waw+1]-lo[barindex-waw]))+square((cl[2]-lo)-(cl[barindex-waw+2]-lo[barindex-waw]))+square((cl[3]-lo)-(cl[barindex-waw+3]-lo[barindex-waw]))+square((cl[4]-lo)-(cl[barindex-waw+4]-lo[barindex-waw]))+square((cl[5]-lo)-(cl[barindex-waw+5]-lo[barindex-waw]))+square((cl[6]-lo)-(cl[barindex-waw+6]-lo[barindex-waw]))+square((cl[7]-lo)-(cl[barindex-waw+7]-lo[barindex-waw]))+square((cl[8]-lo)-(cl[barindex-waw+8]-lo[barindex-waw]))+square((cl[9]-lo)-(cl[barindex-waw+9]-lo[barindex-waw]))))if best<tbest thentbest=bestind=cl[barindex-waw-fut]-cl[barindex-waw]elsif best<tbest2 thentbest2=bestind2=cl[barindex-waw-fut]-cl[barindex-waw]elsif best<tbest3 thentbest3=bestind3=cl[barindex-waw-fut]-cl[barindex-waw]endifsep2=sepif time[barindex-(waw-(sep-4))]=starttime thensep2=sep-4elsif time[barindex-(waw-(sep-3))]=starttime thensep2=sep-3elsif time[barindex-(waw-(sep-2))]=starttime thensep2=sep-2elsif time[barindex-(waw-(sep-1))]=starttime thensep2=sep-1elsif time[barindex-(waw-(sep+1))]=starttime thensep2=sep+1elsif time[barindex-(waw-(sep+2))]=starttime thensep2=sep+2elsif time[barindex-(waw-(sep+3))]=starttime thensep2=sep+3elsif time[barindex-(waw-(sep+4))]=starttime thensep2=sep+4endifwaw=waw-sep2if waw<200 theni=0endifnextendifgoer1=0goer2=0goer3=0goer1=(ind/square(tbest))goer2=(ind2/square(tbest2))goer3=(ind3/square(tbest3))goerz=goer1+goer2+goer3return goerz02/08/2017 at 9:07 AM #24287That’s what I talked about yesterday when I was talking about interesting thread I have lost in the forums.. 🙂 I’m subscribing!
Arrays should have been useful here you are right, and I’m impressed by your work to fill the void of it.
Your approach is a kind of Goertzel algorithm? Find an already known frequency in a larger spectrum? Should be a great idea to comment a bit your indicator’s code, otherwise you’ll lost a lot of people..
Many statistical studies have concluded that the ORB (open range breakout), the direction of the price at the opening of the market is significant for 30% of the sample.
1 user thanked author for this post.
02/08/2017 at 11:25 AM #24300I’m glad you like it. It took some thinking to figure out a good approach.
I have not heard of the Geortzel algorithm previously, but since it is used in tone detection, it should be fairly similar.
I can’t edit the code post to add comments, so I will do it below:
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120//////////////////////////////////////////////////////// Wing's Prophecy Boot version 0.1 - (Pattern Matching indicator)//// Made by user "Wing" of ProRealCode.com// https://www.prorealcode.com/user/wing///////////////////////////////////////////////////////// Parameters:sep=92 // Number of bars between each pattern match. Should be equivalent to the amount of bars per day. Roughly 47 for 30m TF, 92 for 15m etcstarttime=090000 // What time each day to compare current pattern to previous daysfut= 16 // How many bars from best pattern match to look forward and see if pattern results in price increase/decrease. Must be less than 'sep'.hindsight=95 // Number of days to look back. DO NOT CHANGE THIS PARAMETER unless you receive an error message. Generally, it can't look back more than 100 days.//////////////////////////////////////////////////best=0waw=barindextbest=100000tbest2=100000tbest3=100000lo=lowest[10](close) // Set a base line. Each sequence is compared by placing their lowest close on the same level.cl=closeif time=starttime then// correct the day separator, so the right time is compared. Some days can have 98 bars, others 96 bars etc.sep2=sepif time[(sep-4)]=starttime thensep2=sep-4elsif time[(sep-3)]=starttime thensep2=sep-3elsif time[(sep-2)]=starttime thensep2=sep-2elsif time[(sep-1)]=starttime thensep2=sep-1elsif time[(sep+1)]=starttime thensep2=sep+1elsif time[(sep+2)]=starttime thensep2=sep+2elsif time[(sep+3)]=starttime thensep2=sep+3elsif time[(sep+4)]=starttime thensep2=sep+4endifwaw=waw-sep2// Check each dayFOR i = hindsight DOWNTO 0 DO// compare each sequence to the current one, and create an accuracy score. The lower score the betterbest=(0.01*(square((cl-lo)-(cl[barindex-waw]-lo[barindex-waw]))+square((cl[1]-lo)-(cl[barindex-waw+1]-lo[barindex-waw]))+square((cl[2]-lo)-(cl[barindex-waw+2]-lo[barindex-waw]))+square((cl[3]-lo)-(cl[barindex-waw+3]-lo[barindex-waw]))+square((cl[4]-lo)-(cl[barindex-waw+4]-lo[barindex-waw]))+square((cl[5]-lo)-(cl[barindex-waw+5]-lo[barindex-waw]))+square((cl[6]-lo)-(cl[barindex-waw+6]-lo[barindex-waw]))+square((cl[7]-lo)-(cl[barindex-waw+7]-lo[barindex-waw]))+square((cl[8]-lo)-(cl[barindex-waw+8]-lo[barindex-waw]))+square((cl[9]-lo)-(cl[barindex-waw+9]-lo[barindex-waw]))))// save the three best matchesif best<tbest thentbest=bestind=cl[barindex-waw-fut]-cl[barindex-waw]elsif best<tbest2 thentbest2=bestind2=cl[barindex-waw-fut]-cl[barindex-waw]elsif best<tbest3 thentbest3=bestind3=cl[barindex-waw-fut]-cl[barindex-waw]endifsep2=sepif time[barindex-(waw-(sep-4))]=starttime thensep2=sep-4elsif time[barindex-(waw-(sep-3))]=starttime thensep2=sep-3elsif time[barindex-(waw-(sep-2))]=starttime thensep2=sep-2elsif time[barindex-(waw-(sep-1))]=starttime thensep2=sep-1elsif time[barindex-(waw-(sep+1))]=starttime thensep2=sep+1elsif time[barindex-(waw-(sep+2))]=starttime thensep2=sep+2elsif time[barindex-(waw-(sep+3))]=starttime thensep2=sep+3elsif time[barindex-(waw-(sep+4))]=starttime thensep2=sep+4endifwaw=waw-sep2if waw<200 theni=0endifnextendifgoer1=0goer2=0goer3=0// the three best matches. the price change is divided by the preceding sequence's similarity to current sequence. A low similarity score means more similarity, which results in a stronger signal.goer1=(ind/square(tbest))goer2=(ind2/square(tbest2))goer3=(ind3/square(tbest3))// add the three best matches together, for a better predictiongoerz=goer1+goer2+goer3return goerz02/08/2017 at 11:29 AM #24304I will also add that there is the possibility to create a REVERSE pattern matcher. I have that code somewhere, and I might post it later if I find it.
A reverse pattern matcher creates composite images of all patterns that preceded event A. For example, event A could be price moving over 1 % upwards within a span of 10 hours. The average of all preceding price movements to event A can then be compared to the current price movement and maybe predict when new such events will occur.
This approach is much less taxing on the PRT servers, and therefore let’s you take more data into account, compared to the code posted above. However, it is less versatile and accurate.
02/08/2017 at 12:03 PM #24308I changed the bars comparison sequence with a loop to have more bars than 10:
12345678// compare each sequence to the current one, and create an accuracy score. The lower score the better//best=(0.01*(square((cl-lo)-(cl[barindex-waw]-lo[barindex-waw]))+square((cl[1]-lo)-(cl[barindex-waw+1]-lo[barindex-waw]))+square((cl[2]-lo)-(cl[barindex-waw+2]-lo[barindex-waw]))+square((cl[3]-lo)-(cl[barindex-waw+3]-lo[barindex-waw]))+square((cl[4]-lo)-(cl[barindex-waw+4]-lo[barindex-waw]))+square((cl[5]-lo)-(cl[barindex-waw+5]-lo[barindex-waw]))+square((cl[6]-lo)-(cl[barindex-waw+6]-lo[barindex-waw]))+square((cl[7]-lo)-(cl[barindex-waw+7]-lo[barindex-waw]))+square((cl[8]-lo)-(cl[barindex-waw+8]-lo[barindex-waw]))+square((cl[9]-lo)-(cl[barindex-waw+9]-lo[barindex-waw]))))barstocompare=11result=0for a = 0 to barstocompare doresult = result+square((cl[a]-lo)-(cl[barindex-waw+a]-lo[barindex-waw]))nextbest = 0.01*resultI fell into the infinite loop bug of the platform if I set “barstocompare” to 20, only tested 11, currently testing more bars. Is it still accurate for you? Maybe adding this code into an external indicator as a function would reduce the phenomena.
edit: 15 bars is workings ok.
02/08/2017 at 2:26 PM #24336Nice loop. My goal is of course to increase the amount of bars per sequence, and days checked.
I have no idea what the maximum bars are, but the more the merrier. The maximum iterations limit counts everything called in the program I believe.
And yes, adding things as external indicators and calling them in an indicator or system sometimes helps.
02/08/2017 at 4:01 PM #24367So in your own development, you were trying to compare pre-open price behaviour to predict the whole daily movement or only for a short period of time?
Did you try to compare the London session instead to predict price direction of the NY one?
02/08/2017 at 4:43 PM #24381My system where I have tried this indicator, I check price change for the whole daily movement. From open to close. Optimal prediction depends on the length of sequence, time of day, market, timeframe etc.
You do have a point with your example of London and NY. Pre-open movements are less reliable. It is something I have to look into.
02/16/2017 at 5:04 AM #25237So far I am unsure if pattern matching works. On the 15/30m timeframe I can create an edge in DAX. But the system has many moving parts (variables) that make biases and over-optimization a risk. It seems it works best on indices though. My next move will be to investigate if it can perform in equities and maybe on new timeframes.
03/01/2017 at 5:52 PM #26829I have abandoned the approach of price pattern matching, since I couldn’t produce any compelling strategies. An edge exists, but not enough to cover spreads and drawdowns. I am however experimenting with indicator movement matching. For example, the movement pattern of an momemtum indicator compared to previous days, combined with also pattern-matching movement patterns for trend indicators etc. This is an attempt to find a similar price movement, but not by comparing price, but by taking other perspectives into account (different indicator types and periods, etc). This approach could be even more complex than the previous one and therefore less reliable and understandable, but I will post more if it turns out to be good.
03/01/2017 at 6:26 PM #26833Your code is an interesting approach and be sure I can help if you need assistance on any project related to pattern matching.
You are right about the existence of an edge somewhere in pattern matching, but reducing it only to candlesticks patterns is almost like trading market noises IMO. I’d like to train an algorithm to find every combination of short term market’s “pause”, sounds tricky though..
05/17/2017 at 11:08 AM #35812This topic is my bag…. Can we discuss offline?
Good job for getting this far with top-down code without arrays @Wing, I am impressed. So just some background – we have systems using multi layer SOMs, running in OpenCL on some very custom hardware. With arrays in PRT this could also be attempted there. The top-down structure is not really adept for this kind of work – you need parallel processing to harness the power of it. But seriously good work. Would like to get in touch.
05/17/2017 at 12:28 PM #35817Thank you Wing for your outstanding work, I have been wandering wheter machine learning algos could be used on PRT, I’ll do my best to help on this
05/17/2017 at 3:08 PM #35853I have a system based on another form of machine learning I can maybe show if you contact me. I have planned to post it publicly but have not had time to do so yet.
1 user thanked author for this post.
-
AuthorPosts
Find exclusive trading pro-tools on