Some thoughts on the Library of Strategies
- This topic has 31 replies, 8 voices, and was last updated 6 years ago by
.
-
I think everyone appreciates all the hard work and ideas that people put in to their strategies being posted but is the Library getting mostly full of curve fitted / over optimised code? Pretty equity curves look nice but can bite!
Should the library be split in to two sections. One for algos that have been curve fitted and have no proof of IN/OUT sampling and one for algos that have IN/OUT sampling performed with evidence to prove it.
Otherwise the library of strategies becomes mostly full of an optimised view in to history without any confidence of them working and making money going forward.
I like the Sandbox ideas from Nicolas that are market agnostic as this at least lets people know what they are getting and they can then do their own due diligence or use the snippets of code in their own code. I do believe that there is a very good reason to keep the curve fitted code though, if separated. Take Reiner’s pathfinder, it’s full of good ideas and code snippets that I often borrow so these algos are great for reference.
I do think that this would really improve the site and make things easier for the new people coming in, learning PRT and creating algos the correct way.
What do you think guys / Nicolas?
1 user thanked author for this post.
You don’t have an idea of how many strategies (or even indicators), I do not allow to be featured into the library. Sometimes I send the post into the forum to discuss about the idea, sometimes I try to get in touch with the person by email, etc.. the choice is often difficult because of these main considerations:
- I do not want to discourage anyone from contributing to the site and the code library. Humanly speaking, it is very difficult to deny someone a free gift to the community by telling them that what he thought was right is actually a disaster. When the post seems to me to be suitable enough (because it embeds a new notion of interesting programming or because it seems to me to be at the limit of the over-optimization, I let pass). Very often, I post the first comment to alert on this, I do not wish and never will let anyone think that a strategy would be as profitable in the future as it is in the past, it would make I, and therefore the site, still a new site of charlatanism on trading, and believe me this is the last of the things I wish!
- What is an over-optimized strategy? Difficult to say, even certain will continue to perform in the future. Of course, and since the appearance of the Walk Forward module in the platform, one can easily infer that a strategy is or not, even if this tool will never be a preacher of the future of the market.
Since the beginning of the site, I have constantly warned all users about the hassle of optimization and the best ways to avoid it (blog post, forums, comments, etc.). I can not be behind every intellectual process that has spawned a new strategy. By the way, who can tell me if a period 21 RSI is better than a period RSI 14 on a 15 minute time unit on the DAX between 8:00 and 12:00 on Monday? 🙂 - The code library is not a free service of plug and play strategies to get rich. Everyone will shop there in whatever is available and make their own opinion.
The spirit of sharing that prevails here, I have rarely seen it elsewhere and if I allow myself to close the door to people who have never heard of over-optimization, then I condemn everyone to miss a mass of Information and knowledge. Believe it or not, even I learn things from the beginner in trading! I never take anybody high, I consider that all ideas are good and that we must let them have their chances.
I hear all the ideas that make the site better and I am very happy that everyone is sharing their ideas. But, if I may, what you propose is to create a library of code for the elite and another for the beginners?
Very honestly, it seems very difficult to classify the strategies to indicate to the visitors of the site: with these, you will make money (and therefore I will give an investment advice) ..
1 user thanked author for this post.
Always appreciate your comments Nicolas. I don’t think we need to put it as harshly as one for beginner and one for elite or ‘this will make you money’ and ‘this won’t’ this is too far. I’m glad you can filter out a lot though 🙂
I still think some categories would be helpful. Sandbox. Fully Optimised. IN/OUT sampled.
Hey, if you think it will be too difficult / not practical to do this then that’s fine. It’s your vision and you have a lot more experience than I but it was on my mind as a suggestion.
1 user thanked author for this post.
I agree on the need to not alienate or discourage new members, we all learn something new everyday (I certainly do !). But there may be a need at the same time to provide some sort of scaling system as the forum grows. How about a two scale (or more) rating system similar to the ones that the credit rating agencies have ? An overall one sometimes may not provide the right “at a glance” information.
eg. A – E for the overall system performance and 1-5 for the optimisation /WF testing confidence score. The banding categories would need to be discussed but could be something like eg:
A – has been tested and verified in Live for at least x months ; B – running in live for <6mths; C – Demo….. E – Only backtesting performed
1 – W/F testing performed and passes within x% efficiency band….. 4 – no Walk forward testing performed, 5 – Over optimised,And then fill in the bands in between. So the highest rated system would be A1
As members improve/test the strategies they can change the ratings accordingly. However, if you were to feed this into an overall rating then the challenge comes if users don’t update as the ratings become stale. It’s not an easy one to address unfortunately. Hopefully, this is a start !
1 user thanked author for this post.
I find the rough ideas more useful than something that is completely finished to suit the portfolio of somebody else.
Yes, that’s why forum threads are quickly opened right after the posts has been published, more convenient to talk in forums than in the comments section of library’s posts. For this particular case, may be a “tag” like the one Cosmic has talked about could be added also: “Sandbox. Fully Optimised. IN/OUT sampled.” …
1 user thanked author for this post.
On this subject, I can code what one wants, whether for the system of ratings or tags! It is a secret, but I am currently reviewing a lot of things to speed up the site and make it more attractive, I foresee a big update by the summer .. So as long as I am I will like to incorporate the scoring system. Feel free to continue discussing it, I must have a global view of the choices available to us, even though they might seem a bit ‘raw’ at first.
1 user thanked author for this post.
It’s not a secret any more Nicolas!!! hahahaha I don’t know how you do it all, you must never sleep!?
Education & organisation (re Cosmic) is a good idea … there is so much good information on this site be good to tie together like that. Also record of evidence supporting scores (re Cosmic) in itself could be educational.
1 user thanked author for this post.
It’s an interesting topic, at first I didn’t want to have an opinion on this, mainly because my preferred kind of trading is running one fairly large programme and keeping all my focus on it, not a cumulative approach of several small programs from library with the purpose of combining their strength and mitigating their weaknesses, so I thought I was not in a good position at all to discuss a type of scoring for these (plus, it’s not my place to comment other people’s strategies).
However, there’s a general point I might want to insist on and I’d join those who already mentioned something similar: before any kind scoring is started, I would welcome “some” sort of harmonization of results presentation, and by this I mean making common some items (so amending strategies pages accordingly): same starting capital (example 10000euros), contract size adjusted to achieve one or several predetermined max drawdown levels before being in gain forever, and no size increase with portfolio increase (because it only hides the true regularity of the P&L, reinvesting all gains is a personal decision, not a strategy performance decision). Then and only then we could have a starting baseline to compare like for like. Note that size increase within strategy decision is not forbidden by this, be it with adding trades with the trend, or with second and/or third entry after a not yet winning first entry, it just needs to be taken into account within the wider parameter of the accepted max drawdown level.
Here’s an example which you’ll surely find very obvious, but it illustrates the point: a strategy displaying a 500 starting capital, reaching a 400 drawdown before showing 1000 capital in 12 months could make you think you can make 100% in 12 months, but 100% of what? Well if I want to work with 4% drawdown only rather than 80% on my capital, I’d need a 10000 starting capital to accept running with 400 max drawdown on intial capital, not the 500 displayed as “money risked”, and I’ll gain the same 500 for the given contract size, so the real performance is 5% in 12 months for a 4% drawdown. Using 10000 is talking in “starting capital”, using 500 as false starting capital and allowing a big drawdown was confusing the starting capital with some sort of “rounded up max drawdown money risked” number to be closely approached whilst not touched by real drawdown… Not so sure all beginners would see that, not just before scoring a strategy using such poor combination of starting capital vs max drawdown, but also (and that would be the worst consequence) before running it with their own capital and seeing it melt by an 80% order of magnitude…
So, I’d say, first and foremost, harmonizing presentation of strategy backtests with a given starting capital (10000 was just a common example, could be whatever…), and backtest(s) with appropriate contract size needed to achieve one or several predetermined max drawdown key levels (5%, 10%, 15%, …) – I keep the possibility of a plural for backtests just because the non-cumulative order stategies might not have a linear behavious for performance vs max drawdown – in other words the minimum harmonization requirement could be summarised as “same starting capital + never display a performance percentage per year without mentionning the associated max drawdown encountered”
And from there I would be comfortable adding scoring criteria discussed throughout the topic to assess on top of this the robustness of the process (WFA etc…), otherwise I might be distracted/impressed by the good “process” score of a low return strategy, or similarly deterred by the poor process score of a promising good return strategy that might see its good performance not suffer too much from following a better process to amend it…
Find exclusive trading pro-tools on 
