Boy oh boy, do I have found a beautifull example that truly demonstrates that something is wrong with the Optimization test. But I guess Nicolas already acknowledged that.
I’ll post it anyway, maybe it can give the PRT crew an extra insight in where things might go wrong.
Exhibit A
We start our example with a combination of which we know produces a top result (SLL: 0,95 – SLS: 0,98)(We’ll call this our Start Combo)
Exhibit B
We run an optimization test for a total of 100 combinations.
As the test is running we see our Start Combo appearing in the list.
Exhibit C
The test is complete.
Not only is our Start Combo nowhere to be found (as are a bunch of other top result combo’s), but the top result is lower than our Start Combo.
Exhibit D
Since the number of combinations is 100, our Start Combo has to be in the Optimize Report, since this is always a top 100 list.
So we filter the Optimize report in an Excel sheet, and what do we see: the 0,95 and 0,96 SLL series are missing.
Coincidentally, or not, the series that produces the best results.
But then it dawned upon me:
If the min value of SLL is 0.9 and the max value is 1.0, and the min and max value are included in the test, that means there are a total of 11 values to be tested for SLL.
Same goes for SLS.
11×11 = 121
That means that the displayed total number of combinations is also incorrect.
When you look at the Excel file, you will see that the min and max values are indeed included, but since there is only room for 100 combinations, it is no wonder that we can’t see all combinations. But as I already mentioned, it seems the best ones go missing.
