Trading: All about Robustness Testing and reliability of an algorithm

Find out why and how robustness testing plays a crucial role in assessing the quality of an algorithmic trading strategy.

Robustness testing, or how to evaluate the reliability of an algorithmic trading strategy

A battery of statistical tests carried out in the most varied contexts possible, the robustness test allows the trader to ensure the efficiency and resilience of his algorithm, whatever the market conditions.

Although software such as ProRealTime makes the process of designing a trading algorithm much easier, it is still vital to test your trading programs methodically and accurately before entrusting them with the management of any of your financial assets.

The robustness test is an essential prerequisite for the successful launch of an automatic trading strategy and allows the trader to evaluate the performance of his algorithm and at the same time avoid the main pitfalls and statistical biases!

What is robustness testing?

Robustness testing is a methodology that aims to determine the level of reliability of a system.

In the context of algorithmic trading, a robustness test is used to verify the performance, consistency and resilience of a trading algorithm. By doing so, the trader can then discard the most fragile strategies and favour the most robust ones.

Although a simple backtesting operation can allow the trader to simulate the performance of an algorithm over a given period and a given market, the robustness test goes further by evaluating the significance of the results obtained.

The objective is therefore a priori very simple since it is a question for the Trader of extracting the luck factor from the equation, in order to obtain the clearest possible vision of the value of his algorithm.

However, distinguishing a lucky guess from a real ability to beat the market can quickly become quite complex. This is where robustness testing comes into its own.

Although there are many robustness tests proposed by the scientific literature (White’s test, Hausman’s test, or Breusch-Pagan’s test), they are not always easy to apply in practice when evaluating an algorithmic trading strategy.

Let’s try to go through some examples of relatively simple robustness tests adapted to online algorithmic trading.

The importance of multiplying test scenarios

Whether they are carried out on historical prices (backtests) or under live conditions but on a simulation account, the tests of your trading strategies must be as numerous and varied as possible to counter the dangers of over-optimization.

Indeed, all things being equal, the more you increase the test period of a strategy, and the more you multiply the diversity of market scenarios encountered, the more significant the result of your test will be.

For example, rather than testing your strategy over a year, why not test it over a decade? Better still, if you have already tested your strategy over a long period, what if you now decide to test it only on even-numbered days? Or to run it only in the autumn and winter months?

As you can see, the idea here is to check that the results of your strategy remain relatively constant whatever the period tested. If the results are relatively stable, you can conclude that your strategy is relatively robust (based on this criterion), and conversely, if the performance of the strategy changes completely, you should be concerned about its reliability!

Finally, this principle of test diversification also applies to the assets on which you test your strategy. Rather than testing your algorithm only on the CAC 40, why not observe its behaviour on other world stock market indices?

NB: Also make sure that the market conditions encountered by your algorithm are varied. Think of including in your backtest periods bullish, bearish and “range” market phases, but also phases of low and high volatility.

Do not neglect sensitivity analysis

While varying the data samples is a particularly useful first test of the robustness of a trading strategy, it is strongly recommended to complement it with one or more sensitivity analyses.

Sensitivity analysis is, as the name suggests, a way of measuring the sensitivity of a trading strategy’s performance to changes in one or more of its variables.

For example, let’s say your strategy is programmed to automatically place a stop-loss order at a distance of 50 points and a take profit at a distance of 100 points.

What happens if you now test it with a stop-loss at 49 points and a take-profit at 101 points? Do the results remain the same or do they change dramatically?

The Monte Carlo method allows you to evaluate the sensitivity of your strategy accurately so that you can get a clear picture of how your algorithm will behave in a wide range of scenarios.

Monitor the right indicators

Finally, when carrying out your robustness tests, make sure you are not hypnotised by the performance of your algorithm alone. Indeed, in the financial markets, a performance only makes sense in the light of the risk taken to achieve it!

You will agree that a strategy with a performance of +50% one time out of ten is meaningless if it causes you to lose all your capital the rest of the time.

Consequently, it is important to follow other criteria such as the Profit Factor, the Maximum Drawdown, the number of consecutive losing trades, the ratio of winning trades to losing trades…

Above all, your robustness tests should allow you to reassure yourself on the fundamental points in your eyes, and allow you to select a strategy in accordance with your personal situation and your own objectives.

In conclusion, testing the robustness of a trading strategy is essential before entrusting it with any of your capital. Many strategies that look very attractive turn out to be particularly fragile when examined under the microscope.

Share this