Exploring the risks of backtesting: Understanding survivor bias in trading strategies

The risks and assumptions of backtesting trading strategies

By simulating trades using past prices and specific trading rules, traders can estimate how a strategy might perform in real market conditions. However, backtesting comes with certain risks and assumptions that can undermine the accuracy of its results. One of the most critical issues is survivor bias. This article explores survivor bias, its impact on backtesting results, and why traders must be cautious when interpreting backtest outcomes.

What is survivor bias, and why should traders be concerned?

Survivor bias is an error that occurs when analysis includes only those stocks or assets that have “survived” over a testing period, excluding those that failed or were delisted. Survivor bias can lead to overly optimistic results, as it focuses on the success stories while ignoring cases where investments performed poorly or ceased trading entirely.

To illustrate this, consider a story from World War II. Aircraft returning from missions often showed damage in particular areas, and engineers initially focused on reinforcing these parts. However, this approach missed a key insight: planes that didn’t return likely suffered damage in other critical areas, making them unable to complete their missions. Once engineers began reinforcing the areas that showed minimal damage in the surviving planes, losses were significantly reduced.

In trading, the principle is similar. When backtesting a strategy over a long period, it’s essential to ensure that the stocks included in the test were indeed tradeable throughout the entire period. Otherwise, the data will be biased towards stocks that survived, leading to unrealistic profit expectations. Excluding delisted stocks from a backtest can skew results, presenting a strategy as more profitable than it would be in real-world conditions where not every stock maintains market listing.

Survivor bias in backtesting: How it distorts results

Backtesting without accounting for survivor bias often leads to inflated profitability estimates. If a test only includes stocks that remained in the market, it effectively ignores all cases where the stock failed, was delisted, or incurred severe losses, which would have negatively impacted the results.

For instance, if a backtest covers a ten-year period but excludes stocks that delisted or dropped significantly in value, it presents a skewed version of reality. This test would only show stocks that successfully “survived” over those years, leading to a selection bias. To accurately test a trading strategy, the backtesting process must include delisted stocks and account for their delisting dates.

The backtesting software should be able to:

1. Identify delisted stocks and include them in the analysis.
2. Record losses for any open trades in delisted stocks, treating them as full losses in the results.

For example, if a trader has an open position in a stock that subsequently delists, the test results should reflect a 100% loss for that trade. Ignoring this loss creates a misleading view of a strategy’s performance and profitability.

Survivor bias in action: Does it make a big difference?

Survivor bias can have a significant impact on backtesting results. Without survivor bias adjustments, a backtest can show profitability figures several times higher than if the analysis had included delisted stocks. This exaggeration occurs because the test fails to account for the natural attrition that occurs in real markets, where companies regularly fail, are acquired, or go bankrupt.

In practice, most backtesting software is not equipped to handle survivor bias effectively. Some data providers even exclude delisted stocks entirely, meaning that a backtest run on their data would omit any stock that didn’t survive the entire testing period. For traders who rely on these results to make investment decisions, this bias can lead to strategies that perform well in testing but fail to deliver similar results in live trading.

Understanding the limits and benefits of backtesting 

Despite its flaws, backtesting remains a widely used tool for testing trading strategies. It allows traders to evaluate the impact of individual trading rules and compare different strategies to see which is superior. However, it’s important to view backtesting results as an approximation rather than a guaranteed outcome.

Backtesting allows traders to:

1. Compare strategies: Testing different rules or strategies side by side can highlight which approach may be more profitable.
2. Save time: Computerised backtesting allows traders to test multiple strategies quickly, far faster than manual calculations.
3. Adjust rules in real-time: Backtesting makes it possible to fine-tune trading rules and evaluate how changes impact results.

That said, markets are dynamic, and historical data does not always reflect future conditions. This limitation is summed up in the familiar phrase, “Past performance is no guarantee of future success.” Each market cycle has unique characteristics, and markets rarely repeat in identical patterns. Therefore, traders must approach backtesting with caution, understanding that real-world results may differ from those predicted by historical analysis.

Applying caution to backtesting results

While backtesting can yield impressive results, traders must be wary of overconfidence in these outcomes. As the saying goes, “If it looks too good to be true, it probably is.” Even if a backtested strategy appears highly profitable, expecting these results to replicate in live trading can be risky.

Instead, traders should approach backtesting results with a critical eye, keeping several factors in mind:

1. Quality of data: Ensure that the data used for backtesting includes delisted stocks and accounts for all historical information.
2. Realistic expectations: Treat backtesting as a comparative tool rather than a definitive prediction of profitability.
3. Paper trading: Test a backtested strategy in a simulated environment before risking real capital. Paper trading helps confirm that the backtesting results hold up under live market conditions.

By using backtesting results as a preliminary assessment, traders can identify promising strategies, but they should always validate them with real-time paper trading. This additional step helps bridge the gap between theory and practice, enabling traders to assess how well a strategy performs in current market conditions.

The role of survivor bias and realistic expectations in backtesting

Backtesting is a valuable tool, but it must be used with a clear understanding of its limitations. Survivor bias is one of the most common sources of error in backtesting, leading to overly optimistic results that don’t accurately reflect real-world market risks. Including delisted stocks in backtests, accounting for potential full losses, and testing in paper trades can help create a more reliable foundation for your trading strategy.

Backtesting should be viewed as an initial guide rather than a definitive prediction. When used carefully and with a balanced view of its benefits and drawbacks, backtesting can help traders refine their strategies and make more informed decisions, even if real-world outcomes may differ from historical performance.

Take advantage of Trade Radar to refine your trading rules, and set up alerts to stay ahead of market changes.