Backtesting Equity Strategies

A backtest is not evidence that a strategy works. It is evidence that a strategy would have worked on a specific historical dataset under specific assumptions — assumptions that are almost always violated in subtle ways that systematically overstate performance. The graveyard of failed algorithmic trading strategies is populated overwhelmingly by strategies that produced spectacular backtests and mediocre or catastrophic live results. The difference between the backtest and live performance is almost never bad luck; it is the cumulative effect of the biases, data errors, and overfitting problems documented in this course. Understanding these is not optional background knowledge. It is the minimum requirement for treating any backtest as analytically credible.

1. Survivorship Bias: The Most Expensive Data Error

Survivorship bias is the systematic exclusion from a historical dataset of securities that ceased to exist during the analysis period — companies that went bankrupt, were delisted, were acquired, or were otherwise removed from the market. A dataset that includes only currently listed companies is survivorship-biased because it excludes all the failures that would have been included in any live trading strategy at the time they were active.

The magnitude of survivorship bias in equity markets is substantial. Academic research estimates that survivorship bias inflates annualised strategy returns by 1.5–4% for long-only equity strategies, with higher distortion for strategies that select from smaller-cap universes where delisting rates are higher. For a strategy that generates a 10% annualised return in backtest, survivorship bias alone could account for 2–3 percentage points of that return — meaning the actual edge is 7–8% or potentially even less after accounting for other biases.

Correcting survivorship bias requires a point-in-time database: a historical dataset that includes all securities that were actively trading at each point in history, not only those that survived to be included in today’s index. Professional-grade survivorship-free databases (Compustat, CRSP, Bloomberg) are expensive and typically beyond retail trader budgets. For retail traders: acknowledge the bias explicitly, be conservative about the magnitude of edge implied by any backtest, and weight live paper trading results heavily when evaluating strategy viability. Alpaca’s API provides access to historical data for paper trading validation that, while not fully survivorship-free, captures a broader universe than manually curated equity lists.

2. Look-Ahead Bias: When the Backtest Cheats

Look-ahead bias occurs when a backtest uses information that would not have been available at the time the simulated decision was made. It is the most insidious form of data contamination because it is often introduced unintentionally through seemingly innocent data handling choices.

Common look-ahead bias sources in equity backtesting:

• Earnings data as of reporting date vs disclosure date. A company’s Q3 earnings are produced for the quarter ending September 30, but they are not reported to the public until the earnings announcement in late October. Using quarterly earnings data as if it were known on October 1 (the period end date) introduces look-ahead bias of 3–6 weeks.
• Index composition as of today, applied to historical periods. A strategy that selects from “current S&P 500 constituents” for backtesting periods in 2015 is using an index composition that includes companies added since 2015 and excludes companies removed since 2015. The historical selection would have been very different.
• End-of-day price data with intraday order execution. If a strategy generates signals using the day’s closing price and is backtested as executing at that same closing price, it has implicitly looked ahead to know the closing price before it was set. Realistic execution requires the closing price to be unknown at signal generation or the execution to occur the following day.
• Adjusted price data applied without care. Historical equity prices are adjusted for splits and dividends, which is appropriate for technical analysis continuity. However, applying today’s adjusted prices retroactively changes what the price “was” at historical points in ways that can distort signal generation if not handled carefully.

Eliminating look-ahead bias requires imposing a strict temporal boundary on every piece of information used in a backtest: each decision can only access data that would have been publicly known and practically accessible at the moment the decision is simulated to have occurred.

3. Overfitting: The Curve-Fitting Trap

Overfitting — the process of optimising strategy parameters to fit historical data so precisely that they capture noise rather than genuine predictive structure — is responsible for the majority of the gap between backtest performance and live performance. A strategy with 15 adjustable parameters can almost always be optimised to produce excellent historical returns on any dataset; this does not mean it has discovered genuine market inefficiency. It means it has discovered the specific pattern of randomness in that particular historical sample.

The quantitative test for overfitting: how many independent tests have been run? If a researcher tests 100 different parameter combinations, finds 5 that produce Sharpe ratios above 1.5, and reports the best one, the probability of that result arising from pure chance is substantial. The Bonferroni correction (dividing the significance threshold by the number of tests) provides a rough adjustment: if 100 combinations were tested at a 5% significance level, the corrected threshold is 0.05 ÷ 100 = 0.0005, meaning the backtest performance would need to be extraordinarily good to achieve statistical significance after accounting for the testing multiplicity.

Practical safeguards against overfitting: (1) use as few parameters as possible — strategies with more parameters have more degrees of freedom to fit noise; (2) test only parameters that have economic or behavioural justification, not arbitrary combinations; (3) use out-of-sample validation (described below); (4) prefer robust strategies that work across a range of parameter settings rather than those that work only in a narrow parameter band. A strategy where slightly changing the moving average from 20 to 21 periods dramatically changes results is almost certainly overfit; a genuine edge should be relatively insensitive to small parameter changes.

4. Out-of-Sample Validation and Walk-Forward Analysis

Out-of-sample (OOS) validation is the fundamental safeguard against overfitting. The principle: divide the historical dataset into two periods. Develop and optimise the strategy on the in-sample period (typically 60–70% of the data), then evaluate it without further modification on the out-of-sample period (the remaining 30–40%). The OOS results are the genuine test of whether the strategy has captured genuine predictive structure or merely fitted historical noise.

A strategy that performs well in-sample but poorly out-of-sample is overfit. A strategy that performs reasonably well in both periods — with OOS performance that is somewhat lower than in-sample (expected due to the absence of parameter tuning) but in the same direction — has passed the basic validation test. The OOS period should be kept genuinely unseen during the development process; examining OOS results and then adjusting parameters based on them converts the OOS period into an additional in-sample period, eliminating its validity as an independent test.

Walk-forward analysis extends this concept to a rolling validation framework. The dataset is divided into many overlapping windows, each with its own in-sample and OOS period. Parameters are re-optimised on each in-sample window, then evaluated on the subsequent OOS window. The aggregated OOS results across all windows provide a more robust assessment of strategy performance than a single train/test split. Walk-forward analysis also reveals how strategy performance degrades as market regimes change — identifying periods where the strategy is structurally mismatched to the prevailing environment.

5. Transaction Costs, Slippage, and Realistic Execution

A backtest that does not include realistic transaction costs is not a backtest — it is a theoretical exercise. Even in the era of zero-commission retail trading, transaction costs remain real and material:

• Bid-ask spread. Every market order crosses the spread — buying at the ask and selling at the bid. For a liquid large-cap stock with a $0.05 spread at a price of $100, the round-trip cost is 0.05%. Over hundreds of trades, this accumulates to meaningful performance drag. For smaller-cap stocks with wider spreads, the impact is substantially larger.
• Market impact / slippage. Any order large enough to move the market fills at prices progressively worse than the first available price. For retail-sized positions in liquid large-caps, market impact is typically negligible. For positions exceeding 0.1–0.5% of average daily volume, slippage becomes material.
• Borrowing costs for short positions. Short positions require securities borrowing, which costs 0.25–300%+ annually depending on stock availability. Any strategy that includes short positions must model borrow costs explicitly. Our short sell calculator quantifies these costs for individual positions.

The practical minimum transaction cost assumption for a retail equity backtest: add 0.1–0.2% per trade as a catch-all for spread, slippage, and market impact, in addition to explicit commissions. For strategies trading illiquid names, double or triple this estimate. If a strategy’s edge evaporates when realistic transaction costs are applied, the strategy has no genuine edge at retail execution quality.

6. Key Backtesting Standards Checklist