Backtest Overfitting: An Interactive Example

What is backtest overfitting? And why should I care?

Graphic on left shows an investment strategy (in blue) making steady profit while the underlying trading instrument (in green) gyrates in price. This website is designed to illustrate that this type of "optimal" investment strategy is all too easy to produce if enough variations are tried, yet it is typically ineffective moving forward, a consequence of backtest overfitting. Graphic on right shows the same investment strategy performing poorly on a different sample of the same trading instrument, which demonstrates that an overfit investment strategy is quite likely to fail in real world investing.

  
By checking this box, you acknowledge that this web site is a demonstration of a mathematical concept known as "backtest overfitting." The software and material on this site should NOT be interpreted as a recommendation to buy or sell any security or securities; as applicable to the specific investment needs of any particular individual or organization; or as applicable to the forecast of future market prices or trends.

Option 1: Repeat the example above

  
Please be patient! The program may take up to two minutes to display the results page after clicking Go! button

Option 2: Try with random parameters

  
Please be patient! The program may take up to two minutes to display the results page after clicking Go! button
     This "quick-start" option uses random (system generated) values for all of the inputs available in Option 3.

Option 3: Try with your own parameters

You can enter values in one or more of the text boxes below. A more detailed explanation of what each box does is provided in the tutorial document below.
Enter values for these parameters and press Go! button
Maximum Holding Period an integer; ranges from 5 to 20. The number of days a stock is held before it is sold if no other exits (e.g., the stop loss) have been triggered.
Maximum Stop Loss an integer %; ranges from 10 to 40. The percentage loss from the initial entry at which point the stock will be sold. Do not enter the '%' sign.
Sample Length an integer; ranges from 1000 to 2000. The number of days simulated. The sample data considered as the daily closing prices.
Standard Deviation an integer; the standard deviation of random numbers used to generate daily price changes; any positive integer could be entered; we recommend 1 or 2
Seed a seed for the pseudorandom numbers used for In Sample Data; it can be any positive integer; enter a nonpositive integer to use the current time as the seed
Please be patient! The program may take up to two minutes to display the results page after clicking Go! button

Option 4: Try with stock market data

You can enter values in one or more of the text boxes below.
Enter values for these parameters and press Go! button
Maximum Holding Period an integer; ranges from 5 to 20. The number of days a stock is held before it is sold if no other exits (e.g., the stop loss) have been triggered.
Maximum Stop Loss an integer %; ranges from 10 to 40. The percentage loss from the initial entry at which point the stock will be sold. Do not enter the '%' sign.
Sample Length an integer; ranges from 1000 to 12000. The number of days simulated. We suggest between 5000-6000 as it makes almost equal length data on IS and OOS sides. The sample data considered as the daily closing prices.
Please be patient! The program may take up to two minutes to display the results page after clicking Go! button

Default values: If you do not enter a value or enter a value that is outside the range mentioned above, a default value will be used. The default values are: maximum holding period = 7; stop loss = 10; sample length = 1000; and standard deviation = 1.

About execution time: The values for maximum holding period, stop loss and sample length significantly affect the number of iterations performed by the program; the larger these values are, the longer the program will run.

A more detailed explanation of options is provided in the tutorial.


A few recent articles explaining the backtest overfitting in the financial press:

Related documents:

Tutorial of the online tool:

Questions or comments:

Please send any comments or questions for this site to: Amir Salehipour.

Credits:

This web page and program were constructed by Stephanie Ger, Amir Salehipour, Alex Sim, John Wu and David H. Bailey, based on an earlier Python program developed by Marcos Lopez De Prado. This program in turn is based on the following research paper:

We gratefully acknowledge the helpful comments and suggestions from colleagues and friends in shaping this web site. In particular, the suggestions from Mr. David Witkin of StatisTrade, Bin Dong of LBNL, and Beytullah Yildiz of Turkey were extensive and very helpful in improving readability of the web pages. Thanks!