Backtesting

Backtesting refers to a strategy used to evaluate the quality of a forecasting model. In this process, the model's predictions for a past period (e.g., the previous year) are examined and compared to the known, actual values that occurred during that period.

Essentially, backtesting follows the following procedure:

1. The given data history is divided into a training period and a test period.
2. The observations from the training period are used to optimize the model parameters (training the forecasting model).
3. The trained model is used to generate forecasts for the test period.
4. The generated forecasts are evaluated using appropriate performance metrics, such as MAPE, MAE, or MSE.

To generate forecasts for the test period, different strategies are available. The most relevant one is the rolling 1-step forecast, which will be illustrated with the following example:

1. The available data history from January 2010 to December 2016 is divided into a training period (January 2010 – December 2015) and a test period (January 2016 – December 2016).
2. The model is trained on the training period, based on data up to December 2015, and a 1-step forecast is made for January 2016.
3. Now, an additional data point, the actual value of the time series for January 2016, is added to the training period. The model is retrained, based on the data up to the end of January 2016, and a 1-step forecast is made for February 2016.
4. This process is repeated as described above, until a 1-step forecast has been generated for each month in 2016.
5. The forecasts generated in this way for 2016 are called rolling 1-step forecasts. A comparison of these forecasts with the actual values that occurred in 2016 provides an evaluation of the model in terms of its 1-step forecasting accuracy.

The above method can also be used to evaluate higher forecasting steps.