← dawsonwilhelmi.com

Forecast Method Comparison

Which forecasting method should a buyer trust for this series? Up to eleven methods are re-fitted and tested period by period on data they have not seen, then scored on six error metrics. Pick a series or upload your own.

Seasonality
Your data
CSV or Excel (.xlsx): a column of numbers, plus a column of dates or periods if you have one. Most date formats and any time unit (hours to years) work. Files are read in your browser and never uploaded.

Actuals, backtest and 12-month forecast

Dots inside the shaded window are each method's 1-step-ahead forecasts from the backtest. Lines after the last actual are the forecasts from the model fitted on all data.

Error metrics, all methods

Ranked by MASE (lowest first). ★ marks the best value in each column. Click a column to sort. Hover a header for its definition.

Error grows with horizon

MAE at each step ahead. A method that wins at 1 step can lose at 6; match the horizon to your purchasing lead time.

How the test works

Rolling origin. For each of the last N periods, every method is fitted only on data before that point and forecasts the next h periods. Forecasts are compared with what actually happened. This mimics real use and avoids judging a model on data it was fitted to (Tashman, 2000; Hyndman & Athanasopoulos, 2021, §5.10).

Fitting. Holt-Winters fits its three weights and its starting level, trend and seasonal pattern together by minimising one-step squared error (Nelder–Mead search). Seasonal ARIMA is fitted by conditional least squares. The simpler smoothing methods pick their weights from a grid (0.05 to 0.95); full fitting changed their accuracy by under 1% on these series. Every number was re-computed by an independent Python implementation and matches; accuracy is in line with the statsmodels library's maximum-likelihood fits.

No method is favoured. The backtest on your series picks the winner. Two combinations (simple averages of other methods) are entered as candidates because averaging often reduces error (Bates & Granger, 1969; Makridakis et al., 2020), but they only win when the data says so. The page also checks whether the same method wins with 12, 24 and 36-period windows.

Ranking. MASE decides the winner because it compares methods on a common, scale-free basis and does not break when values approach zero (Hyndman & Koehler, 2006).

What the metrics mean

  • MAE: average miss in the series' units. Easy to explain to a supplier or a CFO.
  • MSE / RMSE: square the misses first, so one big miss counts heavily. Use when large errors are costly (stockouts, budget overruns).
  • MAPE: average miss as a % of actual. Intuitive, but undefined at zero, explodes near zero and penalises over-forecasts more than under-forecasts.
  • sMAPE: divides by the average of actual and forecast; bounded at 200%, but still unstable near zero.
  • MASE: MAE divided by the in-sample MAE of a seasonal-naive forecast. Below 1 means the method beats that simple benchmark.
  • Bias (ME): average of actual minus forecast. Persistently positive = under-forecasting.

References

  1. Hyndman, R. J., & Athanasopoulos, G. (2021). Forecasting: Principles and Practice (3rd ed.). OTexts. otexts.com/fpp3
  2. Hyndman, R. J., & Koehler, A. B. (2006). Another look at measures of forecast accuracy. International Journal of Forecasting, 22(4), 679–688. doi:10.1016/j.ijforecast.2006.03.001
  3. Tashman, L. J. (2000). Out-of-sample tests of forecasting accuracy: An analysis and review. International Journal of Forecasting, 16(4), 437–450. doi:10.1016/S0169-2070(00)00065-0
  4. Holt, C. C. (2004). Forecasting seasonals and trends by exponentially weighted moving averages. International Journal of Forecasting, 20(1), 5–10. (Reprint of 1957 ONR memorandum.) doi:10.1016/j.ijforecast.2003.09.015
  5. Winters, P. R. (1960). Forecasting sales by exponentially weighted moving averages. Management Science, 6(3), 324–342. doi:10.1287/mnsc.6.3.324
  6. Box, G. E. P., & Jenkins, G. M. (1970). Time Series Analysis: Forecasting and Control. Holden-Day.
  7. Bates, J. M., & Granger, C. W. J. (1969). The combination of forecasts. Operational Research Quarterly, 20(4), 451–468. doi:10.2307/3008764
  8. Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2020). The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1), 54–74. doi:10.1016/j.ijforecast.2019.04.014
  9. Nelder, J. A., & Mead, R. (1965). A simplex method for function minimization. The Computer Journal, 7(4), 308–313. doi:10.1093/comjnl/7.4.308
  10. Gardner, E. S., Jr., & McKenzie, E. (1985). Forecasting trends in time series. Management Science, 31(10), 1237–1246. doi:10.1287/mnsc.31.10.1237

Data

Data snapshot: FRED pull of Nov 2026; latest observations 2026-09, 2026-10. Monthly series, Jan 2018 onward.