📝 SummarXiv

Abstract

Forecast stability, that is, the consistency of predictions over time, is essential in business settings where sudden shifts in forecasts can disrupt planning and erode trust in predictive systems. Despite its importance, stability is often overlooked in favor of accuracy. In this study, we evaluate the stability of point and probabilistic forecasts across several retraining scenarios using two large retail demand datasets (M5 and VN1) and ten different global forecasting models. To analyze stability in the probabilistic setting, we propose a new model-agnostic, distribution-free, and scale-free metric that measures probabilistic instability: the Scaled Multi-Quantile Change (SMQC). Furthermore, we also evaluate the effects of retraining on various ensemble configurations based on forecast pooling. The results show that, compared to continuous retraining, less frequent retraining not only preserves but often improves forecast stability, challenging the need for continuous retraining. The study promotes a shift toward stability-aware forecasting practices, proposing a new tool to effectively evaluate forecast stability in probabilistic settings, and offering practical guidelines for building more robust prediction systems.