Conference Agenda

Forecasts accuracy is definitively a crucial topic for industrial companies. Indeed, its impacts are huge especially for finance and production departments. It can occur high costs for the company if the forecasts are not accurate, due to stock-outs or excesses of inventory, for example.

Therefore, the purpose of this study is to optimize accessories forecasting for a medium-sized Swiss enterprise. To do that, different forecasting techniques are tested, and a comparison is made between statistical methods and machine learning algorithms. The results have been adjusted thanks to the key account managers (KAM) expertise.

In this paper, a comparison between exponential smoothing, seasonal autoregressive integrated moving average (SARIMA), SARIMAX (SARIMA with exogenous regressors) and Machine Learning algorithms such as k-nearest neighbors (k-NN), LASSO regression, linear regression and even random forest is presented.

To compare these different methods, two measures of statistical dispersion are computed: mean absolute error (MAE) and root mean squared error (RMSE). These results have been standardized for a better comparison. It comes out that for our dataset SARIMAX (with the KAM’s expertise as exogenous variable) gives better results that all the machine learning algorithms tested.

In most fields of business, accurate predictions are the basis for future planning, whereby combining predictions of different forecasters and/or forecasting models usually generates more accurate predictions than any individual forecaster or model alone.

While simply averaging forecasts using equal weights (EW) has proven to be a robust strategy in practice, an alternative approach applied in several recent papers is to learn so-called optimal weights (OW), that minimize the mean squared error (MSE) on past (training) data, and shrinking these weights towards EW. This strategy aims to learn structures from training data while mitigating overfitting and to avoid high prediction errors with novel forecasts.

However, estimating OW and shrinkage levels on training samples is still subject to uncertainty and can be highly unstable especially for smaller datasets and larger sets of forecasters. This turns out to be a key problem of such approaches, which usually do not systematically beat EW approaches in practical settings.

We introduce a new procedure to obtain more stable weighting schemes. The procedure learns OW on randomly drawn subsets of the training data and determines the optimal shrinkage towards EW on the respective omitted observations, resulting in varying shrunk weight vectors. Subsequently, these vectors are averaged so that the final weight each forecaster receives corresponds to the average (shrunk) optimal weight over all subsets and is asymptotically less extreme.

We evaluate the procedure on synthetic datasets, where it shows benefits compared to EW as well as OW approaches in terms of the out-of-sample MSE.

While forecast combination generally improves forecast accuracy, a persisting research question is how to weight individual forecasters. One natural approach is to determine weights that minimize the mean squared error (MSE) on past error observations. Such weights can be computed from the past errors’ sample covariance matrix, which is, however, an unstable estimator of the true covariance matrix, i.e., its variance is high unless there are many error samples – often not given in forecast situations. Hence, the estimation error associated with a sample covariance matrix often leads to overfitted weights and decreased accuracy of novel forecasts derived with such weights, quickly overwhelming potential accuracy gains due to learning weights. A remedy of this overfitting problem is the (Steinian) shrinkage of the sample covariance matrix to a rather inflexible target like the Identity matrix. This decreases the matrix's variance, albeit at the cost of introducing some bias. To apply Steinian shrinkage, two decisions must be made upfront. First, a suitable structure of the target needs to be set. Second, a shrinkage level must be determined that solves the bias—variance trade-off associated with the sample covariance and target matrix. While Steinian shrinkage exhibits promising outcomes in synthetic data experiments, we are not aware of data-driven approaches to select the target and determine the shrinkage level to be used in forecast combination.

In this paper, we propose machine learning-based tuning procedures for selecting targets and tuning shrinkage levels, where experimental analyses show promising results in terms of MSE reductions on unseen data.