Cfrgtkky

First of all, I realise that my question is very broad and that it may be hard to answer this question because of it.

Do you have any advice on how to approach a 'problem' where you need to make forecasts/predictions for 2000+ different products? In other words, each product requires a different forecast/prediction. I have 2 years of historical data on week level (i.e. demand per week per product).

I need to do this in a short time period: I have about a week to do this, hence I am looking for ways that I can quickly make relatively good prediction models. Creating a model for each product and inspecting its performance closely, one by one, would be too time-consuming.

I thought of segmenting the products based on the variance, so that I can employ simple models for products that have a low variance. While this is probably not ideal, it would be a quick way to narrow down the number of models I need to create.

It would be greatly appreciated if you have any practical advice for me on approaching this problem.

A follow up to @StephanKolassa 's answer:

• I concur with Stephan that ETS() from the forecast package in R is probably your best and fastest choice. If ETS doesn't give good results, you might want also want to use Facebook's Prophet package (Auto.arima is easy to use, but two years of weekly data is bordering not enough data for an ARIMA model in my experience). Personally I have found Prophet to be easier to user when you have promotions and holiday event data available, otherwise ETS() might work better. Your real challenge is more of a coding challenge of how to efficiently iterate it over a large number of time series. You can check this response for more details on how to automate forecast generation.




• In demand forecasting, some form of hierarchical forecasting is frequently performed. I.e you have 2000 products and you need a separate forecast for each separate product, but there are similarities between products that might help with the forecasting. You want to find some way of grouping the product together along a product hierarchy and then use hierarchical forecasting to improve accuracy. Since you are looking for forecasts at the individual product level, look at trying the top-down hierarchical approach.




• Something a little bit more farfetched, but I would like call it out: Amazon and Uber use neural networks for this type of problem, where instead of having a separate forecast for each product/time series, they use one gigantic recurrent neural network to forecast all the time series in bulk. Note that they still end up with individual forecasts for each product (in Uber's case it is traffic/demand per city as opposed to products), they are just using a large model (an LSTM deep learning model) to do it all at once. The idea is similar in spirit to hierarchical forecasting in the sense that the neural network learns from the similarities between the histories of different products to come up with better forecasts. The Uber team has made some of their code available (through the M4 competition Github repositories), however it is C++ code (not exactly the favorite language of the stats crowd). Amazon's approach is not open source and you have to use their paid Amazon Forecast service to do the forecasts.

With regards to your second comment: You need to differentiate between forecasting sales and forecasting demand. Demand is unconstrained, if suddenly an item is popular and your customers want 200 units, it doesn't matter that you have only 50 units on hand, your demand is still going to be 200 units.

In practice it is very difficult to observe demand directly, so we use sales as proxy for demand. This has a problem because it doesn't account for situations where a customer wanted to purchase a product but it was unavailable. To address it, along with the historical sales data, information about inventory levels and stock outs is either directly included in a model or used to preprocess the time series prior to generating a model for forecasting.

Typically an unconstrained forecast is generated first by a forecast engine and then passed on to a planning system which then adds the constrains you mention (i.e demand is 500 units but only 300 units are available) along with other constraints (safety stock, presentation stock, budgetary constraints, plans for promotions or introductions of new products etc...) - however this falls under the general rubric of planning and inventory management, not forecasting per se.

We will only be able to give you very general advice.

• Are there any strong drivers, like promotions or calendar events, or seasonality, trends or lifecycles? If so, include them in your models. For instance, you could regress sales on promotions, then potentially model residuals (using exponential smoothing or ARIMA).


• There are software packages that do a reasonably good job at fitting multiple time series models to a series. You can then simply iterate over your 2000 series, which should not take much more runtime than a cup of coffee. I particularly recommend the ets() function in the forecast package in R. (Less so the auto.arima() function for weekly data.


• At least skim a forecasting textbook, e.g., this one. It uses the forecast package I recommend above.


• What is your final objective? Do you want an unbiased forecast? Then assess point forecasts using the MSE. Will your bonus depend on the MAPE? Then this list of the problems of the MAPE may be helpful. Do you need forecasts to set safety amounts? Then you need quantile forecasts, not mean predictions. (The functions in the forecast package can give you those.)

If you have more specific questions, do post them at CV.

Segmenting based on the variance of the original series makes no sense to me as the best model should be invariant to scale. Consider a series ..model it and then multiply each value in the time series by 1000 .

In terms of mass producing equations that may have both deterministic structure (pulses/level shift/local time trends ) OR either auto-regressive seasonality and arima structure you have to run a computer-based script . Beware of simple auto arima solutions that assume no deterministic structure OR fixed assumptions about same.