Forecasting regional poverty: new learnings
Jeremy Kwok
5th June 2025
Predicting monthly claimant counts: univariate vs. panel models
In the wake of COVID-19, accurately forecasting unemployment-related indicators at a local level has become more critical than ever. This blog summarises recent work I’ve undertaken to investigate whether univariate autoregressive (AR) models or panel AR models with fixed effects better predict monthly claimant count rates across England’s ~32,000 Lower-layer Super Output Areas (LSOAs) – a key input for metrics like Autonomy’s Annual Deprivation Index (ADI).
A univariate AR model builds a forecast for each area using only that area’s past data – like asking each neighbourhood to predict its own future based on its own history.
In contrast, a panel AR model with fixed effects combines data from all areas to find shared patterns, while still allowing each one to have its own average level – like assuming every area follows a similar trend but starts from a different baseline.
Using monthly claimant data across a decade (2011 to 2022), we fit AR models of orders 1 to 6 separately for each LSOA and compared them with pooled panel AR models that assume a shared dynamic structure, but allow each area its own baseline (fixed effect).
An AR(p) model uses the last p months of data to predict the next value – so AR(1) looks back one month, while AR(6) uses up to six months of history. Testing different lag lengths matters because it helps capture how persistent or short-lived changes in local unemployment are over time. Forecasts are generated recursively for the post-pandemic period (Jan 2021–Dec 2022) and evaluated on standard metrics: Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Mean Error (bias).
Key findings
Across all lag orders, univariate AR models consistently outperformed panel models. The best-performing model, AR(4), reduced forecast error by around 15% compared to its panel counterpart and was virtually unbiased. In contrast, panel models exhibited slightly higher negative bias – systematically under-predicting claimant rates, particularly during the recovery period.
The study also explored the impact of lag length. For univariate models, forecast accuracy improved up to lag 4, beyond which gains plateaued or reversed. For panel models, increasing the lag to 6 improved performance and reduced bias but never quite matched the accuracy of univariate models.
What does this mean for policymakers and data practitioners?
Firstly, when forecasting local economic distress, this brief study shows that it’s crucial to capture idiosyncratic patterns at the neighbourhood level. Panel models, by design, pull forecasts towards a shared trajectory, which can dilute signals from high-risk areas – potentially leading to under-preparedness. In contrast, univariate models “listen” to each area’s own past, enabling better tracking of atypical local trends.
Secondly, while panel models are computationally efficient and robust to outliers, their forecast bias suggests caution. In contexts like the ADI, where claimant forecasts inform real-world resource allocation, even small under-predictions can mean under-serving the areas that need support most.
That said, the panel approach isn’t without merit. It may still offer advantages when data per unit is sparse, or when consistency across forecasts is prioritised over accuracy in any one area. It is also far more efficient to estimate a single panel model than to fit over 30,000 separate univariate models—making it a practical choice when computing time or resources are limited.
In conclusion, this head-to-head comparison challenges the assumption that pooling is always beneficial. When modelling highly localised, volatile indicators like claimant counts, letting each area speak for itself via univariate AR models yields superior forecast accuracy – a lesson with practical implications for local economic monitoring, deprivation indexing, and targeted policymaking.
Jeremy works within the Autonomy Data Unit (ADU) as a doctoral researcher (co-funded by Autonomy) focusing on nowcasting the UK economy.