Distribution shifts, including covariate shifts, are ubiquitous in real-world applications of machine learning. Empirical evidence suggests that, across a variety of models with and without downstream adaptation, ensembles tend to be more robust to covariate shift than single models.
Theoretical extension
We extend the theory of covariate shift severity from single models to ensembles, providing theoretical justification for the empirical trends.
New discrepancy measures
We introduce new discrepancy measures of covariate shift severity to control the risk of partition-based kernel estimators.
Risk convergence under covariate shift
We then show that with sufficient ensemble size, these models can achieve minimax-optimal risk convergence rates under covariate shift.
Our results suggest that modelers can expect to need larger ensemble sizes for more severe covariate shifts.