Context
Non-determinism in deep learning algorithm design and implementation leads to performance variation, meaning model performance is not a single value, but rather a distribution. These model performance distributions are underexplored despite their impact on robustness.
Objective
We investigate the robustness of deep learning performance to sources of non-determinism, specifically focusing on how performance distributions differ across various architectures and tasks.
Method
- We conducted 186 experiments on image classification and time series forecasting architectures.
- Each experiment was run 100 times with different random seeds to generate performance distributions, resulting in 18,600 runs.
- Robustness was quantified using metrics for spread, symmetry, and tail risk.
Results
- Performance distributions are frequently non-Gaussian, particularly in time series forecasting.
- Model size does not systematically affect robustness -- larger image classification models show fewer outliers but not lower spread, while smaller time series models show lower spread but more extreme underperformers.
- Training duration does not scale linearly; early stopping effectively balances performance and robustness.
- Mean performance does not predict robustness - time series forecasting shows moderate correlation while image classification shows none.
- Time series models produce nearly three times more underperforming outliers than image classification models, indicating substantially higher tail risk.
Conclusions
Tail risk poses serious concerns for Trustworthy AI in high-stakes applications. Models performing well on average may exhibit long tails and extreme outliers revealed only through distributional analysis. Mean performance alone should not guide model selection; assessment of spread, symmetry, and tail risk is essential for reliable model assessment where consistent performance is critical.