Deploying deep learning models on edge CPUs is bottlenecked by computational and memory constraints.
Mixed-precision quantization promises to reduce inference latency while preserving accuracy.
Challenges in Quantization
However, quantization affects different layer types in inconsistent ways, so identifying where accuracy loss is minimized and latency reduction is maximized is critical, as the effect accumulates over a full deployment into substantial savings or unacceptable task degradation.
Nevertheless, widely used metrics fail systematically on modern architectures.
Empirical Study
We present a systematic empirical study of 13 sensitivity metrics for layer-wise INT8 quantization across four distinctly different neural networks, and validate the resulting policies on two ARM64 platforms.
Gradient-based sensitivity methods fail on 4 out of 8 model-hardware configurations and weight-based statistics on 2.
In contrast, the Jensen-Shannon Divergence achieves zero catastrophic failures, reliably isolating the layers that cannot be safely quantized.
Policy Development
A sensitivity metric alone does not define a policy, and the fixed thresholds typically used for that step are fragile over the highly skewed distributions of modern architectures.
We address this with K-Means clustering, achieving near-lossless accuracy and a mean speed-up of $1.81\times$ over the full-precision model.
Layer Exclusion Considerations
Finally, we reveal that excluding from quantization the layers whose speed-up is negligible, regardless of their sensitivity, can be counterproductive, as it induces computational graph fragmentation and disables operator fusion.
Practical Implications
Our results yield concrete allocation policies for practitioners and researchers deploying quantized vision models on heterogeneous edge CPUs, without GPU access or gradient computation.