Fairness audits in production ML typically occur once, at deployment, on a single domain.
Both fail in practice: fairness can shift after retraining or a changing user base, and interventions validated on one dataset are rarely tested across the heterogeneous domains an organization deploys.
We present FAPE (Fairness Auditing for Production Environments)
a four-stage framework evaluating a single post-processing intervention, Fairlearn's ThresholdOptimizer, across eight domain evaluations:
- criminal justice
- income prediction
- legal admissions
- credit lending
- agricultural lending
- a multi-domain benchmark corpus
- healthcare
- education
Each is scored on demographic parity and equalized odds difference, plus disparate impact ratio and accuracy cost where computable.
Intervention effectiveness tracks baseline disparity magnitude:
across model-domain pairs the constraint improved disparity in 9 of 14 high-disparity cases and worsened it in 3 of 4 near-fair ones.
Each of the five high-disparity exceptions reverses under one of two measurement checks, a minimum group size or thresholds fit on held-out data.
A CUSUM monitor started at deployment
tested on a simulated shift, separates constrained models that never met a 0.1 parity convention from those that met it and later regressed.
A single deployment-time audit is therefore an unreliable guide, which argues for baseline-disparity screening and continuous monitoring.