Learn · Bias & Fairness
Bias & fairness in ML
The same model scores two demographic groups. Their score distributions are not identical, so a single decision threshold lands differently on each group. Move each group's threshold, or force one shared cut-off, and watch selection rate, recall and false-positive rate diverge. The distributions are illustrative — the fairness maths is real.
Group score distributions
Illustrative distributions. Group B's scores sit lower overall — a common real-world artefact of biased features or historical data. Anything at or above the threshold is predicted positive.
A single cut-off looks "equal" but treats unequal distributions unequally — that is where fairness metrics disagree.
Group-level rates
| Metric | Group A | Group B |
|---|---|---|
| Selection rate (predicted positive) | — | — |
| True positive rate (recall) | — | — |
| False positive rate | — | — |
Fairness metrics
Demographic parity asks for equal selection rates → the disparate impact ratio should be ≥ 0.8 (the US "four-fifths rule"). Equalised odds asks for equal TPR and equal FPR across groups → both gaps should be ≈ 0.
Why you can't satisfy everything
Demographic parity vs equalised odds
Parity fixes who gets selected (equal selection rates). Equalised odds fixes how accurately each group is treated (equal recall and equal false-alarm rate). Try to equalise selection rates with the sliders and the TPR/FPR gaps usually grow — and vice versa.
The impossibility result
When base rates differ between groups, it is mathematically impossible to satisfy demographic parity, equalised odds and calibration all at once (Kleinberg et al., 2016; Chouldechova, 2017). No threshold choice makes every fairness definition true simultaneously.
Fairness is context-dependent
Which definition matters depends on the decision. Lending, hiring and healthcare each weigh a missed opportunity against a wrongful selection differently. That is a values judgement, not a metric a model can optimise on its own.
Human judgement, with PRISM
bigspark's PRISM responsible-AI framework surfaces these trade-offs explicitly, records the chosen fairness stance and keeps humans accountable for the decision rather than hiding it inside a threshold.
The honest caveat: real score distributions are never two tidy bell curves, group labels are themselves contested, and "fair" cannot be reduced to a single number. This demo shows the tension, not the answer — picking a fairness definition is a responsible-AI decision we work through with stakeholders. See the Fraud threshold demo for the single-group version, or the Learn shelf for the basics.