auditing-subgroup-fairness

Auditing Subgroup Fairness

An aggregate pass can hide a group the model fails. For de-identification that failure has a name: under-protection — PHI that leaks more often for one demographic group than another. openmed.eval.fairness_report slices leakage and recall by gold-span group so disparities surface before deployment, not after a breach.

When to use this skill

• You want per-subgroup recall and leakage for a de-id or NER model.
• You suspect (or must rule out) that one group is under-protected.
• You need disparity numbers for a clinical AI governance review.
• You need to document which subgroups you couldn't evaluate (the data gap).

What it measures

For each surrogate group fairness_report returns:

• leakage_rate — fraction of that group's gold PHI characters left exposed (the de-id harm metric).
• recall — fraction of that group's gold spans detected.
• leakage_disparity — max - min leakage across groups (the gap to close).
• worst_group / worst_group_leakage — the most-failed group.

Group membership comes from a group tag in each gold span's metadata (keys group, demographic_group, or surrogate_group); ungrouped spans fall into unspecified.

Quick start

from openmed.eval import fairness_report

# Gold fixtures must tag spans with a surrogate group, e.g.
#   {"start": 4, "end": 12, "label": "PERSON", "metadata": {"group": "female"}}
fair = fairness_report(
    "OpenMed/Privacy-PII-Detection",
    "golden",                 # named suite, or pass a list of fixtures
    device="cpu",
)

print("leakage disparity:", fair.leakage_disparity)
print("worst group      :", fair.worst_group, fair.worst_group_leakage)
for group, m in sorted(fair.per_group.items()):
    print(f"  {group:14s} recall={m.recall:.3f}  leakage={m.leakage_rate:.4f}")

# Under-protection alarm: any group leaking more than the rest.
LEAKAGE_GAP_LIMIT = 0.0       # leakage-first: ideally zero leakage everywhere
assert fair.leakage_disparity <= LEAKAGE_GAP_LIMIT or fair.worst_group_leakage == 0

FairnessReport.to_dict() is JSON-ready and PHI-free — drop it straight into a model card.

Workflow

1. Tag the gold corpus by group. Add a synthetic group to each PHI span's metadata (sex, age band, race/ethnicity surrogate). Use synthetic surrogates, not real protected attributes (see building-gold-corpus).
2. Run fairness_report on the model + suite.
3. Read leakage first, recall second. For de-id, a group with higher leakage is under-protected — that is the headline finding.
4. Compute the disparity (leakage_disparity) and locate worst_group. Equalized-odds framing: equal true-positive (recall) and equal leakage across groups.
5. Document the gap. If race/ethnicity surrogates are absent, report that the audit could not cover them — most clinical NLP studies omit race entirely, so silence is the default failure mode, not equity.
6. Feed it forward. Put per-group numbers and the gap into the model card and the governance review.

Hand-off to / from OpenMed

• From building-gold-corpus: supplies group-tagged synthetic fixtures.
• From evaluating-with-leakage-gates: an aggregate RELEASABLE decision should be paired with this audit — overall pass, subgroup fail is exactly the trap this catches.
• To authoring-model-cards: FairnessReport.to_dict() fills the quantitative-analysis / subgroup section.
• Pairs with benchmarking-clinical-ner: same run, different slice (label vs group).

Edge cases & gotchas

• Under-protection is the de-id harm; lead with leakage. A group with equal recall but higher leakage is still failed.
• The race documentation gap is the norm. Most clinical NLP corpora don't record race/ethnicity, so most fairness audits silently can't measure it. Report the absence explicitly — don't let missing data read as parity.
• unspecified is not a real group. A pile of spans in unspecified means your gold isn't tagged; fix the corpus before trusting the disparity.
• Small groups give noisy rates. Report span counts (span_count, total_chars) alongside rates; a 1-of-2 leak isn't a 50% population rate.
• Synthetic surrogates only. Never store real protected attributes in eval fixtures; use fabricated group labels for slicing.
• Disparity ≈ 0 with high leakage everywhere is not "fair". Equal failure is still failure — check absolute leakage, not just the gap.

Standards & references

• STANDING Together — reporting standards for health-dataset diversity & documentation: https://www.datadiversity.org/
• Hardt, Price, Srebro, Equality of Opportunity in Supervised Learning (equalized odds): https://arxiv.org/abs/1610.02413
• Chen et al., Ethical ML in Health Care (subgroup performance in clinical NLP): https://doi.org/10.1146/annurev-biodatasci-092820-114757
• OpenMed source of truth: openmed/eval/fairness.py (fairness_report, FairnessReport, FairnessGroupMetrics).