deidentifying-clinical-text

De-identifying clinical text

openmed.deidentify detects PHI/PII and rewrites the text so it can be shared, stored, or analyzed without exposing patients. It runs fully on-device after a one-time model download — no network calls, no telemetry, no raw PHI leaving the process. This is the single most important OpenMed entry point for privacy work; everything else (policies, audit, multilingual, date-shifting) layers on top of it.

When to use this skill

Reach for deidentify when you need to transform text — replace, mask, remove, hash, or date-shift the identifiers. If you only need to locate PHI spans without changing the text, use extract_pii (see extracting-pii-entities). To restore masked text later, use reidentify (see reidentifying-text).

Quick start

import openmed

note = (
    "Patient John Doe (MRN 1234567) was seen on 2024-03-02 by Dr. Alice Reed. "
    "Contact: [email protected], 617-555-0142."
)

result = openmed.deidentify(
    note,
    method="mask",                 # mask | remove | replace | hash | shift_dates
    confidence_threshold=0.7,      # safety default; raise to reduce false negatives' impact
    policy="hipaa_safe_harbor",    # optional bundled profile (see below)
)

print(result.deidentified_text)
# Patient [NAME] (MRN [ID_NUM]) was seen on [DATE] by Dr. [NAME]. ...

for e in result.pii_entities:
    # NEVER log e.text / e.original_text — those are raw PHI. Use offsets + label.
    print(e.canonical_label, e.start, e.end, round(e.confidence, 3))

deidentify returns a DeidentificationResult with these fields (note the exact names):

Field	What it holds
.deidentified_text	the rewritten, PHI-safe string (your output)
.pii_entities	list[PIIEntity] — each has start, end, canonical_label, confidence, action, surrogate; original_text/text hold raw PHI
.mapping	redacted→original dict, only when keep_mapping=True (secret)
.method	the method actually applied
.metadata	run metadata (model, policy, counts)

The five methods

method=	Effect	Reversible?	Use when
"mask"	John Doe → [NAME]	with keep_mapping=True	default; clear that redaction happened
"remove"	deletes the span entirely	no	minimal-footprint output
"replace"	type-matched fake value (John Doe→Mark Lee)	with keep_mapping=True	keep notes readable/parseable (see generating-synthetic-surrogates)
"hash"	stable hash per value, links repeats	no (one-way)	cohort linkage without revealing identity
"shift_dates"	moves dates, preserves intervals	n/a	research needing temporal structure (see shifting-clinical-dates)

Workflow

1. Pick a method and a policy. Start from a bundled policy= profile (hipaa_safe_harbor, gdpr_pseudonymization, research_limited_dataset, …) so per-label actions are set for you. See configuring-privacy-policies.
2. Set confidence_threshold deliberately. Default is 0.7. For de-id, prefer over-redaction: a missed identifier is a breach, an over-redacted token is just noise. The bundled safety sweep catches structured IDs (SSN, MRN-like, emails) even below threshold.
3. Run deidentify. Inspect result.pii_entities by offset and label, not raw text, to confirm coverage.
4. For stable surrogates, pass consistent=True, seed=<int> so the same input maps to the same fake value every run (reproducible pipelines).
5. For reversibility, pass keep_mapping=True and store result.mapping in a secured vault — never alongside the de-identified output.
6. Verify, don't assume. Check residual risk with audit=True (auditing-deidentification-runs) and the 18-identifier checklist (auditing-safe-harbor-checklist).

Consistent surrogates and reversibility

# Same fake identity for every mention of the same person, reproducibly:
r = openmed.deidentify(note, method="replace", consistent=True, seed=42)

# Reversible de-id (keep the mapping secret and separate from output):
r = openmed.deidentify(note, method="mask", keep_mapping=True)
restored = openmed.reidentify(r.deidentified_text, r.mapping)
assert restored == note

Hand-off to / from OpenMed

• Detect only: openmed.extract_pii(text) → PredictionResult with .entities (spans, no rewrite). Use it to preview coverage first.
• Restore: openmed.reidentify(deidentified_text, mapping) — requires keep_mapping=True at de-id time and proper authorization.
• Policies: configuring-privacy-policies to choose/customize a policy=.
• Audit: deidentify(..., audit=True) → AuditReport with offsets, hashes, detector provenance, and residual-risk — never plaintext.
• Other surfaces (same engine): MCP tool openmed_deidentify; REST POST /pii/deidentify. There is no CLI de-id command.

Edge cases & gotchas

• Attribute names. It is result.deidentified_text and result.pii_entities — not .text/.entities. (extract_pii returns a PredictionResult whose spans are at .entities.)
• Raw PHI never leaves the span objects. PIIEntity.text and .original_text contain real identifiers. Do not print, log, or cache them. Audit and logs use offsets, canonical_label, and hashes only.
• Threshold is a safety dial, not an accuracy dial. Lowering it redacts more; in de-id, false positives are cheap and false negatives are breaches.
• shift_dates is for dates only; combine with keep_year/date_shift_days (see shifting-clinical-dates). It does not touch names or IDs.
• keep_mapping output is sensitive as PHI. The mapping re-identifies everyone — store it encrypted, access-controlled, and apart from the output.
• Multilingual: pass lang= (and locale= for surrogates) for non-English notes; see deidentifying-multilingual-text. Do not run English models on other languages.
• De-id is verified, not assumed. Gate releases on leakage/residual-risk, not F1 alone.

Standards & references

• HIPAA De-identification, 45 CFR 164.514(b) — Safe Harbor & Expert Determination: https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html
• HHS Safe Harbor 18-identifier guidance: see auditing-safe-harbor-checklist.
• OpenMed source: openmed/core/pii.py (deidentify, extract_pii, reidentify, DeidentificationResult, PIIEntity).