跳到正文
FunCoding

deidentify-a-dataset

De-identify selected free-text columns in a local CSV, JSONL, or Parquet dataset with OpenMed and produce a separate redacted dataset plus a PHI-free aggregate summary. Use when an agent must prepare a clinical dataset for analysis or sharing without overwriting the source or exposing cell values in logs.

Star
5,664
分类
文档与办公
许可证
Apache-2.0
最近提交
2026年10月13日

skills/deidentify-a-dataset/SKILL.md

在 GitHub 上查看

De-identify a dataset

Keep the source local, name the free-text columns explicitly, and write to a different destination. Never infer columns or print source and redacted cell values.

Procedure

  1. Confirm that the input is CSV, JSONL/NDJSON, or Parquet.
  2. Confirm which columns contain free text. Do not scan or log values to guess.
  3. Choose a policy and language. Prefer strict_no_leak when recall is the governing safety requirement.
  4. Write to a new path; never overwrite the input.
  5. Inspect only result.summary, which contains aggregate counts and rates.
  6. Validate recall and residual leakage on representative synthetic or approved evaluation fixtures before releasing the output.

Runnable synthetic example

Install the model runtime first with python -m pip install "openmed[hf]".

import csv
from pathlib import Path

from openmed import redact_dataset

source = Path("synthetic-notes.csv")
destination = Path("synthetic-notes.redacted.csv")

with source.open("w", newline="", encoding="utf-8") as handle:
    writer = csv.DictWriter(handle, fieldnames=["record_id", "note"])
    writer.writeheader()
    writer.writerows(
        [
            {
                "record_id": "SYNTH-001",
                "note": (
                    "Taylor Example called 212-555-0198 about a "
                    "metformin refill."
                ),
            },
            {
                "record_id": "SYNTH-002",
                "note": (
                    "Send the synthetic follow-up to "
                    "[email protected]."
                ),
            },
        ]
    )

result = redact_dataset(
    source,
    text_columns=["note"],
    output_path=destination,
    policy="strict_no_leak",
    lang="en",
)

print(result.output_path)
print(result.summary.to_dict())  # Aggregate counts only; no cell contents.

Use the equivalent CLI for an existing dataset:

openmed redact-dataset notes.csv \
  --text-columns note,comment \
  --policy strict_no_leak \
  --output notes.redacted.csv

Safety checks

  • Keep model inference and files on infrastructure the user controls.
  • Do not print input rows, detected entity surfaces, reversible mappings, or exception payloads that may contain source text.
  • Keep source and output paths separate and access-controlled.
  • Treat the aggregate summary as evidence, not as proof of compliance.
  • Never commit real clinical data or restricted evaluation corpora.

Repository example

Read and run the offline dataset walkthrough when you need a bundled synthetic fixture and first-run download controls.

相似的 Skill

精选

pptx

生成、读取和修改演示文稿，也能合并拆分幻灯片。

ANanthropics/skills181k近 7 天新增 1,159 个 star

精选

doc-coauthoring

按步骤协作撰写技术方案、提案和决策文档，逐轮修改。

ANanthropics/skills181k近 7 天新增 1,159 个 star

精选

docx

创建和编辑 Word 文档，支持目录、页码、页眉等排版。

ANanthropics/skills181k近 7 天新增 1,159 个 star

精选

canvas-design

按设计理念生成海报和视觉作品，输出 PNG 或 PDF。

ANanthropics/skills181k近 7 天新增 1,159 个 star

精选

pdf

提取文本和表格、合并拆分、填写表单，也能对扫描件做 OCR。

ANanthropics/skills181k近 7 天新增 1,159 个 star

精选

xlsx

读写 Excel 和 CSV，处理公式、格式、图表与数据清洗。

ANanthropics/skills181k近 7 天新增 1,159 个 star