portaljs-check-data-quality

PortalJS — Check Data Quality

Overview

Run a read-only quality audit of one CSV or TSV file, local or remote, and return a structured JSON report. The audit profiles every column — null/blank counts, inferred value types, numeric ranges, likely year/date fields — and flags duplicate rows, duplicate values in identifier-like columns, ambiguous overlapping year columns (e.g. calendar year vs fiscal year), and mixed-type columns. It never edits the source file, datasets.json, or any other project file; it only reads the target file (a remote URL is downloaded to a temp file that is deleted before the run ends) and prints a report. Use it before publishing a dataset with portaljs-add-dataset, or to diagnose why a showcase renders wrong.

Prerequisites

• python3 on PATH — the audit logic runs as an embedded Python script; nothing is installed.
• One CSV or TSV file, given as a local path or an http/https URL. Only one file per run.

Instructions

The canonical, full step-by-step workflow is .claude/commands/portaljs-check-data-quality.md — the single source of truth. Read and follow it when executing. Summary:

1. Gather input — the file path or URL to audit. If missing, ask for it; never dead-end.
2. Resolve the source: if it's an http/https URL, download it to a temp file first; otherwise use the local path as given.
3. Validate the extension is .csv or .tsv. If not, or the file is missing, or the header row is empty, stop and surface the error JSON as-is — do not guess a fix.
4. Profile every column: null/blank counts, distinct values, sample values, inferred per-value type (boolean/integer/float/date/string), numeric min/max, and year range for columns whose name looks year-like.
5. Derive findings from the profiles — duplicate rows, missing-value ratios, invalid year values, mixed types, suspect negative values, duplicate identifier values, and ambiguous overlapping year columns — each tagged critical, warning, or info.
6. Assemble the JSON report (status, file metadata, findings, recommendations, column_profiles), print it, and clean up the temp file if one was created.
7. Relay the report to the user as-is; do not modify the source file, datasets.json, or any other project file based on the findings — that's a separate, explicit step.

Output

A single JSON object printed to stdout:

• status — ok, warning, or critical.
• file, file_name, source_type (local or url), row_count, column_count.
• findings — structured issues, most severe first.
• recommendations — de-duplicated suggested next steps.
• column_profiles — per-column summary (nulls, blanks, distinct count, sample values, inferred types, numeric/year ranges).

No files are created or modified. A remote URL's temp download is removed on exit, success or failure alike.

Error Handling

Symptom	Cause	Fix
"File ... is not available."	Local path is wrong, or the URL download failed	Verify the path or URL is reachable and retry.
"Only CSV and TSV files are supported right now."	File extension isn't .csv/.tsv	Convert the file, or point to its tabular source instead.
"... does not contain tabular headers."	File is empty or the header row is malformed	Open the file and confirm it has a valid, non-empty header line.
Command hangs on a URL	Remote host is slow or blocks non-browser requests	Download the file manually and audit the local copy instead.
python3: command not found	Python 3 isn't installed or not on PATH	Install Python 3, or run the audit where it's available.
Report looks truncated in the terminal	Large report wrapped/paginated by the shell	Redirect to a file (> report.json) and open it separately.

Examples

Example 1 — Audit a local CSV before publishing

/portaljs-check-data-quality ./public/data/trash.csv

Example 2 — Audit a remote CSV over HTTPS

/portaljs-check-data-quality https://example.com/trash.csv

Example 3 — Audit a TSV and save the report for review

bash scripts/check-data-quality.sh ./data/emissions.tsv > /tmp/emissions-quality.json

Example 4 — Read a critical status report

{
  "status": "critical",
  "findings": [
    { "severity": "critical", "check": "duplicate_rows", "message": "42 duplicate rows found." }
  ],
  "recommendations": ["Review and deduplicate repeated rows if they are not intentional."]
}

Fix the flagged rows/columns, then re-run the audit before publishing.

Resources

• Full workflow: .claude/commands/portaljs-check-data-quality.md
• Detailed check catalog and troubleshooting: references/reference.md
• Related skills: portaljs-add-dataset, portaljs-define-schema
• Python csv module (parsing behavior this audit relies on): https://docs.python.org/3/library/csv.html