Agent Skills

---

name: 01-auto-arena

description: Automatically evaluate and compare multiple AI models or agents without pre-existing test data. Generates test queries from a task description, collects responses from all target endpoints, auto-generates evaluation rubrics, runs pairwise comparisons via a judge model, and produces win-rate rankings with reports and charts. Supports checkpoint resume, incremental endpoint addition, and judge model hot-swap. Use when the user asks to compare, benchmark, or rank multiple models or agents on a custom task, or run an arena-style evaluation.

---

name: 01-graders-and-pipeline

description: Build custom LLM evaluation pipelines using the OpenJudge framework. Covers selecting and configuring graders (LLM-based, function-based, agentic), running batch evaluations with GradingRunner, combining scores with aggregators, applying evaluation strategies (voting, average), auto-generating graders from data, and analyzing results (pairwise win rates, statistics, validation metrics). Use when the user wants to evaluate LLM outputs, compare multiple models, design scoring criteria, or build an automated evaluation system.

---

name: 01-paper-review

description: Review academic papers for correctness, quality, and novelty using OpenJudge's multi-stage pipeline. Supports PDF files and LaTeX source packages (.tar.gz/.zip). Covers 10 disciplines: cs, medicine, physics, chemistry, biology, economics, psychology, environmental_science, mathematics, social_sciences. Use when the user asks to review, evaluate, critique, or assess a research paper, check references, or verify a BibTeX file.

---

name: 02-bib-verify

description: Verify a BibTeX file for hallucinated or fabricated references by cross-checking every entry against CrossRef, arXiv, and DBLP. Reports each reference as verified, suspect, or not found, with field-level mismatch details (title, authors, year, DOI). Use when the user wants to check a .bib file for fake citations, validate references in a paper, or audit bibliography entries for accuracy.

---

name: 02-ref-hallucination-arena

description: Benchmark LLM reference recommendation capabilities by verifying every cited paper against Crossref, PubMed, arXiv, and DBLP. Measures hallucination rate, per-field accuracy (title/author/year/DOI), discipline breakdown, and year constraint compliance. Supports tool-augmented (ReAct + web search) mode. Use when the user asks to evaluate, benchmark, or compare models on academic reference hallucination, literature recommendation quality, or citation accuracy.

---

name: 02-rl-reward

description: Build RL reward signals using the OpenJudge framework. Covers choosing between pointwise and pairwise reward strategies based on RL algorithm, task type, and cost; aggregating multi-dimensional pointwise scores into a scalar reward; pairwise tournament reward for GRPO on subjective tasks (net win rate across group rollouts); generating preference pairs for DPO/RLAIF; and normalizing scores for training stability. Use when building reward models, scoring rollouts for GRPO/REINFORCE, generating preference data for DPO, or doing Best-of-N selection.

---

name: 03-ref-hallucination-arena

description: Benchmark LLM reference recommendation capabilities by verifying every cited paper against Crossref, PubMed, arXiv, and DBLP. Measures hallucination rate, per-field accuracy (title/author/year/DOI), discipline breakdown, and year constraint compliance. Supports tool-augmented (ReAct + web search) mode. Use when the user asks to evaluate, benchmark, or compare models on academic reference hallucination, literature recommendation quality, or citation accuracy.

---

name: ab-test-plan

description: Plan an A/B test by script: sample size per variant, days to run, stopping rules. "how many visitors per variant"

---

name: ad-creative

description: Write ad copy variants to each platform's specs, policy-checked and in brand voice. "write ad copy for Meta"

---

name: add-integration

description: Add a custom MCP integration: scaffold the server, write .mcp.json, test it. "add a new MCP server"

---

name: aeo-audit

description: Run a one-time AEO audit of six AI answer engines, scored per surface. "check our AI search visibility"

---

name: aeo-geo

description: Plan AEO/GEO strategy to get cited by AI: content, entities, schema, 90-day plan. "how do we get cited by AI"

---

name: agency-dashboard

description: Build the agency dashboard across all client brands: health, pacing, risk. "how are all our clients doing"

---

name: agent-readiness-audit

description: Audit agent readiness by script: AI-crawler rules, product schema, no-JS HTML, feeds. "can AI agents use our site"

---

name: align-human

description: Use when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic evaluation can replace human review, or build a human-reduction roadmap. Also use when the user mentions calibration, TPR/TNR, judge validation, inter-rater agreement, Cohen's kappa, bias detection, or "is my automatic evaluation trustworthy." Merges the calibrate and align functions into one skill.

---

name: analytics-insights

description: Design the analytics layer with script-backed benchmarks: KPI trees, report templates, MMM, cookieless. "define our KPIs"

---

name: anomaly-scan

description: Scan connected platforms for anomalies against stored baselines, with likely causes. "why did our CPA spike"

---

name: attribution-model

description: Design an attribution model and credit rules. Real path data → attribution-report. "set up multi-touch attribution"

---

name: attribution-report

description: Run attribution on real conversion paths, comparing models. "which channels actually drive revenue"

---

name: audience-intelligence

description: Research audiences: personas, JTBD maps, RFM segments. One persona doc → audience-profile. "run a JTBD analysis"

---

name: audience-profile

description: Profile 2-4 buyer personas as narrative documents: drivers, objections, journey. "create a buyer persona"

---

name: autopilot-status

description: Show autopilot status: health, auto-corrections, guardrails, escalations. "what did the autopilot change"

---

name: backlink-gap

description: Find backlink gap domains linking to competitors, not you, scored by script. "who links to our competitors but not us"

---

name: bootstrap

description: Use when the user has nothing — no traces, no labels, no eval set — and needs to build a v0 evaluation from scratch. Also use when the user says "I need to start evaluating my app but don't know where to begin," "I want to set up eval for a new product," or has just identified failure modes and needs to turn them into principles. Outputs a v0 grader in 30 minutes using OpenJudge SimpleRubricsGenerator, plus a roadmap to reach calibrated evaluation.