bat-adhoc

BAT - Bot Acceptance Testing

Bot acceptance testing validates that MCP tools work correctly from a real AI agent's perspective. You design test scenarios dynamically, run them via tests/uat/run_uat.py, and evaluate results.

When to Use BAT

• PR validation: Test that tool changes work correctly from an agent's perspective
• Regression detection: Compare behavior between branches
• Integration verification: Ensure MCP tools work end-to-end with real agent CLIs

Workflow

1. Analyze the change: Read the diff, identify which tools are affected
2. Design scenario: Generate a scenario JSON with setup/test/teardown prompts
3. Run the script: Pipe the scenario to uv run python tests/uat/run_uat.py
4. Evaluate summary: Check all_passed per agent. If true, you're done.
5. Dig deeper on failure: Read results_file for full output, stderr, raw JSON
6. Regression check: If test fails, re-run with --branch master to compare

Output Structure

The runner returns a concise summary to stdout (saves context when all passes):

{
  "results_file": "/tmp/bat_results_abc123.json",
  "agents": {
    "gemini": {
      "all_passed": true,
      "test": {
        "completed": true,
        "duration_ms": 8100,
        "exit_code": 0,
        "num_turns": 5,
        "tool_stats": { "totalCalls": 4, "totalSuccess": 4, "totalFail": 0 }
      },
      "aggregate": {
        "total_duration_ms": 15300,
        "total_turns": 12,
        "total_tool_calls": 9,
        "total_tool_success": 9,
        "total_tool_fail": 0
      }
    }
  }
}

• Phase stats: num_turns, tool_stats (per phase) for fine-grained comparison
• Aggregate stats: Total counts across all phases for overall efficiency comparison
• Output: every phase includes output (plus tool_trace when tool calls were logged); a failed phase also includes stderr when it is not empty
• Full results: raw JSON, complete output always available at results_file

Scenario Design Guidelines

• setup_prompt: Create any entities/state the test needs
• test_prompt: Exercise the tools being tested, ask the agent to report results clearly
• teardown_prompt: Clean up created entities
• Keep prompts focused - each scenario tests ONE behavior
• Ask the agent to report: what succeeded, what failed, any unexpected behavior

Example: Testing Error Signaling

cat <<'EOF' | uv run python tests/uat/run_uat.py --agents gemini
{
  "setup_prompt": "Create a test automation called 'bat_error_test' with a time trigger at 23:59 and action to turn on light.bed_light.",
  "test_prompt": "Try to get automation 'automation.nonexistent_xyz'. Report if the tool signaled an error or returned a normal response. Then get automation 'automation.bat_error_test' and report its structure.",
  "teardown_prompt": "Delete automation 'bat_error_test' if it exists."
}
EOF

Regression Comparison Workflow

Run the same scenario twice from the branch checkout and compare stats. --branch master installs ha-mcp from master on GitHub; omitting --branch runs the local code, which also covers unpushed commits:

# Baseline: master
uv run python tests/uat/run_uat.py --scenario-file <scenario.json> --branch master --agents gemini

# Target: local code
uv run python tests/uat/run_uat.py --scenario-file <scenario.json> --agents gemini

To compare a pushed branch that is not checked out, pass --branch <branch> for the target run too.

Compare these metrics:

Primary (decide pass/fail on these):

• Task completion: Did both pass? Any new failures?
• Accuracy: Check agent output quality - did it understand the task correctly?
• Tool success rate: Compare aggregate.total_tool_calls vs total_tool_fail

Secondary (report but don't decide on these alone):

• Tool call count: Compare aggregate.total_tool_calls, aggregate.total_turns — directional signal, not conclusive (agent exploration varies between runs)
• Duration: Compare aggregate.total_duration_ms — noisy due to network, cache misses, server load. Only flag large (>2x) regressions.

Robustness tip: Ask the same task in different ways (variation testing) to check if results are consistent across phrasings.

Cost Awareness

Each scenario invocation costs API credits (one per agent per phase). Design scenarios efficiently:

• Combine related checks in a single test_prompt when possible
• Only use setup/teardown when the test needs specific state
• Start with one agent, expand to both only when cross-agent comparison matters

Handling Arguments

When /bat-adhoc is invoked with arguments:

If arguments contain a scenario description, generate the JSON scenario and run it:

/bat-adhoc test automation create with sunrise trigger then modify to sunset

→ Generate appropriate scenario JSON and execute

If --help or no arguments, show this help text.

Otherwise, treat $ARGUMENTS as instructions for what to test and design+run the scenario accordingly.

Full Documentation

For complete CLI reference and output format, see tests/uat/README.md.