# Behavioral Evaluations & EDK Guide > This guide introduces the Eval Development Kit (EDK) and details how to write, validate, run, and report on behavioral e… - 网址：https://funcoding.ai/agents/gemini-cli/behavioral-evals/ - 来源：Gemini CLI 官方文档原文（英文），Apache-2.0 许可，同步于 2026-10-11 - 官方原文：https://geminicli.com/docs/behavioral-evals/ --- This guide introduces the **Eval Development Kit (EDK)** and details how to write, validate, run, and report on **behavioral evaluations** in the Gemini CLI codebase. --- ## Overview Behavioral evaluations are automated tests designed to assert on the **behavior** of the Gemini CLI agent (e.g., verifying which tools are called, checking call ordering, or avoiding destructive commands) rather than checking the final prose output. Evaluating agent behavior is critical because: 1. Model responses are non-deterministic, making exact prose matching highly fragile. 2. We must ensure the model utilizes the most efficient tools (e.g., batching files via `read_many_files` instead of sequential `read_file` calls). 3. We must enforce safety boundaries (e.g., preventing execution of raw shell commands when safe alternatives exist). All behavioral evaluations are stored under the `evals/` directory. --- ## EDK Developer Commands The EDK provides CLI tools under `scripts/` to help contributors audit, check, and monitor evals. ### 1. `npm run eval:inventory` Scans all eval files under `evals/`, statically parses them, and provides a structured overview of what exists in the repository. - **Usage:** ```bash npm run eval:inventory ``` - **JSON Output:** For CI integration or inventory indexing, generate a machine-readable JSON report: ```bash npm run eval:inventory -- --json ``` - **Custom Root:** Run against another directory or repository: ```bash npm run eval:inventory -- --root /path/to/other/repo ``` --- ### 2. `npm run eval:validate` A lint-like checker that validates eval source files against standard structural guidelines and best practices. - **Usage:** ```bash npm run eval:validate ``` - **Custom Scopes:** Validate a specific file: ```bash npm run eval:validate -- evals/my-test.eval.ts ``` #### Validation Rules & Severities | Rule ID | Severity | Description | | :------------------- | :---------- | :--------------------------------------------------------------------------------------------------------------------- | | `file-naming` | **Error** | File must match `*.eval.ts` or `*.eval.tsx` naming conventions. | | `valid-policy` | **Error** | Policy must be one of `ALWAYS_PASSES`, `USUALLY_PASSES`, or `USUALLY_FAILS`. | | `suite-metadata` | **Error** | Both `suiteName` and `suiteType` must be present as static string literals. | | `prompt-presence` | **Error** | Every eval case must have a non-empty `prompt` string. | | `case-name-static` | **Error** | The case name must be a static string literal, not computed dynamically. | | `invalid-tool-refs` | **Error** | All tools referenced in assertions must match known built-in or legacy tools. | | `positive-assertion` | **Error** | Evaluation cases must assert on at least one tool call (e.g., check `waitForToolCall` has been invoked). | | `workspace-setup` | **Error** | Workspace behaviors (like file-system edits/reads) must set up a `files` object. | | `new-evals-policy` | **Warning** | New evals must not use `ALWAYS_PASSES` policy initially (they should be promoted after nightly data proves stability). | Warnings (`new-evals-policy`) will be logged with `⚠` and will **not** cause the CLI process to exit with status `1`. Errors (`✗`) will block CI builds and return exit status `1`. --- ### 3. `npm run eval:report` Aggregates local vitest `report.json` artifacts, maps them against inventory policies, and summarizes the pass rates per model. - **Usage:** ```bash npm run eval:report ``` By default, it scans `evals/logs/` recursively for `report.json` files. - **Specifying Directory:** ```bash npm run eval:report -- /path/to/logs ``` - **JSON Output:** ```bash npm run eval:report -- --json ``` --- ## Contributor Workflow When writing a new behavioral evaluation, adhere to this workflow to ensure high-quality, non-flaky test runs. ### Step-by-Step Guide 1. **Identify the Target Behavior**: Determine which tool calls need verification (e.g., `web_fetch` must be called). 2. **Author the Eval File**: Create your file under `evals/.eval.ts` naming it properly. 3. **Configure Workspace Files**: If the eval reads or edits files, define them inside the `files` metadata field. 4. **Assert Behavior, Not Prose**: Ensure the `assert` block checks tool interactions using `rig.waitForToolCall` or similar. Do not check final prose. 5. **Run Locally**: ```bash RUN_EVALS=true npx vitest run evals/my-test.eval.ts ``` 6. **Deflake**: Run your eval at least 3 times locally to verify it does not fail due to model variance. 7. **Run Validation**: Run `npm run eval:validate` to ensure no linting errors are present. ### Acceptance Criteria Checklist - [ ] **Naming**: File ends with `.eval.ts` or `.eval.tsx`. - [ ] **Policy**: New evals start as `USUALLY_PASSES`. - [ ] **Metadata**: Static `suiteName` and `suiteType` (e.g. `'behavioral'`) are specified. - [ ] **Assertions**: Uses `rig.waitForToolCall` or asserts tool arguments explicitly. - [ ] **Clean workspace**: Does not write to files outside `rig.testDir`. ### Common Anti-Patterns to Avoid - **Restricting core tools**: Never override `settings.tools.core` to limit tools. Evals must run against the default toolset. - **Checking model prose**: Avoid `expect(result).toContain('something')` since model wording is non-deterministic. - **Integration-only testing**: Evals that only write files without checking realistic model prompts are integration tests and belong under `integration-tests/`. --- ## CI & Dashboard Integration You can easily automate behavioral evaluations or compile dashboard data using EDK's JSON reporters. ### CI Validation Block Add a step in your PR checks or GitHub workflows to automatically lint new evals and block pull requests containing validation errors: ```yaml - name: Run Eval Validator run: npm run eval:validate ``` ### Publishing to a Dashboard To record nightly performance metrics across multiple models: 1. Configure your workflow to run evaluations with the JSON reporter: ```bash cross-env GEMINI_MODEL=gemini-2.5-pro npx vitest run --config evals/vitest.config.ts --reporter=json --outputFile="evals/logs/eval-logs-gemini-2.5-pro/report.json" ``` 2. Aggregate all test runs using the reporting tool: ```bash npm run eval:report -- evals/logs --json > aggregated_report.json ``` 3. Upload `aggregated_report.json` to your dashboard storage backend to visualize pass rates over time.