cherry-regression-test

Cherry Regression Test

Run deterministic Playwright E2E tests against one driver-owned Cherry Studio process per platform. The tested product includes Chat, Agents, MCP, Skills, knowledge bases, translation, image generation, and code tools; an LLM test agent does not control the test run.

CI contract

Use .github/workflows/e2e-regression-test.yml as the entry point. It:

1. Resolves a trusted branch or release tag.
2. Initializes an isolated directory under the GitHub runner temporary folder.
3. Installs the application and the code tools under test.
4. Launches one owned Electron process with CDP enabled.
5. Runs the ten files in tests/e2e/regression/ from simple to complex.
6. Continues after a failed phase so later results are still collected.
7. Produces English platform and aggregate reports, then enforces the verdict.
8. Stops only the Electron process recorded in the isolated run directory.

Do not add an LLM tool loop, MCP control server, turn limit, or a second Electron launch for each test. A restart is allowed only where the case contract explicitly verifies persistence or switches from the clean startup profile to the authenticated shared profile.

Configuration

The workflow reads these repository variables and secrets:

• CHERRY_TEST_CUSTOM_PROVIDER_BASE_URL
• CHERRY_TEST_CUSTOM_PROVIDER_ANTHROPIC_BASE_URL
• CHERRY_TEST_CUSTOM_PROVIDER_API_KEY
• CHERRY_TEST_CUSTOM_PROVIDER_CHAT_MODEL
• CHERRY_TEST_CUSTOM_PROVIDER_EMBEDDING_BASE_URL
• CHERRY_TEST_CUSTOM_PROVIDER_EMBEDDING_API_KEY
• CHERRY_TEST_CUSTOM_PROVIDER_EMBEDDING_MODEL
• CHERRY_TEST_CHERRYIN_CHAT_MODEL
• CHERRY_TEST_CHERRYIN_IMAGE_MODEL
• CHERRY_TEST_CHERRYIN_ACCOUNT
• CHERRY_TEST_CHERRYIN_PASSWORD

The custom chat provider requires both URLs: CHERRY_TEST_CUSTOM_PROVIDER_BASE_URL fills OpenAI, and CHERRY_TEST_CUSTOM_PROVIDER_ANTHROPIC_BASE_URL fills Anthropic. Both endpoints share CHERRY_TEST_CUSTOM_PROVIDER_API_KEY.

Chat and embedding providers are independent. Never print literal credentials, write them to fixtures, attach them to Playwright artifacts, or pass them to an unrelated action.

Test organization

Read scenario organization and the controller contract before making changes.

Register each case from the manifest:

test(...caseDefinition('S-01'), async ({ mainWindow }) => {
  // Assert the user-visible outcome.
})

cases.ts owns case IDs, titles, task tags, phases, and capability requirements. The workflow accepts a task ID and delegates selection to the controller. Prefer accessible roles, labels, placeholders, test IDs, and visible text. Native dialogs and cross-application interactions must use the repository-owned helpers in systemAutomation.ts.

Record assertions in Playwright, not prose. The custom reporter writes case and phase status into run.json and feeds the English Markdown/JUnit reports. The fixture saves failure screenshots. Executor errors and interrupted phases block a passing verdict. Do not enable Playwright Trace for credential-bearing tests because action parameters can expose secrets. A passing result does not depend on a model's judgment.

Focused execution

With an initialized run directory and its owned Electron process running:

pnpm exec tsx scripts/e2e/regression/cli.ts run-phase \
  --run-dir /absolute/run-directory --phase 02-basic-features

The task selected when initializing the run determines which cases execute. For a Notes-only run, initialize with --task notes. Enumeration is read-only: set CHERRY_TEST_RUN_DIR to an absolute path, but no initialized run directory or running Electron process is required because --list does not execute fixtures.

CHERRY_TEST_RUN_DIR=/tmp/cherry-regression-list pnpm test:e2e:regression --list

Do not call the regression cleanup command for an Electron instance owned by cherry-electron-dev; cleanup is only for an app record created by this driver.

Verification when changing the framework

Run the focused script suite and enumerate Playwright cases:

pnpm exec vitest run --project scripts scripts/e2e/regression
CHERRY_TEST_RUN_DIR=/tmp/cherry-regression-list \
  pnpm test:e2e:regression --list
pnpm typecheck:e2e
pnpm test:lint

Do not use pnpm test or pnpm build:check for this focused workflow change.