AI browser tests

Generate Playwright tests from a URL with Autopilot, run them on a schedule, and let healing repair broken selectors. Covers planning, credentials, regeneration and CI.

Autopilot writes browser tests for you. Point it at a URL, and it explores the site, plans a set of scenarios, generates Playwright specs, and reruns them on a schedule.

The dashboard calls this Autopilot. The CLI calls the same objects suites (obs suite). One suite is one target site plus the tests generated for it.

Generate a suite#

From the dashboard, open Suites and choose New. From the terminal:

obs suite generate https://example.com --name "Smoke Tests" --max-tests 5

Generation runs in two stages. First the planner explores the site and writes a plan: a list of scenarios worth testing, one planned file each. Then generation turns each planned file into a Playwright spec.

--max-tests accepts 1 to 30 and defaults to 10. Your plan may cap it lower, and also caps how many suites you can save.

Use --plan-only when you want to read the plan before spending anything on generation:

obs suite generate https://example.com --plan-only

Steer the planner#

--instructions appends persistent guidance to the planner prompt, up to 4000 characters. It is saved on the suite and reused every time the plan is rebuilt. It reaches the planner only, so regenerating an existing planned file will not pick up a change to it.

obs suite generate https://example.com \
  --instructions "Log in via the /login form before testing checkout"

--allow-form-submit lets the agents submit non-auth forms. Leave it off when the target has forms you would rather not fire in anger, like a contact form that emails a real inbox.

Credentials#

Tests that need a login take variables. Pass --var KEY=VALUE for a value, or --var KEY alone to be prompted with the input masked. For anything real, keep them in an uncommitted file:

obs suite generate https://example.com --var-file .env.secrets

Update them later with obs suite secrets <id>.

Run and schedule#

obs suite run <id> triggers an execution, obs suite status <id> reports on it, and obs suite wait <id> <executionId> blocks until it finishes, which is what you want in a pipeline.

Give a suite a cron at generation with --cron, or manage it afterwards:

obs suite schedule <id> --cron "0 */6 * * *" --enable
obs suite schedule <id> --disable

Each suite execution spends one AI verification. Planning, generation and healing spend AI credits, a separate allowance. On the lower plans credits arrive as a one-time grant rather than monthly, so a credit pack is how you keep regenerating.

Healing#

When a run fails because a selector no longer matches, healing rewrites that selector and saves a new version of the spec. Edits are restricted to the locator, so a heal cannot quietly change what the test asserts and hand you a green run that proves nothing.

Trigger one with obs suite heal <id>, and read what happened with obs suite heal-history <test-id>.

Editing the plan#

The plan is a file you can own. Pull the suite down, edit PLAN.md, push it back:

obs suite pull <id>
obs suite push <id>

A pushed edit marks the planned files it affects as stale. obs suite get <id> counts them, and obs suite regenerate <id> regenerates exactly those. Add --dry-run to see the targets first. --all is a fallback: when nothing is stale or missing it regenerates every non-dismissed planned file rather than doing nothing.

To drop a scenario you do not want:

obs suite dismiss-planned <suite-id> --planned-file <file>
obs suite restore-planned <suite-id> --planned-file <file>

CI#

Install the GitHub app and pick the repo in the web UI. The CLI manages the integration afterwards:

obs suite ci status <id>
obs suite ci webhook-token <id>   # rotates, invalidating the previous token
obs suite ci disconnect <id>

Next steps#

Related comparisons

Picking between the tools mentioned here? Side-by-side breakdowns below.

Swap out part of your stack

Looking for a different fit? Start here.

Was this page helpful?