A flaky test is one that passes and fails without the code changing. In a local run you shrug and hit re-run. In CI it blocks a deploy. In production monitoring it pages someone at 3am for an outage that never happened. Same test, three very different costs.

Playwright is better at this than most frameworks. Auto-waiting, web-first assertions, and trace viewer eliminate whole categories of flake that Selenium users still fight. But "better" is not "immune." The flake that survives Playwright's defenses is the flake worth talking about, because the usual advice ("just add retries") makes it worse, not better.

This is a working playbook: the failure modes that actually cause flake in a Playwright suite, how to tell them apart, and what to do about each, in local CI and in production where you can't add --retries=3 and call it a day.

Why retries are a trap, not a fix#

retries: 2 in playwright.config.ts is the first thing everyone reaches for. It is the right tool for exactly one job (absorbing genuinely non-deterministic infrastructure noise, a CDN cold start, a one-in-ten-thousand TLS handshake) and the wrong tool for everything else.

Here is the problem. A retry that turns a red build green also deletes the evidence. The test that failed once and passed twice is now reported as "passed." Your flake rate looks like zero. Meanwhile the underlying race condition ships, and the same race condition that flaked your test is now flaking for real users, except they don't get two retries.

// playwright.config.ts
export default defineConfig({
  // Reasonable: absorb infra noise in CI only
  retries: process.env.CI ? 2 : 0,

  // Critical: make retries visible, not invisible
  reporter: [
    ["html"],
    ["json", { outputFile: "results.json" }], // mine this for retry counts
  ],
});

// playwright.config.ts
export default defineConfig({
  // Reasonable: absorb infra noise in CI only
  retries: process.env.CI ? 2 : 0,

  // Critical: make retries visible, not invisible
  reporter: [
    ["html"],
    ["json", { outputFile: "results.json" }], // mine this for retry counts
  ],
});

The rule: retries are allowed to hide infrastructure flake. They are not allowed to hide application flake. The rest of this post is about telling those two apart and killing the second kind.

The five failure modes that survive auto-waiting#

Playwright auto-waits for elements to be actionable. That eliminates the classic Selenium flake ("element not yet in DOM"). What it does not do is wait for your application's asynchronous state to settle. These five are what's left.

1. The data-not-the-DOM race#

Playwright waited for the button to be visible and clickable. It did not, cannot, wait for the network request behind that button to finish writing to the database before the next assertion reads it.

// Flaky: the row may not be queryable yet even though the toast showed
await page.getByRole("button", { name: "Create project" }).click();
await expect(page.getByText("Project created")).toBeVisible();
await page.goto("/projects");
await expect(page.getByText("My New Project")).toBeVisible(); // ← races the write

// Flaky: the row may not be queryable yet even though the toast showed
await page.getByRole("button", { name: "Create project" }).click();
await expect(page.getByText("Project created")).toBeVisible();
await page.goto("/projects");
await expect(page.getByText("My New Project")).toBeVisible(); // ← races the write

The fix is not a waitForTimeout. It's asserting on the thing that actually gates readiness (usually the network response) before navigating:

const created = page.waitForResponse(
  (r) => r.url().includes("/api/projects") && r.request().method() === "POST",
);
await page.getByRole("button", { name: "Create project" }).click();
await created; // the write is acknowledged before we move on
await page.goto("/projects");
await expect(page.getByText("My New Project")).toBeVisible();

const created = page.waitForResponse(
  (r) => r.url().includes("/api/projects") && r.request().method() === "POST",
);
await page.getByRole("button", { name: "Create project" }).click();
await created; // the write is acknowledged before we move on
await page.goto("/projects");
await expect(page.getByText("My New Project")).toBeVisible();

2. Animation and the "visible but not settled" element#

toBeVisible() resolves the instant opacity crosses zero. If your modal slides in over 300ms and the test clicks at 50ms, the click can land on a half-positioned element or get intercepted by the backdrop. This flake is timing-sensitive, so it shows up on slow CI runners and never on your laptop.

Prefer asserting on the post-animation state (focus trap engaged, content interactive) rather than mere visibility, and disable animations in test where you can:

await page.emulateMedia({ reducedMotion: "reduce" });
// or inject CSS that zeroes transition/animation duration in the test env

await page.emulateMedia({ reducedMotion: "reduce" });
// or inject CSS that zeroes transition/animation duration in the test env

3. Time, timezones, and the test that fails only after midnight UTC#

Any test that asserts on a relative date ("Created today", "2 minutes ago") is a time bomb that detonates on a schedule. It passes for 23 hours and fails in the hour where the runner's clock and the assertion's assumption disagree. This is the single most common cause of "it only fails on the nightly run."

Freeze the clock. Don't assert against a moving target.

await page.clock.install({ time: new Date("2026-05-17T12:00:00Z") });
// now "today" is deterministic regardless of when CI runs

await page.clock.install({ time: new Date("2026-05-17T12:00:00Z") });
// now "today" is deterministic regardless of when CI runs

4. Test isolation leaks#

Test B passes alone and fails when test A ran first. Shared state (a seeded user, a leftover cookie, a database row, a feature flag toggled and not reset) leaked across the boundary. Parallel sharding makes this non-deterministic because the order changes per run.

The tell: the failure rate correlates with --workers, not with the code. The fix is per-test isolation (fresh storage state, unique fixture data keyed by testInfo, transactional rollback), never "run this suite with --workers=1," which just hides the leak behind serialization.

5. Third-party and network non-determinism#

Your test renders a Stripe element, a map tile, an analytics pixel, an embedded video. Any of those can be slow or down, and your test fails for a reason that has nothing to do with your product. In CI this is annoying. In production monitoring it's an outage alert for someone else's outage.

Decide explicitly per dependency: mock it (page.route to stub the response), or assert around it (don't gate your assertion on the third-party element rendering at all).

A triage workflow that scales past 50 tests#

Once you have more than a handful of tests, "fix flaky tests" stops being a task and becomes a process. The process is: detect, quarantine, attribute, fix, in that order, never skipping quarantine.

Detect. Parse the JSON reporter output and count any test whose status is passed but whose retry is greater than 0. That is your real flake list. It is invisible in the HTML report's green checkmarks.

// scripts/flaky-scan.ts: run after the suite in CI
import results from "../results.json";

const flaky = results.suites
  .flatMap((s) => s.specs)
  .flatMap((spec) => spec.tests)
  .filter((t) => t.results.some((r) => r.retry > 0 && r.status === "passed"))
  .map((t) => t.title);

if (flaky.length) {
  console.log("::warning::Flaky (passed on retry):", flaky.join(", "));
}

// scripts/flaky-scan.ts: run after the suite in CI
import results from "../results.json";

const flaky = results.suites
  .flatMap((s) => s.specs)
  .flatMap((spec) => spec.tests)
  .filter((t) => t.results.some((r) => r.retry > 0 && r.status === "passed"))
  .map((t) => t.title);

if (flaky.length) {
  console.log("::warning::Flaky (passed on retry):", flaky.join(", "));
}

Quarantine. Tag the test (test.describe('@flaky', ...)) and exclude it from the merge-blocking job while keeping it running in a non-blocking job. This is the step everyone skips, and skipping it is why teams end up with a suite nobody trusts: a flaky blocking test trains engineers to re-run CI reflexively, and once they do that, a real failure also just gets re-run.

Attribute. Open the Playwright trace for the failed attempt, not the passing one. The trace viewer's timeline shows you exactly which action stalled and what the DOM looked like at that millisecond. Ninety percent of the time the failure mode is one of the five above and the trace makes it obvious which.

Fix, then de-quarantine. A test leaves quarantine only when you've identified the failure mode and addressed the cause, not when it "passed a few times in a row."

The part CI can't tell you: production#

Everything above assumes a CI run you control, where retries and quarantine are options. Production synthetic monitoring is the same Playwright tests pointed at the live environment on a schedule, and there, flake is indistinguishable from a real incident unless you've designed for it.

A flaky production check is worse than a missing one. It generates alerts that are wrong often enough that the on-call engineer learns to ignore them, and the one time the alert is real, it gets ignored too. Alert fatigue is just flake with a pager attached.

Three things make Playwright tests survive contact with production:

Confirm before alerting. A single failed run on a 1-minute schedule should not page anyone. Require N consecutive failures, or a failure that reproduces from a second region, before the alert fires. One-off network blips are the production equivalent of CI's "infra flake": absorb them the same way, visibly.
Separate "the assertion failed" from "the script errored." A timeout reaching your login page is an availability signal. A selector that didn't resolve because someone shipped a markup change is a test maintenance signal. Conflating them means every UI refactor looks like an outage.
Heal the selector flake instead of paging on it. The single largest source of production-monitoring flake is not your app. It's your test's selectors drifting as the UI evolves. A test that breaks because a class name changed is not an incident, and it shouldn't wake anyone.

That last point is the one most teams underinvest in, and it's where the cost compounds: every UI change is a potential false page, forever, for every test.

Where ObserveOne fits#

This is the problem ObserveOne is built around. You bring your existing Playwright tests (no rewriting them into a proprietary DSL) and run them as scheduled production monitors. When a selector drifts because the UI changed, the healer agent inspects the live page, finds the element the test meant, patches the script, and versions the diff so you can review what changed. The check stays green for the right reason instead of paging you for the wrong one.

It does not paper over application flake. The data races and isolation leaks in this post are still your bugs to fix, and you should fix them. What it removes is the selector-drift flake that turns every frontend refactor into a week of false alerts, and it gives you confirm-before-alert and region confirmation out of the box so a single blip doesn't become a page.

If your Playwright suite already passes in CI but you don't trust it enough to point at production, that gap is exactly the thing to close. You can paste a URL and have a suite generated, or bring the playwright.config.ts you already have.

The one-paragraph version#

Retries hide flake; make them visible by mining retry counts from the JSON reporter. Most surviving Playwright flake is one of five things: data-vs-DOM races, unsettled animations, relative-time assertions, isolation leaks, and third-party noise, and each has a real fix that is never waitForTimeout. At scale, run a detect → quarantine → attribute → fix loop and never skip quarantine. In production, confirm before you alert, separate availability failures from test-maintenance failures, and auto-heal selector drift so a UI refactor doesn't read as an outage.

Catching Flaky Playwright Tests in Production CI (Before They Catch You)