Skip to content

skillTesting

Flaky Test Diagnosis

Reproduces an intermittent test failure, finds the shared state or timing cause and fixes it, rather than hiding it behind retries or longer waits.

npx workspacetemplates add flaky-test-diagnosis --tool claude-code

Writes 1 file to .claude/skills/flaky-test-diagnosis/

.claude/skills/flaky-test-diagnosis/SKILL.md27 lines · 1.5 KBFile
---
name: flaky-test-diagnosis
description: Reproduces an intermittent test failure, finds the shared state or timing cause and fixes it, rather than hiding it behind retries or longer waits.
---

# Flaky test diagnosis

A flaky test is a real bug in the test or the code. Never "fix" it with a retry, a longer timeout or a skip.

## Steps

1. Get the exact test name and one failing log. Note the runner and whether CI runs tests in parallel.
2. Measure. Run the single test 50 times in a loop and count failures. Then run the whole file, then the whole suite, and compare. Failing only in the suite points to shared state or ordering.
3. Force the suspected cause: run in random or reversed order, with one worker and with many, under CPU load (`stress-ng` or a busy loop), and with the clock frozen or shifted.
4. Classify the cause: leaked global or database state, real time or timezone, unawaited promise or thread, shared port or temp file, unseeded randomness, network call, or dependence on map or set ordering.
5. Fix the cause: isolate state per test, inject the clock, await the work, use ephemeral ports and unique temp dirs, seed the random generator and print the seed on failure, sort before asserting.
6. Rerun the 50-iteration loop. Done only at zero failures.

## Rules

- Never add sleeps. Wait on a condition or an event.
- Change one thing between measurements.
- If the production code is racy, say so and fix it there.

## Report

Failure rate before and after, the classified cause, the diff, and any sibling tests with the same pattern.

What it does

A test that fails one run in twenty teaches a team to ignore red builds. This skill makes the agent treat flakiness as a bug with a cause: loop the test until it fails, then change one variable at a time until the failure rate moves.

The usual culprits are leaked state between tests, real clocks, unawaited async work, port and file collisions, and ordering assumptions. The agent names which one it found and shows the failure rate before and after the fix.

Example

you › tests/invoice.test.ts fails about once in ten CI runs. Find out why.

agent › Loop of 50 runs: 6 failures, all when "totals by month" ran after "late fees". Cause: both mutate a module-level currency cache. Fix: reset the cache in beforeEach and stop sharing the object. After: 0 failures in 100 runs. Two other files import the same cache; I listed them but left them alone.

Questions

Why not just enable retries in CI?
Retries make a red build green but keep the defect, and they hide a real race in your code. The test still costs time and still fails for users.
What if the failure never reproduces locally?
It tries CPU load, a single core, random order and shifted clocks to mimic CI. If it still cannot fail, it asks for the CI logs and reports what it could rule out.
  • Contract Testing

    SkillTesting

    Free
  • E2E Test Design

    SkillTesting

    Free
  • Integration Testing

    SkillTesting

    Free