Ask AI to Write Useful Tests and Verify They Catch Bugs

By /

A passing test is useful only if it would notice a real problem. AI can write tests that look convincing without checking the result you care about.

You will design browser tests for a small reading-list app and deliberately introduce a regression to check them. Allow 45–60 minutes. You need a working local app, Node.js, Playwright and an AI assistant. Follow the current Playwright installation instructions for your environment.

Test plan comparing an item-add check in working code with the same check after deliberately removing the list update.
This is the planned pass/fail check, not a screenshot of an executed Playwright run.
Read this example as text

Working app

  • Add a title
  • Check the item appears
  • Test passes

Deliberately broken copy

  • Remove the list update
  • Run the same test
  • It should fail on the missing item

See the finished example

Open the complete example

Read the complete finished example

Actual executed checks

The accompanying Node harness executes the actual app script with a simulated DOM and storage. It also removes the append line and the save line in disposable in-memory copies, then confirms the relevant assertions fail.

Read the executed verification record

What remains to run

The Playwright suite is complete but has not been run in a browser here. Follow README.txt for setup and real browser checks. Layout, native keyboard behaviour and browser storage policies remain separate checks.

Expected mutation failures

Change in disposable copy Assertion that should fail
Remove APPEND_POINT A submitted title creates one visible list item.
Remove SAVE_POINT The same title is restored after reload.

Setup and mutation instructions

1. Start from behaviours

List what a user should be able to do: add a title, reject empty input, complete a single entry, remove it after confirmation and restore saved entries on reload. Decide how the app labels its controls before selecting test locators.

Write Playwright tests for this reading-list app.
[PASTE THE HTML, BEHAVIOUR REQUIREMENTS AND LOCAL START COMMAND]

Test user-visible outcomes using accessible roles and labels.
Include adding a title, rejecting whitespace, changing only one of two
duplicate-title entries, and persistence after reload. Isolate storage
between tests. Do not use arbitrary sleeps or assert only that a
function was called. Explain which requirement each assertion checks.

2. Check one test before adding more

For an app whose input is labelled “Book title” and button “Add book,” a basic test can be:

import { test, expect } from '@playwright/test';

test('adds a title and clears the input', async ({ page }) => {
  await page.goto('http://localhost:8080');
  await page.getByLabel('Book title', { exact: true }).fill('The Hobbit');
  await page.getByRole('button', { name: 'Add book', exact: true }).click();
  await expect(page.getByRole('listitem')).toHaveCount(1);
  await expect(page.getByRole('listitem')).toContainText('The Hobbit');
  await expect(page.getByLabel('Book title', { exact: true })).toHaveValue('');
});

The address must match your running app. The app must begin this test with an empty list. Adapt selectors to actual labels rather than guessing. This sample has not been executed against your app.

3. Add meaningful failure cases

Ask for a spaces-only submission check and verify that it asserts the list remains unchanged. For persistence, add a title, reload and assert it remains. Test duplicate entries independently; simply finding their shared text is not enough.

Playwright actions and assertions support waiting for expected states. Prefer those checks over fixed pauses that make tests slow or unreliable. Writing Playwright tests

4. Check that the tests catch a deliberate bug

In a disposable copy of the sample app, remove the code that appends the new list item. Run the add test. It should fail on the missing item, not because the server was never started. Restore the code and run it again.

Then disable saving and check that the reload test fails. Record the deliberate change, expected failure and actual result. Do not make these experiments in a production application.

5. Review reliability

Run the suite again with a clean test context. Look for dependence on order, stale storage or external services. A passing run does not establish accessibility, security or correctness outside the behaviours it checks.

Everything passes even with broken code: inspect the assertions. Tests fail randomly: examine state isolation and asynchronous behaviour. The assistant changes tests to fit a bug: return to the acceptance criteria; the intended behaviour is the reference.

Scroll to Top