All posts

Testing an AI-built app: the first tests worth writing

You ask your AI tool for a small change. It works. Two days later you find that checkout, or login, or an old form no longer works. Nobody asked the tool to touch those. It is a common complaint about AI-built apps. The fix is not more careful prompts. The fix is a few tests. Here is which ones to write first, in order of risk, and how to run them. Tool facts come from each tool's own docs, and versions change, so open the linked page before you copy a command.

Short answer: test what costs you money or trust first: the price, who can see which data, and the payment webhook. Add one browser test for the main path, and one check against the live site after each deploy. Run it all on every push.

1. Why AI-built apps break things that worked

An AI tool does not remember what your app must keep doing. Each prompt can change files you did not mention: a shared helper, a database rule, a route. The change looks fine on the screen you were checking. Something else breaks, and you find out from a customer.

A test is a small program that checks one thing and says pass or fail. Together they are the memory of what must keep working. A red result after a change tells you what it broke, while the change is still fresh.

2. What to test first, in order of risk

1. The money path

Start where a bug costs real money. The rule: the price comes from your server, never from the browser. Test the server function that creates the checkout, not Stripe itself.

  • Call the function with a plan name. Check that it uses the price ID you expect for that plan.
  • Call it with a made-up price or amount in the request. Check that the function ignores it or rejects it.

In your test, replace the Stripe call with a fake that records what it was given. Then you assert on the recorded price ID. The rest of the money setup is in Stripe payments in an AI-built app.

2. Permissions

Next, who can see whose data. Create two test users, A and B, and let A create a record. Then check these:

  • B asks for A's record by its ID. The answer must be an error or an empty result, never the data.
  • B tries to update or delete A's record. It must fail.
  • A visitor who is not signed in asks for either. It must fail.

Run this against your real data rules, on a test database. Why this matters is covered in database security checks for AI-built apps.

3. Webhooks

A webhook is a call from another service to your app, such as Stripe saying "this customer paid". Two tests matter.

  • A fake event is rejected. Send a request with a missing or wrong Stripe-Signature header. Your endpoint must refuse it and change nothing. Stripe's webhook docs say to verify the signature, and that without it an attacker could send fake events to trigger actions like granting account access.
  • The same event twice does not double-apply. The same docs say an endpoint might occasionally receive the same event more than once, and suggest logging the event IDs you have processed. Send one event twice. Check that the user is upgraded once and credits are added once.

4. Input validation

Every form and API endpoint takes input from strangers. Write a few tests that send bad input and check the answer: an empty field, a very long text, a wrong type, a missing required value. The endpoint should answer with a clear error and save nothing.

5. One end-to-end test in a real browser

Unit tests check pieces. This one checks that the pieces work together. It opens a real browser, signs up or signs in, and does the main thing your app is for. Playwright does this. Its intro page calls it an end-to-end test framework for modern web apps, and says it runs Chromium, Firefox and WebKit. Keep it to one test at first. If the main path breaks, you know right away.

6. A smoke check against the live site

The last test runs after a deploy, against the real URL. It is small: do the key pages answer, is the key text on them, does a page that should not exist return a 404, does a protected API route refuse a request with no login? It catches deploys that went out broken, such as a missing environment variable. Pair it with uptime checks and error alerts, which keep watching afterwards.

3. Tooling basics

Vitest for unit tests

The Vitest guide calls it a testing framework powered by Vite. It reads your vite.config file by default, so a Vite project needs little extra setup. Install it with npm install -D vitest. Test files need .test. or .spec. in the name, such as price.test.js. Plain vitest keeps watching for changes. To run once and exit, as you want in CI, use vitest run.

Playwright for the browser

Run all tests with npx playwright test. Playwright runs headless by default and, by default, across several browsers in parallel. On a small laptop, that is heavy. The intro page shows that you can limit it to one browser with --project=chromium, or run one file such as npx playwright test tests/example.spec.ts. The parallelism page adds --workers=1 to run one test at a time. Both keep a local run light.

GitHub Actions to run it on every push

GitHub's Actions overview explains the idea: a workflow is a YAML file in .github/workflows in your repository. It starts on an event, and pushing a commit is one such event. It runs one or more jobs on a runner, which is a server GitHub provides. A first workflow checks out your code, installs packages, and runs npm run test:run.

On cost, GitHub's billing page says standard GitHub-hosted runners are free for public repositories. Private repositories get a monthly amount of included usage that depends on your plan, and when it is used up, runs are blocked unless you have a payment method on file. Read that page for your plan.

4. What not to do

  • Do not chase 100% coverage. Coverage counts lines that ran, not lines that matter. Ten tests on money and permissions beat a hundred on button colors.
  • Do not test the framework or other people's APIs. You do not need to prove that Stripe works or that Vue renders. Playwright's best practices say the same: only test what you control, and replace external services with a fake response.
  • Do not leave flaky tests red. A test that fails at random teaches you to ignore red. Fix it or delete it the same day. The Playwright page says its auto-waiting and web-first assertions, which retry until a condition is met, are the main tools for stable tests.

What I do

The test counts on my own projects: my travel app HappyRoam has 1,326 automated tests (Vitest), my AI-visibility tool Foreground about 1,000, my job board Remote Job Radar 502, and my astrology site WuXing Zodiac 268. My TestSwap site has a smoke script, npm run smoke, that checks prerendered pages, 404s, the API's 401 and 404 answers and more against a deploy.

On Nov 10 2026, before I changed how my sites reload after a deploy, I wrote a test that simulates a deploy in a real browser. I ran it first to confirm the bug, then again after the fix. A test that fails first proves it can catch the problem.

Your first week

  1. Day 1: money path tests. Price from the server, right price ID, no price from the browser.
  2. Day 2: permission tests with two users. B cannot read, change or delete A's data.
  3. Day 3: webhook tests. A bad signature is rejected and the same event twice counts once.
  4. Day 4: one Playwright test: sign in and do the main action.
  5. Day 5: a GitHub Actions workflow that runs the tests on every push. Add the live smoke check if you have time.

After that, add a test each time something breaks: write the test that fails, then fix the bug. Before a launch, also go through the launch-ready checklist.

Want this fixed for you?

I finish and fix apps built with Lovable, Bolt, Cursor or Replit, including a first set of tests for payments, permissions and webhooks, a browser smoke test and tests that run on every push. Or send me the details on the hire page and I'll reply within 24 hours.