How do you test AI-generated code?
Testing AI-generated code means checking the change against requirements your team agreed independently, so the code and its tests do not simply repeat the same mistaken assumption. Several methods then check the change, and mutation testing shows whether the tests notice deliberately introduced faults.
Separate the requirement from the implementation.
Suppose a ticket says delivery is free from £50. A function using > and tests covering £49.99 and £50.01 can agree with each other while missing exactly £50. This is a deliberately constructed teaching example, rather than a recorded evaluation of a coding assistant. It demonstrates the review problem regardless of who wrote the change.
// Deliberately seeded boundary defect: exactly £50 incorrectly costs 500p.
export function deliveryPence(basketPence) {
return basketPence > 5000 ? 0 : 500;
}assert.equal(deliveryPence(5000), 0);Use checks that can disagree with the original tests.
Start from the requirement, then ask a different question. The generated property says every basket from 5000p to 10000p should get free delivery. Its local fast-check run found 5000p at seed 20261015 and path 26. The Cucumber examples also failed at that boundary. Against the correct code, Stryker’s >= to > mutant survived the two-example suite and was killed after the boundary assertion was added.
- The property found the concrete failing input in its sampled domain.
- The scenario exposed that failure as an agreed business example.
- Mutation testing showed why the original two-example suite could miss it.
- These methods describe the same boundary fault; they are not three independent product defects.
npm ci --ignore-scripts
npm run verifyKeep the recorded output with the change.
The verification harness expects the deliberately buggy property and scenario commands to fail, checks the counterexample and compares both native mutation reports. It exits successfully only when those expected outcomes are observed. The corrected property passed 1,000 generated cases; the strong suite killed five out of five mutants in this fixture. Inspect all source and recorded results. For separate evidence from Cue April’s actual engines, inspect the owned local product run, which records unit, API and generated-input outcomes against a tested product revision.
fast-check, buggy: counterexample [5000], seed 20261015, path "26"
fast-check, fixed: 1000 generated cases passed
Cucumber, buggy: 2 scenarios passed, 1 failed
Cucumber, fixed: 3 scenarios passed
Stryker, weak: 4 mutants killed, 1 survived
Stryker, strong: 5 mutants killed, 0 survivedReview this checklist before merging.
A useful review checks the change and its evidence against the ticket. GitHub’s responsible-use documentation likewise says generated tests need review. Keep human judgement on the expected behaviour and the implications of a failure.
- Write the intended rule and important exceptions before evaluating the generated assertions.
- Review the diff for permission changes, validation, external effects and data handling.
- Test exact boundaries and denied actions, not only representative successful cases.
- Check that an assertion would fail if the relevant behaviour broke.
- Reproduce a reported failure and keep its input, requirement and tested revision.
- Keep errors, missing cases and blocked checks separate from passes.
- Rerun relevant checks after the fix or a new commit; follow the repository’s merge rules.
Choose the next method for the risk in the change.
Use property-based testing for a stated rule across inputs, mutation testing for the strength of its assertions and Gherkin scenarios for agreed examples. Add API or browser checks when the risk crosses those interfaces. The input domain, environment and expected result should be explicit; no finite set of checks guarantees that every defect has been found.
Bring your own code into a scoped evaluation.
Cue April can coordinate approved checks and retain findings, revision identity and incomplete work for your team to review. Account setup and repository connection are separate from approving execution. Supported tools, qualified runners and paid terms are confirmed for your scope. Compare self-serve setup and managed testing, or see how evidence fits a pull request.
Primary documentation
Framework behaviour and configuration are documented by their maintainers.