L2 · 10 · AI-Assisted Testing & Code Review

AI writes tests in seconds, but many of them prove nothing. This page teaches you to get strong tests from AI: property-based tests, mutation testing to measure test strength, realistic test data, and a review process where AI catches issues without becoming a rubber stamp.

Why AI tests are often weak

Property-based testing

Instead of a few examples, you state properties that must hold for all inputs, and a library generates hundreds of random cases, shrinking any failure to a minimal example. AI is very good at proposing properties. You judge which are true.

Here is `applyDiscount(priceCents: Long, percent: Int): Either[String, Long]` and its spec:
percent must be 0..100; result is rounded half-up to the nearest cent; never negative.
Propose 5 properties that should hold for all valid inputs and 2 for invalid inputs.
Then write them with munit-scalacheck. Don't restate the implementation's formula in the properties.

//> using scala 3
//> using test.dep org.scalameta::munit-scalacheck:1.2.0
import munit.ScalaCheckSuite
import org.scalacheck.Prop.forAll
import org.scalacheck.Gen

class DiscountSuite extends ScalaCheckSuite:
  val prices   = Gen.chooseNum(0L, 10000000L)        // up to 100,000.00
  val percents = Gen.chooseNum(0, 100)

  property("result is between 0 and the original price"):
    forAll(prices, percents) { (p, pct) =>
      Discount.apply(p, pct).exists(r => r >= 0 && r <= p)
    }

  property("0% changes nothing, 100% makes it free"):
    forAll(prices) { p =>
      Discount.apply(p, 0) == Right(p) && Discount.apply(p, 100) == Right(0L)
    }

  property("a bigger discount never gives a higher price"):
    forAll(prices, percents, percents) { (p, a, b) =>
      val (lo, hi) = (a min b, a max b)
      (for x <- Discount.apply(p, lo); y <- Discount.apply(p, hi) yield y <= x).getOrElse(false)
    }

  property("invalid percentages are rejected"):
    forAll(prices, Gen.oneOf(Gen.chooseNum(-1000, -1), Gen.chooseNum(101, 1000))) { (p, bad) =>
      Discount.apply(p, bad).isLeft
    }

//> using scala 3
object Discount:
  def apply(priceCents: Long, percent: Int): Either[String, Long] =
    if percent < 0 || percent > 100 then Left(s"percent must be 0..100, got $percent")
    else Right(priceCents - (BigDecimal(priceCents) * percent / 100).setScale(0, BigDecimal.RoundingMode.HALF_UP).toLong)

from decimal import Decimal, ROUND_HALF_UP

import pytest
from hypothesis import given, strategies as st


def apply_discount(price_cents: int, percent: int) -> int:
    if not 0 <= percent <= 100:
        raise ValueError("percent must be 0..100")
    off = (Decimal(price_cents) * percent / 100).quantize(Decimal(1), rounding=ROUND_HALF_UP)
    return price_cents - int(off)


prices = st.integers(min_value=0, max_value=10_000_000)
percents = st.integers(min_value=0, max_value=100)


@given(prices, percents)
def test_bounded(p: int, pct: int) -> None:
    assert 0 <= apply_discount(p, pct) <= p


@given(prices, percents, percents)
def test_monotonic(p: int, a: int, b: int) -> None:
    lo, hi = sorted((a, b))
    assert apply_discount(p, hi) <= apply_discount(p, lo)


@given(prices, st.one_of(st.integers(max_value=-1), st.integers(min_value=101)))
def test_rejects_invalid(p: int, bad: int) -> None:
    with pytest.raises(ValueError):
        apply_discount(p, bad)

Mutation testing: are the tests any good?

Mutation testing makes small deliberate bugs ("mutants") in your code, such as < → <=, + → -, or removing a condition, and runs your tests against each one. If the tests still pass, the mutant survived: you found a gap. The mutation score (killed / total) is a far better quality signal than line coverage.

AI's role: paste the surviving mutants and ask "write the minimal tests that kill these mutants, deriving expectations from the spec". This is one of the highest-value uses of AI in testing. Mutation runs can be slow, so run them per module or nightly.

Test data and fixtures

• Realistic but fake: "Generate 20 realistic orders as JSON: mixed SKUs, some pickups, some returns, edge cases (zero-price items, max quantity, unicode names)."
• Never real customer data in prompts or fixtures. Ask for synthetic data that matches the schema.
• Generators over fixtures: ask the AI to write ScalaCheck Gens or Hypothesis strategies for your domain types, then reuse them everywhere.
• Flaky tests: paste the test, the failure output from several runs, and the code. Ask for causes ranked by likelihood (time, ordering, shared state, concurrency) and how to make it deterministic.

AI-assisted code review

Locally, before you open the PR

1. Self-review with a fresh reviewer agent (L2 page 08) on git diff main.
2. Targeted passes instead of one generic review: security, concurrency, error handling, performance, API compatibility. Each is a separate prompt with a checklist.
3. Fix or consciously dismiss each finding. Note dismissals in the PR description.

Review this diff ONLY for concurrency and resource-safety issues in Cats Effect code:
blocking calls on the compute pool, missing `IO.blocking`, unbounded parallelism (`parTraverse` on
large lists), leaked resources (not using `Resource`), shared mutable state, race conditions on Ref,
missing timeouts on external calls. For each finding: file:line, why it matters, a concrete fix.
If you find nothing, say so. Do not comment on style.

In CI

• Many platforms offer AI review bots that comment on PRs. Treat their comments as suggestions from a junior reviewer: often useful, sometimes wrong, never a substitute for an approving human.
• Put your team's review checklist in the bot's instructions (or the repo's AGENTS.md / review rules) so it reviews against your standards.
• Measure: track how many bot comments lead to changes. Tune or switch off noisy categories.