30 April 2026 / Applied AI / 8 chapters

Write fixtures for the rules

From Designing an auditable AI capability assessment

Fixtures are small assessment cases with known inputs and expected rule outcomes. They give subject-matter and technical reviewers concrete examples to argue about, which is easier than both of them staring at rule expressions.

Start with one fixture per level or profile, then add cases around every threshold, prerequisite and exception. Include incomplete evidence, contradictory answers, not-applicable values and inputs that should fail validation. If more than one rule can produce the same label, cover each route to it.

A useful fixture has:

  • a short note on what it's testing
  • the methodology version
  • the structured answer values
  • which rules should match and which shouldn't
  • expected intermediate values
  • the expected result and reason codes
  • any warning or validation error you expect

Keep raw conversation text out of most scoring fixtures, because the scoring suite should be testing the deterministic part directly. Keep a separate set of mapping examples for the conversation layer: a raw answer, the structured proposal you'd expect, its ambiguity status and whether it needs confirmation. Model behaviour varies, so judge those on acceptable mapping outcomes rather than exact wording.

When a rule changes, run the whole fixture suite and look at every result that moved. If you just update the expected outputs until the tests go green, you've thrown away the reason for having them. Each changed fixture needs an explanation from the methodology owner. Some changes are intended, some show a knock-on effect nobody thought about, and some show the rule was implemented wrong.

Add property checks where the method supports them. If increasing a clearly positive input should never lower a dimension, test that across generated values. Prerequisites and penalties can mean a better input doesn't always raise a score, though, so only test properties the documented method actually promises.

All articles