Practice · Answer evaluation

Build answer fixtures before testing a model

Use a source-aligned answer and deliberately altered fixtures to check an evaluator’s decisions before measuring model outputs.

Jason Colapietro · Suede AI SEO · · Updated

Give the evaluator a known answer

An answer evaluator needs acceptance criteria you can inspect. The Suede AI example uses the public credential inventory as its reference and four hand-authored answer fixtures as inputs. One matches the source. Three deliberately change a category total, remove a source URL or rewrite the relationship.

The dated run records eight checks per fixture and four expected decisions matching the actual accept-or-review decisions. That result tests the evaluator on these saved examples. The runner uses zero paid model calls, so its output is a useful starting point before you collect responses from a model.

Write criteria from the source fields

Read the named checks in the public results.json alongside the authored answer fixtures. They cover the holder, course total, training achievement total, membership total, Microsoft award types, provider attribution, source URL and relationship. Each saved result has a name and a Boolean decision.

The source-aligned fixture is accepted only when all checks pass. The deliberately conflated fixture fails three checks: course total, training achievement total and Microsoft award types. Keeping the check names in the output makes the failure useful for correcting the answer or refining the input contract.

Inspect the saved fixture decisions

Use Node 22 or newer to read the recorded accept-or-review decisions. Open the linked fixtures and credential inventory beside the report. This command displays saved results; implement your own evaluator against the criteria when testing a new answer.

report_file="$(mktemp)"
curl --fail --show-error --location --max-time 30 https://suedeai.ai/seo/assets/demonstrations/results.json --output "$report_file" &&
REPORT_FILE="$report_file" node --input-type=module <<'JS'
import { readFileSync } from 'node:fs';
const result = JSON.parse(readFileSync(process.env.REPORT_FILE, 'utf8'));
for (const fixture of result.evaluation) {
  console.log(fixture.name, fixture.accepted, fixture.checks);
}
JS

Add cases that expose an actual gap

This evaluator expects a structured answer. It is a small, readable rubric for this inventory. Its provider check accepts a non-empty subset of known providers, so it does not enforce a complete provider list. That distinction belongs in your acceptance criteria if completeness matters.

Before expanding it, write the new expected behavior and add a fixture that currently fails. For example, a workflow requiring every issuer should compare the required set explicitly. A free-text answer needs an extraction or review step before these structured fields can be checked.

Move from authored fixtures to model responses

Use the published credential-answer-prompt.md as a starting input. Retain the model identifier, settings, complete prompt, date and original response when you collect a model answer. Apply the named acceptance criteria to the response and retain each failed check before summarizing the sample.

Building with the Claude API, Craft Precise Prompts for AI Models and Academy Accreditation - Generative AI Fundamentals are the issued training records connected to this exercise. The practical deliverable is a reference set, a readable rubric and a repeatable decision. Claude’s evaluation guidance is a primary reference for defining measurable criteria; your application supplies the actual cases and success threshold.

Issued training connected to this exercise

Inspect the full training inventory · Open the qualifications sheet

Source and recorded result

The demonstration report is dated 2026-10-21. The commands and checklists above explain how to inspect that saved record or capture a new observation from a public page.

Put the workflow to work

Discuss an evaluation workflow. Bring the workflow, source page or answer your team needs to check so the first deliverable has clear acceptance criteria.

Check credential counts and relationships · All three working demonstrations