Testing an AI narrator when no two runs match

Testing an AI narrator starts by giving up on expected output. The same move, in the same scene, with the same memory behind it, comes back as different prose every time, so there is no string to compare against. What you test instead are properties the scene has to satisfy no matter which words arrive.

• A narrator test asserts properties, not text: a scene passes when it carries the right facts, keeps its promises, and honors the move that was written.
• Deterministic checks catch structure and format cheaply; graded checks, run by another model, are the only way to ask whether a meaning survived.
• A single run proves nothing, so every property is measured as a pass rate across many runs of the same frozen case.
• Regressions in generated prose appear as a drifting rate, not as a failed assertion.
• No automated check yet tells you whether a scene was worth reading.

Why expected output fails for generated scenes

A unit test holds a value still. Generated prose will not hold still, and two runs of one scene can share a meaning without sharing a sentence. Asserting on the string would fail on every harmless rewrite and pass on nothing that mattered. The useful question is not what the narrator said but what the scene did.

There is a worse failure than wrong words. A scene can be finely written and still forget that you buried the ring, still answer a question you did not ask, still quietly drop a promise a character made eight scenes ago. Those are the faults readers actually notice. None of them are visible to a string comparison, and all of them are visible to a check that knows what the memory held.

Temperature zero is not a fixture

Narrowing the sampling narrows the variation; it does not pin the text. Batching on the provider’s side, floating-point order, and the model updates that arrive underneath you all move the words. We treat that variation as a property of the system to be measured rather than a setting to be fought, which changes what a test suite is for: not proving one output, but bounding a distribution.

Which properties are worth asserting on

Our list came from reading failures, not from a framework. Each entry is something a scene must do for the story to stay coherent, stated so a check can answer yes or no. Fabledrift, an interactive fiction app for Android, hands the narrator a structured record of facts, characters, places and promises on every turn, which means each of those records is also something the next scene can be tested against.

Property	What the test asserts	How it is checked
Facts carried	Facts in memory that the scene touches are still true in the new prose	Graded check against the fact list
Move honored	The written action was attempted, and a spoken line came back in quotes	Deterministic checks plus one graded
Promise kept	An open promise is advanced, paid off, or left alone, never contradicted	Graded check against the promise record
Openings offered	The scene ends with two to four suggestions, none a repeat of the last set	Deterministic count and comparison
Voice held	The page reads as a book: no chat framing, no stat talk, no menu language	Regex plus a graded style check
Note obeyed	Where a reader’s note contradicts the story record, the note wins	Graded check against the note

The split in that third column is the whole economy of the suite. A deterministic check is cheap, exact and narrow: count the openings, match a quotation mark, reject the word “inventory.” A graded check costs a model call and returns a judgment, which is the only way to ask whether a buried ring survived a paraphrase. We spend determinism wherever it will stretch, because it is the half that never argues.

Where the records themselves are thin, no property can be stated. That is the quiet dependency between testing and authoring: designing a world for an AI narrator is also deciding what will later be checkable.

How testing an AI narrator works run to run

Each case is a frozen starting point — a world, a memory snapshot, a scene, one reader move — run many times and scored on the same properties every time. The result is not pass or fail but a rate: nineteen of twenty scenes carried the fact, sixteen of twenty kept the promise. A case goes green above a threshold set per property, strictest on memory.

Thresholds are a judgment, so we write down why each sits where it does. Forgetting a buried ring is worse than offering a thin suggestion, so facts carried must hold near the ceiling while the openings check tolerates the occasional sparse scene. When a rate drifts down after a prompt change, the change is the suspect. When it drifts with nothing changed on our side, the model underneath has moved.

The cases that earn their keep

The valuable cases are the awkward ones. A reader attempting something the world cannot allow, which should become a scene rather than an error. A reader who steps back and takes a different line, where the narrator has to honor the earlier state and not the abandoned one. A note that contradicts the record. A promise made in scene three and tested in scene eleven. We keep the long cases because story promises in interactive fiction only break at distance.

What the existing eval tools give you, and what they do not

The tooling has improved. Promptfoo’s assertion reference states that “assertions are used to compare the LLM output against expected values or conditions,” and splits them into deterministic checks — contains, regex, is-json, custom javascript — and model-graded ones such as llm-rubric, factuality and similar. That is exactly the two-layer arrangement a narrative suite needs. DeepEval describes itself as “similar to Pytest but specialized for unit testing LLM apps,” and ships metrics for hallucination, faithfulness and bias.

The discipline underneath both is older than language models. Hypothesis, the property-based testing library for Python, has you “write tests which should pass for all inputs in whatever range you describe” and then chooses the inputs itself. Swap generated inputs for generated outputs and the method transfers intact: describe the range, state the invariant, let the machine hunt for the case that breaks it.

What none of this knows is your story. A faithfulness metric assumes a retrieved document to be faithful to; a narrator is faithful to a record that changed last turn and will change again. A generic grader scoring a summary cannot tell you that a character who died in scene four has just spoken. So the rubric is the part you write yourself, and writing it is most of the work.

What still needs a person to read it

Every property can pass and the scene can still be dead on the page. Pacing, the shape of a sentence, whether a line of dialogue sounds like the person who said it — no check we have scores those honestly, and a model asked to grade prose rewards fluency over surprise. So one of us reads scenes every week, cold, with the scores turned off until afterward.

That reading is where new properties come from. A scene that felt wrong gets read twice: once for the fault, once for the sentence that would have caught it. About half the time the fault is already nameable — a dropped fact, an unhonored move — and a case goes into the suite that afternoon. The other half stays a note until it happens again.

Readers find the rest, which is why the repair is built into reading rather than into a support queue. Correcting an AI narrator with a note fixes the story in front of you, and the pattern of what people correct tells us which property we have not written yet.

What we check before a narrator ships

A prompt change runs the full case set on all three narrators before anyone sees it, and we compare rates to the last green run rather than to an absolute bar. A drop on memory stops the change outright. A drop on voice gets argued about, usually for a day. Then someone reads six scenes from The Emberwood and six from Ironreach Station end to end.

That last step is not ceremony. The suite measures scenes, and a reader lives in a story, which is a longer and more fragile object than anything a case can freeze. Property checks tell us the story will still make sense in scene forty. Only reading tells us anyone would want to get there.