Architecture

Store the Probability, Never the Verdict

September 23, 2026 3 min read

One design decision has paid for itself more than any other in the small AI systems I build: when a model makes a decision, store the probability, never the verdict.

It sounds like a database detail. It is really a decision about who gets to change their mind later, and how expensive that is.

The setup

The example comes from a side project, a live-stream avatar that reacts to what is being said. When a new word comes up in the transcript, the system has to decide whether that word deserves its own illustrated sprite. Generating a sprite is slow, costs an image generation and is capped per stream, so the decision matters.

The decision is made by a small judge model, Jev, from TypeSafe. It does not write anything. It answers one question, does this word deserve a sprite, with a probability between 0 and 1. A call uses about 375 input tokens and costs about $0.000016 at the public price of $42 per billion input tokens.

The tempting mistake

The obvious implementation compares the probability to a threshold, say 0.5, and caches the result: yes or no. Next time the word appears, you read the cache and skip the call.

The problem shows up the first time you move the threshold. And you will move it, because nobody picks the right threshold on day one. The moment you do, every cached verdict is wrong in a way you cannot see. Say a word scored 0.48: it was cached as "no", and it stays "no" after you lower the threshold to 0.45. To fix it, you have to call the model again for everything.

A verdict binds each decision to the threshold of the moment. A probability does not.

What storing the probability buys you

I keep a journal: one line per decision, with the word, the probability and the context. The journal survives restarts, because a threshold should be calibrated on a whole stream, not on the last three words.

On top of it sits a slider. Move it, and the interface highlights every word that would flip, and in which direction, if the threshold were there. No model is called. When the cut looks right, one button writes the new threshold to the live configuration, without a restart.

Calibrating a model's threshold went from "rerun everything and hope" to a short review of real decisions.

The cache also turned out to matter more than I expected. Across eight real sessions there were 16,098 candidate words, only 2,127 of them distinct. That is 87 percent repetition. Each call is almost free, but 87 percent of them never need to happen, and the ones that do not happen do not add latency to a live stream either.

Two related lessons

The first is that a model's confidence is not its correctness. The speech-to-text layer that fed this system sometimes looped on silence and produced the same phrase over and over. Those looping transcripts carried a higher median confidence than normal speech, 0.83 against 0.74. The model was most sure of itself exactly when it was stuck. A confidence threshold cannot catch that failure; only the shape of the transcript can.

The second is about failure modes. If the judge is unreachable, what should the filter do? My first answer was to fail open and let words through, so the stream stays lively. But every word let through triggers an image generation that cannot be taken back and eats into a cap of 50 per stream. So the filter fails open for a while, and after eight consecutive failures it inverts and starts rejecting, with a loud error in the log. The right fallback depends on which mistake is reversible.

Beyond a side project

The same rule applies anywhere a model makes a yes-or-no call inside software: content moderation, lead scoring, ticket routing, fraud flags, relevance filters in retrieval. Keep the number. Decide at read time. Log the threshold that was in force separately, so you can explain an old decision without confusing it with the current rule.

It costs a few bytes per decision. In exchange, every future change of mind is free.