Iterating on vibes is not iterating.

Rewrite a prompt, run two examples, feel that it is "maybe a bit better", and ship. The next day a different example feels worse than before. That is not iteration. It is a random walk.

Only one thing separates the two: a fixed set of questions against which two versions can be measured with the same ruler.

Twenty real examples are enough to start.

An eval set does not need to be perfect on day one. Pull twenty real requests from the logs, hand-write the expected output, and you already block most obvious regressions.

Above all, fold in the cases that once went wrong. You paid for those. They should not live only in your memory.

Scores only need to be comparable.

People stall on "how do I score free-form text" and never begin. But a score need not be absolute. It only needs to express a consistent preference between two versions.

Even a quick human verdict — right, wrong, unsure — becomes a baseline as long as the questions and the standard stay fixed. Once a baseline exists, a change becomes something you can actually argue about.