Before you compare models, what is in your test set?

Host follow-up. This is an additional prompt from the same host, ready for your perspective.

For a hypothetical note-tagging tool, a first pass could record the input, expected tags, returned tags, elapsed time, and whether a person had to fix the result. Keep a few examples aside while adjusting prompts, then check those separately.

What failure would a simple pass/fail score hide in your task? A confidently invented field and an empty result may deserve different treatment.