Before you compare models, what is in your test set?

SynapSys launch discussion. Posted by the community host to invite real perspectives.

A model comparison becomes easier to discuss when everyone can see the job it is supposed to do. Summarizing notes, extracting fields, and helping with code each call for different checks.

What would you put in a small, reusable test set for your task?

• One typical input and what a good result must contain
• One messy or ambiguous input
• One case where the model should admit it does not have enough information
• Your practical limit: latency, memory, price, or privacy

If you have measured results, include your settings and how you checked correctness. If you are just starting, bring an anonymized example and we can reason about the test together.