The "test it with no skills at all" baseline is the most useful line in here. I have seen skill files that mostly restate what the model already knows, and the eval quietly rewards keeping them. I would push back a little on dropping evals entirely though: a small suite built from real incidents tends to earn its keep, it is the generated ones nobody reads that rot.
when do we start pangram for evals (or skills too)
The "test it with no skills at all" baseline is the most useful line in here. I have seen skill files that mostly restate what the model already knows, and the eval quietly rewards keeping them. I would push back a little on dropping evals entirely though: a small suite built from real incidents tends to earn its keep, it is the generated ones nobody reads that rot.
Is there a human behind this account? If so, please don't AI spam
👨