How to Test a Decision Model in Your Coding Loop

Takeaways

• Jev and winnow:e4b matched the reference on all 12 Claude Code fixes, Claude in a plain message matched 9, and a filename rule matched 3.
• Only one of the 12 fixes left a bug behind, and all four pickers caught it, so the test measures fit and says nothing yet about catching more bugs.
• The 7.6-microsecond filename rule caught that bug too, even though its targeted pick missed the reference.
• A 0.50 cutoff set with --min-confidence would have stopped 9 of winnow:e4b's 36 picks, all matching the reference, and none of Jev's, whose lowest confidence was 0.62.
• On a first look, a plain Claude call costs $0.0105, about 150 times Jev's per call, since the $0.0017 cached price only came from resending the same packet.