I'm building a binary classifier to decide whether to run a small LLM (2B parameters) on a question or skip it. Ground truth is whether the model can answer correctly (~15% of questions). can answer cannot answer run 1 4 skip 2 0.5 Costs are slightly arbitrary, based on relative costs to run and a 2/1 ratio for wrong answers. I tried to build something logic, with expected asymetry. This gives optimal threshold p* = 3.5 / 4.5 ≈ 0.78 via Elkan...
Originally asked by Lucas Morin on Stack Exchange (ai) (CC BY-SA 4.0).
