I've been using JEV in my n8n automation workflows for the past 2-3 weeks. One thing I still haven't been able to figure out is the reason behind the confidence scores. It really matters what input/prompt you provide and exactly how you use it, whether if it's for classification, judgment, routing, etc. But I have yet to figure out the confidence score problem...
Thanks for this interesting and very thorough overview. It looks promising but as you note their tests could be more in depth. I'm sure I'd have some use case for it but off-hand I cannot think of one. A local version would be welcome.
The reliability test in the post judges only five saved weather-agent responses 100 times each, so the 500 matches with the human reviewer really show repeatability on five items. Low variance on the same input does not by itself show the scores are calibrated across a wider set of outputs. Have you seen Jev run on a larger and more varied set of responses and compared against human labels?
I've been using JEV in my n8n automation workflows for the past 2-3 weeks. One thing I still haven't been able to figure out is the reason behind the confidence scores. It really matters what input/prompt you provide and exactly how you use it, whether if it's for classification, judgment, routing, etc. But I have yet to figure out the confidence score problem...
Thanks for this interesting and very thorough overview. It looks promising but as you note their tests could be more in depth. I'm sure I'd have some use case for it but off-hand I cannot think of one. A local version would be welcome.
The reliability test in the post judges only five saved weather-agent responses 100 times each, so the 500 matches with the human reviewer really show repeatability on five items. Low variance on the same input does not by itself show the scores are calibrated across a wider set of outputs. Have you seen Jev run on a larger and more varied set of responses and compared against human labels?