Jev as a Judge
An evaluation harness comparing Jev with LLM judges on the same recorded agent runs.
An evaluation harness comparing Jev with LLM judges on the same recorded agent runs.
The experiment evaluates five frozen weather-agent runs with two Jev judgments: a binary pass/fail decision and a continuous quality score. It compares repeated decisions with a human pass/fail label and measures score repeatability across repetitions.
Does the final answer stay grounded in the supplied search evidence and avoid inventing weather details?
One evaluation of a fixed weather-agent run. State fields are user_question, expected_behavior, final_answer, tool_calls, and search_evidence; the search-evidence list defaults to empty if absent.
Run the weather agent on five fixed dataset cases and store outputs and evidence.
Evaluate the same frozen runs with Jev and comparator judges over repeated runs.
Review each fixed response against the same pass/fail rubric.
Compare binary decisions with human labels and calculate variance for continuous quality scores.
Summarize accuracy, reliability, latency, and cost with the experiment’s dataset limitations.
Freezing each agent run isolates variation in the judges, while the project separates agreement with the human labels from reliability of the continuous score. Its README scopes results to a small five-case dataset and one human reviewer.