Small decisions. Interesting possibilities.Submit contentSubmit

How it uses Jev

The experiment evaluates five frozen weather-agent runs with two Jev judgments: a binary pass/fail decision and a continuous quality score. It compares repeated decisions with a human pass/fail label and measures score repeatability across repetitions.

What Jev decides

Judge whether the final weather answer is groundedExample answers · not a recorded Jev response · Source ↗
Question 1 · is grounded
YOUR APP
INSTRUCTION

Does the final answer stay grounded in the supplied search evidence and avoid inventing weather details?

STATE

One evaluation of a fixed weather-agent run. State fields are user_question, expected_behavior, final_answer, tool_calls, and search_evidence; the search-evidence list defaults to empty if absent.

JEV · NOUL
YesNo

App workflow

  1. Freeze agent outputs

    Run the weather agent on five fixed dataset cases and store outputs and evidence.

  2. Repeat judge evaluations

    Evaluate the same frozen runs with Jev and comparator judges over repeated runs.

  3. Collect human labels

    Review each fixed response against the same pass/fail rubric.

  4. Measure outcomes

    Compare binary decisions with human labels and calculate variance for continuous quality scores.

  5. Report scoped results

    Summarize accuracy, reliability, latency, and cost with the experiment’s dataset limitations.

Why it is interesting

Freezing each agent run isolates variation in the judges, while the project separates agreement with the human labels from reliability of the continuous score. Its README scopes results to a small five-case dataset and one human reviewer.