Can Jev Grade Open-Response Assessments? We Tested It Against a Real Human Grader

This is a real test of AI scoring open response assessment, not a vendor demo. We ran Jev against 51 real exam answers that already had a human grade attached, logged every raw response, and compared the two. Jev matched the human grader exactly on 35.3% of answers, and landed within one rubric level on all of them.

That is the honest result, sample size attached, not a marketing claim. There is also a gap we will not hide: the dataset gives one grader per answer, not two, so there is no human-to-human agreement rate to compare Jev against. We explain why below.

This follows our broader look at the Jev typesafe AI system and Jev for learning and development.

Why is open-response assessment rare in corporate training?

Most corporate training assessment is multiple choice because it scores instantly at zero grading labor. Open-response questions, asking a learner to explain a decision or write a procedure, need a human to read and judge every answer, which does not scale past a small cohort.

A compliance course running a thousand learners a quarter cannot put a subject-matter expert behind every “explain why you would escalate this incident” question without turning training into a full-time job. So most LMS content defaults to recognition questions, cheap to score but weak at measuring judgment, the kind that matters in compliance training with audit evidence or competency-based training in aviation. Organizations avoid open response not because it is a worse test, but because grading it at scale has never been cheap.

How does a model that cannot read aloud actually mark an answer?

Jev scores an open-response answer against a rubric of graded descriptions, not a single “correct answer” string, and returns a level plus a confidence value per call. It grades on whether content matches the rubric level, not tone or length.

For this test we wrote a three-level rubric matched exactly to the human grader’s own 0-2 scale, not a four-level one that would make the comparison meaningless: does not address the question or is substantially incorrect; partially correct, incomplete or with errors; and fully correct and complete. That matching is the same principle behind how to write Jev questions and rubrics: the rubric does the work, and a sloppy one produces a sloppy score regardless of the model. Every call used the pinned model version jev-1.13.0, so the numbers below are reproducible.

How did we test Jev’s assessment scoring?

We tested Jev against 51 real open-response exam answers from a public dataset, each with one human-assigned grade, using a rubric matched to that scale, and logged every raw response. No internal marked assessment data existed, so we sourced a public substitute instead of fabricating one.

We used the DigiKlausur/ASAG-Dataset on GitHub, genuine exam short answers from a university neural-networks course, graded 0 to 2 by one human evaluator. This is computer-science exam material, not workplace compliance content, and we will not pretend otherwise.

The field’s standard benchmark, the Mohler ASAG corpus, uses two independent graders per answer and would have let us report human-to-human agreement alongside Jev’s. That corpus is mirrored on Hugging Face, but was unreachable from our test environment’s network on the day we ran this, so we used the DigiKlausur set instead. That is the honest reason for the substitute, not a design choice we would defend as ideal.

Every answer was scored once, pinned to jev-1.13.0, on 22 September 2026, with no context beyond the question and reference material provided.

What did Jev agree with the human grader on?

Jev matched the human grader’s exact score on 35.3% of the 51 answers, and every answer fell within one rubric level of it. On a three-level scale, that “within one level” number is a softer bar than it sounds.

Here are the full results from the run.

The reason 100% within-one-level sounds impressive and is not: on a three-level rubric, “within one level” covers two of three outcomes for any answer, a far lower bar than the same phrase on the five-level rubrics many competency assessments use. Report exact agreement alongside it, or the within-one figure will mislead.

Why is there no human-to-human baseline in this test?

There is no human-to-human baseline here because the dataset has one grader per answer, and the two-rater benchmark that would have supplied one, the Mohler ASAG corpus, was unreachable from our test environment on the day we ran this. We are stating that plainly rather than letting 35.3% imply a comparison that does not exist.

Without a second grader, we cannot say whether 35.3% is close to, better, or worse than two humans would agree with each other. Published inter-rater work on short-answer grading generally puts trained graders well short of perfect agreement themselves, so a real baseline matters, and we did not have one. That is a limitation of this test, not a claim that AI scoring equals a baseline we never measured.

We are saying this loudly because most vendor claims about AI grading skip this step. One AI grading vendor’s own published figures cite “90% learner satisfaction,” which is not agreement with a trained grader. Admitting a missing baseline, and explaining why, is more credible than a claim that never shows its method at all.

Where did Jev get it wrong?

Jev disagreed with the human grader on 33 of the 51 answers, and every disagreement was off by exactly one rubric level, never two. The model was never wildly wrong, just one level off, both directions.

The pattern is not a simple “too generous” or “too harsh” story. Jev under-scored fuller, technically correct answers more often than it over-scored weak ones, but it did both, sometimes at similar confidence. Three real examples show the range.

On “Assume you want to binary classify linear separable data points using a perceptron. Write down the learning algorithm in pseudo code!”, the answer had correct, working pseudocode. The human grader scored it a 2. Jev scored it a 1, despite its own confidence being 0.82.

On “Describe how learning based on k-nearest neighbors works, use pseudo code!”, the answer correctly defined the algorithm. The human grader again gave a 2; Jev gave a 1, at 0.80 confidence.

Those two under-scored a correct answer. A third example went the other way. For a question about Restricted Boltzmann Machines, one answer read only “Neurons have to states e.g. on or off. Each neuron has a probability to flip between them,” vague and incomplete. The human grader scored it a 0; Jev scored it a 1, at 0.84 confidence.

Put those three together and the finding is not one consistent bias: Jev’s roughly 0.8 confidence band held both an under-scored correct answer and an over-scored vague one. A confidence score in the low-0.8s did not tell you which kind of error, if any, was coming.

What did it cost, and how long did it take?

Scoring the 51 answers cost roughly $0.0187 per 1,000 answers, and each call took about 0.38 seconds, a different order of magnitude from a human grader averaging well over a minute per answer. Cost and speed are not why you would hesitate to automate this. Accuracy at trustworthy confidence levels is the real constraint, covered next, and if you track either metric over time, that is a number a reporting and analytics setup built around data that matters should capture, not just pass rates.

At what confidence level could you actually automate this?

The most surprising finding of this project is that Jev’s roughly 0.8 confidence band, not its low or middling ones, was the least reliable range measured. Across a bucketed accuracy curve run over this project’s tests, the 0.8 bucket came in at 9.1% accuracy, while 0.0-0.2 and 0.9-1.0 were well calibrated at 90-100%.

That is close to the opposite of what “80% confident” should mean. Six calls in this test’s 51 answers landed in the 0.75-0.84 range, rounding into that 0.8 bucket, and every one was wrong. Sounding moderately confident was, in this sample, one of the worst signals Jev produced.

The two ends of the curve behaved the way a confidence score should: 0.9-1.0 was well calibrated, and so was 0.0-0.2, though only the 0.0 bucket had a sample large enough to clear a 95% accuracy bar. The failure sits in the 0.3-0.8 middle range, where a score that sounds reasonable is one of the least trustworthy things Jev reports.

Practically, a mid-band confidence score is not a green light to auto-finalize a grade. It is closer to a flag for a second look. We go deeper on this in Jev’s limitations and confidence calibration, required reading before setting any auto-approval threshold.

What does this not solve?

This test does not show Jev can replace a human grader, and it does not solve feedback: Jev returns a score and a confidence value, no rationale. A learner marked wrong with no explanation cannot learn from the result, defeating the point of an assessment meant to build a skill.

That makes this a triage tool on the evidence we have, not a marking replacement for anything that needs to explain itself. It fits flagging answers a human should look at first, or high-volume low-stakes checks where a wrong call is cheap to fix, and a poor fit wherever the learner needs to understand what they got wrong, which is most compliance and competency contexts. See Jev’s known limitations around calibration and confidence for the fuller picture.

It also has not been tested on the content most L&D teams grade. Neural-network pseudocode is not a customer-service de-escalation script, and nothing here should stretch to cover those without a fresh test. If your assessments live inside a microlearning module builder with short scenario questions, run this method on your own content first.

Should you use AI to mark assessments?

Use it to triage, not to finalize. With no rationale returned and a confidence score least reliable in the middle of its range, Jev is not ready to stand in for a marker anywhere the learner needs to know why they were scored the way they were. Where it earns its cost is volume: pre-sorting a batch of open responses and surfacing the ones a human should look at first. That is a real, useful job, and a different job from grading the assessment.

Conclusion

The number to remember is 35.3% exact agreement against one human grader on 51 answers, with no second grader to compare against, and a confidence score that misleads in its middle range more than it helps. That is not a reason to dismiss AI-assisted marking. It is a reason to build the workflow around what the evidence shows, not what a sales page implies.

If you are evaluating automated scoring for your own assessments, run this same test on a sample of your own graded content first: same rubric-matching, same logging of raw responses, same honesty about what baseline you do or do not have. That is the only way the result tells you something about your learners, not about a neural-networks exam.