MedJev: Evaluating Zero-Shot System-One Decision Making in Medicine

MedJev is an open medical language model small enough to run on a laptop, evaluated across ten clinical benchmarks as a fast, zero-shot decision maker. It approaches published GPT-4 accuracy on medical examination questions and predicts in-hospital mortality well, but is weaker on emergency-department triage and poorly calibrated for patient-level predictions; a two-parameter recalibration fitted on 50 to 200 labelled examples reduces the calibration error.