Next-visit diagnosis prediction forecasts the diagnoses documented at a patient's next encounter, turning retrospective electronic health records into prospective clinical prediction.
However, conventional aggregate evaluation obscures its multifaceted nature:
- distributionally, models must handle variable-size diagnosis sets and long-tailed labels;
- temporally, they must distinguish persistent, newly recorded, and recurrent diagnoses;
- and both vary with observed history depth.
We introduce an evaluation framework along these dimensions and apply it to:
- rule-based heuristics
- LLM-based prompting
- retrieval
- memory
- multi-agent approaches
This reveals that repeating the last visit outperforms most LLM-based approaches overall despite predicting no new or recurrent diagnoses, and that retrieval and memory exhibit distinct profiles whose relative strengths vary across temporal transitions and backbones.
Motivated by this, we propose LONGINUS, a retrieval-memory multi-agent adjudication framework combining
- evidence retrieved from patient records
- structured memory of the patient trajectory
Across MIMIC-III and MIMIC-IV with four LLM backbones, LONGINUS achieves the best transition-specific performance among LLM-based methods in nearly all settings while remaining competitive with them on aggregate metrics.