Process Reward Models (PRMs) play a central role in test-time scaling for large language model (LLM) reasoning.
However, PRMs may provide inaccurate process rewards for test-time scaling, which we identify as arising from two sources of uncertainty:
- (1) PRM uncertainty that arises from insufficient supervision data and imperfect PRM training;
- (2) policy uncertainty that results from the mismatch between the policy to collect supervision data and the policy for test-time scaling, leading to systematic reward overestimation.
Previous studies fail to jointly consider both uncertainties.
To bridge this gap, we propose BayesianPRM, which treats the process reward as a random variable within a reward hypothesis space and maintains a Bayesian belief over it.
Since exact Bayesian inference over the continuous hypothesis space is generally intractable, we discretize the reward hypothesis space with a reward ensemble and introduce two complementary belief updates, each targeting one source of uncertainty:
- to address PRM uncertainty, we introduce reliability-aware belief updating that measures how well each ensemble member explains the supervision data and updates the belief with an amortized belief network optimized by the evidence lower bound (ELBO);
- to tackle policy uncertainty, we propose conservatism-aware belief calibration which further reweights the belief toward lower rewards to suppress overestimation.
These two updates yield a reliability- and conservatism-aware Bayesian belief, whose expectation determines the process reward.
Experiments on multimodal reasoning benchmarks indicate that BayesianPRM consistently yields more accurate process reward estimates while enhancing test-time scaling performance.