What reusable computation should a tabular foundation model learn when every table defines a new supervised task?
We develop in-situ representation refinement: support labels guide updates to the episode's representations, and these updates transfer to unlabeled queries without changing model parameters.
A regularized leave-one-out objective yields a support correction and its query extension.
The leading term separates attention-based reading from state-dependent scaling, motivating RefineICL: an attention-gated, FFN-free contextual stack with selected low-rank feature interaction and typed memory.
RefineICL-L24 reaches 0.93836 OVR-AUC and 0.87173 accuracy on AMLB29.
A benchmark-informed continuation reaches 1644.8 Elo on the 38-dataset TabArena snapshot, 31.4 Elo above TabPFN-3 under the same evaluation.
It also improves all four reported metrics over TabPFN-v3 on both TabZilla views.
In a matched 100K-update depth grid, an expanded FFN gives no consistent validation benefit and uses 60.2% more peak inference memory at L8.
Internal interventions show that support representations are more than a static source of labels: removing one intermediate support update, while preserving the query output, increases final query cross-entropy in all 72 tested episodes.
Together, the derivation and interventions explain how attention-gated updates can construct a task-specific predictor in context.