We interpret Transformer networks as learned approximations to a hybrid probabilistic-inference procedure that combines max-product Loopy Belief Propagation (LBP) with Expectation--Maximization (EM).
In this view, a Transformer performs inference in partially observed Markov Random Fields (MRFs) whose structure and parameters are not given a priori but are inferred from data.
We show that self-attention implements a differentiable approximation to max-product LBP on an input-conditioned, dynamically structured MRF, with attention weights encoding learned structural affinities.
We further establish an EM correspondence at the level of depth: each layer updates latent representations in an E-step-like forward pass, while the transition $θ^{(l)} \to θ^{(l+1)}$ across layers acts as an implicit M-step.
Building on this interpretation, we introduce meta-models, a design paradigm in which inputs generate instance-specific parameters, unifying mechanisms such as self-attention, mixture-of-experts, and gated units within a single framework.
Empirical results support the proposed perspective.