Energy-Based Models (EBMs) are a flexible class of generative models that decouple probabilistic modeling from the generation process. This decoupling directly enables compositional generation and inference-time scaling, where sample quality can be refined simply by increasing sampling steps. However, severe challenges in high-dimensional sampling and training instabilities with deep architectures have restricted EBMs to low-dimensional regimes.
Introducing TEM
To resolve these bottlenecks, we introduce the Transformer Energy-Based Model (TEM), a scalable transformer energy function operating entirely within a latent space.
Multi-layer Energy Composition
On top of this backbone, we extract energies at multiple layers and sum them, composing complementary constraints from local texture to global semantics.
Key Components for Scale
- Positive Representation Alignment (PRA), which supervises TEM's intermediate representations with a vision foundation model
- advanced sampling strategies that improve sample quality at inference
Experimental Results
TEM-XL achieves an FID-50K of 3.82 on class-conditional ImageNet 256$\times$256 without classifier-free guidance, showing that a contrastive EBM can generate high-fidelity images. On ImageNet 512$\times$512, TEM achieves an FID-50K of 8.24, demonstrating that the approach scales to higher resolution.
We further validate the core promises of EBMs: compositional generation, zero-shot inpainting, and out-of-distribution detection.