This paper develops a stochastic dynamical-systems perspective on three classes of generative AI models.
First: Large Language Models as High-Dimensional Nonlinear Autoregressive Models
First, we formulate large language models (LLMs) as high-dimensional nonlinear autoregressive models with attention-based dependencies. This formulation connects pretraining by conditional maximum likelihood to sequential token generation and provides a probabilistic interpretation of LLM operation.
Second: Networks of Interacting LLMs and Mean-Field Dynamics
Second, we consider networks of interacting LLMs and characterize their collective behavior through mean-field ordinary differential equations. These dynamics provide a framework for understanding emergent preferential attachment and persistent structural biases, including type-dependent glass-ceiling effects.
Third: Neural Stochastic Differential Equations as Continuous-Time Generative Models
Third, we consider neural stochastic differential equations (SDEs) as continuous-time generative models for stochastic trajectories, including those observed at irregular time intervals. We discuss their training with particular