Large language models (LLMs) are typically aligned with population-level preferences, despite substantial variation across individual users.
We introduce POPI, a user-level personalization framework that separates the problem into two components connected by a natural-language interface:
- a shared inference model that distills heterogeneous user signals into a concise preference summary,
- a shared generator that conditions on this summary to produce personalized responses.
Both components are trained under a unified preference-optimization objective, with reinforcement learning handling the non-differentiable inference step.
This objective decomposes into generator approximation error and summary informativeness, revealing how a single loss simultaneously drives accurate generation and informative summarization.
Because the interface is natural language, learned summaries can be inferred once per user and reused across different generators -- including frozen, black-box commercial APIs.
Across four personalization benchmarks, POPI generally improves personalization quality while reducing context overhead by up to an order of magnitude.