Large language models provide a variety of responses at inference (generation) that conventionally depend on a temperature parameter.
Using a conditional mutual information approach, a quantitative, entropy based analysis of response variety is given.
Three limiting behaviors are defined:
- arbiter
- independent (conditionally)
- neutral (deterministic)
that span no to maximal mutual information.
We give examples of each behavior and suggest their application to:
- temperature analysis
- language analysis
- prompt compression
- knowledge graphs