Human--LLM interaction datasets shape our understanding of AI use and provide a foundation for downstream research, including training and evaluation of user models.
In recent years, a growing number of datasets have sought to capture a representative picture of human--LLM interactions.
But how different are the pictures these datasets provide, and what do those differences mean for research built on them?
We study these questions across seven conversation datasets, spanning in-the-wild chat logs and human preference data.
We begin by revisiting the dataset classification experiment of Torralba & Efros
We find that neural network classifiers identify the source of a conversation from user messages alone well above chance, indicating distinctive dataset signatures.
This separability persists after matching datasets on the dimensions of human-designed taxonomies, implying subtle differences that these taxonomies do not capture.
We then examine the implications for user modeling
- How dataset signatures propagate to the outputs of user models trained on these datasets;
- How dataset choice influences evaluations of user model quality and subsequent evaluations of LLM assistants paired with these user models;
- How dataset classifiers can guide data selection for training user models.
While each dataset is meant to capture a slice of 'real-world' interactions, our findings reveal the extent to which these slices diverge, and the consequences of those differences for research built on these foundations.