Given the increased use of large language models (LLMs) for reading and seeking information and the varying needs and backgrounds of users, direct manipulation of text can be a valuable mechanism for end-user control.
However, enabling direct manipulation requires that models generate text consistent with user-requested changes.
We focus on one such setting, manipulating the complexity of scientific information, and ask if models can generate multiple summaries along a continuum of text complexity that enable effective user selection and control.
Through a within-subjects user study, we identify reliable complexity measures and a criterion for how models should conform to these measures.
We evaluate five LLMs of different families, sizes, and reasoning abilities on $557$ scientific queries.
We find that while models can vary complexity across responses, most changes remain inconsistent, with models shifting the complexity measures in the correct direction at most $33.67$% of the time.