首页 > AI前沿 > When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation

When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation

arXiv自然语言 2026-05-20 02:16 5 阅读 查看原文

Automatic evaluation of long-form literary writing remains challenging, particularly for creativity-related dimensions such as originality and flexibility.

We address the lack of large-scale supervision for structured Torrance Test of Creative Writing (TTCW)-based evaluation by constructing a dataset of 263{,}911 long-form stories, each paired with scalar scores and meta-synthesised review comments across 14 TTCW-based dimensions.

Using this dataset, we study whether reviewer-style reasoning traces improve structured literary review generation across Qwen3-4B, Qwen-8B, and Nemotron-8B.

Across all three models, no-trace supervision yields higher score imitation fidelity and near-complete report generation, whereas trace supervision reduces completion to $85.9$--$90.9\%$.

Failure analysis shows that $88.6\%$ of failed reasoning generations contain semantic repetition loops inside the trace.

These loops concentrate in particular story--criterion interactions and progressively harden after onset, while comparable repetition is rare in the supervision data.

Our results suggest that, for long-form tasks requiring strictly structured outputs, explicit reasoning traces can introduce generation instability that outweighs their overall benefit.