Large language models (LLMs) have demonstrated several emergent behaviors with scale, including reasoning and fluency in long-form text generation.
However, they continue to struggle with tasks requiring precise spatial and positional reasoning.
ASCII art, a symbolic medium where characters encode structure and form, provides a unique probe of this limitation.
We introduce ASCIIBench, a novel benchmark for evaluating both the generation and classification of ASCII-text images.
ASCIIBench consists of a filtered dataset of 5,315 class-labeled ASCII images and is, to our knowledge, the first publicly available benchmark of its kind.
Alongside the dataset, we release weights for a fine-tuned CLIP model adapted to capture ASCII structure, enabling the evaluation of LLM-generated ASCII art.
Our analysis shows that cosine similarity over CLIP embeddings fails to separate most ASCII categories, yielding chance-level performance even for low-variance classes.
In contrast, classes with high internal mean similarity exhibit clear discriminability, revealing that the bottleneck lies in representation rather than generational variance.
These findings position ASCII art as a stress test for multimodal representations and motivate the development of new embedding methods or evaluation metrics tailored to symbolic visual modalities.
All resources are available at https://github.com/ASCIIBench/ASCIIBench.