Sometimes less is more: Investigating inductive biases of MLPs and Transformers on noisy k-parity features

We empirically investigate how MLPs and Transformers differ when trained on data sets containing one or more features encoded by noisy k-parity. Noisy k-parity gives us hard synthetic data sets in which we can control the predictivity and availability of every feature. Across a range of data distributions, data scales and model sizes, we find large differences in the training dynamics and inductive biases of the two architectures: Transformers are more robust to noisy features but struggle to learn complex, low-availability features, whereas MLPs show the opposite trend and improve steadily with more data. We also find that, under a fixed training budget, increasing the number of unique training examples can slow Transformer optimisation enough to reduce final accuracy, so that a smaller data set performs better. Training-accuracy results show this is an optimisation-speed effect; it shrinks, but does not vanish, when the budget is scaled with the data set size. By giving precise control over feature predictivity and availability, this testbed offers a clear starting point for understanding when and why Transformers and MLPs learn differently, and for designing data and training budgets that exploit those differences.