Trainable at Depth, Solvable at Small Size: What the Loss Surface of a Polynomial Activation Dictates for Designing, Initializing and Training Networks

What makes a deep network train, and what shape is its loss surface?

We answer both with one construction at two scales: a polynomial activation f (z) = z + λz³, clamped and projected onto the unit sphere, plus an unnormalized identity path.

As a deep stack it trains to one hundred layers on MNIST with no normalization machinery; as a small model its square loss is a polynomial system we solve exactly.

The small scale explains the large one: it prescribes how to design, initialize and train the stack.