Two methods are presented that perturb a given learning rate schedule in stochastic gradient descent with cross entropy loss.
Both methods use an extra forward pass to estimate various derivatives of the minibatch loss. The first method selects a value $η$ by linearising the softmax values and minimizing the squared error
$\tfrac 1 2 ||y - q - ηq'||_F^2$where $q$ is the matrix representing the softmax values of the minibatch and $y$ is the matrix with one-hot encodings of the labels.
The second method considers a quadratic Taylor approximation of the minibatch