A Mathematical Framework for Transformer Circuits (2021)

Algorithm for measuring marginal loss reduction of nth order terms Step 1: Run model, save all attention patterns. Step 2: Run model, forcing all attention patterns to be the version you recorded, and instead of adding attention head outputs to residual stream, save the output, and then replace it with a zero tensor of the same shape. Record resulting loss. Step n: Run model, forcing all attention patterns to be the version you recorded, and instead of adding attention head outputs to residual stream, save the output, and replace it with the value you saved for this head last time. Record resulting loss.