Extracting Concepts from GPT-4

Scaling Sparse Autoencoders to 16 Million Features

Using new techniques for scaling sparse autoencoders, we automatically identified 16 million patterns in GPT-4's computations.