Activation steering controls large language models at inference time, but its effectiveness depends on the choice of reference data.
We introduce genesis steering, which derives a genesis vector from contrastive examples in the model's own fine-tuning data.
When these data are unavailable, we draw on the membership inference literature to identify likely training examples and construct a proxy dataset.
We establish two theoretical results on optimal displacement to the decision boundary and reduced proxy-estimation bias.
We evaluate genesis steering and membership-based proxy construction on Tülu-3-8B, whose fine-tuning data are publicly available, and Llama-3.1-8B-Instruct as a control.
We find that safety data from Tülu-3-8B's training mixture yield stronger steering than external datasets on both models.
On Tülu-3-8B, membership scores largely agree in identifying training examples, and vectors built from top-ranked examples achieve competitive performance against random subsets of the same size.
In contrast, the results on Llama-3.1-8B-Instruct hint that a MIA method will only have a positive effect if the mean direction shift with MIA-filtered in-distribution data is much less severe than the direction shift with OoD data.