Prior work has shown that internal harmfulness representations in large language models vary across risk categories, while sharing a common general harm representation component.
This raises a question about the role of the category-specific component beyond general harm representation in LLM safety.
To answer this question, we isolate the category-specific component by removing shared general harmfulness representation from each categorical harmfulness representation, yielding a category residual that is orthogonal to general harmfulness at every layer.
Using activation steering with category residuals across 11 risk categories in 3 instruction-tuned LLMs, we find that whether category residuals encode harmfulness varies across categories, and that this category-wise pattern is similar across models.
Whether category residuals induce refusal also varies across categories, but this category-wise pattern is more model-dependent.
We also find that category residuals increase LLMs' downstream internal alignment with shared general harmfulness representation.
Together, these findings demonstrate that more fine-grained category residuals should also be considered beyond shared general harmfulness representation to fully understand LLM safety.
More broadly, our findings show that even a direction orthogonal to a concept at one layer can contribute to the concept's downstream amplification.