Decomposing Task Vectors for Refined Model Editing

Large pre-trained models have revolutionized machine learning, yet adapting these models effectively to exhibit precise, task-specific behaviors remains a significant challenge. Task vectors, defined as the difference between fine-tuned and pre-trained model parameters, provide a mechanism for steering neural networks toward desired behaviors. This has given rise to large repositories dedicated to task vectors tailored for specific behaviors. The arithmetic operation of these task vectors allows for the seamless combination of desired behaviors without the need for large datasets. However, the mechanism by which these vectors interact within these operations has yet to be thoroughly explored. We hypothesize that task vectors encapsulate sub-behaviors within their underlying subspaces. To explore this, we propose a task vector decomposition method that, given a set of task vectors exhibiting specific behaviors, separates shared subspaces from task-specific components. This way, the arithmetic operations can be performed without degrading the performance towards the desired outcome. Through extensive experiments, we show the potential of this approach in both visual and language domains while using off-the-shelf, pre-trained, and fine-tuned models. We show the application of our approach in visual recognition, style mixing in diffusion models, and removing toxicity in language models, leading to significant improvements.

1

Submitted to International Conference on Computer Vision and Pattern Recognition (CVPR) 2025