Abstract:Fine-tuning has become the standard practice for adapting pre-trained models to downstream tasks. However, the impact on model robustness is not well understood. In this work, we characterize the robustness-accuracy trade-off in fine-tuning. We evaluate the robustness and accuracy of fine-tuned models over 6 benchmark datasets and 7 different fine-tuning strategies. We observe a consistent trade-off between adversarial robustness and accuracy. Peripheral updates such as BitFit are more effective for simple tasks -- over 75% above the average measured by the area under the Pareto frontiers on CIFAR-10 and CIFAR-100. In contrast, fine-tuning information-heavy layers, such as attention layers via Compacter, achieves a better Pareto frontier on more complex tasks -- 57.5% and 34.6% above the average on Caltech-256 and CUB-200, respectively. Lastly, we observe that the robustness of fine-tuning against out-of-distribution data closely tracks accuracy. These insights emphasize the need for robustness-aware fine-tuning to ensure reliable real-world deployments.
| Comments: | Accepted to International Conference on Computer Vision, ICCV 2025 |
| Subjects: | Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2503.14836 [cs.LG] |
| (or arXiv:2503.14836v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2503.14836 arXiv-issued DOI via DataCite |
Submission history
From: Kunyang Li [view email]
[v1]
Wed, 19 Mar 2025 02:35:01 UTC (670 KB)
[v2]
Mon, 14 Jul 2025 17:14:45 UTC (463 KB)
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.