HR-OTTA: Robust Online Test-Time Adaptation with Hybrid Fine-Tuning

Test-time adaptation (TTA) aims to adapt a model trained on a source domain to an unlabeled target domain without requiring source data. Traditional TTA assumes that the entire target domain data is available for observation during adaptation. In contrast, online test-time adaptation (OTTA) processes target domain data sequentially in mini-batches, enabling real-time model updates as new data arrives. While recent studies have demonstrated the potential of OTTA for various computer vision tasks, several challenges remain. This paper addresses two key issues: (1) While existing methods adapt different model components (e.g., batch normalization (BN) layers, fully connected (FC) layers, and the whole feature extractor), the most effective strategy is unclear. We propose a hybrid fine-tuning approach that modulates only the shallow convolutional (Conv) layers and all BN layers, enabling a more precise and effective adaptation compared to conventional methods. (2) The objective function plays a crucial role in enhancing prediction accuracy. Unlike previous TTA frameworks that focus solely on entropy minimization, we introduce a robust objective function that combines a diversity-promoting regularization term to balance certainty and diversity, along with a confidence-enhanced loss term that prioritizes high-confidence predictions. Our experiments on various corruption datasets, using both CNN and vision transformer (ViT)-based models, demonstrate that our approach outperforms state-of-the-art methods across a wide range of applications. Our implementation is publicly available.