Two-Level Test-Time Adaptation in Multimodal Learning

Test-time adaptation (TTA) aims to adjust the parameters of a pre-trained source model using samples from the target domain, without requiring access to the source data. While recent studies have shown the potential of TTA across various computer vision tasks, most TTA methods are limited to uni-modal adaptation, and the domain shift caused by unimodal data corruption in multimodal tasks is not adequately addressed. Although some recent approaches have reduced cross-modal information discrepancy through modality-sharing modules, the domain adaptation for modality-specific modules has been overlooked. In this paper, we introduce a two-level test-time adaptation method (2LTTA) that accounts for both intra-modal distribution shifts and cross-modal reliability bias in multimodal learning (MML). Unlike conventional TTA methods, which focus primarily on fine-tuning normalization layers, 2LTTA modulates all normalization layers, self-Attention modules of the encoder related to the corrupted modality, and the modality-sharing block. Additionally, we design a two-level objective function that addresses both intra-modal distribution shift and cross-modal reliability bias in the modality fusion block. First, Shannon entropy with sample reweighting is used to mitigate intra-modal distribution shifts caused by data corruption. Second, a diversity-promoting loss is incorporated to reduce cross-modal information discrepancy. Our experiments show that 2LTTA outperforms baseline methods across various datasets.