PromptCAL: Entropy-Calibrated and Prompt-Tunes Test Time Adaptation for Semantic Segmentation

Test-time adaptation (TTA) aims to improve the robustness of segmentation models to an unlabeled target domain without requiring access to the source data. While existing TTA methods have achieved promising results on image classification, they often fail to translate effectively to semantic segmentation due to the spatial complexity and fine-grained nature of dense predictions. We propose PromptCAL, a lightweight and effective TTA framework tailored for semantic segmentation, built upon the SegFormer architecture. Our method addresses two central challenges: (1) Which model component to adapt remains underexplored. Using Grad-CAM visualization and sensitivity analysis, we identify Stage 2 of the transformer backbone as the most domain-sensitive and restrict adaptation to this stage. (2) How to identify reliable supervision during adaptation is critical. We introduce a confidence-aware self-training mechanism based on per-pixel entropy filtering to guide pixel selection for model adaptation, ensuring label quality and model transferability. In addition, we incorporate lightweight prompt injection to enhance the adaptability of mid-level features. Our method achieves competitive improvements over the state-of-the-art while maintaining high adaptation efficiency and significantly reducing runtime overhead. Extensive experiments on corrupted semantic segmentation benchmarks, including ACDC (A-fog, A-night, A-rain, and A-snow), Cityscapes-foggy (CS-fog) and Cityscapes-rainy (CS-rain) demonstrate that PromptCAL achieves comparable or superior accuracy to state-of-the-art TTA baselines, while reducing adaptation time by over 50% per domain. This makes it a practical solution for efficient TTA in smart cities and edge-deployed vision systems. The source code is available at https://github.com/ml4papers/PromptCAL.