Model-Agnostic Uncertainty Calibration for Noisy Constraint Modeling in Bainitic Steel Optimization

Real-world optimization problems often involve complex physical or industrial process constraints that must be met to ensure feasible solutions. Constrained Bayesian Optimization (CBO) provides an efficient framework for such optimization tasks that relies on uncertainty-aware surrogate models to approximate both the objective function and the constraints. Existing approaches for modeling the constraints often assume homoscedastic Gaussian noise and utilize Gaussian processes (GPs), which can lead to miscalibrated uncertainty estimates, particularly in the presence of model mismatch. To address these challenges, we propose a novel framework that improves both the calibration and adaptivity of constraint modeling. First, we use conformal prediction (CP) to construct prediction intervals with guaranteed calibration, independent of the noise distribution. Next, we improve local adaptivity by modeling residuals with a nonparametric kernel density estimator, enabling the intervals to adjust dynamically to heteroscedastic and non-Gaussian noise. Finally, we use a distance-based uncertainty heuristic to detect distribution shifts, improving robustness in regions with sparse or out-of-distribution samples. Our framework is model-agnostic and can be applied to any prediction model. We validate our method on a synthetic CBO task and on real-world data by modeling constraints for carbite-free bainitic steel optimization [1]. Our results show that standard GP models often produce miscalibrated uncertainty estimates, while our method yields improved calibration, and reduced prediction intervals.