Topology-Aware Communication Optimization for CFD Simulations in OpenFOAM

Parallel computational fluid dynamics (CFD) simulations are commonly executed on modern high-performance computing systems with hardware topologies, where communication costs vary significantly across memory hierarchies and network links. OpenFOAM, a widely used open-source CFD framework, provides a scalable MPI-based parallel infrastructure. However, its default process placement and communication strategies are largely topology-agnostic, which can lead to suboptimal performance on NUMA-based cluster architectures. We present a topology-aware communication optimization for OpenFOAM that targets both local and global communication in iterative linear solvers. Local communication is optimized by reassigning MPI ranks to physical cores based on the communication structure induced by the domain decomposition and the underlying hardware topology. In addition, a custom global reduction schedule is introduced that aligns collective communication with the hierarchical organization of compute nodes, NUMA domains, and NUMA nodes. Our experimental results show a substantial redistribution of communication toward faster hardware paths, leading to better runtime with the largest improvements observed for communication-intensive solver configurations. These findings demonstrate that topology-aware communication strategies can significantly enhance the performance of OpenFOAM on modern HPC systems.