RUNMON-RIFT: Adaptive configuration and healing for large-scale parameter inference

Gravitational wave parameter inference pipelines operate on data containing unknown sources, and run on distributed hardware with widely varying configurations and stochastic transient errors. For one specific analysis pipeline (RIFT), we have developed a flexible tool (RUNMON-RIFT) to mitigate the most common challenges introduced by uncertainties in source parameters and computational hardware. On the one hand, RUNMON provides mechanisms to adjust pipeline-specific run settings, including prior ranges, to ensure the analysis completes and encompasses the physical source parameters. On the other, it provides tools for identifying and adjusting to the realities of hardware uncertainties. We demonstrate both general features with controlled examples.