Native HYPRE linear solver

With VertexCFD_ENABLE_HYPRE=ON, the top-level Linear Solver list can select Hypre as its Stratimikos solver. HYPREDRV then owns both the Krylov solver and its preconditioner. Existing Belos, Teko and HYPRE preconditioner configurations remain available.

Build requirements

Use a compatible MPI/compiler stack for Trilinos, HYPRE, HYPREDRV and Vertex-CFD. The CI dependency script builds HYPRE v3.2.0 and HYPREDRV v0.3.0; the CMake minimum for HYPREDRV is 0.3.0. HYPRE must use mixed integers: HYPRE_ENABLE_MIXEDINT=ON and HYPRE_ENABLE_BIGINT=OFF, giving 64-bit HYPRE_BigInt global indices and 32-bit HYPRE_Int local indices. Vertex-CFD rejects incompatible integer types during configuration.

For example, after loading the Trilinos/MPI environment, build the CPU dependencies and configure Vertex-CFD from its source directory:

deps_dir=$(mktemp -d /tmp/vertex-hypre-deps-XXXXXXXX)
source scripts/ci/build-hypre-deps.sh
CI_BUILD_JOBS=$(nproc) build_hypre_deps cpu "$deps_dir"
build_dir=$(mktemp -d /tmp/vertex-hypre-cpu-build-XXXXXXXX)
cmake -S . -B "$build_dir" \
    -DTrilinos_ROOT=/path/to/Trilinos-16.2 \
    -DVertexCFD_ENABLE_HYPRE=ON \
    -DVertexCFD_ENABLE_TESTING=ON \
    -DHYPRE_ROOT="$HYPRE_ROOT" -DHYPRE_DIR="$HYPRE_DIR" \
    -DHYPREDRV_ROOT="$HYPREDRV_ROOT" -DHYPREDRV_DIR="$HYPREDRV_DIR"
cmake --build "$build_dir" --parallel "$(nproc)"

Keep required dependency installations in persistent storage when they must outlive the temporary build. On Frontier, execute builds and tests on an allocated compute node and use an accessible CPU Trilinos installation for CPU testing. CPU results do not validate CUDA compiler diagnostics.

Input configuration

Set Preconditioner Type to None because HYPREDRV creates its own preconditioner. Place the native solver options under Linear Solver Types and Hypre within the top-level Linear Solver list:

<ParameterList name="Linear Solver">
  <Parameter name="Linear Solver Type" type="string" value="Hypre"/>
  <Parameter name="Preconditioner Type" type="string" value="None"/>
  <ParameterList name="Linear Solver Types">
    <ParameterList name="Hypre">
      <Parameter name="HypreDrive YAML File" type="string" value="hypre_solver.yml"/>
      <Parameter name="Number of DOFs" type="int" value="1"/>
    </ParameterList>
  </ParameterList>
</ParameterList>

HypreDrive YAML File is required in native solver mode. Relative paths are resolved from the executable’s working directory. For a scalar diffusion problem, this example selects FGMRES with BoomerAMG:

general:
    statistics: off
solver:
    fgmres:
        max_iter: 200
        relative_tol: 1.0e-12
preconditioner:
    amg:
        print_level: 0

The optional string HypreDrive Preconditioner Preset overrides the preconditioner specified in YAML with a preset supported by the installed HYPREDRV. Number of DOFs defaults to 1, must be positive, and must evenly divide every rank’s local row count. For multiple fields, the matrix must use interleaved DOF ordering matching that value.

Convergence and diagnostics

YAML selects the Krylov algorithm, stopping tolerance, maximum iterations and any scaling. NOX/Thyra convergence requests do not change these YAML settings. If an explicit request is supplied, Vertex-CFD checks the returned solution’s true residual against it; choose a sufficiently strict YAML tolerance for the nonlinear solve.

Only forward solves are supported. Supported Thyra measures are the default criterion and the residual 2-norm relative to the RHS or initial residual 2-norm. Each RHS in a multivector is solved separately; the reported achieved tolerance is the largest relative residual. The iteration count and timings are the latest values reported by HYPREDRV, corresponding to the final RHS. The NOX iteration observer reports these native HYPRE diagnostics.

Solver sessions persist across matrix updates with unchanged configuration. To change the YAML path, preset or DOF count after initialization, create a new solver/factory. Simultaneously live solver operators maintain independent matrix state. Distributed matrices with noncontiguous global IDs are remapped to contiguous HYPRE numbering without changing local vector ordering.

Validation

The native solver unit tests cover multiple RHSs, zero RHSs, true residual criteria, nonconvergence, persistent-session reuse, independent operators, noncontiguous/64-bit global IDs and invalid configurations:

ctest --test-dir "$build_dir" --output-on-failure \
    -R 'VertexCFD_(HypreLinearOpWithSolve|HypreLinearOpWithSolveFactory|HypreUtils|HypreDriveLoopAnnotation)_test'

The push regression TestVariableFlux::test_2d_heated_flow runs with both Belos and native HYPRE and checks the same physical wall-temperature values through Tempus and NOX. The HYPRE case is skipped automatically in builds without HYPRE support.