I'm trying to activate GPU support in PETSc (Bug#1082816),
which PETsc does via kokkos.

PETSc is getting a build error.  I'm not sure at this point if the
error is due to the new petsc configuration that needs to be added, or
due to some configuration error in kokkos. So I'm opening this bug for
further discussion.

I've added
  --with-kokkos=1 --with-kokkos-include=/usr/include/kokkos --with-kokkos-lib="-lkokkoscore"
to the petsc configuration in its debian/rules.

But the petsc build then fails, with error:

mpicxx -c -g -O2 -ffile-prefix-map=/projects/petsc/build/petsc=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fPIC -std=c++20 -fPIC -I/pr
ojects/petsc/build/petsc/include -I/projects/petsc/build/petsc/x86_64-linux-gnu-real-debug/include -I/usr/include/hypre -I/usr/include/suitesparse -I/usr/include/superlu-dist -I/usr/include/superlu -I/u
sr/include/kokkos -I/usr/include/scotch -I/usr/include/hdf5/openmpi -MMD -MP /projects/petsc/build/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o x86_64-linux-gnu-real-debug/obj/src/sys/objects/kokkos
/kinit.o
In file included from /usr/include/kokkos/OpenMP/Kokkos_OpenMP.hpp:168,
                 from /usr/include/kokkos/decl/Kokkos_Declare_OPENMP.hpp:8,
                 from /usr/include/kokkos/KokkosCore_Config_DeclareBackend.hpp:7,
                 from /usr/include/kokkos/Kokkos_Core.hpp:32,
                 from /projects/petsc/build/petsc/include/petsc/private/kokkosimpl.hpp:6,
                 from /projects/petsc/build/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx:2:
/usr/include/kokkos/OpenMP/Kokkos_OpenMP_Instance.hpp:10:2: error: #error "You enabled Kokkos OpenMP support without enabling OpenMP in the compiler!"
   10 | #error \
      |  ^~~~~
make[5]: *** [gmakefile:208: x86_64-linux-gnu-real-debug/obj/src/sys/objects/kokkos/kinit.o] Error 1

I'm testing against libkokkos-dev 5.2.2-1 from experimental.
Given the error "without enabling OpenMP" I first assumed that kokkos
simply hadn't been built with openmp support, but that's not the case.
The 5.2.2 kokkos build does use -DKokkos_ENABLE_OPENMP=ON

So I'm confused why the PETSc build is triggering this error in Kokkos.
The failing condition at l.10 in Kokkos_OpenMP_Instance.hpp is

#include <Kokkos_Macros.hpp>
#if !defined(_OPENMP) && !defined(__CUDA_ARCH__) && \
    !defined(__HIP_DEVICE_COMPILE__) && !defined(__SYCL_DEVICE_ONLY__)
#error \
    "You enabled Kokkos OpenMP support without enabling OpenMP in the compiler!"
#endif

but it's not obvious to me who (petsc or kokkos?) should be setting
these symbols, and where. If kokkos was already built with
-DKokkos_ENABLE_OPENMP=ON, then I'd expect it would already have
_OPENMP defined, at least.

(For context, I'm ultimately interested in activating HIP support for
the GPUs. Not sure if activating HIP in kokkos would set
__HIP_DEVICE_COMPILE__, but nevertheless it's unexpected to get this error about
openmp)

It might be that I misunderstand how petsc is intended to activate
kokkos support.  Is KOKKOS_ENABLE_OPENMP (or _OPENMP, and/or KOKKOS_ENABLE_HIP)
supposed to be set by petsc when building against kokkos, or should
these symbols have already been defined when kokkos was built?

It looks like kokkos is expecting applications to compile using the
-fopenmp flag to avoid this error.

It can be fixed in the petsc configuration by including
--with-openmp=1  alongside  --with-kokkos=1

One might ask why petsc is not automatically adding openmp if kokkos
needs it.  I guess the answer is that kokkos does not "need" it.
But since we've explicitly configured kokkos with openmp support, then
petsc also needs to be likewise configured with openmp if it wants to
use kokkos.

One might then ask why petsc does not inspect the kokkos configuration
to automatically determine whether it needs openmp or not. My guess
for this is that the PETSc config system just checks direct package
availability, which doesn't extend to indirect dependencies of the
kind that we have here. PETSc doesn't use cmake.

Looks like petsc: --with-openmp=1 is probably enough for resolving
this bug, but I'll leave open a little longer to review the situation,
including interaction with the HIP configurations.