With some hacking, we can start attempting to build hypre with CUDA
support (private build, or perhaps a separate hypre package in contrib).

Brief notes:

- The version of nvcc (nvidia-cuda-toolkit) in debian non-free is 8.0.61-3
nvcc does not yet support gcc 7, and nvcc 8 does not support gcc 6
To also use mpiCC (so mpi.h is known, etc), need OMPI_CXX
So need
  CC=nvcc CUDACXX=mpiCC dh_auto_configure -- $(extra_flags) --enable-shared

- nvcc does not recognise dpkg-buildflags flags (-W flags in
particular), so need to set an empty LDFLAGS and friends

- must hack src/config/configure.in where nvcc flags are defined,
replace -fPIC with --compiler-options '-fPIC'

- build with g++-5  fails at src/utilities/gpuMem.c:
$nvcc -O --compiler-options '-fPIC' -DUSE_NVTX -DHYPRE_USE_GPU -DHYPRE_USE_MANAGED  -ccbin=mpiCC -expt-extended-lambda -arch compute_35 --std=c++11 -Xcompiler -fopenmp -Xcompiler   -DHYPRE_USE_CUDA  -DUSE_NVTX -DHYPRE_USE_GPU  -DHYPRE_USE_MANAGED -I /usr/local/cuda/include  -DHAVE_CONFIG_H -I.. -I./.. -I.    -c gpuMem.c
  gpuMem.c: In function ‘getstreamOlde’:
  gpuMem.c:335:23: error: storage size of ‘s’ isn’t constant
     static cudaStream_t s[MAXSTREAMS];
                         ^
  gpuMem.c: In function ‘getdomain’:
  gpuMem.c:351:31: error: storage size of ‘h’ isn’t constant
       static nvtxDomainHandle_t h[MAXDOMAINS];
                                 ^
  gpuMem.c: In function ‘getevent’:
  gpuMem.c:364:22: error: storage size of ‘s’ isn’t constant
     static cudaEvent_t s[MAXEVENTS];


This gpuMem.c does compile cleanly under OMPI_CXX=clang++-3.8
Pass through using export OMPI_CXX=clang++-3.8 in debian/rules.

Finally then, using
  export OMPI_CXX=clang++-3.8
  CC=nvcc CUDACXX=mpiCC dh_auto_configure -- $(extra_flags) --with-cuda --enable-unified-memory --with-nvcc --enable-shared

the libHYPRE_utilities.so build proceeds but fails at link time
providing the soname,
  nvcc -shared -o libHYPRE_utilities-2.13.0.so amg_linklist.o binsearch.o exchange_data.o fortran_matrix.o hypre_ap.o hypre_complex.o hypre_memory.o hypre_printf.o hypre_qsort.o memory_dmalloc.o mpistubs.o qsplit.o random.o threading.o timer.o timing.o umalloc_local.o hypre_mpi_comm_f2c.o hypre_error.o F90_HYPRE_error.o hypre_prefix_sum.o hypre_merge_sort.o hypre_hopscotch_hash.o gpuErrorCheck.o gpuMem.o -lmpi -lm -Wl,-soname,libHYPRE_utilities-2.13.0.so -Wl,-z,defs
  nvcc fatal   : Unknown option 'Wl,-soname,libHYPRE_utilities-2.13.0.so'

Again, the problem is that -W is not recognised by nvcc.  Would have
to hack the soname patch to encapsulate it inside --compiler-options

Is it worth it?

Can get further using nvcc --linker-options to carry -Wl options.

But "CC=mpiCC CUDACXX=mpiCC dh_auto_configure" seems better than
"CC=nvcc", together with export OMPI_CXX=g++-5 to set mpiCC.

With other sundry hacks, gets to src/struct_mv and stalls at
assumed_part.c, apparently because of inconsistencies in C/C++
standards between nvcc and CC.Depending on whether g++-5 or clang++-3.8
is used, whether -std=c++11 or c++14 or c++98 is passed to gcc, get
either

 - conflict in multiple definitions of isnan, conflicting between
CUDA's /usr/include/math_functions.h and /usr/include/x86_64-linux-
gnu/bits/mathcalls.h

 - or similar multiple definitions in abs(float)

 - or  /usr/include/cuda_fp16.h:985:68: error: C does not support default arguments
__CUDA_FP16_DECL__ __half2 __shfl(__half2 var, int delta, int width=warpSize);

- or inconsistent C/C++ definitions
    ./_hypre_struct_mv.h:1464:8: error: expected identifier or '('
    extern "C++" {
           ^
    ./_hypre_struct_mv.h:1488:22: error: unexpected OpenMP directive '#pragma omp parallel for'
    #pragma omp parallel for schedule(static)

Following up this bug for GPU support via HIP (ROCm) since CUDA is non-free.

To be activated through -DHYPRE_ENABLE_HIP=ON and related options.

But it needs the GPU to be specified for a general build.
(i.e. when not building for a local GPU).

Then this is a problem:

The problem with support for alternative GPUs raised in the previous
message can be worked around by configuring the build for just one GPU
architecture, e.g. -DCMAKE_HIP_ARCHITECTURES=gfx1201 alone.

libHYPRE.so then builds successfully, and test programs (e.g. ij) build.

But running tests then fails, if the GPU in question is not actually
present, which would be the case for instance on debian build daemons.

$ ./ij
HIP ERROR (code = 100, no ROCm-capable device is detected) at /build/hypre/src/utilities/general.c:307
[cirdan:452875] *** Process received signal ***
[cirdan:452875] Signal: Floating point exception (8)
[cirdan:452875] Signal code: Integer divide-by-zero (1)
[cirdan:452875] Failing at address: 0x7f6144a7aca8
[cirdan:452875] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x406b0) [0x7f60206926b0]
[cirdan:452875] [ 1] /build/hypre/obj-x86_64-linux-gnu/lib/libHYPRE.so.302(hypre_bind_device_id+0x68) [0x7f6144a7aca8]
[cirdan:452875] [ 2] ./ij(+0xf388) [0x55a247f5d388]
[cirdan:452875] [ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x29fba) [0x7f602067bfba]
[cirdan:452875] [ 4] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x87) [0x7f602067c0d7]
[cirdan:452875] [ 5] ./ij(+0xe541) [0x55a247f5c541]
[cirdan:452875] *** End of error message ***
Floating point exception   ./ij

When inspecting available devices, hipGetDeviceCount returns
hipErrorNoDevice if it doesn't find a device.
https://rocmdocs.amd.com/projects/HIP/en/latest/reference/hip_runtime_api/modules/device_management.html#_CPPv417hipGetDeviceCountPi

And hypre treats any response other than hipSuccess as a fatal error.
So it does not fall back to a non-HIP execution path.

The issue is raised at https://github.com/hypre-space/hypre/issues/1624

This makes it difficult to manage the GPU builds, if a separate
libHYPRE.so is needed for each one. It also means build-time testing
would not be possible (but that's effectively the case anyway,
building for a GPU without that GPU being present).

A general observation is that the infrastructure for GPU usage is not
fully mature yet, which impedes general package builds. For comparison
capabilities of CPUs within a given architecture family can be
inspected at runtime e.g. checking for avx512 availability.
simd (xsimd) usage uses this runtime capability.
The GPU situation is more like multiple families of CPU architectures,
amd64 vs arm64 vs ppc64le etc, which means building for all GPUs
separately gives an N×M profilation of library binary files.

cmake itself allows for multiple GPU architectures listed in
CMAKE_HIP_ARCHITECTURES, so possibly hypre is making it more
complicated than other libraries by allowing only one GPU arch at a
time.