[flang][openmp] real(10) prevents compilation even when it is not used in device code

[VeeEM]

At least for amdgpu, code containing real(10) can no longer be compiled with offloading enabled after 2bfb3ba.

Compiling a program containing a real(10) variable used to print a warning, now it's an error:

program prog1
real(10) :: x
end program

$ flang-new -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa --offload-arch=gfx90a prog1.f90
error: Semantic errors in prog1.f90
./prog1.f90:2:1: error: REAL(KIND=10) is not an enabled type for this target
  real(10) :: x
  ^^^^^^^^^^^^^

Using a module containing real(10) is also an error now, previously this did not even print a warning:

module m
contains
subroutine f(x)
real(10) :: x
end subroutine
end module

program prog2
use m
end program

$ flang-new -c m.f90
$ flang-new -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa --offload-arch=gfx90a prog2.f90
error: Semantic errors in prog2.f90
./m.mod:5:1: error: REAL(KIND=10) is not an enabled type for this target
  real(10)::x
  ^^^^^^^^^^^

I'm thinking real(10) should probably only be an error if used on device.

[llvmbot]

@llvm/issue-subscribers-flang-frontend

Author: Ville-Markus Yli-Suutala (VeeEM)

At least for amdgpu, code containing real(10) can no longer be compiled with offloading enabled after 2bfb3ba.

Compiling a program containing a real(10) variable used to print a warning, now it's an error:

program prog1
real(10) :: x
end program

$ flang-new -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa --offload-arch=gfx90a prog1.f90
error: Semantic errors in prog1.f90
./prog1.f90:2:1: error: REAL(KIND=10) is not an enabled type for this target
  real(10) :: x
  ^^^^^^^^^^^^^

Using a module containing real(10) is also an error now, previously this did not even print a warning:

module m
contains
subroutine f(x)
real(10) :: x
end subroutine
end module

program prog2
use m
end program

$ flang-new -c m.f90
$ flang-new -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa --offload-arch=gfx90a prog2.f90
error: Semantic errors in prog2.f90
./m.mod:5:1: error: REAL(KIND=10) is not an enabled type for this target
  real(10)::x
  ^^^^^^^^^^^

I'm thinking real(10) should probably only be an error if used on device.

[eugeneepshteyn]

I'm thinking real(10) should probably only be an error if used on device.

I think this is what's happening.

I don't have access to AMG GPU machine, but I tried the following on a host with NVIDIA GPU:

$ flang -c -fopenmp -fopenmp-targets=nvptx64-nvidia-cuda real10.f90
flang-21: warning: OpenMP support in flang is still experimental [-Wexperimental-option]
flang-21: warning: OpenMP support in flang is still experimental [-Wexperimental-option]
flang-21: warning: OpenMP support in flang is still experimental [-Wexperimental-option]
error: Semantic errors in real10.f90
./real10.f90:2:1: error: REAL(KIND=10) is not an enabled type for this target
  real(10) :: x
  ^^^^^^^^^^^^^

The warning is outputted three times, which tells me that flang is called three times with different args to produce the GPU offload code. Let's see what is actually called:

$ flang -### -c -fopenmp -fopenmp-targets=nvptx64-nvidia-cuda real10.f90
flang version 21.0.0git (https://github.com/llvm/llvm-project 6e7da07c73c179396e21fb729ac14d6b2a1c3152)
Target: x86_64-unknown-linux-gnu
Thread model: posix
...
Build config: +assertions
flang-21: warning: OpenMP support in flang is still experimental [-Wexperimental-option]
flang-21: warning: OpenMP support in flang is still experimental [-Wexperimental-option]
flang-21: warning: OpenMP support in flang is still experimental [-Wexperimental-option]
 "flang" "-fc1" "-triple" "x86_64-unknown-linux-gnu" "-emit-llvm-bc" "-fcolor-diagnostics" "-mrelocation-model" "pic" "-pic-level" "2" "-pic-is-pie" "-target-cpu" "x86-64" "-fopenmp" "-resource-dir" ".../lib/clang/21" "-fopenmp-targets=nvptx64-nvidia-cuda" "-mframe-pointer=all" "-o" "/tmp/real10-71d5c6.bc" "-x" "f95-cpp-input" "real10.f90"
 "flang" "-fc1" "-triple" "nvptx64-nvidia-cuda" "-S" "-fcolor-diagnostics" "-mrelocation-model" "pic" "-pic-level" "2" "-target-cpu" "sm_80" "-fopenmp" "-resource-dir" ".../lib/clang/21" "-fopenmp-host-ir-file-path" "/tmp/real10-71d5c6.bc" "-fopenmp-is-target-device" "-mframe-pointer=all" "-o" "/tmp/real10-sm_80-133515.s" "-x" "f95-cpp-input" "real10.f90"
 "/usr/local/cuda/bin/ptxas" "-m64" "-O0" "--gpu-name" "sm_80" "--output-file" "/tmp/real10-sm_80-fa9762.o" "/tmp/real10-sm_80-133515.s" "-c"
 ".../bin/clang-offload-packager" "-o" "/tmp/real10-69103a.out" "--image=file=/tmp/real10-sm_80-fa9762.o,triple=nvptx64-nvidia-cuda,arch=sm_80,kind=openmp"
 "flang" "-fc1" "-triple" "x86_64-unknown-linux-gnu" "-emit-obj" "-fcolor-diagnostics" "-mrelocation-model" "pic" "-pic-level" "2" "-pic-is-pie" "-target-cpu" "x86-64" "-fopenmp" "-resource-dir" ".../lib/clang/21" "-fembed-offload-object=/tmp/real10-69103a.out" "-fopenmp-targets=nvptx64-nvidia-cuda" "-mframe-pointer=all" "-o" "real10.o" "-x" "ir" "/tmp/real10-71d5c6.bc"

There are several invocations of flang, the first one is for the host, the second one is for the GPU. Explicitly calling flang for the host results in no errors:

$ "flang" "-fc1" "-triple" "x86_64-unknown-linux-gnu" "-emit-llvm-bc" "-fcolor-diagnostics" "-mrelocation-model" "pic" "-pic-level" "2" "-pic-is-pie" "-target-cpu" "x86-64" "-fopenmp" "-resource-dir" ".../lib/clang/21" "-fopenmp-targets=nvptx64-nvidia-cuda" "-mframe-pointer=all" "-o" "/tmp/real10-71d5c6.bc" "-x" "f95-cpp-input" "real10.f90"

Explicitly calling flang for GPU offload results in the error:

$ "flang" "-fc1" "-triple" "nvptx64-nvidia-cuda" "-S" "-fcolor-diagnostics" "-mrelocation-model" "pic" "-pic-level" "2" "-target-cpu" "sm_80" "-fopenmp" "-resource-dir" ".../lib/clang/21" "-fopenmp-host-ir-file-path" "/tmp/real10-71d5c6.bc" "-fopenmp-is-target-device" "-mframe-pointer=all" "-o" "/tmp/real10-sm_80-133515.s" "-x" "f95-cpp-input" "real10.f90"
error: Semantic errors in real10.f90
./real10.f90:2:1: error: REAL(KIND=10) is not an enabled type for this target
  real(10) :: x
  ^^^^^^^^^^^^^

I think flang works exactly as it's supposed to.

Here's the flang revision I'm using:

$ flang --version
flang version 21.0.0git (https://github.com/llvm/llvm-project 6e7da07c73c179396e21fb729ac14d6b2a1c3152)
Target: x86_64-unknown-linux-gnu
Thread model: posix
...

[eugeneepshteyn]

I was going to suggest that you use the following to select between REAL kinds available on the target platform:

program prog1
use, intrinsic :: iso_fortran_env, only: real64, real80

integer, parameter :: rk = merge(real80, real64, real80 > 0)

real(kind=rk) :: x
print *, "Use real kind", rk

end program

I expected that real80 should be negative for platforms that don't support REAL(10). Unfortunately, I'm getting the same error about REAL(10) on GPU target, so it seems that real80 still resolves to 10 on GPU targets. This, I think is a bug.

real80 is defined in terms of selectedReal80:

selectedReal80 = selected_real_kind(18, 4931)
safeReal80 = merge(selectedReal80, selected_real_kind(0,0), selectedReal80 >= 0)
real80 = merge(selectedReal80, merge(-2, -1, selectedReal80 >= 0), digits(real(0,kind=safeReal80)) == 64)

I think selectedReal80 should return negative value for platforms that support reals that are narrower than 80 bits.

[VeeEM]

The error does come from the invocation for the GPU but I don't think it should be an error in this case, as no real(10) code is actually being generated for the GPU.

If I revert 2bfb3ba and use llvm-dis to translate the two bitcode files being generated, I get the following for the host side:

; ModuleID = '/tmp/prog1-b85984.bc'
source_filename = "FIRModule"
target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"

define void @_QQmain() #0 {
  %1 = alloca x86_fp80, i64 1, align 16
  ret void
}

declare void @_FortranAProgramStart(i32, ptr, ptr, ptr)

declare void @_FortranAProgramEndStatement()

define i32 @main(i32 %0, ptr %1, ptr %2) #0 {
  call void @_FortranAProgramStart(i32 %0, ptr %1, ptr %2, ptr null)
  call void @_QQmain()
  call void @_FortranAProgramEndStatement()
  ret i32 0
}

attributes #0 = { "frame-pointer"="all" "target-cpu"="x86-64" }

!llvm.module.flags = !{!0, !1, !2, !3}
!llvm.ident = !{!4}

!0 = !{i32 2, !"Debug Info Version", i32 3}
!1 = !{i32 7, !"openmp", i32 11}
!2 = !{i32 8, !"PIC Level", i32 2}
!3 = !{i32 7, !"PIE Level", i32 2}
!4 = !{!"flang version 21.0.0 (https://github.com/llvm/llvm-project.git 86d623b07276ce57dd028bd166f04134017419e1)"}

And for the device:

; ModuleID = '/tmp/prog1-gfx90a-ffe706.bc'
source_filename = "FIRModule"
target datalayout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128-p9:192:256:256:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9"
target triple = "amdgcn-amd-amdhsa"

@__omp_rtl_debug_kind = weak_odr hidden addrspace(1) constant i32 0
@__omp_rtl_assume_teams_oversubscription = weak_odr hidden addrspace(1) constant i32 0
@__omp_rtl_assume_threads_oversubscription = weak_odr hidden addrspace(1) constant i32 0
@__omp_rtl_assume_no_thread_state = weak_odr hidden addrspace(1) constant i32 0
@__omp_rtl_assume_no_nested_parallelism = weak_odr hidden addrspace(1) constant i32 0
@__oclc_ABI_version = weak_odr hidden local_unnamed_addr addrspace(4) constant i32 500

!llvm.module.flags = !{!0, !1, !2, !3, !4}
!llvm.ident = !{!5, !6, !6, !6, !6, !6, !6, !6, !6}
!opencl.ocl.version = !{!7, !7, !7, !7, !7, !7, !7, !7}

!0 = !{i32 2, !"Debug Info Version", i32 3}
!1 = !{i32 7, !"openmp-device", i32 11}
!2 = !{i32 7, !"openmp", i32 11}
!3 = !{i32 8, !"PIC Level", i32 2}
!4 = !{i32 1, !"wchar_size", i32 4}
!5 = !{!"flang version 21.0.0 (https://github.com/llvm/llvm-project.git 86d623b07276ce57dd028bd166f04134017419e1)"}
!6 = !{!"clang version 21.0.0git (https://github.com/llvm/llvm-project.git 86d623b07276ce57dd028bd166f04134017419e1)"}
!7 = !{i32 2, i32 0}

There is no fp80 stuff generated for the device and this program does successfully compile.

If I modify the program and add a target region where I use the real(10)

program prog3
real(10) :: x
x = 2
!$omp target map(x)
x = x*x
!$omp end target
print *, x
end program

, then fp80 code will be generated for the device:

; ModuleID = '/tmp/prog3-gfx90a-9e6f45.bc'
source_filename = "FIRModule"
target datalayout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128-p9:192:256:256:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9"
target triple = "amdgcn-amd-amdhsa"

%struct.ident_t = type { i32, i32, i32, i32, ptr }
%struct.DynamicEnvironmentTy = type { i16 }
%struct.KernelEnvironmentTy = type { %struct.ConfigurationEnvironmentTy, ptr, ptr }
%struct.ConfigurationEnvironmentTy = type { i8, i8, i8, i32, i32, i32, i32, i32, i32 }

$_QQclXbbdc7fc8a8efd639b567777c8d17ff9f = comdat any

@__omp_rtl_debug_kind = weak_odr hidden addrspace(1) constant i32 0
@__omp_rtl_assume_teams_oversubscription = weak_odr hidden addrspace(1) constant i32 0
@__omp_rtl_assume_threads_oversubscription = weak_odr hidden addrspace(1) constant i32 0
@__omp_rtl_assume_no_thread_state = weak_odr hidden addrspace(1) constant i32 0
@__omp_rtl_assume_no_nested_parallelism = weak_odr hidden addrspace(1) constant i32 0
@_QQclXbbdc7fc8a8efd639b567777c8d17ff9f = linkonce constant [45 x i8] c"/home/acc/Documents/Fortran/real80/prog3.f90\00", comdat
@0 = private unnamed_addr constant [55 x i8] c";/home/acc/Documents/Fortran/real80/prog3.f90;x;2;13;;\00", align 1
@__omp_offloading_10304_1a41110__QQmain_l4_exec_mode = weak protected addrspace(1) constant i8 1
@llvm.compiler.used = appending addrspace(1) global [1 x ptr] [ptr addrspacecast (ptr addrspace(1) @__omp_offloading_10304_1a41110__QQmain_l4_exec_mode to ptr)], section "llvm.metadata"
@1 = private unnamed_addr constant [23 x i8] c";unknown;unknown;0;0;;\00", align 1
@2 = private unnamed_addr addrspace(1) constant %struct.ident_t { i32 0, i32 2, i32 0, i32 22, ptr @1 }, align 8
@__omp_offloading_10304_1a41110__QQmain_l4_dynamic_environment = weak_odr protected addrspace(1) global %struct.DynamicEnvironmentTy zeroinitializer
@__omp_offloading_10304_1a41110__QQmain_l4_kernel_environment = weak_odr protected addrspace(1) constant %struct.KernelEnvironmentTy { %struct.ConfigurationEnvironmentTy { i8 1, i8 1, i8 1, i32 1, i32 256, i32 -1, i32 -1, i32 0, i32 0 }, ptr addrspacecast (ptr addrspace(1) @2 to ptr), ptr addrspacecast (ptr addrspace(1) @__omp_offloading_10304_1a41110__QQmain_l4_dynamic_environment to ptr) }
@__oclc_ABI_version = weak_odr hidden local_unnamed_addr addrspace(4) constant i32 500

; Function Attrs: mustprogress
define weak_odr protected amdgpu_kernel void @__omp_offloading_10304_1a41110__QQmain_l4(ptr %0, ptr %1) #0 {
entry:
  %2 = alloca ptr, align 8, addrspace(5)
  %3 = addrspacecast ptr addrspace(5) %2 to ptr
  store ptr %1, ptr %3, align 8
  %4 = call i32 @__kmpc_target_init(ptr addrspacecast (ptr addrspace(1) @__omp_offloading_10304_1a41110__QQmain_l4_kernel_environment to ptr), ptr %0)
  %exec_user_code = icmp eq i32 %4, -1
  br i1 %exec_user_code, label %user_code.entry, label %worker.exit

user_code.entry:                                  ; preds = %entry
  %5 = load ptr, ptr %3, align 8
  br label %omp.region.after_alloca

omp.region.after_alloca:                          ; preds = %user_code.entry
  br label %outlined.body

outlined.body:                                    ; preds = %omp.region.after_alloca
  br label %omp.target

omp.target:                                       ; preds = %outlined.body
  %6 = load x86_fp80, ptr %5, align 16
  %7 = fmul contract x86_fp80 %6, %6
  store x86_fp80 %7, ptr %5, align 16
  br label %omp.region.cont

omp.region.cont:                                  ; preds = %omp.target
  call void @__kmpc_target_deinit()
  ret void

worker.exit:                                      ; preds = %entry
  ret void
}

declare i32 @__kmpc_target_init(ptr, ptr)

declare void @__kmpc_target_deinit()

attributes #0 = { mustprogress "amdgpu-flat-work-group-size"="1,256" "kernel" "omp_target_thread_limit"="256" "target-cpu"="gfx90a" "target-features"="+16-bit-insts,+atomic-buffer-global-pk-add-f16-insts,+atomic-fadd-rtn-insts,+ci-insts,+dl-insts,+dot1-insts,+dot10-insts,+dot2-insts,+dot3-insts,+dot4-insts,+dot5-insts,+dot6-insts,+dot7-insts,+dpp,+gfx8-insts,+gfx9-insts,+gfx90a-insts,+gws,+image-insts,+mai-insts,+s-memrealtime,+s-memtime-inst,+wavefrontsize64" "uniform-work-group-size"="true" }

!llvm.module.flags = !{!0, !1, !2, !3, !4}
!llvm.ident = !{!5, !6, !6, !6, !6, !6, !6, !6, !6}
!omp_offload.info = !{!7}
!opencl.ocl.version = !{!8, !8, !8, !8, !8, !8, !8, !8}

!0 = !{i32 2, !"Debug Info Version", i32 3}
!1 = !{i32 7, !"openmp-device", i32 11}
!2 = !{i32 7, !"openmp", i32 11}
!3 = !{i32 8, !"PIC Level", i32 2}
!4 = !{i32 1, !"wchar_size", i32 4}
!5 = !{!"flang version 21.0.0 (https://github.com/llvm/llvm-project.git 86d623b07276ce57dd028bd166f04134017419e1)"}
!6 = !{!"clang version 21.0.0git (https://github.com/llvm/llvm-project.git 86d623b07276ce57dd028bd166f04134017419e1)"}
!7 = !{i32 0, i32 66308, i32 27529488, !"_QQmain", i32 4, i32 0, i32 0}
!8 = !{i32 2, i32 0}

prog3 does lead to a compiler error if we try to compile it all the way to an executable and this is a case where I think a nice error earlier is appropriate. But I think it would be right to allow using real(10) on the x86 host that supports them as long as it's not being used on the GPU.

I'm personally not that interested in using real(10) together with offloading. I actually came across this error with a program that uses the hdf5 library and OpenMP offloading in the same module, and hdf5 being compiled with real(10) support was triggering the error through its module files.

I tested calling selected_real_kind in a target region on gfx1100 with this program

program r80

integer :: selectedReal80, safeReal80

!$omp target map(selectedReal80, safeReal80)
selectedReal80 = selected_real_kind(18, 4931)
safeReal80 = merge(selectedReal80, selected_real_kind(0,0), selectedReal80 >= 0)
!$omp end target

print *, "selectedReal80", selectedReal80
print *, "safeReal80", safeReal80
end program

, it prints

 selectedReal80 -3
 safeReal80 2

, so I think that means it is working correctly.

[bcornille]

FYI I have a very trivial reproducer for this.

program real10
  use ieee_arithmetic
  implicit none
end program

>> flang -fopenmp --offload-arch=gfx90a real10.f90
error: Semantic errors in real10.f90
/share/contrib-modules/Core/amd-staging/2025-03-03_21.0-0/lib/llvm/bin/../include/flang/ieee_arithmetic.mod:143:1: error: REAL(KIND=10) is not an enabled type for this target
  real(10),intent(in)::x
  ^^^^^^^^^^^^^^^^^^^^^^
...

This will almost assuredly affect OpenACC as well once it is mature enough to start invoking the device target triple.

[bcornille]

#124655 is the offending PR.

[eugeneepshteyn]

It seems that the intrinsic module files need to be compiled for different targets, so that the REAL kinds not supported on a particular target don't sneak into the compiled module file for targets that don't support them.

[llvmbot]

@llvm/issue-subscribers-flang-runtime

Author: Ville-Markus Yli-Suutala (VeeEM)

At least for amdgpu, code containing real(10) can no longer be compiled with offloading enabled after 2bfb3ba.

Compiling a program containing a real(10) variable used to print a warning, now it's an error:

program prog1
real(10) :: x
end program

$ flang-new -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa --offload-arch=gfx90a prog1.f90
error: Semantic errors in prog1.f90
./prog1.f90:2:1: error: REAL(KIND=10) is not an enabled type for this target
  real(10) :: x
  ^^^^^^^^^^^^^

Using a module containing real(10) is also an error now, previously this did not even print a warning:

module m
contains
subroutine f(x)
real(10) :: x
end subroutine
end module

program prog2
use m
end program

$ flang-new -c m.f90
$ flang-new -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa --offload-arch=gfx90a prog2.f90
error: Semantic errors in prog2.f90
./m.mod:5:1: error: REAL(KIND=10) is not an enabled type for this target
  real(10)::x
  ^^^^^^^^^^^

I'm thinking real(10) should probably only be an error if used on device.